Skip to content

Volume Shield Troubleshooting

Volume Shield fails closed by design: when integrity, key access, or freshness cannot be verified, reads and writes return EIO and components crash-abort rather than serving unverified or plaintext data. Most "outages" in this page are the system refusing to do something unsafe — the fix is to restore the safety condition, never to bypass it.

Quick health checklist

# 1. Broker DaemonSet present on all nodes
kubectl get ds -n cloudtaser-system -l app.kubernetes.io/component=vs-broker

# 2. Sidecar injected and ready in the workload pod
kubectl get pod <pod> -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[?(@.name=="cloudtaser-vs-sidecar")].ready}'

# 3. Sidecar logs (structured JSON)
kubectl logs <pod> -c cloudtaser-vs-sidecar --tail=50

# 4. Broker logs on the pod's node
NODE=$(kubectl get pod <pod> -o jsonpath='{.spec.nodeName}')
kubectl logs -n cloudtaser-system -l app.kubernetes.io/component=vs-broker \
  --field-selector spec.nodeName=$NODE --tail=50

Health probes are HTTPS: broker on :8081, sidecar on :8082. The sidecar's /readyz returns 200 only when every configured mount is serving — a pod with two encrypted PVCs is not ready until both FUSE servers are up.

Symptom → cause table

Symptom Likely cause Action
Pod stuck Init / sidecar not ready Broker not running on that node, or fd-pass socket not reachable Check broker DaemonSet coverage (taints/nodeSelector); check broker logs for sidecar socket not ready
Sidecar log: fetching DEK from vault error Vault unreachable, role misconfigured, or missing policy on cloudtaser/data/volumes/* Verify vaultAddr, K8s auth role binding, policy; test login with the sidecar SA token
Sidecar log: creating nonce budget / creating trusted freshness error nonceBudget.mode/freshness.mode set to vault but the role lacks cloudtaser/data/volumeshield-state/* Grant read + CAS write/update on the state path (fail-closed by design)
Application gets EIO on read Ciphertext failed authentication: tamper, corruption, or (freshness mode) digest mismatch / tombstone / rollback Treat as an integrity event, not a bug — see below
Application gets EXDEV on directory rename Trusted freshness mode: subtree moves need a multi-file trusted-state transaction (unsupported) Most tools (mv, shutil.move, Java Files.move, Kafka) fall back to copy+delete automatically on EXDEV. Workloads needing ATOMIC directory renames (ClickHouse, Prometheus TSDB, VictoriaMetrics): annotate the PVC cloudtaser.io/vs-freshness: "local" (opts out of rollback detection)
Hard link creation fails with EOPNOTSUPP Trusted freshness mode: per-path freshness state doesn't support two paths sharing one backing file's identity Workloads that hard-link for snapshotting (e.g. Cassandra nodetool snapshot) hit this natively under vault mode. Same opt-out as directory renames: annotate the PVC cloudtaser.io/vs-freshness: "local"
mmap fails with ENODEV PVC is in strict I/O mode Expected — switch to cached if the workload mmaps (reference)
Writes fail after very heavy sustained I/O Nonce budget exhausted (2³² GCM seals, fail-closed per NIST SP 800-38D) Rotate the DEK / re-encrypt; enable nonceBudget.mode: vault so budget is tracked across remounts and warning logs (at 2³¹) arrive early
Files owned by wrong uid/gid on the app side Pod-level fsGroup was set — the webhook strips it because kubelet's recursive chown would corrupt ciphertext metadata Ownership is preserved at the FUSE layer per caller; remove reliance on fsGroup for these volumes
Second PVC's mount empty / dead in an older deployment Multi-PVC pods require newer wrapper (sidecar) and operator versions; older sidecars served only the first PVC Use wrapper v0.2.168 or newer and operator v0.13.153 or newer. Chart v1.0.264 bundles operator v0.13.153; the separate latest standalone operator release is v0.13.155.

Understanding EIO: crash-abort is the contract

An EIO from a Volume Shield mount means authentication failed for the exact bytes on the backing store:

  • Without freshness mode: a chunk failed AES-GCM verification — bit rot, storage corruption, or deliberate tamper (swap/reorder/splice/truncate).
  • With freshness mode: additionally, the ciphertext doesn't match the trusted digest in vault — including valid but old bytes (rollback) and stripped-header plaintext substitution.

Do not attempt to "repair" by disabling freshness or copying raw files around: that converts a detected integrity event into a silent one. Instead:

  1. Snapshot the raw volume for forensics (/var/lib/cloudtaser/raw/<pvc-name> via the broker pod or node).
  2. Determine whether infrastructure corruption or tampering explains the event.
  3. Restore affected files from a trusted backup through the mount (writing through FUSE re-encrypts and, in freshness mode, commits new digests).

Bulk migration and TOFU-window graduation

To encrypt pre-existing plaintext data in place, and — since the 2026-07-06 trusted-freshness default flip (roadmap#197) — to graduate a volume out of its trust-on-first-use (TOFU) window, run the sidecar binary in migrate mode as a Job or init container with the same env/identity the sidecar gets (vault address, role, namespace, PVC UID):

volume-shield-sidecar --migrate            # migrates every configured mount
volume-shield-sidecar --migrate --raw-path /var/lib/cloudtaser/raw/<pvc-name>   # single path

The Job walks the target path and, per file: encrypts remaining plaintext (files already in CTVS format are skipped, so this step alone is idempotent — unchanged from before the default flip), and baselines every encrypted file into vault trusted-freshness state. It prints a JSON result (encrypted / skipped / errors) and exits non-zero if any file failed.

Graduating from the TOFU window

Graduation uses the exact same --migrate invocation above — there is no separate command. What's new since the default flip is what happens after a clean walk:

  • Refusal on errors. The volume's baseline_mode marker flips from the TOFU state to strict only if the walk completed with zero errors. A single unreadable or corrupt file blocks graduation for the whole volume until resolved — the migration never partially-graduates a volume onto a false sense of full coverage.
  • Idempotent, safe to re-run. Files already baselined are skipped on a re-run; only files not yet tracked are baselined. Re-running after fixing the error that blocked a prior attempt is the normal recovery path.
  • Audit trail. A successful flip to strict is logged as its own audit line, distinct from the per-file baseline WARNs — this is the durable record that an operator made the deliberate, reviewable decision to close the TOFU window for that volume.
  • No remount required. The broker and sidecar already serving the volume pick up the strict flip on their next state check; no pod restart or remount needed.

Verifying graduation succeeded:

  1. Confirm the Job/init-container exited 0 and its JSON result reports zero errors.
  2. Confirm the audit log line for the strict flip is present for that PVC's pvc_uid/namespace.
  3. Confirm cloudtaser_vs_freshness_baselined_files_total{pvc_uid="<uid>",namespace="<ns>"} has stopped incrementing. A graduated (strict) volume never baselines a new file again — strict fails closed on missing state instead of baselining it — so a metric still climbing after you believe you've graduated means either the walk didn't cover every raw path (check --raw-path scoping on multi-mount pods) or the flip didn't happen (check the walk's error count). The helm monitoring stack ships a PrometheusRule alert on this metric staying non-zero past an expected graduation window.
  4. Confirm the per-file baseline WARN logs for that volume have stopped appearing in sidecar logs.

Until graduated, a volume stays in the upgrade cohort described in the configuration reference — see that page for the precise security boundary during the TOFU window, and the threat model for how that boundary is scoped against the overall adversary model.

Known operational limits

  • Directory renames under trusted freshness return EXDEV, triggering standard copy+delete fallbacks (file renames work natively; hard links return EOPNOTSUPP — both hit workloads like ClickHouse/Prometheus TSDB/VictoriaMetrics (renames) and Cassandra nodetool snapshot (hard links); the cloudtaser.io/vs-freshness: "local" annotation is the supported opt-out for both).
  • DEK rotation currently requires re-encryption of the volume; there is no transparent key-version rotation yet.
  • DEK revocation is destructive and documented separately in the crypto-shredding runbook.
  • Chunk size is fixed at first write per file; changing the PVC annotation affects new files only.
  • Legacy plaintext passthrough: files without the CTVS header are served as-is until first write (zero-downtime adoption). In default/local mode this is also the plaintext-injection surface — enable freshness mode to close it.

Escalation data to collect

When filing an issue, include: sidecar and broker logs, kubectl describe pod, the PVC annotations, volumeShield Helm values (redact addresses if needed), whether freshness/nonce modes are enabled, and the first 4 bytes (head -c 4) of an affected raw file.