Kubernetes CSI Driver

zerofs-csi exposes a ZeroFS filesystem to Kubernetes as dynamically provisioned persistent volumes. A ZeroFS gateway (a leader/standby pair) backs a StorageClass; each volume is a directory on that filesystem, and pods mount their directory over 9P.

How It Works

ZeroFS is single-writer per bucket/prefix, so the driver shares one gateway across all volumes of a StorageClass rather than running an instance per volume. The gateway is a leader/standby pair: a leader serves the filesystem while a standby replicates it and takes over through a bounded handoff if the leader fails, with at most one node writing at any time (see High Availability).

• Two single-replica StatefulSets run zerofs run against one bucket. The leader serves 9P to the nodes and a gRPC admin API to the controller; the standby receives the leader's writes and stands ready. Manifest-backed serving authority and the post-fence handoff grace prevent overlapping successful reads/acknowledgments, while writer-epoch fencing independently guarantees that only one writer term can commit durable database state.
• The controller plugin translates CreateVolume into a CreateDirectory call for /volumes/pvc-<uuid> on the admin API, and DeleteVolume into RemoveDirectory. It points at both nodes and provisions against whichever is leading; a standby is not serving, so it is skipped. Removal renames the directory into a trash area and returns immediately; a background task deletes the contents and the garbage collector reclaims the extents.
• The node plugin publishes a volume by spawning zerofs mount <both-gateways> <target> --aname /volumes/pvc-<uuid>, the userspace 9P client described in 9P File Access, attached directly to the volume's directory. The attach is rooted at that directory: the mount cannot name anything above it, and .. at its root resolves to itself.

Each published volume is its own zerofs mount process pointed at both nodes, so mounts inherit the client's failover behavior: when the leader fails over (or a node restarts), the client follows the serving node, operations block through the takeover gap, then linked open file handles resume by inode id. A failover does not normally invalidate an existing mount. The exception is an open file whose final link was removed: that handle is connection-local, and a disconnect makes the mount's entire logical session terminal with ESTALE; recreate the pod or mount to establish a new session.

ReadWriteMany works because any number of pods on any number of nodes each run their own client against the same directory; the gateway arbitrates, including POSIX byte-range locks.

Installation

Prerequisites: an object store bucket with credentials for the gateway, and /dev/fuse on the nodes (the node plugin runs privileged and mounts it from the host).

1. The gateway

The example manifest creates the zerofs namespace and, for each node, a config Secret, a per-node Service (9P on 5564, admin RPC on 7000, replication on 9000), and a single-replica StatefulSet. The two nodes form the leader/standby pair over one bucket:

curl -fsSLO https://raw.githubusercontent.com/Barre/ZeroFS/main/zerofs/zerofs-csi/deploy/gateway-example.yaml
# Edit both nodes' Secrets: identical storage url and encryption_password, plus credentials.
kubectl apply -f gateway-example.yaml

Each node is configured like any other ZeroFS instance (see Configuration) plus a [replication] section naming its role and its peer; sizing the cache is the main throughput lever for everything mounted through it. The two nodes must share the same storage url and encryption password.

2. The driver

DEPLOY=https://raw.githubusercontent.com/Barre/ZeroFS/main/zerofs/zerofs-csi/deploy
kubectl apply -f $DEPLOY/csidriver.yaml -f $DEPLOY/rbac.yaml \
              -f $DEPLOY/controller.yaml -f $DEPLOY/node.yaml

The controller is a Deployment running the CSI controller service next to the external-provisioner sidecar. The node plugin is a DaemonSet on every node (it tolerates all taints, since any node can run a pod with a volume). The ghcr.io/barre/zerofs-csi image is published for amd64 and arm64 and carries the same zerofs binary as the standalone image.

3. The StorageClass

kubectl apply -f $DEPLOY/storageclass-example.yaml

adminEndpoint and gateway each carry the pair's node addresses, comma-separated; the data-plane mount and the controller probe the list and follow the serving leader, rerouting on failover.

DeleteVolume receives no StorageClass parameters (the CSI specification gives it only the volume id and secrets), so the StorageClass must also reference a Secret carrying adminEndpoint through csi.storage.k8s.io/provisioner-secret-name and -namespace. The example includes it.

One StorageClass per gateway. Several StorageClasses, each with its own gateway pair, bucket, and encryption password, can coexist in one cluster.

Using Volumes

Nothing ZeroFS-specific appears in workloads:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  storageClassName: zerofs
  accessModes: ['ReadWriteMany']
  resources:
    requests:
      storage: 10Gi

Creating the claim provisions the directory; scheduling a pod that uses it mounts the directory at the pod's mountPath. There is no attach step (attachRequired: false), so pods start as soon as the node plugin has the mount up.

Failure Behavior

• Leader failover: when the leader fails, heartbeat silence triggers a durable marker-claim attempt; the winner remains in Claiming for the nine-second pre-open grace, then enters Opening, opens the database to epoch-fence the old writer, reconciles the retained tail, and activates. Published mounts block through the gap and resume linked open handles by inode id, following the new leader without a remount; the controller's next provisioning call routes to it too. An open-unlinked handle is the exception described above: it cannot be replayed, and its loss terminally fails that mount session with ESTALE. Writer-epoch fencing prevents the deposed leader from committing after takeover. This is the same model documented under High Availability; the data-plane client behavior is Client access.
• Node plugin restart: the FUSE clients are children of the node plugin container, so a restart breaks the published mounts on that node until the affected pods are recreated. The DaemonSet therefore uses an OnDelete update strategy: upgrades are deliberate, per-node, coordinated with draining.
• Node loss: volume directories live on the gateway, so rescheduled pods mount the same data from any other node.

The pair is two nodes, one standby: availability through failover, not write scale-out. See High Availability for the replication model, guarantees, and limits.

Security

None of the gateway ports is authenticated; the trust boundary is network reachability, the same model as NFS with AUTH_SYS. Each must be restricted to its legitimate clients:

• 9P (5564) is the data plane. The attach aname is namespacing, not authentication: any client that reaches this port can attach to any directory, including another volume's or the filesystem root. Pods never speak 9P themselves (they see only a kernel mountpoint), so this port is open to the node plugins.
• Admin RPC (7000) is a root-equivalent control plane. It runs every call as uid 0 and can trash any volume (RemoveDirectory). Its only client is the controller, so it is locked to the controller pod alone.
• Replication (9000) is the leader → standby stream, heartbeats, and role discovery. Its only clients are the gateway pods themselves, so it is open just to them (each node ships to the other).

networkpolicy-example.yaml ships in the deploy bundle and enforces all three. Apply it alongside the gateway (adapt the labels to your manifests):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: zerofs-gateway
  namespace: zerofs
spec:
  podSelector:
    matchLabels:
      app: zerofs-gateway
  policyTypes: [Ingress]
  ingress:
    - ports: [{ port: 5564 }] # data plane, from every node plugin
      from:
        - podSelector:
            matchLabels:
              app: zerofs-csi-node
    - ports: [{ port: 7000 }] # control plane, from the controller only
      from:
        - podSelector:
            matchLabels:
              app: zerofs-csi-controller
    - ports: [{ port: 9000 }] # replication, between the gateway pods
      from:
        - podSelector:
            matchLabels:
              app: zerofs-gateway

A NetworkPolicy is a no-op on a CNI that does not enforce them (vanilla flannel, default k3s); there, no port has in-cluster protection, so do not expose the gateway to untrusted pods. Data is still encrypted before it reaches the object store; the encryption password is per gateway, i.e. per StorageClass.

Limitations

• Capacity is not enforced. PVC sizes are labels; the gateway's max_size_gb caps the filesystem as a whole.
• No per-volume snapshots. Checkpoints cover the whole gateway filesystem, not one volume directory.
• StorageClass mountOptions are rejected. The mount takes no pass-through options; a capability carrying them fails with InvalidArgument rather than being silently ignored.
• Mount volumes only. volumeMode: Block is not supported and not planned. A workload that wants a fixed-size disk image can create one inside its filesystem volume (truncate -s 10G /data/disk.img) and consume it directly (e.g. a file-backed VM disk) or, given sufficient privilege, attach it as a loop device. A first-class block volume would only add a raw device in exchange for giving up the filesystem's ReadWriteMany and reconnect-tolerant recovery.

Next Steps