Architecture

This page describes how ZeroFS is structured: the protocol servers, the shared filesystem core, and the two storage planes beneath it.

System Overview

The NFS, 9P, and NBD servers run in one userspace process and share a single filesystem layer. Below that layer, storage splits into two planes. Metadata (inodes, directories, extent pointers) lives in an LSM tree on the object store. File contents do not pass through the LSM: they are packed into immutable segment objects written directly to the object store. Both planes read through one shared raw-parts cache.

Storage Engine

Data Plane

File data is split into 32 KiB extents. On write, each extent is compressed, then encrypted into a self-describing frame (XChaCha20-Poly1305; see Encryption), and appended to an in-memory open segment. The write path issues no object-store writes. Appends and full-extent overwrites issue no object-store requests at all; a partial overwrite of an extent not held in cache performs a read-modify-write and may issue one ranged GET. When the open segment reaches 256 MiB, when a client issues a durability barrier (fsync, NFS COMMIT, NBD flush), or when the periodic flush fires (default every 30 seconds), it is sealed and uploaded as one immutable object:

segments/<shard>/<epoch>/<counter>

Segments are written once and never modified. The shard is the low byte of the counter, so consecutive segments round-robin across 256 key prefixes.

Metadata Plane

The LSM tree stores metadata only; file bytes never enter it:

The LSM tree also provides writer fencing (a monotonic writer epoch that doubles as the epoch in segment object keys) plus the manifest that anchors durability, checkpoints, and read replicas. Its write-ahead log is permanently disabled: metadata becomes durable only through the flush barrier, which seals and uploads the open segment first. A durable manifest therefore never references a segment that is not in the object store.

Read Path

A read looks up the extent's pointer and issues one ranged GET for exactly the frame's bytes. Adjacent extents stored contiguously in one segment coalesce into a single GET. ZeroFS never fetches whole segment objects.

Caching

One raw-parts cache (memory + disk, 128 KiB parts) sits below both planes and holds object bytes as fetched: compressed and encrypted on the local cache disk. A separate cache holds decoded metadata blocks. See Caching.

Space Reclamation

Reclamation runs in-process on the writer. A segment GC pass runs every 60 seconds, or every 5 seconds while a saturated backlog meets an idle store. Each pass deletes segments with zero live bytes and repacks fragmented ones, re-sorting live extents by file and offset so they become physically contiguous. Metadata LSM compaction is embedded in zerofs run as well. See Garbage Collection.

Choosing a Protocol

Because the servers share one filesystem layer, protocols mix on a single instance: NBD device files in .nbd/, for example, are created and resized with regular file operations through an NFS or 9P mount. Which protocol to mount with depends on the client and the workload:

The fsync difference is the deciding factor for databases. NFS COMMIT semantics allow fsync to return before data reaches stable storage. Over 9P (kernel client or zerofs mount) fsync returns only after data is durable.

Filesystem Limits

The limits below are design constants of the on-disk format:

In practice, S3 provider limits, performance with billions of objects, and storage cost take effect long before these values.

df reports the configured quota (max_size_gb) as the filesystem's total size; without a quota, the total is 16 EiB. The NFS server caps the sizes it reports at 8 EiB because some NFS clients cannot handle larger values, while the 9P server reports the full quota, so df can show different totals for the same filesystem depending on the mount protocol.