Durability & Consistency

This document describes the durability and consistency guarantees provided by ZeroFS, including write atomicity, operation ordering, and crash recovery semantics.

Overview

ZeroFS implements a transactional storage model on object storage. File contents live as compressed, encrypted frames inside immutable segment objects written directly to the object store. Metadata (inodes, directory entries, and one 32-byte FrameLoc pointer per 32 KiB extent) lives in an LSM-tree database. This architecture provides three properties:

1. Write Atomicity: All filesystem operations are atomic at the transaction level
2. Strict Ordering: Operations are totally ordered and this order is preserved across crashes
3. Recovery-Free Startup: No filesystem check or repair is required after unexpected termination

Write Atomicity

Each filesystem operation executes within a single database transaction. A transaction bundles all related modifications into an atomic unit that either commits entirely or has no effect.

// Create transaction
let mut txn = db.new_transaction();

// Seal extents into the open segment buffer,
// stage FrameLoc pointers in the transaction
extent_store.write(&mut txn, id, offset, data, old_size);

inode.size = new_size;
inode.mtime = now;
inode_store.save(&mut txn, id, &inode);

directory_store.add(&mut txn, ...);

// Atomic commit via the single-writer coalescer
write_coordinator.commit(txn).await;

Operation Ordering

ZeroFS provides a total order over all write operations. This order is preserved exactly across process termination and restart.

// caller
write_coordinator.commit(txn).await;

// worker (single task, FIFO drain)
let mut batch = vec![rx.recv().await];
while let Ok(m) = rx.try_recv() { batch.push(m); }
let mut merged = WriteBatch::new();
for (txn, _) in &batch { txn.apply_to(&mut merged); }
db.write(merged).await;

visible(B) → visible(A)

Crash Recovery

ZeroFS requires no recovery procedure after unexpected termination.

Durability Semantics

ZeroFS follows standard POSIX durability semantics. Write operations are buffered in memory and persisted to durable storage upon explicit synchronization.

// POSIX interface
write(fd, data, len);   // Buffer in memory
fsync(fd);              // Persist to storage

# Python
f.write(data)
f.flush()
os.fsync(f.fileno())

// Go
f.Write(data)
f.Sync()

// Rust
file.write_all(data)?;
file.sync_all()?;

Verified fsync

A successful fsync is verified: it means every write the descriptor acknowledged before it is durable on object storage. Acknowledged file data lives in the in-RAM open segment buffer, and its metadata in the memtable, until the next flush, so a crash before that flush loses them. ZeroFS does not let that loss pass silently. A fsync issued after a restart, over writes made before it, returns a stale-handle error (ESTALE) rather than a false success.

Each acknowledged write is tagged with a durability lineage token naming the current unbroken durable lineage. A restart reopens the database and, with no way to prove it inherited the un-flushed buffers, regenerates the token. A fsync presents the token its writes were made under; if the database has since regenerated, the tokens differ and the fsync fails. The first fsync after a write is the durability boundary, so a failure reports real loss: redo the writes and fsync again.

This holds on a single node and under replication alike. With a replicated standby a clean failover can prove it inherited the writes (the standby receives un-flushed extent frames over replication and materializes their segments at takeover) and keeps the token, so the fsync succeeds transparently; the failover cases are diagrammed in High availability. The check is per descriptor and matches POSIX: it covers the whole file across every open handle, speaks for no other file, and verifies data, metadata, directory entries, and rename.

Eliminating the Durability Window

By default, writes between fsync calls live in the open segment buffer and memtable and become durable only when the next periodic flush or explicit fsync persists them. Committed-but-unflushed file data is bounded by the 256 MiB open buffer plus up to 4 × 256 MiB of in-flight segment uploads. For workloads that need every write durable on return, set sync_writes = true in the [lsm] section.

When enabled, the commit coordinator forces a flush after every coalesced write batch before returning success to the caller. Concurrent writes still merge into a single batch and a single flush.

The trade-off is per-operation latency. The WAL is permanently off, so each batch waits for the full durability barrier: the open segment is sealed and uploaded, then the memtable flushes to an L0 SST. No cheaper WAL-append path exists. This is expensive for chatty workloads.

[lsm]
sync_writes = true

Protocol Considerations

The durability guarantees available to applications depend on the access protocol.

Verification Through Crash Testing

ZeroFS verifies its consistency guarantees through systematic crash simulation using failpoints injected throughout the data path.

verify_all()
  enumerate_inodes()
  enumerate_tombstones()
  enumerate_orphans()
  walk_directory_tree()
  verify_directory_counts()
  verify_nlink_counts()
  verify_directory_nlinks()
  find_orphaned_inodes()
  verify_orphan_set_drained()
  verify_stats_counters()
  verify_tombstones()
  verify_file_extents()
  verify_inode_counter()
  verify_orphaned_extents()
  verify_dir_entry_scan_consistency()
  verify_orphaned_directory_metadata()
  verify_dir_cookie_counters()

Test Methodology

For each failpoint, the test suite:

1. Performs a filesystem operation with the failpoint enabled
2. Aborts the operation at the injection point and tears down the filesystem instance, discarding all in-memory state
3. Restarts the filesystem from object storage
4. Runs the full consistency checker
5. Verifies either complete rollback or complete commit, never partial state

This verifies that the atomicity and ordering guarantees hold at each injected crash point.

Deterministic Simulation Testing

The data path, namespace mutations, segment garbage collection, compaction, and crash recovery also run under deterministic simulation. A simulated world executes on a single-threaded runtime with virtual time: every object-store operation is assigned a pseudo-random latency from a seeded generator, and those latencies decide the order in which concurrent tasks wake. One seed defines one exact schedule, hours of simulated activity execute in seconds, and the same seed replays the same run. Seeds are drawn fresh from OS entropy on every run and printed, so repeated runs accumulate coverage and any failure is reproducible from its printed seed. A companion test replays one seed and compares the two runs' complete execution traces, so replay determinism is itself under test.

Each world races ordinary file writers (writes, truncates, hole punches, verified reads, whole-file scans, and fsyncs), two disjoint-region writers on one shared inode, a namespace actor, and live garbage-collection passes. The namespace actor creates and removes files and directories, mutates file content, renames and overwrites entries, creates hardlinks, symlinks, and special nodes, retries idempotent creates, and exercises open-unlink reclamation. The world runs over an in-memory object store in the production durability configuration and behind the production retry layer. Some worlds also inject seeded transient store faults, including puts and deletes whose effect lands but whose response is lost, so retries collide with their own already-applied writes. At a seeded instant the world crashes mid-flight. Every task is cancelled at whatever await point it happens to occupy, the store rejects all in-flight I/O (a dead process completes no writes), and a fresh instance recovers from the surviving objects.

At every quiesced point and after every crash, the state must satisfy a model-based oracle:

1. Each ordinary file and shared-inode region equals the replay of an operation prefix no older than the last acknowledged fsync. One in-flight operation of unknown fate is permitted per actor at the cut, and the shared file's durable size is fixed.
2. The namespace tree is structurally isomorphic to a prefix of its operation log: paths, node kinds and values, file contents, and hardlink identity must all match.
3. Every referenced extent still resolves: the filesystem is re-read in full, so a dangling frame pointer or a wrongly deleted segment surfaces immediately. Modeled data files also require a key for every nonzero extent and reject keys beyond EOF.
4. Segment accounting reconciles through typed snapshots: each counter's live bytes equal the byte sum of the extent pointers referencing it, live bytes never exceed appended bytes, and every referenced segment has a counter.
5. All incremental footprint gauges (segment count, appended bytes, live bytes, and reclaimable bytes) equal an authoritative scan.
6. The full consistency checker passes, including directory and link invariants, extent readability and EOF bounds, orphan metadata, inode and directory-cookie counters, and the global stats behind statfs.

Garbage-collection passes run with seeded per-pass tuning so the rarely-taken paths execute too: checkpoint-pinned passes, the write-cold gates (tail scrub, chain compaction), small round budgets that exercise the gather caps, and multi-batch drains.

A companion mode composes the simulation with failpoints. Some windows are too narrow for a timer to land in (a read-decide-act gap with no await inside it), so each round can arm a failpoint that crashes the world at that exact code point, and can widen such a gap by a few virtual milliseconds so concurrent commits land inside it before the code acts. This combination reproduces historical bugs whose window is the space between a garbage-collection decision and the irreversible action it drives. The failpoint suite remains alongside as the deterministic pointwise guard for each window.

CI runs the simulation as a dedicated test target with the required SlateDB and Tokio configuration flags. A scheduled job additionally soaks fresh seeds for 50 minutes every hour.

export DST_RUSTFLAGS="--cfg dst --cfg tokio_unstable --cfg io_uring_skip_arch_check"
RUSTFLAGS="$DST_RUSTFLAGS" cargo test --test dst

# A timed soak: fresh random seeds, fanned across all cores, until the
# budget elapses. Scale depth with DST_ROUNDS / DST_OPS / DST_CRASH_PCT.
DST_WALL_CLOCK_SECS=3600 RUSTFLAGS="$DST_RUSTFLAGS" \
  cargo test --test dst -- --nocapture seeds

# Failpoint-placed crashes and widened windows
RUSTFLAGS="$DST_RUSTFLAGS" \
  cargo test --features failpoints --test dst crash_points

# Reproduce a failure from its printed seed
DST_SEEDS=16039913183294875603 RUSTFLAGS="$DST_RUSTFLAGS" \
  cargo test --test dst -- --nocapture

Continuous Integration Testing

In addition to crash simulation, CI runs pjdfstest, stress-ng, Linux kernel builds, Jepsen's local-fs suite, and a ZFS integration test on every commit.

Model-Based Testing

Jepsen (local-fs): Generates random sequences of filesystem operations and checks each result against a reference model, shrinking any divergence to a minimal failing case. It runs over a 9P mount. A crash mode injects the loss of un-fsynced writes: it kills the server mid-run, dropping the in-memory open segment buffer and memtable, then recovers from the object store and verifies the surviving state is consistent with the last fsync.

Layered Filesystem Testing

The CI suite includes a test that creates a ZFS pool on a ZeroFS NBD block device:

1. Create a 3GB block device file via 9P
2. Connect via NBD and create a ZFS pool
3. Extract the Linux kernel source (~80,000 files)
4. Compute checksums of all files
5. Export the ZFS pool and restart ZeroFS
6. Reimport the pool and verify all checksums match

This test validates data integrity through multiple filesystem layers and across process restarts.

Test Matrix

All tests run on every pull request and merge to the main branch.

Conditional Writes and Fencing

ZeroFS uses conditional writes (put-if-not-exists) for fencing to prevent split-brain scenarios when multiple instances access the same storage backend. This ensures that only one writer can be active at a time.

AWS S3, Azure Blob Storage, and Google Cloud Storage support conditional writes natively. For S3-compatible object stores that do not support conditional puts, ZeroFS can use Redis as a coordination backend:

[aws]
conditional_put = "redis://localhost:6379"

When configured, ZeroFS uses Redis to coordinate conditional write operations, providing the same fencing guarantees as native conditional put support. See the Configuration Guide for details.

Backend Durability

ZeroFS delegates storage durability to the object storage backend.

These services maintain multiple replicas across independent failure domains. Once data is persisted to object storage, it is protected against hardware failures, facility outages, and other localized events without additional configuration.

Summary

ZeroFS provides the following durability and consistency guarantees: