The mount points may look similar, but the buckets make different promises: S3 Files preserves ordinary objects for the surrounding S3 ecosystem; ZeroFS treats object storage as the private substrate of the filesystem.</p>

Making POSIX filesystems replicated and highly available

2026-06-28T00:00:00+00:00

ZeroFS stores a POSIX filesystem in a log-structured database in S3. On one node, acknowledged but unflushed writes live only in memory, and the filesystem is unavailable whenever that node is down. A standby must cover both gaps without weakening the durability promised by fsync</code>.</p>

Where a single node runs out</h2> Object storage is durable, but each request is slow and billable. Committing every filesystem operation directly would add tens of milliseconds and a PUT charge to each write. ZeroFS instead buffers writes in an in-memory memtable and flushes batches on fsync</code>, when the memtable fills, and on a periodic timer. A write()</code> returns once the data reaches the memtable; fsync()</code> pushes it to S3.</p> This creates two failure modes. A crash before a flush loses acknowledged writes, which POSIX permits because fsync</code>, not write</code>, is the durability boundary. Data already flushed remains in S3, but no server exposes the filesystem while the node is down. A standby can retain the in-memory tail and take over serving, provided it never becomes a second writer.</p> Only one writer, ever</h2> Two nodes over one bucket can split-brain: both believe they are the leader and write concurrently. Once their updates interleave, the filesystem is corrupt. Preventing that cannot depend on timing, network reachability, or synchronized clocks.</p> The storage layer supplies the final boundary. Opening a database for writing conditionally updates its manifest, increments a writer_epoch</code>, and fences the previous holder. The old leader's next manifest refresh or write observes the newer term and stops. Any SST files it uploaded in the meantime remain unreferenced and are later reclaimed by garbage collection. Only one writer epoch can commit durable state.</p> A second mechanism controls who may serve. A conditional marker object beside the database moves through Active</code>, Claiming</code>, and Opening</code>. Every transition names one generation and uses the exact object-store version it read. A candidate must win the marker claim, wait out the previous leader's bounded serving authority, open the database to publish a higher writer epoch, reconcile its tail, and become Active</code> before accepting clients.</p> Fencing and marker ownership use conditional writes, supported directly by S3, Azure Blob, and Google Cloud Storage. For S3-compatible stores without that primitive, ZeroFS can use Redis. At startup, each node also asks its peer which one is active instead of trusting its configured role.</p> </div> Why two nodes, not three</h2> Raft and Paxos clusters use an odd number of members because the nodes vote on leadership and committed state. ZeroFS has no node quorum: exact conditional updates in the object store order marker ownership and writer epochs.</p> The supported HA topology is exactly two participants. One node serves and the sole standby supplies the heartbeat acknowledgement and complete-tail proof used by the leader's fast path. A third participant would not add a vote; it would violate that sole-standby assumption. Independent read-only nodes remain available for read scaling.</p> Keeping a standby in step</h2> Replication is semi-synchronous and ordered. While the pair is Connected, the leader sends each mutation to the standby, waits for an acknowledgement, applies it locally, and then answers the client. The receiver accepts sequence N</em> only when durable storage and its retained tail account for every earlier sequence. A restarted receiver that lost its in-memory base rejects a torn suffix, causing the leader to flush the missing base and retry the same sequence.</p> File contents need more than metadata replication. Until an immutable segment reaches object storage, a replicated write carries its sealed, compressed and encrypted frame bytes with the extent pointer. On takeover the standby materializes any missing segment at the original offsets before serving.</p> If the standby falls behind or disconnects, the leader enters Solo instead of blocking the filesystem. Replication resumes when the standby returns. Before acknowledging its first Solo write, the leader marks the current durability lineage non-inheritable. Solo writes then have single-node durability: anything not yet flushed has no second copy. Fencing still prevents another writer from committing durable state.</p> When the leader dies</h2> The leader sends a coverage heartbeat every 100 ms. It includes the exact writer epoch and the applied and durable replication frontiers; the standby acknowledges only when shared storage plus its retained tail cover the complete applied frontier. After roughly two seconds without coverage, the standby may attempt a handoff. Silence is a suspicion signal, not permission to serve.</p> sequenceDiagram participant C as Client participant L as Leader participant S as Standby participant O as Object storage Note over C,O: Connected C->>L: mutation L->>S: sequence N and any sealed frame bytes S->>S: validate complete prefix, retain tail S-->>L: accepted L->>L: apply sequence N L-->>C: result L->>S: coverage heartbeat S-->>L: exact-epoch coverage ACK Note over L: Leader crashes or is partitioned L--xS: heartbeats stop Note over S: no coverage for ~2 s S->>S: quiesce heartbeat ACKs S->>O: exact CAS Active to Claiming Note over S,O: wait the full 9 s claim grace S->>O: exact CAS Claiming to Opening S->>O: open writer, publish a higher epoch S->>S: validate and replay retained tail S->>O: exact CAS Opening to Active S->>O: validate exact Active identity C->>S: reconnect, replay session, resend same op-id S-->>C: definitive result (deduplicated) Note over L: a returning old leader stays fenced </pre></div> A leader failure end to end: commit-before-apply replication, phased takeover, and session replay.</figcaption> </figure> The claim waits nine seconds before entering Opening</code>. That interval is derived from the previous leader's last three-second authority grant, a three-second final marker-validation opportunity, a two-second response drain, and one second of scheduling margin. These values belong to one static policy rather than independent runtime knobs. Opening the writer then publishes the higher epoch that independently fences durable commits. Only after tail reconciliation and an exact Active</code> validation does the successor serve.</p> The old leader must stop reads as well as writes. During healthy replication, an exact-epoch heartbeat ACK no more than 300 ms old renews serving authority without polling the marker. Once ACKs are stale, the leader immediately validates its exact Active</code> marker on a one-second cadence. Each read has a separate three-second allowance, while authority remains anchored to the request start. An expired grant suspends successful responses while one final validation runs; a changed marker, final error or timeout, or writer fence retires the process.</p> The bundled mount and the Python, TypeScript, and Go clients accept both node addresses as one comma-separated target set; Rust also exposes connect_multi</code>. Targets are probed concurrently and re-probed after a disconnect. Session replay rebinds linked open handles by inode id and re-acquires recorded byte-range locks before publishing the new connection. An open file whose final link was removed cannot be replayed, so its loss makes that logical session terminal with ESTALE</code>.</p> A request in flight during failure may already have committed. Every replay-sensitive mutation therefore carries a stable operation id, attempt state, and origin epoch, while the replicated batch carries its exact result. A successor with complete coverage can answer a retried mkdir</code> or rename</code> without applying it twice.</p> When fsync can't prove durability</h2> Replication reduces but cannot eliminate the un-fsync</code>'d window. Losing both nodes, or losing a leader in Solo, discards acknowledged writes that never reached S3. A successful fsync</code> means every write acknowledged on that descriptor is in object storage and, under HA, survives failover. If recovery did not carry those writes, fsync</code> returns ESTALE</code>.</p> A lineage token</strong> identifies an uninterrupted durable history. A connected standby that replays the leader's tail retains the token. A cold restart or takeover after Solo gets a new token because it cannot prove that it inherited every write. Before acknowledging its first Solo write, the leader marks the lineage un-inheritable in object storage. The client tags un-fsync</code>'d changes with the current token; on fsync</code>, the leader flushes and compares the oldest outstanding token with the live one. A mismatch fails the call.</p> What happened</th> Token</th> fsync result</th></tr></thead> Connected leader fails</td> Kept: the standby held the writes and replayed the tail</td> Succeeds after failover</td></tr> Both nodes lost before a flush</td> Regenerated: a cold restart can't prove it inherited anything</td> ESTALE</code>: the writes were only in memory</td></tr> Takeover after Solo</td> Regenerated: the Solo leader tainted the lineage first</td> ESTALE</code>: the new leader never got the writes</td></tr> </tbody> </table> </div> The token check follows the descriptor POSIX makes responsible. An fsync</code> on a file covers that file's data and metadata across every open handle; an fsync</code> on a directory covers its directory-entry changes, and a cross-directory rename</code> marks both directories. On a lineage mismatch, the next responsible fsync</code> returns ESTALE</code>: work since its last successful durability boundary is gone. Client libraries expose this as a distinct stale error.</p> Trying to break it</h2> ZeroFS tests these guarantees with Jepsen</a>. The single-node suite generates filesystem operations over a 9P mount and checks them against a reference model. Its crash mode kills the server, drops the memtable, recovers from object storage, and verifies the result against the last fsync</code>. This suite runs in CI on every change.</p> The HA suite runs a leader and standby over MinIO while a nemesis kills the leader, the standby, or both; partitions the replication link; pauses the object store; and freezes then thaws a leader after a successor promotes. The workloads cover concurrent add and remove operations, file contents, filesystem statistics, and lineage-aware fsync</code> for file data, metadata, multiple handles, directory entries, and cross-directory rename. The checkers reject lost, resurrected, or corrupted acknowledged state and any successful fsync</code> that approved a lineage recovery which dropped its writes. The suite is in the repository</a>.</p> What it doesn't do</h2> HA improves availability, not throughput. There is still one writer, reads are not distributed across the pair, and exactly two HA participants are supported.</p> Crash failover is not a two-second event. The two-second suspicion window is followed by the nine-second claim grace, writer open, tail reconciliation, activation, and client reconnect. Clients block and retry during that interval. The shared object store and its conditional-update path also remain availability dependencies.</p> Advisory byte-range locks live in server memory and are not replicated. The bundled client records and re-acquires them before completing session replay, but ownership is not continuous: another session may acquire a lock during the gap. Applications that need fencing across failover cannot use an advisory file lock as a distributed mutex.</p> fsync</code> remains the durability boundary. A connected standby also protects unflushed writes, but applications should not rely on that extra copy. Configuration and a complete failure-case table are in the high availability documentation</a>.</p> The phased marker retires the old serving term before the next one opens, while the writer epoch remains the independent storage fence. Replication covers ordinary Connected failover; lineage tokens turn failures outside that coverage into ESTALE</code> instead of false success.</p>

ZeroFS

ZeroFS vs. Amazon S3 Files

Making POSIX filesystems replicated and highly available