High Availability

ZeroFS high availability runs two nodes over one object store. One node serves as leader while the other retains its unflushed write tail and is ready to take over. While the pair is connected, a single-node failure preserves acknowledged writes, including writes that have not yet been fsync'd.

Model

ZeroFS batches metadata and file data before uploading it to object storage. This keeps object-store latency and request cost off the write path, but leaves recent writes in memory until fsync or the next background flush. HA adds a second in-memory copy and another process that can reopen the same database.

The pair has two operating modes:

• Connected: the leader ships each mutation before applying it locally and replying to the client. The standby retains a complete ordered tail.
• Solo: the leader cannot ship to the standby but keeps serving. This favors availability; writes return to the same durability window as a standalone node until replication recovers.

The topology is always two HA participants and one data-database writer. It provides failover, not write scale-out.

Write path

Commit before apply. In Connected mode, the standby accepts sequence N only if durable storage and its retained tail cover every earlier sequence. The leader applies the mutation locally only after that acceptance. If a restarted receiver has lost part of its in-memory base, it rejects the suffix; the leader flushes the missing base and retries the same sequence.

Exact mutation results. The replication batch includes the operation-result ledger used by the private 9P2000.L.Z dialect. A request retried after disconnect keeps its operation id and original writer epoch, so a direct successor can return the recorded result instead of applying the mutation twice. Automatic retries stop after the 120-second protocol horizon if the outcome remains ambiguous.

File data ships as frames. Metadata points into immutable segment objects. Until a segment is uploaded, the leader sends the compressed, encrypted frame bytes with the pointer. During takeover the standby materializes any missing segment at the original offsets before serving. Once a segment is durable, the pointer is sufficient because both nodes use the same object store.

The standby prunes tail entries as leader flushes make them durable. Promotion is refused if the retained tail does not cover a complete prefix of the leader's applied state.

Authority and fencing

The durable .zerofs_ha_leader marker serializes startup and handoff. Each transition uses the exact object version returned by the object store.

Safety has three separate layers:

Writer-epoch fencing is the final storage-safety boundary and does not depend on clocks. The marker and bounded serving lease prevent a deposed process from continuing to return successful responses while the successor opens.

Failover, client recovery, and mutation retry use one static timing policy. These values are release policy, not configuration knobs, and do not change in response to observed latency. Durations that exist to cover other phases are derived rather than repeated as independent constants.

The 300 ms ACK window is three heartbeat periods. The 5-second abandoned-handoff window is five renewal periods. The 9-second claim grace is derived as 3 s authority + 3 s final recovery + 2 s response drain + 1 s scheduling margin. The client and dedup policy is checked with the same contract: a request may spend at most 64 seconds proving that its connection is alive, the conservative request/takeover/probe/replay sequence is 108 seconds, automatic mutation retry ends at 120 seconds, and the completed result remains available for 150 seconds.

An authority grant is measured from the validation request's start, so storage latency consumes the grant rather than extending it. Fresh exact-epoch heartbeat ACKs suppress marker reads. Once ACKs are stale, the leader validates its exact Active identity immediately and schedules each subsequent read from the one-second cadence. A read may take up to three seconds; the cadence no longer doubles as its timeout. If authority expires first, successful responses remain suspended during one final bounded validation; mismatch, timeout, error, or writer fencing retires the process.

The proof does not require synchronized host clocks. Suspending a host or restoring a whole-process or VM snapshot is outside the strict no-overlap guarantee; fence or destroy the old host before another node opens the database.

Failover

Heartbeat silence is only a reason to attempt takeover; it grants no authority. A failed marker compare-and-swap leaves the standby non-serving. The nine-second claim grace covers the predecessor's last cached authority, final recovery attempt, response drain, and scheduling margin. Opening the writer then fences the previous epoch before the new leader serves.

A node that restarts asks its peer for the live role instead of trusting the configured role. A live Claiming or Opening owner renews its exact record. If it dies mid-handoff, another eligible participant must observe the same record contents and physical object version unchanged for five seconds, install a new claim, and wait a new full grace period.

Clients configured with both targets block through the gap, reconnect to the new leader, and rebuild their session. Crash failover therefore includes roughly the two-second suspicion window, the nine-second grace, writer open, tail reconciliation, and client reconnect; it is not instant.

Durability

ZeroFS uses a lineage token to prevent false fsync success after recovery.

Before acknowledging its first Solo write, a leader marks the current lineage non-inheritable in object storage. A successor that cannot prove it received the complete lineage generates a new token. A client presenting the old token then receives ESTALE, which reports that its un-fsync'd work did not survive; it is not a transient retry signal.

Lineage verification follows the descriptor responsible under POSIX. fsync on a file covers that file's data and metadata across its open handles; directory-entry changes are covered by fsync on the directory. See Durability and Consistency for the full flush model.

Configuration

Both nodes must use the same storage URL, prefix, encryption password, and object-store credentials. Give each node a stable unique node_id, a local cache, and both sides of the replication connection.

# node-a
[replication]
node_id = "node-a"
role = "leader"
replication_listen = "10.0.0.1:9000"
peers = ["10.0.0.2:9000"]

# node-b
[replication]
node_id = "node-b"
role = "standby"
replication_listen = "10.0.0.2:9000"
peers = ["10.0.0.1:9000"]

role is a bootstrap hint, not a permanent assignment. Configure both replication_listen and peers on both nodes so either can receive replication after a role swap. During normal operation, ZeroFS rejects a one-sided configuration instead of accepting a pair that cannot complete the reverse role swap. The forced-recovery procedure below is the explicit exception.

The storage backend must support exact conditional object updates, either natively or through the configured conditional-write adapter. The replication and client ports are unauthenticated; keep them on a trusted network. node_id, replication_listen, and peers support the same ${VAR} substitution as the rest of the configuration.

Forced recovery

force_recovery bypasses normal stale-handoff observation. It is a break-glass assertion that all former participants are dead and their object-store requests are quiesced. A replacement node cannot distinguish this condition from a network partition or a suspended process. The unreachable process may still hold the only copy of acknowledged writes in its in-memory tail or resume a writer-open request. Normal failover is automatic only when a surviving standby can prove complete coverage. Without that proof, takeover requires an operator who has fenced every former participant.

[replication]
node_id = "node-a"
role = "leader"
replication_listen = "10.0.0.1:9000"
peers = []
force_recovery = true

Within the handoff state machine, the flag skips only the five-second unchanged-record observation for an abandoned Claiming or Opening owner. It does not skip the nine-second Claiming grace, writer-epoch fencing, tail reconciliation, or the exact Opening-to-Active transition.

Upgrade the two HA peers together. New releases understand the legacy leader marker, but old releases cannot participate after the phased marker has been written, and the replication handshake is release-coupled.

Clients and bindings

Pass both 9P addresses as one target set. The bundled mount and foreign-language bindings use a comma-separated string; Rust also exposes connect_multi. Targets are probed concurrently, and the first node that proves it is serving becomes the connection.

The client tries the targets until it finds the serving leader. During an outage, undispatched calls block while the session reconnects. After takeover it replays fids by stable inode id, reopens linked handles, and attempts to reacquire recorded byte-range locks. Replay is published only if the whole session succeeds.

An open file whose final directory link has been removed cannot be recovered on a new connection. If such a handle exists during reconnect, the logical client or mount becomes terminal with ESTALE; establish a new session and reopen the remaining paths. Server lock state is not replicated, so reacquisition is not an uninterrupted distributed lock guarantee and a competing session may acquire a lock during the gap.

The Linux kernel 9P client takes one endpoint and does not use the private reconnect protocol. NFS and NBD connections also do not reroute between the two nodes. Use zerofs mount, a client library, CSI, or an external connection-routing layer when transparent client failover is required.

Kubernetes (CSI)

The CSI driver uses the same client behavior. Set both data-plane and admin endpoints on the StorageClass:

parameters:
  gateway: node-a.zerofs.svc:5564,node-b.zerofs.svc:5564
  adminEndpoint: http://node-a.zerofs.svc:7000,http://node-b.zerofs.svc:7000

Each published volume runs zerofs mount against both gateways. The controller probes both admin endpoints and sends provisioning calls to the leader. Gate gateway pod readiness on the replication port, not 9P: the standby opens replication early but does not serve 9P until takeover.

Operations and limits

• Two participants only. Never attach a third HA process to the database. Use read replicas for independent read scaling.
• Object storage remains shared infrastructure. Both nodes need the same database and conditional-update service. HA does not make an unavailable object store available.
• The standby tail is held in RAM. If the receiver cannot keep up or loses coverage, the leader enters Solo or flushes the missing base before Connected acknowledgements resume. An uncovered standby refuses promotion.
• Failover pauses clients. The suspicion window, claim grace, writer open, reconciliation, and reconnect all contribute to the gap.
• Advisory locks are not fencing. The bundled client attempts to reacquire them, but ownership is not continuous across reconnect and the server does not persist the lock table.
• Host lifecycle matters. Externally fence suspended, snapshotted, or partitioned hosts before replacing them.

Use Prometheus metrics and logs from both nodes to watch replication mode, coverage, authority, and failover. The replication listener must remain reachable in both directions after a role swap.