9P Protocol Extensions

ZeroFS serves standard 9P2000.L, the Linux dialect of the protocol. Its own client selects one private dialect, 9P2000.L.Z, which includes fewer round trips on metadata-heavy work, idempotent retry across a reconnect, fid rebinding that survives a server restart, durability-aware fsync, and atomic fallocate operations. This page documents what that dialect does and how it is framed on the wire.

Overview

The private dialect is separate from plain 9P2000.L. A stock kernel 9P client proposes 9P2000.L and receives the standard protocol. ZeroFS's own clients propose 9P2000.L.Z: zerofs mount, the client libraries, and the Web UI file manager all use the same zerofs-client core (compiled to WebAssembly in the browser), so they share one wire format and one negotiation path.

Every extension was added to remove a specific cost in the standard protocol:

• 9P2000.L makes the client walk a path and then stat it, or open a file and then read it, as separate requests. The fast paths fold those pairs into one round trip.
• 9P2000.L has no way to retry a mutating request safely: a mkdir whose reply was lost cannot be re-sent without risking a double-apply. An op-id makes retries idempotent.
• A reconnecting client cannot re-bind a fid to a still-linked file that was renamed while it was disconnected. Rebinding by inode id fixes that.
• 9P Tfsync returns success whether or not the connection's writes are still the live durable copy. The durability-verified variant returns success only when they are.
• 9P2000.L has no fallocate request. Tfallocate carries Linux range-allocation modes to an atomic server-side operation.

Negotiation

Dialect selection happens in the standard Tversion/Rversion exchange. There are exactly two accepted version strings:

The bundled client proposes exactly 9P2000.L.Z and requires the server to echo that exact string. The name is indivisible: .Z identifies the current ZeroFS wire generation layered on the Linux dialect; it is not a capability list, and there is no substring matching or downgrade to a partial private protocol. A released server that recognizes only an older .zerofs* layout will answer with a different token, so negotiation fails cleanly before either side decodes an incompatible frame. A future incompatible generation will use a new token such as .Z2. This remains a private interface; upgrade the client, server, Web UI, and HA peers as one release set.

Selecting 9P2000.L.Z enables the whole private type set and the mutation envelope described below. A plain 9P2000.L session uses standard framing and cannot send the private messages. There is no feature-by-feature negotiation within the private dialect.

Fewer round trips

These extensions exist because the common client patterns in 9P2000.L cost two requests where the server already has both answers in hand. Each one returns the second answer with the first.

Walk then stat

A path lookup in 9P is a Twalk followed by a Tgetattr on the resulting fid. Twalkgetattr does both in one message: it clones fid to newfid along wnames and returns the walk qids together with the final entry's full stat. It is all-or-nothing: on any component miss the server replies Rlerror rather than a partial walk.

Twalkgetattr  fid[4] newfid[4] nwname[2] nwname*( wname[s] )
Rwalkgetattr  nwqid[2] nwqid*( wqid[13] ) stat[Stat]

Directory listing with attributes

Treaddirattr is readdirplus: each entry comes back with its full stat inline, so a listing that needs attributes (the usual case for ls -l or a stat-walk) does not pay a Tgetattr per entry.

Treaddirattr  fid[4] offset[8] count[4]
Rreaddirattr  count[4] count*( DirEntryPlus )
DirEntryPlus  qid[13] offset[8] type[1] name[s] stat[Stat]

Compound create and open

The private dialect adds messages that fold a fixed client-side sequence into one round trip:

• Tlopenat is Twalk(clone) + Tlopen. It opens fid's inode on a fresh newfid and leaves fid untouched, so the client keeps a handle on the directory it walked from.
• Tlcreateattr is Twalk(clone) + Tlcreate + Tgetattr. It creates and opens the child on newfid (without mutating the directory fid, unlike plain Tlcreate) and returns the child's full stat.

The remaining stat-carrying messages reuse the standard request layout but reply with the post-operation stat that the server computes anyway and the standard reply throws away: Tmkdirattr, Tsymlinkattr, Tmknodattr, Tlinkattr, and Tsetattrattr. Without them the client has to follow every create or attribute change with a Tgetattr to refresh its cache.

Tlopenat  fid[4] newfid[4] flags[4]
Rlopenat  qid[13] iounit[4]

Open then read

Tlopenatread goes one step further for reads. It opens fid's inode on newfid exactly like Tlopenat, and also prefetches up to count bytes from offset 0 in the same round trip. A file that fits in the prefetch window is read with no follow-up Tread at all.

The inline read is best-effort and never affects the open. On a read error the reply still carries the open result with empty data, and the client falls back to a normal Tread. The eof flag tells the client whether data reached the end of the file: when set, the whole file fit in count and the client may serve the file's reads entirely from the prefetched buffer; when clear, it discards the partial prefetch and reads normally.

Tlopenatread  fid[4] newfid[4] flags[4] count[4]
Rlopenatread  qid[13] iounit[4] eof[1] count[4] data[count]

Idempotent retries

Standard 9P has no safe way to retry a mutating request. If a Tmkdir is sent and the connection drops before its reply arrives, the client cannot know whether the server applied it. Re-sending risks a double-apply (a second mkdir that fails with EEXIST, or worse, a second rename or unlink that acts on the wrong state). This matters most for the bundled client, which reconnects on its own and would otherwise have to surface an error for every in-flight mutation at the moment a server restarts.

The private dialect attaches a 25-byte operation envelope to every reorder-sensitive mutation. The envelope is outside the message body: it appears immediately after the tag, and the frame's size includes it.

ordinary request   size[4] type[1] tag[2] body...
covered mutation   size[4] type[1] tag[2] op_id[16] flags[1] origin_epoch[8] body...
                                            ^ 25 bytes after the standard
                                              7-byte header

op_id identifies one logical mutation. A frame with no flag bits set is its initial attempt (FIRST); after that attempt becomes ambiguous, every resend sets the RETRY bit and retains both the op-id and its original writer epoch. In standalone mode origin_epoch is zero. In HA it lets a coverage-proven direct successor recognize a retry that originated at its fenced predecessor.

The server keeps a time-bounded ledger keyed by op-id. It serializes concurrent copies through a single-flight gate and retains each completed operation's exact result, including errors, so a known retry receives the original outcome without applying again. An unseen RETRY fails closed with ESTALE rather than being reinterpreted as new work, except during the narrowly scoped promotion window for the named predecessor epoch. An all-zero envelope opts out of deduplication.

Automatic retry is bounded: the bundled client retries one ambiguous op-id for up to 120 seconds, while the server keeps its completed result for an additional 30-second margin. Both values come from the same static timing policy as client recovery and HA handoff, which checks that the retry horizon covers their conservative 108-second recovery budget. If the result is still unknown at the horizon, the client reports an ambiguous disconnect instead of starting a fresh attempt that might apply twice.

The covered requests are the create family (Tlcreate, Tsymlink, Tmknod, Tmkdir, and Tlink), Trename, Trenameat, Tunlinkat, Twrite, Tsetattr, Tfallocate, and the stat-carrying create and setattr variants. Writes, setattr, and fallocate are included even where repeating the operation would produce the same immediate state: a delayed copy could otherwise execute after a later mutation and roll state backward. Reads, walks, attach, clunk, and the open-only fast paths carry no envelope.

The envelope is what makes zerofs mount reconnect cleanly. On a dropped connection the client re-sends in-flight mutations with their original attempt state, and the client-side failover path preserves it across a re-route so a retried write, create, or rename has one authoritative outcome even when it reaches a successor after failover.

Reconnect by inode id

When the bundled client reconnects, it has to rebuild its open fids on the new session. Re-walking the original path does not work if a still-linked file was renamed. Trebind binds a fresh fid directly to an inode by its server-assigned id rather than by a path, so linked open handles survive a rename or hardlink shuffle across the outage.

Trebind  fid[4] inode_id[8] root_inode[8] flags[1] uname[s] n_uname[4]
Rrebind  qid[13]

root_inode preserves the immutable attach root that governed the old fid, while uname and n_uname reconstruct its credentials. A root-self request first re-establishes an attach root by stable inode id, so renaming that directory during the outage does not strand the session. The REPLAY flag marks automatic session recovery. OPENED is valid only together with REPLAY; it marks a fid whose open state the client will restore after rebinding it.

The server verifies that a rebound inode is still available to the session governed by its declared attach root. Reconnect does not resurrect an inode whose final namespace link has been removed.

An open-unlinked handle therefore belongs to its current connection. It continues to work after the unlink, as POSIX requires, but a later disconnect or failover makes it unrecoverable. Replay is an indivisible session transition: if any recorded open handle can no longer be rebound, the client permanently fails the entire logical session with ESTALE instead of publishing a partial recovery. This is session-wide, not a per-fid error; the caller must establish a new client or mount.

Durability-verified fsync

A plain 9P Tfsync returns Rfsync once the server has flushed, but it says nothing about whether the connection's writes are still the live durable copy. After a failover, an old leader can flush and report success for writes that a new leader has already superseded. The private dialect closes that gap by tying fsync to a durability lineage token.

At connect time, the client asks for the connection's lineage with Tgetlineage, off the hot path. Rgetlineage returns both the durability token and the current HA writer epoch. The client tracks the token of its oldest un-fsync'd write and presents it on Tfsyncdur; it copies the writer epoch into the operation envelopes described above. The server flushes and replies Rfsync only if the token is still the live durable lineage; otherwise it returns Rlerror(ESTALE). A successful fsync therefore implies that every write the client acknowledged before it is durable and will survive a crash or failover. A 0 token means the client has nothing un-fsync'd, and a 0 writer epoch identifies a standalone server.

Tgetlineage                          (empty body)
Rgetlineage  token[8] writer_epoch[8]

Tfsyncdur    fid[4] datasync[4] token[8]
             -> Rfsync            if token is the live durable lineage
             -> Rlerror(ESTALE)   otherwise

This is the protocol-level half of the durability and consistency guarantee, and it is what lets the bundled client verify fsync durability through a high-availability failover. With ignore_fsync, the administrator deliberately disables the barrier: Tfsyncdur returns success without flushing or checking lineage.

Atomic fallocate

The private dialect adds Tfallocate, since standard 9P2000.L has no message for Linux fallocate(2). The request targets an open file fid and carries the byte range and Linux mode bits. The server applies extent edits, logical-size growth, quota accounting, timestamps, and cached directory metadata in one transaction.

Tfallocate  fid[4] offset[8] length[8] mode[4]
Rfallocate  (empty body)

Supported modes are plain allocation (0), PUNCH_HOLE | KEEP_SIZE, and ZERO_RANGE with optional KEEP_SIZE. Zero-filled ranges are represented as sparse holes while logical growth remains quota-accounted. Standalone keep-size preallocation and the collapse, insert, unshare, and write-zeroes modes return EOPNOTSUPP.

Message reference

Every private message and its 9P type id. Reply types are paired with their request, and all are available only in 9P2000.L.Z.

The operation envelope is not a separate message. It is a 25-byte framing field carried by the request types listed under idempotent retries.

Next steps