Troubleshooting

Diagnostic steps for common ZeroFS problems: mount failures, startup errors, S3 connectivity errors, cache and configuration issues, encryption passwords, NBD devices, runtime storage errors, and debug logging.

Common Issues

Mount Failures

# Error: mount.nfs: access denied by server
# Solution: Check NFS port and permissions

# Verify ZeroFS is running
ps aux | grep zerofs

# Check if port is listening
netstat -tlnp | grep 2049
# or
ss -tlnp | grep 2049

# Try mounting with verbose output
sudo mount -v -t nfs -o vers=3,tcp,port=2049 127.0.0.1:/ /mnt/zerofs

S3 Connectivity Issues

# Error: InvalidAccessKeyId or SignatureDoesNotMatch
# Solution: Verify AWS credentials in config

# Test credentials directly
aws s3 ls s3://your-bucket/ --debug

# Check configuration file
cat zerofs.toml | grep -A 5 '\[aws\]'

# Verify values are correct:
# [aws]
# access_key_id = "your-key"
# secret_access_key = "your-secret"
# region = "us-east-1"
# endpoint = "https://s3.wasabisys.com"  # For S3-compatible

Cache Problems

# Error: creating foyer hybrid cache dir at <path>
#    or: creating parts cache dir at <path>
# Solution: Check permissions and disk space

# Verify cache directory from config
grep -A 3 '\[cache\]' zerofs.toml

# Check if directory exists
ls -la /var/cache/zerofs

# Check disk space
df -h /var/cache/zerofs

# Fix permissions
sudo mkdir -p /var/cache/zerofs
sudo chown $USER:$USER /var/cache/zerofs
chmod 755 /var/cache/zerofs

# Verify write access
touch /var/cache/zerofs/test && rm /var/cache/zerofs/test

Startup Errors

Conditional Write Check Fails

Startup fails with:

Storage provider does not support conditional writes. PutMode::Create succeeded when it should have failed. This feature is required for fencing in ZeroFS.

Before every read-write server start (zerofs run without --read-only/--checkpoint), ZeroFS writes a probe object named .zerofs_compatibility_test_<uuid> to the database path and attempts a second, create-only write to the same key. The store must reject the second write with an already-exists error. A store that accepts it lacks put-if-not-exists support, which fencing requires, so ZeroFS refuses to start. A message starting with Storage provider precondition check failed means the probe itself returned an unexpected error; the underlying store error follows the colon.

For S3-compatible stores without conditional put support, set conditional_put to a Redis URL:

[aws]
conditional_put = "redis://localhost:6379"

See the Conditional Put Support section under Storage Backends for configuration and Conditional Writes and Fencing for the fencing model.

Config Parse Fails on wal_enabled or max_unflushed_gb

After an upgrade, startup fails with an unknown-field error pointing at the offending line:

Failed to parse config file: zerofs.toml: TOML parse error at line 10, column 1
   |
10 | wal_enabled = true
   | ^^^^^^^^^^^
unknown field `wal_enabled`, expected one of `l0_max_ssts`, `max_concurrent_compactions`, `flush_interval_secs`, `sync_writes`

The [lsm] section rejects keys it does not define, and two keys were removed:

Delete the keys from the config file. Neither has a replacement setting.

Volume from an Earlier Release Refused on Open

Startup fails with:

segment extractor configuration mismatch (persisted: Some("zerofs-meta-chunk-v1"), configured: Some("zerofs-meta-extent-v1"))

ZeroFS stamps a key-layout name into the database manifest at creation and checks it on every open. This release writes and requires zerofs-meta-extent-v1. Volumes created before file contents moved into segment objects carry the previous name zerofs-meta-chunk-v1, or no name at all (persisted: None), and are refused. The check prevents opening a database whose existing keys the configured layout would misroute.

This release contains no migration tool for these volumes. They remain readable and writable by the release that created them.

compactor Subcommand and --no-compactor Flag Not Recognized

Scripts or systemd units from an earlier release fail with:

error: unrecognized subcommand 'compactor'

or:

error: unexpected argument '--no-compactor' found

The standalone compaction worker no longer exists. Metadata compaction always runs inside zerofs run, and space reclamation for file data is handled by the in-process segment garbage collector. Remove zerofs compactor service units and the --no-compactor flag; no replacement invocation is needed.

Related output change for scripts: zerofs debug list-keys prints file-data keys as extent_index where earlier releases printed chunk_index.

Configuration Issues

Missing Configuration File

# Error: Failed to load config
# Solution: Create or specify config file

# Initialize a new config
zerofs init

# Run with specific config file
zerofs run --config /path/to/zerofs.toml

# Or use short form
zerofs run -c zerofs.toml

Encryption Issues

Password Problems

# Wrong password or corrupted data

# Verify password in config file:
[storage]
url = "s3://bucket/path"
encryption_password = "correct-password"

# Values undergo environment variable expansion.
# Write a literal $ as $$; this sets the password to p@$w0rd!
encryption_password = 'p@$$w0rd!'

# Or use environment variable substitution:
encryption_password = "${ZEROFS_PASSWORD}"

Changing Passwords

# Change password using the CLI
# The new password is read from stdin
echo "new-password" | zerofs change-password --config zerofs.toml

# After changing, update your config file:
# [storage]
# encryption_password = "new-password"

NBD Issues

Device Connection

# Error: nbd-client connection failed
# Solution: Check NBD configuration and availability

# Verify NBD is configured in zerofs.toml:
# [servers.nbd]
# addresses = ["127.0.0.1:10809"]

# Check if port is listening
ss -tlnp | grep 10809

# Ensure NBD module is loaded (Linux)
sudo modprobe nbd
lsmod | grep nbd

# Create NBD device file
sudo mkdir -p /mnt/zerofs/.nbd
sudo truncate -s 10G /mnt/zerofs/.nbd/my-device

# Connect to the device
sudo nbd-client 127.0.0.1 10809 /dev/nbd0 -N my-device

Runtime Errors

Reads Stall with Repeated Short-Body Warnings

Reads hang while the log repeats:

object store returned a short body; re-fetching

ZeroFS checks the body of every object store read against the length the store reported. When the body comes back shorter, ZeroFS discards it and fetches again rather than return truncated data. There is no retry limit: the read blocks until a full body arrives, with backoff starting at 100 ms, doubling per attempt, and capping at 800 ms. The warning fields report the object location, the expected and received byte counts, and the attempt number.

Repeated warnings for the same object mean something between ZeroFS and the storage backend truncates responses, either a proxy or the provider itself. Reads recover without intervention once full responses arrive.

Process Exits with a Fatal Write Error

The process exits with code 1 after logging:

Fatal write error, exiting: <underlying error>

Any failed write to the database terminates the process: after a write failure, the database state is unknown, and a restart rebuilds from the last committed state. Run ZeroFS under a supervisor that restarts it; the systemd unit under System Integration in the configuration guide sets Restart=always.

The text after the colon is the underlying storage error. Diagnose that error; the exit is the recovery mechanism, not the fault.

Debugging Techniques

Enable Detailed Logging

# Maximum verbosity
export RUST_LOG='zerofs=trace,slatedb=debug'

# Log to file for analysis
export RUST_LOG='zerofs=debug'
zerofs run -c zerofs.toml 2>&1 | tee zerofs.log

# Filter specific components
export RUST_LOG='zerofs::nfs=trace,zerofs::nbd=debug'

debug list-keys Fences a Running Instance

zerofs debug list-keys --config zerofs.toml prints every key in the database with a per-type summary. It opens the database in read-write mode, and fencing allows one writer per database. When it opens a database that a running ZeroFS instance is serving, it takes over as the writer: the instance's next write fails and the instance exits with Fatal write error, exiting.

Stop the instance before running debug list-keys against its database, or accept that a supervisor with Restart=always restarts the instance after the exit.