Backup and restore
FeltDB supports verified offline bundles, write-consistent online bundles, and point-in-time selection at completed backup boundaries. It does not synthesize a recovery point between backups; backup frequency therefore determines the recovery-point objective.
Create and verify
feltdb-server backup create \
--data /var/lib/feltdb/state.log \
--keys /var/lib/feltdb/api-keys.json \
--audit /var/lib/feltdb/state.audit.log \
--output /backups/feltdb-2026-08-11
feltdb-server backup verify /backups/feltdb-2026-08-11
Paths may be absolute, ./-relative, or bare relative (feltdb-server backup create --output backup);
all resolve against the working directory and behave identically, and missing parent directories are
created. feltdb-server backup <path> is shorthand for backup create --output <path>, and
backup verify <path> for backup verify --archive <path>.
create reports success only after the bundle verifies independently of the code that wrote it; the
exit status is the outcome. With --json, both commands print the result:
{
"operation": "create",
"status": "ok",
"path": "/backups/feltdb-2026-08-11",
"backup_id": "b239c69f…",
"source": { "data": "/var/lib/feltdb/state.log", "state_sha256": "d735f346…", "server_version": "0.2.0", "durable_format": 2 },
"format": 1,
"created_ms": 1790339429136,
"files": 3,
"bytes_written": 1642,
"verification": "verified",
"duration_ms": 4
}
backup_id is the SHA-256 of the manifest and identifies the bundle. Verification checks that the
manifest is readable and in a supported format, that every entry's length and SHA-256 match, that no
file absent from the manifest is present, and that the bundle restores into a database this build can
open (on a private copy). A bundle that fails any of these is reported failed with exit status 1 and
is never described as valid. feltdb-server doctor --data <log> --backup <bundle> runs the same checks
as part of a health report.
Creation writes to a .partial sibling and atomically renames only after every file and the manifest
are durable. Existing output and stale partial directories cause failure. The format includes the
canonical state log, checkpoint, replication acknowledgements, membership and recovery proposal,
lease promises, API-key hashes, audit log, and content-addressed object tree. Missing optional files
are omitted explicitly from the manifest.
Online backup
Create an API key with the cluster:write scope and request a backup from the running server:
curl -fsS -X POST http://127.0.0.1:7700/admin/backups \
-H "Authorization: Bearer $FELTDB_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"output":"/backups/feltdb-2026-08-11-live"}'
The server blocks mutations while it exports a hash-verified state snapshot with its causal version vector and copies membership, lease, authentication, audit, and content artifacts. Reads remain available. The output path is resolved on the server and must not already exist. A successful response is the completed manifest; verify the bundle before transferring or restoring it.
Online bundles contain database.snapshot.json in place of a live state log. Restore verifies the
bundle and installs this snapshot into a pristine native state log, including its causal metadata.
Restore to a point in time
Keep completed bundles as direct children of a catalog directory, then supply a Unix timestamp in milliseconds:
feltdb-server backup restore \
--archive /backups/feltdb-catalog \
--at-ms 1786442400000 \
--output /var/lib/feltdb-restored
FeltDB selects the newest completed bundle whose manifest timestamp is at or before the requested time. It verifies every manifest-bearing bundle in the catalog before selection. A corrupt candidate or the absence of an eligible recovery point fails the restore; FeltDB never silently falls back to older customer state. The selected snapshot boundary, rather than individual row wall-clock values, defines the recovery point and preserves causal metadata after compaction.
Restore
feltdb-server backup restore \
--archive /backups/feltdb-2026-08-11 \
--output /var/lib/feltdb-restored
feltdb-server --data /var/lib/feltdb-restored/files/state.log \
--keys /var/lib/feltdb-restored/files/api-keys.json \
--audit /var/lib/feltdb-restored/files/audit.log
Restore verifies the format, path safety, byte lengths, and SHA-256 of every entry before staging any installation. It refuses to overwrite an existing target. Corrupt, partial, traversal-bearing, and unsupported-format bundles are never installed.
Recovery objectives
Recovery point objective (RPO): your backup interval. FeltDB restores to
the most recent backup and does not create recovery points between backups.
Backing up every hour bounds the loss to an hour of writes; daily backups bound
it to a day. Schedule feltdb-server backup create, or the online
/admin/backups API (cluster:write scope), at the interval you can afford to
lose.
Recovery time objective (RTO): not characterized beyond the test configuration.
On test configurations (single-node, moderate state size), measured RTO is < 1 second including verification. Production RTO scales with:
- Total state size
- Disk I/O throughput
- Network transfer time (for multi-region setups)
Benchmark your environment:
# Create a full backup
time feltdb-server backup create --data state.log --keys api-keys.json --audit state.audit.log --output test-backup
# Verify integrity
time feltdb-server backup verify --archive test-backup
# Measure restore
rm -rf test-restore
time feltdb-server backup restore --archive test-backup --output test-restore
Document and alert on RTO regressions. If measured RTO exceeds 5 minutes, review state size, disk configuration, and network capacity.
Drill policy
Run a restore drill at least monthly and before every production release. Record bundle size, creation time, verification time, restore time, restored state count, content hash results, authentication check, subscription replay, and workflow/agent recovery.
.github/workflows/disaster-recovery.yml runs the authenticated offline and online clean-restore
drill monthly and supports an operator-triggered run; it retains the test report for 90 days.