Docs / Operations Operate
Backup & Recovery
Back up the EKOS home to a file or object storage, restore it, and recover from torn writes and stale indexes.
Backup
ekos home backup (RFC 0183) writes every project's config and compiled data, and the console's database, to one .tar.zst file:
ekos home backup # → <home>/backups/ekos-backup-<time>.tar.zst
ekos --tenant acme home backup # one tenant's projects only
ekos home backup --to s3://my-bucket/ekos/ # or gs://…, https://<account>.blob.core.windows.net/<container>/ekos/
- Each project is copied under its ledger's write lock. If a pipeline stage is writing, the backup stops with that project's name and writes nothing; reads, including agents' queries, are not interrupted.
- The console database is copied as a consistent snapshot while the console runs.
--to(RFC 0232; the image, or a build with theobject-storagefeature) uploads to S3, Azure Blob Storage or Google Cloud Storage with the machine's cloud identity — no key on the command line — and removes the local copy.- Not in a backup: the sources, the binary, the trash. The archive is not encrypted.
- On one server there is no schedule: run it from cron, and copy the file off the machine. The cloud deployments back up to their bucket every night.
A plain workspace outside the home is backed up by copying .ekos/ and ekos.toml after the indexer finishes. Artifacts and caches (artifacts/, llm-cache/, embed-cache/) are rebuildable — the LLM cache is worth keeping to avoid repeating paid calls.
Restore
ekos home restore ekos-backup-20261013T101500Z.tar.zst
ekos home restore --from s3://my-bucket/ekos/ --force # the newest backup at that location
- Without
--forcenothing that exists is replaced: if any project in the archive, or the console database, is already there, the restore stops and changes nothing. --forcefirst moves each existing project, and the console database, to the home'strash/. Nothing is deleted. It is needed even on a fresh console volume, because the console created its own empty database at start; restart the console afterwards.--tenant acmerestores that tenant's projects only; role bindings are in the console database, which a per-tenant archive does not include.- A project's config holds its sources' path: restore where the sources are mounted at the same path, or edit
ekos.toml. - First-start restore: a console with a backup location (
backup_url, which the cloud deployments set) that starts over an empty home restores the newest backup there before opening its database. An empty location starts empty; any other failure stops the start.
Recovery
| Situation | Action |
|---|---|
| Crash during a write (torn active-segment tail) | ekos ledger repair — verifies sealed segments, heals the tail |
| "search index schema is stale" or "index runs need rebuilding" from a read-only open | ekos ledger repair (or any writable command), then reopen |
| Search results missing after a failed commit | ekos ledger repair |
| Lost the state directory entirely | Restore a backup, or rebuild from sources: the pipeline is deterministic, but LLM-derived text and review decisions (events) will not be reproduced |
Review decisions and sign-offs are events in the ledger, so back up the ledger, not just the sources.