EKOSdocs
Docs / Operations Operate

Storage

Sizing, layout and scaling the ledger in production.

An index lives under .ekos/ beside the sources, or, for a project in the EKOS home, under <home>/data/<tenant>/<project>/state/ next to its ekos.toml. Rough guidance from real runs:

Workspace Objects Ledger entries
EKOS's own repository (multi-language) ~16,700 ~204,000

Ledger size grows with history because it is append-only; identical objects are skipped on re-commit.

Choosing a layout

  • New workspaces: the fact engine (default).
  • Existing SQLite workspaces: keep them; migrate deliberately with ekos ledger migrate --v3.
  • Very large or long-lived ledgers: partition by dimension and time bucket ([storage.partition]).
  • Multiple machines: distributed mode — a coordinator, compile workers (lease/heartbeat protocol), query workers, and a gateway that merges per-shard statistics and fuses rankings. The object-store read path was verified against MinIO and a 95-partition workspace.

Disk

Keep the state directory on local SSD storage: the fact engine memory-maps segments, and a commit's synchronous writes wait on the disk's latency. Measured in the cloud runs (devlog_315/316):

Disk Commit cost
Azure premium managed disk a 142k-object commit took 48 minutes, even at P20 — a 32 GiB premium disk is only a P4 (120 IOPS); keep a VM's home_size_gb at 128 or more
Azure Files NFS (Container Apps) about 90–105 ms per object: 142k objects in about 3½ hours, 2,983 objects in about 5 minutes

Most of those 142k objects came from data scripts; excluding them by name in [observe] ignore-patterns gave 2,983 objects with the same tables and relationships. Clear .ekos/artifacts/ with ekos clean if only the artifact cache is large.