Fact Ledger
The append-only store: the fact engine, its segments, indexes and history.
The Semantic Knowledge Ledger is append-only. Three engines exist:
| Engine | Details |
|---|---|
| Fact engine (RFC 0016) | default for new workspaces; each entry decomposed into (entity, attribute, position, value) tuples; tantivy indexes; mmap reads; sealed segments verified on open |
| SQLite v2 (RFC 0015) | zstd-compressed payloads, compact integer/blob columns, contentless FTS |
| SQLite v1 | legacy; readable and writable until migrated |
Entries carry an id, a type, a payload and a written_at time. Point-in-time state is reconstructed by reading entries up to a timestamp — which is what ekos diff, ekos_state at=… and EKL AS OF do.
Reads across processes
A long-lived read-only handle (ekos mcp serve) does not reopen the ledger when a separate writer commits. On every call it does one HEAD read and one stat of the active segment; if the active segment grew it decodes only the appended tail, and it reloads tantivy's reader. A seal, run flush or dictionary change falls back to a cold rebuild through the same code a fresh open uses, so the two are identical. No lock is taken on the read side. On a 5,000-object ledger this measured ~27 µs unchanged and ~106 µs after one appended batch, against ~851 µs for the cold reopen it replaces (RFC 0112).
Point lookups
Each run in the fact engine's index carries a Bloom filter of the entity ids it holds, so a by-id read skips runs that cannot contain the id. Within a run, only the block holding the id is decoded (RFC 0168). On a real ledger this took a point lookup from 1.43 ms to 0.171 ms. See Performance.
Integrity and repair
- Sealed segments are checksummed;
ekos ledger repairverifies each and self-heals a torn active-segment tail or a stale search-index run. - A read-only open cannot rebuild a stale search index and reports it; open once writable (any writable command, or
ekos ledger repair) to heal it.
Provenance
A WriteContext records the run id and stage of every write; ekos ledger audit and ekos_audit read it back. In the fact engine it lives in a provenance.jsonl sidecar.
Scale
Partitioned storage (RFC 0111) splits the ledger by dimension and time bucket; distributed mode (RFC 0113) adds a coordinator, compile workers and query workers. See Storage.