EKOSdocs
Docs / Architecture Learn

Compilation Pipeline

The five stages, what each reads and writes, and how caching works.

The five pipeline stages build, recover, resolve, compile and commit, each with the artifact it writes
The five pipeline stages build, recover, resolve, compile and commit, each with the artifact it writes
Stage Reads Writes Notes
build [[sources]] / observe paths; [[databases]] observation artifacts; File objects; live catalog artifacts redaction runs here; fingerprint cache skips unchanged trees; live databases are read once per build outside that cache, and a failed read fails the build
recover observation artifacts KIR knowledge artifacts analyzers are CompilerPasses scheduled by dependency; --parallel
resolve KIR identity decisions exact merges; fuzzy → unconfirmed SameAs only when names agree word for word; live and structurally keyed objects never merge; conflicts fail unless --force; same-name groups split across the database and different languages' code (a table gl beside a Perl package LedgerSMB::GL) are not conflicts
compile resolved KIR CKM (ckm/model.json.zst) derives concentration risks and ConflictingEvidence; pass output cached by version + inputs; a live catalog keeps only its newest read
commit CKM ledger entries, indexes; integrity index then the post-commit steps below

Passes

A CompilerPass declares a name, dependencies, a version string and cache inputs. The PassManager orders passes by dependency and skips a pass whose version and inputs are unchanged — the skipping pass (cached) log line. Bump a pass's version when its behaviour changes, otherwise stale output is reused.

Analyzers

Analyzer Source
sql_analyzer, sql_transform_analyzer DDL (statement by statement when a whole file fails to parse); SELECT/VIEW/procedures into the Transformation IR
view_analyzer, plpgsql_analyzer, plsql_analyzer, trigger_analyzer views, PL/pgSQL and PL/SQL routines with per-statement reads/writes/calls, triggers
schema_history dated migration scripts → change sets and classified changes
oracle_catalog_analyzer live Oracle, PostgreSQL and ClickHouse catalogs → origin: live objects
pentaho_analyzer .ktr / .kjb
rust_analyzer, python_analyzer, javascript_analyzer, elixir_analyzer, perl_analyzer real AST parsing, symbols and Calls
java_analyzer Java types and methods, embedded SQL, the iDempiere/ADempiere application model
dbt_analyzer dbt models and schema.yml / sources.yml
git_analyzer, github_analyzer, confluence_analyzer history and ticket ids, issues, wiki
local_docs_analyzer, document_semantics_analyzer documents; optional LLM semantics
crate_topology_analyzer, cicd_analyzer, dependency_analyzer, crypto_analyzer structure and dependencies

Post-commit steps

These run after the ledger is written and are not compiler passes. In order:

  1. subsystem rollups;
  2. data-lineage linking (transformation steps to tables);
  3. routine linking (routines to the tables they read and write and the routines they call) and schema changes to the tables they alter;
  4. [semantics] business-semantics synthesis (opt-in, RFC 0170);
  5. [conflicts] (on by default): label mismatches, the code compared with the live database, and the list of current conflicts;
  6. write behaviour per table, and tickets linked to what cites them;
  7. [xref]: the cross-reference integrity index, written to a file, never to the ledger;
  8. [llm-description] (opt-in), then extension steps, then [embeddings] vectors (opt-in).

Run logs

Every pipeline command, and every command that changes something, writes a JSON-lines run log (to EKOS_RUN_LOG or the state directory's runs/): one line per step with its counts, warnings redacted, and a last llm line with the number of model calls the run made. On the iDempiere testbed, all five pipeline commands logged a model-call count of 0. The console stores one log per run and shows it as a steps table.

Determinism

Same inputs, same outputs. Ids are derived, not random, where identity matters: relationship and evidence ids are derived from their content, so a re-run over unchanged input writes no new versions. Extensions that change observer output bump logic_version so the build cache invalidates.