How EKOS answers a BIRD question

EKOS turns a plain-English question into SQL in three steps. It compiles each database once into a knowledge ledger and a data profile. It builds one prompt from that knowledge. Then it lets the model answer, runs the SQL read-only, and sends it back once if the result looks wrong. On BIRD Mini-Dev, with Claude Haiku 5.5, this scored 63.9% on 302 hold-out questions, against 53.3% for the same model given raw DDL.

At a glance

EKOS does its main work before any question is asked. The model sees the result as one prompt, and its SQL is checked once before it counts.

Input: what EKOS receives

For each question, EKOS receives only the question text and the name of its database. It never sees the gold SQL or the expected result while it generates an answer.

The EKOS cells ran without BIRD’s evidence hints, which makes the task harder than the official leaderboard setting. A bug also meant no dictionary description reached the prompt in any measured run. It has since been fixed, but that cell has not been measured yet.

Stage 1 — prepare: compile each database once

ekos eval bench prepare --suite bird-mini-dev runs once, before any question is asked. It makes no model calls. For each of the 11 databases it produces three things.

1. A compiled ledger

1. EKOS reads the table list, columns, types, primary keys and foreign keys from SQLite’s own catalogue.
2. It writes them out as a schema.sql file of CREATE TABLE statements, plus one COMMENT ON COLUMN per dictionary row.
3. It creates an EKOS project around that file and runs the normal compiler pipeline: build → recover → resolve → compile → commit. The SQL analyzer pass parses the DDL with the postgres dialect.
4. The result is an append-only ledger:
   • Table objects, each with its columns, types, primary keys and descriptions;
   • ForeignKey relationships, each recorded as a join path such as qualifying.driverId = drivers.driverId.

This is the same compiler EKOS runs on any enterprise codebase. Nothing in it is specific to BIRD.

2. A column profile

Stored in data/values/<db>.profile.json. For every table EKOS stores the row count. For every column it stores:

• the null count;
• the empty-string count (profile v2);
• the distinct count, and whether the column is unique;
• up to three example values, never blank;
• a list-valued flag. It is set when values such as Artifact,Creature are comma-separated lists whose elements also occur on their own.

3. A value index

Stored in data/values/<db>.json. It holds the distinct text values of every column with at most 5,000 distinct values of up to 80 characters. It is used to spot values the question names.

Profiles and the value index are read straight from the data and kept outside the ledger. The ledger holds only what the schema says.

Stage 2 — the prompt: one message, built from the ledger

For each question EKOS builds a single user message. It has no system prompt, no conversation history and no examples. The message is a template with three slots: the engine (SQLite), the schema text and the question.

The schema text (“the EKOS pack”) is rendered from the ledger, not copied from the database:

1. Each Table object is shown as a CREATE TABLE, preceded by its row count.
2. Each column carries a comment with its profile: nulls, empty strings, distinct count or “unique”, list-column advice, and examples.
3. Each ForeignKey relationship becomes a line under “Join paths”.
4. Values the question names are appended, with every column each value occurs in. A value of one word counts only when the question spells it as stored or quotes it.

An excerpt of what the model saw for formula_1 (cell G2):

-- 6967 rows
CREATE TABLE `qualifying` (
    `qualifyId` INTEGER, -- unique (6967 values); e.g. '23', '24', '25'
    `raceId` INTEGER, -- 319 distinct; e.g. '19', '20', '21'
    `driverId` INTEGER, -- 151 distinct; e.g. '13', '8', '5'
    `q1` TEXT, -- 109 null; 6283 distinct; e.g. '1:35.347', '1:35.645', '1:35.227'
    `q3` TEXT, -- 4935 null; 28 empty strings ('' also means missing: exclude with <> '' as well as IS NOT NULL); 1959 distinct; e.g. '1:35.748', '1:36.230', '1:36.613'
    PRIMARY KEY (`qualifyId`)
);

-- Join paths (foreign keys):
--   qualifying.driverId = drivers.driverId
--   qualifying.raceId = races.raceId

-- Values in the database that the question mentions:
--   'Artifact' in `cards`.`types` (list column: `types` LIKE '%Artifact%')

The table and join lines are verbatim. Some qualifying columns are left out to save space. The last line is illustrative: it shows the value-lookup format for a card_games question. In the measured runs the column comments had no dictionary text; since the fix, each comment starts with BIRD’s description. The whole formula_1 pack is about 14 KB.

The instructions (prompt ekos-v1) are BIRD’s published baseline prompt plus five rules:

• Return exactly the columns the question asks for, in its order, and nothing else.
• Return all of the information asked for, nothing missing and nothing extra.
• Before counting or averaging, check each table’s grain from its row and distinct counts. Count DISTINCT entities when an entity spans several rows.
• Use the exact spelling of values shown in the comments. Match a comma-separated list column with LIKE '%value%'.
• Keep computations in the form the question states, e.g. a rate as a count divided by a count.

The answer format asks for the SQL first, in one ```sql block, then at most three sentences of explanation. With the SQL first, an answer cut short by the token limit still contains its query.

Stage 3 — generating the SQL

The model writes the SQL. EKOS does not build the query from rules or templates. Its job is to give the model the right facts, then check what comes back.

1. Call the model. One request goes to Claude Haiku 5.5 through OpenCode Zen, at temperature 0 and up to 4,096 output tokens. Each call is logged with its processed and billed tokens. A cached answer is counted as cached.
2. Extract the SQL. EKOS takes the last ```sql block. Failing that, it takes the last code block, then the text from the first SELECT or WITH. Trailing semicolons are dropped.
3. Run it, read-only. The query runs on the SQLite file, opened read-only, with a 30-second deadline and a row cap.
4. Judge the result. The query is sent back to the model when:
   • there was no SQL block in the answer;
   • the query failed: the database error is sent back, e.g. no such column: T2.q3;
   • it returned no rows: “If the question expects rows, the filters, joins or value spellings are likely wrong”;
   • it returned a single NULL.
5. One repair round (r = 1). The second request carries the first prompt, the model’s first answer and the feedback, and asks for corrected SQL. Whatever comes back is final, right or wrong.
6. Score, after the answer is final. The predicted and gold SQL both run. Execution accuracy (EX) compares their result sets, following BIRD’s official scorer: order ignored, duplicates collapsed, 1 == 1.0.

The feedback never contains the gold SQL or the expected result. It describes only the query’s own outcome: an error, an empty result or a lone NULL.

Voting was tried as cell H. It drew three candidates from three renderings of the context and kept the result most of them shared. It scored 64.1% against G’s 65.7% on the tune split, at 1.9× the cost, so it was dropped.

Every query EKOS issues

All database access is read-only. The model never connects to the database itself.

None of the queries EKOS writes itself depends on the question. The only question-specific step is the value lookup, a word match against the prepared index in memory.

Tools: what is used, and what is not

What is not in the loop:

• No agent and no tool use. The model gets one prompt and, at most, one repair message. It cannot browse the schema or run queries itself.
• No MCP server, EKL or semantic search at answer time. The pack is rendered straight from the ledger: every table, not a retrieved subset. BIRD databases are small enough for this.
• No few-shot examples. BIRD’s train split is not used yet.
• No evidence hints, no fine-tuning, no larger model.

Inspect the exact schema text any cell shows the model with ekos eval bench context --suite bird-mini-dev --cell zen-haiku/g2 --db formula_1. It makes no model call.

Guards that keep the score honest

• Pinned inputs. The question file and the database archive are checked against SHA-256 hashes in suite.toml.
• Allowlist. Only the database files and their dictionaries leave the archive. Each project’s source is exactly one file, schema.sql.
• Leakage scan. After compiling, EKOS searches every ledger and project source for any question, gold SQL or evidence text. A hit fails prepare.
• Tune / hold-out split. The split is fixed (seed 240, 40/60, stratified). All tuning used the 198 tune questions. The 302 hold-out questions were run once for the final comparison.
• Contamination flag. A cell whose configuration changed after it was run on hold-out is marked contaminated in its report.
• Official-scorer parity. On 704 predictions, our EX verdicts were identical to those of BIRD’s own Python evaluator.

What it cost and what it scored

On the 302 hold-out questions, the same model gained 10.6 points from what EKOS showed it and checked. Both cells used Claude Haiku 5.5 with a 4,096-token output limit and no evidence hints.

G − A-4k is +10.6 points, with a 95% CI of +4.3 to +16.9 and McNemar p = 0.0014. G costs about $0.001 per question. Its input is three times larger, because the profiles are in the prompt.

* G2 was run on hold-out after we had analysed G’s hold-out failures, so it is not a clean hold-out number.

What moved the score, measured as paired differences on the tune split:

• Column profiles: +11.1 points. This also includes raising the output cap from 1,024 to 4,096 tokens.
• Prompt rules and result checks: +11.6 points.
• Value lookup: not significant. It changed the score by −2.5 when added and +2.0 inside G.
• Counting empty strings as missing: formula_1 went from 18 to 26 of 33 on hold-out.

Every question, generated query and token count is in the full report, BIRD Mini-Dev on EKOS. Design notes: RFC 0240 and RFC 0241 in the EKOS repository.