BIRD Mini-Dev on EKOS

Text-to-SQL on the BIRD Mini-Dev benchmark (release “Mini-Dev 1024”, SQLite, 500 questions over 11 databases), answered by Claude Haiku 5.5 through OpenCode Zen, with and without the knowledge EKOS compiles from each database. Every number below is read from the saved run reports and the model router’s billing log.

1 · Source databases and what EKOS compiled from them

Each database is its own EKOS project, built by ekos eval bench prepare. The pipeline reads the database’s catalogue and its dictionary (database_description/*.csv): one generated schema.sql per project, holding CREATE TABLE statements plus one COMMENT ON COLUMN per dictionary row. That file goes through the normal pipeline (build → recover → resolve → compile → commit). Question files and gold SQL never leave the archive. After the build, every ledger is scanned for the text of all 500 questions, gold statements and evidence hints; the scan was clean (1,349 texts, no hit).

The schema text below is exactly what a cell showed the model, printed by ekos eval bench context. Three views per database: the EKOS pack as the measured G2 runs saw it (tables, keys and join paths from the ledger, plus column profiles: rows, nulls, empty strings, distinct values, examples, list columns); the same pack after the v1.2.0 analyzer fix, now carrying the dictionary’s descriptions; and the raw DDL that cells A and A-4k got.

2 · Every hold-out question, the query EKOS generated, and its result

302 hold-out questions. The SQL is cell G2’s answer; ✓ means its result set equals the gold result set (execution accuracy, as BIRD’s official scorer computes it). The A-4k and G columns show whether those cells got the same question right. Each prompt was the template below, with the database’s schema text from section 1, the matched values, and the question. No cell here used BIRD’s evidence hint.

Task Overview:
You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.

Database Engine:
{engine}

Database Schema:
{schema}This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints. Column comments give each column's meaning, its null and distinct counts, and example values.

Question:
{question_block}

Instructions:
- Return exactly the columns the question asks for, in the order it asks for them, and nothing else.
- The generated query should return all of the information asked in the question without any missing or extra information.
- Before counting or averaging, check the grain of each table from its row and distinct counts: when an entity appears in several rows (e.g. one row per printing or per year), count DISTINCT entities if the question is about entities.
- Use the exact spelling of values shown in the schema comments; a comma-separated list column is matched with LIKE '%value%'.
- Keep computations in the form the question states (e.g. a rate as a count divided by a count).

Output Format:
Give the SQL query first, in one code block:
```sql
-- Your SQL query
```
Then explain it in at most three sentences.

3 · Tokens and money

All metered calls went to Claude Haiku 5.5 through OpenCode Zen ($0.10 per million input tokens, $0.50 per million output tokens). The billing log (data/llm/llm-usage.jsonl) records each call that reached the provider. Repeated identical prompts were served from EKOS’s LLM cache and are not billed, so a report’s processed tokens can exceed its billed tokens.

Every run

Cost covers billed calls only. “$ per 1k” is the cost of 1,000 questions at that run’s rate. Runs before the cache accounting was added (the first five) show processed tokens only.

4 · Average price per question

The EKOS prompt is about three times longer on input (1,509,241 against 572,686 input tokens for 302 questions), because the profiles describe every column. Output is the same size, and output carries five times the per-token price, so the EKOS cell costs about 1.5× as much per question: about one tenth of a cent.

5 · Accuracy and speed on the hold-out split

How the EKOS cell was chosen (tune split, 198 questions)

Everything was tuned on a separate tune split (40% of each database × difficulty stratum, fixed seed). The hold-out split was used once per cell.

A = raw DDL with a 1,024-token output limit · E = EKOS pack + column profiles · F = E + value lookup · G = F + the ekos-v1 prompt + result checks · G-nv = G without value lookup · H = G with three varied-context candidates (voting) · G2 = G + empty-string profiles.

6 · Code added to run the benchmark

6,927 lines in 32 files, all in the public EKOS repository (alexeyban/EKOS, v1.2.0). Nothing outside EKOS is needed to run a benchmark. The official BIRD scorer is used only by the parity test, from a separate checkout, and is never copied into the repository.

7 · Configuration

The suite pins its downloads by SHA-256. A cell is one TOML file: what the model is shown, the prompt, the generation strategy and the model route.

evals/bench/bird-mini-dev/suite.toml

# RFC 0240 — BIRD Mini-Dev, SQLite, the "Mini-Dev 1024" release (2026-10-28 update).
#
# Questions come from the Hugging Face dataset at a pinned revision; the databases from the
# original Mini-Dev package (the SQLite databases did not change in the update — the README says
# the original dev databases serve the SQLite version). `ekos eval bench prepare` downloads both,
# checks these checksums and extracts only <db>/<db>.sqlite and <db>/database_description/*.csv:
# the package's own question and gold files never leave the archive.
name = "bird-mini-dev"
adapter = "bird"
version = "Mini-Dev 1024"
dialect = "sqlite"
new-dev = false
notes = "birdsql/bird_mini_dev @ 5b256b5a (data/manifest.json revision colleague_review_fixes_20261025)"

[questions]
file = "mini_dev_sqlite.jsonl"
url = "https://huggingface.co/datasets/birdsql/bird_mini_dev/resolve/5b256b5a9c17478f06ed6912612e9a14d17065c8/data/mini_dev_sqlite.jsonl"
sha256 = "926af2a0d1b47e5d0d15a7aa8a34d4721007f4550a881555b6a89496b54a2606"

[databases]
file = "minidev.zip"
url = "https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip"
sha256 = "cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f"
prefix = "minidev/MINIDEV/dev_databases/"

# Fixed seed, stratified by database and difficulty: 40% tune / 60% hold-out (RFC 0240 §6).
[split]
seed = 240
tune-fraction = 0.4

evals/bench/bird-mini-dev/configs/zen-haiku/g2.toml

# RFC 0241 — G with profile v2. Motivated by the formula_1 hold-out analysis; validate on the tune split.
name = "G2"
description = "G with profile v2: empty-string counts, and that '' means missing"
context = "ekos-pack"
evidence = false
value-lookup = true
few-shot = 0
n = 1
r = 1
max-tokens = 4096
ves-iterations = 3
timeout-seconds = 30
profiles = true
profile-version = 2
value-lookup-version = 2
prompt = "ekos-v1"
result-checks = true
vary-context = false

[llm]
# OpenCode Zen, Claude Haiku 5.5 ($0.10 / $0.50 per million tokens). Metered: needs --allow-metered.
provider = "zen"
model = "claude-haiku-5-5"
api-key-env = "OPENCODE_API_KEY"
budget-usd = 0.6

evals/bench/bird-mini-dev/configs/zen-haiku/a-4k.toml

# RFC 0241 — the baseline for the final comparison: cell A with max-tokens 4096, so the difference to G/H is EKOS alone.
name = "A-4k"
description = "Cell A (raw DDL, baseline prompt, no evidence, no lookup) with the output allowance of the RFC 0241 cells"
context = "raw-ddl"
evidence = false
value-lookup = false
few-shot = 0
n = 1
r = 1
max-tokens = 4096
ves-iterations = 3
timeout-seconds = 30

[llm]
# OpenCode Zen, Claude Haiku 5.5 ($0.10 / $0.50 per million tokens). Metered: needs --allow-metered.
provider = "zen"
model = "claude-haiku-5-5"
api-key-env = "OPENCODE_API_KEY"
budget-usd = 1.0

Commands, from the repository root

ekos eval bench prepare --suite bird-mini-dev           # download, verify, extract, build 11 projects, leakage scan
ekos eval bench context --suite bird-mini-dev --cell g2 --db formula_1   # print what the model is shown
ekos eval bench run     --suite bird-mini-dev --cell evals/bench/bird-mini-dev/configs/zen-haiku/g2.toml \
                        --split hold --allow-metered    # needs OPENCODE_API_KEY
ekos eval bench compare <report-g2.json> <report-a-4k.json>
ekos eval bench export  <report.json> --target bird-submission
ekos eval history                                       # every run, newest last

8 · How it works, and what it costs to run

Pipeline

1. Prepare. Check the archive checksums. Extract only <db>.sqlite and its dictionary CSVs. Write one schema.sql per database, from SQLite’s own catalogue and the dictionary. Build one EKOS project per database through the standard pipeline (child processes, with API keys removed, so preparing never calls a model). Record each ledger’s fingerprint, and scan every ledger for leaked benchmark text. Build a column profile and a distinct-value index from the database file; these are kept beside the project, never in a ledger.
2. Context. For an EKOS cell, the schema text is rendered from the project’s compiled ledger: Table objects with columns, types, keys and descriptions, plus ForeignKey edges as join paths. The column profile is added, and the values the question mentions are looked up.
3. Generate. One model call with the cell’s prompt. If the SQL does not run, the error goes back once. If it returns no rows or a single NULL (result checks), that fact goes back once. The model is never shown anything about the gold answer.
4. Score. The predicted and gold SQL run against the same read-only database with a 30-second limit. Result sets are compared with Python’s equality, as the official scorer does (1 == 1.0). Each answer gets an outcome category. Three timed runs feed R-VES.
5. Report. Each report stores the cell’s configuration and its hash, the prompt hash, ledger fingerprints, the split and a hash of its question ids, and n with a bootstrap interval. It also stores tokens (billed and cached), cost, latency, and every question’s SQL and verdict. compare pairs two runs question by question.

Machine and resources

The machine was a 12-core Linux workstation with 62 GB RAM and a 4 GB GTX 1650. The model calls run remotely (Zen) or on the local GPU (Ollama). The harness itself is single-threaded and light.

What the results say

• EKOS’s compiled pack beats raw DDL by 10–14 points with the same model, prompt and output limit: G 63.9% (a clean hold-out) and G2 66.9%, against 53.3%. Both differences are significant on 302 paired questions.
• What moved the score (on the tune split): column profiles +11 points; the ekos-v1 prompt with result checks another +12. Raising the output limit from 1,024 to 4,096 tokens removed answers with no SQL at all (9% of the baseline). Voting between three candidates did not pay; value lookup is neutral.
• formula_1 was worse under the pack (G 18/33 against 26/33) until the profile also counted empty strings. Its lap-time columns record “no time” as both NULL and ''. A profile that reported only the NULLs made the model drop the <> '' filter it writes when it sees raw DDL. G2 fixed it: 26/33.
• The scorer matches BIRD’s official one. 704 recorded predictions gave identical verdicts from bird-bench/mini_dev’s scorer (Python 3.10, SQLite 3.37.2). One Soft-F1 value differed because SQLite before 3.45 cannot parse 16 nested replace() calls.
• Not yet measured: the pack with BIRD’s column and value descriptions (fixed in v1.2.0), few-shot examples from BIRD’s train split, a stronger model.