PostgreSQL to Apache Iceberg Pipelines
EKOS Lakehouse generates and governs PostgreSQL to Apache Iceberg medallion pipelines: bronze reconciled on every run, tested silver, schema changes as pull requests, one-command install.
EKOS Lakehouse turns a PostgreSQL database into Apache Iceberg tables in bronze, silver and gold layers, and keeps them correct as the source changes. It is an open-source extension of EKOS (github.com/alexeyban/ekos-lakehouse, MIT). EKOS reads the source and compiles it into evidence-backed knowledge. From that knowledge the extension generates pipeline code into a Git repository you own. It never moves data itself: Trino moves rows, Dagster schedules runs, and people approve what is lossy, breaking or business-facing.
Install in one command
On Linux x86_64 with Docker:
curl -fsSL https://raw.githubusercontent.com/alexeyban/ekos-lakehouse/main/install.sh | sh
The installer downloads a verified package, starts the stack (Trino, the Polaris Iceberg catalog, S3 storage, Dagster), creates a sample project, runs its pipeline once and opens the EKOS web console on its Lakehouse tab. For your own database, add --source host:5432/database --source-user reader. The password is read once and kept in owner-only files, never in configuration.
What it builds
| Layer | What EKOS Lakehouse does |
|---|---|
| Bronze | Every table lands in Iceberg on a schedule. A load is incremental only where a watermark is proven to move on every update; otherwise it is a full load. Every run is reconciled against the source — row counts, per-column aggregates, checksums — with a planted difference that must be caught, or the run is marked unverified. |
| Silver | A generated dbt-trino project: keys from declared and inferred foreign keys, deduplication, masking of personal data on request, history (SCD2) on request, and data-quality tests from what the source states. |
| Schema changes | Each change in the source is classified into one of nine classes. Safe ones (a new nullable column, a widened type) are applied; the rest become a branch or pull request with their impact on silver, and the table pauses until it is merged. |
| Gold | Marts from a written description. Every term must resolve to a real column or concept or it is asked back; drafted SQL is checked so that it states nothing the description did not. |
| Monitoring | Health per table and layer (healthy, late, unverified, failing, schema pending) with the facts behind it, webhook alerts, OpenTelemetry metrics and read-only MCP tools for agents. |
The console tab
In the EKOS web console each project gets a Lakehouse tab: health per table and layer, changes and marts waiting for a merge, how each table loads, tables that could not be landed and why, and where the stack runs. Its buttons run doctor, sync, check, plan and generate as jobs, with the same roles and confirmations as every other command.
Why a knowledge compiler underneath
Pipelines usually encode what someone believed about the source when they were written. EKOS Lakehouse generates them from what EKOS has compiled about the source: its catalog, keys, triggers, comments and lineage, each with evidence. So the question "is this table right, and if not, why" has an answer from one place, and every decision — a load mode, a type mapping, a refused column — cites the fact it rests on.
Status
Version 0.1.0, released 2026-10-17. Linux x86_64 with Docker is the tested platform. The design is in EKOS RFCs 0190–0199.