PostgreSQL to Apache Iceberg: CDC vs Scheduled Loads
How to get PostgreSQL tables into Apache Iceberg: log-based CDC tools (Debezium, Flink CDC, RisingWave, Estuary, Airbyte, Amazon Data Firehose) compared with scheduled, reconciled loads, and when each one fits.
There are two ways to keep a PostgreSQL database in Apache Iceberg. Log-based change data capture (CDC) reads PostgreSQL's write-ahead log and applies every insert, update and delete to Iceberg as it happens. Scheduled loads query the tables on a timetable and write what changed since the last run, or the whole table. CDC gives the lowest latency. Scheduled loads have fewer moving parts and leave nothing running against the database between runs.
This page compares the common tools for both approaches. One of them, EKOS Lakehouse, is ours, and it is a scheduled tool, not a CDC tool. Claims about other tools are taken from their own documentation, linked below, as read on 2026-10-21. Check those pages for current details.
CDC or scheduled loads?
| Question | Log-based CDC | Scheduled loads |
|---|---|---|
| How fresh is Iceberg? | seconds to minutes | the schedule (for example hourly) |
| What runs against PostgreSQL? | a logical replication slot, all the time | queries during each run |
| PostgreSQL setup | wal_level = logical, a replication slot, a publication |
a read-only role |
| Hard deletes | captured from the log | seen by full loads and by reconciliation, not by a watermark |
| Iceberg side | frequent small commits and delete files: compaction is required | fewer, larger commits |
| Main operational risk | a stalled consumer: the slot keeps WAL, and the disk fills | a run that is late or slow on large tables |
The replication slot risk is real. PostgreSQL keeps every WAL segment a slot has not confirmed, so a consumer that stops for a weekend can fill the server's disk. max_slot_wal_keep_size caps it, at the cost of breaking the slot. If you choose CDC, monitor slot lag from the first day.
Choose CDC when downstream users need changes within minutes, when you must capture every intermediate version of a row, or when tables are too large to scan on a schedule and have no reliable updated_at column.
Choose scheduled loads when hourly or daily freshness is enough, which covers most analytics. They are also the better fit when you cannot change wal_level (managed databases, tight change control) or would rather not run streaming infrastructure.
The tools
| Tool | Approach | How it reaches Iceberg | Runs where |
|---|---|---|---|
| Debezium + Kafka | CDC (logical decoding, pgoutput) |
Kafka Connect with the Apache Iceberg sink, or a Flink or Spark job | self-hosted, open source |
| Flink CDC | CDC (replication slot, pgoutput or decoderbufs) |
the Iceberg pipeline sink: at-least-once, primary keys required, one source database per pipeline | self-hosted Flink, open source |
| RisingWave | CDC into a streaming database | an Iceberg sink, append-only or upsert on a primary key | self-hosted or managed cloud |
| Estuary | CDC (logical replication) | Spark jobs on EMR Serverless (or another configured backend) merge changes; Glue, S3 Tables and Snowflake Open Catalog | managed SaaS, private deployment or BYOC |
| Airbyte | scheduled syncs (the Postgres source can read CDC) | the S3 Data Lake destination: REST, Glue, Nessie or Polaris catalogs; merge-on-read dedup; some schema changes need a full refresh | self-hosted or cloud |
| Amazon Data Firehose | CDC from RDS, Aurora and EC2 databases | managed delivery to Iceberg tables in S3 | AWS. The documentation marks database sources as preview, and the AWS News Blog says the preview ended on 2025-09-30. Check availability before planning on it. |
| EKOS Lakehouse | scheduled loads, reconciled every run | Trino writes Iceberg through the Polaris REST catalog; dbt-trino builds silver and gold | self-hosted (Linux x86_64, Docker), open source (MIT) |
Where EKOS Lakehouse fits, and where it does not
EKOS Lakehouse covers the scheduled column. What it adds is proof that the copy is right, plus the layers above bronze:
- Reconciliation on every run. Bronze is compared with PostgreSQL by row counts, per-column aggregates and bucketed checksums. Each run plants a difference the comparison must catch, so a check that could not fail never reports healthy.
- Incremental only when it is safe. A table loads incrementally only where a watermark column is proven to change on every update (the column plus a trigger that sets it). Otherwise it is reloaded in full.
- Schema changes as pull requests. Safe changes are applied. A rename, a drop or a narrowed type becomes a branch showing its impact on silver, and the table waits until someone merges it.
- Silver and gold generated. A dbt-trino project with keys, deduplication and tests, and optional masking and SCD2 history. Gold marts are drafted from written descriptions. Pentaho jobs translate to dbt.
It is not the right tool if you need sub-minute freshness, if your source is not PostgreSQL, or if you need a managed service. It is version 0.1.1 and tested on Linux x86_64 with Docker. For those cases, pick from the CDC rows above. A CDC tool for bronze and EKOS for silver, gold and documentation can also work together, since EKOS compiles the PostgreSQL schema either way.
Frequently asked questions
What is the best CDC tool for PostgreSQL to Apache Iceberg?
It depends on what you already run. With Kafka in place, Debezium plus the Apache Iceberg Kafka Connect sink is the standard open-source path. With Flink, use Flink CDC's PostgreSQL source and Iceberg sink. If you would rather run one system, a streaming database such as RisingWave reads PostgreSQL CDC and writes Iceberg. Managed options include Estuary. If hourly freshness is enough, you may not need CDC at all.
Do I need CDC to load PostgreSQL into Iceberg?
No. If hourly or daily freshness is enough, scheduled loads are simpler. They need only a read-only role, no logical replication slot, and no streaming infrastructure. CDC is worth its cost when users need changes within minutes or you must capture every row version.
What PostgreSQL settings does CDC require?
Log-based CDC needs wal_level set to logical, a replication slot and usually a publication, and a role with replication rights. Monitor the slot: an idle consumer makes PostgreSQL keep WAL until the disk fills, unless max_slot_wal_keep_size caps it.
How do I verify that Iceberg matches PostgreSQL?
Compare row counts, then per-column aggregates, then checksums over buckets of rows, and narrow any difference to the keys that differ. EKOS Lakehouse does this on every run and also plants a known difference that the check must catch, so a broken comparison cannot report success.
Is EKOS Lakehouse open source?
Yes, under the MIT license. It installs with one command on Linux x86_64 with Docker and starts Trino, the Polaris Iceberg catalog, S3-compatible storage and Dagster.