A BI extract has two jobs: keep the source system out of the query path, and be fast. Tableau’s Hyper and Power BI’s VertiPaq do both, and both are formats nobody else can read. That second property is not an accident; it is the moat. We wanted the opposite.
Why not a database file
DuckDB has its own file format and it is excellent for a local cache. It is a poor distribution format: it changes across major versions, it is one opaque blob (no partial refresh, no partial download) and it takes a single writer. Parquet has none of those problems and every engine on earth can read it.
Partitioned by the columns dashboards filter on, an extract answers a Q3 question by reading Q3. DuckDB’s read_parquet('…/orders/**/*.parquet', hive_partitioning=true) does the pruning; the compiler wraps every model table in that call as a CTE.
The version is the cache key
A refresh never overwrites. It writes a new version prefix, builds the manifest, and only then moves the datasource’s pointer. Readers never see a half-written extract; the previous version stays for rollback until it expires. And because a version is immutable, the result cache is keyed on (extract id, SQL, params) and needs no invalidation logic at all — a refresh changes the id, and every stale entry is simply never asked for again.
The bugs that went away
Stale dashboards after a refresh. A widget reading table A from the new extract and table B from the old one. A cache serving yesterday’s number with today’s timestamp. All of them are versions of “the data changed underneath the reader”, and none of them can happen when data never changes underneath anyone.
Where the data comes from
For Postgres and MySQL, DuckDB attaches the source directly and copies straight to Parquet: COPY (SELECT * FROM src.public.orders) TO 's3://…/orders' (FORMAT PARQUET, PARTITION_BY (year, month)). One statement, no intermediate. For other dialects the worker streams through Arrow batches into DuckDB and copies from there. For CSV and Parquet drops, DuckDB reads the files. In every case the manifest is written last, and only when it exists is the version real.
Iceberg, and why not yet
We looked hard at Iceberg as the write format. For one writer per datasource on a schedule, everything it adds — multi-writer ACID, snapshots, partition evolution — is something the version prefix already gives us, at the cost of a catalogue service that would break the “open it on a laptop” story. Iceberg belongs on the read side: when a customer already has a lakehouse, the server runtime queries it directly and no extract exists. If we ever need a table format under extracts, the files are already Parquet; we would be adding metadata, not rewriting data.
The one rule about clients
An extract that reaches a browser or a desktop is data that person can read, whatever the row policy says. So a datasource can be marked server-only, and presigned URLs are never issued for it. Everything else is a size threshold and a TTL. Client-side security is user experience; the flag is the security.