InsightViz on Azure

The deployment is a container, an object store and a Postgres database on every cloud. What genuinely differs between them is which of that cloud's data can be read live rather than copied, and on Azure the answer is unusually clear-cut.

What runs where

The engine opens an extract by creating a DuckDB secret of type AZURE with PROVIDER CREDENTIAL_CHAIN, which resolves to the container's managed identity. Scope that identity to the extracts container and nothing else.

xychart-beta horizontal
    title "Synapse dedicated pool, GBP per month"
    x-axis ["Before", "After"]
    y-axis "GBP per month" 0 --> 5200
    bar [4900, 2450]

The Microsoft warehouse story, plainly

The engine can attach some databases and read them without copying. The attach paths are PostgreSQL, MySQL and Redshift — which means, on Azure:

If your estate is Synapse and Azure SQL — which is most Microsoft estates — you are extracting. Plan the refresh cadence deliberately rather than discovering it after go-live.

The reporting latency improvement was expected. The ETL overruns disappearing was not, and it was worth more to the operations team than the money — three or four mornings a month where the overnight load had not finished before people arrived.

Why extract-first is usually the right answer anyway

It reads like a constraint. In practice it is the architecture most reporting estates should have chosen regardless, for a reason that has nothing to do with any product:

flowchart LR
    SY["Synapse / Azure SQL"] -->|"scheduled refresh"| EX["Extract
Parquet in Blob"]
    EX --> API["The API
Container Apps"]
    API --> D["Dashboards"]
    API --> E["Explore"]
    API --> A["Ask"]
    SY -.->|"never on a
dashboard open"| D
    style EX fill:#eef1ff,stroke:#c7cdf2
    style SY fill:#f7f8fb,stroke:#d8dbe8

The trade is a staleness budget. An extract is as current as its last refresh, and that should be a visible, owned setting rather than something buried. For most reporting the budget is hours; for an operational screen that must be current to the second, use one of the two databases that attach live and accept that it will be slower.

Datasource

Entra ID, and where roles come from

Sign-in goes through Entra, and roles come from your directory rather than being maintained twice. That matters more than it sounds for the thing Azure customers usually ask about next: when someone changes team, their group membership changes and their access follows.

What does not come from Entra is which rows they see. That is a row policy on the datasource, keyed on a principal — a role or a person — and applied inside the query on every path.

Access policies

What it costs

Three things to get right on Azure

1. Put a lifecycle rule on the extracts container on day one. Every refresh writes a new version and keeps the last. That is the rollback story, and without a rule it is also an unbounded storage bill.
2. Scope the managed identity to the container, not the account. The API never needs to enumerate storage accounts and should not be able to.
3. Decide the refresh cadence per datasource, with the business. “Nightly” is a decision, not a default. The people reading the dashboard should know how old the numbers can be, and the product tells them — a panel painted from a stored result says so on screen.

What we have not proven

• We have not run this in production. The code paths for Azure Blob are real and exercised; the surrounding Azure deployment is a design. Ask us for a proof of concept rather than a date.
• No Fabric-native integration. OneLake is read as a source on refresh. There is no shortcut or direct-lake path.
• Power BI import is not built. The report layer is readable JSON; the model needs the XMLA endpoint, which needs Premium. If your estate is Power BI rather than Tableau, talk to us before assuming a migration path exists.

The reference deployment, on AWS →  ·  Running across more than one →

← All posts   Request a demo