InsightViz on AWS - InsightViz Blog

InsightViz is a container, an object store and a Postgres database. That is the whole deployment, and it is worth saying plainly before any architecture diagram, because most BI platforms are considerably more than that and the difference shows up in what your team has to operate.

This is the AWS shape — the one we run in production for insightaiviz.com, so the numbers below are what it actually takes rather than a sizing guess.

What runs where

flowchart TB
    U["Browser"] --> CF["CloudFront + S3
site and console"]
    U --> API["ECS Fargate
the API"]
    API --> PG[("Postgres
models, dashboards,
policies, audit")]
    API --> S3[("S3
extracts as Parquet
SSE-S3, private")]
    API -->|"refresh"| SRC["Your warehouses
Redshift, Snowflake,
Postgres, Oracle…"]
    API -->|"live query"| RS["Redshift / Postgres / MySQL
attached read-only"]
    style S3 fill:#eef1ff,stroke:#c7cdf2
    style PG fill:#eef1ff,stroke:#c7cdf2
    style RS fill:#e6f7f2,stroke:#9fd8c9

Four managed services and one container. There is no cluster to size, no separate query engine to run, and no vendor holding your data.

Piece	What it is on AWS	Why
The API	One ECS Fargate task — 1 vCPU, 4 GB, 40 GB ephemeral, no public IP, behind a target group on `/api/health`.	The query engine is embedded in the process, so the task is the whole compute tier.
The extracts	A private S3 bucket, SSE-S3, with a lifecycle rule: uploads expire at 14 days, extract versions at 60.	Parquet in your own account, in a format any tool reads. The lifecycle is what stops versioned extracts growing without bound.
The metadata	Postgres — models, dashboards and their versions, access policies, the audit log.	Small, transactional, and the thing you actually back up.
The console and site	Static build on S3 behind CloudFront, with Route 53 in front.	No server to run for the front end.
Secrets	The task role, via the default credential chain.	No access keys in configuration — see below, this is the part that matters.

The credential chain is the whole security story

When the engine opens an extract it creates a DuckDB secret with PROVIDER CREDENTIAL_CHAIN rather than an access key. On Fargate that resolves to the task role. Nothing in the configuration, the database or the container image holds a long-lived S3 credential.

What to scope the task role to

Read and write on the extracts bucket prefix, and nothing else. The API never needs to list buckets, never needs IAM, and never needs access to the warehouses it reads — those use their own connection credentials, stored encrypted and never shown again after they are entered.

Which of your AWS data can be queried live

This is the question that decides your architecture, and the answer is sharper than most vendors admit. The engine can attach some databases directly and read them without copying. Everything else has to be extracted on a schedule.

AWS data source	Live query	Notes
Amazon Redshift	Yes — attached read-only	Speaks the Postgres wire protocol, so it attaches natively. Queries go to your cluster and you pay your cluster for them.
Amazon RDS / Aurora PostgreSQL	Yes	Attached read-only. The most common live source.
Amazon RDS / Aurora MySQL	Yes	Attached read-only.
S3 (Parquet, CSV)	Effectively	Read directly through httpfs. An S3-backed datasource is already the native shape.
Apache Iceberg on S3	Yes	Scanned by metadata location. Never cached, because the catalogue decides what “now” means, not us.
Amazon Athena	Extract only	No attach path. Extract on a schedule.
Snowflake on AWS	Extract only	Streamed through SQLAlchemy into Parquet on refresh.
Databricks on AWS	Extract only	Same. Live Databricks needs a passthrough we have not built yet.

If your warehouse is Redshift or Aurora, you have a genuine choice between live and extracted. If it is Athena, Snowflake or Databricks, you are extracting — and you should plan the refresh cadence rather than discover it.

Extracting is usually the right answer anyway. A dashboard reading aggregated Parquet does not touch your warehouse, which means reporting load stops competing with the workloads that earn money. The performance argument, in full →

A datasource page showing status ready, 600 rows in one table, the extract size and version, and that the runtime may be served to a browser or desktop — **Every refresh is a new version.** The previous one is kept until the lifecycle rule expires it, so a bad refresh is a rollback rather than an incident. The version string is the S3 prefix.

What it costs

Approximate monthly, us-east-1, for a deployment serving a few hundred readers. Your warehouse costs are separate and unchanged — except that they usually go down, because dashboards stop querying it.

Component	Shape	Rough monthly
ECS Fargate	1 task, 1 vCPU / 4 GB, always on	~$35
S3	Extracts, Parquet, aggregated — usually low gigabytes	~$1–5
CloudFront + S3	Static console and site	~$1–10
Postgres	Small instance, or an existing one	~$15–50, or $0 if shared
Data transfer	Mostly CloudFront egress	Low
Per reader	—	Nothing

The last row is the point. There is no per-session cost for the vendor to recover, because there is no vendor hosting your data. Why that changes the economics →

1 taskthe entire compute tier

SSE-S3extracts encrypted at rest, in your bucket

60 daysextract version retention, by lifecycle rule

$0per reader, per month

Two places an AWS deployment goes wrong

Both of these are ours, learned the hard way, and both are now checked automatically.

A worked example

The afternoon we thought we had shipped

A routine API deploy. The script waited for the ECS service to stabilise, reported success, and went green. Production ran the previous image for four hours.

Four hours

14:02
Deploy runs
New image pushed, task definition revision 4 registered, service updated. The script waits for the service to reach a steady state.
14:06
“Deployed”
The service is stable. It stabilised by rolling back to revision 3, because the new tasks would not start. ECS does this quietly and the service state does not distinguish it from a successful deployment.
18:20
Someone notices
A feature that had shipped was not there. The container log had the reason the whole time: an import failed at startup because a platform package was present in one install path and absent from the other.
The fix, part one
Check the revision, not the state
Compare the running task definition against the one you registered. If they differ, the deploy failed — whatever the service says — and print the container log.
The fix, part two
Fail at build, not at start
The image now runs an import of every platform package at build time. A missing dependency now costs three minutes of build rather than four hours of believing you have shipped.

Illustrative composite. The figures are internally consistent and every step is one the product supports today.

1. ECS rolls back silently

When a task will not start, ECS reverts the service to the previous revision and reports nothing useful. A deploy script that waits for the service to stabilise will happily tell you it succeeded while production stays on the old build.

WHAT WE DIDWait for stability, report success

The service stabilised — on the previous revision. Production ran the old image for hours while the log said the deployment completed.

WHAT TO DOCheck the rollout state, not just stability

Compare the running task definition against the one you registered. If they differ, the deployment failed, whatever the service state says. Print the reason from the container log.

2. A missing dependency reaches the image

A platform package present in one install path and absent from the other produced an image that built cleanly and died on startup. The fix is a build-time import of every package the app needs, so a missing one fails the build rather than the deployment.

The general rule

Anything that can fail at container start should be made to fail at container build instead. A build failure costs three minutes and a deployment failure costs an afternoon, and on ECS specifically it can cost an afternoon during which you believe you have shipped.

Identity and the things your security team will ask

Sign-in through your identity provider — Okta, Microsoft Entra or Google — with roles coming from your directory rather than being maintained twice.
The data never leaves your account. Extracts are Parquet in your bucket. If you removed InsightViz tomorrow, every file would still be readable by anything that reads Parquet.
Row and column rules are applied when the query is compiled, on every path — dashboard, exploration, export, AI answer. How that is set up →
The task needs no inbound access from the internet. It sits behind a target group with no public IP.

Sizing, honestly

One Fargate task with 4 GB handles a few hundred readers on aggregated extracts, because a dashboard query reads a few megabytes of columnar data and most panels are served from the stored result while fresh numbers are fetched. The two things that actually force a bigger task:

Large refreshes. Extracting a wide table streams through the task. This is the memory-hungry operation, not serving dashboards.
Live queries against a slow source. The task holds the connection while the warehouse thinks. Concurrency here is bounded by your warehouse, not by the container.

Neither is solved by adding tasks first. Measure which one you have before scaling.

What this deployment does not include

A CodePipeline. The CodeBuild project exists; the pipeline is not wired up, so deployments are currently run from a laptop or from CI you provide.
Multi-region. Single region, single task. Adequate for reporting; not a high-availability design, and we would not claim otherwise.
Live Databricks. Extract only, today. It is on the list.

The same architecture on Azure and GCP → · The business case →

← All posts Request a demo