How to tell whether it worked

Twelve months after a BI migration, somebody asks whether it was worth it. The honest answer in most organisations is “probably, and we cannot show you”, because nobody wrote down the before.

Take the baseline in week one. It costs an afternoon and it is the difference between a business case and an anecdote.

A worked example

A baseline worksheet, filled in

Taken in week one at a 1,200-person insurer, before anything was deployed. Four numbers, one afternoon, and the twelve-month figures alongside. This is the whole business case.

Metric	Week 1 baseline	Month 12	How it was taken
Licence spend per active reader	£214,000 ÷ 486 = £440	£0 for readers	Invoice, divided by distinct logins in the prior quarter
Time from question to answer	median 2.5 days, p90 19 days	median 40 min, p90 17 days	20 tickets sampled from the BI queue, opened to delivered
Distinct definitions, top 5 metrics	13 across 5 metrics	5	Searched the estate for each metric name, counted distinct formulas
Dashboards opened in 90 days	147 of 512 (29%)	121 of 133 (91%)	Usage logs. The denominator fell because 379 were archived, not migrated

Illustrative composite. The figures are internally consistent and every step is one the product supports today.

Look at the second row. The median collapsed from two and a half days to forty minutes; the 90th percentile barely moved, from nineteen days to seventeen. That is the result, not a disappointment. The small questions left the queue; the genuinely hard ones still need an analyst and always will.

And the fourth row is the one that surprised the sponsor: the improvement came mostly from the denominator. They did not migrate 512 dashboards. They archived 379 that nobody had opened and migrated 133.

The metric we will not give you

Let us do this one first, because it is the one every vendor leads with.

“Hours saved” is unfalsifiable

It is computed by multiplying a guess about time-per-request by a guess about requests avoided, and no part of it can be checked. It makes every other number on the slide look like the same kind of thing. A business case containing one unfalsifiable metric is weaker than the same case with it removed — which is why we leave it out.

If someone needs a headline figure, use licence spend. It is on an invoice, it is checkable, and it is usually the largest number anyway.

Four numbers that are real

Metric	How to take the baseline	What good looks like
Licence spend per reader	Today’s invoice divided by people who opened a dashboard last quarter. Most estates have never computed this.	Falls to zero for readers
Time from question to answer	Sample twenty requests from the BI queue. Ticket opened to answer delivered, median and 90th percentile.	The median collapses; the tail stays
Distinct definitions per key metric	Search the estate for your top five metrics. Count the distinct formulas in circulation.	Goes to one, and stays there
Dashboards opened, of dashboards owned	Usage logs, last 90 days. Count dashboards with at least one view.	Rises — because you retire the rest

The second one has a trap worth naming. The median collapses and the tail does not, because the small questions stop being tickets while the genuinely hard ones still need an analyst. If you only track the median you will overstate the effect; if you only track the 90th percentile you will miss it entirely. Track both, and expect them to diverge.

flowchart LR
    B["Before
median 3 days
p90 3 weeks"] --> A["After
median under an hour
p90 still ~3 weeks"]
    A --> W["The gap is the point:
small questions left the queue,
hard ones still need a person"]
    style A fill:#e6f7f2,stroke:#9fd8c9
    style W fill:#eef1ff,stroke:#c7cdf2

A flat p90 is not a failure. It is the signal that the queue is now made of real analysis rather than re-cuts — which was the goal.

xychart-beta horizontal
    title "Time from question to answer, working days"
    x-axis ["Median before", "Median after", "p90 before", "p90 after"]
    y-axis "Working days" 0 --> 20
    bar [2.5, 0.08, 19, 17]

The pair to watch. The median collapses because the small questions leave the queue; the 90th percentile barely moves because the hard ones still need an analyst. A flat p90 is the design working, not a shortfall.

The number almost nobody tracks

Here is the one that usually produces the largest finding, and it costs nothing to measure because the answer is already in your logs.

What fraction of your dashboards was opened by nobody at all in the last ninety days?

In most estates the answer is somewhere between half and four fifths. Nobody has looked, because the tooling makes it awkward and nobody is rewarded for deleting things.

What the figure tells you	Implication
The migration scope you thought you had is wrong	You were going to rebuild dashboards nobody opens. This is the single biggest cost overrun available in a BI migration.
Your licence count is wrong too	Seats are often allocated to owners of unopened dashboards.
Your metric-drift problem is smaller than it looks	Many of the conflicting definitions live in artefacts nobody reads. Settle the ones that are actually in use.
You have a retirement backlog, not a migration	Retiring is faster than migrating and improves the estate more.

Take this one before you scope anything

It changes the plan more than any other measurement, it takes an afternoon, and it works on your current tool without buying anything. If you read one paragraph of this post, this is it.

Two things worth counting afterwards

Softer, but they move first and they predict the rest.

Questions asked in plain words, per week. Not a value metric on its own — volume is not virtue — but a leading indicator. It rising means people trust the answers enough to keep asking. It flat after month two means they tried it and stopped, and you should find out why before reporting anything else.
Refusals, and what they were. The system logs questions it could not answer and fields it refused. That log is a prioritised list of what your model is missing, written by your users without being asked.

AI Ask answering which region has the weakest profit ratio, with a month-by-month breakdown, a key takeaway and a trend chart — **The adoption signal to watch is the second question, not the first.** Anyone will try it once. People asking follow-ups means the answers survived contact with what they already knew.

What twelve months should look like

By	Expect	If not
Month 1	Baseline taken. Readers migrated. Licence line identified.	Nothing else on this list will be provable
Month 3	Top metrics settled with named owners. Median request time down sharply.	The definitions phase stalled — escalate to whoever owns the P&L
Month 6	Unopened dashboards retired rather than migrated. Ask volume rising.	You migrated scope instead of choosing it
Month 12	Licence saving realised. One definition per key metric, holding.	Drift resumed, which means a definition has no owner

Half to four fifthsof dashboards opened by nobody, typically

Mediancollapses — the small questions leave the queue

p90does not, and should not

Not trackedhours saved

A worked example

The slide that got signed off, and the one that did not

The same programme, twelve months in, presented twice. The first version was sent back by the CFO; the second was approved in the meeting. Nothing about the underlying results changed.

Version one — sent back.

“Saved 1,400 analyst hours” — computed from a guess about time-per-request multiplied by a guess about requests avoided
“340% productivity improvement”
“Democratised data across the organisation”
£214,000 licence saving

The CFO asked how the 1,400 hours were counted, got an answer involving two assumptions, and stopped reading. The £214,000 was real and checkable, and it went down with the rest of the slide.

Version two — approved.

Licence line: £214,000 → £0 for readers. Invoice, before and after.
Median time from question to answer: 2.5 days → 40 minutes. Twenty sampled tickets, both times, same method.
p90: 19 days → 17 days. Unchanged, and stated as unchanged.
Definitions in circulation for the top five metrics: 13 → 5.
New cloud cost: +£9,400 a year.

Version two includes a number that works against the case and one that did not improve. That is why it was believed.

Illustrative composite. The figures are internally consistent and every step is one the product supports today.

Three ways to measure this badly

Counting dashboards built. Output, not outcome. An estate that builds more dashboards faster may simply be generating shelfware more efficiently.
Counting AI answers. Volume is not value. A hundred questions a day means nothing if nobody acted on any of them, and it can be inflated by a tool that is frustrating to use.
Declaring success from month one. Month one is the honeymoon: new tool, motivated pilot group, the clean workbooks. Month six on the messy ones is the measurement. A month where everything went well is a sample, not evidence.

The two-line version

Before you start: count dashboards opened by nobody in ninety days, count distinct definitions of your top five metrics, and sample twenty tickets for time-to-answer.

Twelve months later: the same three numbers, plus the licence line. If all four moved and you can show the workings, that is a business case. If you have to reach for hours saved, it is not — and the people reading it will know.

The ninety-day sequence → · Where the licence spend actually goes →

← All posts Request a demo