AI adoption reporting

The state before

The client had rolled out Claude, ChatGPT, Copilot, and Otter across a workforce of several thousand in several operating companies. The executive question was the obvious one. Are people using this, and is it worth what we are paying for it.

Answering it cost a person an hour or three every week: pull exports from each vendor console by hand, paste them into a spreadsheet with hard-coded columns, reconcile the names that did not match. That is a small enough tax to keep paying and a large enough one to make the number arrive on somebody’s schedule rather than on the question’s. A hand-assembled number is also one somebody has to vouch for every time it gets shown.

Why the obvious approach fails

Automating the export is the obvious move. Here is why that wasn’t ideal.

The vendors do not agree on what a user is. Each console has its own definition of activity, its own reporting window, and its own idea of who counts as provisioned. Adding four such numbers together produces a total that is wrong in a direction you cannot estimate. Any comparison across tools has to be built on a definition you own, computed from message-level or session-level facts, not on each vendor’s headline figure. Not all of them hand you an API for that either, so some of the ingestion runs off compliance exports and had to be worked out with the client’s data engineering team.

A total is not an answer. “A thousand weekly active users” cannot be acted on. A ranking by department can, because it names who to go talk to first, and in what order. That requires every usage row to carry a department, a subsidiary, and where possible a project, and none of the vendor consoles know any of those things.

Licenses are pooled, so cost is not a lookup. Spend does not decompose per person by itself. Attributing it back to a team means modeling it: averaging across the pooled population per tenant, and saying plainly that the result is an allocation rather than an invoice. Anyone who checks it against the bill and finds it presented as fact will conclude the whole dashboard is wrong.

What I built

Ingestion from vendor admin APIs and compliance exports into the governed data layer, then usage facts joined to the conformed employee and project entities that layer already maintains, then rollups by department, subsidiary, and project. Because the identity reconciliation lives one layer down, this work did not have to solve it again, which is most of why it was tractable.

The product decisions mattered more than the pipeline. I settled on weekly active users and message volume as a pair rather than either alone, since active count alone treats a person who opened a tab like a person who works in it, and message volume alone tracks a handful of power users. Rollups were keyed to the org chart rather than to vendor groups so that the numbers slice the way leadership actually thinks. Custom assistants got their own facts, because an assistant somebody built for one job and kept using is a different signal from general chat usage, and it is the one that says a workflow has actually changed. Cost attribution was modeled per user across tenants and labelled as an allocation.

The dashboard reads pre-aggregated endpoints rather than querying the layer live, so the view stays fast as the fact tables grow.

What changed

A few hours of assembly every week became a thing that is simply there, and it slices by department, subsidiary, and project. It reports over a thousand weekly active users and tens of thousands of messages a week, alongside an internal ecosystem of several hundred purpose-built assistants carrying their own user and message counts. Cost per user is attributed across tenants on the same layer, so a conversation about spend and a conversation about adoption are finally reading off the same numbers.

It is also the background the rollout runs against. Which tool to push to which group is a question the usage data answers rather than a matter of taste, and the groups sitting at the bottom of the ranking are usually not resistant so much as underserved: the tool does not fit the way they work yet. That is a product problem, and knowing which groups have it is how the product effort gets pointed at them.

What I take from it

The number that changes behavior is never the total. It is the comparison: this group against that one, this month against last. Building the total is a day of work. Building something the comparison can stand on is the whole job, and it happens a layer below where anybody is looking.