Legible EventsThe business case for software factoriesDec 1
← Newsletter

Your Agent Is Only as Good as Its Underlying Data

Hi there,

The topic du jour over here at Legible’s global HQ is data engineering. It’s becoming increasingly clear to us that AI agents are getting better and better, but their performance will only be as good as the data they run on.

Let me give you an example. My cofounders and I run a couple of different products in addition to our consulting day job. We are increasingly automating the blocking and tackling of running these businesses: the support, the invoicing, the lead qualification, etc.

We are now pushing hard on fully automating first-line support, so we reached out to an acquaintance of mine who is doing it at scale.

He runs three AI agents that handle 400 to 500 support tickets a day for a consumer app with well over a million users. What stuck with me was how many systems his lead agent draws from: the help desk, billing, the code repository, and the cloud logs. He trained his AI teammates on two years of tickets, plus every escalation and whether it actually got fixed. And when he found context that lived nowhere, like which customers had already received a goodwill refund, he had the agent build and maintain its own table to track it.

That is data engineering, and it turns out to be most of the job.

We’re finding the same thing with our own agent that supports Ambient, our AI note-taking product. The context our AI support agent (“Max”) needs is scattered across Intercom, years of email, our billing system, our product database, and the logs from a third-party meeting recorder. Each system holds a piece of the answer to “what happened to my meeting?” None of them holds the whole thing. My cofounder Taylor used a fleet of subagents to distill hundreds of support emails into skills; this was the easy part. Stitching the live systems together is where it gets tricky.

Then there’s the quality of the data itself. Our internal dashboard is a bit lazy with the error codes around what went wrong. If your error messages are too vague to guide a person, they are definitely too vague to guide an agent. Rewriting those error states in plain language is now on our list before the agent goes live.

So we need a real data engineering strategy: one place where an agent can find a unified picture of a customer. We’re evaluating Snowflake for the longer term. In the meantime, we’re hacking it together with Google Drive, which is enough to get started.

The payoff for getting this right is big. My acquaintance trained his lead agent on a premium model at about $4 a run, then moved her to one that costs 13 cents. Quality held, because the intelligence lived in the context. The model became a cost decision.

If you’re standing up agents of your own, I’d start with an inventory. Where does the context each agent needs actually live, and how much of it can the agent read cleanly today?

We’ll dig into how the teams keeping pace organize their context at our session on Thursday, November 5. Grab a spot.

I read every reply. Let me know what I missed.

Lawrence Coburn

Originally sent by email. View on Beehiiv ↗