How do we evaluate and monitor AI performance

I have a bad answer around how we evaluate and monitor AI performance. It's to use maths. But let's put that to one side for the moment and get to it later.

At this point everyone knows collecting data is valuable. We've all gotten used to the idea that data lakes exist and that things like Google Analytics are helpful. Data's just a part of the story of how you actually monitor and evaluate that AI is doing what you're expecting it to. If I'm feeling brave on a call I'll try and dig into what we actually do with the data but I've learnt this tends to make everyone glaze over.

The problem with evaluating AI is that it's very hard to ask "is it right?". AI doesn't neatly have 2 + 2 = 4 answers that can give us our golden reference. And with large language models their main value is that they're really good at dealing with ambiguous situations. We can instead answer, "does this match our expectations". To some people that sounds a bit underwhelming. "Does it match our expectations" doesn't feel neatly objective and sounds a bit like we're bringing beliefs into it. But it's actually a stronger position to be in because it allows us to properly evaluate what the data is showing us.

In the old 2 + 2  = 4 world of software our monitoring and tests should show that nothing has changed. There's only one correct answer, the software either runs or it doesn't. We obviously - by the way - still do this to monitor the non-AI software we're running. This all feels nice and objective but is actually based on the belief that nothing should change.

Enter maths. There are two ways to think about statistics and they're easy to mix up. On one side you have “how likely is this data if nothing’s changed?” with “how likely is it that something’s changed, given this data?”. This is confusing and, for many, feels counterintuitive. It's made somewhat worse because "how likely is this data if nothing's changed?" was the dominant way to think about statistics for much of the 20th century and only more recently that "how likely is it that something's changed, given this data?" has started to be recognised as the more useful way to think about numbers. Welcome to Frequentism vs Bayesianism. Needless to say in order to evaluate AI the question has to be about whether what we're looking at has moved. If you want to think about AI you want to think like a Bayesian.

To understand whether what AI is doing matches our expectations we need to run a lot of tests. This gives us a baseline for our expectations. This changes based on the project and based on our belief. For example I've very strong confidence that any LLM will correctly tell me that the Battle of Hastings happened in 1066 so probably don't need to test it. I have lower confidence that it'll correctly link the question ID from one audience group in one hospital with a related audience group in another hospital based on the wording of a question. That does require testing, alongside the iteration towards making the LLM be more and more predictable in how well it completes the task.

The scale of tests change. With Learning Lab - the offshoot from Cancer Academy at Guys St Thomas NHS Trust - we ran through close to one billion tokens to validate that the 'personality' of synthetic patients would remain stable across different contexts and circumstances based on how they'd been configured. It was importantly, obviously, that any clinician get the same experience when interacting with a given synthetic patient even if they were using different words or talking about different topics. By contrast to test that Goose correctly created documents we didn't create a specific test suite but instead used user-acceptance testing since we had very high confidence that the LLM would be able to successfully call the document_request tool. In a current live project we're testing to ensure the response is correct 8 times out of 10, that is, the project can be tolerant to a ~20% error rate and still be useful. A one-size-fits-all won't work and tests need to be appropriate for the work being done.

With a baseline for our expectations we can then properly look at the data. Again, the question we're asking is, "how likely is that something's changed, given this data". We have our prior in the form of the baseline, evidence from our data, and a posterior that we update based on our prior and evidence.

Goose, the thinking partner for the heritage sector, is a busy project with many people having conversations every day. We have a live monitor of the 'personality' of Goose. That's based around sycophancy, numbers of questions asked, length of response etc. We eval the health of the model's responses on a macro level using an Exponentially Weighted Moving Average and at a micro level have deviation alerts on individual conversations. We use - if you really want to get into the weeds - a Region of Practical Equivalence to try and stop it being too noisy. The data's updated after each message. Beyond personality it also allows us to monitor capabilities and if it's misbehaved (e.g. answered a niche question that's likely not adequately in its training data without first doing a websearch). I can't tell you it's working 'perfectly' because there's no concept of 'perfect' to reference it against but I can, with very high confidence, tell you that it's working exactly as expected for the vast majority of people using it and the eval allows us to fix things for the long tail of cases where it isn't working as expected.

There's no way to avoid maths in evals. But it's a fuzzy sort of maths. It won't say "it's right" but rather will let you say, "this is how sure we are that it's behaving the way we expected". It also means that there's a version of it that's accessible for people who are fans of arts and humanities. You don't need to start thinking about evals by calculating moving averages. You can just start writing down what you're hoping it'll do, how accurate you need it to be for it to be useful, how you might aim to detect when it's being inaccurate etc. In maths jargon it's your prior belief and it's what everything in the eval can be built on.

Note: If you want to fall down the rabbit hole with this "Bernoulli's Fallacy" by Aubrey Clayton, "Everything is Predictable" by Tom Chivers and "The Art of Uncertainty" by David Spiegelhalter are all books worth reading.