Writing my first evals

I was drinking my morning maté, listening to the latest and greatest music releases, proud of having shipped my first production AI agent, when my friend asked me: “did you write evals?”

I’d heard the word before, probably in a Dwarkesh podcast episode, so I had a vague notion of what it could be but no practical experience.

I need less than that to embark on a learning challenge. Plus I was itching to try a specific “skill” written by Matt Pocock, /teach. It turns your coding agent into a tutor that writes lessons as you learn into html files and tailors the course to exactly what you need to learn.

The app I’d just shipped is a business intelligence agent: it gathers, summarizes and filters recent news from our sector, so my company can keep a pulse on innovation and an eye on our competitors’ moves.

I learn by building and solving real problems, that’s my privileged channel. So I used /teach to understand evals while implementing them in the live app. Six lessons, one exercise each, and at the end I shipped my first batch of evals to production.

Then the interesting part: the evals showed me my agent wasn’t judging article relevance the way I thought it was. I was proud of something that wasn’t quite doing its job. That’s the gift of evals, they hold up a mirror to your own system.

But the thing that stuck with me most is simpler: I really like programming with natural language. It leans on reading, writing and judgment rather than pure determinism. It makes building incredibly stimulating and more connected with the world of words and abstract ideas, in which I feel very at ease.