This is Part 1 of a series where we build a small shell in C from scratch. By the end of this post, you'll have a working shell that can run any program on your system. Later parts add builtins, redirection, chaining, and pipelines.

Teams are shipping AI and agents but are unsure how to test them correctly. AI evals are the discipline that turns vibe coding into engineering. In this post I'll lay out my current understanding and opinions about how to go about it. After some brief definitions, my primary goal is to answer high-level questions: what to evaluate, when to evaluate, and how to evaluate.

spek is a side project: a small LLM-powered coding agent that takes a plain-language description of a program and produces a working, tested Python package. You write a markdown file describing what you want, point spek at it, and a few minutes later there's an installable package on disk that builds and passes its own tests.

This post walks through one example end-to-end, then digs into the design questions I had to answer along the way and what I picked for each.

The full source is at github.com/mkarots/spek.

I've been poking around ChatGPT's "sandbox" — the code execution environment behind the scenes — to understand what it can and can't do in practice.