New Roles Series · AI Evaluation Engineer

Learn LLM evaluation the way it's actually done.

Real numbers, runnable labs, and the judgment the job demands — not blog-post hand-waving. Start with a full chapter, free.

Labs run free, no API key · One-time purchase · Learn online

The AI Evaluation Engineer — course cover

Why it's different

It teaches. It doesn't summarize.

1

Real numbers, not names

Position bias isn't just defined — it's measured: 9/50 pairs flipped = 18%. You see the failure, then the fix.

2

Labs you actually run

Every chapter ends with a hands-on lab using the real Anthropic SDK — and every one runs free, no API key needed. See the free labs on GitHub →

3

Quizzes that test judgment

Scenario diagnosis, not recall — the way a real interview or design review would push you. Every answer explains the why.

The course

The AI Evaluation Engineer

12 chapters, each a deep worked read → a lab you run → a worked solution → a reasoning quiz. Written for working engineers who want the delta, fast.

The AI Evaluation Engineer course
$29 · one-time

Also available: New Grad edition · Manager & Founder edition