Field notes

Writing on LLM cost and reliability

Short technical studies, each built from numbers measured on a live production system. No vendor talking points, no benchmarks borrowed from a paper and passed off as mine. Where a figure comes from, it says so.

Eval engineering, a field studyJune 17, 20269 min read

Your LLM judge drifts 2 points between runs. Mine almost reverted a real cost win

A test retest sigma of 0.42 inside a session hid a 2.4 point swing between sessions. Here is the measurement, and the paired grading fix.