August 26, 2026
Released: Grade it like an audit
Grade it like an audit is now installable as a Claude Code plugin, with a four-page site covering how it works, what it cannot do, and how it compares to everything else we could find.
The short version: it is a grading tool whose purpose is feedback. It drafts evaluations and proposes grades, but a proposed grade is arithmetic — what is left after every deduction has named a specific issue and cited a source the instructor can open. Blind auditors read one submission each, a single lead pass reads across all of them (because fairness is a property of the set, and no blind reader can see the set), and every lesson is written back into the rulebook so the same mistake cannot recur.
The number worth quoting is the unflattering-sounding one. Across fourteen audit rounds and seventy independent reviews, no finding has ever changed a grade. Every catch improved the evidence or the wording instead. That is the design: the loop protects what the student reads, and the outcome stays with the instructor’s judgment.
The failures are published at the same length as the successes, including a run that went nine rounds without converging and a defect that survived all forty-five of its reviews until a human read the material cold. That defect was a claim about the world, which no source can falsify — the permanent hole in the method, and the reason the last reader has to be a person.
There is a twelve-to-eighteen-minute demo on entirely synthetic data, which plants seven defects for itself to hunt and then shows you the one it cannot catch. Free, CC BY 4.0, and originated by Ira Winder.