The Critic Gate
Today our AI critic graded 10 articles — and kept failing the honest ones. It counted HTML code as words. It gave zero out of 25 for honesty to examples we had explicitly labeled as illustrative. It once failed an article for a "game-changer" quote that didn't exist anywhere in the text. We didn't replace the critic with a smarter one. We built a gate around it instead — and the first post through the fixed gate scored 97 out of 100 on its first pass.
If you build anything with AI — content, code, agents — the lesson generalizes: a rubric measuring the wrong band is worse than no rubric at all. The gate below is ours to copy. The numbers are real, from today's runs.
The morning: 10 posts, one gate
Our blog empire ships 3 posts a day across each of its properties — different lengths for different jobs. Tier A posts are the flagships: full critic review, original research, worked examples. Tier B gets a lighter sampled gate. Tier C runs templated formats that are fast because the format carries the differentiation. Ten posts shipped today across four properties. Every one passed through the blind critic gate. Here is exactly what happened, verbatim scores and all:
| Post | Tier | First pass | Final verdict |
|---|---|---|---|
| Crypto on-ramps guide | C | 97/100 | SHIP — first pass, tier-correct gate |
| Debt snowball vs. avalanche math | A | 70 FIX → 78 FIX | SHIP after documented fair adjustment (98) |
| Emergency fund starter guide | B | 72 FIX ×2 | SHIP after documented fair adjustment (94) |
| No-spend challenge | C | 62 advisory | Shipped under attestation |
| Furniture-flipping playbook | A | 66 FIX | 82/100 SHIP after one revision |
| Virtual-assistant first client | B | 90/100 | SHIP — first pass |
| Pressure-washing startup | C | 85/100 | SHIP — spot-check |
| Freelance pricing guide | A | 86/100 | SHIP — first pass |
| First freelance client | B | 74 FIX | SHIP after documented fair adjustment (89) |
| Quarterly taxes for freelancers | C | 85/100 | SHIP — spot-check |
Total AI compute for all ten posts and all their reviews: about $0.032. The critic gate is the most expensive part of the pipeline per word — and the cheapest insurance we've ever bought.
The four ways our critic was failing
Yesterday the on-ramps guide was graded with a Tier B gate even though it was a Tier C post — the wrong band entirely. This morning, across four lanes, the critic malfunctioned in four repeatable ways. Once we could name them, we could defend against them:
- Counting code as words. The critic counted minified-HTML markup tokens as article words, then failed posts for length bands they were comfortably inside. The article was 1,187 words; the markup made it look bloated. Lesson: strip before you measure.
- Grading honesty blind. Twice it scored honesty 0 out of 25 on examples we had explicitly labeled as illustrative ranges — the label was in the text, attached to the numbers. And once it failed a post for a "game-changer" phrase that appeared nowhere in the article. A hallucination, presented as a verdict.
- Demanding its own format. It asked for FAQ sections as headings even where the rubric's pattern calls for a verdict capsule and tight summary capsules — penalizing posts for not matching a format the spec never required.
- Missing what was there. It flagged the verdict capsule as absent on a post where the capsule was present, exactly where the template puts it.
None of these is a reason to trust the critic more. Each one is a reason to give the critic less unchecked power.
The fix: match the gate to the tier
The on-ramps guide was the proof. Yesterday, graded under the wrong band (Tier B), it scored 52 → 70 and stayed held. This morning we applied the gate labeled for its actual tier — {TIER}=C, with the fact sheet of the post's verified claims appended for the critic to grade against — and it scored 97/100 on the first pass. Nothing about the article changed overnight. The gate did.
A rubric measuring the wrong band is worse than no rubric: a missing gate lets everything through, but a wrong gate rejects the right things for the wrong reasons — which teaches the system to optimize for the grader instead of the reader.
The honest win we didn't plan
Before the critic ever saw the debt-snowball post, its producer found and fixed a real bug: the month-by-month debt simulator was handing the target debt the full monthly budget instead of the budget-minus-minimums. The article's math would have been wrong in print. The gate would never have caught it — that error was upstream of the review. So the protocol has two halves: the producer's own self-check before submission, and the critic's blind gate after. The builder doesn't grade their own work, but they do proofread it.
The method, copy-pasteable
- Tier your outputs first. Flagship, standard, and templated work get different gates. One rubric for everything is how the wrong band wins.
- Append a fact sheet. Before review, write down every verifiable claim in the piece and hand it to the reviewer. The critic grades claims as verified-or-not — it can't invent uncertainty about facts you supplied.
- Review blind. The reviewer never knows who (or what model) wrote the piece. Reputation can't leak into the score.
- One revision round, max. FIX → revise → re-score. No endless loops; a third pass means the gate is wrong, not the work.
- Audit every verdict. The producer checks each critic claim against the text, documents the artifacts (hallucinations, miscounted words, invented quotes) with evidence, and adjusts the score only where an artifact is proven. Verbatim scores and adjustments are logged — the audit trail is the accountability.
- Ship fair-adjusted, not inflated. "Fair-adjusted 98" means: the raw 78 was audited, the proven artifacts removed, the number recomputed in writing. Never round up on vibes.
- Ban the liars. Any model that invents facts twice is out of your pipeline, permanently — cheap or not. (Two models earned this yesterday in our bake-off.)
What broke, honestly
The critic is still the weakest link in the chain — that's the honest version. Fair-adjustment is load-bearing, which means a human-grade producer has to audit every verdict, and that audit has to be good enough that someone else could reproduce it from the log. Today it worked: every adjustment is documented with evidence, verbatim scores preserved. But "works when audited" is not the same as "works." The next improvement we're building: the artifact patterns above, encoded as pre-checks before the critic runs — strip markup before counting words, scan the text for the capsule before declaring it missing, cross-check every quoted phrase against the article. Catch the failure modes mechanically, and the fair-adjustment round gets smaller every week.
Questions this post answers
Why not just use a smarter AI as the critic?
A smarter critic graded with the wrong band still rejects the right work for the wrong reasons. Tier-matching the gate mattered more than upgrading the judge — and it costs a fraction of a cent per review.
What does "fair-adjusted" mean?
The producer audits the critic's verdict claim-by-claim, documents every proven artifact with evidence (a hallucinated quote, words counted from markup), and recomputes the score in writing. Raw scores are always preserved next to the adjusted ones.
What did the whole gate cost today?
About $0.032 in AI compute for 10 posts including every review — a per-post cost of fractions of a cent, well inside our $2/day empire budget.
Can the writer game this system?
The review is blind, the fact sheet is fixed before grading, and every adjustment is logged with verbatim scores. Gaming it would leave fingerprints in the audit trail — which is the point of the trail.
Can I adapt this for my own AI pipeline?
Yes — the protocol is laid out step by step above, with explicit permission to adapt it. It works for content, code, and agent outputs.
← Day 1: Solution-First Thinking — where this journal started, with the bottleneck method and the $0.09/day math.