WhetstonePrompt Optimizer

Whetstone rewrites a prompt round after round and only keeps a change when the numbers honestly say it is better. Most prompt tweaking is vibes: change a line, eyeball three outputs, call it better. I wanted a loop that measures like an experiment instead.

Claude mode is covered only by fake-fetch tests: headers, retries on 429 and 529, refusals and the budget stop. It was never run against the real API. The UI buttons were checked by screenshots and DOM dumps, not by hand. The simulator is a stand-in for a real model, so the 44% to 100% climb shows the loop works, not what a real prompt would gain.

Claude Code built it in about half an hour of wall clock on September 23. I set the scope from the spec: honest measurement first, a sealed test split, and a budget cap checked before any money gets spent. Claude Code wrote the pure core (splits, graders, judges, statistics, a cost ledger, a version tree and the loop) plus 316 assertions across nine small test files. Everything ran with fakes. No live API call was made during the build.

The part I like most is the simulator. The simulated model has hidden "skills" (JSON only, exact keys, a category list, priority rules, an order id format, summary length) and only uses them when the prompt asks. The proposer can only discover them through grader failure messages. That means the whole loop visibly improves with no API key. The demo task turns 40 messy customer emails into JSON support tickets, and it climbs from 44% to 100% on validation over 5 rounds, with rejected candidates along the way.

• Significance tests change what "better" means. With only 10 validation cases, pass/fail scoring was too coarse for a paired bootstrap to see real gains. Scoring each case as a weighted mean of graders gave it enough signal.
• A budget cap has to be enforced before the call, not after. The ledger reserves the worst-case cost, refuses the call if it would cross the cap, then settles from real usage. The test shows spend never passes the cap.
• A judge is only as good as its evidence. The rubric judge has to quote the output, and the core checks the quote is really there. In the demo about one score in eight gets thrown out as ungrounded.
• The pairwise judge runs twice with the positions swapped, so a judge that always says "A" comes out tied.
• The proposer has to know what it already tried. Without that, a rejected round produced the exact same candidates again and the loop stalled.

The idea of checking a model's evidence instead of trusting it shows up all over this last stretch. #92 keeps only candidate ids Claude was actually sent, #98 checks every quote against the chunk it cites, and #100 rejects any AI rewrite that adds a fact. Whetstone also added a verify check that fails on any em dash in its folder, after two slipped into test files.

I would add a small held-out canary check inside the loop to catch overfitting to validation earlier. Then I would test Claude mode against the real API with a tiny budget before calling it done.