One green smoke test, then a test run where almost every paid order ended in a refund

I run Amplituda, a small agency. In October I launched eight tiny paid services on Claude Haiku 5.5, from subtitle translation to test-matrix drafts. They are new pilots with no customers yet.

Before opening sales I ran a live smoke test on the subtitle pilot: order, payment, result. It passed. On October 20 I ran a proper series of about 250 orders with varied inputs through the same live path. Before the fixes, Screenshot Review delivered 0 of 8 paid orders and Brief to Diagram 0 of 19. Everything else went to refund_pending.

No real customers paid: payment was simulated with a live Stripe Checkout session plus a webhook I signed myself.

The paid path

Every pilot follows the same shape. The code computes an upper bound on model cost and refuses the order if it exceeds 25% of the price. That bound is reserved in a shared SQLite budget journal before Checkout, under per-product and all-pilots caps. Only a signed checkout.session.completed webhook marks an order paid, after HMAC, a 5-minute window and binding checks on amount, currency, account and order. Event ids are deduplicated; Checkout and refunds use stable idempotency keys. A worker makes at most two model calls per order. Any validation failure sends the order to refund_pending, and refunds need explicit admin confirmation.

The system fails closed, so every bug below showed up as a refund, not as a broken file.

What the series found

1. The thinking block. Haiku 5.5 uses adaptive thinking by default. When it thinks, content starts with a thinking block and then the text. Our providers accepted text only, so they rejected the reply, retried, rejected again, and refunded. Why the smoke test missed it is my guess, not a proven fact: in Catalog Copy, the only order that got through before the fix was the platform sample, the exact input a smoke test uses, and on such a simple input the model probably answers without thinking. The fix was to skip thinking/redacted_thinking blocks and still refuse anything else that isn't text. Two groups also send thinking: {"type": "disabled"}. The third keeps adaptive thinking because, on a Spec Check replay, thinking off left 3 of 12 inputs failing checks, while adaptive thinking passed 11 of 12.

2. SQLite lock at boot. The pilots share one budget journal. When seven containers start together, database is locked killed the background worker while HTTP kept answering. Docker restarts crashed containers, not unhealthy ones. The fix: a 30 s busy_timeout, bounded retries for idempotent steps, and a supervisor with bounded restarts that reports worker_failed in /healthz.

3. One rate-limit bucket. In some pilots, uvicorn didn't trust proxy headers, so every client looked like the Docker gateway. 135 requests with distinct X-Forwarded-For values started getting 429 at request 121. After adding --proxy-headers --forwarded-allow-ips, the same test got zero 429s.

4. Validators rejecting replies over trivia. The model miscounted code-point offsets in diagram citations; now it returns a verbatim quote and code locates it. The CSV dictionary rejected harmless words like "category". A non-breaking space got normalized inside a quote. Brief to Diagram went 0 → 6 → 16 of 19, CSV Brief 0 → 10 → 15 of 15.

What is still weak

Russian slang in subtitles sometimes flips meaning. Spec Check coverage is unstable. Diagram customers pay before learning the model refused a cyclic process.

Cost and lessons

Model spend for the whole campaign was under $1.

A smoke test on one or two inputs tells you the path is wired, not that the product works. Keep sales closed until a varied series passes the real paid path. Make mocks return what the API actually returns. Cap budget and attempts so a total failure costs cents.

SRT Rescue page: contractkit.amplituda.pro. As with the other pilots, check the result yourself.