LinkedIn draft — 2026-10-20

Pick a hook

1. Your AI judge is probably lying to you at a 0.5 threshold.
2. The cheapest eval judge that clears your bar beats the best one.
3. Evals stopped being a craft this week. They're infrastructure now.

Post

Evals stopped being a craft this week. They're becoming infrastructure.

Three things landed within days of each other:

→ Google published a staged playbook for agent evals. Read traces by hand, turn your notes into pass/fail judges, then gate every PR on a pass rate with no regressions.

→ Google also open-sourced AQuA, an agent that reviews your production agent's sessions, clusters failures and proposes fixes for a human to approve.

→ Hamel Husain showed that cheap "decision models" can replace LLM judges, but only after you tune each model's threshold. Default 0.5 cut-offs compare models unfairly, and one model predicted ~8% failure for traces where 60% actually failed.

My takeaway as a PM: the question is no longer "do we have evals?" It's "what does our eval system cost, who owns it, and what's our tolerance for misses versus false alarms?"

That last part is a product decision, not an engineering one. A missed failure in a payments agent and a false alarm in a writing assistant shouldn't share a threshold.

If you own an AI feature, try this: write precision and recall floors for your top three checks, with one sentence each on what a miss costs the user. You'll learn more about your product than from any leaderboard.

Where is your team on this: vibe checks, offline evals, or evals gating releases?

AIProductManagement #LLMEvals #AIAgents #ProductManagement #GenerativeAI

Source: https://hamel.dev/blog/posts/decision-model-evals/

Before posting

• Rewrite the first line in your own voice
• Add one detail from your own experience
• Check the source link still works