Top‑1% AI PM competency map¶
The daily brief trains against these ten competencies. Each month, score yourself 1–5 on each and aim your deep reading and weekend builds at the weakest one.
| # | Competency | Top‑1% looks like | Canon to start with |
|---|---|---|---|
| 1 | Model literacy | Explains how pretraining, post-training (RLHF/RLAIF, RL on verifiable rewards), reasoning models, context windows and tool use shape product behavior, and predicts what next quarter's models make possible | Karpathy "Intro to LLMs" & "Deep Dive into LLMs"; Lilian Weng's blog; Sebastian Raschka's Ahead of AI |
| 2 | Evals & quality | Designs eval suites before the feature: error analysis on real traces, binary pass/fail judges calibrated against humans, regression sets in CI, and online metrics tied to offline evals | Hamel Husain "Your AI product needs evals" & "Field guide to rapidly improving AI products"; Eugene Yan on LLM evaluators; Shreya Shankar "Who validates the validators?"; Anthropic "Demystifying evals for AI agents" |
| 3 | Agentic product design | Knows when a workflow beats an agent, how to design tool surfaces, human-in-the-loop checkpoints, memory, recovery from failure, and how to scope autonomy | Anthropic "Building effective agents"; OpenAI "A practical guide to building agents"; Chip Huyen "Agents"; MCP spec |
| 4 | AI UX | Designs for probabilistic output: calibrated trust, showing uncertainty, steerability, graceful failure, latency masking, and feedback capture that feeds evals | Google PAIR Guidebook; Microsoft HAX guidelines; Linus Lee's interface essays; Simon Willison on prompt injection |
| 5 | Economics & infra | Builds a unit-economics model (tokens × price × calls per task), uses routing, caching and batching, and knows how inference cost curves change pricing strategy | SemiAnalysis; Epoch AI trends; provider pricing pages; a16z "LLMflation" |
| 6 | Governance, safety & risk | Writes a risk register for a launch, maps it to the EU AI Act / NIST AI RMF / ISO 42001, plans red-teaming, and ships a model/system card | NIST AI RMF; EU AI Act summaries (IAPP); Anthropic RSP; METR reports; frontier safety frameworks |
| 7 | Data strategy & moats | Knows what data flywheel the product creates, how to get feedback signal, and what still defends you when the base model gets 10× better | Stratechery on aggregation; Eugene Yan on recsys and data flywheels |
| 8 | Strategy at the frontier | Separates durable value from model-wrapper risk, reads lab roadmaps, positions against commoditization, and handles build/buy/partner decisions | Interconnects; Latent Space; Benedict Evans; lab launch posts |
| 9 | Execution in uncertainty | Ships with eval-gated launches, staged rollouts, kill switches and incident playbooks, and writes AI PRDs with explicit quality bars | Lenny's Newsletter AI PM episodes; company engineering postmortems (Anthropic, GitHub, Uber) |
| 10 | Point of view & influence | Has a public, evidence-backed thesis people quote, and explains complex trade-offs to execs in 3 sentences | Your own LinkedIn posts and talks, built with this repo |
Hands-on ladder (weekend builds)¶
- Call a model via the API, then add structured outputs and prompt caching, and measure cost per call.
- Build a 50-example eval set for a real task, write an LLM judge, and measure its agreement with your own labels.
- Build a tool-using agent (3 tools), log traces, run error analysis, and fix the top failure mode.
- Ship an MCP server for a tool you use.
- Run a red-team session on your agent (prompt injection, data exfiltration) and write the mitigations.
- Compare three models on your eval and build a router, then chart quality vs cost.
- Write a one-page system card and risk register for your agent.
- Write a PRD for an AI feature with eval-based launch criteria.