PMaker home
Search
中EN
System
Color theme
空格的键盘
空格的键盘空格的键盘产品经理 · AI Builder

Think product. Make products.

How products are imagined, designed, and built —
in the age of AI.

AI lets more people turn ideas into products. The faster you build, the more you need to think clearly about what to make, how to make it, and how to improve it after launch.

AI Basics

Build a shared vocabulary and a map of the industry: where AI comes from, how far it's gotten, and what each role in the value chain does.

1建立坐标先看懂名词、历史与当下
Terms aren't a scattered pile; each one lives on a specific stage of a single callLoad the inputSystem prompt, conversation history, retrieved fragments, and the user's words go together into a context window measured in tokens.The model computesThe only mechanism is predicting the next word, repeated hundreds of times to finish a reply. Capability ceilings are set at training.It outputs textGenerated on the spot, not fetched from a database. Where it doesn't know, it fills with the most plausible-sounding content—that's hallucination.It actsIt emits tool calls, your code executes, results go back into stage one, and the loop keeps turning—that's the agent loop.
01

A Map of AI Terms

Put the high-frequency terms on one map: what each is, which problem it solves, and how it relates to you.

Each generation's dividing line is "which stretch still has to be done by humans"A line that keeps movingEvery era of AI is two parts: the stretch done by humans and the stretch the machine learns on its own. Progress is the line moving right; the motive is always that the human part became too much.Four shiftsHumans write rules (expert systems) → machine learning learns statistics → deep learning learns features → train one foundation and change its use with prompts. The bottleneck never vanishes; it just moves.Six milestones1950 Turing, 1956 Dartmouth, 1970s-80s expert systems, 2012 AlexNet, 2016-17 AlphaGo and the Transformer, 2022 onward conversational LLMs.Three reminders for buildersCapability and reliability progress separately; old paradigms settle into proper roles; each expansion trades in one bottleneck: high-quality context, verifiable feedback, controllable action boundaries.
02

A Short History of AI

From symbolic rules and machine learning to LLMs. What each paradigm shift replaced.

We assume "it can do the hard thing, so the easy thing is no problem" — that doesn't hold for modelsAlready reliableRewriting, summarization, translation, extraction, code completion, drafts; use as a copilotUsable with guardrailsKnowledge Q&A, document analysis, controlled tool use, short agent flows; pair with eval and human fallbackStill needs cautionUnattended runs, high-risk approvals, medical/legal conclusions, open web browsing; key actions need human confirmationWhat decides the tier is the consequence of failure, not the technology
03

Where AI Stands Today

What's stable and usable now, what's still changing fast, and what demos well but isn't reliable in production.

Which layer your product sits on decides your cost structure and your moatApplicationsFace end users directly. Closest to the money, lowest technical barrierMiddlewareModel platforms, vector stores, agent frameworks. Earns by lowering the cost of integrationModelsTrain and sell models. Burns the most cash, fights the worst price warsComputeChips and clouds. Tight supply, hard to replace—the strongest pricing powerEvery layer is the customer of the layer below it; money flows downward
04

The Four Layers of the AI Industry

Compute, models, platforms, and applications. See what each layer sells, pays for, and builds its moat on.

Leaderboards turn over every few months; these axes don'tTask performanceAccuracy and stability on your kind of task20–50 real samples, same prompts for every model, human gradingReal costThe cost of one complete call, not the per-million-token rateMeasure with real requests — inputs, outputs, re-sendsStability & complianceRate limits, timeouts, data residency, ability to sign a contractHave legal read the data-processing terms, not the marketing pageReplaceabilityHow much code and prompting changes if you switchPick two, not one — and actually run the backupMeasure with your own tasks; this table survives the next quarter
05

How Model Providers Differ

Skip the vendor rankings. Make a real choice by capability, cost, ecosystem, deployment, and compliance.

The same model, five paths to it. The real difference is who you call when it breaksYour productModel calls wrapped in your own layerOfficial APIFirst access to new versions, clearest rate limits and SLA, official supportCloud platformContracts, invoices, compliance — the smoothest path to productionAggregatorOne key for many models — best for comparing and switchingSelf-hosted gatewayOfficial prices plus unified management, at the cost of running itAPI resellerCheap and tempting, but data passes through a stranger — not for productionWrap model calls in your own layer — switching providers will happen eventually
06

Where to Get Models

Official APIs, cloud platforms, model hubs, and aggregators differ in control and responsibility.

2看懂行业谁提供算力、模型、平台与应用

LLMs

How large language models work. From the single mechanism of "predict the next token", every strength and every flaw follows.

1认识先看清大模型在做什么
In one generation, this is the only thing it doesReadRead the whole existing sequence inScoreCompute the probability distribution of the next tokenSampleDraw one by probability — not the highest-scoring tokenAppend & repeatAppend it to the sequence and start over from step oneMaking it talk more is making it compute more rounds
01

How LLMs Actually Work

Predicting the next token — that's the whole mechanism. Every strength and every flaw grows out of it.

Search engines give you a location; an LLM gives you a conclusionSearch engineThe source stays present — index, match, rank, return "where it is"Click through and verify the original textLLMThe corpus is compressed into weights; answers are reassembled from patternsConfuses facts, has a cutoff, and gives no provenanceSearch-then-answerFetch a few real passages into the context, then generate from themYou get sources — but it is still generatingNeed facts and sources? Use search. Need transformation and rewriting? Use a model.
02

LLMs vs. Search Engines

Search hands you the bookshelf; an LLM hands you a conclusion. Answering from memory means there's no source to check.

Training happens once; inference happens on every call — the boundaries of cost and capability sit on either side of this lineTrainingFeed massive text and guess the next token trillions of times: months, tens of thousands of GPUsCapability, language, and style are locked in here — you cannot touch itInferenceFeed input through the weights to compute an answer: seconds, fractions of a cent, every callPrompt, context, temperature, tools all yours — 90% of the work lives hereFine-tuningOne small extra training round on existing weights: style, format, domain framingSits between the two — do not reach for it first"Train our own model" usually means wanting the inference-side things
03

Training vs. Inference

One feeds the model, the other uses it. The boundaries of cost and capability sit on either side of this line.

The rough makeup of training corpora — ratios differ by vendor; the scale is about rightWeb pagesEncyclopedias, forums, news, blogs, product pages — the bulkBreadth, but also errors, stale info, and marketing copyBooks and papersLong coherent arguments and structuresCopyright-limited; less of it than you would thinkCodeStrict syntax, machine-checkable textBuilds structural sense and reliable JSON outputDialogue and Q&AGives it a human voiceAlso teaches it to agree with youYour internal dataCompany docs, business conventions, client lists, latest processesNever in the corpus; only you can supply itThe corpus decides what the model favors and what it is missing
04

Where Training Data Comes From

What the corpus is made of decides whose bias the model carries and which parts it's missing.

Six kinds of models, six different billing unitsTextText in, text out — billed by token; output costs several times more than inputThe familiar one; the workhorse of most AI featuresImage / videoBilled per image and per second, with stepwise price jumpsThe two most expensive kinds; budget them separatelySpeech TTS / ASRBilled per character and per audio duration — nothing to do with tokensSame duration, same price, regardless of how much is saidEmbeddingsTurn meaning into coordinates; generate nothing — the base of semantic searchThe cheapest kind, by one or two orders of magnitudeEstimate the six kinds separately — one wrong unit and the whole budget is off
05

Six Kinds of Models

Text, image, video, speech synthesis, speech recognition, and embeddings. Input, output, and billing units differ.

Two models, one question: writing more is computing moreRegular modelPredicts the next token straight to the conclusion: fast, cheap, stable latencyMulti-step tasks go wrong easily; there is no chance to correct mid-wayReasoning modelWrites a long chain of thought first: breaks down the problem, lists intermediate steps, checks assumptionsMulti-step tasks are steadier, but slower, pricier, and latency turns unpredictableTurn it on only when the task needs multi-step reasoning
06

Reasoning vs. Regular Models

Deep thinking means generating a chain of thought before answering. Which tasks justify that extra cost.

"Open-source model" usually means the middle piece is public — the weightsWhat's openTraining data, weights, and the training recipe — usually only the weights are releasedYou can use it, but can't reproduce it or see what it learnedLicensesPublic isn't free to use — commercial use, scale limits, derivative modelsRead the license text, not the marketing pageThe self-host mathGPUs, power, ops, concurrency — mostly fixed costsAt low volume, per-call cost beats the API in the wrong directionOpen source sells you control, not cheapness
07

Open vs. Closed Weights

What public weights mean, and when self-hosting actually pays off.

The same word "strawberry": you see characters, the model sees tokens — the unit is neither a letter nor a wordWhat you seestrawberry is 10 characters; r appears 3 times — you can count instantlyYou assume the model can count characters, spellings, and words tooWhat the model seesThe same text is split into str, aw, berry — three tokens, letters sealed insideIt cannot count words, arithmetic wobbles, character-level tasks failMany "why can't the model do this?" moments trace back to this mismatch
08

Tokens: The Billing Unit

LLMs don't bill by characters. The price gap between languages comes from how text is tokenized.

Three numbers that are constantly mixed up, and actually have nothing to do with each otherParameter countHow many weights the model has — sets the capability ceilingDoesn't decide how well it does this task; most models no longer publish itContext lengthHow many tokens one request can holdFitting it in doesn't mean using it well — the middle gets ignoredBenchmark scoreAccuracy on standard question setsQuestions may have leaked into training; task types may not match your caseThe fourth number that matters: run your own 20 real cases on each candidate
09

How to Read Params and Benchmarks

Parameter count, context length, and benchmark scores are three different things. Don't mix them.

One candidate distribution, and temperature only reshapes it — each bar is a candidate tokenLow temperatureSharpens the distribution; the model sticks to its most confident candidatesConservative, repetitive, safe — nearly identical every timeHigh temperatureFlattens the distribution; long-tail candidates get liftedFresher wording, but the odds of nonsense rise with itCreativity and reliability are two ends of the same dial; you cannot have both
10

Temperature and Randomness

Getting a different answer each time is by design, not a bug. When to turn it off.

One call needs only three things—and switching providers usually changes only the first twoOne model callWhere · Which · Whose billbase_urlWhere to send it: the address of the model service, filled differently for official, cloud, or self-hosted gatewaysModel nameWhich one: an identifier. Pin production to an exact version and watch out for -latest aliasesAPI keyWhose bill: an identity credential and billing attribution in one; must never live in the frontendIntegration isn't the hard part; what follows is: cost control, failure handling, and quality acceptance
11

What Calling a Model Takes

base_url, model name, and API key. Why everyone is compatible with the OpenAI format.

You don't need to memorize every parameter — sort them by which part of the call they controlCall parametersGrouped by what they controlWhat it sayssystem, temperature, top_pHow muchmax_tokens, stopHow it's deliveredstream, response_format, seedDefaults are almost never what you want — review all three groups for every feature you ship
12

Common Call Parameters

max_tokens, top_p, stream, system — what each parameter means for your product.

Not every request deserves the most expensive modelTier by taskLight, mid, and heavy tiers — static assignment by feature entry is usually enoughSimple features on the expensive tier = a bloated billPrimary/backupAuto-switch to a backup when the primary fails — and actually run the backupA backup never verified collapses exactly when you need itFixed test setKeep a fixed test set and run it side by side on every model changeRely on feel, and a quality drop goes unnoticedWrap it in a layerTiering, fallback, and verification all live in your own wrapper layerSDKs scattered through business code make every fix expensiveRate limits, timeouts, a version de-listed one day — normal, not an accident
13

Selection and Fallbacks

Tier models by task, and auto-failover when the primary model goes down.

It doesn't have an "I don't know" defaultWhy it's inevitableEvery step draws a token from the candidates, and nothing checks whether the sentence is true. It has no "I don't know" state, patterned content is easiest to invent, post-training amplifies confidence, and it follows your assumptions.Where it shows up mostObscure specialist topics, specific numbers and dates, citations and links, anything past the training cutoff, your internal concepts, and the tail of long answers.Four defensesConstrain the source, demand citations, cross-validate, and make "I don't know" legitimate. Put the burden of verification in your system, not in the user's vigilance.
14

Why Hallucinations Happen

Hallucination is an inevitable byproduct of the generation mechanism, not a bug. You can only defend, not fix.

The world the model knows stops at a certain dateWhat the cutoff isTraining corpora stop on a cutoff date; months of training, evals, and internal testing follow before release. The model you get knows less than today—and often less than its own release date.Why it's wrong with confidence"I don't know" isn't a state the model has. Past the cutoff it has no memory, only patterns for how such things look, so it extrapolates: bump the version, follow the convention. Plausible, just not true.How to feed it todayAnything that can go stale must come from you: put today's date in the system prompt, retrieve facts first, route real-time data through tools, pin versions. Don't expect it to remember.
15

Knowledge Cutoff

An LLM doesn't know what happened yesterday unless you feed it yesterday.

An image is chopped into patches, encoded, and spliced into the same sequence as textResizeScale the image to a size the model acceptsThe ceiling on visible detail is fixed here — small text blurs firstPatchCut it into tiles; a vision encoder turns each tile into vectorsMap & joinMap the vectors into the same space as text tokens and append them to the sequenceKeep predictingThe model no longer tells text from image — it keeps guessing the next tokenEvery text-model failure shows up in image reading tooIf accuracy has hard requirements, don't use a general model — dedicated OCR is sharper and cheaper
16

Multimodality: How Images Get Read

Multimodality is a retrofit. How it was added decides how closely the model can look.

2种类有哪几种模型，各管什么
3度量按什么计费、按什么衡量
4接入怎么调用、配置与降级
5边界哪些地方一定会出事

Prompting

A prompt isn't an incantation — it's a requirements document. The clearer you write it, the more it gets right.

1认识提示词到底是什么
What a prompt does is narrow the range of what it might writeJust "write copy"Who for, what product, how long, what tone — all unstatedIt falls back to the most common pattern in its corpus: generic copy+ for a 3-month-old userRules out every new-user approachThe range starts narrowing+ under 100 charactersLength fixed, structure mostly settledThe range narrows further+ no exclamation marks, no marketing voiceWhat is left is close to what you wanted"Think hard" narrowed nothing at allPrompt engineering is not finding a magic phrase; it is systematically saying everything that matters
01

A Prompt Is Not an Incantation

It's a requirements document. Wherever you didn't specify, the model fills in its own answer.

Four parts, each blocking a typical failureA promptFour parts, none optionalContextWho it's for and the scenario; without it, output fits anywhere and helps nowhereRequirementsWhat to actually produce; without it, it picks the most generic readingConstraintsWhat's off-limits; without it, it improvises and inventsAcceptanceWhat "done" looks like; without it, the format changes every timeMiss one part, and it drifts on exactly that one
02

The Four Parts of a Prompt

Context, requirements, constraints, and acceptance criteria. Miss one, and it goes off the rails on that one.

Every request sends a whole sequence, not just the sentence the user just typedSystem promptEverything that holds every turn, resent each round; the stable prefix can hit cachingDynamic content mixed in breaks the cacheHistoryAlternating user and assistant turns, tool results included, growing every roundInflates fastest in agent setupsLatest messageThis round's request and material; all dynamic content belongs hereUser-controlled text must never enter the system promptFixed content first, changing content last
03

System vs. User Prompts

What to carry in every turn, what to say once, and how priority works.

The same content reads very differently to a model depending on how it's writtenOne run-on blockAll boundary information is lost; where topics change, what's parallel, what's material — the model has to guessThe longer and messier the prompt, the more it guesses wrongStructured sectionsHeadings, lists, and fences mark the boundaries explicitlyWhich part is instructions and which is material is obviousStructure is information, not aesthetics
04

Why Write Prompts in Markdown

Structured text is parsed more accurately. This isn't a matter of taste.

One example carries all the requirements you can't put into wordsAdjectivesCompressed information that everyone decompresses differently. Say "professional" and the model takes the most common phrasing in its corpus—the averageWrite ten; it still returns the averageOne exampleAn input-output pair that carries length, tone, addressing, and conclusion order—every implicit requirement at onceGive one; it gets every detailTrapsExample bias gets amplified; classification has order and ratio effects; examples harden into a block nobody dares touchKeep diversity, balance counts, annotate each exampleExamples should come from real data, not from something you invent
05

Examples Beat Explanations

How few-shot examples work and their traps: how many to give, which ones, and when they hurt.

Three layers, reliability rising — output read by programs shouldn't rely on layer one alone1. Ask in the promptState the format and give a sample — about 90% reliableThe missing 10% is hundreds of daily failures in production2. Enforce via APIresponse_format plus a schema constrains output at generation timeGuarantees syntax, not content correctness3. Validate yourselfRetry, fall back, refuse; track the parse-failure rateThis layer can't be skipped, even with the first two in placeThe three layers stack — they're not an either/or
06

Controlling Output Format

From "return JSON" to enforced formats, and the trade-offs when streaming.

Four tricks, four mechanisms — understand the principle and you'll know when to use eachHigh-leverage tricksEach treats a different problemAsk firstWhen your requirements are fuzzy, have it ask 3 to 5 questions before startingAnchor to a referencePoint to a concrete reference instead of using adjectivesThink before answeringWrite out the reasoning steps explicitly for extra chances to correctRole framingShifts tone and wording, not capability or accuracyThese aren't magic spells — they work only when the scenario fits
07

A Few High-Leverage Tricks

Asking questions first, reference anchoring, think-before-answering — and what role prompts really do.

Two ways to change a prompt, and they lead to very different resultsTuning by feelTry one input, it looks better, ship itAnother class of inputs regresses and you can't say which line caused itIterating on casesFixed test set, one change at a time, run the full suiteEvery change has evidence; you can revert and traceGet a ruler first, then tune
08

How to Iterate on Prompts

Change one thing at a time and keep a set of test cases. Tuning by feel doesn't work.

The same rule, "don't fabricate data," goes two very different ways depending on where you keep itInlined in codeFour files, four phrasings, one of them missing the ruleNo one dares edit, fixes miss copies, rollback impossibleManaged centrallyDefined once, referenced everywhere, changes diff cleanlyOne edit applies site-wide; you can test, roll back, and traceA prompt is an asset, not a string literal
09

Storing and Versioning Prompts

Prompts scattered through code will eventually become a blob no one dares touch.

2写法让它稳定听懂
3迭代从能用到稳定

Context & RAG

LLMs have no memory. First see how retrieval finds things, then how to fit material into a limited window.

1检索机制机器怎么判断「意思接近」
Put every sentence on one map, and finding answers becomes "finding the nearest neighbors"What it solvesKeyword search matches literals; "refund," "money back," and "I don't want it anymore" share no words but one meaning. Vectors ignore wording and look at meaning—similar ones end up close.How text becomes coordinatesA dedicated embedding model turns text into hundreds or thousands of numbers. Coordinates are learned from huge corpora: words that appear in similar contexts get pulled close. Distance is measured with cosine similarity.What it means for your productChunk and embed documents offline first; updates require recomputing; switching embedding models means recomputing everything. At query time you embed the question once and compare distances.Limits you should knowIt measures "similar," not "correct"; it's insensitive to negation; exact matching is weak; long text gets diluted.
01

Embeddings and Semantic Similarity

Turn a sentence into coordinates; similar meanings land close together. The whole retrieval stack is built on this.

A user asks: "Can I still return it after seven days?" The top three by similarityTable of contentsHeaders full of "return" and "FAQ" — heavy word overlap, highest scoreTitles only, no body text; not a single useful sentenceReverse clause"This category does not support 7-day no-reason returns" — nearly identical wordingIt says "not supported," so the model may answer the oppositeThe right answerThe passage that actually answers the question ranks thirdTake only the top two and the right answer never reaches the modelSimilarity measures how alike the wording is; reranking judges whether it answers
02

Similar Is Not Relevant

The most similar passages aren't necessarily the answer to your question. This is the biggest error source in retrieval.

The two methods have complementary strengths, so real systems mix themKeyword searchMatches literal text: order numbers, model codes, legal citations, names, and new terms all hit exactlyRephrase it and it misses: "I don't want it" vs. "return" share zero charactersVector searchMatches meaning: synonyms, casual phrasing, and related concepts are all findableExact strings, negation, and new terms all fail; it handles "close enough" wellHybrid searchEach path recalls a batch; merge, dedupe, and rank togetherAlmost always better than either alone, especially in enterprise knowledge basesRunning one method means giving up half your recall
03

Keyword, Vector, and Hybrid Retrieval

Three ways to search, each with blind spots. Real systems almost always mix them.

Everything placed on this table adds up, and the total cannot exceed the window limitContext windowEverything in one requestSystem promptProduct setup and rules, resent every roundHistoryAll previous exchanges, growing steadilyRetrieved materialDocument chunks fed in by RAG — often the largest blockUser inputThis round's question and attachmentsOutput headroomThe most forgotten block — output also consumes the windowReserve output headroom first; only what remains is available for input
04

The Context Window

Everything a single request can hold — that's the size of the table you get.

The same key information is used with very different probability depending on where it sitsStartTheme, setup, task goalHigh probability of being used correctlyMiddleDevelopment, background, fillerIncreasingly ignored as the context growsEndConclusion, current focus, latest instructionsHigh probability of being used correctlyPut the important stuff at both ends, repeat critical constraints, and split long tasks
05

Lost in the Middle

The start and end are read clearest; the middle fades as the context grows.

When a conversation outgrows the window, three approaches exist — and each drops different thingsTruncationDrop the earliest rounds, keep the most recentThe earliest rounds are often the most important — the model starts to drift once they're goneSummarizationCondense the first N rounds into a short summary and replace themBetter than truncation, but lossy — and the summary itself can be wrongLayered retentionHard constraints verbatim, middle process summarized, recent rounds verbatimMore work up front, but a much bigger difference in stabilityWhat must be remembered stays exact; what can be fuzzy gets fuzzy
06

Conversation Compaction

Long chats quietly drop content. What drops first is usually the constraints set early on.

"It remembers me" is really your system storing and retrieving on its behalfExtractWhen a conversation ends, a model call picks out what is worth keeping for laterKeep too much and the store fills with triviaStoreWrite to the memory store and resolve conflicts: new facts overwrite old onesMixing up temporary and permanent is the easiest place to slipInjectAt the next session, pull the relevant memories into the system promptIf a user would tell you something willingly, make it a setting — don't make the AI guess
07

How Long-Term Memory Works

Extraction, conflicts, injection. What it costs to make a model recognize you.

Retrieve first, then answer — the model learns nothing new without retrainingIndexChunk documents, embed each chunk, store in a vector database; done when documents changeChunking quality caps retrieval qualityRetrieveEmbed the question and find the most relevant passagesMature setups add hybrid search plus rerankingAssemble contextPassages, question, and constraints become one requestMaterial status and placement decide successAnswerThe model answers from the material and cites sourcesWith citations, errors become findableRAG isn't a model; it's an engineering pipeline
08

RAG: The Three Steps

Retrieve first, then answer. The three stages, and what it actually solves.

The same policy document, three chunking strategies, three different outcomesToo smallThe condition and its exception land in different chunks; the exception never gets retrievedConfidently, cited, and wrong — and hard to catch in testsToo bigSeveral topics collapse into one embedding, none of them distinctRecalls a bit of everything, accurate on nothingBy structureOne chunk covers one idea, exceptions includedThe smallest retrieval unit is exactly one complete meaningThe chunk is the smallest unit retrieval can return
09

Chunking: RAG's First Do-or-Die Step

Chunk too small and you lose context; too big and the key point gets diluted. This step decides everything.

When the answer is wrong, don't touch the prompt first — walk forward in this order, and most problems sit in the first two stagesIs it in the index?Search the index directly and pull up the raw passageMany "wrong answers" are content that was never writtenWas it recalled?Log every passage this query retrievedThe right passage may never have entered recallWhat rank?Rank 18 with top-5 taken means it wasn't recalled at allThat's a reranking problemRight material, wrong answer?Paste the passages and the question into a chat manuallyOnly this failure points at the promptTouching the prompt when the first three stages failed is pure waste
10

Where RAG Goes Wrong

Chunking, recall, and reranking — every stage can fail, each with its own symptoms.

One line to keep straight: fine-tuning teaches "how to say it," RAG supplies "what to say"Fine-tuningChanges the model itself: continue training on input-output samples, adjust the weights, shift output tendenciesRight for stable format, style, and domain register; can't add knowledge, no provenance, slow updates, may degrade other abilitiesRAGChanges the material it sees: the model is untouched; relevant material is handed over at query timeAdds knowledge, can cite sources, updates fast; doesn't change how it speaksMissing knowledge means RAG; off tone means try prompting first; fine-tuning comes last
11

Fine-tuning or RAG

To add knowledge, it's almost always RAG. When fine-tuning is actually the right call.

2上下文工程把资料装进有限的窗口
3知识库 RAG把检索和上下文接起来

Agents & Skills

From chat to getting things done — the layers in between: tool calling, loops, Skills, and MCP.

1认识从聊天到干活，差在哪
"AI" isn't one thing; it's three layers stacked togetherApplicationInterface, flows, permissions, data, copy — the only layer users seeAgentTools, loops, memory, skills, permission boundaries — makes the model actModelDoes one thing: predicts the next token from its inputThe model only takes the blame for "the materials were right and it still answered badly"
01

Model, Agent, Application

A diagram that separates the three layers. Confuse them and you'll treat a product problem as a model problem.

A model's "call" is really a request — your code does the actual work1. Send the tool listYour code sends each tool's name, parameter shape, and usage descriptionHow well the description is written decides whether the model uses it right2. Model writes a callThe model emits a structured request in its reply, usually JSONThe format follows the API spec; underneath it is just text3. Parse and executeYour code parses, validates arguments, calls the function, catches errorsArguments cannot be trusted; validation belongs at execution4. Feed the result backWrite the result or error back into the context so it can plan the next stepErrors that never return cause retries until the agent hangsThree of the four boxes are your code — it writes requests, does the work, and hands back what it needs to see
02

Tool Calling

The model can't do anything by itself. You lend it your hands; only then can it touch the world.

Think, act, observe, think again — each round writes its result into the contextThinkRead the context, decide: call a tool or answer directlyActExecute the tool call and get a resultObserveWrite the result back into the context — don't skip thisThink againRe-decide with one more round of information: continue or wrap upAll of the loop's memory lives in the context: if the window holds it, it remembers; if it gets squeezed out, it forgets
03

The Agent Loop

Think, act, observe, think again. That's how anything running dozens of rounds spins.

Two words people keep mixing up — the split is actually simpleSkillInstructions for the model: steps, formats, boundaries, examples; touches no external systemSolves "it doesn't know how" — you change it by editing a document, zero codeMCPA standard way to connect: wraps databases, order services, email into one shapeSolves "it can't reach" — connection only, does not fix usage"Doesn't know how" means Skill; "can't reach" means MCP; most real agents use both
04

Skills and MCP

One carries knowledge, the other carries interfaces. They solve different problems; don't mix them up.

The payoff of splitting is tempting, but the cost is real — count it firstOrchestratorA lead agent understands the goal, splits the task, delegates; workers report backCost: the lead becomes a new bottleneck and single point of failurePipelineStrict order A to B to C; each step has one role and can be evaluated aloneCost: total latency is the sum of all stepsPeerEach agent does an independent task; results are compared at the endCost: who judges, and by what standard, is often never settledIf a single agent isn't hitting context, role, or parallelism problems, splitting is pure cost
05

Multi-Agent Collaboration

When it's worth splitting into several, and where the added cost shows up.

Three layers, each governing a slice — mixed into one blob, they fight each otherSystem promptDefines who you are, global rules, and the security floor; sent at the front of every roundRarely changes; one edit affects everythingStanding rulesSkills, memory, and project conventions, loaded on demandConfigures behavior inside the floor, can't override itOne-off instructionsWhat the user wants this round — short, concrete, valid oncePicks actions within the framework, can't change itThe top layer decides what's allowed; the bottom decides what happens
06

Layers of Prompts

System prompts, standing rules, and one-off instructions each govern a slice, with different priorities.

Drift isn't the model turning bad; it's a loop losing its constraints in three placesGoal driftAs the context grows, the model starts treating subtasks as the goalStop it: endpoint conditions, high-attention placement, permission to declineTool misuseToo many vague tools, or external content steering it to the wrong callStop it: clear tool docs, tier risky tools, validate args at executionNon-converging loopFailures don't carry a reason, so it keeps retrying in placeStop it: hard step cap, retry with the failure reason, convergence ruleEvery restraint comes from the constraints you build, not from the model waking up
07

Three Ways Agents Drift

Goal drift, tool misuse, and loops that don't converge — each with its own way to stop it.

Agent permissions must be graded — not "give or not give" but "how far do we go"Read-onlyRead docs, data, and code; mistakes can simply be redoneNo gate neededLow-risk writesDrafts and temp files; auto-approve but keep a logReversibleHigh-risk writesDelete, overwrite, send, or pay; denied by defaultIrreversibleHuman checkpointActions outside scope stop and wait for approvalA human decidesThe more irreversible the impact, the more it depends on a person
08

Permissions and Human Checkpoints

Which steps to gate so it can't go and delete the database.

Measuring an agent means more than checking whether the final answer is right — all three layers countOutcome layerCompleteness, correctness, side effects; human scoring or a stronger model, judged periodicallyProcess layerTool order, retry count, blocked actions; automatic from logs, run on every changeCost layerRounds, total tokens, duration, call count; fully automatic, run oftenAfter the first run, the critical step is keeping the record as a baseline — without one, no later change can be compared
09

Evaluating Agents

No eval means running blind. A minimal acceptance plan you can actually run.

2组织给它装能力的几种方式
3失控别让它自己跑太远

Cost & Safety

How the token bill is calculated, how to defend against prompt injection, and which judgments should never be handed over.

1账单这笔钱到底怎么算
Reading one call's bill comes down to a single formulaInputSystem prompt, history, retrieved materialBilled at the input price, discounted on cache hitsOne callInput x input price + output x output priceOutputThe answer the model generatesUsually 3–5x more expensive than input Compare the bills two implementations produce on the same task, not the numbers on the price sheet
01

How a Call Is Billed

Input, output, and cache are priced differently — by several times. Read the price sheet before choosing a model.

Images aren't billed per image—they're converted into equivalent tokens by resolutionHow images are billedThe model cuts an image into patches, understands each, and stitches them together. The patch count is set by resolution, so billing tiers by the long edge and jumps at thresholds.Why it's expensiveThe cost is in understanding: pixels are information-sparse but compute-dense. Multiple images plus text makes the bill climb even faster.Ways to cut the billLower resolution to what the task needs, send only the relevant part, crop out noise, use discount lanes for batch jobs, and keep image understanding separate from image generation.
02

Why Image Generation Costs 10x

How image tokens are counted, and how a resolution change jumps the bill up a tier.

Prefix caching: the unchanged part bills at a deep discount, the changed part at full price — put the stable stuff firstStable prefixFixed content goes first, identical on every requestCache hitThe matching part bills at a discount, as low as a tenth of the input priceCost dropsA typical support agent cuts input cost by over 80%Track cache hit rate as a metric; moving it from 0 to 80% may be your cheapest optimization
03

Cache Hits and Saving Money

A cache hits only when the prefix is unchanged. Reorder your prompt and the bill can halve.

A sentence inside external content gets treated by the model as an instruction to itselfHidden in contentA webpage, email, or document carries a line that looks like a commandAnyone who can put text into your context can do itMixed into contextExternal content reaches the model alongside the user's requestRetrieved chunks and tool results can carry it tooExecuted as a commandThe model can't separate instructions from data, so it compliesIt only sees tokens; there is no internal flagDamageLeaks internal information, performs actions it shouldn'tIndirect injection: the attacker never talks to your systemStop it with permissions and process, not by asking the model to behave
04

Prompt Injection

The model can't tell which words are instructions and which are data. This is a flaw at the principle level.

The same sentence goes to different places depending on the tier you're usingWeb appConsumer products, loosest policy: conversations may improve the model, human reviewers may sample themMost convenient for personal use, worst place for sensitive materialAPIDeveloper access; most mainstream providers don't train on it by default, but retention and encryption details varyThe default choice for building AI features; terms differ by providerEnterpriseCommercial agreements: explicit no-training promise, NDAs, agreed data residencyThe only tier where data boundaries go into a contract; near-mandatory for complianceCheck the terms of the exact product you use—don't generalize
05

Where Conversation Data Goes

Web, API, and enterprise tiers route your data to different places.

AI writes code fast and often — and it writes these three old flaws fluently tooHardcoded secretsAPI keys and database passwords written directly into code and committedOnce in the cloud, the secret is public — and the bill lands on youBroken access controlChecks "are you logged in" but not "can you see this record"A logged-in user can view others' orders or edit their profilesInjectionUser input concatenated straight into SQL, commands, or HTMLTreats data as instructions — the code-world cousin of prompt injectionAI is the assembly line; you're the inspector. A fast line doesn't mean the product passed
06

Common Flaws in AI Code

Hardcoded secrets, broken access control, injection — the ones AI leaves most often.

Hand it to AI? Ask three questions; three seconds, you have your answer1. Reversible?Can the email be recalled? The file recovered? The money refunded?Irreversible: no autonomous execution2. How wide?One person and one record, or all users and the whole production system?Everyone affected: keep a human checkpoint3. Verifiable?When it is done, can you tell right from wrong? Is there a source or a standard answer?It could fabricate and you would never know: be carefulOne "no" among the three — downgrade: add a human confirmation, or shrink the scope
07

What to Hand to AI

A trust test you can run in three seconds.

No need to verify line by line — three moves filter out most fabricationThree movesCheck the source, find a counterexample, recompute key numbersCheck the sourceFor every citation, figure, name, or paper, open the originalFind a counterexampleFor every "always" or "never," imagine a case where it failsRecompute key numbersPercentages, amounts, dates, conversions — calculate them yourselfFor the rest, ask one question: would it hurt me if this is wrong?
08

Three Checks on an AI Answer

No need to verify line by line; three moves filter out most fabrication.

AI is an excellent executor and a poor decision-maker—the boundary sits right thereImplementationCompleting code, generating documents, batch organizing, translating and rewriting, quick first drafts. Fast, tireless, unlimited, never boredHand it to AI and maximize efficiencyJudgmentNo stance, no values, no motive, no capacity to bear consequences. Goals, trade-offs, and accountability can't come from itAlways your job; never hand it overAI is your colleague—not your boss, and not your scapegoat
09

The Human Boundary

AI can complete the implementation, not the judgment. Which part is always your job.

2安全它会信错谁
3判断它的活和你的活

做产品

Basics

What everyone in this trade is assumed to know. Skip if you already do.

1认识这一行先知道自己在什么盘子里
A PM doesn't ship code or design files. It ships judgmentIncomingMarket, users, business, techFour conflicting inputsPMTranslate + trade offOutgoingWhat, how, what firstDecisions you can act on The same need produces a pile of features or a business that runs itself, depending on judgment
01

What a PM Does

The boundaries of the role, and where it sits in a company's org chart.

Sixteen steps, each phase with a clear deliverableThink it throughUsers, problem, value, competitorsThe phase most often skippedDesign itFlows, screens, prototype, planSkipping review and heading to devBuild itDevelopment, integration, test, acceptanceRequirements keep moving until the last minuteRun itLaunch, rollout, metrics, retroLaunch is treated as the finish lineEvery step you skip, you pay for somewhere downstream
02

The Product Workflow

The stages from idea to launch, and when each role shows up.

Consumer and enterprise differ not in the interface but in decisions and accountabilityConsumerUsers can leave at any momentEvery extra step is churnRemoving steps is the default moveEnterpriseUsers can't leave, but every step sits on a real process and a real line of accountabilityRemoving a step can break thingsAdded steps need confirmationThe same feature can differ between the two by a single 'Confirm' button
03

Product Types & Directions

Consumer, enterprise, platform, tool, content — each judges success differently.

Picking an industry is picking a set of yardsticksMature industriesE-commerce, social, content, toolsNewcomers get the scrapsEmerging groundGoing global, enterprise software, AI appsHarder to judgeHow to chooseStage first, then size; take the edge you have; count whose lunch you're stealingDon't judge new ground with mature-industry thinkingThe industry sets your ceiling and decides who you fight
04

The Internet Industry Map

Where e-commerce, social, SaaS, content, and tools stand today.

A business model isn't how you make money; it's who pays, why, and whether you earn more or less as you scaleYour productUsers ≠ payersAdvertisingUsers pay nothing; advertisers paySubscriptionPay per cycle, buying ongoing valueTake ratePlatform matches; splits a cut of the transactionFreemiumBasic is free; advanced features chargeWhat decides success isn't the charging method itself, but four economic laws underneath it
05

Common Business Models

Where the money comes from: ads, subscriptions, commissions, freemium — and how to validate each.

You don't need to write code, but you do need to know which layer something happens onBrowserPages, interactions, inputThe frontend can only change its own UIAPIThe contract between frontend and backendPermission checks must happen hereServerBusiness rules, computation, authMoney, permissions, and others' data only live hereDatabaseTables linked by idsThis is what a data model looks likeEvery request walks this path; when something breaks, first locate which layer
06

Understand Tech

Frontend, backend, APIs, databases, auth, deploy, security — a minimal technical baseline.

There's a lot of jargon, but few terms show up often. Learning them by scenario is enoughGlossaryGrouped by scenario, not alphabetNeeds & usersPRD, persona, scenario, pain pointProduct & designMVP, prototype, interaction, UXDevelopment & launchFrontend, backend, API, rollout, rollbackData & growthConversion, retention, north star, A/BHear a term in a meeting, come back and know which part it belongs to—that's enough
07

Glossary

The jargon of both product and tech — look it up so you're not lost in meetings.

Models don't hand you answers; they work as checklistsThinking modelsIndexed by stageResearchPEST, SWOT for the outside and yourselfRanking needsKANO decides what to build firstExperienceFive Elements locate which layer a problem is inGrowthAARRR breaks the user journey into stagesThe test for whether a model earned its keep: did your decision change?
08

Product Thinking Models

PEST, SWOT, AARRR, KANO, the Five Elements — and when to use which.

Tools solve a fluency problem, not a capability problemThink it throughWhiteboard, mind map, AI chatDon't spend the time on formattingDraw itFlowcharts, prototypes, architecture diagramsOutput must feed the next stepWrite itDocs, PRD, requirementsUse templates, don't start from zeroValidate itCompetitor research, interviews, dashboardsFindings must flow back into judgmentPick a tool by one criterion only: does it cut the cost of moving things between steps?
09

Tools & Templates

Diagrams, docs, competitor research, design files — a handful of go-tos is enough.

Product judgment isn't built by reading; it's built by shipping things that break a few timesMake itTools, a technical base, the workflowThe bottleneck is fluencyMake it rightRequirements, judgment, thinking modelsThe bottleneck is judgmentGet it movingCollaboration, process, making others productiveThe bottleneck is communicationExplain itBusiness, expression, pulling in resourcesThe bottleneck is perspectiveRead one or two books thoroughly per stage—that beats listing twenty
10

Learning Path & Reading List

From beginner to leading a product line — what to learn along the way.

The future of work: agents execute, people set directionRowingWriting code, making decks, organizing data, tracking executionMostly delegable to agentsThe faster it gets, the more you should hand it offSteeringSetting direction, choosing goals, making opinionated callsStays with humans for nowThe abstraction level keeps risingWhen everyone has the same tools, judgment and point of view are what separate people
11

Steer, Don't Row

The fundamental shift in how work happens: agents execute, people set direction.

2懂点技术够跟 AI 和工程师对话就行
3手边的东西随时能翻出来用的

Find Problems

Where needs come from, which are real, which are worth doing.

1认识先搞清楚需求到底是什么
The same thing has three phrasings, and only the middle one is the needWhat the user saysThe solution they can think of, like 'can we add a button to export to Excel'The state they want to reachThis is the need. On monthly reconciliation day, turn scattered revision records into a list they can send straight to the clientGet this middle layer clear and there can be several solutions to choose fromThe feature you plan to buildThe solution you can think of, like one-click statement generation that also exports to ExcelThe top and bottom layers are both solutions; only the middle layer is the need
01

What Is a Need

What users say, what they actually want, and what you plan to give them — three different things.

Where a need comes from determines how it should be validatedNeedNever enters the pool without a sourceUser feedbackSomeone is stuck, and at which stepBusiness goalsWhat the company wants this quarterData anomaliesWhich step dropped harder than usualCompetitor movesWhich direction others bet onCredibility, high to low: data, observed behavior, what users say, competitors — always paired with the original scenario
02

Four Sources of Needs

User feedback, business goals, data anomalies, competitor moves — each with different credibility.

A fake need is the user answering a hypothetical questionWhere fake needs come fromPoor expression, over-designing, wrong assumptions, and tech-driven featuresThe four sievesAsk about facts that already happened, not future intentionsWhen the answer is vague, the verdict is usually outTypical casesUsers ask for "more options"; what improves things is "better defaults"Fail more than two of the four, and don't build it
03

Real vs. Fake Needs

Telling apart what they say they want from what they'll actually use.

People can't predict the future but they remember the past—ask only about the pastAsk intention"Would you use this feature?" "Would you pay for it?"They picture an idealized self and answer with politenessAsk behavior"When did you last hit this? What did you do about it?"You get facts that already happened—the most credible signalHearing "right now I manage it with my own spreadsheet" beats ten "it's important"s
04

Ask About Behavior, Not Intention

Ask how they did it last time, not whether they'd use it.

If a problem is real, users are already solving it with some clumsy methodThe existing workaroundA free requirements specA self-built spreadsheetColumn names are fields, colors are statuses—the clearest needsScreenshots and saved chatsThe real need is retrieval: 'save it now, find it later'A little script they wrotePainful enough to learn to code—your best early usersSomeone they hiredThey're already paying, and the amount is your pricing anchorSeveral tools stitched togetherYour opportunity sits in the seams between those toolsThey gave upReal need, too costly to hack—a possible new marketNo workaround means the problem isn't painful—or it's so painful nobody can cope
05

Find the Existing Workaround

How are they hacking around it today? No workaround means it isn't painful enough.

The three dimensions are multiplied together, not addedPeopleHow many match this description. It sets the ceiling—few people but high willingness to pay can still workFrequencyWhen did they last hit it, and before that? It decides whether a habit can form, and retentionPainWhat were the consequences when it went wrong? It decides whether they'll pay, and how muchA zero in the product can't be offsetIf you can't size it, don't build it yet—not sizing it means you don't know these people well enough
06

Is It Worth Doing

Frequency, pain, audience size — if all three are low, don't.

Abstract needs can't produce a design; concrete situations canWrite the anchorWho, when, what they want to achieve, where they're stuck, and what they do todayMeasure solutions with itRun every proposal through the anchor; the ones that don't fit eliminate themselvesWithout an anchor, it comes down to who talks loudestPut it in the promptHave AI state the hard constraints, the main task, and what common approaches don't fitThe third question is the most useful: it forces AI to say "this doesn't fit the scene"
07

Scene Anchor

Replace a vague "users find it inconvenient" with a specific person in a specific moment.

An interview is a fixed sequence, and your solution never comes outWarm-upAsk what a normal workday looks like — build context, let them relaxRecallAsk when this last happened, and pull them back to one specific dayDon't jump straight to the productProbe"Walk me through what you did first that day" — rebuild the full stepsWrap-upAsk who else is worth talking to, and get the next personThe moment your solution appears, the rest of the hour is wastedYou want facts, not evaluations
08

How to Run a User Interview

Open questions, probe for details, avoid leading — a reusable interview guide.

Keep asking until the answer stops being 'because the system doesn't support it' and becomes a business factLayer 0 · SolutionWhat they want. Usually some interface they've seenLayer 1 · TaskWhat they'll do with the solution—context appearsLayer 2 · GapWhy they must do it by hand—where the system breaksLayer 3 · Structural problemWhy the gap exists—often the original information architectureLayer 4 · Business factWhy it was designed that way originally—you can stop hereAfter reaching the bottom, step back one layer and pick a fix at a cost you can afford
09

The Five Whys

Trace the surface request to the root cause, then decide what to build.

Prioritization isn't about a perfectly correct order—it's about making every cut in line give a reasonBucket with KANO firstFive categories. Bucket first, then rank: whether to do it at all, then which one firstThen apply the scoring formulaPriority = value ÷ cost. Value is a multiplier—if it's zero, the result is zeroMaintaining the poolEvery entry has a source and context, cuts leave a record, prune regularlyIf the five cells can't be filled, it isn't thought through—don't queue itA backlog's value is that it can be read to the end
10

Backlog & Prioritization

KANO categories plus a sorting rule—not whoever shouts loudest.

2原则判断标准
3方法可以照着做的步骤

Define Product

Who it's for, what it solves, where the boundary lies.

1认识一份产品定义包含什么
For whom, solving what, and where it stops—miss one and every decision waversProduct definitionAll three parts requiredTarget userSpecific enough to picture one person: job, device, skill level, daily workCore valueHow they cope today, and why they should switch to youScope boundaryA list of what's in, plus a list of what's explicitly outAfter writing it, run the three questions; if you can't answer one, go back and revise
01

A Definition Has Three Parts

Target user, core value, scope—miss any one and every later decision wobbles.

Ask which stage the product is in first, then decide what to doExplorationIs there a small group that must have it? Watch week-1 retentionGrowthHow do similar people find you faster? Watch acquisition and activationMaturityHow do you keep users and make money? Watch paid conversion and renewalDeclineMilk it as-is or pivot? Watch churn and maintenance costBig rescue investments on old features pay far less than a new directionBuilding solo, you spend most of your time in exploration
02

The Product Lifecycle

Exploration, growth, maturity, decline — each stage has different priorities.

How you charge ends up deciding what the product becomesSubscriptionMust prove "I'm still helping you this month"Homepage shows accumulated value at a glance; retention mattersOne-time purchaseMust prove "this alone is worth the price"Feature completeness is the selling point; "done and gone" is fineAdvertisingMust prove "lots of people stay here a long time"Feeds and infinite scroll—in direct conflict with getting things done fastUsage-basedMust prove "the more you use it, the more you get"Usage must be visible, with quota reminders. Most AI products go this wayFree users sincerely recommend you; paid users feel the money beat the time saved
03

How the Business Model Connects

Your product definition has to answer where the money comes from, or it stalls after launch.

When one sentence fails, you haven't decided what to give upFill the four boxesFor whom, when, from what state, to what state—none can stay emptyTest itRead it to someone with no context; a rough paraphrase means it passesIf you have to explain, it's not right yetRewrite it oftenDirection drifts; rewrite regularly, and stop when all four boxes changeTwo target users in one sentence and it falls apart
04

One-Sentence Product

If you can't say in one sentence what you make and for whom, don't let AI start yet.

A product can do only one thing outstandingly wellFind the main threadThe action users repeat most, the sentence they'd say to a friend, the thing you can't cutSplit the resourcesSeventy percent on the main thread; the other four directions split the restHold the lineEvery new feature answers one question: does it strengthen or dilute the threadSlightly better moves nobody; it has to be felt at a glanceThe thread can change, but only one at a time
05

The Single Thread

Find the product's main task; every other feature is a supporting role.

Built for everyone means built for no oneThe broad approachTarget every freelancer — the potential market looks hugeEveryone thinks "close, but not quite for me" — big market × terrible conversion = no usersThe narrow approachTarget designers who coordinate over WeChat — a much smaller market"This was made for me" — small market × high conversion + word of mouth = real usersBecome irreplaceable to a small group first, then expand outward
06

Narrow the Audience

Built for everyone means built for no one — serve a small group first.

A to-do list is intent; a not-doing list is the actual decisionWhat to doListing it costs you nothingIt is intent — it cannot stop scope creepWhat not to doEvery line costs you somethingIt is a decision — it blocks a dozen "might as well add it" momentsScope blowout is the accumulation of "might as well add it" moments; the not-doing list is what pushes back
07

The Not-Doing List

Writing down what this version won't do is more powerful than writing what it will.

The goal of v1 is not to cover every feature; it's to let one real person finish one thingHorizontal cutBy technical layer: finish all screens first, then all APIsUntil everything is done, you have nothing you can validateVertical cutBy one complete user task: interface, logic, and data pierced through togetherAs soon as the first slice is done, someone can use it and real feedback startsThe test: when this slice is done, can you hand it to a real person and learn something from their use
08

Minimum Slice

Cut down to the smallest scope that runs end-to-end, then add on top.

Borrowing structure saves the user's learning cost; copying features saves your own thinking costWhat to borrow, what not toStructure, interaction, naming—yes. Features—no. Copying saves your own thinking costWhere to borrow fromStart with what users use daily, then mature products with the same structure, then system conventionsNote for the AIOne concrete reference beats a pile of adjectivesAfter borrowing, say where you differ—otherwise users ask "then why you?"Differentiation lives in how you solve the problem, not in what the buttons are called
09

Borrow the Form

Take a proven structure from an existing product instead of inventing from zero.

A blueprint orders what to validateOrder by hypothesesWrite what each version validates; the feature list is an outcome, not a starting pointFour lines per versionThe hypothesis, what you'll build, which number to watch, and what counts as successCan't fill four lines? The version isn't thought throughRe-read after each releaseIf it held, follow the plan; if not, test the same hypothesis a different wayA schedule can't fail; a blueprint gives every version a clear verdict
10

Product Evolution Blueprint

Map the next few versions into one picture, so each release has an answer.

Narrative before product: explain the why first, then decide the whatTell the storyDeep understanding of tech + deep understanding of the buyerThe step most often skippedTest the storyPitch it to 100 people, refine relentlesslyPitch to people who cheer you on, hear no truthThen buildCommit to the product form only after the narrative holdsBuild first, invent the story later — the most expensive reworkHow you talk about the product can — and should — come before building the product itself
11

Nail the Story Before Building

Product marketing fit matters as much as product-market fit — and it comes first.

2原则判断标准
3方法可以照着做的步骤

Design Structure

Information architecture, page maps, data models. How the product is organized.

1认识信息架构是什么，有哪几种
Architecture only works if the user can answer three questions on any screenInformation architectureThe structure that answers those three questionsWhere am IAnswered by nav highlight, breadcrumbs, page title. Without it, users hesitate, unsure they came to the right placeWhere can I goAnswered by siblings, child entries, related links. Without it, users think a feature doesn't exist—they just can't find itHow do I get backAnswered by clickable breadcrumbs, back buttons, persistent top-level entries. Without it, users hit browser back or start overWhen a screen feels off, separate the layers: is the thing misplaced, or is it placed right but hard to operate
01

What Is Information Architecture

Let users always know where they are, where they can go, and how to get back.

The relationships within the content decide the structureHierarchyContent naturally has parent-child relations; one thing belongs to one categoryCost: when category boundaries blur, users have to guess how you classified thingsMatrixOne thing has multiple attributes; users find it from different anglesCost: needs good filters and sorting, or you're handing the hard problem to usersLinearSteps have a strict order; the next step is meaningless before the previous oneCost: needs back-and-save mid-way, or everything is lost when users break offOrganicAssociations matter more than belonging; exploration itself is the valueCost: new users get lost—they need a landing page or recommendationsIf one thing can belong to two categories, it's a matrix—forcing it into a hierarchy leaves items forever unplaceable
02

Four Organizational Structures

Hierarchy, matrix, linear, organic—different content suits different organization.

Desktop's space advantage and mobile's gesture advantage each drive different approachesInformation per screenDesktop fits two or three columns; list and detail share the screenMobile shows one column and splits the same content into two or three pagesNavigation placementDesktop keeps a left or top rail always visibleMobile gets four or five bottom tabs, or a hamburger menuDepth toleranceDesktop handles three to four levels, propped up by breadcrumbsMobile handles two to three; go deeper and users don't come backInput & usage contextTouch targets set the hit area; one hand and interruptions shape interaction and dwell timeAssume mobile users will be interrupted mid-task: save progress and let them pick up where they left off
03

Web vs. Mobile Differences

Screen size, nav position, depth tolerance all differ — structure can't be copy-pasted.

One question behind the choice: how many top-level modules, siblings or nestedLeft nav onlyDocs, notes, a single system, five to nine modulesAll content belongs to one thingTop nav onlyCorporate sites, e-commerce, content sites, few top-level entriesThe page stays open for contentLeft top-level, top second-levelAdmin panels with many modules, more than nine top-level itemsEach top-level item has several childrenTop top-level, left second-levelPortal plus subsystems, the top bar switches independent systemsOn mobile it usually splits into separate entriesThe third and fourth are the easiest to mix up: if the left content changes entirely after a top switch, it's the fourth form
04

Four Navigation Forms

Left-only, top-only, left+top, top+left — depends on how many top-level modules you have.

Feature-complete and clearly prioritized are two different thingsFlat layoutFive equally sized cards, everything weighing the sameUsers face a multiple-choice question on entryOne job per screenOne main task; the rest gives way — smaller, lower, or folded into the navUsers know what to do without thinkingIf users could tap only one thing, which should it be? If you can't answer, the page isn't designed yet
05

One Screen, One Job

Each page serves one main task; everything else steps aside.

How you group things is how users understand the productGroupingBy task, object, role, frequency, system moduleBy taskThe default choice for most cases — Risk: tasks must be stable, or the structure gets rebuilt on redesignBy objectThe product has clear core entities, like projects and customers — Risk: users who don't think in entities find it roundaboutBy roleDifferent roles use completely non-overlapping features — Risk: one person wearing several roles switches constantly, which is annoyingBy frequencyWorks as a secondary sort, not a primary grouping — Risk: frequency changes, and it differs from user to userBy system moduleAlmost never the right time — Risk: you're handing your implementation details to users to understandGroup it wrong, and every future hunt for something makes users guess again
06

Grouping and Naming

How you group things is how users understand the product.

Every extra level loses a batch of peopleGroup visuallySplit a dozen items into three groups on one screen with subheadings or dividersUsers still scan it all at a glance—no extra tapsLayer structurallyLooks the tidiestEvery user makes one more 'which category is this in' guessBreadcrumbs can't fix depth—they only answer 'where am I'
07

Wide and Shallow

Five to nine top-level items, main paths no deeper than three levels — go deeper and nobody arrives.

Start from the information; page count is pushed out by content volumeStart from pagesYou fill in content by copying interfaces you've seen, patching gaps as you go—the page count keeps shiftingHalfway through a page you find a missing field and rework the layoutStart from informationList everything first, then split pages by frequency and length—the result falls out on its ownThe inventory doubles as a data-model draft; the table structure is mostly decidedHigh-frequency and short goes in a list; low-frequency or long goes in a detail view; rarely seen isn't shown at all
08

Content Inventory

List every piece of information to show before deciding how many pages to split.

Open sorting before closed—reverse the order and the round is wastedHow to run itPrepare cards, open sort, closed sort, probe hesitations—five to eight people is enoughHow to read the resultsWatch which cards stick together, which wander, the group names, and the hesitationsHow to do it soloRemote whiteboard, reverse test, check competitors, or let AI take a first passAI gives the common splits of similar products, not your users' thoughtsTheir group names work directly as navigation labels—better than yours
09

Card Sorting

Write features on cards and let users group them—more accurate than guessing alone.

Page count is scope; jump relationships are complexityList pagesEach standalone page becomes a box; modals and drawers belong to their host pageConnect hierarchySolid lines for parent-child relations, matching nav and breadcrumbsConnect jumpsDashed lines for where you go after an action, matching buttonsMark statesFlag pages that need login, need permissions, or can be emptyUnless you draw it, those connections don't exist for the AI
10

Page Map

Draw every page and the jumps between them before you start building.

The order can't be reversed; mashing them into one makes no sense to anyoneFunctional structureWhich feature modules exist and how they divide — a 2-to-3-level tree of function names, for scopingInformation structureWhat information each business object holds — objects as roots, fields as branches, for data modelingProduct structureHow functions and information land on pages — page frames with their content, able to stand in for prototype reviewIf you draw only one, pick the information structure — it's the only one that forces a database rebuild when wrong
11

Three Structure Diagrams

Feature structure, information structure, product structure — draw each separately, don't mash them up.

Changing the interface takes minutes; changing the data model means migrating the data already stored against itFind the nounsCircle the nouns in the content list and scenario: projects, customers, revision recordsJudge entitiesSomething you can create, edit, and delete on its own is an entity; the rest are fieldsDefine relationshipsAsk pairwise: how many Bs can one A have, and how many As can one B haveCheck for duplicatesIf the same information appears in two tables, it should probably be its own entityMany-to-many needs a join table; the later you define it, the harder it is to change
12

Data Model First

Get entities and relationships wrong, and no UI tweak will fix it.

2原则判断标准
3方法可以照着做的步骤

Design Interaction

Paths, states, feedback, edge cases, permissions. Every moment that goes wrong.

1认识交互设计在管什么
What you can see in the picture cannot carry "how to use it"Visible in the pictureLayout, fields, where buttons sitInteraction notesWrite the behavior into the specInvisible in the pictureWhat a click does, when it's disabled, what happens on error If you do not write it down, the builder can only guess
01

Interaction Handles Behavior

A prototype shows the look, not the behavior. Write it down, or the AI can only guess.

Know components by the problem, not by the nameKnow components by problemPick one, pick many, input, too much content, show data—every class has mature answersEasy to get wrongDropdown or radios, modal or drawer, pagination or infinite scrollEvery component has statesDefault, hover, pressed, focus, disabled, loadingFocus is the most missed state—it decides whether keyboard users can use your productThe real skill: look at a need and say instantly "this is a pick-one-from-a-set problem"
02

Common Components

Dropdowns, steppers, pagination, drawers—know them all before choosing.

The hint must match how serious the matter isBadgeHow many items need attention. A small dot with a number on the icon; interrupts nothingToastPoint users to a new feature. A short bubble in the corner; read it or notBannerImportant state changes or urgent info. Top banner or centered modal covering the contentBulletinLong-form info, like maintenance notices or changelogs. A dedicated section users open deliberatelyPushUsers aren't in the app but you need them to know. The system notification center; overuse gets it disabledShow a trivial message in a modal, and users learn to click OK without reading
03

Five Kinds of Hints

Badge, toast, banner, bulletin, push—escalating in intrusiveness.

Every time you fetch data, the result lands in one of four cellsData componentAll four states requiredNormalRendering as usual when data existsEmptyNo data. Distinguish 'never had any' from 'filtered to nothing'LoadingA skeleton screen, not a centered spinnerErrorState the situation + guide the action, with a retry buttonThe AI writes only the normal state by default; the other three have to be pulled out of the spec
04

Cover All Four States

Empty, loading, error, normal. AI writes only the last one by default.

When the user clicks, something on screen must change; without change, their default assumption is that they missedInstant feedbackAppears the moment of the click, answering 'did I click?'. Button color change, loading stateResult feedbackAppears when the operation completes, answering 'did it succeed?'. Toast, error messageState feedbackPersists after the operation, answering 'what's the state now?'. The data actually changed, the flag flippedShow 'saved' while the list still shows old data, and users refresh the page to check
05

Every Action Needs Feedback

After the click, how does the user know it worked?

Confirmation dialogs spread the cost the wrong wayConfirm first, then actA hundred actions, a hundred interruptionsDismissed on reflex after a while — the one real mistake still slips throughAct first, offer an undo windowNormal actions are never interruptedThe one time it goes wrong costs one second to click undoTwo questions decide it: is the action reversible, and how many people does it affect
06

Undoable Beats Confirm

Give dangerous actions one chance to undo, instead of three confirmation dialogs.

What the user sees should equal what they can doThree kindsPage, action, and data permissions work differently; data is the easiest to missAdd a role layerRBAC: users get permissions through roles; adding people or rights becomes configHide by default in the UIDon't render what they can't use; show-but-disable only when they can applyHiding on the frontend is experience, not securityWrite it into project rulesPut the rules in CLAUDE.md; every API checks permissions on its ownThe backend has to gate again
07

Permission as View

Features you can't access shouldn't appear and then error.

The main path is three to five steps long; every step needs an exitSet the start pointWhere the user enters; a different start means different knowledgeList the main pathEach step on the happy path, kept to three to five stepsMark decision pointsAsk if a branch forms here, and draw both routesMore than five steps is a signal to split the task or merge stepsAdd fallbacksInterrupted, failed, or without permission—each lands somewhereDraw return pathsEvery endpoint must lead back to a stable place, no dead endsAfter the path, add an emotion line and the design decisions surface on their own
08

Map the User Journey

Break a complete task into ordered steps, mark each step's exit.

Missing states come from states not appearing in the requirementList by elementLists, buttons, and inputs each have their own states; don't mix them by pageWrite triggers concretely"API returns 500 or times out" is testable; "when an error occurs" isn'tAsk all six questionsEmpty, loading, error, disabled, processing, extreme — plus two more for inputsScan the list yourself before building — adding a state now is just one table row
09

State Checklist

Ask each component what states it can appear in, before you start writing.

The first screen a new user sees is almost entirely empty, and it decides whether they take a second stepThe empty first screenThe screen that decides whether they stayOne value statementWhat will live here, and what it does for themOne primary buttonThe only next step—never three choicesOne shortcutImport sample data, start from a template, connect an existing accountA sense of progressWhich of three steps you're on; knowing what's left helps people finishWhen it's done, walk through it with a brand-new account—the only way to find the problems
10

Designing the First Run

What's on screen when a new user opens the product and there's no data yet?

Every extra field drops another batch of people willing to finishCut fields firstAsk 'what if they don't fill this in'; don't ask for what you can deriveWhere the label goesLeft, top, inline, or right-aligned—each has its placeHandling errorsForgiveness works across three moments; miss one and users get stuckFive easy winsDefaults, autofill, steps, grouping, and the right mobile keyboardCompletion rate, error rate, and time spent—read all three together
11

Restraint in Forms

Every extra field drops another batch of people who won't finish.

The formula: state the current situation plus the next action—both parts requiredState the situationWhat happened, in language the user understands—no status codes exposedGive the next actionWhat they can do now—a way out, a way to recoverA message that reports a problem without a way out is as good as unwrittenMatch the severity levelBlocking errors use a modal, general info a toast, optional hints a bubblePick the form by severity in three tiers, and never jump a tier
12

How to Write Hints

State the situation plus the next step. No exclamation marks, no trailing punctuation.

2原则判断标准
3方法可以照着做的步骤

Design Interface

Layout, hierarchy, color, typography, design systems.

1认识视觉是怎么起作用的
Users scan first and stop at whatever stands out most. You decide what stands outVisual hierarchyFour levers: size, position, whitespace, contrastSizeImportant things bigger, secondary things smaller. The gap must be real—two or three pixels is no gap at allPositionImportant things go up, to the left, and above the fold. The most effective lever, and it adds no visual noiseWhitespaceGive important things more room around them, isolate them. The most underrated lever—space is cleaner than decorationContrastPull apart color, weight, and lightness. Go all the way or not at all—weak contrast just confusesPrefer position and whitespace, then size; reach for color last
01

Where Visual Hierarchy Comes From

Size, position, whitespace, contrast — four things decide where the eye lands.

The more sparingly you use color, the more effective it isThree propertiesHue, value, saturation. Interfaces lean on value mostFour rolesBrand, neutral, semantic, accent—hold the total to five huesChoosing a brand colorMatch the brand, match the content, take from a reference, or generate with a toolCommon pitfallsDon't rely on color alone; body text contrast at least 4.5:1Dark mode isn't inverting the colorsRecord the exact hex and build a value scale—remembering "blue" will slowly drift
02

Color Basics

Hue, lightness, saturation—and why neutral colors fill 90% of an interface.

Chinese has no ascenders or descenders, so its visual density is higherCopy English parametersLine height 1.4, applied straight to Chinese body textLines stick together; reading feels tiringUse Chinese's own parametersLine height around 1.7, spaces between CJK and LatinEach line is visible on its own; finding the next line is easyNine-tenths of an interface is text; get the type wrong and no decoration will save it
03

Typography Basics

Font size, line height, weight, tracking — Chinese and English use different parameters.

Spacing differences should express relationshipsImprovised11, 13, 14, 15 — a pile of near-but-not-equal valuesToo small to look deliberate, too big to ignore: the page feels messy and nobody can say whyA fixed set4 / 8 / 16 / 24 / 40 / 64 — spacing becomes a meaningful signal8 means tightly related, 24 means these are two different groupsFive or six values is enough; once set, spacing is no longer a decision you make each time
04

One Spacing Scale

Lock a set of fixed values; don't eyeball each spot.

Hierarchy comes from gaps, not from the number of levelsSeven or eight sizesAdjacent levels differ by only 1–2pxUsers can't perceive the difference — it just feels unevenThree to four sizesAdjacent levels differ by at least 1.25xOne big heading per screen, everything else in its placeMost of the time, bolding and graying are more restrained than enlarging
05

Type Scale

Three to four sizes is enough — more and the hierarchy scatters.

A page can be interrupted only so many times — once you run out, you are doneA color per moduleSix elements in six colors tells the user all six matter equallyWhich means there is no focal pointKeep just oneColor only appears where the user should look and clickWhere to click is obvious at a glanceColor is a scarce resource; use it everywhere and it works nowhere
06

One Accent Color

Keep one color; leave the rest to neutral grays.

To separate two blocks, first pull them apart; only when distance isn't enough, use a lineDrawn separatorsBorders and dividers, immediate and obviousThey add visual elements, and overused lines stop working—they become background noiseWhitespace separation8 within a group, 24 between groups—spacing expresses groupingNo extra elements added, and the grouping is just as clearDelete first, add back later: strip every border, then restore only where the structure blurs—usually about 30%
07

Whitespace First

Try deleting dividers and borders first; separate with spacing instead.

Every extra starting line adds another visual pauseUse fewer axesCount the page's left edges first, then snap elements to a few axesWhich alignment to pickRight-align numbers, left-align body text—one block uses one methodOptical alignment firstWhen the numbers line up but it doesn't look aligned, trust your eyeCircles, quotes, and icon centers all need nudgingIt costs almost no time, and the effect is obvious
08

Alignment Axes

Align every element on the page to a few axes—it instantly looks clean.

Without unified components, no amount of page-tuning will fix the patchwork lookEach page invents its ownRadius, type size, padding all differ; one style change touches seven or eight filesThe AI only sees the current page and has no global view—it will absolutely make its ownOne component set, reusedSet the radius once and the next few dozen pages never think about it againNew pages become assembly work, no style decisions leftComponents-first turns a design decision from "every time" into "once"
09

Components First, Then Pages

Without unified components, the page will always look stitched together.

Gather the scattered values into named variables, and the styles stay consistent by themselvesDesign tokensA set of named variablesColorOne primary, six gray steps, three semantic colorsSpacingMultiples of 4, six stepsTypeFont sizes with matching line heights, four stepsRadius and shadowThree radius steps, two shadow levels. More starts to look noisyA design system's value is constraint; the more you add, the weaker the constraint
10

Build a Minimal Design System

Color, spacing, type, radius—four variable groups keep the whole project consistent.

Responsive design isn't squashing; it's swapping in a new structure at specific widthsPick the breakpointsDecide the main battlefield first; three tiers are enough: mobile, tablet, desktopDecide what changesTables become cards, side nav becomes bottom tabs, columns reorder by importanceDon't just stack columns into a single oneTell AIHave it spell out the change per element, propose first, then code after your OKTrying to make both ends equally good usually leaves both awkward
11

Making Responsive Work

Don't dump desktop components onto mobile—think through the trade-offs at each breakpoint first.

2原则判断标准
3方法可以照着做的步骤

Work with AI

Translating your judgment into something AI can execute.

1认识它擅长什么，不擅长什么
The dividing line is not difficulty—it's whether one right answer existsHand to itTurning descriptions into code, refactoring, explaining code, matching a referencePlenty of examples, convergent approaches—it is faster than youKeep for yourselfWhat to build and what to skip, whether a solution fits, trade-offs, acceptanceDepends on your situation and your standards—it can't knowThe calls you skip, it will make for you—and make them look plausible
01

What It Can and Can't Do

It can fill in the implementation, not the judgment. Knowing the line tells you which part is yours.

Fitting in the window doesn't mean every part gets read clearlyOpeningSystem prompt, project rules, and your latest words get the most attentionMiddleKey constraints buried in the middle often don't take effect—the easiest part to loseThe longer it gets, the blurrier; accuracy fades gradually, and it's hard to noticeEndRecent conversation and tool output also tend to be seen clearlyImportant rules go either first, or get restated in the current instruction
02

What Context Is

It only knows what you give it; memory doesn't carry across sessions automatically.

Where a sentence belongs depends on how long it needs to stay in forceConstraintsCLAUDE.md — rules that hold for the whole project, like every component implements four states and spacing uses six valuesSpecdocs/specs/module.md — one module's data, states, and edge cases, like two-level replies and soft deletePromptThe text you type — the concrete thing for this round, like moving a button to the top-rightThe spec layer is the easiest to skip; skip it and decisions live only in conversation historyAll three layers apply at once when code is generated; on conflict, the lower layer wins
03

Spec, Prompt, Rules — Divided

One-offs go in specs, recurring ones in rules, temporary ones in prompts.

All four parts are essential; the last one is the most valuableDataWhich entities, key fields, and relationships — get this wrong and everything else is wrongStatesWhat the screen can look like; at minimum normal, empty, loading, errorEdge casesExtra-long content, zero records, concurrent edits, missing permissions, network failureOpen questionsEverything it would have to guess, listed out for you to answerThis part decides whether you redo the workAsk for a one-page spec and the same error shows up in twenty lines in thirty seconds
04

Spec Before Code

Have it write what to build first, then have it write code.

A big task costs you at verification, not generationPick the step sizeIndependently verifiable, output you will actually read, and cheap to throw awaySplit verticallyBy one complete user action, not by frontend, backend, or database layersConfirm after each stepLook right away; don't save up three steps to verifySaving them up means you didn't really splitEvery step is a save point you can roll back to
05

One Thing at a Time

Break the work into small tasks; don't ask for a whole app at once.

Adjectives constrain direction; a reference constrains the resultWhat to anchorStructure, density, tone, code style, plus one negative referenceHow to give itPoint to a specific part and say what to borrow and what not toWithout that, it copies the features tooHave it paraphrase firstAsk it to state what it will borrow and skip; you catch the misread in secondsReferences inside the project are the strongest: zero room for interpretation
06

Reference Anchor

Give it a concrete reference; ten times more effective than adjectives.

Each part owns one job; miss one and you pay for another rework roundGoalWhat state the user can reach and why — not a technical taskThe most common mistake: writing the goal as "add a useEffect"ConstraintsStack, existing components, must-follow conventions, and what's explicitly out of scopeAcceptance criteriaA checklist you can tick item by item, edge cases included if possibleThe third part is the most valuable and the easiest to skip — it's both the self-check list and your acceptance basis
07

Three-Part Prompt

Goal, constraints, acceptance criteria — miss any one and it drifts.

Every piece of irrelevant content steals attention from what mattersWhat to giveProject constraints, module spec, relevant files, this task's instruction—leave about 30% blankState key constraints twice to beat the mid-conversation blind spotWhat not to giveThe whole repo, full error logs, last topic's history, large sample dataWhen to clear and restartIt starts breaking agreed rules, repeating itself, or failing three rounds straight—restartThe self-check: if you're not sure whether something is needed this time, it probably shouldn't be given
08

Context Budget

What to include and what not to—more important than how much.

Building pages with AI: three steps, in this orderStatic page firstConfirm the look: the screen's static structure, all content as placeholdersDon't let it wire data and states along the wayThen wire dataConfirm the shape: swap real samples in, check length, nulls, and formatsFinally cover edgesConfirm the fallbacks: loading, failure, empty, and oversized contentOnly when every step is cheap can you dare to change it
09

Prototype to Code

Static page first to confirm the look, then wire real data, then cover the edge cases.

Rules you repeat belong in a file, but that file has to stay thinWhat to distillRules that cause rework, site-wide design conventions, and explicit non-goalsHow to writeImperatives, sectioned headings, a why on the critical linesDescriptive sentences don't change behaviorHow to keep it thinThe model follows rules probabilistically; longer means weaker. Stay under 200 linesOne level upProcesses you rarely need become callable skill filesAn outdated rule is worse than no rule
10

Distill the Rules

Rules you re-state constantly go into CLAUDE.md and Skills.

2原则判断标准
3方法可以照着做的步骤

Validate & Iterate

Check before launch, read data after launch.

1认识验证在验什么
If the layer below doesn't hold, the conclusion above it is meaninglessUsedDo they come back and would they pay — tracking, retention curves, conversion funnelsUsableCan a stranger get through on their own — five-second test, three to five people trying it while you watchWorksIs the function correct, do abnormal cases crash it — walkthrough checklist, real-data stress, security checksSkip the first layer and look straight at the data, and you get numbers you can't explain
01

Usable, Useful, Used

Three layers of validation, each checking something different.

One number at the top for direction, broken down to the layer a single change can moveTop-level · DirectionMatches the product goal, shifts with the lifecycle, pick a number you can moveSecond level · LocateMaps to the key steps of the main flow, used to find where a problem sitsThird level · ActMovable by one concrete change — stop breaking down at this layerOutcome metrics are for review; process metrics are the handles
02

How to Build a Metrics System

Break down from the North Star to the layer a single feature can move.

Five events per feature answer most of your questionsExposureHow many people saw the entry — it's the denominatorStartHow many clicked in and beganA big drop from exposure to start points to the entry itselfSuccessHow many made it through — the number you actually wantFailureMust carry a reason: validation failed, API errored, balance too lowDrop-offWhere the people who quit midway stoppedThese five events chain into a funnel; fix whichever stage drops the hardest
03

Track Before Launch

A feature shipped without tracking is a feature that didn't ship.

A number can be the North Star only if it meets all fourNorth StarRepresents value the user actually receivedRepresents valueNot "showed up" — "received". Completing one real task is what countsMovableDo one thing this quarter and see it change within two weeksLeads revenueTells you in advance where revenue is heading, instead of following itCannot be gamedExplainable in one sentence; cannot be inflated by splitting pages or changing definitionsThe more metrics you watch, the easier it is to find one going up and convince yourself everything is fine
04

One North Star

Pick one metric that means the product is getting better; the rest are reference.

Only real behavior records settle whether a product worksSmall samples misleadThey hand you the comfortable answer in the direction you were hoping forTwo of three clicked, and you read "sixty percent"Check the four criteriaBehavior over claims, paths over single points, enough sample size, and real scenariosWhen you have no dataInstrument before launch, ship a small version to real users, never use surveys as behaviorInsiders are people who already know they'll use it
05

Judge by Real Data

Small samples and your own team's feelings don't count.

The first screen's communication is decided in five secondsFind someone unfamiliarColleagues and registered users are out—they already 'know what this is'Strictly five secondsShow it and take it away; any longer and it becomes a reading testFive seconds only covers the first screenAsk three open questionsWhat is this, what do I get, what do I do. No options, no hintsThree to five people is enoughA qualitative check to surface problems, not quantitative evidenceIf they can't answer, the first screen isn't communicating
06

The Five-Second Test

Five seconds with a stranger—can they say what this is?

Fixed order: Function → Experience → Data → Security, with security as a single vetoFunctionMain flow completes · all four states · retry after failureExperienceWalk through as a brand-new account · narrow screen · readable messagesDataKey events reported · failures carry a reason · dashboard readyIf an earlier one fails, the later ones do not matterSecurityNo secrets in the frontend · privilege escalation fails · input escapedSingle-veto gateUnder twenty items, walkable in fifteen minutes — only then does it become a habit
07

Launch Checklist

Function, experience, data, security — check item by item before launch.

You develop with "just right" samples—real data is not like thatOversized contentLong titles, comments, nicknames—does the layout break, do cards deformReal users include such people; don't bet they don'tZero rowsJust launched, new account—a designed empty state, or leftover blanks and an errorMalformed dataHTML, emoji, very long numbers, missing fields—rendering errors, API crashesThis isn't performance testing; it's data testing
08

Stress-Test with Real Data

Run it once with absurdly long content and once with zero rows.

AI writes code fast and often—the security check has to keep up, three minutesThree high-frequency trapsHardcoded secrets, broken access control, injection. Know them before you checkThe three-minute checklistWhoever wrote the code, run this checklist before launchDon't just delete the secret—the deleted copy already leakedWhy AI is especially proneIts "normal" happens to be insecure, for three reasons tied to how it worksThe security line was always a human's to hold—faster tools don't mean higher quality
09

Security Check for AI Code

Hardcoded secrets, broken access control, injection—the traps AI leaves most often.

A metric that everyone is responsible for is a metric no one isNorth StarServes as referee, not as taskComeNew activations — add an entry point, adjust a channelUseCore task completion — change onboarding, cut stepsStayWeek-1 retention — improve re-engagement, build habitsReturnActivity after returning — personalized content, recommendationsEvery path gets a metric, a set of actions, and an owner
10

Splitting the North Star

Split one top-line metric into several paths you can each act on.

Code is validated by output, knowledge work by processValidating codeRun tests, check results, see if it worksThe output is the evidenceTests pass, it passesValidating knowledge workExamine process, inputs, citations, reasoningThe output alone proves nothingA 90% number says nothing about the reasoningMake AI show its proof of work, and you can actually trust its answer
11

How to Validate Knowledge Work

Code is validated by output; knowledge work by process. Different validation means products must be designed for visible process.

2原则判断标准
3方法可以照着做的步骤

Forward Deployed Engineering

Rethinking FDE

Define the role: what it is, what it is not, the capabilities it needs, and where it sits in a team.

Models are abundant; turning them into outcomes is scarceWhat models doA polished demoWorks out of the boxFDETranslate + deliverWhat customers needData, access, workflowsA result that runs FDE turns an ambiguous need into a system a customer team can keep using
01

What Is a Forward Deployed Engineer

The role that closes the gap between model capability and customer outcomes.

The finish line, pricing, and deliverable define the roleSalesBefore signingWin the dealA convincing deckStaffingBy the hourPeople leave; system stopsDelivered laborConsultingAdviceNo operating ownershipA project reportFDEAfter signingMake it runCapability remainsFDE owns the outcome and feeds what it learns in the field back into the product
02

FDE vs. Sales, Consulting, and Implementation

Use the finish line, pricing, and deliverable to separate neighboring roles.

FDE is broader, better at translation, and more willing to own the resultOwnership + edgeOwn the result; challenge bad assumptionsWithout it, work gets done but nobody owns itBusiness translationTurn technical work into customer valueWithout it, features ship but value stays vagueTechnical breadthCode, APIs, data, cloud, and enterprise systemsWithout it, the field problem cannot be solvedThe three layers together make an FDE; coding alone does not reach the customer outcome
03

Three Capabilities, Not One

Technical breadth, business translation, and ownership are a single operating system.

Field information converges in one FDE and flows back into four worldsFDEDiagnose · build · feed backCustomerEmbed in the workProductTurn patterns into capabilitySalesUse results to earn trustOrganizationBuild independent teamsThe connector is not a messenger; it turns field problems into running, reusable results
04

One Person, Four Battlefields

The FDE connects customer, product, sales, and organization.

Choose the Right Battlefield

Screen the problem, go to the field, and narrow the validation to a valuable slice within institutional constraints.

Validate problem-solution fit before deciding to buildSpecific painVisible in the fieldA broad direction cannot be acceptedEconomicsLoss and gain can be countedPainful does not always mean valuableFeasibilityData, quality, and timeline are knownNo data means no deploymentThe three gates help an FDE reject an endless project before it starts
05

PSF: Ask If It Is Worth Doing

Screen a deployment opportunity through specific pain, economics, and feasibility.

The real problem lives in the workflow, not in the translated requirementObserveFollow the customer’s real workWatch actions before explanationsGo around translationsReach the pain and data sourceEvery retelling loses informationWork togetherDebug side by sidePeople reveal more while doingShadowing produces a deliverable problem definition, not a transcript
06

Go to the Field, Not the Meeting Room

Pain appears in the workflow; shadowing is the FDE’s field method.

MVD keeps the depth and narrows the slice to where value can appearReal dataUse data the customer actually works withA sample working is not a field resultDeep narrow sliceOne workflow, end to endDo not make MVD a crippled planFixed deadlineVerify in weeks, not half a yearLong timelines invite scope growthThe judge is continued use in the real workflow, not a convincing demo
07

Minimum Viable Deployment

Verify real value with real data, a narrow slice, and a fixed deadline.

Institutional constraints require FDEs to understand business, compliance, and infrastructure togetherEnterprise / governmentInstitutions are the main buyersThe decision chain is not one personState-owned engineEnergy, finance, and manufacturing leadFind both business owners and procurementPrivate and trustedData, compute, and networks have boundariesAdaptation is part of deliveryCore productivityMove from pilot to operationsValue must enter the business ledgerThe China FDE basic skill is making a result run inside institutional constraints
08

China’s Battlefield: Enterprise and Private Deployment

Understand institutional buyers, trusted infrastructure, and the path from pilot to core operations.

Earn Trust

Choose a lighthouse customer first, then earn durable trust through real results, action, and honesty.

The first customer determines whether the playbook can be copiedReal painSpecific and consequentialWorth investing inNot generic AI interestInternal ownerBusiness leader with influenceSomeone pushes itNot only procurement or ITReal dataWilling to validate togetherCan move quicklyNot a free demo huntA lighthouse illuminates the path; a demand hopper consumes FDE capacity
09

How to Choose a Lighthouse Customer

Choose a reference customer whose delivery can become evidence and a reusable path.

Enterprise trust must be tested in the real environmentGood promisesPlans, commitments, demosThe customer still doubtsFDETurn words into resultsCredible workReal data, real environmentThe customer keeps using it Action creates trust, honesty protects it, and content amplifies it
10

The Trust Dividend

Action, honesty, and useful content turn one delivery into durable trust.

Make the System Stick

Launch is not activation. Use hot fixes, read-old-write-new, and change management to reach real usage.

Launch is an event, activation is the result, and delivery must cover real usageLaunchThe system enters realityDeployment does not mean useHot fixesImprove while people use itSilent failure still breaks trustActivationThe team changes its workflowValidate with behavior and evalsThe milestone is when the customer starts depending on the system, not when the system is deployed
11

Launch Does Not Mean Activation

Use hot fixes and evaluation to move from deployment to real adoption.

Accept the old world first, then put new value behind a controllable boundaryLegacy systemMainframes, shared drives, private APIsIt will not move for youFDERead old · write newNew workflowInterfaces, evals, gradual replacementValue starts running Be compatible in data, independent in architecture, and respectful of user habits
12

Read Old, Write New

Stay compatible with legacy data while keeping the new architecture independent.

Adoption is not training; it is getting key people to stand behind the systemFind alliesBusiness, leadership, and frontline usersDo not work only with procurement or ITWin advocatesLet key people use the resultSeeing beats being toldPilot to scaleUse a local win to move the organizationMake it their systemWhen the customer promotes the system, FDE becomes a catalyst for organizational capability
13

Make the Organization Your Ally

Turn the customer organization into an adoption network rather than a source of resistance.

Grow the Relationship

Use health scores as an early warning, charge for outcomes, and expand from one department to the network.

Renewal is seen in daily usage signals, not discovered at the deadlineHealth scoreActivity, usage, value, spreadWatch each component, not only the totalEarly interventionAct before the signal turns redDo not wait for renewal seasonOrganizational spreadCapability enters roles and workflowsDo not depend on one contactA healthy customer does not depend on one person; the organization has made the result part of daily work
14

The Renewal Early-Warning System

Turn renewal from reactive firefighting into proactive intervention.

Once pricing is tied to outcomes, delivery must define what success meansBind to outcomePay for the result receivedBoth sides alignUnused features lose priorityMeasure earlyLock acceptance metrics before workValue is testableBuild the exam firstDeepen the accountExpand from one team to a networkValue creates a handleUse data to open doorsOutcome pricing is powerful and honest: delivery must answer whether the customer succeeded
15

Charge for Outcomes

Tie the commercial model to the result the customer actually receives.

Expansion turns a validated playbook into organizational capabilityExpansion cadenceOpen a new field after health is provenDo not expand before value is deepCopy boundaryStart with similar departmentsRepair failed assumptions before reuseDepartment to networkMove common capability into the platformDo not repeat every integrationThe evidence of expansion is that department N starts faster than department N-1
16

From One Department to a Network

Use a deliberate rhythm to deepen an existing account and expand the proven playbook.

Turn the Playbook into a Product

Capture each success in a playbook, then turn it into reusable platform capability.

Turn one success into capability that can be called againCapture the fieldRecord pitfalls, moves, and decisionsWrite it down while the work is freshReuse firstCheck the playbook before redesigningReview successful projects tooCreate the loopDeliver, capture, share, winMake experience a content assetHand off by templateLet a new teammate run the checklistReduce handoff and trial costA playbook is organizational memory that improves delivery quality with every account
17

The Delivery Playbook

A reusable operating system that makes the next deployment faster.

If the same problem repeats, stop solving it by handField problemRepeated gaps, connectors, workflowsEvery delivery gets heavierFDEFind patterns · feed productPlatform capabilityTemplates, connectors, evalsThe next customer starts faster More delivery makes the product stronger; a stronger product makes delivery lighter
18

Turn Delivery into Product

Bring repeated field learning back as standard product capability.

Different companies and markets keep rediscovering the same frontline delivery logicPalantir’s originBattlefield to bootcampReal data, real resultField work feeds platformChinese enterpriseCore work, not demosValue in the ledgerPrivate, domestic stackFailure logsRMB 2M, zero go-livesSpend without adoptionProcess and data fail firstIndustry definitionOpenAI, Ramp, LayerXProduct into the fieldNot a salesperson codingThe formula: connect real data, pick a narrow workflow, build the evaluation, drive adoption, feed it back
19

Frontline Case Studies

Palantir’s origin, Chinese samples, failure logs, and how the industry defines the role.

Under one name, four pieces differ by an order of magnitude in difficultyCopy: field resultsPrototype in days with real customer dataProvable within weeksNo platform foundation needed to startCan't copy: the platformConnectors, permissions, eval definitions become a foundationYears of engineering investmentWithout it, custom deployments spiral out of controlCopy wrong: rename titlesRebrand pre-sales and customer success as FDESame cost, pricier storyThe metrics never changedWhat can't be copied isn't the posture — it's the platform and discipline underneath it
20

When Not to Copy Palantir

Copying the title is cheapest and worst; people on site without a platform is outsourcing at double the price.

Decide the organizational shape first, then hireFDE teamDecide all four at once; miss one and it degradesReporting lineTo product or delivery leadership, not salesStaffing unitPairs: engineer + mission ownerMetricsReusable assets and activation, no sales quotasLoad ceilingHow many active customers one person can carryWhat gets replicated isn't heroes — it's these four constraints
21

How to Staff an FDE Team

Reporting line, staffing, metrics, load ceiling — decide the shape first, then hire.

Agent Engineering

Pick the Right Complexity

Start at the lowest complexity that works: when you do not need an agent at all, and which rung to start from when you do.

Pick the lowest rung that does the jobPlain codeStop when branches are expressibleModel callStop if it only judges or extractsTool loopMove up when next step is unknownWorkflowMove up for state machines and gatesMulti-agentMove up for isolation or real parallelismEvery step up must answer: why is the lower rung not enough
01

Do You Need an Agent at All

A model in the loop is not an agent, and many steps is not many agents. Pick the lowest rung.

Six domains decide if the model can shipHarnessControl layer between model and environmentIdentityRole, permissions, forbidden actionsOrchestrationPaths, dependencies, human nodesGateEntry checks and output acceptanceRecoveryState, retry, and rollbackKeep components replaceable and ablatable
02

The Harness Around the Model

The model judges. The harness turns judgment into something you can ship and recall.

Classify the layer before you buildL0 publicAlready in the model — write none of itL1 platformWait for the platform; mark shims for removalL2 org processFacts no model can invent — your leverageL3 judgmentResponsibility stays with humans, enforced by machinesRules that patch model gaps expire on upgrade
03

What Belongs in Your Harness

More rules is not more safety. Write only what the model cannot invent and you cannot delegate.

Verification capacity caps autonomyModel says donePolite, unverifiableLooks finishedMechanical criteriaTests green, artifact exists, exit 0Trusted deliveryReviewable and rollback-ableSafe to widen autonomy When verification is short, expand it before granting autonomy
04

Autonomy Ends Where Verification Ends

Autonomy is capped by how cheaply and reliably you can check the result.

Context Engineering

Context is the agent's only working memory: what goes in, how it is layered, and how to stop it from degrading as a run grows.

Every source needs a trust level and a pruning ruleSystem and safety rulesSYSTEM: highest rank, never overriddenOrg and project rulesORGANIZATION: admins only, versionedUser input and historyUSER: shapes tasks, not permissionsRetrieved and tool outputEXTERNAL: needs source and timestampIntermediate resultsAPPLICATION: drop if recomputableContext is a budget, not a transcript
05

What Actually Goes Into Context

Twelve sources, six trust tiers, and why a bigger window is not more capacity.

Noise accumulates with each step, models do not degradeBloatRaw output enters context in fullHistory eats the budget earlyDilutionUseful instructions fall to 10%Argument error rates climbCorruptionCorrected errors stay in contextRetries push bloat back upAnxietyModel wraps up and declares doneWork ends before it is finishedMeasure context noise before changing models
06

How Context Gets Dirty

Dilution, corruption, and anxiety: three decay modes, and why compaction cannot fix the third.

Model reads intent, system moves bytesExternalizePast 8000 chars, store as artifactOtherwise only 3-5 items surviveReferenceKeep refId plus a bounded summarySummaries must trace to sourcePartial readOutline and search, not full textReading it all back defeats itBindRuntime injects the argumentModels truncate and invent UUIDsExact values travel through the system, not the model
07

Stop Making the Model Carry Data

Models truncate, mix up, and invent IDs. Let the system be the pipe and the model be the judge.

More resident rules is not more safetyCoreEvery run: objective and stop conditionsScoped rulesBy project: directory and stackPhase contextOn entry: contract and checklistOn-demand referenceOn trigger: specs and examplesArtifact dataPartial reads: results and logsGive the model a map, not a thousand-page manual
08

Layered Loading and Handoffs

Five layers loaded on demand, structured handoffs, and why a stable prefix keeps your cache alive.

Tools and Environment

Tools are contracts and the environment is the capability boundary. Together they decide how much real feedback the agent can get.

Cut tools by task, not by endpointAPI wrapperForty endpoints, forty toolsHard to pick, hard to changeTask-shaped toolCut and named around the taskPicks right, fills rightDeclared contractRisk, timeout, idempotency, trimmingFails safely, recoversDeclare everything the model cannot infer
09

A Tool Is a Contract, Not an API Wrapper

Descriptions drive selection accuracy, output trimming drives context cost, error contracts drive recovery.

Feedback sets the ceiling, not promptingInternal environmentOpaque, slow to verifyQuality slips silentlyVerifiable feedbackCompilers, tests, snapshotsAgent capabilityVerifiable problems fall firstHard work becomes shippable Fix the environment, then the prompt
10

Environment Feedback Sets the Ceiling

The environment is the invisible quality killer: it never crashes, it just quietly gets worse.

The model judges; the program loopsDeclare dependenciesMap the graph, find independent workFetch in parallelLoad shared context exactly onceRun the batchLoops, filters, and retries in codeReport onceEvery run lands in a replayable traceCollapse homogeneous batches into one round trip
11

Put Loops and Batches in Code

Round trips issued one at a time scale latency, tokens, and drift linearly with N.

Choose by whether you need discovery or executionMCPDynamic discovery across hostsCost: schemas stay residentTyped service toolsStable operations and side effectsCost: a written contractCLI and scriptsBuilds, tests, migrations, checksCost: versioned upkeepMCP for discovery, never for reliable execution
12

Where MCP and Skills Belong

MCP costs are resident, skill risk is supply-chain grade, and both need per-agent allowlists.

State and Long Tasks

A good run is not a delivery: state layering, interrupt recovery, side-effect receipts, and the bar for going multi-agent.

Different lifecycles cannot share one arrayRun stateCurrent step, budget, interruptSession historyMessage continuity and summariesWorkflow checkpointResume, replay, human pauseLong-term memoryPreferences and verified factsDecide recovery semantics first, storage second
13

Four Kinds of State, Four Lifecycles

One array cannot hold four lifecycles. Mixing them costs you both recovery and deletion.

Define done first, then design recoveryWrite the goal contractOutcome, constraints, verificationAdvance in stepsOne small, named task at a timeLeave artifactsPlan, implement, documentationReach an end stateBlocked or out of budget, stopArtifacts decide whether work survives a session
14

Long Tasks Need Handoffs and End States

The context window will run out. What carries long work across sessions is artifacts on disk.

Side effects need a key, a receipt, and phasesBefore dispatchRecord intent, mint the keyAfter dispatchMark it sent, await confirmationResult confirmedWrite back the receiptReconcileCheck first, never replay blindWhen the result is unknown, reconcile instead of retrying
15

An Answer Is Not Accountability

A wrong answer costs a retry. A wrong action has already sent the mail and charged the card.

Multi-agent is a cost, not a defaultSingle agentOne context, one ownerCost: no answer when context overflowsStart here by defaultWorker poolParallel similar work, no coordinationCost: with no hierarchy, only small fixesUse when work is independentCoordinator plus specialistsPlan, build, and verify have ownersCost: extra tokens and latencyWorth it when done differsSplit only where engineering responsibility differs
16

When Multi-Agent Actually Pays

Every extra agent adds tokens, latency, failure surface, and ambiguity about who owns the result.

Evals and Verification

Define done before you build: eval sets, four metric layers, the online and offline tracks, and keeping verification ahead of generation.

Optimizing without a defined decision is optimizing nothingName the decisionWhich decision this eval supportsTier the metricsOutput, safety, ops guardrailsPick gradersDeterministic checks for hard metricsOne variableChange one thing at a timePromote to regressionGraduated capability evalsAnswer the decision, then choose metrics and graders
17

Define Done Before You Write Code

Tune prompts without a decision behind them and you just get lost faster.

Two agents can both be right and differ completelyOutcomeTask finished, output usableLucky hits score as successProcessSound plan, stable stepsBreaks on the next batchEfficiencyTime, tokens, tool callsCostly and cheap score alikeRiskOverreach, bad side effectsRare severe events vanishFour layers separate reproducibility from luck
18

Four Layers of Evaluation

Getting it right is not the same as being right. Four layers make the path visible.

Online alarms, offline provesOnline evalSampled production traces, a benchmark not ground truthSignals whether something droppedOffline evalCurated dataset with ground truthProves the change really helpedOnly oneOnline cannot verify fixes; offline misses driftOne guesses, one goes blindA flywheel turns production traces into the next test set
19

Online and Offline Evals

Treating production scores as acceptance is the mistake agent teams make most often.

Three options when output outruns verificationStatic and typesSecond-level: syntax, types, lintMisses runtime behaviorUnit testsSecond-level: function behaviorLarge happy-path blind spotIntegrationMinute-level: contracts, real depsFlaky environments misfireEnd-to-endMinute-level: browser, full pathSlow, costly to locate failuresHuman reviewHour-level: only what machines cannot judgeLowest throughput, the bottleneckNever by default: expand, slow down, or lower the bar
20

Verification Must Keep Up With Generation

Nobody declared what was relaxed. Review just turned from reading into skimming.

Safety, Cost, and Shipping

What matters once it runs: containment and credential boundaries, cost attribution, launch metrics, and feeding failures back into the system.

Model layers cut probability; the environment is the floorSandbox boundaryLimit filesystem and egressLeast-privilege credsScoped per tool and resourceData boundaryExternal content is never instructionTiered approvalGate only high-risk actionsAudit and identityAI-BOM plus verifiable delegationSandbox decides reach; approval decides timing
21

Sandboxes, Credentials, and Human Boundaries

What the model layer cannot stop, the filesystem, the network, and credential scoping have to.

Baseline first, optimize second, guard with qualityBudgetResolved at run start, soft and hardTiered routingPick models by capability and riskCache hitStable prefix plus a deterministic frontAttributionSplit by run, model, context sourceCost per successTotal spend over successful tasksGuard every saving with quality and safety
22

Attribute Cost Down to Every Step

Cost you cannot split only shows direction. Cost you can split tells you what to change.

A passing demo is not a launch caseDemo metricsOne success, fluent reference answerRetries and takeovers stay hiddenLaunch metricsSuccess rate, all-pass@k, quality floorVariance and the long tail show upCost per successTotal spend over successful tasksSavings must clear a quality barProve stability before you scale
23

What to Measure Before You Ship

A demo proves the capability exists. Launch means proving it holds on the real distribution.

Change the structure, do not add a warning lineIncidentRecord scene, expected, actualRoot causeFix structure, not the wordingCandidate ruleMark CANDIDATE, scope itRegressionOffline eval plus canaryMechanical enforcementWire the gate into CIFailures you do not compound get charged again
24

Turn Failures Into Engineering Assets

Editing a prompt is not a fix. A gate, a validator, or a regression test is.

Runtime Discipline

The discipline after launch: classify failures before counting them, tier recovery, consume events at safe points, and treat launch as a process, not a switch.

Classify, then count, then think about circuit breakersRetry on any errorRate limits and bad parameters treated alikeNon-retryable errors exhaust every attemptLayered classificationAPI, tool, context, control flow, one mapping eachOnly the retryable half gets retriedLiveness monitoringA watchdog on streams that are connected but not flowingSilent hangs and broken pairings surfaceThe error-to-policy table is the runtime's first asset
25

Classify Failures Before You Count Them

Blind retrying counts errors that no retry can fix. Classify first; only then do metrics and circuit breakers mean anything.

Escalate in tiers, keep every path boundedSilent retryExponential backoff with jitter, foreground and background apartDegrade and resumeRaise the cap, continue from the cut, switch to backup modelExpose to the userShow which recovery actions were already triedGlobal backstopIteration ceiling, session budget, escalate after repeated failureThe error-handling boundary is the whole recovery loop, not one request
26

Tier Your Recovery, Then Circuit-Break

Three escalating tiers, each more transparent than the last; breaker ceilings read off production data.

The three strategies differ only in how they treat safe pointsEvent loopTake events, reason, run tools, return to the boundarySafe pointA stretch of reasoning ends, a tool returnsCancellationManufacture a safe point for urgent eventsQueuingWait for the natural boundary, consume in batchParallelLightweight independent query in its own session, appended back with a markerKeep the trajectory synchronous; interrupting is a necessary compromise
27

Consume Events Only at Safe Points

Never sever mid-reasoning or mid-tool: events queue to the loop boundary, and cancellation just creates one early.

A release is a sequence of gates, not a momentSwitch-style launchFull traffic the instant the deploy finishesFailure surface equals every userGate-style launchOffline tests, shadow, canary, then full rolloutEvery gate keeps an exitVersion pinningPrompt, model and tool endpoint recorded per turnReproducible failures can be attributedThe gate sequence and the rollback hook must exist before launch
28

Launch Is Not a Switch

Teams that treat deployment as a moment hand users surprises as features — a release is a series of gates with exits.

Continuous Evolution

Getting better with age, not messier: first-deviation attribution, tending the eval set, turning experience into learning, and braking the evolution loop.

The primary cause is the earliest error that explains the restReplay the traceMark the step and evidence where behavior first became unacceptableLocate the deviationIt may raise no error and may not even be a tool callAssign the layerA missing observation channel, or a model that cannot do itPrescribe the fixRepair the harness, change the workflow, or add dataTurn it into a regressionThe same record feeds the replay caseThe attribution record is also the work order
29

Attribute Failure to the First Deviation

End-to-end scores only say it failed; attribution says from which step — and only the first deviating error earns that role.

An eval set iterates like a productStatic eval setBuilt once, then only consumedScores climb, reality driftsTendingLayering, leak resistance, failure feedbackLiving assetKept close to the real task distributionChange decisions have evidence Today's production failure is tomorrow's regression case
30

Tend the Eval Set Like a Product

Five redesigns in tau-bench, 300-plus issues in OSWorld: every trustworthy benchmark was tended into being.

Separate recording from sortingRaw tracesImmutable, for audit and re-analysisSingle-run analysisThis run's outcome, strategy and draft lessonCross-trace inductionCluster and contrast into knowledge with an evidence tableTransfer validationMeasured on new tasks that took no part in the distillationReflection by itself is not evidence
31

Saving Experience Is Not Learning From It

Store the evidence, induce offline, then prove transfer on new tasks — drop one of the three and it is only archiving.

Beyond proving it is stronger, limit who may change whatEvidence apart from instructionsUntrusted evidence never becomes a long-lived capabilityCandidates apart from productionNew capability first lives where it cannot serve real trafficSafety mechanisms closed to self-editVerifiers, gates, audit log and stable release are outside proposal rightsThe guarantee comes from a layer the modifier cannot reach
32

An Evolution Loop Needs Three Brakes

Prompt injection must not travel through experience, and regression must not travel through the verifier — three hard boundaries around automatic evolution.