Whatever you wrote down as the finish line is what your team, and your agents, are actually building toward.
The fourth or fifth time I watched somebody pick up the same card twice, I finally understood what we were doing.
My wife and I designed a card game. Sixty eight cards, six rounds, two to six players, and not one line of code in it. No analytics, no event stream, nothing that could tell us anything after the fact. Just a table, some people, and the question every product team treats as unanswerable: is this any good?
We could not measure fun. What we could do was watch. A card read twice means the wording failed. A table going quiet in the middle of a round means the round is too long. A player who checks out the moment their set gets blocked means the scoring is punishing them for something they cannot control. None of those is fun. Every one of them is visible from across a table, and each one points at exactly one card to fix.
More than a hundred people played it before we called it finished, and the game got fixed off those signals rather than off anybody's opinion of it.
We were not measuring the game. We were building something a person could check.
I have spent the months since finding the same move everywhere, including in products that do have instrumentation and are mostly worse off for it.
The work is not unmeasurable
Every product organization keeps a pile of work it has quietly classified as unmeasurable. Quality. Taste. Whether the design is any good. Whether the decision was sound. Whether the customer actually trusts us. We say these things need human judgment, and we say it as though that settles the matter.
It settles nothing. It moves the work somewhere nobody is accountable for it.
Here is the reframe I would offer. Almost none of that work is genuinely unmeasurable. It is unrepresented. It exists in people's heads as an impression, and impressions cannot be counted, compared across weeks, handed to a new hire, or given to a machine. Change how the thing is written down and a surprising amount of it becomes checkable without becoming any less human.
Here is the version of that at work, on the hardest thing we are building. We have a voice agent that calls hotels during a disruption, checks availability and reports back. Our CEO calls it the company-making feature and he may well be right. It is also genuinely difficult in a way that is easy to underestimate until you sit and listen to the calls. A hotel line answers with an automated menu, or a voicemail greeting, or a person who picks up halfway through a sentence, and the agent has to work out which of those just happened before it opens its mouth.
So I put a number in front of it. It does not enter the development pipeline until half of its calls run cleanly end to end. When I set that, roughly one call in ten did.
Nothing about fifty percent is scientific and I would not defend the exact figure. What the number does is give everyone the same picture of a problem that is nearly impossible to see from a demo. A recording of one good call tells you the thing works. A success rate tells you what a customer would actually get. Nobody had to change their mind about the bet, and nobody has. We stopped estimating readiness and started counting it, which is a much easier thing to be patient about.
A card table and a voice agent, about as far apart as two products get. Neither one was made more mechanical. Both were made legible, which is a different move, and it is available almost everywhere.
Then the optimization starts
Now the trap, and it arrives the moment you succeed.
Whatever you make checkable becomes the thing everyone optimizes, including the people you trust, including you, and very much including any agent you point at the problem. A check is not a neutral observer. It is a definition of progress, and everything in the system will move toward it, whether or not it is moving toward what you meant.
I did this to myself this year on garmin-local-mcp, an open source tool I built and maintain. It has a demo mode so people can try it without connecting their own account, and the demo generates a realistic looking health dataset. My target was, roughly, make the demo good. The first version was excellent. Two of the metrics came out correlated at 0.86, clean and obvious, a perfect little story. My own real data, from my actual body, sits closer to 0.68. Messier.
So I spent an evening making the demo worse on purpose until it matched. Under my original target, every step in the wrong direction would have felt like progress, and I would have kept polishing until the demo was beautiful and dishonest. A demo that looks better than the real thing tells people you have never looked at the real thing.
The same failure has an expensive enterprise version, and I heard it described precisely at the CPO Summit this June. Arvita Tripathi, a chief product officer who has shipped in regulated markets for eighteen years, argued that enterprise AI pilots die as product design failures rather than sales failures, and named the mechanism: the buyer usually cannot articulate what would kill the pilot, so nobody writes the criteria down, and the organization's default criteria win by absence. Her line was that evaluation design is a procurement question and that it will cost you, so you had better make it a product question. Her fix was to define kill criteria up front in three classes, evidence, adoption and governance, each with a date attached.
Read that alongside the demo story and it is one lesson at two scales. The pilot did not fail because the product was bad. It failed because the only criteria in the room were the ones that happened to be written down, and product was not there when they were written.
The check you choose is the strategy you get. Not the strategy on the slide. The one on the dashboard.
What makes a check hold
If a check is going to carry that much weight, it needs to survive contact with people who are trying to satisfy it. Three properties do most of that work, and I have watched all three fail.
It has to be expensive to fake. The three signals at the card table share one property, and it is the reason they worked: not one of them is an opinion. A playtester who wants to be encouraging will tell you they enjoyed it, and they will mean it. They cannot decide to stop rereading a card whose wording did not land. Ask people whether the game is good and you get generosity. Watch what their hands do and you get the truth, because it was never theirs to give you.
It has to come from outside the work. This is the structural claim I made in Everyone Drafts. Dominant Skills Finish. about who should audit an artifact, and it applies with more force to a machine than to a person. Nothing grades its own output honestly for long. The auditor has to be whoever breaks if the artifact is wrong.
It has to be checking the thing that is actually running. I lost real time this year to a bug in code that was correct. The code was right and the tests passed. The connector was launching the published version of my package rather than my local checkout, so none of my edits ever reached the running server. What made it genuinely hard to see is that one of the tools read the new data perfectly well while two others insisted the field did not exist, so the thing looked half alive. A check pointed at the wrong artifact is indistinguishable, from the inside, from no check at all. It is worse, because it produces confidence.
You cannot import someone else's checks
Here is the part I did not expect, and it is the reason this is a leadership problem rather than a tooling one.
The checks that work are discovered, not designed. Nobody handed us the three signals we ended up trusting at that card table. We noticed the repeated card by accident, well into the playtesting, and only then worked out what it was telling us. Every good check my team runs at work arrived the same way, out of a specific failure in our own system, and each one encodes assumptions about that system nobody wrote down and nobody could have. The two-value column matters because of how our issues arrive. The zero-charge rule matters because of what a missing charge means in our particular flow of money. Neither of them would be the first thing I would think to build somewhere else.
This is why the published playbooks disappoint. Somebody posts the workflow that transformed their team, you run it against your work, and it produces nothing. Not because they lied. Because their loop was tuned to their stack, and the tuning is most of the value. The visible artifact travels. The fit does not.
Which has an uncomfortable corollary for anyone selling an AI operating model, and I include my own last issue in that. The seven-step process I described there is not the transferable part. The transferable part is the habit of noticing where your own work goes quiet, and writing that specific silence down in a form somebody or something can count. You have to find your own, and finding them costs a quarter, not an afternoon.
That is also the honest answer to the question every leader is being asked right now about why the AI spend has not shown up in the results. The tools arrived. The checks did not, because the checks were never anybody's deliverable.
What it costs, because it isn't free
Honesty section, as always.
A green check is not the same as a working thing. garmin-local-mcp, the tool from the demo story above, has ninety five tests and they all pass, with continuous integration green on every commit. At the end of July a dependency I had pinned loosely shipped a new major version that removed the interface I was building on, and every fresh install failed to launch. Ninety five passing tests, not one of them wrong, and the product was completely broken for anybody who installed it that week. My tests checked my code. Nothing checked the environment my code would land in. Every check has an edge, and the failure is always waiting just past it.
The first version of a check is usually wrong, and you will defend it anyway. Writing a check is a decision, and decisions get attached to the person who made them. It took me an embarrassingly long time to accept that my demo target was the problem rather than my demo.
And the artifact I never wrote a check for is this one. This newsletter published near weekly through the start of July and then went quiet for five weeks, while the list of things I wanted to write grew past eighty items. I have a definition of done for a reconciliation case, a test suite, a PRD and a pull request. I never wrote one for an issue of this newsletter, so no issue was ever ready. I have been diagnosing a pattern I was standing inside, which I suspect is the only way anyone ever notices one.
Somebody else's cairns
On unfamiliar ground above treeline, you navigate by cairns, small stacks of rock left by people who came through before you. They are genuinely useful and they are also the most confident liars on the mountain. A cairn tells you that somebody went this way. It does not tell you where they were going, what the conditions were, how strong the party was, or whether they got there.
Follow enough of somebody else's cairns and you will be a long way off route while feeling entirely reassured, because at every single step there was a marker confirming you were fine.
The fix is not to ignore them. It is to carry your own bearing and check it against the ground, and to notice the moment the markers and the terrain stop agreeing. Transformation is earned in the foothills, and this is one of the things it is earned as: not the ability to follow a route, but the ability to tell whether the route is still yours.
I argued in Your Roadmap Is Lying to Everyone that a number can be an artifact rather than a fact. This is the same argument aimed one step earlier, at the moment the number gets defined. We spent two years getting very good at producing work and almost no time deciding what would tell us the work was any good. That decision is cheap, it is available this quarter, and nobody is going to hand it to you.
Pick one thing your team currently calls a matter of judgment. Write down the smallest fact that would distinguish the good version from the bad one. Put it somewhere it gets recorded at the moment it happens, by someone who gains nothing from flattering it.
Then watch what your team optimizes, because that is now the strategy.
Below the feature line, as ever.
Originally published on LinkedIn. View the original →