Intel's Nikolay Petrov: Measure What Your SKILL.md Adds

How Nikolay Petrov Measures What a Skill Teaches the Model

Nikolay Petrov of Intel starts from a problem anyone with a folder of agent skills has. You have a working skill and it passes your evals, but it is long, models keep changing, and you want it to use fewer tokens. Which lines can go? Editing line by line and rerunning evals is brute force. His talk describes a cheaper first pass: measure which parts of the skill the model already knows.

Two kinds of skills: Petrov separates skills that bring the model new knowledge from steering or process skills that only make a sequence of actions repeatable. "Both are useful." His method applies to the first kind.

Evals first: The baseline test is a two-arm eval. Define tasks the agent should solve, run them with no skill loaded and again with the skill, and compare. In his example the agent solves three of five tasks without the skill and all five with it. Token use and time are compared the same way. That shows a skill works, but not which parts of it do the work.

Atoms, borrowed from FActScore: He splits a skill into items, around 25 to 30 per skill, each with a question and an expected answer. Some are general propositions and some are contracts with an exact expected action. Each question is asked with and without the skill, and a judge, ideally from a different model family or a panel, scores the answers. The questions are checked so they don't leak the answer. The model can be tested with no tools, to probe what it knows internally, or with tools for a more practical read.

Four outcomes per atom: If the model answers correctly both ways, the skill adds nothing there. If it answers correctly only without the skill, the skill is hurting. If it answers correctly only with the skill, that is the goal: "Teach model something new." If it fails both ways, the topic may matter but the skill isn't written clearly enough to get it across. Harmful atoms are the easiest fix and are rare in popular skills. He found one among roughly 400 atoms in Anthropic's skills, and says smaller, less popular skills tend to have more.

General ideas versus exact details: A lenient check asks whether the model has the general idea. A strict check asks for exact keys, parameters, values and steps. The gap is wide in frontier models. Opus drops from 70% understanding of a skill's content on the lenient check to 22% on the strict one. Smaller models know less than larger ones, and the share varies a lot from skill to skill. His reading: new models generalize well on concepts, and it is still worth giving them exact details and data.

What it costs: Running the analysis over 29 skills cost about $14 in Sonnet and Haiku calls, with Sonnet as the judge.

Novelty is a starting point: "It's basically initial vector for optimization." Once you know which atoms the model already has, remove some or all of them and rerun the evals. If the score holds, the skill is smaller and uses less context. If it drops, you know what to put back. Petrov says the measurements were made with a framework for comparing skills, agent instruction files, MCP servers and tools; the transcript does not give its name clearly enough for us to cite it.

5 Takeaways From Nikolay Petrov on Trimming Agent Skills

• Test each claim, not the whole file - Break a skill into question-and-answer atoms and score each with and without the skill
• Delete what hurts first - Atoms where the model does worse with the skill are the simplest win
• Keep the specifics - Models know the concepts in most skills, but far fewer of the exact parameters, values and steps
• Recheck per model - What a small model needs spelled out, a larger one may already know, and the answer shifts with each release
• Evals have the last word - Novelty scores say where to cut, and only the task evals say whether the cut was safe

What Skill Novelty Scoring Means for Teams That Run on Agent Skills

Skills are how most teams now encode how their company does a piece of work, and they accumulate. Every line loaded into context costs tokens and competes for the model's attention. Petrov's method gives skill owners a measurable question to ask of each line: does the model answer differently because this is here? For a company whose procedures live in skills, that turns skill maintenance from taste into something closer to testing. For background, see Agent Skills and Skill Engineering.