feat(skill): tune judgement from eval feedback and add skill evals

.gitignore

@@ -7,3 +7,4 @@ examples/*/export/
 site/dist/
 site/promo/dist/
 .wrangler/
+skills/*-workspace/

CHANGELOG.md

@@ -8,6 +8,7 @@ Until the first release, changes are recorded under [Unreleased].
 
 ### Added
 
+- Skill evals in `dev/evals/evals.json`: five cases against public repos (a build story, a release story, a thin fix, standard practice asked for as a deep dive, and a small PR inside a larger stack that should be written as the stack), each with checks a grader can verify. The skill-creator workspace they run in (`skills/*-workspace/`) is ignored.
 - The website's build writes `sitemap.xml` (the home page, the examples page, each example and its slides) and a `robots.txt` that names it, when given `--url`, as the deploy is.
 - `.gitattributes`: shell scripts (`*.sh`) check out with LF line endings on every platform.
 - Video chapters: a whole-cut MP4 or WebM from `motion.mjs` carries one chapter per scene, named by its label (scenes in a row with the same label make one), and `<story>-motion.vtt` beside it holds the same chapters. `export.mjs` takes them as `--chapters <file>`. The example videos carry them, and the site's example pages play it in their own player: its scrubber has a segment per chapter (click, drag, arrow keys, Page Up and Page Down by chapter) with the chapter's name on hover, plus play, mute and full screen, and the chapters are listed under it. Without JavaScript the browser's own controls stay. Tests in `dev/test/motion.test.mjs` and `dev/test/site.test.mjs`.

README.md

@@ -85,12 +85,12 @@ Three flavors. The skill picks one from the request and the size of the change,
 | | **Release story** | **Build story** (default) | **Technical deep dive** |
 |---|---|---|---|
 | Reader | Users and prospective users | Developers who know the product | Engineers who want the mechanism |
-| Answers | What's new, why it matters, how to use it | What we built, how it works, what shaped it | How it works inside, the trade-offs, the edge cases |
+| Answers | What's new, why it matters, what changes for you | What we built, how it works, what shaped it | How it works inside, the trade-offs, the edge cases |
 | Visuals | Screenshots, before/after, one simple flow | Diagrams, motion, before/after | Diagrams, terminal runs, charts |
 | Code | A usage example at most | Only where the code is the idea | Where it helps |
 | Length | Up to 1,200 words | Up to 2,500 words | Usually under 3,000 words |
 
-Word counts are ceilings, not targets. A small change gets a **note** of a paragraph or three. A typo fix or a dependency bump gets an honest "not worth a post", and if it is one piece of a bigger idea, the skill names that idea and asks whether to write about it instead.
+Word counts are ceilings, not targets. A small change gets a **note** of a paragraph or three. A typo fix, a dependency bump, or standard practice done the usual way gets an honest "not worth a post", even when asked for a deep dive. If it is clearly one piece of a bigger story, the skill writes that story and says so; if the bigger idea would change the subject, it names it and asks first.
 
 ## Visuals
 

dev/evals/evals.json

@@ -0,0 +1,78 @@
+{
+  "skill_name": "buildnotes",
+  "note": "Run with the skill-creator skill: each case once with the current skill and once with a snapshot or no skill. Outputs and grading go to skills/buildnotes-workspace/ (ignored).",
+  "evals": [
+    {
+      "id": 1,
+      "name": "build-story-hono-error-middleware",
+      "repo": "hono",
+      "prompt": "Write up how we built routable error and not-found middleware in Hono (PR #5541) \u2014 why we went with routing-based onError()/notFound() over the alternatives.",
+      "expected_output": "Build story that shows the before (old error/notFound handling) then the after, names the decision and the alternative that lost, plain headings.",
+      "assertions": [
+        "Post HTML file exists in outputs",
+        "The old handling (before) appears earlier in the post than the new handling (after), and is labelled as the before",
+        "Names the chosen design and at least one alternative from #5229 that lost, with a reason taken from the record",
+        "Every chapter heading is a plain phrase readable on first pass (no imperative riddles or code names)",
+        "Title does not contain the product name 'Hono'",
+        "No clock times or PR-by-PR narration in the body",
+        "Any code block matches code in the repo at the merge commit and is <=15 lines",
+        "Hand-off lists at most three open questions"
+      ]
+    },
+    {
+      "id": 2,
+      "name": "release-story-uv-publish-prompts",
+      "repo": "uv",
+      "prompt": "Short launch-style post for uv users about PR #22593: uv publish no longer prompts for credentials when env vars or keyring can already authenticate.",
+      "expected_output": "Short release story or note: leads with what users no longer have to do, no edge-case inventory, brief.",
+      "assertions": [
+        "Output exists (post HTML or a hand-off proposing a note/rollup)",
+        "Headline says what users get (no unnecessary prompt), not the internal mechanism",
+        "Title does not contain the product name 'uv'",
+        "Body (if a post) is under 600 words",
+        "Does not enumerate edge cases or every credential source beyond what a user needs",
+        "Considers nearby related work (other publish/auth changes) \u2014 either rolls it in or mentions it in the hand-off",
+        "No PR numbers, review quotes or clock times in the body"
+      ]
+    },
+    {
+      "id": 3,
+      "name": "thin-subject-hono-gridgap",
+      "repo": "hono",
+      "prompt": "Write a blog post about the gridGap px fix we made in hono/jsx (PR #5487).",
+      "expected_output": "Declines a full post; says briefly why; proposes a bigger arc from nearby JSX/v5 work or a changelog line; asks before proceeding.",
+      "assertions": [
+        "Does not produce a full multi-section post about the gridGap fix alone",
+        "Hand-off says briefly why the fix alone is not worth a post",
+        "Hand-off names a bigger idea from nearby work, or offers a changelog line / short note",
+        "Hand-off asks the author whether to proceed with the alternative"
+      ]
+    },
+    {
+      "id": 4,
+      "name": "standard-practice-hono-serve-static",
+      "repo": "hono",
+      "prompt": "Technical deep dive on why Hono's serve-static middleware now only serves GET and HEAD requests (PR #5514).",
+      "expected_output": "Recognizes this is standard practice; declines the deep dive or offers a line in a larger release / a rollup, asks before proceeding.",
+      "assertions": [
+        "Does not produce a multi-chapter deep dive about serve-static method handling alone",
+        "Hand-off says this is standard/expected behaviour that readers already understand",
+        "Offers an alternative: a line in a larger release, a rollup of related v5 changes, or a short note",
+        "Hand-off asks the author before writing the alternative"
+      ]
+    },
+    {
+      "id": 5,
+      "name": "widen-scope-uv-safeurl",
+      "repo": "uv",
+      "prompt": "Write a post about uv PR #22619, which implements RemoteSource for SafeUrl.",
+      "expected_output": "Recognizes #22619 is one step in the SafeUrl series merged the same day and writes the series story without asking first, noting the widened scope in one sentence of the hand-off.",
+      "assertions": [
+        "Writes a post (post HTML exists) rather than stopping to ask",
+        "The post's subject is the SafeUrl / URL-safety series, not RemoteSource alone",
+        "Hand-off says in about a sentence that the scope was widened and why",
+        "Hand-off does not ask permission to widen the scope"
+      ]
+    }
+  ]
+}

skills/buildnotes/SKILL.md

@@ -37,8 +37,9 @@ The rest of the request is the brief: the subject, plus anything about flavor, r
 
 Each rule carries the fault that taught it, and links to the reference that holds the detail where there is one.
 
-- **A starting point, not the manual.** Go only as deep as it takes to get the idea across; the docs carry the rest. Show one way to try it and link the docs, unless the author asks for a full guide. *(Fault: a closing chapter touring every setup path, a menu, an agent tool's argument, two CLI flags and a help screen, for a feature you set up once.)*
-- **Earn every detail.** Keep a date, number, quote, tool name, review comment or code excerpt only if the reader would understand the feature less without it. Nothing in the record is owed a place. *(Fault: an adversarial-review quote that added color and no understanding.)*
+- **A starting point, not the manual.** Go only as deep as it takes to get the idea across; the docs carry the rest. Show one way to try it and link the docs, unless the author asks for a full guide. A post says what changed in a way that stays true; the docs hold the scenarios. Listing every case where a behaviour applies (each credential source, each retry, each flag) reads as a support page and goes stale with the next release, so keep it directionally right and link the docs for the cases. A deep dive the author asked for is different: that reader wants the mechanism and its cases, and the post can sit between a launch post and a reference page. *(Fault: a closing chapter touring every setup path, a menu, an agent tool's argument, two CLI flags and a help screen, for a feature you set up once.)*
+- **Earn every detail.** Keep a date, number, quote, tool name, review comment or code excerpt only if the reader would understand the feature less without it. Nothing in the record is owed a place. Outside a deep dive, leave out edge-case behaviour (no JavaScript, reduced motion, keyboard shortcuts) and don't list everything a UI contains; readers assume a familiar control works the familiar way. *(Fault: a launch post for video chapters that explained what happens with motion reduced and walked through every key.)* *(Fault: an adversarial-review quote that added color and no understanding.)*
+- **Before, then after.** When the change replaces something, show the old way first and the new way next, labelled, ideally side by side near the top, so the reader is anchored before the explanation starts. Give a sentence of what came before; don't start cold, and don't recount the whole history. *(Fault: the new figure in chapter one and the old one two chapters later, so a reader scrolling past saw the change in reverse.)*
 - **Lead with what people can do.** The headline is the capability users see, not the mechanism behind it. The mechanism leads only in a deep dive, when the author asks, or when it *is* what users touch. [brief.md](references/brief.md#the-six-fields) *(Fault: a post about one-click preview links whose centerpiece was how the scheduler pools build machines.)*
 - **Don't name yourself in the title.** The title says plainly what shipped; the angle and the play go in the dek and the opening. The title and chapter headings are about what the reader gets; the product and company name stay out of them, and the prose names the product once where a newcomer needs it. In a feed where every title names its own company, the name stops meaning anything. [craft.md](references/craft.md) *(Fault: "Tell Releases what you care about".)*
 - **Ideas, not chronology.** Chapters are concepts, challenges or parts of the system, never days or PRs. Mention timing only when it is the point, and then in days. Iteration labels ("round one", "as it ships") are fine; clock times and commit-by-commit narration are not. [story.md](references/story.md#altitude) *(Fault: "At 14:56 UTC the decision went the other way; the API half merged at 15:33.")*
@@ -62,7 +63,7 @@ Pick one in Scope. It sets the reader, the level of detail and the defaults belo
 | | **Release story** | **Build story** (default) | **Technical deep dive** |
 |---|---|---|---|
 | Reader | Users and prospective users | Developers who know the product | Engineers who want the mechanism |
-| Answers | What's new, why it matters, how to use it | What we built, how it works, what shaped it | How it works inside, the trade-offs, the edge cases |
+| Answers | What's new, why it matters, what changes for you | What we built, how it works, what shaped it | How it works inside, the trade-offs, the edge cases |
 | Detail | Product and outcomes; internals in a sentence | Concepts and decisions; internals only where they explain something the reader sees | Mechanisms, data structures, failure modes |
 | Visuals | Screenshots, before/after, one simple flow | Diagrams, motion, before/after | Diagrams, terminal runs, charts |
 | Code | A usage example at most | Only where the code is the idea | Where it helps |
@@ -72,7 +73,7 @@ Pick one in Scope. It sets the reader, the level of detail and the defaults belo
 
 **Size it to the subject, not the flavor.** Word counts are ceilings with no minimum (release story up to 1,200, build story up to 2,500, deep dive usually under 3,000; chapters, visuals and sizing by scope in [story.md](references/story.md#length)). A small change gets a **note**: a paragraph or three, no chapters, no masthead, at most one visual.
 
-**Say when it isn't worth a write-up.** If the subject is thin (a typo, a one-line fix, a dependency bump, a refactor users never see), check the nearby work first. If the thin change is one piece of a bigger idea, that is a change of subject and worth stopping for: say in a sentence why the change alone isn't a post, name the bigger idea and the angle you'd lead with, and ask whether to write that instead (check it first with `--pr`). If there is no bigger idea, say plainly it's not worth a post and offer a short note or a changelog line.
+**Say when it isn't worth a write-up.** A subject is thin when it is small (a typo, a one-line fix, a dependency bump, a refactor users never see) and also when it is standard practice: work any capable team would do the usual way, like self-hosting web fonts, adding chapters to a video player or writing a sitemap. Readers already know how those work, so explaining one at length talks down to them. This holds even when the author asks for a deep dive; standard practice earns a post only if it went somewhere unusual (a measured result, a surprising failure, a choice most teams would make differently). For a thin subject, check the nearby work first. If the thin change is clearly one piece of a bigger idea (the same release, the same effort, work the author would plainly see as one story), write that post and say in the hand-off, in a sentence, that you widened the scope and why. The author would rather have a draft to react to than a question. Stop and ask only when the bigger idea is a real change of subject: a different feature, a much larger post, or work the author might not want public. Then say in a sentence or two why the change alone isn't a post (the author needs the verdict, not the full case), name the bigger idea and the angle you'd lead with, and ask whether to write that instead (check it first with `--pr`). Often the bigger idea is an arc across several modest changes from the same weeks ("three ways our examples got easier to follow"), where each change is a section, not a post. If there is no bigger idea, say plainly it's not worth a post and offer a short note or a line in a larger release.
 
 ## Workflow
 

skills/buildnotes/references/craft.md

@@ -37,6 +37,8 @@ The title says plainly what shipped, in the reader's words: "Release webhooks fo
 - **The reader test.** Someone who sees only the title should know what changed. "Various improvements" and a version number fail.
 - **The outcome, not a generic verb.** Lead with the specific thing and its new state. "Worktrees no longer drop unpushed commits" beats "Fixes worktree bug dropping unpushed commits". No filler ("a number of", "various").
 - **Headings make the argument.** A reader skimming only the headings should learn what shipped and why. "One call per release" passes; "A smarter approach", "Overview" and "Background" don't. A heading says what a thing does, not its code name or version; in a deep dive, or a post the author asked to be about the engineering, a heading may name the mechanism when the mechanism is the subject.
+- **Plain words, everyday order.** Write a heading as a phrase a person would say: "Labels sit next to the parts", not "Read the name where the part is". Aim for the reading level of a good product page; if a heading needs a second read, rewrite it.
+- **Name the feature, not the example.** A title tied to the one place it showed up ("Jump to any chapter of the example videos") reads narrower than the change. Name what the reader gets wherever they meet it.
 - **Short.** About eight words for the title, fewer for headings. Cut the subtitle-after-a-colon formula unless both halves earn their place.
 
 From real drafts:

skills/buildnotes/references/story.md

@@ -70,7 +70,7 @@ Don't open with the history of the codebase, a definition, a date, or a question
 
 ## The ending
 
-End on what readers can do now (one way to try it, and the docs for the rest) or on what's next. A command they'll paste goes in a command line (`<div class="cmd">`), which has a copy button ([visuals.md](visuals.md#components)). A post is a starting point: don't tour every way to set the feature up unless the author asks for a guide. Where it fits, the way to try it proves the feature's promise, not just its setup: the smallest sequence that shows the thing working ("send two prompts; the second reuses the tab the first opened"). *(Fault: a post about a browser that persists between turns, whose try-it was a setup snippet with no second turn.)* No summary of what they just read, and no sign-off ("stay tuned", "more to come", "we can't wait"). A colophon line about how the post was made is optional; use one if the author wants it.
+End on what readers can do now (one way to try it, and the docs for the rest) or on what's next. Not every post needs a call to action. A quality-of-life fix that users get just by upgrading needs no try-it section; one line saying which release has it is enough. Likewise, when the change is a new default that arrives on its own, end with what changes for the reader and how to keep the old behaviour if that matters, in a sentence or two, not a try-it chapter. Skip it too when the main user is an agent rather than a person. *(Fault: a "Try it" chapter for a changed figure default, built around a JSON option nobody needed.)* A command they'll paste goes in a command line (`<div class="cmd">`), which has a copy button ([visuals.md](visuals.md#components)). A post is a starting point: don't tour every way to set the feature up unless the author asks for a guide. Where it fits, the way to try it proves the feature's promise, not just its setup: the smallest sequence that shows the thing working ("send two prompts; the second reuses the tab the first opened"). *(Fault: a post about a browser that persists between turns, whose try-it was a setup snippet with no second turn.)* No summary of what they just read, and no sign-off ("stay tuned", "more to come", "we can't wait"). A colophon line about how the post was made is optional; use one if the author wants it.
 
 ## Voice
 

skills/buildnotes/scripts/lint.mjs

@@ -72,7 +72,7 @@ const BANNED = [
   "comprehensive", "robust", "seamless", "seamlessly", "powerful", "blazing", "world-class", "game-changing",
   "cutting-edge", "leverage", "leverages", "leveraging", "supercharge", "elevate", "delve", "journey",
   "deep dive", "unlock", "unlocks", "it's worth noting", "worth noting", "let's take a look", "in this post",
-  "as mentioned above", "here's the thing", "it turns out", "enter:", "game changer", "best-in-class",
+  "as mentioned above", "here's the thing", "it turns out", "enter:", "game changer", "best-in-class", "earns its keep", "earns its place", "pulls its weight",
   // Grading the work instead of stating it, filler for what changed, and sign-offs.
   "doubling down", "double down", "leap forward", "transformative", "next-generation", "continues to evolve",
   "substantial improvements", "exciting new", "utilize", "utilizes", "utilizing", "facilitate", "facilitates",