"A true master teaches not by telling, but by refining." - The Skill Sensei
Sensei is an evidence engine for Agent Skills. It validates portable frontmatter, checks host compatibility, analyzes the full skill resource graph, suggests authoring improvements with the Ralph loop pattern, and optionally measures observed host behavior against a baseline.
Specification versus advice: score keeps portable diagnostics in specChecks
and optional Sensei recommendations in checks, including license/version suggestions.
Additive findings and profiles distinguish portable specification, host
compatibility, information architecture, behavioral evidence, safety/supply
chain, and lifecycle health.
The standard requires only name and description; 150 characters, 60 words,
WHEN:, type prefixes, and the 500-token target are heuristics, not normative rules.
Static scores and proof artifacts are neither observed activation nor security audits.
Unavailable evidence stays unknown; it never becomes a success-shaped default.
See scoring and host portability
for constraints, runtime limitations, and the separately owned Copilot extension work in #5.
- Overview
- Quick Start
- Distribution and Release Status
- Prerequisites
- How It Works
- Evidence Profiles
- Repository Analysis
- Configuration
- Scoring Criteria
- Examples
- Troubleshooting
- Contributing
Skills can have valid frontmatter and still fail in practice. Common issues include:
- No triggers - Agent doesn't know when to activate the skill
- Ambiguous scope - Similar skills compete for the same request
- Vague descriptions - Length alone does not establish useful context
- Token bloat - Oversized skills waste context window
- Broken resources - Missing, escaping, deep, or orphaned references undermine progressive disclosure
- Unproven behavior - Static prose looks good without improving routing or task quality
- Hidden risk - Scripts, permissions, dependencies, and release state are not captured by a frontmatter score
Sensei implements the "Ralph Wiggum" technique:
- Read - Load the skill's current state and token count
- Score - Evaluate portable compliance and the selected host profile
- Improve - Clarify positive scope and test close near-misses
- Verify - Run tests to ensure changes work
- Check Tokens - Analyze token usage, gather suggestions
- Analyze - Check the resource graph, scripts, trust boundaries, and lifecycle evidence
- Summary - Display before/after findings and evidence limits
- Prompt - Ask user: Commit, Create Issue, or Skip?
- Repeat - Until the intended gates and tests pass
Run sensei on my-skill-name
Run sensei on my-skill-name --fast
Run sensei on skill-a, skill-b, skill-c
Run sensei on all Low-adherence skills
Run sensei on all skills
Run sensei on my-skill-name --gepa
Run sensei score my-skill-name
Score-only mode runs without LLM calls.
GitHub is the primary distribution channel. Pin release and Action tags; npm is an optional channel and may lag.
$package = 'https://github.com/spboyer/sensei/releases/download/v1.6.3/spboyer-sensei-1.6.3.tgz'
npm install --global $package
sensei score .
sensei score . --profile=copilot
sensei analyze skills/my-skill --root .
sensei check --root . --config .token-limits.json --strict--root resolves paths; --profile selects portable or Copilot frontmatter;
--config selects limits; --emit-proof writes sensei-audit.md;
.sensei.json enables repo policy. The release page also includes the committed
source bundle, provenance JSON, and checksum manifests.
steps:
- uses: actions/checkout@v4
- uses: spboyer/sensei@v1.6.3
with:
command: check
root: .
path: .
config: .token-limits.json
strict: 'true'import { scoreSkillContent } from '@spboyer/sensei/score';
import { parseFrontmatter } from '@spboyer/sensei/parse';
import { checkNameCompliance } from '@spboyer/sensei/checks';
import { analyzeSkillRepository } from '@spboyer/sensei/analyze';
const result = scoreSkillContent(renderedSkillMarkdown, {
path: 'skills/my-skill',
moduleCount: 2,
profile: 'copilot'
});
const repositoryReport = analyzeSkillRepository('skills/my-skill', { repositoryRoot: '.' });
const frontmatter = parseFrontmatter(renderedSkillMarkdown);
checkNameCompliance(frontmatter?.name ?? '');path is metadata; moduleCount defaults to 0. parseFrontmatter returns
null for absent/unclosed delimiters, and optional errors for invalid delimited
YAML. Nested metadata and scalar types are preserved; inspect diagnostics before
using typed convenience fields.
npm run --silent tokens -- eval manifest.json --root=/path/to/project --format=jsonRun from a current Sensei checkout with script dependencies installed. Compare repeated positive, near-miss, sibling-collision, and held-out host observations against a without-skill or prior-version baseline. Assertions can inspect output, JSON shape, artifacts, tool trajectories, or trusted executable checks. Improvement claims require an explicit positive lift threshold. Unavailable evidence cannot pass; GEPA keyword fitness is not observed activation. No provider or Python dependency is added. See the trusted adapter contract, exit codes, and protocol smoke test.
Sensei does not collapse unlike evidence into one grade.
| Profile | What it establishes |
|---|---|
| Portable specification | Agent Skills frontmatter compliance against commit-pinned upstream fixtures |
| Host compatibility | Portable or Copilot host fields without mislabeling extensions as portable |
| Information architecture | Resource pointers, progressive disclosure, workflow criteria, and advisory sprawl signals |
| Behavioral evidence | Observed activation, routing metrics, task assertions, trajectories, variance, and baseline lift |
| Safety and supply chain | Path containment, script capability surface, dependency locks, permissions, and resource limits |
| Lifecycle health | Ownership, changelog, eval coverage, mirrors, versions, and release evidence |
Every normalized finding reports pass, fail, warning, unknown, or
not-applicable, plus its basis, evidence level, remediation, and limitations.
The legacy specChecks and checks arrays remain available.
sensei analyze skills/my-skill --root . --format=jsonanalyze builds a contained manifest for SKILL.md, references, assets,
scripts, and linked repository resources. It checks Markdown and Copilot
#file: references, reference depth, symlinks, aggregate MCP resource limits,
script readiness and capability signals, dependency locks, ownership,
changelog, and skill-specific eval evidence. Static findings preserve explicit
limitations and do not certify runtime safety.
uv run scripts/src/gepa/auto_evaluator.py score --skill my-skill --skills-dir skills --tests-dir tests
uv run --with-requirements scripts/src/gepa/requirements.txt scripts/src/gepa/auto_evaluator.py optimize --skill my-skill --skills-dir skills --tests-dir testsScore-only mode needs no GEPA installation. Optimization requires the optional
version in scripts/src/gepa/requirements.txt and model access; if that version
is unavailable from your package index, optimization cannot run. GEPA uses
keyword/proxy fitness; it does not observe whether a host loaded a skill.
| Flag | Description |
|---|---|
--fast |
Skip tests for faster iteration |
--gepa |
Use GEPA evolutionary optimization instead of template-based improvements |
--skip-integration |
Skip integration tests (unit + trigger tests only) |
⚠️ Note: Using--fastspeeds up the loop significantly but may miss issues. Consider running full tests before final commit.
Version 1.6.3 contains the evidence profiles, calibrated authoring checks, host profiles, repository analyzer, stronger behavioral evaluation, and Aspire pilot described here. GitHub is the authoritative release channel. Each release includes a compiled CLI tarball, source archive, provenance JSON, and checksums. npm is optional and may remain behind until its trusted-publisher binding is configured.
GitHub Release assets include
spboyer-sensei-<version>.tgz, sensei.tar.gz, source.json, SHA256SUMS,
and RELEASE-SHA256SUMS. Release assets do not expire. Workflow artifacts
retain the source bundle for 30 days. Neither path updates downstream dotfiles
automatically.
-
Node.js 22.x (22.12.0+), 24.x, or 26.0.0+ - For running the current source CLI and contributor toolchain
node --version
-
Git - For commits and comparisons
git --version
-
Test Framework - Jest, pytest, or similar for trigger tests
-
Python 3.10+, uv, and GEPA - Optional proxy-based optimization
Option 1: Copilot CLI skill and source bundles
mkdir -p "$HOME/.copilot/skills"
git clone https://github.com/spboyer/sensei.git "$HOME/.copilot/skills/sensei"
cd ~/.copilot/skills/sensei/scripts && npm installThis is the personal Copilot CLI location, not a guarantee of cloud availability. Check the surface/location matrix. Invoke with:
Run sensei on my-skill-name
For project-specific installation:
mkdir -p .github/skills
git clone https://github.com/spboyer/sensei.git .github/skills/sensei
cd .github/skills/sensei/scripts && npm installFor CLI use without npm registry publication:
npm install --global https://github.com/spboyer/sensei/releases/download/v1.6.3/spboyer-sensei-1.6.3.tgz
sensei check .cd ~/.copilot/skills/sensei && npm run tokens -- check
sensei check .Read the skill and tests, count baseline tokens, then validate YAML and portable
fields with npm run tokens -- score <skill-dir>. Review optional authoring
suggestions separately. Scaffold missing tests, clarify positive scope, and test
realistic prompts plus close near-misses. GEPA mode optionally proposes changes
using proxy fitness, not observed activation.
Run the target project's tests and npm run tokens -- check <skill-dir>/SKILL.md.
Configured token warnings are separate from optimization suggestions. Summarize
changes, test evidence, token deltas, and remaining recommendations. Ask whether
to commit, create an issue, or skip without discarding edits unasked. Repeat until
the advisory target and tests pass, or report a timeout after five iterations.
See the detailed loop for framework-specific guidance.
When running on multiple skills:
- Skills are processed sequentially
- Each skill goes through the full loop
- User prompted after each skill: Commit, Create Issue, or Skip
- Summary report at the end shows all results
| Setting | Default | Description |
|---|---|---|
| Skills directory | skills/ or .github/skills/ |
Where SKILL.md files live |
| Tests directory | tests/ |
Where test files live |
| Max iterations | 5 | Per-skill iteration limit before moving on |
| Target score | Medium-High | Advisory workflow target |
| Token soft limit | 500 | SKILL.md target token count |
| Token check limit | .token-limits.json |
Configurable local threshold; spec recommends <5000 |
| User prompt | After each skill | Commit, Create Issue, or Skip |
| Continue on failure | Yes | Process remaining skills if one fails |
Override defaults in your prompt:
Run sensei on my-skill with skills in src/ai/skills/ and tests in spec/
This is the workflow's authoring rubric, not a computed CLI level. The CLI reports individual static checks. Do not pad descriptions to hit a minimum length.
| Level | Description | Criteria |
|---|---|---|
| Low | Vague description | Purpose or scope unclear |
| Medium | Has trigger keywords | Purpose recognizable, routing ambiguous or noisy |
| Medium-High | Has WHEN: or USE FOR: | "WHEN:" (preferred) or "USE FOR:" with ≤60 words |
| High | Routing clarity | Medium-High + useful routing guidance (INVOKES/FOR SINGLE OPERATIONS) |
-
Name validation
- Lowercase ASCII letters, digits, and hyphens; no leading/trailing or consecutive hyphens
- Matches directory name
- ≤ 64 characters
-
Description length
- No specification minimum beyond non-empty; do not pad short useful descriptions
- Maximum: 1024 characters (spec limit)
-
Trigger phrases
- Contains "WHEN:", "USE FOR:", or "Use this skill when"
- Lists specific keywords and phrases
-
Scope boundaries (evaluate on the target host)
- Prefer concise positive routing by default
- Test negative boundaries as near-miss prompts; no mandatory exclusion syntax
-
Routing clarity (for High score)
- Skill type prefix:
**WORKFLOW SKILL**,**UTILITY SKILL**, or**ANALYSIS SKILL** INVOKES:lists tools/MCP servers the skill callsFOR SINGLE OPERATIONS:guidance for when to bypass skill
- Skill type prefix:
For this advisory target, aim for:
- ✅ Clear intent and scope without padding
- ✅ Explicit trigger phrases ("WHEN:" preferred, or "USE FOR:")
- ✅ Description ≤ 60 words
- ✅ SKILL.md < 500 tokens (soft limit, monitored)
To reach High, add routing clarity:
- ✅ All Medium-High criteria
- ✅ Skill type prefix (
**WORKFLOW SKILL**, etc.) - ✅
INVOKES:listing tools/MCP servers used - ✅
FOR SINGLE OPERATIONS:bypass guidance
For skills that invoke MCP tools, review this authoring checklist from MCP Integration Patterns. It is not a computed CLI sub-score:
| Check | Purpose |
|---|---|
| MCP Tools Used table | Documents tool dependencies in skill body |
| Prerequisites section | Lists required tools and permissions |
| CLI fallback pattern | Provides fallback when MCP unavailable |
| Name collision review | Compare skill names with actual installed tool names |
See references/mcp-integration.md for detailed patterns.
- SKILL.md: < 500 tokens (soft), < 5000 (hard)
- references/*.md: < 2000 tokens each
- Score skill:
npm run tokens -- score [dir] - Check with:
npm run tokens -- check - Get suggestions:
npm run tokens -- suggest
---
name: pdf-processor
description: 'Process PDF files for various tasks'
---Problems:
- No trigger phrases
- Scope is not specific enough to distinguish nearby tasks
- Agent doesn't know when to activate
---
name: pdf-processor
description: "Extract, rotate, merge, and split PDF files. WHEN: \"extract PDF text\", \"rotate PDF pages\", \"merge PDFs\", \"split PDF\", \"PDF to text\"."
---Improvements:
- ~160 characters (informative but under limit)
- Clear description of purpose
- Explicit WHEN: trigger phrases with distinctive quoted strings
---
name: azure-deploy
description: "**WORKFLOW SKILL** - Deploy prepared Azure applications. WHEN: \"deploy to Azure\", \"azd up\", \"publish to Azure\". INVOKES: Azure deployment tools. FOR SINGLE OPERATIONS: use the deployment CLI directly."
---High score achieved with:
- Skill type prefix (
**WORKFLOW SKILL**) INVOKES:lists MCP tools usedFOR SINGLE OPERATIONS:guides when to bypass skill
Before (empty):
const shouldTriggerPrompts = [];
const shouldNotTriggerPrompts = [];After:
const shouldTriggerPrompts = [
'Extract text from this PDF',
'Rotate this PDF 90 degrees',
'Merge these PDF files together',
'Split this PDF into pages',
'Convert PDF to text',
];
const shouldNotTriggerPrompts = [
'Create a new PDF document',
'Extract images from this PDF',
'OCR this scanned document',
'What is the weather today?',
'Help me with AWS S3',
];Use realistic in-scope prompts and close near-misses derived from intended scope. Do not merely mirror literal keywords or add negative frontmatter syntax to satisfy tests.
Separate spec failures (such as description >1024 characters) from optional recommendations. Investigate overlapping scope with real host observations.
git reset --soft HEAD~1 # Undo last commit- Edit
SKILL.mdfor instruction changes - Edit
references/*.mdfor documentation changes - Test tokens:
npm run tokens -- check - Test on a sample skill before committing
- Document the rule in
references/scoring.md - Add examples in
references/examples.md - Update scoring criteria in SKILL.md
- Create template in
references/test-templates/{framework}.md - Document usage in references/configuration.md
Sensei supports Waza-style trigger accuracy testing. See the Waza test template.
Open an issue with skill name, starting state, and git log --oneline -10.
- Ralph Loop Pattern - Sensei's iterative improvement workflow
- Distribution - Commit-pinned source bundles
- Behavioral Evaluation - Trusted adapters and observed evidence
- Adapter Mappings - Copilot/Vally and Promptfoo mappings
- Aspire Pilot - Corpus results and false-positive controls
- Host Portability - Installation and runtime differences
- Anthropic Skills Documentation - Writing guidance
- MCP Integration Patterns - MCP integration best practices
- Waza Trigger Test Template - Skill trigger accuracy testing
- GEPA - Evolutionary optimization for skills
- skill-creator - For creating new skills from scratch
Sensei - "The path to compliance begins with a single trigger." 🥋