feat(productivity): add write-naturally and public evals

.claude-plugin/marketplace.json

@@ -27,6 +27,11 @@
       "description": "Transparent spreadsheet financial modeling with reusable calculation blocks and validated outputs",
       "source": "./plugins/finance",
       "category": "finance"
+    },
+    {
+      "name": "productivity",
+      "description": "Clear technical writing with behavioral evaluation cases",
+      "source": "./plugins/productivity"
     }
   ]
 }

.github/plugin/marketplace.json

@@ -27,6 +27,11 @@
       "description": "Transparent spreadsheet financial modeling with reusable calculation blocks and validated outputs",
       "source": "./plugins/finance",
       "category": "finance"
+    },
+    {
+      "name": "productivity",
+      "description": "Clear technical writing with behavioral evaluation cases",
+      "source": "./plugins/productivity"
     }
   ]
 }

.gitignore

@@ -1,3 +1,6 @@
 node_modules/
 .cache/
 coverage/
+.env
+.promptfoo/
+plugins/**/evals/**/outputs/

README.md

@@ -24,6 +24,7 @@ Claude Code and Copilot CLI can use this repository's native marketplace manifes
 | --- | --- |
 | [deepwiki](plugins/deepwiki) | Public repository documentation tools and guidance |
 | [engineering](plugins/engineering) | Interactive PR review skill |
+| [productivity](plugins/productivity) | Direct technical writing and behavioral evals |
 | [finance](plugins/finance) | Financial modeling guidance and an original fictional XLSX example |
 | [tradingview](plugins/tradingview) | TradingView market data MCP configuration |
 
@@ -52,6 +53,16 @@ bun run validate
 bun test
 ```
 
+The productivity plugin includes a writing eval suite. Install development dependencies with `bun install`, then use `bun run eval:validate` or `bun run eval:write-naturally`. Its 28 fixtures are fictional and cover team messages, customer updates, planning notes, reviews and technical designs. The writer and judge run in a read-only workspace with network access disabled inside the model workspace; model-service requests still use the configured Codex provider. The suite never posts reviews. The writer defaults to `gpt-6.1-sol`; the grader defaults to `gpt-5.6-luna`. Use `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to override them separately. Focus on review comments with `bun run eval:write-naturally --filter-metadata group=review-regression --max-concurrency 2`.
+
+Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private).
+
+### Humanizer comparison
+
+Eight fictional cases inspired by [Humanizer](https://github.com/blader/humanizer/blob/225a6f39ac85f76ee48dbad772ea4abe4ed6c9d8/SKILL.md) cover rhetorical framing, inflated claims, formatting, voice and replies to readers who already know the context. Controls keep useful checklists and plain technical wording. Run the same cases with the skill using `bun run eval:write-naturally --filter-metadata group=humanizer`, then without it using `bun run eval:write-naturally:baseline`. Both conditions use the same writer model, reasoning setting and graders. The baseline rejects recorded tool calls, including file reads, and missing traces. This checks the model turn; it cannot rule out instructions supplied by the surrounding Codex configuration.
+
+Readability has separate style criteria for these cases; fact preservation is graded independently. Calibration answers stay in metadata and are never supplied to the writer. Compare individual outputs and repeat runs before attributing a gain to the skill.
+
 Changes are reviewed through pull requests. See [AGENTS.md](AGENTS.md) for contribution guidance.
 
 ## License

package.json

@@ -3,6 +3,13 @@
   "private": true,
   "scripts": {
     "validate": "bun run scripts/validate-catalogue.ts",
-    "test": "bun test"
+    "test": "bun test",
+    "eval:validate": "promptfoo validate config -c plugins/productivity/evals/write-naturally/promptfooconfig.yaml",
+    "eval:write-naturally": "promptfoo eval -c plugins/productivity/evals/write-naturally/promptfooconfig.yaml --no-cache --no-share",
+    "eval:write-naturally:baseline": "bun run scripts/eval-writing-baseline.ts plugins/productivity/evals/write-naturally"
+  },
+  "devDependencies": {
+    "@openai/codex-sdk": "0.162.1",
+    "promptfoo": "0.124.1"
   }
 }

plugins/productivity/.claude-plugin/plugin.json

@@ -0,0 +1,9 @@
+{
+  "name": "productivity",
+  "description": "Direct, evidence-led technical writing and behavioral evaluations.",
+  "author": {
+    "name": "AllAgents"
+  },
+  "version": "1.0.0",
+  "homepage": "https://github.com/allagentsdev/agent-plugins/tree/main/plugins/productivity"
+}

plugins/productivity/README.md

@@ -0,0 +1,11 @@
+# Productivity
+
+## Write naturally: practical gotchas
+
+- **Long conversations:** an agent can read the skill early and later drift from it. Before the final version, check which guidance it is applying. If the answer is vague or the prose has drifted, ask it to reread the skill and the relevant reference. A claim that it remembers the skill is weaker evidence than a fresh read.
+- **Use your paraphrase:** if you need to ask “so you mean …?”, give the agent your plain-language interpretation, confirm the meaning, then ask it to put that explanation into the text. The next reader should understand it without repeating your clarification.
+- **Separate wording from judgment:** clearer prose cannot establish whether a concern is a blocker. Settle the evidence and intended action, then ask for a final version that carries that decision and only the context its recipient needs.
+
+For example: “So you mean a caller can send text where an integer is required, and validation accepts it?” Once confirmed: “Put that caller example directly in the comment, with what happens and the proposed follow-up.”
+
+The [core skill](skills/write-naturally/SKILL.md) applies to general prose. It loads additional guidance for [reviews](skills/write-naturally/references/reviews.md) or [technical designs](skills/write-naturally/references/technical-designs.md) when needed. The [public eval suite](evals/write-naturally/promptfooconfig.yaml) uses fictional examples, including nontechnical writing. Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private).

plugins/productivity/evals/write-naturally/assertions/baseline-no-tools.mjs

@@ -0,0 +1,17 @@
+// Audit the full Codex turn, rather than Promptfoo's path-based skill heuristic.
+export default function baselineNoTools(_output, context) {
+  const fail = (reason) => ({ pass: false, score: 0, reason });
+  if ((context.metadata?.skillCalls || []).length) return fail('Baseline recorded a skill read.');
+  let turn;
+  try {
+    turn = JSON.parse(context.providerResponse?.raw);
+  } catch {
+    return fail('Baseline has no parseable Codex turn trace.');
+  }
+  if (!Array.isArray(turn?.items) || !turn.items.some((item) => item?.type === 'agent_message')) {
+    return fail('Baseline trace has no recorded agent response.');
+  }
+  const tools = turn.items.filter((item) => !['agent_message', 'reasoning'].includes(item?.type));
+  if (tools.length) return fail(`Baseline recorded tool items: ${tools.map((item) => item?.type || 'unknown').join(', ')}.`);
+  return { pass: true, score: 1, reason: 'Baseline trace records a response and no tool calls.' };
+}

plugins/productivity/evals/write-naturally/baseline-workspace/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

plugins/productivity/evals/write-naturally/extensions/workspace.mjs

@@ -0,0 +1,24 @@
+import { cpSync, mkdirSync, rmSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const suiteRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..');
+const pluginRoot = resolve(suiteRoot, '../..');
+const workspace = resolve(suiteRoot, 'workspace');
+const workspaceAgents = resolve(workspace, '.agents');
+const skillSource = resolve(pluginRoot, 'skills/write-naturally');
+const skillTarget = resolve(workspaceAgents, 'skills/write-naturally');
+
+export function workspaceHook(hookName, context) {
+  if (hookName === 'beforeAll') {
+    rmSync(workspaceAgents, { recursive: true, force: true });
+    mkdirSync(dirname(skillTarget), { recursive: true });
+    cpSync(skillSource, skillTarget, { recursive: true });
+  }
+
+  if (hookName === 'afterAll') {
+    rmSync(workspaceAgents, { recursive: true, force: true });
+  }
+
+  return context;
+}

plugins/productivity/evals/write-naturally/promptfooconfig.yaml

@@ -0,0 +1,116 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: write-naturally structure, fidelity and final review outcomes
+
+extensions:
+  - file://extensions/workspace.mjs:workspaceHook
+
+prompts:
+  - id: file://prompts/rewrite.md
+    label: write-naturally
+
+providers:
+  - id: openai:codex-sdk
+    label: codex-write-naturally
+    config:
+      model: '{{ env.CODEX_MODEL | default("gpt-6.1-sol") }}'
+      model_reasoning_effort: medium
+      working_dir: ./workspace
+      sandbox_mode: read-only
+      approval_policy: never
+      network_access_enabled: false
+      web_search_enabled: false
+      enable_streaming: true
+      maxRetries: 0
+
+defaultTest:
+  assert:
+    - type: skill-used
+      metric: repository-skill-used
+      value: write-naturally
+    - type: javascript
+      metric: repository-skill-path
+      value: |
+        const calls = context.metadata?.skillCalls || [];
+        const expected = '.agents/skills/write-naturally/SKILL.md';
+        const paths = calls.map((call) => String(call.path || '').replace(/\\/g, '/'));
+        const pass = paths.some((path) => path === expected || path.endsWith(`/workspace/${expected}`));
+        return {
+          pass,
+          score: pass ? 1 : 0,
+          reason: pass
+            ? 'Codex read the repository-local eval workspace skill.'
+            : `Expected repository-local skill read; observed: ${paths.join(', ') || 'none'}`,
+        };
+    - type: javascript
+      metric: source-link-destinations
+      value: |
+        const urls = (text) => new Set(
+          (String(text).match(/https?:\/\/[^\s<>()]+/g) || [])
+            .map((url) => url.replace(/[.,;:!?]+$/, '')),
+        );
+        const actual = urls(output);
+        const missing = [...urls(`${context.vars.draft || ""}\n${context.vars.context || ""}`)].filter((url) => !actual.has(url));
+        return {
+          pass: missing.length === 0,
+          score: missing.length === 0 ? 1 : 0,
+          reason: missing.length
+            ? `Source link destinations missing or changed: ${missing.join(', ')}`
+            : 'Source link destinations preserved.',
+        };
+    - type: llm-rubric
+      metric: direct-open-and-faithful
+      provider: &writingJudge
+        id: openai:codex-sdk
+        config:
+          model: '{{ env.CODEX_JUDGE_MODEL | default("gpt-5.6-luna") }}'
+          model_reasoning_effort: medium
+          working_dir: ./workspace
+          sandbox_mode: read-only
+          approval_policy: never
+          network_access_enabled: false
+          web_search_enabled: false
+          maxRetries: 0
+      value: |
+        Judge factual fidelity and the requested stance, not the source design's completeness or correctness.
+        Grade only the candidate in the <Output> block. The source draft below may deliberately contain problems that the candidate fixes; text appearing only in that draft is not a candidate defect. Before quoting a candidate failure, verify the quoted wording appears in <Output>. Missing required facts can still fail.
+        Check each required fact against what the candidate actually says, including the actor, action, affected entity, conditions and limits. A related operation or outcome does not preserve the same guarantee.
+        The current request and supplied context define the intended stance and retained facts. Preserve every applicable source claim and meaning-related requirement. When the request asks for a final comment, apply its selection of relevant material: omit superseded findings and drafting history, and use the current classification. Do not demand their retention or let a stale draft override the current decision. Readability is scored separately.
+        A changed guarantee, omitted constraint or contradictory sentence is a failure even when the surrounding prose is correct.
+        Require only the detail supplied in the source. An explicitly preserved guarantee is sufficient when the source does not explain its implementation; do not demand or invent a mechanism.
+        On failure, name the violated requirement and quote the candidate text responsible, or identify the missing fact.
+        Do not require exact wording or a fixed heading template.
+
+        Requested surface: {{surface}}
+        Pass criterion: {{pass_criterion}}
+        Failure conditions: {{fails_if}}
+
+        Current context (when supplied):
+        {{context | default("")}}
+
+        Source draft:
+        {{draft}}
+    - type: llm-rubric
+      metric: readable-on-first-pass
+      provider: *writingJudge
+      value: |
+        Judge only readability and the requested tone, not factual fidelity or architectural completeness. Factual fidelity is scored separately.
+        Match the requested surface: a short comment can be one conversational paragraph; a longer design needs a visible flow and navigable explanations.
+        For a design, check that the opening gives a working picture, actors and actions are concrete, and each failure stays beside its consequence.
+        Reject a long design collapsed into one dense paragraph, a dense matrix or disconnected slogan-like fragments, even if all source facts remain.
+        When the context says the recipient already knows an explanation, apply the case's requirement to lead with the decision rather than repeating that explanation. Clear wording alone does not satisfy a request to omit known context.
+        Use the case's style requirements below. Do not require exact wording, particular heading names or a word-count target. Already-readable source prose may remain unchanged.
+        Require explanations to remain as concrete as the supplied source, not more technically detailed. Do not fail an explicitly preserved guarantee because its implementation mechanism is absent from both source and candidate.
+        Consider only the style-related parts of the case criteria below; factual substitutions belong to the fidelity grader.
+        On failure, identify the readability problem in the candidate.
+
+        Requested surface: {{surface}}
+        Pass criterion: {{style_criterion | default(pass_criterion)}}
+        Failure conditions: {{style_fails_if | default(fails_if)}}
+
+        Current context (when supplied):
+        {{context | default("")}}
+
+        Source draft:
+        {{draft}}
+
+tests: file://tests/*.yaml

plugins/productivity/evals/write-naturally/prompts/baseline.md

@@ -0,0 +1,15 @@
+For this baseline experiment, do not read or apply any skill files.
+
+Use the current request and supplied context to determine the intended position and retained facts. Preserve load-bearing factual claims; omit superseded findings and drafting history when producing the final comment. Match the structure and level of detail to the requested surface. Return only the rewritten text.
+
+Surface: {{surface}}
+
+Request: {{request}}
+
+Current context (when supplied):
+
+{{context | default("")}}
+
+Draft:
+
+{{draft}}

plugins/productivity/evals/write-naturally/prompts/rewrite.md

@@ -0,0 +1,15 @@
+First run `cat .agents/skills/write-naturally/SKILL.md` and read its output. Then use $write-naturally from that file to write the requested text. Read the reference it selects for the requested surface, if any, before drafting.
+
+Use the current request and supplied context to determine the intended position and retained facts. Preserve load-bearing factual claims; omit superseded findings and drafting history when producing the final comment. Match the structure and level of detail to the requested surface. Return only the rewritten text.
+
+Surface: {{surface}}
+
+Request: {{request}}
+
+Current context (when supplied):
+
+{{context | default("")}}
+
+Draft:
+
+{{draft}}

plugins/productivity/evals/write-naturally/tests/cases.yaml

@@ -0,0 +1,80 @@
+# Stable case IDs; v2 replaces private source material with fictional fixtures.
+- description: '[C1] succinct review comment stays direct and open'
+  vars:
+    surface: Pull-request review comment
+    request: Question whether the Router should require headers.caller. Keep it succinct, direct, and open to a
+      different intended meaning.
+    draft: |
+      Open question
+
+      The Job Contract leaves headers.caller optional. The Router rejects a request unless that header is portal.
+
+      Red
+
+      A portal request with the header omitted fails validation.
+
+      Green
+
+      The Router receives the portal ingress event as PortalRequest. Infer the caller there, validate the request, then start Temporal.
+    pass_criterion: Asks whether the stricter headers.caller rule is needed, explains that Router already knows
+      the Portal ingress and carries PortalRequest, and remains open to caller having separate compatibility semantics.
+      Uses concise conversational prose without headings.
+    fails_if: Uses Open question, Red, or Green headings; turns the comment into a design document; declares the
+      decision closed; loses the optional-versus-required mismatch; or invents facts.
+  metadata:
+    id: C1
+    group: core
+    surface: comment
+    dataset_version: v2-public-fictional
+    fixture_origin: Original fictional fixture; no captured review text or private provenance.
+  assert:
+  - type: javascript
+    metric: comment-no-template-headings
+    value: |
+      const templateHeading = /^(?:#{1,6}\s+|\*\*(?:open question|red|green)\*\*\s*$|(?:open question|red|green)\s*$)/mi;
+      return !templateHeading.test(output);
+  - type: javascript
+    metric: comment-succinctness
+    value: |
+      const words = output.trim().split(/\s+/).filter(Boolean);
+      const questions = (output.match(/\?/g) || []).length;
+      return words.length <= 110 && questions >= 1 && questions <= 2;
+  - type: contains
+    value: headers.caller
+  - type: icontains
+    value: optional
+  - type: icontains
+    value: Router
+  - type: icontains
+    value: PortalRequest
+- description: '[D1] technical design keeps useful navigation'
+  vars:
+    surface: Technical architecture decision
+    request: Rewrite this into a concise technical design that a reviewer can navigate. Keep the decision, precedence,
+      boundaries, and verification distinct.
+    draft: |
+      Configuration has three layers. Values in config.local.yaml override config.yaml, except empty scalar values leave the lower layer active. If both files lack an active value, use repository defaults. Secret storage and deployment policy are outside this change. Check local override, shared fallback, an empty local value and default fallback.
+    pass_criterion: Uses useful Markdown headings to separate the decision and supporting contract, preserves the
+      exact configuration precedence and exclusions, and remains concise.
+    fails_if: Removes headings entirely; turns the design into a conversational comment; changes the precedence;
+      treats an empty scalar as an override; adds secret-storage or deployment commitments; or invents evidence.
+  metadata:
+    id: D1
+    group: core
+    surface: technical-design
+    dataset_version: v2-public-fictional
+    fixture_origin: Original fictional fixture; no captured review text or private provenance.
+  assert:
+  - type: javascript
+    metric: design-navigation
+    value: |
+      const headings = output.match(/^#{1,3}\s+\S.+$/gm) || [];
+      return headings.length >= 2;
+  - type: contains
+    value: config.local.yaml
+  - type: contains
+    value: config.yaml
+  - type: icontains
+    value: empty
+  - type: icontains
+    value: verification