Skip to content

Check that group skills select and dispatch what they should #66

Open

Description

@glennfaison

Category: feature
Area: Evaluation (tests/scenarios)
Impact: The group skills decide which items get worked and what each item is told. A wrong selection or a dropped input is invisible to the item-level scenarios.
Confidence: traced

Problem (the use case)

scripts/scenario.mjs skips group skills. Their contracts promise things worth testing: at most *_batch items, oldest first, each one passing the item skill's requires, stuck and human-merge items excluded, drafts excluded for review, and every dispatch given write_log=false plus the inputs the item also lists. The dispatch-forwarding fix in #51 has no test beyond the contract text.

Proposed solution

Add scenarios for the five group skills under tests/scenarios/<group>/, with a dispatches: expectation listing the items, in order, and the inputs each was given.

  • Selection: a world with more eligible items than the batch, an ineligible one, and a stuck one. Assert the batch and the order.
  • Forwarding: assert write_log=false and every shared input the item lists (runner, unattended, trusted_authors, ...), and none that it does not.
  • Log: one entry whose results match the dispatched items.
  • The checker uses the real item contract's params to decide what must be forwarded.

Acceptance criteria

  • The checker supports dispatches: for group skills.
  • At least three scenarios for each of the five group skills, including nothing eligible.
  • A scenario that drops trusted_authors from a dispatch fails.
  • triage-github-issues is covered, including its call to reconcile-github-issues.

Non-goals

Running the dispatched items.

Size: medium

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions