Skip to content

Prefer native SharePoint routing and add measured evaluations - #242

Closed
Patrick Rodgers (patrick-rodgers) wants to merge 1 commit into
microsoft:mainfrom
patrick-rodgers:sharepoint-native-routing
Closed

Patrick Rodgers (patrick-rodgers) wants to merge 1 commit into
microsoft:mainfrom
patrick-rodgers:sharepoint-native-routing

Conversation

@patrick-rodgers

Copy link
Copy Markdown
Member

Summary

  • Make the official workiq skill prefer supported /sharepoint/ operations for SharePoint discovery, content, metadata, browsing, and changes. Preserve other-workload routing and explicitly requested original-file downloads.
  • Add focused semantic-passage/structured-text guidance, batching and provenance rules, authoritative library-metadata recipes, and correct provider-function JSON-body documentation.
  • Add an independent evaluation harness, a 168-case development/holdout matrix, all 80 advertised native paths, stateful mocks, scorer/provenance/budget/write-boundary tests, and offline-only CI.
  • Leave workiq-preview, hosted endpoint, marketplace registries, and public plugin version unchanged.

Measured routing results

Real gpt-6.1-sol sessions against controlled mocks; the frozen final comparison contains 56 distinct held-out cases with three repetitions per arm. These are observed model choices, not unit-test trigger rates or live-service completion claims.

Held-out metric Baseline Promoted guidance
Task-relevant native routing 17/138 (12.3%) 138/138 (100%)
Native task success 10/138 (7.2%) 136/138 (98.6%)
Appropriate Botdoc use 11/72 (15.3%) 72/72 (100%)
Botdoc content success 9/72 (12.5%) 72/72 (100%)
Overall correctness 155/168 (92.3%) 166/168 (98.8%)
Runs attempting unrequested downloads 2 0
  • Semantic search itself was selected for 13/24 focused questions; the other 11 used suitable structured text. The 100% content-routing result is not a semantic-search-only rate.
  • All 24 other-workload controls and all six explicit-download runs passed in each arm.
  • Catalog breadth improved from 52/76 to 69/76 expected substantive path/schema/scope selections; both arms selected 3/4 helpers. Seven substantive misses remain reported.
  • All 32 fresh installed-package checks passed automatic skill activation, package/reference provenance, and correctness: 28/28 eligible native routes, 16/16 content routes, and zero unrequested downloads. These are development smoke cases, not a second untouched holdout.

Evidence and limitations

Full sanitized results, evolution, confidence intervals, failures, costs, and provenance

The table uses the disclosed amended scorer. Original acceptance failed; a narrowly scoped duration-typography correction made amended acceptance pass without changing candidate instructions or routing/use outcomes. Original scores/evidence and both folder-verification holdout failures are preserved. A separate download-receipt hardening changed zero observed scores. Folder-fixture repairs were checked only in fresh development follow-ups.

This is primarily model/mock evidence, not proof that every advertised path works live. Read-only live checks established site-key resolution, library discovery, and item enumeration; the visible library was empty, so live Botdoc extraction and mutation completion were not demonstrated. Shell/network builtins were excluded from evaluation, so unrestricted direct HTTP was not tested. Legacy CRUD fixture asymmetry also limits baseline correctness comparisons. The report retains these limitations and the increased tool-call/latency costs.

The experiment finished 888 of the authorized 1,000 fresh-session reservations (885 normalized observations and three raw-only diagnostics). Raw traces, tenant content, credentials, and private experiment artifacts are not included.

Validation

  • npm --prefix tests\workiq-sharepoint test - 50/50 passed.
  • npm --prefix tests\workiq-guidance test - 348/348 passed.
  • git diff --check passed; the official plugin was reinstalled and its accepted instruction bytes verified.
  • CI runs offline tests only; it does not launch models, install plugins, contact Microsoft 365, or perform live writes.

Prioritize native discovery, semantic passages, structured document text and authoritative metadata without unrequested original-file downloads. Preserve other-workload routes and explicit-download exceptions.

Add a frozen core/breadth matrix, stateful mock, opt-in fresh-session runner, offline tests and sanitized results with disclosed scorer amendments and installed-package provenance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 368a4b09-7ffc-4ec1-b718-abc44648a275
@patrick-rodgers

Copy link
Copy Markdown
Member Author

Superseded by an internal preview-focused review. Closing this public PR so discussion and validation stay with the replacement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant