[emmavdh]

Pull Request Description

What and why?

site/scripts/generate_chatbot_product_map.py lists at most six related docs per route in site/llm/chatbot-product-map.md. The ranking scored candidates by the length of the matched keyword segment, but on the workflow routes every candidate matches the same workflows segment, so they all tied and alphabetical path order picked the six. Pages from other sections that only mention workflows in their filename (/faq/, /guide/integrations/, /guide/monitoring/, /guide/risk-tiering/) sorted ahead of most of /guide/workflows/, and /guide/workflows/manage-record-stages.html dropped out of the map entirely. Nothing reported what the cap left out.

This PR changes the ranking and makes evictions visible:

• Candidates are ranked by section before the cap is applied: pages in the same directory as the route's primary docs come first, then pages in a section named by a route keyword (for example /guide/workflows/ for /workflows), then pages that only match a keyword in their filename.
• Inside a tier, keyword specificity comes first, then position in the docs sidebars (site/guide/_sidebar.yaml and the other sidebar files), then path. Docs writers already order each sidebar section on purpose, so a route keeps the overview and setup pages listed first. Pages missing from the sidebar sort after the listed ones, so adding a page no longer evicts anything unless it is placed ahead of it in the sidebar.
• The related docs the cap leaves out are written to site/llm/chatbot-product-map-dropped.md, committed next to the map. A new eviction shows up as a changed line in the PR diff. The file is not copied into the LLM corpus (llm/render.sh copies only the map, and site/llm/ is its own Quarto project, so it is not published either). The cap stays at six, now the RELATED_DOCS_CAP constant.
• RELATED_DOC_KEYWORDS gains "tier": ["risk-tiering"]. Without it, /settings/risk-tier-templates lost its own guide to document-template pages, and /settings/risk-tier-stages had no related docs at all.
• "integrations" now maps to /guide/integrations/ only, plus new connections, secrets and data-exports keywords. It used to map to configuration as well, and the longer segment won on specificity, so the three integrations settings screens listed organization setup pages and none of their own guides (also the case on main).
• CI: the generator tests now run on every PR, targeted previews included, and before the map auto-commit in the validate workflow. A guide edit that evicts a pinned page fails the PR that caused it instead of a later one. Both CI checks include the dropped file, and the validate workflow auto-commits it along with the map.
• Regenerated the map and the dropped file with python3 site/scripts/generate_chatbot_product_map.py (the CI command), with site/installation populated the way CI does it. The frontend snapshot is unchanged. site/llm/README.md describes the ranking and the new file.

Before and after for the related docs on the three workflow routes:

Route	Before (main)	After
/settings/workflows (primary: setting-up-workflows)	/faq/faq-workflows.html /guide/integrations/integrations-examples/use-webhooks-with-workflows.html /guide/monitoring/trigger-workflows-on-threshold-breach.html /guide/risk-tiering/set-up-risk-tiering-workflows.html /guide/workflows/assign-workflow-executions.html /guide/workflows/conditional-step-requirements.html	/guide/workflows/introduction-to-workflows.html /guide/workflows/workflow-step-types.html /guide/workflows/conditional-step-requirements.html /guide/workflows/workflow-states.html /guide/workflows/manage-record-stages.html /guide/workflows/configure-workflows.html
/settings/workflow-states (primary: workflow-states)	Same as above	/guide/workflows/introduction-to-workflows.html /guide/workflows/workflow-step-types.html /guide/workflows/conditional-step-requirements.html /guide/workflows/setting-up-workflows.html /guide/workflows/manage-record-stages.html /guide/workflows/configure-workflows.html
/workflows (no primary)	Same as above	/guide/workflows/introduction-to-workflows.html /guide/workflows/workflow-step-types.html /guide/workflows/conditional-step-requirements.html /guide/workflows/workflow-states.html /guide/workflows/setting-up-workflows.html /guide/workflows/manage-record-stages.html

Primary docs are unchanged on every route.

Other routes that change, compared with main:

• Seven inventory settings routes now list the inventory setup pages (setting-up-the-inventory, record types, fields, segments, record intake, working with the inventory) in place of the record-level pages (archive, interdependencies, layout, overview page, editing, import) and the inventory FAQ.
• Four templates and documents routes (/settings/block-library, /settings/document-types, /settings/documents/general, /settings/templates) drop /guide/risk-tiering/manage-risk-tier-templates.html and the document checker page and pick up the templates overview pages (working-with-document-templates, managing-documents).
• /settings/risk-tier-templates and /settings/risk-tier-stages list the five /guide/risk-tiering/ guides first.
• /settings/attestation-templates swaps /faq/faq-attestations.html for /guide/templates/working-with-document-templates.html.
• /settings/integrations/connections, /secrets and /data-exports now list their own guide first (configure-connections, manage-secrets, configure-data-exports), then the other /guide/integrations/ pages, in place of six /guide/configuration/ pages.
• /analytics, /attestations and /settings/attestations only reorder.

How to test

From the repo root, with site/installation populated (CI copies it from validmind/installation; locally make -C site clone copy-installation), otherwise the SSO installation page loses its section list:

python3 site/scripts/generate_chatbot_product_map.py
git diff --exit-code -- site/llm/chatbot-product-map.md site/llm/chatbot-product-map-dropped.md
python3 -m unittest discover -s site/scripts -p 'test_generate_chatbot_product_map.py' -v

The diff should be empty and all 19 tests should pass. New tests cover same-section pages outranking cross-section filename matches, keyword-section pages outranking filename matches on routes without primary docs, sidebar order breaking ties inside a section, reading the sidebar while skipping commented-out entries, the dropped file listing evicted paths, a regression check that the three workflow routes keep manage-record-stages.html, and a regression check that /settings/risk-tier-templates lists its risk-tiering guides ahead of other sections, and a regression check that the three integrations screens list their own guide and only /guide/integrations/ pages.

What needs special review?

• FAQ pages rank below guide pages in a route's own section, so they drop off routes whose section fills the six slots (faq-workflows, faq-inventory, faq-attestations on /settings/attestation-templates). This is intended. They are listed in the dropped file.
• Docs PRs that change site/guide/_sidebar.yaml can now change the map: moving a page up a section can push another page off a route. The dropped file shows it in the diff, and for manage-record-stages on the workflow routes the regression test fails with a message pointing at the sidebar.
• Guide-only PRs now run the generator tests (about 0.1 s). If one goes red on test_workflow_routes_keep_manage_record_stages, the fix is a sidebar or ranking decision, not a code bug in that PR.
• Integrations settings screens no longer list /guide/configuration/ pages such as private connectivity and organization setup. If any of those belong there, they need their own keyword.
• Section ranking only helps where a route keyword names the section. The risk-tiering gap was one case, and the dropped file is the place to spot others.

Dependencies, breaking changes, and deployment notes

Changes .github/workflows/validate-docs-site.yaml (generator tests run on every PR, before the map auto-commit) and .github/workflows/full-docs-validation.yaml (tests run first, and the dropped file is checked). No deployment steps.

Release notes

Internal: no release notes.

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[juanmleng]

Review: #1600 at ee6e908 (merge-base 8ed2521)

Computed by the calibrated-review skill, run by @juanmleng.

Merge with recommendations · no blocking · 2 small fixes · 1 question for you

The ranking fix works: the three workflow routes keep manage-record-stages, and the new tier keyword matches only the two risk-tier routes.

Small fixes: F1 run the generator tests on guide-only PRs (today they're skipped, so the next eviction is auto-committed and fails a later, unrelated PR), F2 drop the brittle first-position assertion in the risk-tier test.

❓ @emmavdh — should new evictions show in the PR diff? Options: commit the dropped list as a sidecar file, or warn only when a route's dropped list changes.

Fix later: F4 alphabetical tiebreak inside a section.

_{calibrated-review v2 · review · run by @juanmleng · head ee6e908}

Agent handoff · calibrated-review v2 · review · ee6e908

calibrated-review: 2
mode: review
pr: validmind/documentation#1600
head: ee6e9086ef83e10bd04acac2cee4a301242f7645
merge_base: 8ed252152bf3f99e1ccb57d3dcb6638e5540968f
run_by: juanmleng
verdict: merge-with-recs
downgraded: 1 # F1 fails gate 2: the eviction needs a future guide PR
items:
  - id: F1
    class: fix-now
    type: correctness
    direction: risky
    title: Generator tests are skipped on guide-only PRs, so the regression test fires on the wrong PR
    scenario: >-
      A PR adds site/guide/workflows/approve-workflow-steps.qmd and
      archive-workflows.qmd. The preview selector marks it targeted, so the
      "Test chatbot product map generator" step is skipped. The map step
      regenerates the map without manage-record-stages on the three workflow
      routes and the bot auto-commits it. The PR merges green. The next PR that
      runs full validation fails test_workflow_routes_keep_manage_record_stages
      for an eviction it did not cause. The PR body says the regression test
      will flag this case.
    evidence:
      - .github/workflows/validate-docs-site.yaml:419 gates the test step on targeted != 'true'
      - .github/workflows/validate-docs-site.yaml:386 map step is ungated and auto-commits the regenerated map
      - .github/scripts/select_docs_preview_targets.py exits 0 (targeted) for "added site/guide/workflows/archive-workflows.qmd"
      - with the two pages added, test_workflow_routes_keep_manage_record_stages fails for all three routes
    fix: >-
      Remove the targeted gate from the test step (no render needed, ~0.1 s)
      and run it before the map auto-commit, so a failing test stops the
      commit. If that is out of scope, correct the PR body's claim.
    done_when: a guide-only PR runs the generator tests before the map is committed
  - id: F2
    class: fix-now
    type: hygiene
    direction: safe
    title: Risk-tier regression test pins the alphabetically first page
    scenario: >-
      Adding any risk-tiering guide whose name sorts before "configure-"
      (about-..., assign-...) fails
      test_risk_tier_templates_route_keeps_risk_tiering_guide although the
      route still lists its own guides.
    evidence:
      - site/scripts/test_generate_chatbot_product_map.py:178 assertEqual(related[0], ".../configure-risk-tier-calculation.html")
      - the assertIn on the line above already covers the docstring's intent
    fix: >-
      Delete the related[0] assertion, or assert that the leading entries are
      all under /guide/risk-tiering/.
    done_when: the test asserts membership or section, not alphabetical position
  - id: F3
    class: question
    type: risk-appetite
    direction: safe
    title: The drop report is not visible during review
    scenario: >-
      The report goes to stderr in a CI step that passes, and it lists all 20
      routes the cap already trims on every run. A new eviction looks the same
      as the standing ones, and a reviewer reading the PR diff does not see it.
      The story asked for the eviction to show during review.
    evidence:
      - site/scripts/generate_chatbot_product_map.py:795 prints the full report to stderr
      - regenerated at head, the report lists 20 routes
    fix: >-
      Owner @emmavdh decides: commit the dropped list as a sidecar file next to
      the map (kept out of the LLM corpus) so evictions appear in the diff, or
      emit a ::warning:: only for routes whose dropped list changed.
  - id: F4
    class: fix-later
    type: scope-expansion
    direction: safe
    title: Within a section, alphabetical order still picks the six
    scenario: >-
      On routes whose own section has more than six matching pages (workflows,
      inventory, attestation templates), path order decides what the assistant
      sees. The author notes this under special review.
    fix: >-
      Ticket: add a small per-route pin list or rank by keyword match in the
      page title, so the canonical how-to for a screen is chosen on purpose.
accepted_risks:
  - checked: removal of "route.related_docs or" in the render loop
    why: route.related_docs is only ever assigned from suggest_related_docs, so the output is unchanged
  - checked: the "tier" keyword is a substring match
    why: across the frontend snapshot it matches only /settings/risk-tier-templates and /settings/risk-tier-stages
  - checked: FAQ pages losing their slot on crowded routes
    why: disclosed by the author under special review, and it shows in the drop report
verification:
  - run: python3 -m unittest discover -s site/scripts -p 'test_generate_chatbot_product_map.py'
    result: 16 passed at head
  - run: python3 site/scripts/generate_chatbot_product_map.py
    result: one-line diff on the installation page; site/installation not populated locally
  - run: add approve-workflow-steps.qmd and archive-workflows.qmd under site/guide/workflows, rerun the tests
    result: 3 subtest failures in test_workflow_routes_keep_manage_record_stages
  - claim: "Before and after" route table (PR body), workflow rows
    result: matches the regenerated map
  - claim: "Before and after" route table (PR body), other route swaps
    result: NOT CHECKED
  - claim: "the drop report and the regression test will flag that" (PR body)
    result: false for guide-only PRs, see F1

[github-actions]

Validate docs site

✓ INFO: A live preview of the docs site is available — Open the preview

[github-actions]

Lighthouse check results

✓ INFO: No site pages to audit in this PR.

Commit SHA: ee6e908

[cachafla]

Checked head ee6e908: related docs are now ranked by section (the primary doc's own section first, then sections a keyword names, then docs that match only by filename) before the six-entry cap. Docs the cap drops are reported on stderr. The regenerated map looks right: manage-record-stages stays on the workflow routes, and risk-tiering guides are no longer listed under the document settings routes. CI is green, and all 16 tests in test_generate_chatbot_product_map pass locally.

[emmavdh]

Thanks @juanmleng, this was useful. All four are addressed in 5ae1769.

F1: Agreed, and I reproduced it. The generator tests now run on every PR, targeted previews included, and before the map auto-commit, so an eviction fails the PR that caused it. full-docs-validation runs them first too.

F2: Done. The risk-tier test checks that the route keeps manage-risk-tier-templates and that the risk-tiering guides come before any other section, without pinning a specific page.

F3: Yes, I want evictions in the diff. The generator now writes the dropped list to site/llm/chatbot-product-map-dropped.md instead of stderr. CI checks it and auto-commits it with the map. It stays out of the LLM corpus and isn't published.

F4: I folded this into the PR rather than a follow-up. Inside a tier, ties now break on position in the docs sidebars before path, so each route keeps the pages docs writers list first. Pages that aren't in the sidebar sort last. With your two example pages added, nothing is evicted unless they're placed ahead of manage-record-stages in the sidebar. When they are, the regression test fails with a message pointing at the sidebar. This changes the lists on the workflow, inventory and templates routes, and the PR description has the updated before and after.

[juanmleng]

Thanks Emma — checked 5ae1769: tests now run on every PR before the map auto-commit, the risk-tier test checks order by section, the dropped list is committed and stays out of the LLM corpus, and sidebar order handles the tiebreak. All four items resolved, nothing new from me.

[emmavdh]

One more change in 15c9e77. The three integrations settings screens (Connections, Secrets, Data exports) listed only /guide/configuration/ pages, on main too. The integrations keyword also mapped to configuration, and the longer segment won on specificity. It now maps to /guide/integrations/ only, with keywords for each screen so it lists its own guide first. Only those three routes change, and there's a regression test for them.

[juanmleng]

Checked 15c9e77 too — only the three integrations routes change, each lists its own guide first, tests pass. Still good to merge.