Skip to content

fix(ruby-llm): contain recalled tracing safety gaps - #1091

Merged
nik13 merged 2 commits into
mainfrom
fix/opus-followup-docs-1077
Nov 2, 2026
Merged

nik13 merged 2 commits into
mainfrom
fix/opus-followup-docs-1077

Conversation

@nik13

@nik13 nik13 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #1077 after reproducing both recall findings with released gems and synthetic loopback traffic.

  • Qualify native error coverage: RubyLLM 2.1.0–2.1.1 can finish outer-timeout, Interrupt and killed-thread requests with UNSET/no error.type. Document HTTP timeout configuration and cooperative stream cancellation without promising hard-deadline coverage.
  • Add lifecycle regressions for all three limitations, HTTP timeout, chat.cancel and tool exceptions. Verify actual OTLP status, privacy and span counts.
  • Replace wildcard metadata namespace trust in the retained legacy fixture with an exact attribute allowlist. Add custom request/response/usage attribute regressions and protobuf-wire assertions, retaining event removal and metadata-only status sanitization.
  • Remove the copied legacy exporter from the customer guide. Explain how to pause legacy tracing without deleting only its privacy safeguard, preserve the migration warnings, and identify the old example as a regression fixture rather than an installable integration.
  • Add network-isolated legacy CI for RubyLLM 1.15.0/1.16.0 on Ruby 3.2/3.4; retain the native 2.1.0/2.1.1 matrix.

Explicit remaining work

This is containment, not a complete legacy integration release. The namespace bypass is fixed in the repository fixture, but this PR does not deliver a maintained, installable Ruby privacy adapter. Existing copied installations do not automatically receive that fix. The customer legacy route remains disabled until migration to native tracing or delivery of that package. No new copied helper is recommended and no package availability is invented.

The upstream native interruption behavior is unchanged; this PR corrects its documented coverage and provides tested alternatives, rather than monkeypatching RubyLLM.

Verification

All local execution used the network-denying sandbox, synthetic credentials and disposable homes. No provider or Future AGI Cloud calls.

  • Before: native recall probes failed the ERROR expectation for outer timeout, Interrupt and thread kill on both 2.1.0 and 2.1.1. The 1.16.0 legacy wire probe exported synthetic private content under gen_ai.request.parameters. New page contracts and legacy namespace regressions also failed before the fix.
  • After: native suite passes 10 tests / 257 assertions per version, RubyLLM 2.1.0 and 2.1.1, Ruby 3.2.2.
  • After: legacy recipe, opt-in and wire suites pass 14 tests / 94 assertions per version, RubyLLM 1.15.0 and 1.16.0, instrumentation 0.7.1. The exact recall loopback probe also passes against the hardened fixture on both versions.
  • Page contracts: 4/4. Integration/scaffolding tests: 20/20. Link audit: zero broken nav/content links.
  • Astro build and Pagefind pass, 797 pages indexed. Isolated Chromium render: HTTP 200, no page errors; inspected lifecycle warning, timeout code block and legacy hold warning.
  • Ruby 3.4 is configured in PR CI, not claimed as a local run.

Re-run native checks from the repository root with node --test scripts/ruby-llm-page.test.mjs and RUBY_LLM_VERSION=2.1.1 ruby scripts/ruby-llm-native.test.rb after installing the versions listed in the native test header. Run the legacy checks with bundle exec rake test in examples/frameworks/ruby-llm, setting RUBY_LLM_VERSION before installing its Gemfile. Use loopback-only networking for both suites.

E2E coverage

Not applicable (no e2e/ harness). Runtime integration behavior is covered by real released RubyLLM gems and decoded OTLP; browser validation covers the rendered guide. No live backend UI mapping is claimed.

No registry publication, release, merge, or default-branch push is included.

🤖 Generated with Claude Code

nik13 and others added 2 commits November 2, 2026 05:12
Qualify native lifecycle error coverage and test interruption, HTTP timeout, cancellation and tool exceptions. Replace the legacy fixture's namespace trust with an exact attribute allowlist and regression tests. Hold the copied customer route pending a maintained legacy package rather than claim the fixture is an installable replacement.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Test the documented two-turn session in a fresh process without the legacy patch, while preserving unrelated shared-provider tracing.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 merged commit 1bdd912 into main Nov 2, 2026
10 checks passed
@nik13
nik13 deleted the fix/opus-followup-docs-1077 branch November 2, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant