Skip to content

Add inline script environment lifecycle telemetry (PEP 723 PR 15/16) - #1723

Open
Stella Huang (StellaHuang95) wants to merge 4 commits into
microsoft:mainfrom
StellaHuang95:pep723-pr15-telemetry-v2
Open

Add inline script environment lifecycle telemetry (PEP 723 PR 15/16)#1723
Stella Huang (StellaHuang95) wants to merge 4 commits into
microsoft:mainfrom
StellaHuang95:pep723-pr15-telemetry-v2

Conversation

@StellaHuang95

@StellaHuang95 Stella Huang (StellaHuang95) commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Part of #1602 (PEP 723 inline script env support). Design doc: #1601.

Builds on the merged creation, uv-fallback, and persistence work in #1656, #1696, and #1697. This branch is rebased on current main.

Roadmap context

This is PR 15 of 16 in the PEP 723 inline-script roadmap. It adds the remaining lifecycle telemetry for the internal manager without changing routing, creation, or cache policy.

Phase 5: Lifecycle and polish PR Status
PR 13: clear inline-script cache separate
PR 14: opportunistic TTL eviction follow-up
PR 15: lifecycle telemetry this PR
PR 16: status-bar decision resolved; no code PR

Why this PR

The manager can create, reuse, and persist inline-script environments, including consent-gated uv/Python fallback, but those lifecycle outcomes are not observable as a coherent feature funnel.

This PR adds low-cardinality telemetry that answers:

  • whether setup built a new environment or reused a validated cache entry;
  • how long actual environment construction took;
  • how many normalized dependencies were involved; and
  • which stable failure category prevented setup.

The instrumentation is deliberately attached to the underlying coalesced operation rather than every caller, and it excludes script/package/interpreter content.

What this PR does

Adds three typed lifecycle events

  • inlineScript.envCreated
  • inlineScript.envReuseHit
  • inlineScript.envError

The telemetry constants and GDPR declarations use typed event/property mappings so call sites cannot send undeclared fields.

Reports verified creation only

  • Emits envCreated only after environment creation succeeds, ownership/version validation passes, and .meta.json is persisted.
  • Measures the actual build/rebuild interval rather than lock waiting or cache inspection.
  • Reports normalized, deduplicated dependency count rather than raw dependency values.
  • Emits one creation event for the underlying coalesced operation, even when multiple callers await it.

Reports validated cache reuse only

  • Emits envReuseHit only after sidecar, interpreter, ownership, and compatibility checks complete successfully.
  • Does not report a hit for uncertain, stale, malformed, or partially validated entries.
  • Preserves the existing cache reuse and lastUsedAt behavior.

Classifies lifecycle failures without leaking details

envError uses stable low-cardinality categories for outcomes such as:

  • environment discovery failure;
  • no compatible Python;
  • declined compatible-Python installation;
  • uv/Python installation failure;
  • general setup, cache-validation, or metadata-persistence failure;
  • package-install cancellation;
  • cache-lock timeout; and
  • retained, orphaned, or otherwise unavailable lock state.

Errors remain logged through the existing paths; telemetry sends no exception text.

Preserves uv consent and compatibility behavior

  • Adds detailed internal uv lookup/install result types so the manager can distinguish available, declined, failed, and installed outcomes.
  • Keeps existing compatibility wrappers for callers that only need boolean/path results.
  • Does not change prompt text, consent requirements, install selectors, restart-required behavior, or uv's existing telemetry.

Keeps event emission coalesced and deterministic

  • Same-key concurrent creation/reuse callers receive one lifecycle result for the shared operation.
  • Cache inspection, lock acquisition, fallback selection, cancellation, and cleanup retain their existing control flow.
  • Non-applicable create() calls and pre-validation exits do not emit success-shaped lifecycle events.

Event payloads and privacy

Event Data
inlineScript.envCreated build duration; normalized dependency count
inlineScript.envReuseHit normalized dependency count
inlineScript.envError stable failure category

The events send no:

  • script URI or filesystem path;
  • requirement or dependency value;
  • package name;
  • Python/interpreter version;
  • cache key;
  • prompt text; or
  • exception/error-message content.

Script-controlled metadata therefore cannot create unbounded telemetry dimensions.

Lifecycle examples

validated cache hit
→ emit one inlineScript.envReuseHit
cache miss or stale entry
→ start build timer
→ build + install dependencies
→ validate ownership/version
→ persist sidecar
→ emit one inlineScript.envCreated
no compatible Python
→ user dismisses consent prompt
→ emit one inlineScript.envError with the declined category

Tests

Coverage includes:

  • creation emitted only after verified sidecar persistence;
  • validated reuse and rebuild behavior;
  • one event for concurrent/coalesced callers;
  • duration boundaries excluding lock wait/cache inspection;
  • normalized/deduplicated dependency counts;
  • discovery, compatibility, consent, installation, cancellation, setup, and lock categories;
  • no lifecycle telemetry for non-applicable calls;
  • direct detailed uv results for available, declined, failed, and installed outcomes; and
  • preservation of existing uv wrapper behavior.

Validation on the rebased branch:

  • npm run compile-tests
  • npm run compile
  • npm run lint
  • focused lifecycle/detailed-uv suites: 39 passing

The full Windows unit run reaches 1612 passing and 5 pending; the existing concurrent writeMetaJson rename test can still intermittently fail with EPERM on Windows. That writer is unchanged by this PR and the same failure is reproducible on main.

Performance

  • No activation work, scan, timer, watcher, or new filesystem operation is introduced.
  • Instrumentation performs constant-size event construction around operations that already occur.
  • Coalesced setup emits once rather than once per waiter.
  • No user-controlled strings are normalized or transmitted beyond the dependency count already needed for the cache operation.

User impact

No default-path user impact. The inline manager remains behind the undeclared, default-off python-envs.inlineScripts.enabled flag.

When the internal flag is manually enabled, prompts, environment creation/reuse, cancellation, error propagation, and cache behavior remain unchanged. This PR only records privacy-safe lifecycle outcomes.

Scope and follow-up

This PR intentionally does not implement:

  • script detection or automatic routing;
  • activation-time cache discovery;
  • project registration or user-facing setup UX;
  • cache clearing or TTL eviction; or
  • status-bar behavior.

The telemetry is ready for those later entry points to consume once the feature is intentionally exposed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895
Cover detailed uv outcomes and normalized dependency counts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Keep telemetry-only helpers scoped to their consuming test suite.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
@rchiodo

Rich Chiodo (rchiodo) commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

Comment thread src/common/telemetry/constants.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
@rchiodo Rich Chiodo (rchiodo) added the review-auto:changes-requested Automated review: posted blocking findings to address. label Aug 18, 2026
Report reuse dependency counts and preserve accurate final failure outcomes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added review-auto:approved Automated review: no blocking findings (approval posted). and removed review-auto:changes-requested Automated review: posted blocking findings to address. labels Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature-request Request for new features or functionality review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants