Use this interface for manual collection and inspection. For agent tool calls, use OpGuard Harness; the machine-readable backend contract is documented in AgentTools.md. Installation and native extension options are in Installation.md.
The interactive CLI keeps session state in ~/.opguard_state.yaml. Use a dedicated workload checkout: instrumentation edits Python source. Do not edit those files or run harness instrumentation concurrently in that checkout.
opguard --help
opguard start --help
opguard run --help
opguard compare --help| Command | Purpose |
|---|---|
doctor [--json] [--require-native] |
Inspect Torch, CUDA, CUPTI and native extension availability |
configure |
Show the current configuration |
dumpto DIRECTORY |
Select a capture output root |
start [DIRECTORY] |
Emit shell activation commands and enable discovery |
stop |
Finish discovery and generate an instrumentation configuration |
instrument |
Apply the selected instrumentation configuration |
replay |
Rerun the recorded workload with instrumentation |
run SCRIPT [SCRIPT ...] |
Run the interactive discovery/capture workflow for scripts |
compare RUN_A RUN_B |
Materialize captures and generate Perfetto comparisons |
sync |
Request synchronization of locally staged captures |
clear |
Restore instrumentation and clean interactive session state |
replay and run execute workload code. Include discovery and capture executions in the workload budget. Prefer the harness's explicit entrypoint specification when launch provenance and bounded retries matter.
Optional repository helpers are documented in scripts/README.md.
Shell activation commands must be evaluated in the shell that launches the workload:
eval "$(opguard start /absolute/path/to/new-capture)"
python train.py
eval "$(opguard stop)"
opguard replayAutomatic discovery uses Python's usercustomize startup hook. An interpreter that disables the user site, including many isolated virtual environments, may not load this hook. Check python -c 'import sys; print("usercustomize" in sys.modules)' in that environment. If it is unavailable, use the explicit opguard.tools worker through the harness instead; that path does not depend on the startup hook.
stop disables autostart for subsequent processes. Ordinary tracing preserves the workload's numerical execution settings. Explicit deterministic interventions are described in Installation.md.
For the quick workflow, supply executable Python or shell scripts:
opguard run run_a.py run_b.py --gpu 0 --disable-cupti --disable-cudaapiThe command creates captures and comparisons; inspect its output and workload logs. It does not return a root-cause diagnosis or numerical equivalence verdict.
For manual instrumentation, prepare a configuration for your workload and run the low-level instrumenter from a separate working copy:
python -m opguard.instrument_all_files config.yaml
python train.py
python -m opguard.instrument_all_files --mode revert --dry-run
python -m opguard.instrument_all_files --mode revertConfiguration paths resolve relative to the configuration file. Backups and a manifest are maintained under .opguard/ in the working directory. The regular revert path checks changes; inspect a mismatch before deciding how to reconcile it. Interactive opguard clear performs forced restoration, so it is not a substitute for reviewing concurrent user edits. Harness capture uses its separate source transaction and restoration protocol.
To convert an existing callsite artifact into a configuration, the supported module and argument are:
python -m opguard.tools.callsite_to_instrumentation \
--callsite-file /path/to/callsites.json \
--base-dir /path/to/workload \
--output /path/to/capture.yamlSee Preprocessing.md for the compression and configuration pipeline.
opguard compare /path/to/run-a /path/to/run-b --gpu 0For already materialized rank directories, the exporter can write a selected output directly:
python -m opguard.trace_export.perfetto \
--run1 /path/to/run-a/exec_trace/gpu_000 \
--run2 /path/to/run-b/exec_trace/gpu_000 \
--out /path/to/comparison.json.gzOpen the generated trace in Perfetto. Reported pairs are correspondence candidates, and matching recorded fingerprints do not prove tensor equality. AlignmentMaps.md describes explicit reviews. Capture.md describes local staging, distributed paths, transport completeness and comparison evidence.