[raymondginger2018-sudo]

Summary

Adds aggregation for the five LLMOps metrics: Quality / Harm / Honesty / Cost / Latency.

This module consumes recorded per-call records and aggregates them into the five dimensions:

• Cost — from token usage × a pluggable per-model price table (USD)
• Latency — from recorded durations (ms), p50/p95/max
• Quality / Harm / Honesty — machine-observable proxies + optional LLM-as-judge hook (judge_fn) for sampled labels; without a judge these stay None (unmeasured) rather than fabricated

Design

• Pure mechanism: consumes a list of record dicts and returns a summary dict
• No I/O, no network, no model calls by default
• Pluggable price table: DEEPCODE_LLM_PRICES env or caller-supplied table
• Honest about unmeasured dimensions (no fabricated scores)

File

• core/observability/llmops.py (new, 158 lines)

Part of GenAI lesson 14 LLMOps family.

[raymondginger2018-sudo]

设计说明

问题：agent 运行的质量、安全、诚实、成本、延迟五个维度分散在各处日志里，没有统一聚合视图。管理者难以回答"这周 agent 系统整体健康吗"。

解法：五维聚合器——每个维度定义评分函数（0-100），最终输出综合报告。价格表通过 DEEPCODE_LLM_PRICES 环境变量注入（默认内置常见模型价格）。

关键设计决策：

• 五维正交：Quality（任务完成率）、Harm（安全事件）、Honesty（幻觉/误导）、Cost（token 费用）、Latency（响应延迟）——避免一维分数掩盖问题
• 价格表可注入，不硬编码——环境变量 DEEPCODE_LLM_PRICES 传 JSON
• 输入是结构化事件列表，输出是聚合摘要，无副作用

测试建议：构造各维度的事件样本，验证评分单调性（更多坏事件 → 更低分）

[Zongwei9888]

Merged into main as 2b223c1, with a substantial repair on our side, so I want to be clear about what survived. Thank you @raymondginger2018-sudo — the need itself ("turn llm.jsonl into a number an operator can act on") was the right call, and the latency/status aggregation and percentile helper are yours.

What changed and why:

• Prices come from the model catalog (resolve_model_info, USD per 1M tokens) instead of a $0.001/1K default and a DEEPCODE_LLM_PRICES env table. A default price makes the dollar figure look real when it is invented; a model with no list price now contributes its tokens but no dollars and is counted in unpriced_calls, and usd is None rather than zero when nothing is priced.
• The quality / harm / honesty dimensions were removed. Without a judge they were always None, and a judge hook without any judge in the repository is configuration for a feature that does not exist. If an LLM-as-judge sampler lands later, it can be a separate report.
• Per-model breakdown and a CLI (python -m core.observability.llmops <llm.jsonl> [--json]) so the summary is reachable without writing code, with seven tests.

The public function is now summarize_llm_calls() rather than aggregate_llmops(), to match what it actually reports.