Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ site](app/site/README.md).
| [Judgement](docs/judgement.md) | What a judge model is asked |
| [The Nullius standard](docs/standard.md) | Ports, kits, setups, controls, proofs; importing |
| [Architecture](docs/how-it-works.md) | Execution paths, a run on disk, the pages |
| [Settings](docs/settings/README.md) | Providers, System One, LLM judge, hosts, keys |
| [Settings](docs/settings/README.md) | Providers, Decision, LLM judge, hosts, keys |
| [Benchmarks](docs/benchmarks/supported/README.md), [Agents](docs/agents/supported/README.md), [Capabilities](docs/capabilities/supported/README.md) | What is supported; adding one |
| [CLI](docs/cli/README.md) | Every command, by topic |
| [Metric field guide](docs/field-guide.md) | Every metric and outcome |
Expand Down
18 changes: 9 additions & 9 deletions app/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -33,20 +33,20 @@ OPENAI_API_KEY=
OPENROUTER_API_KEY=
# An Anthropic key, for a run of the Nullius standard whose agent speaks Anthropic's Messages, and for the LLM judge: the calls go to Anthropic's own base with this key. Only the relay and the judge hold it, never an agent, a recipe or a log.
ANTHROPIC_API_KEY=
# A TypeSafe key, for the System One judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.
# A TypeSafe key, for the Decision judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.
# e.g. ts-…
TYPESAFE_API_KEY=
# The provider the System One judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.
SYSTEM_ONE_PROVIDER=
# The decision protocol System One is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.
SYSTEM_ONE_PROTOCOL=
# The System One model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.
SYSTEM_ONE_MODEL=
# The provider of the LLM judgement, the second judgement beside System One: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom:<ID> (a custom provider), asked with that provider's own key and base from the model providers above. local and local:<ID> are read as custom. Unset: the first of them that is set, Anthropic last.
# The provider the Decision judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.
DECISION_PROVIDER=
# The decision protocol Decision is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.
DECISION_PROTOCOL=
# The Decision model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.
DECISION_MODEL=
# The provider of the LLM judgement, the second judgement beside Decision: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom:<ID> (a custom provider), asked with that provider's own key and base from the model providers above. local and local:<ID> are read as custom. Unset: the first of them that is set, Anthropic last.
LLM_JUDGE_PROVIDER=
# The protocol the LLM judge is asked in, on a provider that serves both: chat (OpenAI Chat Completions) or messages (Anthropic Messages). Unset, or one the provider does not serve: the first the provider serves, chat before messages.
LLM_JUDGE_PROTOCOL=
# The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as System One and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.
# The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as Decision and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.
LLM_JUDGE_MODEL=
# Additions to the OpenRouter provider, as a JSON object without the key: endpoints of protocols the preset does not list, {"endpoints": {"<protocol>": "/path" or "https://full/url"}}. In the environment, a partial object merged over this one.
OPENROUTER=
Expand Down
2 changes: 1 addition & 1 deletion app/nullius/judge/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""Judgement: structured, confidence-scored answers about a recorded trial
from a judge model (TypeSafe.ai System One), shown beside the deterministic
from a judge model (TypeSafe.ai Decision), shown beside the deterministic
evidence and never in its place.

The contract: the question
Expand Down
2 changes: 1 addition & 1 deletion app/nullius/judge/batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ def ask_and_store(ws_dir: str, job_id: str, trial_name: str, states: dict, key,
rec = ask_states(states, key, post=post, calibration=calibration)
if rec is not None:
# a call over some of the kinds keeps the earlier readings for the others, as the answers are kept
lane = getattr(key, "lane", "system-one") # the LLM judge's records apart from System One's
lane = getattr(key, "lane", "decision") # the LLM judge's records apart from Decision's
earlier = store.load(ws_dir, job_id, trial_name, version=rec["questions"], lane=lane, started=started) or {}
rec["hygiene"] = {**(earlier.get("hygiene") or {}), **rec.get("hygiene", {})}
store.save(ws_dir, job_id, trial_name, rec, lane=lane, started=started)
Expand Down
80 changes: 42 additions & 38 deletions app/nullius/judge/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
one Settings chose); each asks a provider of the registry (``nullius.providers``)
that serves its protocol:

- **System One**, a decision model on a decision protocol: ``systemone``
- **Decision**, a decision model on a decision protocol: ``systemone``
(TypeSafe's Jev, on TypeSafe or OpenRouter), ``decisions`` (OpenRouter's,
for many makers' models) or ``openai-decisions``. Its answers are the
model's own: a Noul's probability, a Choice's distribution over its
Expand Down Expand Up @@ -43,37 +43,42 @@
CALIBRATION_MIN_CHARS = 4_000 # a stored call this small says nothing about a large request's ratio: it is kept out of the band


LANES = ("system-one", "llm") # the two judgements a trial can have, each with its own records
LANES = ("decision", "llm") # the two judgements a trial can have, each with its own records
# The protocols each judge speaks, in its order of preference; the providers come from the registry.
LLM_PROTOCOLS = ("chat", "messages")
SYSTEM_ONE_PROTOCOLS = ("systemone", "decisions", "openai-decisions")
# System One's models as each provider names them: the first is its default. Another is typed in Settings.
SYSTEM_ONE_MODELS = {"typesafe": [MODEL],
DECISION_PROTOCOLS = ("systemone", "decisions", "openai-decisions")
# Decision's models as each provider names them: the first is its default. Another is typed in Settings.
DECISION_MODELS = {"typesafe": [MODEL],
"openrouter": ["~typesafe/jev-latest", "typesafe/jev-1.14", "typesafe/jev-1.13", "typesafe/jev-router", "jaredpalmer/kev-4b"],
"openai": ["gpt-6-luna"]}


def system_one_models(choice: str, fetch: bool = False) -> list:
"""System One's models on a provider, its default first: on OpenRouter, its decision models
def decision_judge_models(choice: str, fetch: bool = False) -> list:
"""Decision's models on a provider, its default first: on OpenRouter, its decision models
(``providers.decision_models``, cached; asked only when ``fetch``), TypeSafe's next; else, or with
none, ``SYSTEM_ONE_MODELS``."""
listed = SYSTEM_ONE_MODELS.get(choice, [])
none, ``DECISION_MODELS``."""
listed = DECISION_MODELS.get(choice, [])
live = providers.decision_models(fetch=fetch) if choice == "openrouter" else []
if not live:
return list(listed)
return [listed[0]] + sorted(set(live) - {listed[0]}, key=lambda m: ("typesafe/" not in m, m))


def system_one_protocol(choice: str, model: str, env=os.environ) -> str:
"""The decision protocol System One speaks on a provider: ``SYSTEM_ONE_PROTOCOL`` when the provider
def _setting(env, suffix: str) -> str:
"""``DECISION_<suffix>``, else its name before the rename (``settings.RENAMED``), stripped."""
return (env.get(f"DECISION_{suffix}") or env.get(f"SYSTEM_ONE_{suffix}") or "").strip()


def decision_judge_protocol(choice: str, model: str, env=os.environ) -> str:
"""The decision protocol Decision speaks on a provider: ``DECISION_PROTOCOL`` when the provider
lists it; else, on one that serves both ``systemone`` and ``decisions`` (OpenRouter), ``systemone``
for TypeSafe's models and the ones listed for it, and ``decisions`` for another maker's; else the
first the provider lists."""
rec = providers.record(choice, env)
listed = [p for p in SYSTEM_ONE_PROTOCOLS if providers.serves(rec, p)]
if (env.get("SYSTEM_ONE_PROTOCOL") or "").strip() in listed:
return env["SYSTEM_ONE_PROTOCOL"].strip()
typesafe = "typesafe/" in model or model.startswith("jev") or model in SYSTEM_ONE_MODELS.get(choice, [])
listed = [p for p in DECISION_PROTOCOLS if providers.serves(rec, p)]
if _setting(env, "PROTOCOL") in listed:
return _setting(env, "PROTOCOL")
typesafe = "typesafe/" in model or model.startswith("jev") or model in DECISION_MODELS.get(choice, [])
if "systemone" in listed and "decisions" in listed:
return "systemone" if typesafe else "decisions"
return listed[0] if listed else "systemone"
Expand Down Expand Up @@ -107,11 +112,11 @@ class Judge:

def __init__(self, provider: str, key: str, model: str, url: str = "", lane: str = "", name: str = "", protocol: str = ""):
self.provider, self.key, self.model, self.url, self.name = provider, key, model, url, name
self.lane = lane or ("system-one" if provider == "typesafe" else "llm")
self.protocol = protocol or ("systemone" if self.lane == "system-one" else "chat")
self.lane = lane or ("decision" if provider == "typesafe" else "llm")
self.protocol = protocol or ("systemone" if self.lane == "decision" else "chat")

def __bool__(self) -> bool:
if self.lane == "system-one":
if self.lane == "decision":
return bool(self.key) and bool(self.model)
return bool(self.model) and bool(self.url) and (bool(self.key) or self.provider.startswith("custom"))

Expand All @@ -129,9 +134,8 @@ def label(self) -> str:
@property
def missing(self) -> str:
"""What Settings still needs for this judge, in a phrase; '' when it can be asked."""
if self.lane == "system-one":
return " and ".join(([f"the {self.who} key"] if not self.key else [])
+ (["a model"] if not self.model else []))
if self.lane == "decision":
return "" if self.key and self.model else "a model"
need = []
if not self.url:
need.append("an endpoint (MODEL_API_URL)")
Expand All @@ -142,31 +146,31 @@ def missing(self) -> str:
return " and ".join(need)


def system_one_providers(env=os.environ) -> dict:
"""System One's providers: every one that lists one of its protocols, each with its models
(``system_one_models``, never asked here; the first, ``model``, its default; ``live`` when the
def decision_judge_providers(env=os.environ) -> dict:
"""Decision's providers: every one that lists one of its protocols, each with its models
(``decision_judge_models``, never asked here; the first, ``model``, its default; ``live`` when the
provider has a list to ask) and the protocol it would be asked in."""
out = {k: dict(d, models=system_one_models(k), model=(SYSTEM_ONE_MODELS.get(k) or [""])[0], live=k == "openrouter")
for k, d in _table(SYSTEM_ONE_PROTOCOLS, env, known=True).items()}
out = {k: dict(d, models=decision_judge_models(k), model=(DECISION_MODELS.get(k) or [""])[0], live=k == "openrouter")
for k, d in _table(DECISION_PROTOCOLS, env, known=True).items()}
for k, d in out.items():
d["protocol"] = system_one_protocol(k, (env.get("SYSTEM_ONE_MODEL") or "").strip() or d["model"], env)
d["protocol"] = decision_judge_protocol(k, _setting(env, "MODEL") or d["model"], env)
return out


def system_one(env=os.environ) -> Judge:
"""The System One judgement: on ``SYSTEM_ONE_PROVIDER`` (unset, or one
that serves no System One protocol: TypeSafe when its key is set, else
def decision_judge(env=os.environ) -> Judge:
"""The Decision judgement: on ``DECISION_PROVIDER`` (unset, or one
that serves no Decision protocol: TypeSafe when its key is set, else
OpenRouter when its is), with that provider's key, and the model
``SYSTEM_ONE_MODEL`` names or the provider's default."""
table = system_one_providers(env)
chosen = _choice(env.get("SYSTEM_ONE_PROVIDER"))
``DECISION_MODEL`` names or the provider's default."""
table = decision_judge_providers(env)
chosen = _choice(_setting(env, "PROVIDER"))
if chosen not in table:
chosen = "openrouter" if not env.get("TYPESAFE_API_KEY") and env.get("OPENROUTER_API_KEY") else "typesafe"
d = table[chosen]
model = (env.get("SYSTEM_ONE_MODEL") or "").strip() or d["model"]
protocol = system_one_protocol(chosen, model, env)
model = _setting(env, "MODEL") or d["model"]
protocol = decision_judge_protocol(chosen, model, env)
return Judge(chosen, env.get(d["key"], ""), model, providers.url(providers.record(chosen, env), protocol),
lane="system-one", name=d["label"], protocol=protocol)
lane="decision", name=d["label"], protocol=protocol)


def llm_providers(env=os.environ) -> dict:
Expand Down Expand Up @@ -203,7 +207,7 @@ def settings_placeholder() -> str:


def judge_for(lane: str, env=os.environ) -> Judge:
return llm_judge(env) if lane == "llm" else system_one(env)
return llm_judge(env) if lane == "llm" else decision_judge(env)


class JudgeError(Exception):
Expand Down Expand Up @@ -426,7 +430,7 @@ def ask(state: str, ids, key: str, truncated: bool = False, post: Optional[Calla
key = key.key
if not key:
raise JudgeError(401, "no TypeSafe.ai key: set TYPESAFE_API_KEY in Settings, Judge" if provider == PROVIDER
else f"no {who} key for System One: set it in Settings, Model providers")
else f"no {who} key for Decision: set it in Settings, Model providers")
asked = questions if questions is not None else Q.request_questions(ids)
openai = protocol == "openai-decisions"
body = {"model": model, "input": state, "questions": openai_questions(asked)} if openai else {"state": state, "model": model, "questions": asked}
Expand Down
12 changes: 6 additions & 6 deletions app/nullius/judge/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@
_SAFE = re.compile(r"[^A-Za-z0-9._@+-]")


ROOTS = {"system-one": "judgements", "llm": "judgements-llm"} # one directory per judgement
ROOTS = {"decision": "judgements", "llm": "judgements-llm"} # one directory per judgement


def _root(ws_dir: str, lane: str) -> str:
return os.path.join(ws_dir, ROOTS.get(lane, "judgements"))


def _path(ws_dir: str, job_id: str, trial_name: str, lane: str = "system-one") -> str:
def _path(ws_dir: str, job_id: str, trial_name: str, lane: str = "decision") -> str:
return os.path.join(_root(ws_dir, lane), _SAFE.sub("_", job_id), _SAFE.sub("_", trial_name) + ".json")


Expand All @@ -52,7 +52,7 @@ def _pick(data: dict, version: Optional[str], started: Optional[str] = None) ->
return None


def load(ws_dir: str, job_id: str, trial_name: str, version: Optional[str] = None, lane: str = "system-one",
def load(ws_dir: str, job_id: str, trial_name: str, version: Optional[str] = None, lane: str = "decision",
started: Optional[str] = None) -> Optional[dict]:
"""The record under `version`; with none named, the current version or
the newest earlier one that has a record. `started`: the trial's own
Expand All @@ -69,7 +69,7 @@ def is_current(record: Optional[dict]) -> bool:
return bool(record) and record.get("questions") == Q.VERSION


def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "system-one",
def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "decision",
started: Optional[str] = None) -> None:
"""The record under its question version; other versions kept. A
record's answers merge over the version's earlier answers, so judging
Expand All @@ -94,7 +94,7 @@ def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "s
os.replace(tmp, path)


def for_job(ws_dir: str, job_id: str, version: Optional[str] = None, lane: str = "system-one",
def for_job(ws_dir: str, job_id: str, version: Optional[str] = None, lane: str = "decision",
started: Optional[dict] = None) -> dict:
"""Every judged trial of a job: {trial_name: record}, each the current
version's record or the newest earlier one, as `load` picks it.
Expand Down Expand Up @@ -129,7 +129,7 @@ def count(ws_dir: str) -> int:
return sum(n.endswith(".json") for lane in ROOTS for _, _, names in os.walk(_root(ws_dir, lane)) for n in names)


def all_records(ws_dir: str, job_id: Optional[str] = None, lane: str = "system-one"):
def all_records(ws_dir: str, job_id: Optional[str] = None, lane: str = "decision"):
"""Every stored record, every version: yields (job_id, trial_name,
version, record). One job's when `job_id` is given."""
root = _root(ws_dir, lane)
Expand Down
4 changes: 2 additions & 2 deletions app/nullius/onboarding.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
{"id": "where", "text": "Choose where runs happen", "fact": "where",
"target": [["#hosts", "Add a host, or run on this machine with Docker or Podman"]]},
{"id": "judge", "text": "Pick a judge", "fact": "judge", "optional": True,
"target": [["#judge", "Choose System One or an LLM judge"]]},
"target": [["#judge", "Choose Decision or an LLM judge"]]},
]},
{"id": "bench", "name": "Pick a benchmark", "page": "benchmarks_page", "where": "Benchmarks", "subs": [
{"id": "browse", "text": "Browse the catalogue", "visit": r"^/benchmarks/?$",
Expand Down Expand Up @@ -86,7 +86,7 @@ def facts(env, jobs: Iterable[dict] = (), judged: bool = False, docker: Optional
return {
"provider": bool(providers.available(env)), # a run provider: a TypeSafe key serves the judge only
"where": bool((env.get("HOSTS") or "").strip()) or found,
"judge": bool(client.system_one(env)) or bool(client.llm_judge(env)),
"judge": bool(client.decision_judge(env)) or bool(client.llm_judge(env)),
"control_run": any(a in CONTROLS for a in agents),
"agent_run": any(a and a not in CONTROLS for a in agents),
"judged": bool(judged),
Expand Down
Loading
Loading