diff --git a/README.md b/README.md
index 0c587b99..77440b41 100644
--- a/README.md
+++ b/README.md
@@ -143,7 +143,7 @@ site](app/site/README.md).
| [Judgement](docs/judgement.md) | What a judge model is asked |
| [The Nullius standard](docs/standard.md) | Ports, kits, setups, controls, proofs; importing |
| [Architecture](docs/how-it-works.md) | Execution paths, a run on disk, the pages |
-| [Settings](docs/settings/README.md) | Providers, System One, LLM judge, hosts, keys |
+| [Settings](docs/settings/README.md) | Providers, Decision, LLM judge, hosts, keys |
| [Benchmarks](docs/benchmarks/supported/README.md), [Agents](docs/agents/supported/README.md), [Capabilities](docs/capabilities/supported/README.md) | What is supported; adding one |
| [CLI](docs/cli/README.md) | Every command, by topic |
| [Metric field guide](docs/field-guide.md) | Every metric and outcome |
diff --git a/app/.env.example b/app/.env.example
index 4fb3b953..feda3fbc 100644
--- a/app/.env.example
+++ b/app/.env.example
@@ -33,20 +33,20 @@ OPENAI_API_KEY=
OPENROUTER_API_KEY=
# An Anthropic key, for a run of the Nullius standard whose agent speaks Anthropic's Messages, and for the LLM judge: the calls go to Anthropic's own base with this key. Only the relay and the judge hold it, never an agent, a recipe or a log.
ANTHROPIC_API_KEY=
-# A TypeSafe key, for the System One judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.
+# A TypeSafe key, for the Decision judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.
# e.g. ts-…
TYPESAFE_API_KEY=
-# The provider the System One judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.
-SYSTEM_ONE_PROVIDER=
-# The decision protocol System One is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.
-SYSTEM_ONE_PROTOCOL=
-# The System One model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.
-SYSTEM_ONE_MODEL=
-# The provider of the LLM judgement, the second judgement beside System One: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.
+# The provider the Decision judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.
+DECISION_PROVIDER=
+# The decision protocol Decision is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.
+DECISION_PROTOCOL=
+# The Decision model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.
+DECISION_MODEL=
+# The provider of the LLM judgement, the second judgement beside Decision: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.
LLM_JUDGE_PROVIDER=
# The protocol the LLM judge is asked in, on a provider that serves both: chat (OpenAI Chat Completions) or messages (Anthropic Messages). Unset, or one the provider does not serve: the first the provider serves, chat before messages.
LLM_JUDGE_PROTOCOL=
-# The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as System One and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.
+# The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as Decision and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.
LLM_JUDGE_MODEL=
# Additions to the OpenRouter provider, as a JSON object without the key: endpoints of protocols the preset does not list, {"endpoints": {"": "/path" or "https://full/url"}}. In the environment, a partial object merged over this one.
OPENROUTER=
diff --git a/app/nullius/judge/__init__.py b/app/nullius/judge/__init__.py
index 8299ed5f..0ec9c22d 100644
--- a/app/nullius/judge/__init__.py
+++ b/app/nullius/judge/__init__.py
@@ -1,5 +1,5 @@
"""Judgement: structured, confidence-scored answers about a recorded trial
-from a judge model (TypeSafe.ai System One), shown beside the deterministic
+from a judge model (TypeSafe.ai Decision), shown beside the deterministic
evidence and never in its place.
The contract: the question
diff --git a/app/nullius/judge/batch.py b/app/nullius/judge/batch.py
index 4b207638..6410e3be 100644
--- a/app/nullius/judge/batch.py
+++ b/app/nullius/judge/batch.py
@@ -116,7 +116,7 @@ def ask_and_store(ws_dir: str, job_id: str, trial_name: str, states: dict, key,
rec = ask_states(states, key, post=post, calibration=calibration)
if rec is not None:
# a call over some of the kinds keeps the earlier readings for the others, as the answers are kept
- lane = getattr(key, "lane", "system-one") # the LLM judge's records apart from System One's
+ lane = getattr(key, "lane", "decision") # the LLM judge's records apart from Decision's
earlier = store.load(ws_dir, job_id, trial_name, version=rec["questions"], lane=lane, started=started) or {}
rec["hygiene"] = {**(earlier.get("hygiene") or {}), **rec.get("hygiene", {})}
store.save(ws_dir, job_id, trial_name, rec, lane=lane, started=started)
diff --git a/app/nullius/judge/client.py b/app/nullius/judge/client.py
index 7208038e..496b4d6e 100644
--- a/app/nullius/judge/client.py
+++ b/app/nullius/judge/client.py
@@ -3,7 +3,7 @@
one Settings chose); each asks a provider of the registry (``nullius.providers``)
that serves its protocol:
-- **System One**, a decision model on a decision protocol: ``systemone``
+- **Decision**, a decision model on a decision protocol: ``systemone``
(TypeSafe's Jev, on TypeSafe or OpenRouter), ``decisions`` (OpenRouter's,
for many makers' models) or ``openai-decisions``. Its answers are the
model's own: a Noul's probability, a Choice's distribution over its
@@ -43,37 +43,42 @@
CALIBRATION_MIN_CHARS = 4_000 # a stored call this small says nothing about a large request's ratio: it is kept out of the band
-LANES = ("system-one", "llm") # the two judgements a trial can have, each with its own records
+LANES = ("decision", "llm") # the two judgements a trial can have, each with its own records
# The protocols each judge speaks, in its order of preference; the providers come from the registry.
LLM_PROTOCOLS = ("chat", "messages")
-SYSTEM_ONE_PROTOCOLS = ("systemone", "decisions", "openai-decisions")
-# System One's models as each provider names them: the first is its default. Another is typed in Settings.
-SYSTEM_ONE_MODELS = {"typesafe": [MODEL],
+DECISION_PROTOCOLS = ("systemone", "decisions", "openai-decisions")
+# Decision's models as each provider names them: the first is its default. Another is typed in Settings.
+DECISION_MODELS = {"typesafe": [MODEL],
"openrouter": ["~typesafe/jev-latest", "typesafe/jev-1.14", "typesafe/jev-1.13", "typesafe/jev-router", "jaredpalmer/kev-4b"],
"openai": ["gpt-6-luna"]}
-def system_one_models(choice: str, fetch: bool = False) -> list:
- """System One's models on a provider, its default first: on OpenRouter, its decision models
+def decision_judge_models(choice: str, fetch: bool = False) -> list:
+ """Decision's models on a provider, its default first: on OpenRouter, its decision models
(``providers.decision_models``, cached; asked only when ``fetch``), TypeSafe's next; else, or with
- none, ``SYSTEM_ONE_MODELS``."""
- listed = SYSTEM_ONE_MODELS.get(choice, [])
+ none, ``DECISION_MODELS``."""
+ listed = DECISION_MODELS.get(choice, [])
live = providers.decision_models(fetch=fetch) if choice == "openrouter" else []
if not live:
return list(listed)
return [listed[0]] + sorted(set(live) - {listed[0]}, key=lambda m: ("typesafe/" not in m, m))
-def system_one_protocol(choice: str, model: str, env=os.environ) -> str:
- """The decision protocol System One speaks on a provider: ``SYSTEM_ONE_PROTOCOL`` when the provider
+def _setting(env, suffix: str) -> str:
+ """``DECISION_``, else its name before the rename (``settings.RENAMED``), stripped."""
+ return (env.get(f"DECISION_{suffix}") or env.get(f"SYSTEM_ONE_{suffix}") or "").strip()
+
+
+def decision_judge_protocol(choice: str, model: str, env=os.environ) -> str:
+ """The decision protocol Decision speaks on a provider: ``DECISION_PROTOCOL`` when the provider
lists it; else, on one that serves both ``systemone`` and ``decisions`` (OpenRouter), ``systemone``
for TypeSafe's models and the ones listed for it, and ``decisions`` for another maker's; else the
first the provider lists."""
rec = providers.record(choice, env)
- listed = [p for p in SYSTEM_ONE_PROTOCOLS if providers.serves(rec, p)]
- if (env.get("SYSTEM_ONE_PROTOCOL") or "").strip() in listed:
- return env["SYSTEM_ONE_PROTOCOL"].strip()
- typesafe = "typesafe/" in model or model.startswith("jev") or model in SYSTEM_ONE_MODELS.get(choice, [])
+ listed = [p for p in DECISION_PROTOCOLS if providers.serves(rec, p)]
+ if _setting(env, "PROTOCOL") in listed:
+ return _setting(env, "PROTOCOL")
+ typesafe = "typesafe/" in model or model.startswith("jev") or model in DECISION_MODELS.get(choice, [])
if "systemone" in listed and "decisions" in listed:
return "systemone" if typesafe else "decisions"
return listed[0] if listed else "systemone"
@@ -107,11 +112,11 @@ class Judge:
def __init__(self, provider: str, key: str, model: str, url: str = "", lane: str = "", name: str = "", protocol: str = ""):
self.provider, self.key, self.model, self.url, self.name = provider, key, model, url, name
- self.lane = lane or ("system-one" if provider == "typesafe" else "llm")
- self.protocol = protocol or ("systemone" if self.lane == "system-one" else "chat")
+ self.lane = lane or ("decision" if provider == "typesafe" else "llm")
+ self.protocol = protocol or ("systemone" if self.lane == "decision" else "chat")
def __bool__(self) -> bool:
- if self.lane == "system-one":
+ if self.lane == "decision":
return bool(self.key) and bool(self.model)
return bool(self.model) and bool(self.url) and (bool(self.key) or self.provider.startswith("custom"))
@@ -129,9 +134,8 @@ def label(self) -> str:
@property
def missing(self) -> str:
"""What Settings still needs for this judge, in a phrase; '' when it can be asked."""
- if self.lane == "system-one":
- return " and ".join(([f"the {self.who} key"] if not self.key else [])
- + (["a model"] if not self.model else []))
+ if self.lane == "decision":
+ return "" if self.key and self.model else "a model"
need = []
if not self.url:
need.append("an endpoint (MODEL_API_URL)")
@@ -142,31 +146,31 @@ def missing(self) -> str:
return " and ".join(need)
-def system_one_providers(env=os.environ) -> dict:
- """System One's providers: every one that lists one of its protocols, each with its models
- (``system_one_models``, never asked here; the first, ``model``, its default; ``live`` when the
+def decision_judge_providers(env=os.environ) -> dict:
+ """Decision's providers: every one that lists one of its protocols, each with its models
+ (``decision_judge_models``, never asked here; the first, ``model``, its default; ``live`` when the
provider has a list to ask) and the protocol it would be asked in."""
- out = {k: dict(d, models=system_one_models(k), model=(SYSTEM_ONE_MODELS.get(k) or [""])[0], live=k == "openrouter")
- for k, d in _table(SYSTEM_ONE_PROTOCOLS, env, known=True).items()}
+ out = {k: dict(d, models=decision_judge_models(k), model=(DECISION_MODELS.get(k) or [""])[0], live=k == "openrouter")
+ for k, d in _table(DECISION_PROTOCOLS, env, known=True).items()}
for k, d in out.items():
- d["protocol"] = system_one_protocol(k, (env.get("SYSTEM_ONE_MODEL") or "").strip() or d["model"], env)
+ d["protocol"] = decision_judge_protocol(k, _setting(env, "MODEL") or d["model"], env)
return out
-def system_one(env=os.environ) -> Judge:
- """The System One judgement: on ``SYSTEM_ONE_PROVIDER`` (unset, or one
- that serves no System One protocol: TypeSafe when its key is set, else
+def decision_judge(env=os.environ) -> Judge:
+ """The Decision judgement: on ``DECISION_PROVIDER`` (unset, or one
+ that serves no Decision protocol: TypeSafe when its key is set, else
OpenRouter when its is), with that provider's key, and the model
- ``SYSTEM_ONE_MODEL`` names or the provider's default."""
- table = system_one_providers(env)
- chosen = _choice(env.get("SYSTEM_ONE_PROVIDER"))
+ ``DECISION_MODEL`` names or the provider's default."""
+ table = decision_judge_providers(env)
+ chosen = _choice(_setting(env, "PROVIDER"))
if chosen not in table:
chosen = "openrouter" if not env.get("TYPESAFE_API_KEY") and env.get("OPENROUTER_API_KEY") else "typesafe"
d = table[chosen]
- model = (env.get("SYSTEM_ONE_MODEL") or "").strip() or d["model"]
- protocol = system_one_protocol(chosen, model, env)
+ model = _setting(env, "MODEL") or d["model"]
+ protocol = decision_judge_protocol(chosen, model, env)
return Judge(chosen, env.get(d["key"], ""), model, providers.url(providers.record(chosen, env), protocol),
- lane="system-one", name=d["label"], protocol=protocol)
+ lane="decision", name=d["label"], protocol=protocol)
def llm_providers(env=os.environ) -> dict:
@@ -203,7 +207,7 @@ def settings_placeholder() -> str:
def judge_for(lane: str, env=os.environ) -> Judge:
- return llm_judge(env) if lane == "llm" else system_one(env)
+ return llm_judge(env) if lane == "llm" else decision_judge(env)
class JudgeError(Exception):
@@ -426,7 +430,7 @@ def ask(state: str, ids, key: str, truncated: bool = False, post: Optional[Calla
key = key.key
if not key:
raise JudgeError(401, "no TypeSafe.ai key: set TYPESAFE_API_KEY in Settings, Judge" if provider == PROVIDER
- else f"no {who} key for System One: set it in Settings, Model providers")
+ else f"no {who} key for Decision: set it in Settings, Model providers")
asked = questions if questions is not None else Q.request_questions(ids)
openai = protocol == "openai-decisions"
body = {"model": model, "input": state, "questions": openai_questions(asked)} if openai else {"state": state, "model": model, "questions": asked}
diff --git a/app/nullius/judge/store.py b/app/nullius/judge/store.py
index 93e012b5..fe35cb87 100644
--- a/app/nullius/judge/store.py
+++ b/app/nullius/judge/store.py
@@ -21,14 +21,14 @@
_SAFE = re.compile(r"[^A-Za-z0-9._@+-]")
-ROOTS = {"system-one": "judgements", "llm": "judgements-llm"} # one directory per judgement
+ROOTS = {"decision": "judgements", "llm": "judgements-llm"} # one directory per judgement
def _root(ws_dir: str, lane: str) -> str:
return os.path.join(ws_dir, ROOTS.get(lane, "judgements"))
-def _path(ws_dir: str, job_id: str, trial_name: str, lane: str = "system-one") -> str:
+def _path(ws_dir: str, job_id: str, trial_name: str, lane: str = "decision") -> str:
return os.path.join(_root(ws_dir, lane), _SAFE.sub("_", job_id), _SAFE.sub("_", trial_name) + ".json")
@@ -52,7 +52,7 @@ def _pick(data: dict, version: Optional[str], started: Optional[str] = None) ->
return None
-def load(ws_dir: str, job_id: str, trial_name: str, version: Optional[str] = None, lane: str = "system-one",
+def load(ws_dir: str, job_id: str, trial_name: str, version: Optional[str] = None, lane: str = "decision",
started: Optional[str] = None) -> Optional[dict]:
"""The record under `version`; with none named, the current version or
the newest earlier one that has a record. `started`: the trial's own
@@ -69,7 +69,7 @@ def is_current(record: Optional[dict]) -> bool:
return bool(record) and record.get("questions") == Q.VERSION
-def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "system-one",
+def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "decision",
started: Optional[str] = None) -> None:
"""The record under its question version; other versions kept. A
record's answers merge over the version's earlier answers, so judging
@@ -94,7 +94,7 @@ def save(ws_dir: str, job_id: str, trial_name: str, record: dict, lane: str = "s
os.replace(tmp, path)
-def for_job(ws_dir: str, job_id: str, version: Optional[str] = None, lane: str = "system-one",
+def for_job(ws_dir: str, job_id: str, version: Optional[str] = None, lane: str = "decision",
started: Optional[dict] = None) -> dict:
"""Every judged trial of a job: {trial_name: record}, each the current
version's record or the newest earlier one, as `load` picks it.
@@ -129,7 +129,7 @@ def count(ws_dir: str) -> int:
return sum(n.endswith(".json") for lane in ROOTS for _, _, names in os.walk(_root(ws_dir, lane)) for n in names)
-def all_records(ws_dir: str, job_id: Optional[str] = None, lane: str = "system-one"):
+def all_records(ws_dir: str, job_id: Optional[str] = None, lane: str = "decision"):
"""Every stored record, every version: yields (job_id, trial_name,
version, record). One job's when `job_id` is given."""
root = _root(ws_dir, lane)
diff --git a/app/nullius/onboarding.py b/app/nullius/onboarding.py
index 92510219..1b4047c1 100644
--- a/app/nullius/onboarding.py
+++ b/app/nullius/onboarding.py
@@ -30,7 +30,7 @@
{"id": "where", "text": "Choose where runs happen", "fact": "where",
"target": [["#hosts", "Add a host, or run on this machine with Docker or Podman"]]},
{"id": "judge", "text": "Pick a judge", "fact": "judge", "optional": True,
- "target": [["#judge", "Choose System One or an LLM judge"]]},
+ "target": [["#judge", "Choose Decision or an LLM judge"]]},
]},
{"id": "bench", "name": "Pick a benchmark", "page": "benchmarks_page", "where": "Benchmarks", "subs": [
{"id": "browse", "text": "Browse the catalogue", "visit": r"^/benchmarks/?$",
@@ -86,7 +86,7 @@ def facts(env, jobs: Iterable[dict] = (), judged: bool = False, docker: Optional
return {
"provider": bool(providers.available(env)), # a run provider: a TypeSafe key serves the judge only
"where": bool((env.get("HOSTS") or "").strip()) or found,
- "judge": bool(client.system_one(env)) or bool(client.llm_judge(env)),
+ "judge": bool(client.decision_judge(env)) or bool(client.llm_judge(env)),
"control_run": any(a in CONTROLS for a in agents),
"agent_run": any(a and a not in CONTROLS for a in agents),
"judged": bool(judged),
diff --git a/app/nullius/providers.py b/app/nullius/providers.py
index f2f80427..4e05b26b 100644
--- a/app/nullius/providers.py
+++ b/app/nullius/providers.py
@@ -1,7 +1,7 @@
"""Where model calls go: the one definition of each model provider, and the builder's steps.
Every feature reads the registry below (``record``, ``records``, ``serves``) and asks for the
-protocol it speaks: a run what its agent speaks, the LLM judge ``chat`` or ``messages``, System One
+protocol it speaks: a run what its agent speaks, the LLM judge ``chat`` or ``messages``, Decision
a decision protocol. A provider is a preset (``PRESETS``) or a custom provider, ``custom``: the
OpenAI-compatible endpoint at MODEL_API_URL, and one per id in Settings (``settings.endpoint_ids``).
A recipe names one by ``provider`` (``PROVIDERS``, the ones a run is sent to) and, for a custom one,
@@ -38,7 +38,7 @@
"chat": {"label": "OpenAI Chat Completions", "path": "/chat/completions", "auth": "authorization"},
"responses": {"label": "OpenAI Responses", "path": "/responses", "auth": "authorization"},
"messages": {"label": "Anthropic Messages", "path": "/messages", "auth": "x-api-key"},
- "systemone": {"label": "TypeSafe System One", "path": "/systemone", "auth": "authorization"},
+ "systemone": {"label": "TypeSafe Decision", "path": "/systemone", "auth": "authorization"},
"decisions": {"label": "OpenRouter Decisions", "path": "/decisions", "auth": "authorization"},
"openai-decisions": {"label": "OpenAI Decisions", "path": "/decisions", "auth": "authorization"},
}
diff --git a/app/nullius/settings.py b/app/nullius/settings.py
index f241a09a..90bea904 100644
--- a/app/nullius/settings.py
+++ b/app/nullius/settings.py
@@ -41,13 +41,13 @@
_TEXT_OK = re.compile(r"^[^\x00-\x1f]{1,400}$")
_MODEL_OK = re.compile(r"^~?[A-Za-z0-9][A-Za-z0-9._:/@+-]{0,159}$") # an OpenRouter model id: [~]vendor/model[:variant]
# a provider choice each judge can use: a preset that serves its protocols, or custom (local, its old name);
-# System One's custom provider is a named one, the only kind whose record lists a decision protocol
+# Decision's custom provider is a named one, the only kind whose record lists a decision protocol
_ID = r"[A-Z0-9]+(_[A-Z0-9]+)*"
-_S1_OK = re.compile(rf"^(typesafe|openrouter|openai|(custom|local):{_ID})$")
+_DECISION_PROVIDER_OK = re.compile(rf"^(typesafe|openrouter|openai|(custom|local):{_ID})$")
_LLM_JUDGE_OK = re.compile(rf"^(openrouter|openai|anthropic|(custom|local)(:{_ID})?)$")
-# the protocols each judge speaks (``judge.client``: LLM_PROTOCOLS, SYSTEM_ONE_PROTOCOLS)
+# the protocols each judge speaks (``judge.client``: LLM_PROTOCOLS, DECISION_PROTOCOLS)
_LLM_PROTOCOL_OK = re.compile(r"^(chat|messages)$")
-_S1_PROTOCOL_OK = re.compile(r"^(systemone|decisions|openai-decisions)$")
+_DECISION_PROTOCOL_OK = re.compile(r"^(systemone|decisions|openai-decisions)$")
_ENVNAME_OK = re.compile(r"^[A-Z_][A-Z0-9_]*$")
_BOOL = {"0", "1", "true", "false", "yes", "no"}
@@ -149,7 +149,7 @@ class Variable:
name: str
group: str
purpose: str
- kind: str # secret | localkey | endpoint | mirror | provider | path | paths | url | repo | name | bool | text | envname | hosts | constraints | model | llmprovider | s1provider | llmprotocol | s1protocol | envnames
+ kind: str # secret | localkey | endpoint | mirror | provider | path | paths | url | repo | name | bool | text | envname | hosts | constraints | model | llmprovider | decisionprovider | llmprotocol | decisionprotocol | envnames
example: str = "" # shown as a placeholder, written to .env.example as a comment
@property
@@ -182,9 +182,9 @@ def validate(self, value: str) -> Optional[str]:
"constraints": _constraints,
"model": _check(_MODEL_OK, "a model id as OpenRouter lists it, vendor/model"),
"llmprovider": _check(_LLM_JUDGE_OK, "a provider: openrouter, openai, anthropic, custom or custom:"),
- "s1provider": _check(_S1_OK, "a provider: typesafe, openrouter, openai or custom:"),
+ "decisionprovider": _check(_DECISION_PROVIDER_OK, "a provider: typesafe, openrouter, openai or custom:"),
"llmprotocol": _check(_LLM_PROTOCOL_OK, "a protocol: chat or messages"),
- "s1protocol": _check(_S1_PROTOCOL_OK, "a protocol: systemone, decisions or openai-decisions"),
+ "decisionprotocol": _check(_DECISION_PROTOCOL_OK, "a protocol: systemone, decisions or openai-decisions"),
"envnames": _agent_key_names,
}[self.kind](value)
@@ -213,13 +213,13 @@ def validate(self, value: str) -> Optional[str]:
Variable("OPENAI_API_KEY", GROUPS[0], "An OpenAI key, for a recipe whose provider is OpenAI: that run is sent to OpenAI's own base with this key, never with the model API key or its endpoint. It reaches the run as the model key does, and never a recipe or a log.", "secret"),
Variable("OPENROUTER_API_KEY", GROUPS[0], "An OpenRouter key, for a recipe whose provider is OpenRouter: that run is sent to OpenRouter's own base with this key, never with the model API key or its endpoint. It reaches the run as the model key does, and never a recipe or a log.", "secret"),
Variable("ANTHROPIC_API_KEY", GROUPS[0], "An Anthropic key, for a run of the Nullius standard whose agent speaks Anthropic's Messages, and for the LLM judge: the calls go to Anthropic's own base with this key. Only the relay and the judge hold it, never an agent, a recipe or a log.", "secret"),
- Variable("TYPESAFE_API_KEY", GROUPS[0], "A TypeSafe key, for the System One judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.", "secret", "ts-…"),
- Variable("SYSTEM_ONE_PROVIDER", GROUPS[0], "The provider the System One judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.", "s1provider"),
- Variable("SYSTEM_ONE_PROTOCOL", GROUPS[0], "The decision protocol System One is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.", "s1protocol"),
- Variable("SYSTEM_ONE_MODEL", GROUPS[0], "The System One model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.", "model"),
- Variable("LLM_JUDGE_PROVIDER", GROUPS[0], "The provider of the LLM judgement, the second judgement beside System One: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.", "llmprovider"),
+ Variable("TYPESAFE_API_KEY", GROUPS[0], "A TypeSafe key, for the Decision judgement on TypeSafe's own base: sent as a bearer header there and nowhere else.", "secret", "ts-…"),
+ Variable("DECISION_PROVIDER", GROUPS[0], "The provider the Decision judgement is asked on: one that serves a decision protocol, such as typesafe or openrouter, with that provider's own key. Unset: TypeSafe when its key is set, else OpenRouter when its key is.", "decisionprovider"),
+ Variable("DECISION_PROTOCOL", GROUPS[0], "The decision protocol Decision is asked in, on a provider that serves more than one: systemone, decisions or openai-decisions. Unset, or one the provider does not serve: on OpenRouter systemone for TypeSafe's models and decisions for another maker's, else the first the provider serves.", "decisionprotocol"),
+ Variable("DECISION_MODEL", GROUPS[0], "The Decision model, as the provider names it: on TypeSafe jev-latest, on OpenRouter ~typesafe/jev-latest or typesafe/jev-1.14. Unset: the provider's default.", "model"),
+ Variable("LLM_JUDGE_PROVIDER", GROUPS[0], "The provider of the LLM judgement, the second judgement beside Decision: openrouter, openai, anthropic, custom (the endpoint at MODEL_API_URL) or custom: (a custom provider), asked with that provider's own key and base from the model providers above. local and local: are read as custom. Unset: the first of them that is set, Anthropic last.", "llmprovider"),
Variable("LLM_JUDGE_PROTOCOL", GROUPS[0], "The protocol the LLM judge is asked in, on a provider that serves both: chat (OpenAI Chat Completions) or messages (Anthropic Messages). Unset, or one the provider does not serve: the first the provider serves, chat before messages.", "llmprotocol"),
- Variable("LLM_JUDGE_MODEL", GROUPS[0], "The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as System One and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.", "model"),
+ Variable("LLM_JUDGE_MODEL", GROUPS[0], "The model of the LLM judgement, as its provider names it (on OpenRouter, vendor/model). It is asked the same questions as Decision and answers them as JSON; its numbers are what it states, so a choice or a score carries no confidence.", "model"),
*(Variable(name, GROUPS[0], f"Additions to the {title} provider, as a JSON object without the key: endpoints of protocols the preset does not list, "
'{"endpoints": {"": "/path" or "https://full/url"}}. In the environment, a partial object merged over this one.', "provider")
for name, title in (("OPENROUTER", "OpenRouter"), ("OPENAI", "OpenAI"), ("ANTHROPIC", "Anthropic"), ("TYPESAFE", "TypeSafe"))),
@@ -260,7 +260,11 @@ def validate(self, value: str) -> Optional[str]:
# says which of its own names was meant - for HOST and PORT, a start-up
# option, read at start. OPENAI_API_KEY and OPENROUTER_API_KEY are its own:
# the keys a recipe on OpenAI or on OpenRouter is sent with.
-INSTEAD: Dict[str, str] = {"API_TOKEN": "MODEL_API_KEY", "API_URL": "MODEL_API_URL", "OPENAI_BASE_URL": "MODEL_API_URL",
+# The Decision judge's settings before it had that name: still read (``judge.client``), and taken out of
+# the file when the page saves the new one, so a saved choice survives the rename.
+RENAMED: Dict[str, str] = {"SYSTEM_ONE_PROVIDER": "DECISION_PROVIDER", "SYSTEM_ONE_PROTOCOL": "DECISION_PROTOCOL",
+ "SYSTEM_ONE_MODEL": "DECISION_MODEL"}
+INSTEAD: Dict[str, str] = {**RENAMED, "API_TOKEN": "MODEL_API_KEY", "API_URL": "MODEL_API_URL", "OPENAI_BASE_URL": "MODEL_API_URL",
"API_KEY_TYPESAFE": "TYPESAFE_API_KEY", "JUDGE_MODEL": "LLM_JUDGE_MODEL", "HOST": "APP_HOST", "PORT": "APP_PORT"}
@@ -283,7 +287,7 @@ def parse(path: str) -> List[Tuple[str, Optional[str], Optional[str]]]:
for k, v in values.items()
):
raise ValueError("Settings must be an object of variable names and single-line strings.")
- return [(f"{key}={value}", key, value) for key, value in values.items()]
+ return _renamed([(f"{key}={value}", key, value) for key, value in values.items()])
for raw in fh:
line = raw.rstrip("\n")
s = line.strip()
@@ -292,6 +296,20 @@ def parse(path: str) -> List[Tuple[str, Optional[str], Optional[str]]]:
continue
key, _, value = s.partition("=")
out.append((line, key.strip(), value.strip().strip('"').strip("'")))
+ return _renamed(out)
+
+
+def _renamed(entries: List[Tuple[str, Optional[str], Optional[str]]]) -> List[Tuple[str, Optional[str], Optional[str]]]:
+ """A setting written under its old name (``RENAMED``) read as its new one, its line too, so the page
+ shows it set and the next save writes the new name. A file that holds both keeps the new name's
+ value; the old line stays unread until a save of the new one takes it out."""
+ present = {k for _, k, _ in entries if k}
+ out = []
+ for line, key, value in entries:
+ new = RENAMED.get(key or "")
+ if new and new not in present:
+ line, key = line.replace(key, new, 1), new
+ out.append((line, key, value))
return out
@@ -641,8 +659,15 @@ def save_many(path: str, changes: Dict[str, Optional[str]]) -> None:
problems.append(f"{name} must be {why}.")
if problems:
raise ValueError(" ".join(problems))
+ # a setting saved under its new name takes its old one out of the file
+ old = {o for o, n in RENAMED.items() if n in changes and o not in changes}
lines, done = [], set()
for line, key, _ in parse(path):
+ if key in old:
+ if key in LOADED_KEYS:
+ os.environ.pop(key, None)
+ LOADED_KEYS.discard(key)
+ continue
if key in changes:
value = changes[key]
if value is not None and key not in done:
diff --git a/app/site/mkdocs.yml b/app/site/mkdocs.yml
index 82ac2df5..ce75b8cd 100644
--- a/app/site/mkdocs.yml
+++ b/app/site/mkdocs.yml
@@ -109,7 +109,7 @@ nav:
- Settings:
- Overview: settings/README.md
- Model providers: settings/model-providers.md
- - System One provider: settings/system-one.md
+ - Decision provider: settings/decision.md
- LLM judge provider: settings/llm-judge.md
- Custom keys: settings/agent-keys.md
- Model prices: settings/model-prices.md
diff --git a/app/static/settings/autosave.js b/app/static/settings/autosave.js
index 72c2809c..7cd43b8f 100644
--- a/app/static/settings/autosave.js
+++ b/app/static/settings/autosave.js
@@ -33,7 +33,7 @@
} else if (key.startsWith('value:')) names.add(key.slice(6));
}
// A model edit always carries the displayed provider, including a sole auto-selection.
- for (const prefix of ['SYSTEM_ONE', 'LLM_JUDGE']) {
+ for (const prefix of ['DECISION', 'LLM_JUDGE']) {
if (body.has('value:' + prefix + '_MODEL') || body.has('clear:' + prefix + '_MODEL')) {
const provider = current.get('value:' + prefix + '_PROVIDER');
if (provider) body.set('value:' + prefix + '_PROVIDER', provider);
diff --git a/app/static/settings/judges.js b/app/static/settings/judges.js
index 7f0864a6..c01f5b58 100644
--- a/app/static/settings/judges.js
+++ b/app/static/settings/judges.js
@@ -47,13 +47,13 @@
if (model) model.placeholder = on?.dataset.default || '';
}
// A provider with no list of its own here: its catalogue, asked once when the field is first used; a live list
- // (System One on OpenRouter) replaces the one drawn with the page. Typing a model still works.
+ // (Decision on OpenRouter) replaces the one drawn with the page. Typing a model still works.
const catalogues = new Map();
async function offerCatalogue(card, model) {
const on = chosen(card);
if (!on || (on.dataset.models && !('live' in on.dataset)) || !model.list) return;
const provider = on.value.replace(/^local(?=$|:)/, 'custom'); // local:, a named endpoint, is the builder's custom:
- const url = 'live' in on.dataset ? '/settings/system-one/models' : '/new-run/models';
+ const url = 'live' in on.dataset ? '/settings/decision/models' : '/new-run/models';
if (!catalogues.has(url + provider)) {
catalogues.set(url + provider, nulliusFetch(url, {method: 'POST', headers: {'Content-Type': 'application/json',
'X-CSRF-Token': document.querySelector('#settings-form [name=csrf]').value}, body: JSON.stringify({provider})})
diff --git a/app/templates/compare/compare.html b/app/templates/compare/compare.html
index ee3fe3fb..2f7a98ff 100644
--- a/app/templates/compare/compare.html
+++ b/app/templates/compare/compare.html
@@ -144,7 +144,7 @@
Task by task
- {% elif judgement.unjudged %}System One not configured — set its key in Settings{% endif %}
+ {% elif judgement.unjudged %}Decision model not configured: needs a model — choose one in Settings{% endif %}
{% endif %}
diff --git a/app/templates/runs/feasibility.html b/app/templates/runs/feasibility.html
index 1ba7cd28..af04743d 100644
--- a/app/templates/runs/feasibility.html
+++ b/app/templates/runs/feasibility.html
@@ -68,7 +68,7 @@
- {% elif triage.unjudged %}System One not configured — set its key in Settings{% endif %}
+ {% elif triage.unjudged %}Decision model not configured: needs a model — choose one in Settings{% endif %}
{% endif %}
diff --git a/app/templates/runs/job.html b/app/templates/runs/job.html
index 97a571b7..4a6128a0 100644
--- a/app/templates/runs/job.html
+++ b/app/templates/runs/job.html
@@ -298,7 +298,7 @@
{% if (judgement.unjudged) or (judgement.llm.unjudged) or (tq and tq.unjudged) %}
{# one button per judge: each judges apart, so a person asks for the one they want #}
- {% for lane, name, n, keyed in [('system-one', 'System One', judgement.unjudged, judgement.keyed), ('llm', 'LLM judge', judgement.llm.unjudged, judgement.llm.keyed)] if n %}
+ {% for lane, name, n, keyed in [('decision', 'Decision model', judgement.unjudged, judgement.keyed), ('llm', 'LLM judge', judgement.llm.unjudged, judgement.llm.keyed)] if n %}
{% if keyed %}
@@ -80,7 +80,7 @@
Settings
{% for b in benchmark_access %}{% set access_search.words = access_search.words ~ ' ' ~ b.title %}{% for s in b.needs %}{% set access_search.words = access_search.words ~ ' ' ~ s.name ~ ' ' ~ s.summary %}{% endfor %}{% endfor %}
{{ section_link('data-providers', 'Data providers', 'cloud-down', 'hugging face huggingface hf_token docker hub login registry token dataset pull limit') }}
{{ section_link('custom-keys', 'Custom keys', 'key', access_search.words) }}
- {{ section_link('judge', 'Judges', 'scale', 'typesafe system one llm model judgement') }}
+ {{ section_link('judge', 'Judges', 'scale', 'typesafe decision llm model judgement') }}
{{ section_link('prices', 'Model prices', 'dollar', 'cost price estimate usd tokens list update') }}
@@ -186,7 +186,7 @@
Add a provider
Judges
Choose the judges that evaluate trial evidence.
- {% set s1, llm = judges %}
+ {% set decision, llm = judges %}
{#- Provider, then Protocol when the provider lists two this judge speaks (else a quiet line), then Model. -#}
{% macro judge_rows(prefix, choices, judge, label) %}
@@ -218,18 +218,18 @@
Judges
{% endmacro %}
- {% set srow = by.get('SYSTEM_ONE_PROVIDER') %}
- {% if srow and by.get('SYSTEM_ONE_MODEL') %}
-
', s1)
+ s1, llm = (html[html.index(f'id="{p}_PROVIDER"'):html.index(f'id="{p}_MODEL"')] for p in ("DECISION", "LLM_JUDGE"))
+ self.assertIn('
via TypeSafe Decision
', s1)
self.assertIn('
via Anthropic Messages
', llm)
self.assertIn('id="LLM_JUDGE_PROTOCOL" data-source="unset" data-secret="no" hidden>', llm)
self.assertNotIn("messages<", llm); self.assertNotIn(">chat<", llm) # names, never keys
diff --git a/app/web/judge.py b/app/web/judge.py
index 06abdce8..95ab677d 100644
--- a/app/web/judge.py
+++ b/app/web/judge.py
@@ -16,9 +16,9 @@
from . import settings as settings_web
from . import traces
-def key(lane: str = "system-one") -> client.Judge:
+def key(lane: str = "decision") -> client.Judge:
"""A judgement's judge, with its key and model: true when it can be
- asked. System One unless the LLM judgement is named; passed down to
+ asked. Decision unless the LLM judgement is named; passed down to
every call as the key is."""
return client.judge_for(lane, os.environ)
@@ -73,12 +73,12 @@ def started_by_name(job) -> dict:
return {store._SAFE.sub("_", t.trial_name): started_of(t) for t in job.trials}
-def judgement_of(job, trial, lane: str = "system-one") -> dict | None:
+def judgement_of(job, trial, lane: str = "decision") -> dict | None:
d = ws_dir()
return store.load(d, job.job_id, trial.trial_name, lane=lane, started=started_of(trial)) if d else None
-def panel(job, trial, lane: str = "system-one") -> dict:
+def panel(job, trial, lane: str = "decision") -> dict:
"""What the trial page renders: each question with its answer or 'not
judged', the record's provenance, whether a Judge action is possible.
A record asked under an earlier vocabulary is shown under its own
@@ -271,7 +271,7 @@ def judge_trial_route(job_id, trial_name):
"""One click, one trial: the page's questions asked and stored."""
_guard()
job, trial = traces._trial_evidence(job_id, trial_name)
- lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "system-one"
+ lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "decision"
try:
rec = judge_trial(job, trial, key(lane))
except client.JudgeError as e:
@@ -289,11 +289,11 @@ def judge_trial_route(job_id, trial_name):
@app.route("/judge/job/", methods=["POST"])
def judge_job_route(job_id):
"""Judge the job's unjudged trials with one judge, the form's ``lane``
- (System One unless it names the LLM judge): the click is the request."""
+ (Decision unless it names the LLM judge): the click is the request."""
job = shared._find_job(shared._jobs_for_request(), job_id)
if job is None:
abort(404)
- lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "system-one"
+ lane = request.form.get("lane") if request.form.get("lane") in client.LANES else "decision"
d = ws_dir()
judged = store.for_job(d, job_id, lane=lane, started=started_by_name(job)) if d else {}
todo = [t for t in job.trials if t.has_trajectory and store._SAFE.sub("_", t.trial_name) not in judged]
@@ -301,7 +301,7 @@ def judge_job_route(job_id):
k = key(lane)
done, failed = run_batch(todo, lambda t: judge_trial(job, t, k), lambda t: t.trial_name)
from flask import flash
- flash(f"{'LLM judge' if lane == 'llm' else 'System One'}: judged {done} of {len(todo)} trial(s)." + (" Failed: " + "; ".join(failed[:3]) if failed else ""), "error" if failed else "")
+ flash(f"{'LLM judge' if lane == 'llm' else 'Decision model'}: judged {done} of {len(todo)} trial(s)." + (" Failed: " + "; ".join(failed[:3]) if failed else ""), "error" if failed else "")
return redirect(url_for("job", job_id=job_id) + "#judgement")
@@ -347,7 +347,7 @@ def job_summary(job) -> dict:
def _llm_count(d, job) -> dict:
"""The LLM judge's own count for the job's button: it judges apart from
- System One, so a trial judged by one is still unjudged by the other."""
+ Decision, so a trial judged by one is still unjudged by the other."""
judged = store.for_job(d, job.job_id, lane="llm", started=started_by_name(job)) if d else {}
return {"unjudged": len([t for t in job.trials if store._SAFE.sub("_", t.trial_name) not in judged]), "keyed": bool(key("llm"))}
diff --git a/app/web/settings.py b/app/web/settings.py
index f7601ae6..b74eb2ba 100644
--- a/app/web/settings.py
+++ b/app/web/settings.py
@@ -54,8 +54,8 @@ def settings_page():
prices=prices.state(),
machines=_machines(), constraint_table=constraints.TABLE,
provider_cards=provider_cards({r["name"]: r for r in rows}),
- ssh_hosts=hosts.scan_ssh_config(), judges=(client.system_one(os.environ), client.llm_judge(os.environ)), llm_providers=client.llm_providers(os.environ), s1_providers=client.system_one_providers(os.environ),
- judge_protocols={"SYSTEM_ONE": client.SYSTEM_ONE_PROTOCOLS, "LLM_JUDGE": client.LLM_PROTOCOLS}, protocols=providers.PROTOCOLS, agent_keys=settings.agent_key_names(), reserved_agent_keys=settings.reserved_agent_keys(),
+ ssh_hosts=hosts.scan_ssh_config(), judges=(client.decision_judge(os.environ), client.llm_judge(os.environ)), llm_providers=client.llm_providers(os.environ), decision_providers=client.decision_judge_providers(os.environ),
+ judge_protocols={"DECISION": client.DECISION_PROTOCOLS, "LLM_JUDGE": client.LLM_PROTOCOLS}, protocols=providers.PROTOCOLS, agent_keys=settings.agent_key_names(), reserved_agent_keys=settings.reserved_agent_keys(),
tested={k: _shown(v) for k, v in providers.tests().items() if isinstance(v, dict)})
@@ -78,14 +78,14 @@ def settings_provider_test():
return jsonify(_shown(got))
-@app.route("/settings/system-one/models", methods=["POST"])
-def settings_system_one_models():
- """System One's models on OpenRouter for its card, asked after the page has loaded: OpenRouter's
- public list of decision models, kept in the cache (``client.system_one_models``)."""
+@app.route("/settings/decision/models", methods=["POST"])
+def settings_decision_judge_models():
+ """Decision's models on OpenRouter for its card, asked after the page has loaded: OpenRouter's
+ public list of decision models, kept in the cache (``client.decision_judge_models``)."""
shared._require_csrf()
if not _local_client():
abort(403)
- return jsonify(ok=True, models=[{"id": m, "name": m} for m in client.system_one_models("openrouter", fetch=True)])
+ return jsonify(ok=True, models=[{"id": m, "name": m} for m in client.decision_judge_models("openrouter", fetch=True)])
def _size(mb: int) -> str:
diff --git a/app/web/traces.py b/app/web/traces.py
index 03df52dc..e41a4fa9 100644
--- a/app/web/traces.py
+++ b/app/web/traces.py
@@ -310,7 +310,7 @@ def trial_attempts(job_id, trial_name):
imported_from=_imported_from(j))
-def _judgement_panel(job, trial, lane="system-one"):
+def _judgement_panel(job, trial, lane="decision"):
from . import judge
return judge.panel(job, trial, lane)
diff --git a/docs/judgement.md b/docs/judgement.md
index 8796dc21..b7ef3368 100644
--- a/docs/judgement.md
+++ b/docs/judgement.md
@@ -6,10 +6,10 @@ Two judgements per trial, a tab each, records in `judgements/` and `judgements-l
| Judgement | Answers |
| --- | --- |
-| [System One](settings/system-one.md) | A decision model's answers: a yes/no's probability, a choice's distribution, a score, each with the model's confidence when it gives one; one call per state |
+| [Decision](settings/decision.md) | A decision model's answers: a yes/no's probability, a choice's distribution, a score, each with the model's confidence when it gives one; one call per state |
| [LLM judge](settings/llm-judge.md) | What a chat model states, one JSON object per state, no confidence |
-The job, compare, feasibility and task-instruction pages ask System One. A record names the provider and the model that answered, and the questions the model refused. The cost estimate for TypeSafe is calibrated on its own stored calls.
+The job, compare, feasibility and task-instruction pages ask the Decision model. A record names the provider and the model that answered, and the questions the model refused. The cost estimate for TypeSafe is calibrated on its own stored calls.
!!! note "A judge never changes an outcome, a count or a verdict"
@@ -76,7 +76,7 @@ Follow the evidence: a recorded rate limit is a provider's failure whatever the
-- [System One provider](settings/system-one.md)
+- [Decision provider](settings/decision.md)
- [LLM judge provider](settings/llm-judge.md)
- [Read results](results.md)
diff --git a/docs/settings/README.md b/docs/settings/README.md
index ecfefed8..4b5f4e4f 100644
--- a/docs/settings/README.md
+++ b/docs/settings/README.md
@@ -7,7 +7,7 @@ changes save automatically; data providers and benchmark credentials use **Save*
| Page | Sets |
| --- | --- |
| [Model providers](model-providers.md) | A run's key or endpoint |
-| [System One provider](system-one.md) | The System One judgement |
+| [Decision provider](decision.md) | The Decision judgement |
| [LLM judge provider](llm-judge.md) | The second judge |
| [Custom keys](agent-keys.md) | Agent credentials, dataset keys and benchmark tool licences |
| [Model prices](model-prices.md) | What a run's cost is estimated from |
@@ -26,7 +26,7 @@ changes save automatically; data providers and benchmark credentials use **Save*
| `OPENAI_API_KEY` | Runs on OpenAI |
| `OPENROUTER_API_KEY` | Runs on OpenRouter |
| `ANTHROPIC_API_KEY` | Standard runs whose agent speaks `messages`, and the LLM judge on Anthropic |
-| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL`, `TYPESAFE_API_KEY` | [System One](system-one.md) |
+| `DECISION_PROVIDER`, `DECISION_PROTOCOL`, `DECISION_MODEL`, `TYPESAFE_API_KEY` | [Decision](decision.md) |
| `LLM_JUDGE_PROVIDER`, `LLM_JUDGE_PROTOCOL`, `LLM_JUDGE_MODEL` | [LLM judge](llm-judge.md) |
| `AGENT_KEYS` | Comma-separated agent key names; Harbor runs only |
| `HOSTS` | The hosts |
diff --git a/docs/settings/system-one.md b/docs/settings/decision.md
similarity index 64%
rename from docs/settings/system-one.md
rename to docs/settings/decision.md
index f41623d5..9c0098d2 100644
--- a/docs/settings/system-one.md
+++ b/docs/settings/decision.md
@@ -1,11 +1,11 @@
-# System One provider
+# Decision provider
-The provider and model that answer the System One judgement: questions about recorded evidence.
+The provider and model that answer the Decision judgement: questions about recorded evidence.
-System One asks a provider that lists a decision protocol
+Decision asks a provider that lists a decision protocol
([protocols](model-providers.md#protocols)).
-| `SYSTEM_ONE_PROVIDER` | Credential | Protocol | Default model |
+| `DECISION_PROVIDER` | Credential | Protocol | Default model |
| --- | --- | --- | --- |
| `typesafe` | `TYPESAFE_API_KEY` | `systemone` | `jev-latest` |
| `openrouter` | `OPENROUTER_API_KEY` | `systemone` for TypeSafe's models and `jaredpalmer/kev-4b`, `decisions` for other makers' | `~typesafe/jev-latest` |
@@ -14,9 +14,10 @@ System One asks a provider that lists a decision protocol
| Setting | Value |
| --- | --- |
-| `SYSTEM_ONE_PROVIDER` | Unset: TypeSafe when its key is set, else OpenRouter |
-| `SYSTEM_ONE_PROTOCOL` | `systemone`, `decisions` or `openai-decisions`, on a provider that serves more than one; unset, or not served: the protocol in the table above |
-| `SYSTEM_ONE_MODEL` | A model the provider accepts, such as `typesafe/jev-1.14` |
+| `DECISION_PROVIDER` | Unset: TypeSafe when its key is set, else OpenRouter |
+| `DECISION_PROTOCOL` | `systemone`, `decisions` or `openai-decisions`, on a provider that serves more than one; unset, or not served: the protocol in the table above |
+| `DECISION_MODEL` | A model the provider accepts, such as `typesafe/jev-1.14` |
+| `SYSTEM_ONE_PROVIDER`, `SYSTEM_ONE_PROTOCOL`, `SYSTEM_ONE_MODEL` | Old names, still read |
| Reply | Recorded as |
| --- | --- |
@@ -26,7 +27,7 @@ System One asks a provider that lists a decision protocol
## Configure
-Add the connection in **Settings → Model providers** first. **Judges** offers only configured, compatible providers. A sole provider is selected automatically; with several, choose one and set the model. Under the provider, the card names the protocol System One uses. When the provider serves more than one, choose one under **Protocol**. Keys and endpoints stay in Model providers.
+Add the connection in **Settings → Model providers** first. **Judges** offers only configured, compatible providers. A sole provider is selected automatically; with several, choose one and set the model. Under the provider, the card names the protocol Decision uses. When the provider serves more than one, choose one under **Protocol**. Keys and endpoints stay in Model providers.
On OpenRouter, the **Model** suggestions are OpenRouter's public list of decision models. Nullius gets the list without a key when you first use the field, and keeps it in the cache for 6 hours. With no network and no cache, the suggestions are a short built-in list.
@@ -38,8 +39,8 @@ your existing `settings.json` ([location and precedence](README.md#save)).
```json title="settings.json"
{
"TYPESAFE_API_KEY": "YOUR_TYPESAFE_KEY",
- "SYSTEM_ONE_PROVIDER": "typesafe",
- "SYSTEM_ONE_MODEL": "jev-latest"
+ "DECISION_PROVIDER": "typesafe",
+ "DECISION_MODEL": "jev-latest"
}
```
@@ -48,14 +49,14 @@ your existing `settings.json` ([location and precedence](README.md#save)).
```json title="settings.json"
{
"OPENROUTER_API_KEY": "YOUR_OPENROUTER_KEY",
- "SYSTEM_ONE_PROVIDER": "openrouter",
- "SYSTEM_ONE_MODEL": "~typesafe/jev-latest"
+ "DECISION_PROVIDER": "openrouter",
+ "DECISION_MODEL": "~typesafe/jev-latest"
}
```
## Request a judgement
-A trial's or a job's System One tab.
+A trial's or a job's Decision model tab.
!!! warning "A judgement is a paid call"
Configuration alone calls nothing; a batch button says how many calls it will make.
diff --git a/docs/settings/llm-judge.md b/docs/settings/llm-judge.md
index fd4719ce..45cd1d76 100644
--- a/docs/settings/llm-judge.md
+++ b/docs/settings/llm-judge.md
@@ -1,6 +1,6 @@
# LLM judge provider
-The same questions as System One, answered as JSON by a chat model.
+The same questions as Decision, answered as JSON by a chat model.
| Setting | Value |
| --- | --- |
@@ -9,7 +9,7 @@ The same questions as System One, answered as JSON by a chat model.
| Credential | [Model providers](model-providers.md); a custom provider needs its base |
| `LLM_JUDGE_PROTOCOL` | `chat` or `messages`, on a provider that serves both; unset, or not served: `chat` where the provider serves it, else `messages` |
-| Differs from System One | |
+| Differs from Decision | |
| --- | --- |
| Answers | What the model states; no confidence value |
| Out of range | An answer outside its question's options or range is left out |
@@ -39,7 +39,7 @@ your existing `settings.json` ([location and precedence](README.md#save)).
Evidence, storage, cost.
-- [System One](system-one.md)
+- [Decision](decision.md)
The first judgement.
diff --git a/docs/settings/model-providers.md b/docs/settings/model-providers.md
index f26a5bcc..826dddc9 100644
--- a/docs/settings/model-providers.md
+++ b/docs/settings/model-providers.md
@@ -1,7 +1,7 @@
# Model providers
Where model calls go, and with which key. Each provider is defined once, and
-every feature reads that definition: runs, the provider test, System One and
+every feature reads that definition: runs, the provider test, Decision and
the LLM judge. A feature asks for the protocol it speaks and gets the providers
that can serve it.
@@ -28,7 +28,7 @@ and how the key is sent. The feature that calls the provider picks the protocol.
| `chat` | OpenAI Chat Completions | `/chat/completions` | Bearer |
| `responses` | OpenAI Responses | `/responses` | Bearer |
| `messages` | Anthropic Messages | `/messages` | `x-api-key` and `anthropic-version` |
-| `systemone` | TypeSafe System One | `/systemone` | Bearer |
+| `systemone` | TypeSafe Decision | `/systemone` | Bearer |
| `decisions` | OpenRouter Decisions (alpha) | `/decisions` | Bearer |
| `openai-decisions` | OpenAI Decisions (beta) | `/decisions` | Bearer |
@@ -36,7 +36,7 @@ and how the key is sent. The feature that calls the provider picks the protocol.
| --- | --- |
| A run | What its agent speaks (`model.apis` in its definition); a Harbor run, `chat` |
| LLM judge | `chat`, or `messages` on Anthropic |
-| System One | `systemone`, `decisions` or `openai-decisions` |
+| Decision | `systemone`, `decisions` or `openai-decisions` |
Anthropic is offered to agents that speak `messages`. It is not offered to an
agent that speaks only `chat`, to a Harbor recipe, or to a judged benchmark.
@@ -178,7 +178,7 @@ Keys keep their names. `LLM_JUDGE_PROVIDER` still reads `local` and
Precedence, reloads.
-- [System One](system-one.md), [LLM judge](llm-judge.md)
+- [Decision](decision.md), [LLM judge](llm-judge.md)
The judges.