A self-hosted world-event monitor that shows its evidence, and publishes the experiments where its own predictions failed.
It collects public data about world events: news, machine-coded event records, disasters, markets, cyber indicators, satellite fire detections, aircraft presence, normalises it into one row shape, stores it locally, and puts it on a map with the provenance attached. It runs on one machine. Nothing leaves it.
basilsuhail.github.io/OSINT: the map, the filters and the cards, drawn from a frozen snapshot of a running console. Hazards, military aircraft and AIS vessels, all clickable. Nothing there is live: it is one moment saved to a file, and the page says so at the top and again beside anything it had to thin out.
Everything below is how to run the real thing, which fetches, stores, scores and refreshes on your own machine.
Pick your machine. Each one is self-contained: everything from nothing to a running console, in order, nothing to look up elsewhere.
Free disk space is the one thing to check first: about 15 GB to install, and the database grows to a 30 GB cap before it starts trimming its own oldest days.
Raspberry Pi 5 (8 GB)
Two things about the board itself. It needs 64-bit Raspberry Pi OS (the images are arm64 and will not run on the 32-bit build), and it needs the official 27 W supply. A phone charger browns out under load, and the board answers by throttling rather than by saying so:
vcgencmd get_throttled0x0 is healthy. Anything else is the power supply, and no amount of tuning
below will fix it.
Docker and Node:
sudo apt update && sudo apt install -y git curl ca-certificates make python3 && \
curl -fsSL https://get.docker.com | sudo sh && \
sudo usermod -aG docker "$USER" && \
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - && \
sudo apt install -y nodejs && sudo corepack enableThen reboot. Not optional and not cosmetic: the line above adds you to the
docker group, and a group only takes effect on a new login. Until then every
Docker command is refused, and the next few steps do not touch Docker, so the
failure surfaces several minutes later, at make up, looking like Docker is
broken:
sudo rebootAfterwards groups lists docker. Pick the list up again from here.
Ollama:
curl -fsSL https://ollama.com/install.sh | sh
sudo mkdir -p /usr/share/ollama
sudo chown -R ollama:ollama /usr/share/ollama
sudo mkdir -p /etc/systemd/system/ollama.service.d
cd /etc/systemd/system/ollama.service.d
echo '[Service]' | sudo tee override.conf
echo 'Environment="OLLAMA_HOST=0.0.0.0"' | sudo tee -a override.conf
cd -
sudo systemctl daemon-reload
sudo systemctl enable ollama
sudo systemctl restart ollama
sleep 3
systemctl is-active ollama
ss -tlnp | grep 11434Must print active and 0.0.0.0:11434. If not:
sudo journalctl -u ollama -n 20 --no-pagerThe code:
git clone https://github.com/BasilSuhail/OSINT.git
cd OSINT
make envNothing to edit. make env reads how much memory the board has and, at 8 GB,
writes the settings for it: one small model doing every job instead of three
different ones, and memory floors and generation timeouts sized for that model.
It prints what it chose.
Those numbers are in force from the first run, not held in reserve.
BRAIN_TIMEOUT_S is the ceiling on every local generate this board makes (the
gists, the keywords and tags, the severity grades, the situation summary), and
raising it to 300 s is what stops the slowest of them being cut off and
reported as the brain being offline. QA_MIN_FREE_MB is the free-memory floor
a reader-triggered read has to clear, and the reasoned read behind a contested
story still asks for one. They are also the numbers an answer would use, sized
so they are already right if you turn the question box back on.
The question box is the one thing make env turns off on a board this size.
Everything else runs: it fetches, it stores, it scores, it writes the gists and
the tags and the situation summary, and both search boxes answer as fast as
they do on a laptop, because neither has ever gone near a model. What it does
not do is answer typed questions: that costs minutes of a board that has other
work, and the box is not drawn rather than left there to disappoint.
ASK_ENABLED in .env is the setting. It ships blank, which means on; make env writes false into it here. To have the box: write true over that,
empty the NEXT_PUBLIC_ASK_ENABLED line below it (that is the console's own
copy, and make env writes the new value across whenever it is blank), then
run make env && make up. The restart is not optional: the dashboard compiles
the setting into its bundle when it starts, so a running one keeps serving the
build it was started with. make env leaves a true you typed alone from then
on, on every later run, and make env-check says so if the two lines ever
drift apart.
A backstop, in case something still runs the board out of memory. It kills the largest process instead of letting the kernel thrash, which is the difference between a log line and a machine you have to unplug:
sudo apt install -y earlyoom
sudo systemctl enable --now earlyoomThe models. Two on a board this size, about 2.3 GB together: one that writes the gists and the tags, one that turns text into vectors now, so retrieval is warm if you ever turn the question box back on:
ollama pull llama3.2:3b
ollama pull nomic-embed-textmake up pulls these itself if you skip them, and make env has already written
their names into .env. Doing it here means the download happens now rather than
in the middle of the first start.
Check the model actually answers before building anything on it:
ollama run llama3.2:3b "reply with one short sentence"A sentence back means the model works. This is the check worth having: if it fails here, nothing above it is the problem.
Start it:
make up20–40 minutes the first time: images and browser packages, plus the models if you skipped the pull.
Open http://localhost:3000, or make share to reach it from another
device.
Fill it:
make fetch
make newsLinux desktop or server: Debian, Ubuntu
Docker and Node:
sudo apt update && sudo apt install -y git curl ca-certificates make python3 && \
curl -fsSL https://get.docker.com | sudo sh && \
sudo usermod -aG docker "$USER" && \
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - && \
sudo apt install -y nodejs && sudo corepack enableThen reboot. Not optional and not cosmetic: the line above adds you to the
docker group, and a group only takes effect on a new login. Until then every
Docker command is refused, and the next few steps do not touch Docker, so the
failure surfaces several minutes later, at make up, looking like Docker is
broken:
sudo rebootAfterwards groups lists docker. Pick the list up again from here.
Ollama. The systemd override is what lets the containers reach it:
curl -fsSL https://ollama.com/install.sh | sh
sudo mkdir -p /usr/share/ollama
sudo chown -R ollama:ollama /usr/share/ollama
sudo mkdir -p /etc/systemd/system/ollama.service.d
cd /etc/systemd/system/ollama.service.d
echo '[Service]' | sudo tee override.conf
echo 'Environment="OLLAMA_HOST=0.0.0.0"' | sudo tee -a override.conf
cd -
sudo systemctl daemon-reload
sudo systemctl enable ollama
sudo systemctl restart ollama
sleep 3
systemctl is-active ollama
ss -tlnp | grep 11434Must print active and 0.0.0.0:11434. If not:
sudo journalctl -u ollama -n 20 --no-pagerThe code, and the settings for this machine:
git clone https://github.com/BasilSuhail/OSINT.git
cd OSINT
make envNothing to edit. make env measures the memory and writes the settings to match,
then prints the models it chose. On anything with room that is three, about 5 GB:
one writes the summary, one answers questions, one turns text into vectors so
the Ask panel can find the right stories:
ollama pull llama3.2:3b
ollama pull qwen3.5:4b-q4_K_M
ollama pull nomic-embed-textIf make env printed different names, pull those instead, because it has measured this
machine and those commands have not. Skipping this is fine either way: make up
pulls whatever .env names.
Check the model answers:
ollama run llama3.2:3b "reply with one short sentence"A sentence back means the model works. If it fails here, nothing above it is the problem. Then:
make up20–40 minutes the first time: images, browser packages, and any model you skipped.
Open http://localhost:3000, then fill it:
make fetch
make newsmacOS
Homebrew first, if you do not have it:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"On Apple silicon it finishes by printing two eval lines to run. Run them, or
brew is installed but not on your path. Then:
brew install git node && brew install --cask docker && sudo corepack enable && open -a DockerDocker Desktop asks for your password and for you to accept its terms the first time. It has to finish starting before anything below will work.
Wait for the Docker whale in the menu bar to settle, then:
brew install ollama
brew services start ollamaNo override here: Docker Desktop reaches the host without one. Check it took:
curl -s http://localhost:11434/api/tagsAny JSON back means it is running. Connection refused means it is not, and the Ask panel will say the brain is offline while everything else works.
The code, and the settings for this Mac:
git clone https://github.com/BasilSuhail/OSINT.git
cd OSINT
make envNothing to edit. make env measures the memory and writes the settings to match,
then prints the models it chose. On a Mac with room that is three, about 5 GB:
one writes the summary, one answers questions, one turns text into vectors so the
Ask panel can find the right stories:
ollama pull llama3.2:3b
ollama pull qwen3.5:4b-q4_K_M
ollama pull nomic-embed-textIf make env printed different names, pull those instead, because it has measured this
Mac and those commands have not. Skipping this is fine either way: make up
pulls whatever .env names.
Check the model answers:
ollama run llama3.2:3b "reply with one short sentence"A sentence back means the model works. If it fails here, nothing above it is the problem. Then:
make up20–40 minutes the first time: images, browser packages, and any model you skipped.
Open http://localhost:3000, then fill it:
make fetch
make newsWindows: through WSL2
In PowerShell as administrator:
wsl --install -d UbuntuReboot. Install Docker Desktop and turn on Settings → Resources → WSL Integration for Ubuntu.
Everything below runs in the Ubuntu terminal, not PowerShell. No
get.docker.com: Docker Desktop supplies the engine, and a second one fights it:
sudo apt update && sudo apt install -y git curl ca-certificates make python3 && \
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - && \
sudo apt install -y nodejs && sudo corepack enableOllama, on Windows rather than inside Ubuntu. Download it from ollama.com/download.
Then let it answer the containers. Ollama listens only to the Windows machine itself until told otherwise, and the stack reaches it from inside a container, so out of the box it refuses the connection, and the console reports that as the brain being offline while the map and the feed work normally. In PowerShell as administrator:
[Environment]::SetEnvironmentVariable("OLLAMA_HOST", "0.0.0.0", "Machine")Quit Ollama from the system tray and start it again: an environment variable set this way reaches it only on a fresh start. Check, in PowerShell:
curl.exe -s http://localhost:11434/api/tagsJSON back means it is listening.
Everything from here runs in the Ubuntu terminal, and the clone belongs in your
Ubuntu home directory, not under /mnt/c. A repository on the Windows drive is
reached through a translation layer, and the difference is minutes per command
rather than seconds.
The code, and the settings for this machine. In Ubuntu:
git clone https://github.com/BasilSuhail/OSINT.git
cd OSINT
make envNothing to edit. make env measures the memory and writes the settings to match,
then prints the models it chose, usually three, about 5 GB. One writes the
summary, one answers questions, one turns text into vectors so the Ask panel can
find the right stories.
Pull them in PowerShell, because Ollama is the Windows one:
ollama pull llama3.2:3b
ollama pull qwen3.5:4b-q4_K_M
ollama pull nomic-embed-textIf make env printed different names, pull those instead, because it has measured this
machine and those commands have not. Skipping this is fine either way: make up
pulls whatever .env names.
Check the model answers, still in PowerShell:
ollama run llama3.2:3b "reply with one short sentence"A sentence back means the model works. If it fails here, nothing above it is the problem. Back in Ubuntu:
make up20–40 minutes the first time: images, browser packages, and any model you skipped.
Open http://localhost:3000, then fill it:
make fetch
make newsDo not install pnpm yourself on any of them. corepack enable fetches the
version this project pins; a different one resolves the lockfile differently.
make up # start everything
make down # stop everything, keep all data
make share # start it reachable from your phone or another computer
make fetch # fill the map now instead of waiting for the schedule
make news # build the story feed and the written summary
make news-all # gist every story rather than 20, hours on a small box
make logs # watch what the stack is doing
make env-check # say what .env is missing, empty, or typed wrong
make help # every command in the MakefileIt means the API could not get an answer out of the model, and it says the same sentence whichever reason applies: not installed, not reachable, not enough memory, or simply slower than the machine is allowed to wait. Four different faults, one sentence, which is why it is worth checking in order:
ollama psNothing listed and no model loading when you ask means the stack cannot reach
Ollama. Check it is listening where the containers can see it: 0.0.0.0, not
127.0.0.1, using the check in your machine's list above.
make askRuns the retrieval and the model call the console's ask does, from the terminal, and prints the real error instead of the one sentence. It also reports free memory against the floor the model needs, and how long the prompt is.
It goes past ASK_ENABLED deliberately, since running it is itself a decision to ask,
so it still answers on a build where the console's box is gone. It prints the
setting beside the floor, so an answer here and no box on screen is a difference
you can see rather than one you have to work out.
The console is empty at first and that is not a fault: nothing has been fetched
yet. Left alone it fills itself over the next hour. make fetch and make news
skip the wait.
Something wrong: §19. Any step in full detail: §1.
git checkout main
git pull
make env
make upTo test a branch before it merges, swap main for the branch name. Run make env
after every pull: new settings reach your file that way, and skipping it leaves
the feature they switch on quietly off.
Full prerequisites: §3. The walkthrough: §1. Anything that goes wrong: §19.
Read §0 before trusting any number on the screen. The predictive claim this project was built to test has been put through every pre-registered protocol built for it and refused every time. That is stated up front, with the tables, rather than buried.
Licence: PolyForm Noncommercial 1.0.0: source-available, not open source. Security reporting, provider data terms, and attribution are in §25.
4,300 lines. 24 sections. Everything underneath this page.
| The handbook covers | |
|---|---|
| Operate it | Install and first run · read the console · configure · every control · troubleshooting |
| Every formula | Corroboration · divergence · the composite · severity · lead-time gate, each with a worked example and its failure modes |
| What was tested, and what failed | Every pre-registered refusal, with baselines, bootstrap confidence intervals, and the held-out window |
| Whether to trust the data | Bias and provenance, including one country traced end to end with measured counts |
| Rebuild it yourself | Every command that regenerates every number quoted anywhere |
| Reference | Data sources · pipeline · storage · backend · frontend · glossary · code walkthroughs |
Every numbered control above is explained in §2.1 of the handbook, each with a link to the code that produces it.
- 0. What this system is for, and what it is not
- 25. Licence, security, and provider terms
- 26. Repository map
- 27. Documentation index
- 28. SWOT
Section numbers are shared across both files and never reused: 1–24 are in HANDBOOK.md, so a reference to §15 means the same thing wherever you read it.
Read this page first. It states the claim the project makes, the claim it does not make, and what is actually being offered, so nothing later in the handbook has to be inferred.
This is a self-hosted system that collects public data about world events: news, machine-coded event records, disasters, markets, cyber indicators, satellite fire detections, aircraft presence, normalises it into one row shape, stores it locally, and puts it on a map with the evidence attached. It runs on one machine. Nothing leaves it. Every number it shows can be traced back to the row that produced it.
A composite of several open-data signal domains discriminates later instability better than the best single-domain baseline.
That claim was pre-registered, evaluated, and refused. Not once: under every protocol built for it, including on a held-out window reserved specifically for the question:
| Evaluation | Result |
|---|---|
| Incidence, pooled | Composite AUROC ≈ 0.502 against a 0.929 base rate |
| Head-to-head vs single domains | beaten: [], dominates none of them, in either window |
| Held-out test 2023–24 | Same verdict on data reserved for this question |
| Onset | 0.496 / 0.520 / 0.526, a coin flip |
| Within-country concordance | 0.531 best, CI [0.474, 0.582], below the declared 0.55 |
The predictive claim failed. This document does not soften that anywhere, and §15 gives every table.
The composite failed. The machinery that caught it did not.
What this project delivers is an apparatus for not fooling yourself, built around data that makes fooling yourself easy. Concretely:
- Protocols are frozen before results exist. Eligibility, target, horizons, contenders, metrics and the decision rule are written down and dated first. The run happens once. Corrections are amendments, never edits (§15.1).
- Verdicts are computed, not narrated. A threshold decides, in code. The
within-country evaluation prints
NEGATIVEbecause_verdict()said so, not because someone read the table and agreed (§15.7). - A trend that fails the rule is reported as a failure. The composite rises monotonically with horizon. The protocol declared in advance that 0.50–0.55 is a negative rather than a promising trend, so it is recorded as a negative (§15.7).
- Forecasts of a constant are refused entry. 1,101 forecasts of the number 0.5 were once recorded as forecasts. The check is now concentration, not exact flatness (§14.4).
- A hindcast cannot enter the journal. A "prediction" whose window overlaps the known past is skipped, because grading it would fake a track record (§15.10).
- Independence must be positively established. A source with no ownership record contributes nothing to a confidence score, because ten anonymous blogs are one claim told ten times, not ten confirmations (§14.6).
- Absence is not imputed as average. A missing domain is excluded and the weights renormalised, rather than entered as "exactly average" (§14.4).
- The bias is measured and published, including the finding that 54 of 55 news feeds are English and that a country the system reports on can have no domestic feed at all (§16).
The evidence that this apparatus works is that it caught its own author's claim and refused it, repeatedly, on the record. A system that only ever confirms the hypothesis of the person who built it has not been tested. This one has.
- It does not forecast instability. See above. Any future claim requires a new pre-registered version and a new evaluation.
- It does not calculate truth. It shows provenance, corroboration, disagreement and sensor agreement. Six independent outlets can be independently wrong.
- Its scores are not probabilities. They are orderings. Nothing here is calibrated, so 0.875 does not mean "correct 87.5% of the time".
- It does not see the whole world. It sees what open, mostly English-language feeds publish, and it says so with numbers rather than a disclaimer.
- It is not a safety-critical alerting service. Upstream feeds fail, arrive late, change format, and withdraw access.
Stated here rather than left to be discovered:
- The running version has never been evaluated. Everything in §15 grades
v1.0. The live system emitsv3.0, which changed the domain structure and the weights (§14.1). - There is no forward evidence yet. 1,695 predictions issued, 0 graded (§15.10).
- Five of six declared robustness tests have not been run (§15.9).
- Two of the five label families were never built.
P4(market crisis) andP5(hazard disruption) do not exist, so a multi-modal index is currently graded only on whether it predicts conflict (§15.2). - The pre-specified case studies were never filled in.
- Reporting delay has never been measured on this system's own data (§16.4).
It is useful to someone who has to answer "is this story actually being independently reported, or is it one wire item repeated?", "does any physical sensor agree with this claim?", "are two countries telling this differently?", and "how much should I trust what I am looking at?", and who wants each answer to arrive with the evidence attached rather than as a verdict.
It is not useful to someone who wants a risk number to act on. That number was built, tested, and found not to work.
| If you want to | Read |
|---|---|
| Run it | §1 |
| Understand the screen | §2 |
| See how a number is computed | §14 |
| See what was tested and what failed | §15 |
| Judge the data before trusting it | §16 |
| Rebuild every number yourself | §17 |
| Find any file in the repository | §26 |
Find the right document in docs/ |
§27 |
| Know the licence, security policy, and data terms | §25 |
| See a candid read of where this stands | §28 |
PolyForm Noncommercial 1.0.0. Full text in
LICENSE.
This is source-available, not open source. The distinction is not pedantry: every OSI-approved licence permits commercial use, so calling this open source would tell you that you have a right you do not have.
You may use it, run it, study it, fork it, modify it and share your changes, for any noncommercial purpose: personal projects, study and research, and use by charities, educational institutions, public research bodies and government. You may not sell it or use it commercially.
Two reasons, and the second is the real one:
- It is under development and its outputs have been wrong before. The composite is a coin flip in every pre-registered evaluation, and that is written down in §15 rather than hidden. Nothing here is fit to sell.
- It ingests third-party feeds. Several are free for noncommercial or research use and require a separate agreement for anything else. Those terms are not the maintainer's to hand on, so they are not handed on.
The licence covers the code in this repository and nothing else. It does not cover, and cannot cover, the data this software fetches. Those feeds belong to the organisations that publish them, each on its own terms. Nobody here has the right to sub-licence them, so nobody here has granted you anything over them.
Running this software makes you the one fetching the data. Whatever the provider requires (registration, an API key, attribution, a commercial licence, a limit on redistribution), it requires of you, directly, under the agreement you accept when you take the key.
NOTICE.md is the
maintained index of every feed and where its terms live. It is a pointer, not a
legal summary and not legal advice. Terms change. Read the current ones for any
feed you actually enable. Operational guidance on this is in
§9.5.
The bundled gazetteer files under
app/enrichment/data/
are Natural Earth, which is public domain.
Base map tiles, imagery overlays and aircraft presence are fetched by the browser directly from their publishers and are never stored; their attribution is rendered on the map itself.
Report privately. Do not open a public issue. The channels, the information
to include, and the automated tooling in place are in
SECURITY.md.
What runs continuously on this repository:
| Control | What it covers |
|---|---|
| Dependabot | pip and pnpm dependencies, weekly, opens update PRs |
| CodeQL | push, pull request, and a weekly scheduled scan |
pip-audit |
backend dependencies, in the backend workflow |
pnpm audit --audit-level high |
frontend dependencies, in the frontend workflow |
| Dependency review | blocks high-severity dependency regressions on PRs |
| Secret scanning + push protection | enabled at repository level |
The repository-level security baseline, what is enabled and what hardening is
still recommended, is logged in
docs/security.md.
Operational security when you run it is a separate matter and is covered in
this handbook: the API token in §5.4,
and network exposure, including the fact that make share adds no password,
in §5.7.
This repository is public and has been forked; forks copy the full history.
Anything committed is beyond recall the moment it is pushed. The rules for what
may and may not go into a commit, an issue or a pull request are in
AGENTS.md, which is
read by both people and coding agents.
The short version: no personal names, no contact details, no credentials, and one issue → one branch → one pull request → one commit. A pre-commit hook screens staged files and blocks on a hit; a block is a prompt to rephrase, not an obstacle to route around.
Issues are welcome. A good one names what you ran, what you expected, and
what happened, with the relevant lines from bash scripts/dev-logs.sh. If the
console is involved, say which browser. §19
covers the failures that already have known causes, worth a look first, because
several of them look like bugs and are configuration.
Security problems do not go in issues. Use the private channel in §25.4.
Pull requests: one issue, one branch, one pull request, one commit. Read
AGENTS.md first.
It is short, and it is binding on what may appear in a commit message, an issue
or a PR description, because this repository is public and has been forked.
Before opening one:
make verify # or: ruff check . && ruff format --check . && pytest
cd osint-frontend && pnpm exec tsc --noEmit && pnpm exec vitest runA pre-commit hook screens staged files and blocks on a hit. A block is a prompt to rephrase, not an obstacle to route around.
What is most useful right now, in order: an evaluation of the running
v3.0 composite (§14.1),
any of the five unrun robustness tests
(§15.9),
a chance-corrected agreement statistic for the severity grader
(§14.5),
and non-English or non-Anglophone-origin feeds
(§16.2).
The shape first, then the route data takes through it.
OSINT/
├── app/ PYTHON BACKEND: ingest · score · serve
│ ├── api.py FastAPI read-API: /events /scores /ingest-health /stream
│ ├── celery_app.py Celery app instance (broker = Redis)
│ ├── tasks.py Celery tasks + beat schedule (cadence + nightly prune)
│ ├── fetcher_registry.py maps source name → fetcher
│ ├── persistence.py upsert events into Postgres (+ Redis "new rows" tick)
│ ├── events_bus.py Redis pub/sub channel powering the live SSE stream
│ ├── housekeeping.py retention policy (see RETENTION_* in .env)
│ ├── db.py / db_models.py SQLAlchemy engine/session + table definitions
│ ├── settings.py ALL config, read from .env
│ ├── models.py canonical Event/Score shapes
│ ├── watchdog.py ingest health monitor
│ ├── sources/ one fetcher per feed + rss_feeds.json registry
│ ├── composite/ aggregation · normalisation · scoring · backfill
│ ├── corroboration/ per-story confidence + sensor cross-checks
│ ├── disagreement/ cross-country telling divergence
│ ├── divergence/ physical-vs-narrative spike and lead-time gate
│ ├── cii/ Country Instability Index
│ ├── journal/ immutable prediction journal
│ ├── onset/ within/ baselines/ the pre-registered evaluations
│ ├── audit/ nightly source-data audit + expectations
│ ├── brain/ local-model narrate · enrich · ask
│ ├── severity/ grading and measured agreement
│ ├── devx/ lan_share: who on the network may reach this
│ └── enrichment/ geocode · NER · sentiment (+ data/ gazetteers)
│
├── osint-frontend/ NEXT.JS CONSOLE: reads app/api.py
│ ├── app/ routes: page.tsx, layout.tsx, providers.tsx, news/
│ ├── lib/ apiClient.ts · queries.ts · realtime.ts · types.ts
│ ├── components/ MapPane · CardDeck · FilterRail · panels/
│ ├── stores/ zustand filter + selection stores
│ └── __tests__/ frontend suite
│
├── data/ ALL LOCAL STORAGE ($OSINT_DATA_DIR, gitignored)
│ ├── postgres/ the actual database files
│ ├── redis/ Redis append-only file
│ ├── private/ licensed/manual inputs, never commit
│ └── exports/ generated reports (evaluations, audits)
├── backups/ snapshot dumps (gitignored)
├── migrations/ Alembic schema migrations
├── scripts/ dev-up.sh · dev-down.sh · snapshot.py · one-off tools
├── tests/ pytest suite (backend)
├── docs/ specifications, protocols, and evaluation records, see §27
│
├── docker-compose.yml Postgres + Redis services
├── Makefile every command in this handbook
├── env.example copy → .env, then fill in
└── .env YOUR config and secrets (gitignored, never commit)
| I want | Open |
|---|---|
| My config and secrets | .env, from env.example; read via app/settings.py. Every setting explained in §5 |
| The database itself | data/postgres/, relocate with OSINT_DATA_DIR (§5.5) |
| What the console fetches | osint-frontend/lib/apiClient.ts ↔ served by app/api.py |
| To add or adjust a source | app/sources/ + app/fetcher_registry.py, full steps in §12.5 |
| How long data is kept | app/housekeeping.py and RETENTION_* (§11.3) |
A GDELT event's whole life. Every path is a link.
| # | Stage | File | What happens |
|---|---|---|---|
| 1 | fetch | app/sources/gdelt_fetcher.py |
request the newest export |
| 2 | register | app/fetcher_registry.py |
name → fetcher lookup |
| 3 | schedule | app/tasks.py |
Celery Beat fires it on cadence |
| 4 | dedup + store | app/persistence.py |
upsert on stable identity → events |
| 5 | normalise | app/composite/normalization.py |
rolling within-country z-score (§14.4) |
| 6 | score | app/composite/scoring.py |
weighted z → sigmoid → scores |
| 7 | serve | app/api.py |
/events, /scores, SSE stream |
| 8 | fetch in browser | osint-frontend/lib/apiClient.ts |
every backend call in one file |
| 9 | render | osint-frontend/components/ |
map, cards, panels |
| Source | Fetcher | Output |
|---|---|---|
| GDELT live | app/sources/gdelt_fetcher.py |
events, rolling window |
| GDELT history | app/composite/gdelt.py |
monthly checkpoints + daily volume |
| USGS · GDACS · FIRMS · EONET | app/sources/ |
events → footprint enrichment |
| yfinance · FRED | app/sources/ |
events, market and macro |
| ACLED (labels) | local drop folder (§5.3) | labels, ground truth, kept separate |
| RSS, 55 feeds | app/sources/rss_feeds.json |
events → stories |
| Concern | Folder | Formula | Method in |
|---|---|---|---|
| Composite index | app/composite/ |
normalization.py · scoring.py |
§14.4 |
| Corroboration | app/corroboration/ |
score.py |
§14.2 |
| Telling divergence | app/disagreement/ |
tellings.py |
§14.3 |
| Lead-time gate | app/divergence/ |
config.py |
§14.8 |
| Country Instability Index | app/cii/ |
scoring.py |
§14.9 |
| Prediction journal | app/journal/ |
emit.py |
§15.10 |
| Evaluations | app/onset/ · app/within/ · app/baselines/ |
n/a | §15 |
| Retention | app/housekeeping.py |
n/a | §11.3 |
This handbook is the entry point. The files below are the primary records it draws on. Read the protocol before the result, which is the order they were written in.
| Document | What it is |
|---|---|
docs/methodology.md |
Part A: the pre-registered evaluation protocol: ground truth, splits, baselines, metrics, sensitivity programme, reporting checklist. Part B: the literature baseline with citations and reading priority. |
docs/onset-eval.md |
The onset evaluation, frozen 2026-07-10, with its single run and amendment log (§15.6). |
docs/within-country-eval.md |
The within-country evaluation, frozen 2026-07-22, and its NEGATIVE verdict (§15.7). |
docs/disagreement-exam.md |
A forward evaluation, not gradable until enough predictions mature. |
docs/backtest/ |
Lead-time gate reports, including threshold sensitivity (§15.8). |
docs/severity-grading.md |
How headline severity is decided, which model, and the measured agreement (§14.5). |
docs/analytical-agenda.md |
The workstreams: what is actually done with the data: quantify, validate, predict. |
docs/project-direction.md |
What the project is, who it serves, and the long-term path. |
docs/data-coverage.md |
The operational record of what actually landed in storage (§16.3). |
docs/storage.md |
Storage layout, retention, move, back up, restore, wipe (§11). |
docs/acled-non-api-collection.md |
How the label data is obtained without an API. |
docs/security.md |
Repository security baseline: what is enabled, what hardening remains (§25.4). |
docs/architecture/ |
The build specification: 01 overview · 02 storage · 03 ingestion · 04 schema · 05 originality · 06 validation · 07 risks |
docs/architecture/CII-METHODOLOGY.md |
Per-country baseline and the four-component event blend (§14.9). |
docs/architecture/ENRICHMENT-METHODOLOGY.md |
Sentiment, NER, city resolution, news-scope classification. |
docs/audits/ · docs/frontend/ |
Hand-checked audits; console design notes. |
A candid read of where the project stands, kept with the code so it stays current rather than becoming a pitch.
| Helpful | Harmful | |
|---|---|---|
| Internal | Strengths | Weaknesses |
| External | Opportunities | Threats |
- A pre-registration and verdict machine that structurally cannot flatter itself: protocols frozen first, verdicts computed in code, negatives published in full, none buried (§0.3).
- Reproducible, local-first, no cloud. Idempotent backfills, every method version-stamped and never edited in place.
- Genuine multi-modal ingestion with structural deduplication and published coverage bias rather than a disclaimer (§16).
- The corroboration and divergence engine works today and is independent of the composite claim that failed.
- The headline claim failed. The composite beats none of the single-domain baselines, on either window (§15.5).
- The running version has never been evaluated. Every published result
grades
v1.0; the live system emitsv3.0(§14.1). - No forward evidence yet: 1,695 predictions issued, none graded (§15.10).
- The live composite is degenerate at 0.5 from the retention-versus-z-score mismatch, and the label panel is maintained by hand.
- Five of six declared robustness tests have not been run (§15.9).
- Built by one maintainer with heavy assistance from language models; depth of understanding of the internals is the standing risk, and this handbook plus §14 is the mitigation.
- Slow-onset hazards (drought, flood, sustained unrest) are the untested anchor where a sensor could plausibly lead coverage. An open question, not a settled failure.
- The hazard domain is the strongest single indicator measured so far and beats the composite that contains it. Making it survive a fair onset evaluation is a well-defined next task.
- The corroboration and coverage engine has standalone value: it answers "is this independently reported" without needing the composite to work.
- Upstream drift: GDELT gaps, label-source access changes, RSS format changes, and sensor values that turn out to measure something other than what they appear to.
- Retention versus evaluation needs can silently flatten a signal before it is ever measured: the class of defect that produced 1,101 forecasts of a constant.
- The hardest questions to answer are the within-country construction and the 0.5 degeneracy, both of which the composite's published results depend on.
Start here: §0 for what this claims and refuses to claim, Quick start to run it, and HANDBOOK.md for everything else.