Skip to content

feat: AI exposure shows the published score for each role, or says there isn't enough data #81

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
coderback wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: AI exposure shows the published score for each role, or says there isn't enough data #81

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Built with FastAPI, Next.js, [browser-use](https://github.com/browser-use/browse

1. **Accounts & Persona** — sign up, then build a rich persona (contact details, preferences, skills, work history, education, projects, certifications, screening answers, behavioural stories, custom directives).
2. **Upload a CV (PDF)** — parsed into structured data and merged into your persona; the PDF is kept so the browser agent can attach it.
3. **AI Exposure Risk** — scores each skill and job title against the Anthropic Economic Index / O\*NET task-penetration data (~18 000 occupational tasks), with LLM-generated reasoning.
3. **AI Exposure** — shows the Anthropic Economic Index's published score for the occupation each job or target role belongs to, or "not enough data".
4. **Job Feed** — searches a local job store with real filters (search terms, role/domain tags, level, location, remote, visa sponsorship, experience asked for, degree requirement) and ranks results by semantic similarity to your persona. When results are thin, live discovery across job boards runs in the background and the feed fills in as it finishes.
5. **Paste any job link** — scrape an arbitrary listing into the same analysis flow.
6. **Per-job analysis** — fit score (LLM semantic match) and AI risk for a single role.
Expand Down Expand Up @@ -79,11 +79,12 @@ The API refuses to start without `JWT_SECRET` and `CREDENTIALS_KEY`.

### 2. Data files

The economic index and task-penetration data must be present in `./data/` (bind-mounted read-only at `/data`):
The economic index and occupation data must be present in `./data/` (bind-mounted read-only at `/data`):

```
data/economic_index.json — Anthropic Economic Index (O*NET occupation exposure scores)
data/task_penetration.csv — O*NET task-level AI penetration scores (~18 000 rows)
data/economic_index.json — Anthropic Economic Index (published exposure score per occupation)
data/occupation_titles.csv — O*NET job titles and the occupation each belongs to
data/occupation_evidence.json — tasks people hand to AI most and least, per occupation
```

### 3. Run
Expand Down Expand Up @@ -168,13 +169,13 @@ Long-running endpoints stream **Server-Sent Events (SSE)**. Protected endpoints

## AI Exposure Risk Scoring

Scores are deterministic lookups against O\*NET data; the LLM only adds human-readable reasoning afterwards. `score_ai_risk` resolves each skill/title in this order (`agent.py`, `risk_scorer.py`):
Every score is the figure the Anthropic Economic Index publishes for an occupation; nothing is estimated and no model is involved (`risk_scorer.py`). There are no per-skill scores, because the index has none.

1. **Curated exposure table** — a hand-tuned map of common tech skills and job titles.
2. **Keyword task search** — average penetration of all O\*NET tasks containing the term.
3. **TF-IDF semantic match** — cosine similarity against ~1 350 non-zero-penetration task descriptions; requires all content words to appear.
4. **O\*NET word-overlap** — stems and noise-strips the title, then matches the occupation with the most shared content words.
5. **Default 0.5** — neutral when no data differentiates.
1. **Title to occupation** — words that say nothing about the job (graduate, 2027, programme) are dropped, then the title is looked up in O\*NET's list of job titles, plus a short list for titles O\*NET leaves out or spreads across occupations (backend, DevOps, ML engineer). A longer title can contain a known one; if it contains two from different occupations it isn't scored.
2. **Not enough data** — shown when a title doesn't clearly belong to one occupation, or when the index publishes a zero, which means too little usage was seen rather than that the job is safe.
3. **Read at request time** — job scores aren't taken from the stored column, so a data update applies to every job at once.

The score isn't a forecast: it comes from 2025 Claude usage and US occupation data, and Anthropic found only a slight link to projected job growth. Jobs aren't ranked by it. The roadmap gets a short summary of the tasks people hand to AI most and least in the target occupation (one month of chat use) as a hint, and is told not to quote its figures.

Labels: **High ≥ 65 %**, **Medium 35–65 %**, **Low < 35 %**. **Job fit** (`score_job_fit`) is an LLM semantic analysis returning `fit_score` (0–100), `level_match` (`strong`/`ok`/`reach`), `matched_skills`, `skill_gaps` and a one-line rationale.

Expand Down Expand Up @@ -235,7 +236,8 @@ Labels: **High ≥ 65 %**, **Medium 35–65 %**, **Low < 35 %**. **Job fit** (`s
| File | Description |
|---|---|
| `data/economic_index.json` | 756 O\*NET occupations → `overall_exposure` (0–1). Source: Anthropic Economic Index. |
| `data/task_penetration.csv` | ~18 000 O\*NET task descriptions → `penetration` score. |
| `data/occupation_titles.csv` | ~62 000 job titles → occupation. From O\*NET 30.2 (U.S. Department of Labor, Employment and Training Administration), used under CC BY 4.0. Built by `scripts/build_occupation_titles.py`. |
| `data/occupation_evidence.json` | 103 occupations → the tasks handed to AI most and least, from the index's chat usage. Built by `scripts/build_occupation_evidence.py`. |
| `api/app/jobs_fixture.json` | Mock job listings shown only when every live source is empty. |
| `applications.db` (SQLite, `/appdata`) | `users` (bcrypt password hashes, persona JSON, encrypted job-site password, onboarding flag) + `applications`. |
| `jobs.db` (SQLite, `/appdata`) | Cached jobs with semantic vectors, filled by the scheduled collection and the feed's discovery; jobs no source has listed for 3 days are expired. Also the collection runs and the sponsor register. |
Expand Down
112 changes: 17 additions & 95 deletions api/app/agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from .company_names import normalise_company_name as _normalise_company_name
from . import http_client
from . import tracker
from .risk_scorer import lookup_by_title
from .risk_scorer import target_evidence
from .browser_agent import apply_with_browser

logger = logging.getLogger("windrush.agent")
Expand Down Expand Up @@ -102,42 +102,12 @@ def _chat_params(**params) -> dict:
return out


_HARDCODED_EXPOSURE: dict[str, float] = {
# Programming languages
"python": 0.78, "javascript": 0.71, "typescript": 0.68, "java": 0.69,
"c++": 0.62, "c#": 0.65, "go": 0.60, "rust": 0.55, "r": 0.76,
"matlab": 0.72, "scala": 0.63, "kotlin": 0.64, "swift": 0.61,
# AI / ML
"machine learning": 0.82, "deep learning": 0.79, "nlp": 0.84,
"natural language processing": 0.84, "computer vision": 0.81,
"tensorflow": 0.75, "pytorch": 0.76, "keras": 0.74,
"large language models": 0.88, "llm": 0.88, "generative ai": 0.86,
"reinforcement learning": 0.77, "neural networks": 0.80,
"data science": 0.81, "data analysis": 0.80, "statistics": 0.74,
# Web / backend
"react": 0.65, "node.js": 0.67, "django": 0.66, "fastapi": 0.64,
"rest api": 0.70, "graphql": 0.67, "sql": 0.72, "nosql": 0.68,
"postgresql": 0.70, "mongodb": 0.67, "redis": 0.63,
# DevOps / infra
"docker": 0.58, "kubernetes": 0.52, "aws": 0.61, "azure": 0.60,
"gcp": 0.59, "terraform": 0.54, "ci/cd": 0.56, "git": 0.55,
"cloud computing": 0.60, "microservices": 0.58,
# Lower-exposure / hardware / specialised
"fpga": 0.32, "embedded systems": 0.40, "hardware design": 0.35,
"robotics": 0.48, "signal processing": 0.44, "compiler design": 0.38,
"systems design": 0.38, "distributed systems": 0.42,
"cryptography": 0.36, "security": 0.41, "networking": 0.43,
# Soft / domain
"research": 0.45, "technical leadership": 0.28, "agile": 0.42,
"project management": 0.50, "communication": 0.30,
# Job titles
"software engineer": 0.62, "software developer": 0.65,
"data engineer": 0.72, "ml engineer": 0.80, "ai engineer": 0.82,
"graduate software engineer": 0.62, "backend engineer": 0.64,
"frontend engineer": 0.66, "full stack engineer": 0.65,
"data scientist": 0.81, "research engineer": 0.58,
"devops engineer": 0.55, "platform engineer": 0.53,
}
def _evidence_prompt(evidence: dict) -> str:
tasks = lambda key: "; ".join(t["task"] for t in evidence[key])
return (f"Usage evidence for {evidence['occupation']} (Anthropic Economic Index, chat conversations, {evidence['month']}):\n"
f"Handed to AI most: {tasks('most_handed_to_ai')}\n"
f"Handed to AI least: {tasks('least_handed_to_ai')}\n")


async def _llm(system: str, user: str, max_tokens: int = 2048) -> str:
"""Simple single-turn LLM call for internal tool use (CV parsing, cover letter, roadmap)."""
Expand Down Expand Up @@ -1113,48 +1083,6 @@ async def execute_tool(name: str, tool_input: dict) -> dict:
)
return data or {}

elif name == "score_ai_risk":
skills = tool_input.get("skills", [])
if not skills:
return {"skill_risks": []}

# First, get raw scores from our index/hardcoded list
raw_results = []
for skill in skills:
key = skill.lower().strip()
if key in _HARDCODED_EXPOSURE:
exposure = _HARDCODED_EXPOSURE[key]
else:
data = lookup_by_title(skill)
raw = data.get("overall_exposure", 0.0)
exposure = round(raw, 2) if raw and raw != 0.5 else 0.55

raw_results.append({
"skill": skill,
"exposure_score": round(exposure * 100),
"risk_label": "High" if exposure >= 0.65 else ("Medium" if exposure >= 0.35 else "Low")
})

# Use LLM to generate reasoning for these scores
text = await _llm(
system=(
"You are an AI workforce expert. Provide a concise, one-sentence reasoning for the AI exposure risk of each skill. "
"Focus on why the skill is either easily automated or requires human-centric reasoning/physicality. "
"Return ONLY valid JSON with a 'skill_risks' array. "
"Each object MUST have: skill, exposure_score, risk_label, and reasoning. "
"Keep the scores/labels from the input, just add reasoning. "
"Output ONLY JSON."
),
user=f"Skills and Scores: {json.dumps(raw_results)}",
max_tokens=2048,
)

text = text.strip().removeprefix("```json").removeprefix("```").removesuffix("```").strip()
try:
return json.loads(text or f'{{"skill_risks": {json.dumps(raw_results)}}}')
except json.JSONDecodeError:
return {"skill_risks": raw_results}

elif name == "score_job_fit":
jobs = tool_input.get("jobs", [])
persona = tool_input.get("persona", {})
Expand Down Expand Up @@ -1386,17 +1314,12 @@ async def execute_tool(name: str, tool_input: dict) -> dict:
}

elif name == "generate_skill_roadmap":
skill_risks = tool_input.get("skill_risks", [])
persona = tool_input.get("persona", {})
# Without an explicit target, plan for the roles the user is aiming at, not their last employer's industry.
target_job_title = tool_input.get("target_job_title") or ", ".join(
(persona.get("preferences") or {}).get("target_titles") or [])

confirmed_high = [s for s in skill_risks if s.get("exposure_score", s.get("exposure", 0) * 100) > 65]
confirmed_low = [s for s in skill_risks if 0 < s.get("exposure_score", s.get("exposure", 0) * 100) <= 35]
all_skills_str = ", ".join(
f"{s['skill']} ({s.get('exposure_score', round(s.get('exposure', 0) * 100))}%)" for s in skill_risks
)
target_titles = (persona.get("preferences") or {}).get("target_titles") or []
target_job_title = tool_input.get("target_job_title") or ", ".join(target_titles)
evidence = target_evidence([tool_input["target_job_title"]] if tool_input.get("target_job_title") else target_titles)
all_skills_str = ", ".join(s for group in persona.get("skills", []) for s in group.get("skills", []))

core = persona.get("core_info", {})
name_str = f"{core.get('first_name', '')} {core.get('last_name', '')}".strip() or "the candidate"
Expand Down Expand Up @@ -1427,8 +1350,10 @@ async def execute_tool(name: str, tool_input: dict) -> dict:
"STRICT RULES: "
"1. Plan for the TARGET role when one is given: every recommendation must make the candidate more competitive for that role across the employers who hire for it. Past employers' industries are background only — a short internship or placement does not define the candidate's industry. "
"2. Build ON their existing technical strengths — deepen and extend, do not replace them. "
"3. For each skill at confirmed high AI risk, suggest a way to apply it at a level AI cannot yet reach "
"(novel research, production systems, domain-specific expertise, technical leadership). "
"3. If usage evidence is given, favour work people hand to AI least over work they hand to it most, and "
"apply their strengths above the level of the tasks handed over most (novel research, production systems, "
"domain-specific expertise, technical leadership). The evidence is one month of chat use and generic task "
"descriptions, so treat it as a hint: never quote its figures or call a skill safe or at risk. "
"4. Do NOT recommend generic soft skills (leadership, communication, PM) unless the candidate is "
"already working in a non-technical or hybrid role. "
"5. Do NOT recommend skills the candidate already lists. "
Expand All @@ -1441,9 +1366,8 @@ async def execute_tool(name: str, tool_input: dict) -> dict:
f"Experience: {exp_str}\n"
f"Current job titles: {', '.join(job_titles)}\n"
f"Target role: {target_job_title or 'not specified'}\n\n"
f"All skills with AI exposure scores: {all_skills_str}\n"
+ (f"Confirmed high AI-exposure (score > 65%): {', '.join(s['skill'] for s in confirmed_high)}\n" if confirmed_high else "")
+ (f"Confirmed lower AI-exposure (safer foundation): {', '.join(s['skill'] for s in confirmed_low)}\n" if confirmed_low else "")
f"Skills they already list: {all_skills_str}\n"
+ (_evidence_prompt(evidence) if evidence else "")
+ "\nGenerate 6 skill development recommendations that deepen this candidate's technical profile and make them more competitive for their target role."
),
max_tokens=2048,
Expand Down Expand Up @@ -1625,7 +1549,6 @@ async def run_apply(
job_id: str,
job_url: str = "",
cover_letter: str = "",
skill_risks: list | None = None,
session_id: str = "",
instruction_queue: asyncio.Queue | None = None,
frame_queue: asyncio.Queue | None = None,
Expand All @@ -1637,7 +1560,6 @@ async def run_apply(
) -> AsyncGenerator[str, None]:
"""Apply phase: browser automation. `answerer` answers the form's questions (see answers.py)."""
persona = tracker.get_user_persona(user_id)
skill_risks = skill_risks or []

yield _sse("start", {"message": "Starting browser application with your Persona…", "session_id": session_id})

Expand Down
2 changes: 1 addition & 1 deletion api/app/discovery.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
_finished_at: dict[_Pair, float] = {}

# The live search runs in worker *processes*: it fans out to ~70 job boards and does a lot of
# CPU work on the results (multi-MB JSON, HTML-to-text, exposure scoring). In threads that
# CPU work on the results (multi-MB JSON, HTML-to-text). In threads that
# work contends for the GIL with the event loop and stalls the API for ~1s at a time; in a
# separate process it can't. 'spawn' avoids forking a process that has live threads.
_pool: ProcessPoolExecutor | None = None
Expand Down
Loading
Loading