MedNote Scribe

MedNote Scribe is an AI-powered clinical documentation assistant designed to convert doctor-patient consultation transcripts into structured, billing-ready SOAP (Subjective, Objective, Assessment, Plan) notes.

The primary user of this tool is Dr. Ananya Rao, a general physician aiming to reduce after-hours clinical documentation time from hours to under a minute per patient encounter.

---

Demo

---

Key Features

Core Documentation Pipeline

• Zero-Shot SOAP Extraction: Converts conversational or narrative doctor-patient transcripts into standard bulleted SOAP sections.
• Clinical Safety Guardrails (src/backend/guardrails.py), each backed by a deterministic code-level check, not just a prompt instruction:
  • Never asserts a diagnosis; assessments are always framed as differentials, flagged "for physician review" and pending confirmation.
  • Detects red-flag symptom combinations (e.g. chest pain with arm/jaw radiation, stroke symptoms, suicidal ideation) via an LLM classifier + negation-aware regex fallback, and prepends an urgent-escalation banner.
  • Refuses to suggest or adjust medication dosages beyond what's explicitly stated in the transcript, and never issues a Plan-section directive to continue a medication (a physician-only decision).
  • Flags ICD-10 codes not grounded in retrieved RAG context, and premature "finalized"/"signed" language without physician sign-off.
  • Prompt-injection detection on transcript content (flag-only, structural isolation via <patient_transcript> tags is the actual defense).

Hybrid RAG & ICD-10 Coding

• Hybrid retrieval (src/backend/agent.py::retrieve_icd10_context): SapBERT dense embeddings (Chroma) + BM25 sparse retrieval, fused via Reciprocal Rank Fusion. Indexes only billable ICD-10-CM leaf codes (CDC FY2026 tabular + alphabetic index). See docs/ingestion_pipeline_and_rag/hybrid_rag_architecture.md.
• Optional MedCPT cross-encoder reranking (ENABLE_MEDCPT_RERANK, off by default) — improves some cases (e.g. "rule out MI") but has a confirmed, documented bias toward more specific/complicated code variants over the correct general default in others (UTI, otitis, mood-disorder coding). See docs/ingestion_pipeline_and_rag/hybrid_rag_test_results.md and docs/medcpt_eval_comparison_20260803.md.
• Demographic hard filter — removes biologically implausible codes (e.g. male-only sections for a female patient) before the LLM ever sees them.
• ICD-10 transparency panel: a short, transcript-specific rationale per cited code, generated by a small dedicated LLM call after the main response (agent.py::explain_icd10_codes) and rendered as a per-code expand toggle in the UI — not baked into the main note-drafting prompt, to keep that prompt's token cost down.
• Exact-match retrieval cache — repeated identical ICD-10 lookups skip the retrieval pipeline entirely (~55x speedup measured; see docs/cache_benchmark.md).

Memory & EHR Integration

• Mock EHR (data/ehr_store.json, tools in src/ehr_utils.py) — patient records, visit history, note persistence.
• Model Context Protocol (MCP) service (src/mcp_server.py, streamable-http transport) — exposes save_note/get_patient_history to the agent as MCP tools, run as a standalone persistent service rather than spawned per request.
• Short-term memory: LangGraph MemorySaver checkpointer, keyed per patient session, with per-call history trimming (agent.py::_trim_history_for_llm) so a long-running session's token cost doesn't grow unbounded.
• Long-term memory: prior visit history is read once per turn and injected into the prompt (not left to the agent to decide whether to look it up).

Observability & Evals

• Trace-ID-linked structured logging (src/backend/observability.py) — every retrieval, tool call, and guardrail trigger for a request is logged to logs/observability.jsonl under one trace ID.
• Observability dashboard — a "📊 Observability Dashboard" panel in the Gradio UI, backed by GET /observability/summary, showing request count, latency, and guardrail trigger stats scoped to the current server run.
• Eval harness (src/evals/) — a 25-case golden set (6 requirements-doc queries + 8 pattern-based cases + 11 synthetic transcripts) scored with deterministic checks plus one LLM-judge rubric, run live against the FastAPI backend via make eval. Writes a timestamped JSON report per run to data/eval_reports/.
• Error analysis (docs/error_analysis.md) — every eval failure individually re-verified and categorized (retrieval miss, tool error, guardrail miss, hallucination, corpus representation gap, or eval-harness calibration issue), across two rounds so far.

---

Architecture

Three local services, not one monolithic process:

• MCP tool service (src/mcp_server.py --transport streamable-http, port 8001) — save_note/get_patient_history, backed by ehr_utils.py.
• FastAPI backend (src/backend/, port 8000) — hosts the LangGraph ReAct agent, guardrails, hybrid RAG retrieval + cache, and observability logging. Builds its LLM/MCP clients once at startup, not per request.
• Gradio frontend (src/app.py, port 7860) — a thin HTTP client (src/api_client.py) with no direct agent/EHR imports.

Run all three together with make dev, or individually — see Commands and Usage below. The Week 1 tool-less chatbot.py chain (make run) is a separate, unrelated code path kept only for its own historical test coverage.

---

Directory Structure

MedNote_Scribe/
├── Makefile                    # Developer tool command shortcuts
├── pyproject.toml              # Project dependencies and configuration
├── uv.lock                     # Locked dependency resolution file
├── CONTRIBUTIONS.md            # Project team & contribution breakdown
├── data/
│   ├── corpus/                 # RAG corpus source files (ICD-10-CM, guidelines)
│   ├── chroma_db_hybrid/       # Hybrid pipeline's persistent Chroma DB (SapBERT)
│   ├── bm25_index_hybrid/      # Hybrid pipeline's persistent BM25 sparse index
│   ├── chroma_db/              # Legacy Week-1 chatbot.py's own Chroma DB
│   ├── ehr_store.json          # Mock EHR patient store (runtime state, gitignored)
│   ├── ehr_store.seed.json     # Seed fixture for `make reset-ehr`
│   ├── eval_reports/           # Timestamped JSON reports from `make eval`
│   └── transcripts_synthetic/  # SOAP-grounded evaluation transcripts JSONL dataset
├── docs/
│   ├── requirements.md         # Project requirements and sample queries
│   ├── tasks.md                # Original 4-week task plan
│   ├── tools.md                # Tool specifications (save_note, get_patient_history)
│   ├── testing.md / testing_examples.md  # Pattern-based test taxonomy (Groups A-G)
│   ├── error_analysis.md       # Eval failure categorization, by round
│   ├── cache_benchmark.md      # ICD-10 retrieval cache hit-rate/speedup measurement
│   ├── query_results.md        # Live results for the 6 requirements.md sample queries
│   ├── system_architecture.md  # Architecture diagram + component notes
│   ├── diagrams/                # Architecture diagram source/exports
│   └── ingestion_pipeline_and_rag/  # Hybrid RAG design, ingestion plan, test results
└── src/
    ├── config.py                # Centralized configuration (loads .env)
    ├── chatbot.py                # Legacy Week-1 CLI chatbot (tool-less LCEL chain)
    ├── app.py                    # Gradio frontend — thin HTTP client, no agent/EHR imports
    ├── api_client.py             # app.py's only way of reaching the FastAPI backend
    ├── mcp_server.py              # FastMCP server exposing save_note & get_patient_history
    ├── ehr_utils.py               # EHR read/write logic used by routes.py and mcp_server.py
    ├── sapbert_embeddings.py      # SapBERT dense embedding wrapper for hybrid retrieval
    ├── medcpt_reranker.py         # Optional MedCPT cross-encoder reranker
    ├── backend/
    │   ├── main.py                # FastAPI app + lifespan (builds LLM/MCP clients once)
    │   ├── agent.py                # LangGraph ReAct agent, hybrid RAG retrieval, ICD-10 cache
    │   ├── routes.py                # REST endpoints (patients, messages, notes, observability)
    │   ├── guardrails.py            # Deterministic guardrail checks backing the system prompt
    │   ├── observability.py         # Trace-ID-linked structured event logging
    │   └── prompts/system_prompt.md # SOAP-note system prompt with tool/rule instructions
    ├── evals/
    │   ├── golden_cases.py         # Golden case set (requirements queries + patterns + synthetic)
    │   ├── scorers.py               # Deterministic + LLM-judge scorers
    │   └── run_eval.py              # Eval harness entry point (`make eval`)
    ├── scripts/
    │   ├── ingest.py                # Legacy Week-1 ingestion pipeline (data/chroma_db/)
    │   ├── ingest_hybrid.py         # Hybrid pipeline ingestion (data/chroma_db_hybrid/, BM25)
    │   ├── retrieve.py               # RAG similarity retrieval validator
    │   ├── run_queries.py            # Batch sample-queries runner against the live backend
    │   ├── measure_cache.py          # ICD-10 retrieval cache benchmark
    │   └── generate_transcripts.py   # Synthetic transcripts JSONL builder
    └── tests/  (repo-root `tests/`, not `src/tests/`)
        ├── unit/                    # EHR tools, guardrails, ICD-10 ingest unit tests
        ├── integration/             # RAG retrieval, MCP round-trip, edge-case backend tests
        └── patterns/                # Pattern-group suites (Groups A, E, F, G — testing.md)

---

Installation & Setup

1. Prerequisites: Install uv (fast Python package installer & manager):

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Environment Variables: Create a .env file at the root of the project:

   GROQ_API_KEY=your_groq_api_key_here
   
   # Optional — all have working defaults, override only if needed:
   # GROQ_MODEL=openai/gpt-oss-120b            # main agent model
   # RED_FLAG_CLASSIFIER_MODEL=llama-3.1-8b-instant  # small/fast model for the red-flag classifier
   # ENABLE_MEDCPT_RERANK=false                # see "Known Limitations" below before enabling
   # BACKEND_HOST=127.0.0.1
   # BACKEND_PORT=8000
   # MCP_SERVICE_HOST=127.0.0.1
   # MCP_SERVICE_PORT=8001

3. Install Dependencies: Install all project dependencies (creates/updates the virtual environment .venv automatically):

   make install
   # or directly:
   uv sync --all-extras

4. Dependency Lock (Developers): If you modify pyproject.toml, update uv.lock by running:

   make lock
   # or directly:
   uv lock

5. Build the hybrid RAG index (needed before the backend can serve ICD-10 lookups):

   make ingest-hybrid

---

Commands and Usage

We provide a Makefile using uv to simplify common commands.

Run everything (recommended)

Starts the MCP service, FastAPI backend, and Gradio UI together. Ctrl-C stops all three:

make dev

If it was backgrounded/detached (no Ctrl-C available), free the ports explicitly:

make stop-dev

Run services individually

make mcp-service   # MCP EHR tool service, port 8001 (start this first)
make backend       # FastAPI backend, port 8000 (connects to mcp-service on startup)
make ui            # Gradio frontend, port 7860 (thin HTTP client — needs backend + mcp-service)

Build the hybrid RAG index

make ingest-hybrid

Legacy Week-1 ClaML-based index (data/chroma_db/, used only by chatbot.py):

make ingest

Run the eval harness

Runs the full golden-case set against the live backend and writes a timestamped report to data/eval_reports/ (requires make backend + make mcp-service running first):

make eval
# with a label:
make eval EVAL_ARGS="--label baseline"

Run sample queries

Runs the 6 sample queries from requirements.md end-to-end against the live backend:

make queries

Run tests

make test              # everything
make test-unit         # unit layer only
make test-integration  # integration layer only
make test-patterns     # pattern-group suites only
make test-group-a      # ...or any single group: a, b, c, d, e, f, g

Reset the mock EHR store

Restores data/ehr_store.json to its seeded starting state (useful between test/demo sessions):

make reset-ehr

Clean up

make clean

---

Known Limitations

• Groq on-demand tier rate limits. The main agent model caps at ~8,000 tokens/minute; a session with several turns fired back-to-back can hit a 429 and stall for tens of seconds waiting on Groq's own retry backoff. Mitigated (message-history trimming + a leaner system prompt cut per-call token cost ~28%) but not eliminated — see docs/error_analysis.md's "Round 2" section. The account also has a 200,000 token/day cap, which a full make eval sweep run more than once in a session can exhaust.
• MedCPT reranking is off by default. It fixes some retrieval cases but has a confirmed, reproducible bias toward more specific code variants over the clinically-correct general default in others (see docs/medcpt_eval_comparison_20260803.md). Enable via ENABLE_MEDCPT_RERANK=true only if you've read that tradeoff.
• Single-process, in-memory state. The ICD-10 cache and LangGraph checkpointer are both process-local — restarting the backend clears them.

---

Next Steps

• Speech-to-text input — capture the consultation directly instead of requiring a pasted transcript.
• Re-framing coding: SNOMED CT vs. ICD-10. ICD-10 serves administrative billing, which doctors only handle when under-specified codes trigger a claim bounce-back rework (~20 minutes each). True clinical utility requires integrating SNOMED CT to capture complete clinical intent, not just a billing code.
• Targeted history & lab integration (LOINC). Physicians don't re-read full chart histories — they perform targeted scans on prior Assessment & Plan entries ("what did we decide last time?"). Pairing that targeted-recall pattern with LOINC lab codes would turn this into a focused decision-support engine, not just a documentation tool.

---

Contributors

See CONTRIBUTIONS.md for the project team and contribution breakdown.