⚠️ Archived 2026-09-10 — 프로젝트 종료(웹). 앱 레포: CLARA_Call
Daily multimodal capture — voice and video together — and longitudinal reasoning across time. A simulation-powered framework that proves continuous AI monitoring saves lives, then deploys it as a real-world application.
Live Demo Page
CLARA voice call
Quick Start · HAI-DEF Models · Notebooks · Results · Technical Details · Deployment
In oncology trials, clinical monitoring relies heavily on clinic visits scheduled every 12–21 days for patient safety. During this gap, nobody assures the safety of the patient on the trial.
- 400,000+ oncology trial participants are affected by these blind spots globally each year (Izarn et al., ESMO Open, 2025)
- Physician–patient AE grade agreement is only ~50% (Basch et al., Lancet Oncol., 2006)
- Physicians routinely fail to capture 57% of AEs (Di Maio et al., J Clin Oncol., 2015)
- Self-reporting rates for Grade 1–2 Adverse Events (AEs) are exceptionally low, hovering around ~2%
Most patients tend to underreport their discomfort. Missing these implicit signals can lead to severe repercussions—including grade escalation, emergency hospitalization, and loss of life. The delayed attention will also affect the effectiveness of clinical trials if forced dose interruptions occur.
CLARA simulates the complex dynamics of clinical trials, emulating hundreds of patients' daily health statuses and their psychological tendency to underreport symptoms, then quantifies the measurable impact of proactive daily monitoring on patient outcomes.
Drug name in → Simulated trial out → Evidence that daily monitoring works → Deploy as real-world app
CLARA consists of two MedGemma-based agents. We transitioned them from simulation to a production mobile application powered by MedGemma and HAI-DEF models, enabling continuous clinical visibility between visits.
 DCA |
Data Collection Agent (DCA) — Runs on the patient's device. Conducts daily ~60-second CLARA Call check-ins via video + voice interaction. Extracts structured clinical signals through multimodal processing: voice → context representation (MedGemma), audio → cough classification (HeAR), video → on-device feature map → server-side classification (MedSigLIP). Operates strictly within nursing scope: detect, report, refer. |
 DAA |
Data Analysis Agent (DAA) — Runs on the platform. Transforms daily signals into continuous patient timelines with AE/SAE flagging, auto-generated briefings, and report generation (CDISC-compliant CRF records, SAE narratives, FDA MedWatch 3500A, E2B XML). Powered by MedGemma 4B fine-tuned with RLFR (Reinforcement Learning Feature Reward), reducing hallucination rate from 37.3% to 6.7%. |
Try the app: The DCA is available as a standalone mobile application — see CLARA Call.
Live demo: clara.parrotvox.com
Each HAI-DEF model serves a distinct clinical role within CLARA:
| Model | Base Model |
|---|---|
| MedGemma 1.5 4B | DCA — powers the conversational AI medical staff during daily CLARA Call check-ins, generating clinically appropriate questions and assessments |
| MedSigLIP | DCA — medical vision encoder that provides image representations for AE symptom detection from CLARA Call video frames |
| HeAR | DCA — health audio representation model that extracts embeddings for respiratory sound analysis during CLARA Call |
| MedASR | DCA — medical terminology-aware speech recognition for patient speech transcription during CLARA Call |
| Model | Base Model | Role in CLARA | Details |
|---|---|---|---|
| MedGemma 1.5 4B + RLFR | MedGemma 1.5 4B | DAA — whenever a SAE occurs, autonomously generates reports in FDA MedWatch and E2B XML format | MMLU Medical Acc 60.7% → 64.3% Hallucination rate 37.3% → 6.7% |
| MedGemma AE Detection | MedGemma 1.5 4B Anti-Hallucination | DCA — compares baseline and current patient photographs captured during CLARA Call to detect and grade new adverse events using CTCAE criteria | Acc 23.8% → 90.5% |
| HeAR Classifier Head | HeAR | DCA — classifies cough segments from CLARA Call audio, distinguishing dry from wet cough via 2-stage pipeline | Fine-tuned on 956 randomly sampled recordings (478 dry / 478 wet) from the COUGHVID dataset, tested on Gemini-generated synthetic audio |
| MedSigLIP Classifier Head | MedSigLIP | DCA — processes video frames captured during CLARA Call to detect visible AE symptoms (e.g., rash, swelling) and classify their CTCAE grade | MLP classifier head on frozen encoder; Trained on 210 Gemini-generated synthetic patient images (147/21/42 split) Acc 26.2% → Acc 61.9% |
CLARA ships with 5 Jupyter notebooks demonstrating each component:
| # | Notebook | What It Covers | GPU |
|---|---|---|---|
| 1 | medgemma_anti-hallucination |
RLFR fine-tuning — hallucination probe training and RL optimization | ~10 GB |
| 2 | medgemma+medsiglip+HeAR_SAE-detection |
Multimodal AE detection — vision (MedSigLIP) + audio (HeAR) pipelines | ~20 GB |
| 3 | simulate-clinical-trial |
End-to-end trial simulation — rule discovery, patient generation, daily loop | None (API) |
| 4 | application_voice_call |
CLARA Call voice call demo — MedASR + TTS + nurse conversation | ~10 GB |
| 5 | application_SAE_report_generation |
FDA MedWatch 3500A + E2B XML report generation from CRF data | ~10 GB |
Notebooks use notebooks/src/ as a local module providing shared source code — simulation engine, LLM agents, multimodal pipelines, ruleset generation, and the document agent.
All models and data download automatically from HuggingFace on first run:
- Models: newmes/medgemma-ae-detection, newmes/medgemma-4b-antihallu
- Data: newmes/clinical-trial-engine-data
CLARA runs A/B comparisons on identical cohorts (N=100, same seed): Group A (no agent, bi-weekly visits only) vs. Group B (same trial + CLARA Call daily monitoring).
Group A (Standard Visit-Based) Group B (+ CLARA Call)
No daily monitoring Daily ~60s video + voice check-ins
Hospital ──── 13 days ──── Hospital Hospital ── 3 days ── Early Visit ── Hospital
AE hidden AE detected dose adjusted
AE worsens early referral faster recovery
| Metric | Group A (Standard) | Group B (CLARA Call) | Improvement |
|---|---|---|---|
| AE Detection Delay (mean) | 4.6 days | 1.2 days | ↓74% |
| Discontinued (out of 100) | 21 | 17 | ↓19% |
| Deaths (out of 100) | 21 | 16 | ↓24% |
| Grade 3+ AE Duration (mean) | 1.4 days | 1.3 days | ↓7% |
| Metric | Base MedGemma 4B | + RLFR | Change |
|---|---|---|---|
| Hallucination Rate (MedHallu Hard) | 37.3% | 6.7% | -82% |
| MMLU Medical Score | 60.7% | 64.3% | +5.9% |
| Metric | Score |
|---|---|
| Accuracy | 90.5% |
| Weighted F1 | 90% |
| Training data | 210 Gemini-generated images (147/21/42 split) |
| Metric | Score |
|---|---|
| Accuracy | 61.9% |
| Weighted F1 | 57.1% |
| Training data | 210 Gemini-generated images (147/21/42 split) |
Why Simulation? Real-world daily multimodal longitudinal datasets do not yet exist at scale. So we built one — a rule-based simulation engine grounded in real clinical datasets and published drug safety profiles, validated against 7 known drug profiles.
CLARA operates through three mechanisms:
1. Data-Driven Ruleset Generation — Synthesizes historical clinical trial data from 10+ biomedical databases (DailyMed, ClinicalTrials.gov, Project Data Sphere, PubMed, DrugBank, OnSIDES, etc.) to automatically build simulation rulesets covering AE incidence distributions, demographics, efficacy endpoints, and dose modification protocols.
2. Realistic Daily Simulation with Information Asymmetry — The simulation maintains two strict data layers: a Ground Truth (GT) layer computing each patient's actual daily status, and a Hospital Record (HR) layer updated only at clinic visits (typically every 2 weeks). All treatment decisions are made from HR only — never from GT. This gap mirrors real-world blind spots.
3. Data Collection Agent — CLARA Call — Patients complete a short, ~60-second daily check-in. Voice becomes structured context, audio enables cough classification, and video is converted into feature maps for server-side classification. These signals are fused through MedGemma for longitudinal multimodal reasoning, producing structured clinical variables that medical teams can act on.
Each patient goes through this pipeline every simulated day:
| Step | Name | Description |
|---|---|---|
| 1 | Stochastic AE Onset | Hazard function determines new adverse events |
| 2 | AE Grade Transition | Active AEs worsen or improve based on cumulative toxicity |
| 3 | Tumor & RECIST Evaluation | Tumor response on scheduled scan days |
| 4 | Dose Modification | Hold/reduce/withdraw — hospital visit days only, HR-based |
| 5 | Lab Data Simulation | CBC, metabolic panel, liver/kidney function |
| 6 | Vitals Simulation | BP, heart rate, temperature, SpO2 |
| 7 | CDASH/CRF Mapping | Daily records mapped to CDISC clinical trial format |
| 8 | AE Cascade Update | Secondary AE chains triggered by primary events |
| 9 | Dynamic ECOG PS | Performance status adjusted by AE burden |
| 10 | Mortality Assessment | Life-threatening event evaluation |
| Principle | Why |
|---|---|
| LLM sets probabilities, code rolls dice | Prevents mode collapse; guarantees statistical distributions |
| Ground Truth vs. Hospital Record | Dose decisions use observed data only — never omniscient GT |
| Drug-agnostic | Change the drug name → system auto-discovers rules from 10+ databases |
| No fate table | Daily hazard functions replace pre-determined event timelines |
| CLARA Call detects, doctors decide | DCA flags and refers; only physicians modify treatment |
| Seed-reproducible | Same seed → identical simulation for scientific rigor |
The DCA conducts daily CLARA Call check-ins following a 4-turn protocol:
Turn 1: Patient describes how they feel (speech → MedASR transcription)
Turn 2: Nurse asks targeted follow-up questions (MedGemma conversation)
Turn 3: Patient shows affected area on camera (image → MedSigLIP classification)
Turn 4: Nurse summarizes and refers if needed (MedGemma → early visit recommendation)
Multimodal Pipeline:
Voice → Context Representation (MedGemma conversation)
Audio → Cough Classification (Dry/Wet/None) (HeAR audio classification)
Video → On-device Feature Map (MedSigLIP vision analysis)
→ Server-side Classification
Signal Fusion → MedGemma Reasoning → Structured Clinical Variables
Multimodal detection channels cover five categories:
Channel Examples Detection Method
────────────────────────────────────────────────────────────────
lab neutropenia, anemia Blood test (hospital only)
patient_reported nausea, pain, fatigue Patient self-report (mood-dependent)
video_detectable rash, alopecia, edema MedSigLIP vision analysis
audio_detectable cough, dyspnea HeAR audio classification
physical_exam neuropathy, hepatomegaly Doctor examination (hospital only)
The DAA transforms daily signals into continuous patient timelines with AE/SAE flagging, auto-generated briefings, and report generation.
We discovered that MedGemma 1.5 4B is vulnerable to hallucinations due to its multi-modal attribution. CLARA uses Reinforcement Learning Feature Reward (RLFR):
- Train a binary hallucination detection probe on MedGemma's hidden states
- Freeze the probe
- Use probe confidence as the reward signal for RL fine-tuning
This reduced the hallucination rate from 37.3% to 6.7% on MedHallu Hard while improving MMLU medical scores from 60.7% to 64.3%. In medical settings, grounded and stable outputs are not optional — they are essential.
Hazard Function
Instead of a pre-determined fate table, CLARA uses a mixture model to compute daily AE onset probability:
P(onset on day t | no onset before t) = I · f(t) / (1 − I · F(t−1))
Where:
I = patient-adjusted AE incidence rate
F(t) = onset CDF (Normal, LogNormal, or Uniform)
f(t) = F(t) − F(t−1) (discrete probability mass)
AE grade transitions use daily Markov probabilities:
- Base worsen rate: 1.5%/day (increases with cumulative toxicity)
- Base improve rate: 0.5%/day
- CLARA Call intervention: worsen ×0.3, improve ×3.0
Observation Model (GT ↔ HR)
The observation model implements a whitelist-based filter — HR only receives GT information through defined observation channels:
| Observation Point | What Gets Updated | Trigger |
|---|---|---|
| Scheduled visit | Full exam: labs, vitals, physical exam, AE assessment | Treatment cycle day |
| Scheduled scan | Tumor/RECIST data | Protocol-defined intervals |
| Self-report | Patient-reported AEs only | Mood-dependent probability |
| Video call (CLARA Call) | video_detectable + patient_reported AEs | Daily (CLARA Call mode) |
| ER visit | Full exam | Grade 4+ AE or dangerous vitals |
HR never falls back to GT. If a lab value wasn't measured, it stays stale.
7-Dimension Mood Model
Each virtual patient has a persistent psychological profile that evolves over time and directly impacts clinical data collection:
| Dimension | Effect on Simulation |
|---|---|
| Anxiety | High → over-reports symptoms, seeks ER visits early |
| Depression | High → reduced reporting motivation, missed calls |
| Fatigue | High → shorter video calls, less engagement |
| Irritability | High → terminates calls early, refuses visual inspection |
| Hopefulness | High → better treatment adherence, attends visits |
| Defensiveness | High → minimizes symptoms (reports Grade 2 as Grade 1) |
| Trust | High → accurate reporting, engages fully with CLARA Call |
Persona-specific baselines are set at patient generation. Events (new AE, good scan result, CLARA Call empathy) cause daily micro-adjustments.
- Python 3.10+
- NVIDIA GPU with ~10 GB VRAM (NB2 only ~20 GB; simulation CLI uses Gemini API only)
- Google Gemini API key
git clone https://github.com/newmes/CLARA.git
cd CLARA
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .Request access to google/medgemma-4b-it and google/medsiglip-448, then:
python -c "from huggingface_hub import login; login()"cp .env.example .env
# Set GOOGLE_API_KEY (required for NB3, NB5 and CLI simulation)jupyter lab notebooks/
# Run sequentially: 1 → 5# Default: Padcev + Pembrolizumab, 1 patient, 21 days
python src/run_simulation_v2.py
# A/B comparison: 50 patients, 84 days (4 cycles)
python src/run_simulation_v2.py --patients 50 --days 84 --seed 42 --mode both
# Drug-agnostic: any drug works
python src/run_simulation_v2.py --drug "Ozempic" --indication "type 2 diabetes" --patients 5CLARA deploys as a multi-service Docker stack with GPU acceleration.
-
Docker network — create the external network before starting:
docker network create vital-net
-
.envfile — create at the project root:GOOGLE_API_KEY=<your-google-api-key> HF_TOKEN=<your-huggingface-token>
-
Model weights — the following must be available locally:
google/medgemma-1.5-4b-it(HF cache)google/medgemma-4b-it(HF cache)newmes/medgemma-4b-antihallu(HF cache)google/medasr— gated repo, requires access request. Without it, the data-collection-agent starts but audio transcription (MedASR) is unavailable.
-
SigLIP classification head — required by data-collection-agent:
mkdir -p dca_server/models # Download from HuggingFace dataset: python -c " from huggingface_hub import hf_hub_download hf_hub_download( repo_id='newmes/clinical-trial-engine-data', filename='siglip_ft_head/best_model_wf1.pt', repo_type='dataset', local_dir='/tmp/cte_data' ) " && cp /tmp/cte_data/siglip_ft_head/best_model_wf1.pt dca_server/models/siglip_head.pt
docker-compose.yaml volume paths must match your local HuggingFace cache location. The default config assumes /data2/huggingface/hub/.... If your cache is elsewhere (e.g., ~/.cache/huggingface/hub/), update all volume mounts accordingly.
Important: HF cache symlink issue — HuggingFace stores model files as content-addressed blobs, with snapshot directories containing relative symlinks (
../../blobs/...). When mounting into Docker, you must mount the entire model directory (e.g.,models--google--medgemma-1.5-4b-it/) so that relative symlinks resolve correctly inside the container. Mounting only the snapshot subdirectory will result in broken symlinks.
vLLM latest (v0.15+) requires more GPU overhead for torch.compile and multimodal encoder profiling than older versions. If you see No available memory for the cache blocks errors, increase --gpu-memory-utilization (recommended: 0.6 for 24GB GPUs).
# Django frontend only
docker compose up -d django
# All GPU services
docker compose --profile medgemma4b-base --profile medgemma4b-antihallu --profile antihallu --profile data-collection-agent up -d| Service | Port | Default GPU | Purpose |
|---|---|---|---|
django |
19001 | — | Web interface (trial viewer, CRF tables, SAE reports) |
medgemma4b-base |
38004 | GPU 2 | DCA nurse backend (vLLM, medgemma-1.5-4b-it) |
medgemma4b-antihallu-ft |
38002 | GPU 3 | Anti-hallucination model (vLLM, medgemma-4b-antihallu) |
antihallu-server |
38003 | GPU 1 | Hallucination detection API (original vs RLFR side-by-side) |
data-collection-agent |
38005 | GPU 1 | Multimodal detection (SigLIP + HeAR + MedASR + TTS) |
Note:
antihallu-serverloads two full models (original + RLFR fine-tuned) on a single GPU (~16GB total). It must not share a GPU with vLLM services. GPU assignments can be overridden via environment variables:GPU_MEDGEMMA4B,GPU_ANTIHALLU,GPU_CARE_AI.
CLARA/
├── notebooks/ 5 demo notebooks (anti-hallu → SAE reports)
│ └── src/ Local module shared across notebooks
│ ├── agents/ LLM agents (rule, patient, daily, care)
│ ├── engine/ Probability engine (hazard, observation, sampler, mood)
│ ├── doc_agent/ SAE report generation (MedWatch, E2B XML, MedDRA)
│ ├── multimodal/ Multimodal AE detection (SigLIP + HeAR)
│ ├── multimodal_v2/ Multimodal AE detection v2
│ ├── ruleset_generation/ Drug rule discovery from 10+ DBs
│ │ └── ground_truth/ 7 real clinical trial datasets for validation
│ ├── cough_detection/ HeAR-based cough classification
│ └── experiments/ Experiment management
│
├── src/
│ ├── engine/ Probability engine (LLM-independent)
│ │ ├── hazard.py Daily AE onset/grade hazard functions
│ │ ├── observation.py Ground Truth ↔ Hospital Record model
│ │ ├── sampler.py Seed-reproducible random sampling
│ │ ├── mood.py 7-dimension patient psychology
│ │ └── prob_engine.py LLM → rand → LLM orchestration
│ │
│ ├── agents/ LLM agents (Gemini-powered)
│ │ ├── rule_agent.py Phase 0: drug rule discovery
│ │ ├── patient_agent.py Phase 1: virtual cohort generation
│ │ ├── daily_agent.py Phase 2: daily simulation
│ │ └── care_agent.py CLARA Call: 4-turn video calls
│ │
│ ├── multimodal_v2/ Multimodal AE detection pipeline
│ ├── cough_detection/ HeAR-based cough classification
│ ├── orchestrator_v2.py 3-Phase simulation orchestrator
│ └── run_simulation_v2.py CLI entry point
│
├── dca_server/ DCA Nurse API (FastAPI)
│ ├── server.py Endpoints: classify, cough, transcribe, nurse
│ ├── nurse_engine.py Medical conversation engine
│ ├── siglip_classifier.py SigLIP vision classifier head
│ └── cough_classifier.py Cough audio classifier
│
├── frontend/ Django web interface
│ ├── viewer/ Trial viewer, patient state, CRF tables
│ └── templates/ Interactive simulation visualization
│
├── models/ Fine-tuned checkpoints
│ ├── medgemma-4b-ft-antihallu/ RLFR anti-hallucination (hallu 37.3% → 6.7%)
│ └── medgemma-4b-ft-ctcae/ Skin AE classifier (W-F1 = 90%)
│
├── data/ Simulation outputs
│ ├── rule_set.json Default Padcev+Pembro rule set
│ └── runs/ Per-experiment results (JSONL)
│
└── docker-compose.yaml Multi-GPU service orchestration
- Basch, E. et al. (2006). Lancet Oncol., 7(11), 903–909. doi:10.1016/S1470-2045(06)70910-X
- Di Maio, M. et al. (2015). J Clin Oncol., 33(8), 910–915. doi:10.1200/JCO.2014.57.9334
- Izarn, F. et al. (2025). ESMO Open, 10(1), 104086. doi:10.1016/j.esmoop.2024.104086
- Orlandic, L. et al. (2021). Sci Data, 8, 156. doi:10.1038/s41597-021-00937-4
- Atmaja, B. T. et al. (2023). Int. J. Inf. Technol. doi:10.1007/s41870-023-01626-8
Built with
Google Gemini
and the
HAI-DEF
model collection.
Submitted to the MedGemma Impact Challenge on Kaggle.
| Asset | Source | License |
|---|---|---|
| Character sprites (nurse, patients, buildings, decorations) | Sunnyside by Daniel Diggle | Commercial license |
| 1-Bit & Tiny Town tilesets | Kenney | CC0 1.0 |
| CuteRPG tilesets | CuteRPG | Commercial license |
| Synthetic medical images | Generated with Google Gemini | — |