Joining 5 US government datasets to surface where pharma payments, prescribing behaviour, and patient harm intersect
🔗 Live Dashboard: https://cms-command-center-wpbn3mbqgpkwae6syflrbp.streamlit.app/
Three testable theses, proven with real data across 1.6 million US providers:
| Thesis | Finding |
|---|---|
| Opioid payments → higher prescribing | Outliers prescribe at 107.8x the rate of low-scoring peers |
| Brand payments → fewer generics | Brand outliers prescribe 165% above specialty average |
| High billing → worse outcomes | Top 0.6% of billing outliers = 15.4% of all Medicare services |
| Dataset | Source | Size | What it contains |
|---|---|---|---|
| CMS Open Payments | data.cms.gov | 8.8 GB | Every dollar pharma paid to every US doctor |
| Medicare Part D | data.cms.gov | 4.2 GB | Every drug every doctor prescribed |
| Medicare Part B | data.cms.gov | 3.2 GB | Every procedure every doctor billed |
| Hospital Compare | data.cms.gov | 2 MB | Hospital readmission and outcome data |
| NPPES NPI Registry | download.cms.gov | 11 GB | Every licensed US provider |
| Total | 27 GB |
All datasets are free, public, and require no login.
Every CMS dataset uses the National Provider Identifier (NPI) — a unique 10-digit number every US doctor has. This project uses NPI as the primary key to join all five datasets into a single master provider table with 1,614,811 rows.
Raw numbers are meaningless without context. A surgeon prescribing more opioids than a dermatologist is expected — not suspicious. This project computes a z-score for every doctor relative to their own specialty peer group:
- Z-score of 0 = exactly average for their specialty
- Z-score of 1 = above average, unremarkable
- Z-score of 2 = top 2.3% of specialty — warrants review
- Z-score of 3 = top 0.1% — statistical outlier
Three z-scores (opioid rate, brand drug ratio, billing intensity) are averaged into a single composite risk score. Providers are classified into four tiers: Low, Moderate, Elevated, High.
Built with Streamlit + Plotly — 6 interactive tabs:
| Tab | What it shows |
|---|---|
| Command Overview | National risk map, tier distribution, top 20 highest-risk providers |
| Thesis 1: Opioids | Specialty ranking, outlier vs peer comparison, z-score by specialty |
| Thesis 2: Brand Drugs | Brand ratio analysis, specialty scatter plot |
| Thesis 3: Billing | Intensity maps, hospital readmission overlay, worst 15 hospitals |
| NJ Spotlight | New Jersey provider deep-dive — 29,901 NJ providers, 213 high-risk |
| Provider Lookup | Search any provider by name, specialty, or risk tier with gauge chart |
National (1,614,811 providers):
- 13,612 providers in the High risk tier (0.8%)
- 13,138 providers flagged on 2+ dimensions simultaneously
- Surgery and Neurological Surgery top opioid rate rankings
- Rheumatology shows highest billing intensity (19.9 services/patient)
- Top 0.6% of billing outliers account for 15.4% of all Medicare services
NJ Spotlight (29,901 NJ providers):
- 213 providers flagged as high-risk
- Dental Hygienists and Pharmacists show highest brand drug ratios
├── 01_verify_data.py # Confirm all 5 datasets loaded correctly
├── 02_load_and_join.py # NPI join — build master provider table
├── 03_peer_scoring.py # Z-score peer deviation scoring
├── 03c_rebuild_thesis.py # Corrected opioid + brand drug detection
├── dashboard.py # 6-tab Streamlit dashboard
├── requirements.txt
├── outputs/
│ └── sql/
│ ├── thesis1_opioid_by_specialty.csv
│ ├── thesis2_brand_by_specialty.csv
│ ├── thesis3_billing_by_specialty.csv
│ └── nj_provider_spotlight.csv
# 1. Download the 5 CMS datasets (see Phase 1 guide in README)
# 2. Install dependencies
pip install -r requirements.txt
# 3. Run the pipeline in order
python 01_verify_data.py
python 02_load_and_join.py
python 03_peer_scoring.py
python 03c_rebuild_thesis.py
# 4. Launch the dashboard
streamlit run dashboard.pyThis analysis uses statistical peer comparison to surface providers whose behaviour diverges significantly from their specialty peers. High scores indicate patterns that warrant review — not evidence of wrongdoing. Legitimate factors such as patient complexity, rural practice, or specialist referral patterns may explain outlier behaviour. This project is a portfolio demonstration of healthcare data analytics, not an enforcement tool.
Python Pandas NumPy SciPy SQLite SQL Streamlit Plotly Matplotlib
The deployed Streamlit dashboard loads a representative sample for performance reasons — all high-risk and elevated providers are included in full, with a random sample of lower-risk providers. To run the full analysis on all 1.6 million providers, follow the local setup instructions above and download the complete CMS datasets.
Karan Trivedi MS Data Analytics, Webster University LinkedIn · GitHub