🎙️ Voice Authentication System — RawNet3

A production-grade, text-independent speaker verification system built on top of RawNet3 — a state-of-the-art deep neural network that operates directly on raw audio waveforms to extract speaker embeddings.

Users can register by speaking their voice, and later authenticate by speaking again — no password required.

---

🏗️ Architecture Overview

Browser (Next.js)
      │
      │  Audio recording via MediaRecorder API
      ▼
FastAPI Backend  ──► Redis Queue (Celery) ──► GPU Worker
      │                                          │
      │                                    RawNet3 Inference
      │                                    (256-dim embedding)
      │◄─────────────────────────────────────────┘
      │
      ├──► Pinecone (Vector DB)   — stores speaker embeddings
      └──► Supabase (PostgreSQL)  — stores user metadata

Flow	Description
Registration	3+ audio samples → RawNet3 embeddings → mean pooling → dynamic threshold → stored in Pinecone + Supabase
Verification	1 audio sample → RawNet3 embedding → cosine similarity vs stored embedding → accept/reject

---

🧰 Tech Stack

Layer	Technology
ML Model	RawNet3 (PyTorch) — raw waveform speaker verification
Backend API	FastAPI
Task Queue	Celery + Redis
Vector Database	Pinecone — stores 256-dim speaker embeddings
Relational Database	Supabase (PostgreSQL) — user metadata
Frontend	Next.js (TypeScript) + Tailwind CSS
Containerisation	Docker + Docker Compose
Prototype UI	Gradio (Flask-based)

---

📁 Repository Structure

voice-auth/
│
├── app/                         # ✅ Production application
│   ├── backend/                 # FastAPI + Celery backend
│   │   ├── main.py              # API routes: /register, /verify
│   │   ├── database.py          # Supabase + Pinecone integration
│   │   ├── speaker_model.py     # RawNet3 inference wrapper
│   │   ├── preprocessing.py     # VAD audio preprocessing
│   │   ├── celery_app.py        # Celery + Redis broker setup
│   │   ├── tasks.py             # Async embedding tasks
│   │   ├── models/              # RawNet3 model definition
│   │   ├── Dockerfile
│   │   ├── docker-compose.yml
│   │   └── .env.example         # Environment variable template
│   │
│   └── frontend/                # Next.js frontend
│       └── src/
│           ├── app/page.tsx     # Main registration/verification UI
│           ├── components/
│           │   └── AudioRecorder.tsx
│           └── services/api.ts  # API client
│
├── prototype/                   # 🧪 Gradio prototype (v1)
│   ├── app.py                   # Single-file Flask + Gradio demo
│   └── speaker_model.py
│
├── rawnet-reference/            # 📚 RawNet family reference implementations
│   └── python/
│       ├── RawNet1/
│       ├── RawNet2/
│       ├── RawNet2_modified/
│       └── RawNet3/             # Core model used in production
│
├── training/                    # 🏋️ Training framework (VoxCeleb dataset)
│   ├── trainSpeakerNet.py
│   ├── SpeakerNet.py
│   ├── DatasetLoader.py
│   └── configs/
│
└── research/
    └── rawnet3.pdf              # Original RawNet3 paper

---

🚀 Getting Started

Prerequisites

• Python 3.9+
• Node.js 18+
• Docker & Docker Compose
• A Supabase account (free tier works)
• A Pinecone account (free tier works)

---

1. Clone the Repository

git clone https://github.com/YOUR_USERNAME/voice-auth.git
cd voice-auth

---

2. Backend Setup

cd app/backend

# Copy and fill in environment variables
cp .env.example .env
# Edit .env with your Supabase + Pinecone credentials

Run with Docker (recommended):

docker-compose up --build

This starts:

• FastAPI server on http://localhost:8000
• Redis broker
• Celery GPU worker

Or run locally:

pip install -r requirements.txt
# Terminal 1 — API server
uvicorn main:app --reload --port 8000
# Terminal 2 — Celery worker
celery -A celery_app worker --loglevel=info

---

3. Frontend Setup

cd app/frontend
npm install
npm run dev

Open http://localhost:3000

---

4. Environment Variables

Copy app/backend/.env.example to .env and fill in:

# Supabase
SUPABASE_URL=https://your-project-id.supabase.co
SUPABASE_KEY=your-anon-or-service-role-key

# Pinecone
PINECONE_API_KEY=your-pinecone-api-key
PINECONE_ENV=us-east-1-aws
PINECONE_INDEX_NAME=voice-auth-embeddings

# App Config
SIMILARITY_THRESHOLD=0.45

---

⚙️ How It Works

Registration

1. User records 3+ voice samples in the browser
2. Each audio file is sent to the FastAPI backend
3. Audio is pushed into a Redis-backed Celery queue
4. A GPU worker picks up the task and runs RawNet3 inference → 256-dim embedding
5. All embeddings are mean-pooled into a single representative embedding
6. A dynamic threshold is calculated from pairwise cosine similarities between samples (higher variance → stricter threshold)
7. Embedding + threshold stored in Pinecone; username metadata stored in Supabase

Verification

1. User records a single voice sample
2. RawNet3 extracts an embedding via the same Celery pipeline
3. Cosine similarity is computed between the new embedding and the stored one
4. If similarity ≥ personalized_threshold → ✅ Authenticated
5. Otherwise → ❌ Rejected

---

🧪 Gradio Prototype (v1)

The prototype/ folder contains an earlier single-file demo using Gradio + Flask. This was the first working prototype before the production stack was built.

cd prototype
pip install -r requirements.txt
python app.py
# Open http://localhost:5000

---

🏋️ Training (Optional)

The training/ directory contains the training framework for RawNet3 using the VoxCeleb dataset.

cd training
pip install -r requirements.txt

# Download RawNet3 pretrained weights
git submodule update --init --recursive

# Evaluate pretrained model (EER ~0.89%)
python trainSpeakerNet.py --eval --config ./configs/RawNet3_AAM.yaml \
    --initial_model models/weights/RawNet3/model.pt

---

📄 Research

This project is based on:

Pushing the limits of raw waveform speaker recognition
Jung et al., Interspeech 2022
arxiv.org/abs/2203.12414

@inproceedings{jung2022pushing,
  title={Pushing the limits of raw waveform speaker recognition},
  author={Jung, Jee-weon and Kim, You Jin and Heo, Hee-Soo and Lee, Bong-Jin and Kwon, Youngki and Chung, Joon Son},
  booktitle={Proc. Interspeech},
  year={2022}
}

---

📜 License

This project is licensed under the MIT License.
The rawnet-reference/ and training/ submodules retain their respective original licenses.