Skip to content

asr.md

Latest commit

 

History

History
272 lines (196 loc) · 11 KB

asr.md

File metadata and controls

272 lines (196 loc) · 11 KB
Raw

ASR (Automatic Speech Recognition)

Transcribe audio files using speech recognition, powered by sherpa-onnx. All inference runs offline on your machine.

Prerequisites

No external dependencies are required for WAV files. Non-WAV format support via the CLI is deprecated and requires ffmpeg (see COLI_DEP002).

CLI

# Plain text output
coli asr recording.wav

# JSON output
coli asr -j recording.wav

# Select model
coli asr --model whisper recording.wav

# Use a local model file or directory without downloading
coli asr --model-path /path/to/sensevoice/model.int8.onnx recording.wav

# Specify language (sensevoice only)
coli asr --language zh recording.wav

Options

-j, --json     Output result in JSON format
--model        Model to use: whisper, sensevoice (default: sensevoice)
--model-path   Path to a local model file or directory
--language     Language for sensevoice: auto, zh, en, ja, ko, yue (default: auto)

coli asr-stream

Stream speech recognition from stdin. Expects raw 16kHz mono s16le PCM audio piped in.

# From microphone (macOS)
ffmpeg -f avfoundation -i :0 -ar 16000 -ac 1 -f s16le pipe:1 | coli asr-stream

# With VAD
ffmpeg -f avfoundation -i :0 -ar 16000 -ac 1 -f s16le pipe:1 | coli asr-stream --vad

# JSON output (one JSON object per line)
ffmpeg -f avfoundation -i :0 -ar 16000 -ac 1 -f s16le pipe:1 | coli asr-stream --vad --json

# Use local ASR and VAD models without downloading
ffmpeg -f avfoundation -i :0 -ar 16000 -ac 1 -f s16le pipe:1 | coli asr-stream --model-path /path/to/sensevoice/model.int8.onnx --vad --vad-model-path /path/to/silero_vad.onnx

# From a file
ffmpeg -i podcast.m4a -ar 16000 -ac 1 -f s16le pipe:1 | coli asr-stream --vad

Options

-j, --json              Output each result as a JSON line
--vad                   Enable voice activity detection
--model-path <path>     Path to a local SenseVoice model file or directory
--vad-model-path <path> Path to a local VAD model file
--language <lang>       Language for sensevoice: auto, zh, en, ja, ko, yue (default: auto)
--asr-interval-ms <ms>  Recognition interval in ms (default: 1000, ignored with --vad)

JSON output example

{
	"text": "The tribal chieftain called for the boy.",
	"model": "sensevoice-small",
	"lang": "<|en|>",
	"emotion": "<|NEUTRAL|>",
	"event": "<|Speech|>",
	"tokens": ["The", " tri", "bal", " chief", "tain", "..."],
	"timestamps": [0.9, 1.26, 1.56, 1.8, 2.16, "..."],
	"duration": 7.152
}

API

ensureModels(models?)

Download the specified models if not already present. Defaults to ['sensevoice']. Call this before runAsr or streamAsr.

import {ensureModels} from '@marswave/coli';

await ensureModels(); // downloads sensevoice only
await ensureModels(['whisper', 'sensevoice']); // downloads both

readWave(filename)

Read a WAV file and return an AudioData object. Use this to load WAV files for runAsr.

import {ensureModels, readWave, runAsr} from '@marswave/coli';

await ensureModels();

const audio = readWave('/path/to/recording.wav');
await runAsr(audio, {json: false, model: 'sensevoice'});

runAsr(input, options)

Run speech recognition on audio data. Results are printed to stdout.

The input parameter accepts either an AudioData object (recommended) or a file path string (deprecated).

import {ensureModels, runAsr} from '@marswave/coli';

await ensureModels();

// Recommended: pass AudioData directly
await runAsr(
	{sampleRate: 16000, samples: myFloat32Array},
	{json: false, model: 'sensevoice'},
);

// Custom model path: no download, fails if local files are missing
await runAsr(
	{sampleRate: 16000, samples: myFloat32Array},
	{json: false, model: 'sensevoice', modelPath: '/path/to/model.int8.onnx'},
);

// Deprecated: file path input (requires ffmpeg for non-WAV formats)
await runAsr('recording.m4a', {json: false, model: 'sensevoice'});

Options

Property Type Description
json boolean Output JSON (with model name, tokens, timestamps, etc.) instead of plain text
model 'whisper' | 'sensevoice' Which model to use for recognition
modelPath string Path to a local model file or directory. Skips download and throws if required local files are missing.
language SenseVoiceLanguage Language hint for sensevoice: 'auto', 'zh', 'en', 'ja', 'ko', 'yue' (default: 'auto')

getModelPath(model)

Returns the local filesystem path for a given model.

import {getModelPath} from '@marswave/coli';

getModelPath('sensevoice');
// => '/Users/you/.coli/models/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17'

getModelPath('whisper');
// => '/Users/you/.coli/models/sherpa-onnx-whisper-tiny.en'

modelDisplayNames

A mapping from model key to its human-readable display name.

import {modelDisplayNames} from '@marswave/coli';

modelDisplayNames.sensevoice; // => 'sensevoice-small'
modelDisplayNames.whisper; // => 'whisper-tiny.en'

Streaming API

For streaming speech recognition, use the streaming API. It accepts audio as an async iterable of Float32Array chunks (16 kHz mono PCM) and delivers recognition results via the onResult callback as audio accumulates, using the SenseVoice model.

streamAsr(audio, options)

Stream audio in and receive recognition results incrementally. Call ensureModels() first. If using VAD, also call ensureVadModel().

import {ensureModels, ensureVadModel, streamAsr} from '@marswave/coli';

await ensureModels();

const audioSource = createAudioStream(); // AsyncIterable<Float32Array> of 16 kHz mono PCM

// Interval-based (default) — emits partial results at a fixed interval
await streamAsr(audioSource, {
	modelPath: '/path/to/model.int8.onnx',
	onResult(result) {
		console.log(result.text, result.isFinal ? '(final)' : '(partial)');
	},
});

// VAD-based — segments speech automatically, each segment emits a final result
await ensureVadModel();
await streamAsr(audioSource, {
	vad: true,
	onResult(result) {
		console.log(result.text);
	},
});

// VAD with custom parameters
await streamAsr(audioSource, {
	vad: {
		modelPath: '/path/to/silero_vad.onnx',
		threshold: 0.4,
		minSilenceDuration: 0.3,
		maxSpeechDuration: 10,
	},
	onResult(result) {
		console.log(result.text);
	},
});

Options

Property Type Description
onResult (result: AsrStreamResult) => void Callback invoked with each recognition result
sampleRate number Audio sample rate in Hz (default: 16000)
language SenseVoiceLanguage Language hint for sensevoice: 'auto', 'zh', 'en', 'ja', 'ko', 'yue' (default: 'auto')
modelPath string Path to a local SenseVoice model file or directory. Skips download and throws if required local files are missing.
asrIntervalMs number Recognition interval in milliseconds (default: 1000). Ignored when using VAD
vad boolean | VadOptions Enable VAD. Pass true for defaults or a VadOptions object

VadOptions

Property Type Description
modelPath string Path to a local VAD model file. Skips download and throws if missing.
threshold number Speech detection threshold (default: 0.5)
minSpeechDuration number Minimum speech duration in seconds (default: 0.25)
minSilenceDuration number Minimum silence to end a segment in seconds (default: 0.5)
maxSpeechDuration number Maximum speech segment duration in seconds (default: 15)
enableExternalBuffer boolean Use external buffer for VAD speech segments (default: undefined)

Result

Property Type Description
text string Recognized text so far
lang string Detected language tag
emotion string Detected emotion tag
event string Detected audio event tag
tokens string[] Individual tokens
timestamps number[] Timestamp for each token
isFinal boolean Whether the result is finalized

Models

On first run, coli automatically downloads required models to ~/.coli/models/:

ASR Models

Name Model Languages
sensevoice (default) SenseVoice Small int8 Chinese, English, Japanese, Korean, Cantonese
whisper Whisper tiny.en int8 English

VAD Model

Name Model Size
silero_vad Silero VAD (k2-fsa export) ~629 KB

streamAsr uses the SenseVoice model for recognition. VAD uses Silero VAD and is downloaded separately via ensureVadModel().

Supported audio formats

The CLI accepts WAV files directly. For the programmatic API, use readWave() to load WAV files into an AudioData object, or provide your own AudioData from any source. Non-WAV file path input is deprecated (see COLI_DEP002).