Skip to main content

Speaker Diarization

OpenTranscribe identifies different speakers and segments audio by "who spoke when". Speakers can be detected on this server, by the cloud provider that transcribes the file, or by a separate cloud service; you choose which.

Where speakers are detected​

Transcription and speaker detection are two separate choices. Which model transcribes a file is set under Settings → Transcription; who detects its speakers is set under Settings → Speaker Identification → Speaker Detection. Any pairing works on a full deployment, with the limits listed below.

The Speaker Detection options​

Option (as shown in Settings)What happens
Automatic (recommended)The default. See the rules below.
Always on this server's GPUSpeakers are detected on this server, whichever model transcribes. Not available on a lite deployment.
pyannote.ai cloud serviceThe audio is sent to the hosted pyannote.ai service, with the built-in model or a cloud provider. Needs your own pyannote.ai API key, and pyannote.ai bills that key for every file. This is not the PyAnnote engine that can run on this server.
Off (one speaker for the whole recording)No speaker detection. Every segment is assigned to one speaker, and speaker profiles and cross-file matching do not run.

The fifth choice is Another provider's speaker detection: a saved cloud transcription provider used only for its speakers (it needs the capability that lets users save their own providers, and only providers that return speakers are listed). That provider transcribes the whole file again with speaker detection on; OpenTranscribe keeps only its speaker turns and throws its text away. The provider still bills a full transcription of every recording and receives the audio, so the card and the pipeline summary warn about this before you save. If you pick the same provider and model that already transcribes the file, its own speakers are used and nothing is billed twice.

Automatic rules​

  • Built-in model (including the lightweight CPU models): speakers are detected on this server.
  • Cloud model that returns speakers: that provider's own speaker labels are used. Nothing else runs.
  • Cloud model that returns text only: on a full deployment, this server detects the speakers after the transcript is saved. On a lite deployment this is refused with a message asking you to choose pyannote.ai or another provider for speaker detection.

Which cloud models return speakers, and which combinations are valid on each deployment, are listed on Deployment variants (see also which combinations are valid).

When detection runs​

  • Inside transcription. With the built-in GPU model and this server's engine, speakers are detected in the same task, right after the words. With a cloud provider that detects speakers itself, its labels arrive with the transcript. With a cloud transcription provider and a cloud speaker detector (pyannote.ai or another saved provider), both requests run side by side.
  • After the transcript is saved. In every other pairing (for example a cloud transcript with this server's engine, or the built-in model with pyannote.ai), speaker detection runs as a separate step once the transcript is stored. If a cloud speaker detector fails at that point, the transcript is kept without speakers and the reason is recorded; nothing silently falls back to another detector.

How speakers are matched to the words​

The speaker turns are merged onto the transcript in one place, by timing:

  • If the transcription model returns word timestamps, each word gets the speaker who was talking at that moment.
  • If it returns segment timestamps only, each segment gets one speaker. A speaker change inside a segment is then not detected, and the pipeline summary shows a warning.
  • If it returns no timestamps at all, speakers cannot be placed. Choosing speaker detection with such a model is refused with a clear message; pick a model with word or segment timing, or turn speaker detection off.

There is no forced-alignment step. Word timestamps come from the transcription model itself (for the built-in model, from faster-whisper; see Native Word-Level Timestamps), and cloud timestamps are used as the provider returns them.

Voiceprints are always computed on this server​

Whoever detects the speakers, the voiceprints used for cross-file matching and speaker profiles are computed on this server, never taken from a vendor. They come from one interface: today the WeSpeaker model, served by the diar-native sidecar or, when the sidecar is not available, loaded inside the worker. A Rust replacement for this extractor is planned for v0.8.0 (#1231); it plugs into the same interface.

Lite deployments​

A lite deployment has no local speaker detection. Use a cloud model that returns speakers (with Automatic), the pyannote.ai cloud service, another saved provider used only for speakers, or Off. Always on this server's GPU is refused there. Voiceprints are still computed on the lite server's CPU.

Which engine runs on this server​

When speakers are detected on this server, the engine is a deployment-wide choice made by a super admin under Settings → Speaker Identification → Speaker Engine (Diarizer Backend): the diar-native sidecar (the default) or the in-process PyAnnote engine. Users choose where speakers are detected, not which local engine does it. See Native Diarization Engine.

Speaker-count hints​

Your minimum, maximum or exact speaker count is passed to the detector when it can use it:

  • pyannote.ai honours all three.
  • This server's engine honours them only when the in-process PyAnnote engine is selected. The default diar-native engine counts speakers automatically and ignores them.
  • A cloud transcription provider's own speakers honour whatever that provider supports (listed on Deployment variants).

When the detector ignores the hints, the pipeline summary says so.

Seeing the plan before you upload​

The processing pipeline summary in Settings is a read-only view of how your next file will be processed: the transcription step, speaker detection, voiceprints and AI features, where each one runs, and any problem with the combination you chose. It reads GET /api/user-settings/pipeline, which API clients can call directly. Saving a speaker-detection choice that cannot run is refused, and a file whose plan cannot run fails before any work starts, with the same message.

This Server's Engine (PyAnnote v4 pipeline)​

This section describes the engine that runs when speakers are detected on this server. Both local engines (diar-native and in-process PyAnnote) run the same pyannote/speaker-diarization-community-1 pipeline.

State-of-the-art speaker diarization (upgraded to PyAnnote v4):

Core Features​

  • Neural network-based speaker detection
  • Voice activity detection (VAD)
  • Speaker embedding extraction
  • Overlap detection with visual highlighting
  • Clustering for speaker assignment

Recent Improvements (v4 Migration)​

New Capabilities:

  • Speaker Overlap Detection: Automatically identifies and highlights segments where multiple speakers talk simultaneously with primary/secondary speaker classification and confidence scores
  • Warm Model Caching: Models are cached on startup, reducing initialization time by 40-60 seconds on subsequent restarts
  • Fast Speaker Assignment: 273x faster speaker clustering using optimized embedding algorithms

Technical Updates:

  • Embedding Dimension: Upgraded from 192-dimensional to 256-dimensional speaker embeddings for improved voice fingerprinting accuracy
  • Better Noise Handling: Improved performance with background noise and overlapping speech
  • Optimized Processing: Faster GPU utilization with reduced memory overhead

How It Works​

The diagram shows the path when speakers are detected on this server. When a cloud service detects them instead, the speaker turns come from that service and everything from embedding extraction onward still runs here.

Processing Steps​

  1. Voice Activity Detection: Identify speech vs silence
  2. Speaker Embedding: Extract voice fingerprints
  3. Overlap Detection: Find overlapping speakers
  4. Clustering: Group similar voices
  5. Speaker Assignment: Label each segment

Speaker Fingerprinting​

Each speaker gets a unique voice embedding:

  • 256-dimensional vector (PyAnnote v4)
  • Always computed on this server, whichever detector found the speakers
  • Captures voice characteristics with high precision
  • Used for cross-video matching
  • Enables speaker recognition across videos

Speaker Management​

Automatic Detection​

  • Detects 1-50+ speakers automatically
  • No manual configuration required
  • Adaptive to conversation dynamics
  • Handles speaker interruptions

Speaker Profiles​

Create persistent speaker profiles:

  1. Label "Speaker 1" as "John Doe"
  2. System creates global profile
  3. Profile matches across all videos
  4. Voice fingerprint stored for matching

Cross-Video Recognition​

Intelligent speaker matching:

  • Voice embedding comparison
  • Similarity scoring (0-100%)
  • High-confidence auto-linking (75% or more): the matching profile's name is applied
  • Below 75%, no name is applied automatically
  • Speaker appears in multiple recordings

Example:

Video 1: John Doe speaks
Video 2: Unknown speaker detected
System: 92% match → Links the speaker to "John Doe"

Speaker Analytics​

Talk Time Analysis​

  • Total speaking duration per speaker
  • Percentage of conversation
  • Speaking turns count
  • Average turn length

Interaction Patterns​

  • Interruption detection
  • Turn-taking analysis
  • Overlap frequency
  • Silence ratio

Speaking Metrics​

  • Words per minute (WPM): Speaking pace
  • Question frequency: Questions asked
  • Longest monologue: Continuous speaking time
  • Participation balance: Equal participation score

AI-Powered Identification​

With LLM configured, get smart suggestions:

Context-Based Identification​

  • Analyzes conversation content
  • Identifies speakers by role/expertise
  • Detects names mentioned in conversation
  • Provides confidence scores

Example:

Context: "As the CEO mentioned earlier..."
AI Suggestion: "John Smith (CEO)" - 85% confidence

Verification Workflow​

  1. AI suggests speaker name
  2. Review suggestion + confidence
  3. Accept or manually correct
  4. System learns from corrections

Configuration​

Detection Range​

Speaker count expectations are set at three levels:

  • System default range: MIN_SPEAKERS / MAX_SPEAKERS in .env (defaults 1 and 20).
  • Your own range: Settings → Speaker Identification → Speaker Detection → Number of speakers (1 to 50).
  • Per file: the upload wizard's Speakers step, or the reprocess dialog.
# .env configuration (system default range)
MIN_SPEAKERS=1 # Minimum speakers
MAX_SPEAKERS=20 # Maximum speakers

Use Cases:

  • 1-1 interview: MIN=2, MAX=2
  • Team meeting: MIN=3, MAX=10
  • Conference panel: MIN=5, MAX=15

Note: the range is a hint, and not every detector uses it. The default diar-native engine counts speakers automatically and ignores it; see Speaker-count hints.

Quality Settings​

For best results:

  • Clear audio (minimal background noise)
  • Distinct speakers (different voices)
  • Good microphone quality
  • Minimal speaker overlap

Speaker Display​

Visual Representation​

  • Color coding: Each speaker gets unique color
  • Speaker labels: Names shown in transcript
  • Timeline view: Speaker segments visualized
  • Speaker list: All speakers with talk time
  • Filter transcript by speaker
  • Search within speaker's words
  • Export single speaker's content
  • Compare speaker contributions

Advanced Features​

Overlap Detection​

Identifies when multiple speakers talk simultaneously:

Visual Highlighting:

  • Overlapping segments are highlighted in the transcript interface
  • Primary speaker (dominant voice) is identified first
  • Secondary speakers shown with reduced emphasis
  • Duration of overlap clearly marked

Technical Details:

  • Confidence Scoring: Each overlap includes overlap_confidence (0-100) indicating detection certainty
  • Primary/Secondary Classification: PyAnnote v4 automatically ranks speakers by acoustic dominance during overlap
  • Overlap Duration: Precise timing of when speakers overlap, measured in milliseconds

Use Cases:

  • Interruption Analysis: Track when speakers interrupt each other
  • Dialogue Dynamics: Understand conversation flow and turn-taking patterns
  • Quality Control: Identify audio issues or multiple speakers in a single recording
  • Meeting Insights: Analyze discussion intensity and participation intensity

Speaker Attribute Detection (New in v0.4.0)​

OpenTranscribe can automatically detect speaker attributes to improve identification and cluster quality:

Gender Classification:

  • Neural network-based gender prediction from voice characteristics. It is an estimate, always labeled as one in the UI, and is never used to name a speaker
  • Runs automatically after transcription (configurable per-user, and can be switched off)
  • Confidence scoring for each prediction
  • Results stored on speaker profiles for cross-video consistency

Gender-Informed Cluster Validation:

  • Speaker clusters use predicted gender to validate membership
  • Cross-gender cluster assignment requires higher similarity threshold
  • Minority-gender members in a cluster are flagged for review
  • Gender outlier analysis available per cluster

Configuration: Enable/disable speaker attribute detection (Enable speaker attribute detection, Predict gender from voice) and the badges on speaker cards (Show predictions on speaker cards) in Settings → Speaker Identification → Voice Attributes. A switch-off at the deployment or organization level takes precedence over a user's own setting, because the privacy policy only ever restricts.

The same tab has a separate switch, Suggest speaker names automatically with AI. When it is on (the default) and an AI provider is configured, the transcript text is sent to that provider after transcription to suggest speaker names; suggestions are never applied automatically. It works independently of attribute detection, and it does not affect the manual suggest button.

Speaker Pre-Clustering (New in v0.4.0)​

GPU-accelerated speaker clustering runs automatically after transcription:

  • Groups speakers across files into clusters based on voice similarity
  • Uses embedding comparison for accurate cross-video speaker matching
  • Supports batch re-clustering for global speaker management
  • Cluster assignments update automatically as new files are processed

Alias-Based Speaker Indices (New in v0.4.0)​

Speaker embeddings are stored in versioned OpenSearch indices with an alias system for zero-downtime migrations:

  • speakers_v3: 512-dimensional embeddings (PyAnnote v3 / embedding model)
  • speakers_v4: 256-dimensional embeddings (WeSpeaker / PyAnnote v4)
  • speakers alias: Points to the active versioned index
  • Atomic alias swap enables seamless migration between embedding versions

Speaker Embedding Consistency Self-Healing (New in v0.4.0)​

Automatic detection and repair of inconsistent speaker embeddings:

  • Identifies speakers with missing or mismatched embeddings across indices
  • GPU-accelerated batch repair re-extracts embeddings from original audio
  • Admin-triggered via Settings → Data & Storage → Speaker Embedding System
  • Progress tracking with WebSocket notifications
  • Distributed locking prevents concurrent repairs

Turning Speaker Detection Off​

Choose Off (one speaker for the whole recording) under Settings → Speaker Identification → Speaker Detection to skip speaker detection on your uploads. The transcript is still produced; every segment is assigned to one speaker, and speaker profiles and cross-file matching do not run for those files. Use it for dictation, voicemail or single-speaker recordings.

The upload API also accepts a per-file disable_diarization flag; the upload dialog does not offer a per-file switch. There is no server-wide variable for this (the old ENABLE_DIARIZATION is gone): the choice in Settings decides.

pyannote.ai Cloud Diarization (New in v0.4.0)​

OpenTranscribe's independent diarization provider architecture allows routing diarization to an external cloud service rather than running locally:

  • pyannote.ai: The cloud service from the creators of PyAnnote, offering the same algorithms without requiring a local GPU for diarization
  • Each user saves their own pyannote.ai API key in Settings → Speaker Identification → Speaker Detection (shown when users may bring their own provider keys), then chooses pyannote.ai cloud service as the speaker-detection source. The key is stored encrypted and is never shown again
  • When pyannote.ai is selected, speaker diarization is sent to the cloud API, whichever model transcribes. With the built-in (local) model it runs right after the transcript is saved; with a cloud provider it runs beside transcription. pyannote.ai bills your key for every file it processes
  • Choosing pyannote.ai without a saved key is refused, rather than falling back to this server
  • If the cloud service fails, the transcript is kept without speakers and the reason is recorded on the file's speaker-detection task. There is no silent fallback to local diarization; re-run speaker detection (see below) once the service is back

Re-running Speaker Detection​

The reprocess dialog's Re-diarize Only (v4) stage re-runs speaker detection on the existing transcript text without transcribing again. It uses your current speaker-detection choice, not the one the file was first processed with:

  • pyannote.ai cloud service or another saved provider: that service detects the speakers again.
  • Automatic, Always on this server's GPU or Off: this server's engine runs, because a transcription provider's own speakers cannot be fetched again without transcribing the file again.
  • On a lite deployment, a choice that would run this server's engine is refused before anything is cleared.

The dialog offers this stage whichever transcription provider is active, because the choice above (not the provider that transcribed the file) decides what runs. It needs a completed transcript with word-level timestamps; the dialog also shows your saved speaker range for it.

Settings → Processing pipeline shows that same plan as a read-only page: the four steps (transcription, speaker detection, speaker identification, AI features), where each runs (this server or a named cloud vendor), whether audio leaves the server, and every problem with a link to the tab that fixes it. The Speaker Detection card and the provider list use the same vocabulary: Speakers included or Text only, Word timing, Segment timing or No timing (speakers cannot be added to text with no timestamps), and Speakers via adapter (extra billing) for a provider used only for its speaker labels. Every place that picks a transcription model shows the same labels, so you can see what a model returns before you choose it (see What the labels mean).

In the Speaker Detection card, Another provider's speaker detection lists all your saved providers with their labels and says which ones qualify (only models that return speakers do). It also says, before you save, what the choice costs: a provider chosen only for its speakers still transcribes every recording in full, bills you for that transcription and receives the audio. When the provider you pick is the one that already transcribes, the card says its own speakers are used and nothing is billed twice. With Automatic and a cloud model that returns speakers, the card says nothing extra is billed.

Stored setting values​

For API clients: the Speaker Detection choice is stored per user as diarization_source on GET/PUT /api/user-settings/transcription. The values are provider (Automatic), local (this server), pyannote (pyannote.ai), asr_config (another saved provider, named by diarization_asr_config_uuid) and off. An unrecognized value is refused, and a combination that cannot run (for example pyannote with no API key) is refused with its message instead of falling back to another detector. The same holds for transcription: a file planned for a cloud provider that cannot be started (for example a saved key that no longer works) fails with a clear message instead of being transcribed on this server.

Native Diarization Engine (New in v0.5.0)​

Speaker detection on this server (the local path) no longer means "PyAnnote" by default. A from-scratch native diarization engine — a diar-server sidecar built for this project — is now the primary engine for on-box (local) diarization, replacing PyAnnote as the default while staying fully independent of ASR engine selection: you can run cloud ASR with local diarization, local ASR with cloud diarization, or any other combination (a lite deployment has no local diarization at all; see Lite deployments).

  • Backend selection: engine.diarizer_backend (SystemSettings, falls back to env ENGINE_DIARIZER_BACKEND) accepts native (default) or pyannote. If the diar-native sidecar is unreachable when a task needs it, the pipeline automatically falls back to the in-process PyAnnote engine documented earlier on this page — there is no hard dependency on the sidecar being up, unless a super admin turns on Require Diarization Sidecar, which fails the job instead of falling back.
  • The native engine is the recommended default; use strict mode in production. The fallback works, which is also why it can go unnoticed: a deployment can run for days on in-process PyAnnote, which is slower (about half the speed) and holds GPU memory, with every setting still reading "native". It is therefore never silent. Each run on the fallback logs a WARNING with the reason (unreachable, timeout, backpressure, sidecar_not_ready, error), a worker that starts with the sidecar down logs one ERROR, the Prometheus counter diarization_runs_total{backend,reason} counts every run, the system status shows the engine that is actually serving, and the file's processing details record which engine ran. For production, turn on Require Diarization Sidecar (Settings, Speaker Identification; or ENGINE_DIARIZER_REQUIRE_SIDECAR=true as the default): a diarization task then fails with a clear error and is retried when the sidecar is unavailable, instead of running on the slower engine, and the status page reads "unavailable" until the sidecar recovers.
  • Running it: the sidecar ships as an additive Docker Compose overlay, docker-compose.diar-native.yml. It shares the backend image rather than shipping a separate one — the diar-server binary is already inside davidamacey/opentranscribe-backend, so there is nothing extra to pull. That overlay is CPU-safe on its own (no GPU device reservation, DIAR_MODE defaults to cpu); a second overlay, docker-compose.diar-native-gpu.yml, adds the nvidia device reservation and flips DIAR_MODE to cuda. Both opentr.sh (dev) and opentranscribe.sh (self-hosted) load that second overlay automatically whenever they've already detected an nvidia runtime for the rest of the stack — there is no separate CPU/GPU switch to set. Idle GPU footprint is ~2.2 GB of warm ONNX Runtime arena once the sidecar has served a request (measured on diar-server 0.3.1; earlier 0.3.x builds held roughly double that).
  • Provisioning the weights is automatic — there is nothing to run by hand on a normal install. The backend exports the ONNX/PLDA model set itself, on its own first startup (backend/app/transcription/native_provision.py, called from the FastAPI lifespan), given HUGGINGFACE_TOKEN in .env and that same account having accepted the terms at pyannote/speaker-diarization-community-1 (the gate is per-account and auto-approved — a valid token whose account never accepted it fails identically to having no token at all). Measured cold: 483,882,939 bytes in 137 seconds on a warm HuggingFace cache. It's idempotent behind a diar-provision.json marker, so every startup after the first is a stat pass, and the diar-native compose service waits on depends_on: backend: condition: service_healthy so it never starts against an empty /models and crash-loops.
    • No token configured, or the export otherwise fails: never fatal to startup. Diarization runs on the in-process PyAnnote engine instead — logged, not silent, and (see below) a fully supported configuration, not a broken one.
    • Manual escape hatch: ./opentranscribe.sh download-models diar-native runs the same export outside the backend's lifespan. Use it to pre-provision before first start (so the first request doesn't wait on the export), to force a re-export, or on a multi-replica deployment where DIAR_NATIVE_AUTO_PROVISION=false hands the export to a dedicated job instead of racing several backend replicas against the same files.
    • They land in ${MODEL_CACHE_DIR}/diar-native, a sibling of the huggingface/ and torch/ caches the PyAnnote path already uses.
    • This repo's dev checkout: ./opentr.sh start dev auto-loads the sidecar the same way — when the export already exists, or when HUGGINGFACE_TOKEN is set so the backend can produce it on this startup. --with-diar-native / --no-diar-native force it on or off explicitly.
  • Where the weights come from: the same place PyAnnote's do. The sidecar runs ONNX/PLDA artifacts exported from the very same gated pyannote/speaker-diarization-community-1 pipeline the in-process engine loads — it is a Rust/ONNX reimplementation of that pipeline, not a different model. Nothing about this engine is redistributed by OpenTranscribe.
  • If the sidecar is not running, diarization falls back to the in-process PyAnnote engine and everything keeps working — slower, and slightly worse on AMI, but correct. That fallback is a supported configuration, not a degraded one; see issue #572 for why PyAnnote stays.
  • Inline gender classification: while the sidecar already holds the decoded audio for diarization, it can also classify each speaker's gender in the same pass (DIAR_NATIVE_GENDER, on by default). When active, this skips the separate ~87–90s CPU wav2vec2 gender-classification task described below entirely, rather than running both — verified to produce the same labels as the CPU path at higher confidence (e.g. male 0.999/female 0.989 vs. 0.999/0.593).

Speaker Verification Status​

Track identification confidence:

  • ✅ Verified: Manually confirmed
  • AI Suggested: LLM identification
  • Auto-Matched: Voice fingerprint match (75% or more)
  • Unverified: Default detection

Merge & Split (Enhanced in v0.2.0)​

Merge speakers (New UI):

  • Visual speaker merge interface with segment preview
  • Select primary speaker and merge others into it
  • Automatic segment reassignment
  • Consolidate speaker profiles across files
  • Update all segments automatically

Split speakers:

  • Separate incorrectly merged speakers
  • Re-assign segments
  • Create new profiles

Per-File Speaker Settings (New in v0.2.0)​

Configure speaker detection for each upload or reprocess:

  • Upload dialog: Set min/max (or an exact count) in the Speakers step
  • Reprocess dialog: Adjust speaker range for re-transcription or re-diarization
  • User preferences: Speaker count when uploading, under Settings → Speaker Identification → Speaker Detection → Number of speakers
    • Use my range, but let me change it per file: the Speakers step is pre-filled with your range
    • Leave the range empty on upload: the system default range applies
    • Always use my range: your saved range is used on every upload

Whether the range is used depends on the detector; see Speaker-count hints.

Performance​

Accuracy​

  • Speaker detection: ~95% on clear audio
  • Voice matching: ~90% across videos
  • Overlap detection: ~85% accuracy
  • Overlap confidence scoring: Reliable 0-100 scale

Processing Time (PyAnnote v4)​

Startup Impact:

  • Cold start (first transcription): Normal baseline + model loading
  • Warm cache (subsequent transcriptions): 40-60 seconds faster due to pre-loaded models
  • Fast speaker assignment: 273x faster clustering on GPU

Per-File Processing (this server's engine):

  • Base diarization: ~20-30 seconds for 1-hour audio
  • No separate alignment pass: word timestamps come from transcription itself
  • With the built-in GPU model, diarization runs in the same task right after transcription (the transcription model's GPU memory is released first)
  • Diarization adds ~30% overhead to total transcription time

Example Timelines (with warm cache):

  • 30 minutes audio: ~20 seconds diarization
  • 1 hour audio: ~30 seconds diarization
  • 2 hours audio: ~45 seconds diarization

Resource Usage​

  • VRAM: Additional 2GB for diarization (256-dimensional embeddings)
  • Total: 8GB VRAM recommended
  • CPU mode: Slower but functional
  • Model Cache: ~2.6GB total (huggingface + torch models)

Use Cases​

Business Meetings​

  • Identify all participants
  • Track who spoke when
  • Generate speaker-attributed notes
  • Analyze participation balance

Interviews​

  • Separate interviewer and subject
  • Quote attribution
  • Response time analysis
  • Follow-up question tracking

Podcasts​

  • Multi-host identification
  • Guest speaker tracking
  • Episode analytics
  • Automated show notes

Legal/Medical​

  • Court proceedings transcription
  • Medical consultation records
  • Deposition transcripts
  • Accurate speaker attribution

Performance Optimization​

OpenTranscribe v4 includes several optimizations to speed up speaker diarization processing:

Warm Model Caching​

What it does: AI models are loaded into memory on server startup and kept warm between transcriptions.

Benefits:

  • 40-60 seconds faster on first transcription after startup
  • Subsequent transcriptions use pre-loaded models
  • No repeated model initialization overhead
  • Configurable cache location via MODEL_CACHE_DIR environment variable

How it works:

# Models cached in these locations
${MODEL_CACHE_DIR}/huggingface/ # PyAnnote transformer model (~500MB)
${MODEL_CACHE_DIR}/torch/pyannote/ # PyAnnote voice embeddings

Fast Speaker Assignment​

What it does: PyAnnote v4 uses optimized clustering algorithms for speaker detection.

Performance:

  • 273x faster than previous speaker assignment methods
  • Clustering completes in milliseconds vs seconds
  • Particularly noticeable on high-speaker-count files (10+ speakers)

Technical: Uses sklearn's AgglomerativeClustering with optimized similarity thresholds

Native Word-Level Timestamps​

Word-level timestamps are generated natively during transcription via faster-whisper's cross-attention DTW mechanism. No separate alignment model runs: the WAV2VEC2 forced-alignment step the old WhisperX pipeline used is gone, and word timing works for all 100+ languages without any additional processing overhead. Post-processing validates and corrects timestamps for monotonicity and drift.

Technical Deep Dive​

Why Agglomerative Hierarchical Clustering (AHC)?​

PyAnnote's diarization pipeline uses sklearn's AgglomerativeClustering for grouping speaker embeddings. This was chosen over several alternatives after evaluating tradeoffs across accuracy, scalability, and operational characteristics (#144):

AlgorithmK Required?Handles Outliers?Incremental?ComplexityVerdict
AHC (chosen)No (threshold-based)NoNoO(n^2 log n)Best for per-file diarization with unknown speaker count
K-MeansYesNoLimitedO(nKI)Requires knowing speaker count in advance -- unusable for diarization
SpectralYes (or auto-tune)NoNoO(n^3)Cubic complexity prohibitive for long files; used internally by PyAnnote for refinement
DBSCANNo (needs epsilon)YesNoO(n log n)Good for cross-file batch clustering but epsilon tuning is fragile for variable-quality audio
HDBSCANNoYes (noise points)NoO(n log n)Best for periodic batch re-clustering across files; overkill for single-file diarization

AHC with a cosine similarity distance threshold is the standard approach in speaker diarization research because it does not require pre-specifying the number of speakers (K). Instead, it merges clusters until the inter-cluster distance exceeds a tuned threshold. This matches the real-world scenario where the number of speakers in a recording is unknown. NVIDIA NeMo's speaker verification pipeline uses a similar cosine similarity threshold of 0.7 as a reference point.

For cross-file speaker management at scale, OpenTranscribe uses a hybrid approach: real-time kNN assignment via OpenSearch HNSW (O(log n) per speaker) for immediate cluster routing, with periodic HDBSCAN batch re-clustering to discover groups missed by incremental assignment.

Cosine Similarity Thresholds​

Speaker matching thresholds were calibrated against industry benchmarks from Pindrop (5B+ calls processed), NVIDIA NeMo, and internal testing:

ThresholdActionRationale
>= 0.75Auto-accept: the matching profile's name is appliedMatches OpenTranscribe's SPEAKER_CONFIDENCE_HIGH; conservative to prevent false merges
0.65 - 0.75Cluster formationFollows WeSpeaker's UMAP + HDBSCAN recipe threshold; groups likely-same speakers
0.50 - 0.65Weak suggestionPresented as option but not auto-applied
< 0.50Different speakersIndustry standard cutoff

The conservative auto-accept threshold (0.75) was chosen specifically to prevent false positive merges, which are harder to recover from than missed matches. Profile coherence monitoring flags profiles where constituent embeddings have minimum pairwise similarity below 0.60, indicating a potential false merge.

Why 256-Dimensional WeSpeaker Embeddings​

PyAnnote v4 switched from 192-dimensional pyannote/embedding (v3) to 256-dimensional WeSpeaker ResNet34-LM embeddings. This was not simply a version bump -- it reflects a deliberate accuracy-performance tradeoff:

  • Accuracy: WeSpeaker ResNet34-LM achieves state-of-the-art results on VoxCeleb benchmarks. The 256-dim space provides a richer representation than 192-dim while remaining computationally efficient.
  • Speed: 273x faster speaker assignment (from ~3 seconds to ~11ms per speaker) due to optimized ONNX runtime and better clustering algorithms in v4.
  • Storage: 256-dim vectors are 33% larger than 192-dim but 50% smaller than the 512-dim embeddings used by the legacy pyannote/embedding model. At scale (30,000 speakers), the full OpenSearch index remains under 32MB.
  • L2-normalization: All embeddings are L2-normalized before storage, which means cosine similarity reduces to a simple dot product -- enabling efficient HNSW search in OpenSearch.

The 512-dim pyannote/embedding model is retained in the speakers_v3 index for backward compatibility and can be used as a fallback. The alias-based index architecture allows atomic switching between v3 and v4 without downtime (see Embedding Migration).

Gender Classification Model Selection​

Speaker attribute detection uses the prithivMLmods/Wav2Vec2-Gender-Age-Classification model (#141). It classifies voice attributes only; it plays no part in word timing or alignment. This model was selected based on three criteria:

  1. Apache 2.0 licensing: Unlike many voice classification models that use restrictive research-only licenses, this model's Apache 2.0 license permits commercial and production use without legal risk. This was a hard requirement for OpenTranscribe's open-source distribution.
  2. Language independence: The model analyzes acoustic features (pitch, formant frequencies, timbre) rather than linguistic content, so it works across all 100+ supported transcription languages without retraining.
  3. Lightweight inference: Runs on CPU (~1-3 seconds per speaker) as a separate Celery task on the cpu queue, so it never blocks GPU resources needed for transcription and diarization.

Gender-Informed Cluster Validation​

Predicted gender is used as a validation signal for speaker clusters, not as a primary clustering feature. When a speaker cluster contains members with conflicting gender predictions, the system applies a cross-gender penalty:

  • Same predicted gender: Standard cosine similarity threshold (0.65) for cluster membership.
  • Cross-gender assignment: Requires a higher similarity threshold to join the cluster, reducing false merges between speakers of different genders who happen to have similar acoustic profiles.
  • Minority-gender members: If a cluster has a dominant gender prediction (e.g., 8 male, 1 female), the minority member is flagged for manual review as a potential mis-assignment.

This approach improves cluster purity without over-relying on gender prediction accuracy, which can be unreliable for children, whispered speech, or heavily processed audio.

Troubleshooting​

Too Many Speakers Detected​

Solutions:

  • Lower the maximum in your speaker range (Settings, the upload wizard's Speakers step, or the reprocess dialog). The default diar-native engine ignores the range, so this helps only with pyannote.ai, a provider that supports count hints, or the in-process PyAnnote engine
  • Improve audio quality
  • Merge duplicate speakers manually

Too Few Speakers Detected​

Solutions:

  • Raise the minimum, or set an exact count, in your speaker range (same caveat as above)
  • Check audio has multiple distinct voices
  • Verify speakers have different vocal characteristics

Poor Speaker Separation​

Causes:

  • Similar-sounding voices
  • Poor audio quality
  • Heavy background noise

Solutions:

  • Use better microphones
  • Separate speakers physically
  • Post-process manually

Next Steps​