Skip to main content

Multi-GPU Worker Scaling

For systems with multiple GPUs, OpenTranscribe supports parallel GPU workers to dramatically increase transcription throughput.

Overview​

Standard Setup: 1 GPU = 1 worker, about 39x realtime end to end (measured, RTX A6000, large-v3-turbo)

Scaled Setup: 1 GPU = several workers in one container, transcribing several files simultaneously. Total throughput rises with the worker count but sub-linearly (about 51x at 4 workers in the same measurement).

When to Use​

Multi-GPU scaling is ideal for:

  • Batch processing large numbers of files
  • High-throughput production systems
  • Systems with dedicated transcription GPU
  • Workflows with concurrent uploads

--gpu-scale vs. --with-gpu-split​

These are two distinct multi-GPU features with different goals, enabled independently:

--gpu-scale (this page)--with-gpu-split
What it doesRuns N parallel Celery workers in one container against a dedicated GPU, for higher throughput (more files transcribed concurrently)Runs transcription and diarization on separate GPUs for higher per-file performance
Enabled byThe --gpu-scale CLI flag on ./opentr.sh start dev / start prod -- not GPU_SCALE_ENABLED (see Step 1 below)The --with-gpu-split CLI flag on ./opentr.sh (dev), or ENGINE_GPU_SPLIT=true in .env alone on the shipped ./opentranscribe.sh (no --with-* flags there)
Compose overlaydocker-compose.gpu-scale.yml (COMPOSE_PROFILES=gpu-scale)docker-compose.gpu-split.yml (gpu-split profile)
TuningGPU_SCALE_WORKERS, GPU_SCALE_DEVICE_ID, GPU_SCALE_DEFAULT_WORKERGPU_TRANSCRIBE_DEVICE_ID, GPU_DIARIZE_DEVICE_ID

This page covers --gpu-scale only. For --with-gpu-split, see Deployment Configuration. The two flags combine on a 3+ GPU host if you want both behaviors at once.

Hardware Example​

GPU 0: NVIDIA RTX A6000 (49GB) - Local LLM (vLLM/Ollama)
GPU 1: RTX 3080 Ti (12GB) - Default worker (disabled when scaling)
GPU 2: NVIDIA RTX A6000 (49GB) - 4 parallel workers (scaled)

Configuration​

Step 1: Configure Environment​

GPU_SCALE_ENABLED does not enable scaling

Scaling is turned on only by the --gpu-scale CLI flag in Step 2 -- no compose file or startup script reads GPU_SCALE_ENABLED. It's consulted in exactly one unrelated place (tasks/utility.py), to pick which GPU device IDs the system-stats task queries, so a stale value misreports which GPU is in use without changing any scheduling. Don't set it expecting it to turn scaling on or off.

Edit .env to configure which GPU the scaled workers use and how many run:

# Which GPU to use for scaled workers
GPU_SCALE_DEVICE_ID=2

# Number of parallel workers. If left unset it follows DIAR_NATIVE_MAX_INFLIGHT (default 2),
# so raise the sidecar's permits together with this value (see docker-compose.gpu-scale.yml)
GPU_SCALE_WORKERS=4

# Keep the default single-GPU worker running alongside the scaled workers (1) or disable it (0)
# (.env.example ships 1; the compose default when unset is 0)
GPU_SCALE_DEFAULT_WORKER=1

Step 2: Start with Scaling​

# Development
./opentr.sh start dev --gpu-scale

# Production
./opentr.sh start prod --gpu-scale

# Reset with scaling
./opentr.sh reset dev --gpu-scale

Performance​

WorkersMeasured throughput
1 worker39.3x realtime
2 workers47.4x realtime
4 workers51.3x realtime

Figures are full-pipeline throughput (transcription, speaker detection, speaker assignment) on one RTX A6000 with large-v3-turbo, from docs/BENCHMARK_RESULTS.md; a 4-worker run is not 4x faster. Adding workers mainly improves GPU utilisation; per-file time rises as workers compete.

VRAM Requirements​

WorkersRecommended VRAMSupported Models
212GB+large-v2
424GB+large-v2
648GB+large-v2

Monitoring​

# Watch GPU usage
watch -n 1 nvidia-smi

# View scaled worker logs
./opentr.sh logs celery-worker-gpu-scaled

# Monitor task queue
# Open: http://localhost:5175/flower

Troubleshooting​

Out of Memory Errors​

Reduce worker count:

GPU_SCALE_WORKERS=2 # instead of 4

Poor GPU Utilization​

Increase worker count (if VRAM available):

GPU_SCALE_WORKERS=6 # instead of 4

Next Steps​