OpenTranscribe — Celery Queues (inside "Celery Workers")

OpenTranscribe — Celery Queues (inside "Celery Workers") An architecture diagram generated by Archify. Task Router · task_routes + plan · Architecture component Task Router task_routes + plan gpu · local Whisper + speakers · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) · concurrency=1 gpu local Whisper + speakers concurrency=1 cloud-asr · cloud ASR + cloud speakers · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) · concurrency=16 cloud-asr cloud ASR + cloud speakers concurrency=16 cpu · voiceprints, cpu-transcribe · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) · concurrency=8 cpu voiceprints, cpu-transcribe concurrency=8 download · yt-dlp URL/playlist fetch · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) · concurrency=5 download yt-dlp URL/playlist fetch concurrency=5 nlp · summary, topics, speaker-ID · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) · concurrency=4 nlp summary, topics, speaker-ID concurrency=4 embedding · chunk-level search indexing · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) · concurrency=1 embedding chunk-level search indexing concurrency=1 redaction · PII/toxicity detection · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) redaction PII/toxicity detection utility · backups, pipeline errors, index writes · Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu) utility backups, pipeline errors, index writes MinIO · audio, thumbnails, waveforms · Architecture component MinIO audio, thumbnails, waveforms gpu/cpu/download read+write media Celery Workers (lite: both gpu workers at replicas 0; gpu-scale can multiply gpu)

The plan picks the pipeline queues

  • • task_create_missing_queues=False: an unknown queue raises at dispatch instead of creating a phantom queue
  • • pipeline_plan._queues: gpu (or gpu-transcribe/gpu-diarize when split), cpu-transcribe, cloud-asr; the speaker post stage goes to cloud-asr for a vendor, gpu for this server

Voiceprints run on cpu

  • • The voiceprint stage (extract_speaker_embeddings) only embeds known speaker turns, so it needs no GPU
  • • It runs after every cloud or post-stage detector, on full and lite alike

Lite has no gpu consumers

  • • docker-compose.lite.yml sets DEPLOYMENT_MODE=lite and scales both gpu workers to 0; gpu_preferred_queue() then resolves to cpu
  • • The plan never routes local transcription or local speaker detection on lite: it refuses them before dispatch