OpenTranscribe — Celery Queues (inside "Celery Workers")

OpenTranscribe — Celery Queues (inside "Celery Workers") An architecture diagram generated by Archify. Task Router · celery.py task_routes · Architecture component Task Router celery.py task_routes gpu · ASR, diarization · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) · concurrency=1 gpu ASR, diarization concurrency=1 cloud-asr · + cpu-transcribe, dynamic · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) cloud-asr + cpu-transcribe, dynamic cpu · embeddings, thumbnails, reindex · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) · concurrency=8 cpu embeddings, thumbnails, reindex concurrency=8 download · yt-dlp URL/playlist fetch · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) · concurrency=3 download yt-dlp URL/playlist fetch concurrency=3 nlp · summary, topics, speaker-ID · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) · concurrency=4 nlp summary, topics, speaker-ID concurrency=4 embedding · chunk-level search indexing · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) · concurrency=1 embedding chunk-level search indexing concurrency=1 redaction · PII/toxicity detection · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) redaction PII/toxicity detection utility · backups, pipeline errors, index writes · Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards) utility backups, pipeline errors, index writes MinIO · audio, thumbnails, waveforms · Architecture component MinIO audio, thumbnails, waveforms gpu/cpu/download read+write media Celery Workers (docker-compose.gpu-scale.yml can multiply the gpu queue across cards)

task_routes is the single source of truth

  • • task_create_missing_queues=False — an unrouted task raises at dispatch, it never silently lands somewhere wrong
  • • cloud-asr / cpu-transcribe are dynamic: dispatch.py picks the queue per request, not a static route

Why extract_speaker_embeddings is CPU, not GPU

  • • It only reads already-known segments — no diarization model pass needed
  • • Lite mode scales the gpu queue to zero workers; pinning this to gpu left it stuck forever (issue #584)

Workers hit MinIO directly

  • • gpu/cpu/download tasks read source audio and write thumbnails/waveforms straight to MinIO — not proxied through the backend
  • • gpu: one physical card, one job at a time; embedding: the embedding model itself is the bottleneck, not hardware