Operational Runbooks
This page provides step-by-step procedures for diagnosing and resolving common production issues in OpenTranscribe.
Runbook: Stuck Transcription Tasks
Symptoms: Files remain in "Processing" status indefinitely. The progress bar stops updating. No new files are being processed.
Diagnosis:
- Open the Flower dashboard at
http://localhost:5175/flower - Check the Active tab for tasks that have been running longer than expected (on a GPU, transcription runs far faster than real time; see Performance Tuning for measured figures)
- Check the backend logs for errors:
./opentr.sh logs celery-worker
- Query the database for stuck files:
./opentr.sh shell backendpython -c "from app.db.session import SessionLocalfrom app.models.media import MediaFile, FileStatusdb = SessionLocal()stuck = db.query(MediaFile).filter(MediaFile.status == FileStatus.PROCESSING).all()for f in stuck:print(f'{f.id}: {f.filename} - {f.status} (updated: {f.updated_at})')"
Resolution:
Option A: Use the Admin Recovery Panel (recommended)
- Log in as an admin user
- Open Settings > File status > Recovery (the Task Health Monitor panel)
- The system automatically detects stuck tasks (files in processing state for longer than the configured timeout)
- Use Recover Stuck Tasks to reset them to a retryable state
- The built-in
TaskRecoveryServicewill clean up partial transcript segments before retrying
Option B: Manual recovery via API
- Call the admin recovery endpoint:
curl -X POST http://localhost:5174/api/tasks/recover-stuck-tasks \-H "Authorization: Bearer <admin-access-token>"
Option C: Cancel and retry individual files
- In the UI, click Cancel on the stuck file and wait for it to reach
cancelled - Click Retry to queue it again. A retry keeps the file's own upload options (model, speaker counts) and is otherwise planned from the owner's current transcription and speaker-detection settings
Prevention:
- The
TaskRecoveryServiceruns automatically on a schedule and handles most stuck task scenarios - A file waiting for a worker (for example while GPU workers are stopped or scaled to zero) stays in "Processing" and is not a stuck task: recovery only acts on a transcription whose worker stopped heartbeating or whose queue message is gone
- Recovery thresholds are environment variables — see Task Recovery
- Monitor the Flower dashboard regularly for long-running tasks
Runbook: File Refused Before Processing
Symptoms: A file goes to error right after upload or retry, without any transcription
progress. Its error message is a plain sentence about settings, for example "This deployment has
no local speaker detection. Choose pyannote.ai or a provider that returns speakers." or
"pyannote.ai speaker detection needs a pyannote.ai API key. Save your key first, then choose
this option."
Cause: Before anything is queued, OpenTranscribe works out how the file will be processed from the owner's transcription and speaker-detection settings. A combination that cannot run on this deployment is refused at that point, so no worker is involved and there is nothing stuck to recover.
Resolution:
- Read the file's error message; it names what to change.
- Have the file owner fix the setting under Settings → Transcription or Settings → Speaker Identification → Speaker Detection (for example pick a cloud provider on a lite deployment, or save a pyannote.ai key).
- Retry the file.
On a lite (cloud-only) deployment the GPU worker celery-worker is scaled to zero, so its logs
are empty. Cloud transcription and cloud speaker detection run in celery-cloud-asr-worker, and
voiceprints in celery-cpu-worker:
./opentr.sh logs celery-cloud-asr-worker
./opentr.sh logs celery-cpu-worker
Runbook: Queue Backup / Slow Processing
Symptoms: New uploads queue up but are not processed. Flower shows a growing number of tasks in the Reserved or Scheduled state. Processing takes much longer than usual.
Diagnosis:
- Check queue depth in Flower at
http://localhost:5175/flower - Check if the Celery worker is running:
docker compose ps celery-worker
- Check worker logs for errors:
./opentr.sh logs celery-worker
- Check Redis connectivity (the message broker):
docker compose exec redis redis-cli ping# Should respond: PONG
- Check GPU utilization (if applicable):
docker compose exec celery-worker nvidia-smi
Resolution:
Clear stuck tasks first:
- Follow the "Stuck Transcription Tasks" runbook above to clear any blocked tasks
Scale workers if processing is just slow:
- For multi-GPU systems, enable GPU scaling:
# In .env (GPU_SCALE_ENABLED is not the switch):GPU_SCALE_DEVICE_ID=2GPU_SCALE_WORKERS=4# The --gpu-scale flag on the next start is what actually enables scaling./opentr.sh stop./opentr.sh start dev --gpu-scale
Restart the worker if it is unhealthy:
- Restart only the Celery worker:
docker compose restart celery-worker
Flush the queue as a last resort (WARNING: loses queued tasks):
- Purge all pending Celery tasks:
docker compose exec celery-worker celery -A app.core.celery purge -f
Prevention:
- Monitor queue depth via Flower
- Scale GPU workers to match throughput needs
- Set up alerts on queue depth thresholds
Runbook: GPU Out of Memory (OOM)
Symptoms: Celery worker crashes or restarts unexpectedly. Logs show CUDA out of memory, RuntimeError: CUDA error: out of memory, or the container is killed by the OOM killer.
Diagnosis:
- Check worker logs for OOM errors:
./opentr.sh logs celery-worker | grep -i "out of memory\|OOM\|CUDA error"
- Check current GPU memory usage:
docker compose exec celery-worker nvidia-smi
- Check if multiple workers are competing for GPU memory:
docker compose ps | grep celery
Resolution:
Step 1: Reduce concurrency
Lower the number of parallel workers to reduce simultaneous GPU memory usage:
# In .env:
GPU_SCALE_WORKERS=2 # Reduce from 4 to 2
Restart the worker after changing this value. (GPU_SCALE_WORKERS only applies with --gpu-scale; on a single default worker, lower GPU_CONCURRENT_REQUESTS instead.)
Step 2: Use a smaller Whisper model
Switch from large-v3 (about 10GB VRAM) to large-v3-turbo (about 6GB VRAM), either in the admin UI or in .env (the UI wins):
# In .env:
WHISPER_MODEL=large-v3-turbo
Restart the worker. Note: large-v3-turbo does not support translation tasks.
Step 3: Check for GPU memory leaks
If memory usage grows over time without releasing:
# Restart the worker to free all GPU memory
docker compose restart celery-worker
# Monitor memory usage over time
watch -n 5 'docker compose exec celery-worker nvidia-smi'
Step 4: Pin the worker to a dedicated GPU (advanced)
Docker cannot cap GPU memory, but you can keep the worker off a card shared with other workloads
(GPU_DEVICE_ID in .env, or --gpu-device N on ./opentr.sh start). The reservation in
docker-compose.yml looks like:
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
device_ids: ['0']
Prevention:
- Match
GPU_SCALE_WORKERSto your GPU's VRAM capacity - Use
large-v3-turbofor GPUs with less than 10GB VRAM - Monitor GPU memory with
nvidia-smiafter deployments - Avoid running other GPU workloads on the same device
Runbook: Database Connection Issues
Symptoms: Backend returns 500 errors. Logs show connection refused, too many connections, or connection pool exhausted. API requests hang or timeout.
The commands in this runbook use the default POSTGRES_USER=postgres and database opentranscribe. Substitute your own POSTGRES_USER if you changed it.
Diagnosis:
- Check if PostgreSQL is running:
docker compose ps postgres
- Test database connectivity:
docker compose exec postgres pg_isready -U postgres
- Check current connection count:
docker compose exec postgres psql -U postgres -c \"SELECT count(*) FROM pg_stat_activity;"
- Check backend logs for connection errors:
./opentr.sh logs backend | grep -i "connection\|pool\|database"
Resolution:
If PostgreSQL is down:
docker compose restart postgres
# Wait for it to become healthy
docker compose exec postgres pg_isready -U postgres
# Then restart the backend to re-establish connections
docker compose restart backend
If connection pool is exhausted:
- Increase the pool size in
.env:DB_POOL_SIZE=20 # API default is 20 (workers use a smaller pool, WORKER_DB_POOL_SIZE=2)DB_MAX_OVERFLOW=40 # API default is 40 (workers: WORKER_DB_MAX_OVERFLOW=3) - Restart the backend:
docker compose restart backend
If connections are leaking (count keeps growing):
- Kill idle connections:
docker compose exec postgres psql -U postgres -c \"SELECT pg_terminate_backend(pid)FROM pg_stat_activityWHERE state = 'idle'AND query_start < now() - interval '10 minutes';"
- Restart the backend to get fresh connections:
docker compose restart backend
Prevention:
- Monitor active connection count
- Set appropriate
DB_POOL_SIZE/DB_MAX_OVERFLOWfor your workload - Ensure the backend properly closes sessions (SQLAlchemy context managers)
Runbook: OpenSearch Index Corruption
Symptoms: Search returns no results or incomplete results. Admin panel shows index health as "red" or "yellow". Logs show index_not_found_exception or shard allocation errors.
Diagnosis:
- Check cluster health:
curl -s http://localhost:5180/_cluster/health | python3 -m json.tool
- Check index status:
curl -s http://localhost:5180/_cat/indices?v
- Check for unassigned shards:
curl -s http://localhost:5180/_cat/shards?v | grep UNASSIGNED
- Check the speaker alias:
curl -s http://localhost:5180/_cat/aliases?v
Resolution:
For missing or corrupt transcription index:
-
Trigger a full reindex from the settings UI:
- Open Settings > Data & Storage > Search & Indexing
- Check Index Health, and use Repair Indices for a red index
- Click Re-index All Transcripts
- This rebuilds the search index from the database (source of truth)
-
Or via API:
curl -X POST http://localhost:5174/api/search/reindex \-H "Authorization: Bearer <admin-access-token>"
For unassigned shards:
# Retry shard allocation
curl -X POST http://localhost:5180/_cluster/reroute?retry_failed=true
# If single-node, ensure replica count is 0
curl -X PUT http://localhost:5180/_settings -H 'Content-Type: application/json' -d '
{
"index.number_of_replicas": 0
}'
For corrupt speaker index:
The speaker index uses an alias-based architecture (speakers alias pointing to speakers_v3 or speakers_v4).
Unlike the transcript indices, the speakers_v* indices cannot be rebuilt from PostgreSQL, which stores no embedding vectors. Deleting one destroys the speaker embeddings unless a *.voiceprints.ndjson backup artifact exists (see Backup & Restore). Take ./opentranscribe.sh backup first, then try Settings > Data & Storage > Search & Indexing > Repair Indices, which tries close/reopen and force merge before any rebuild. Only delete an index you have an artifact for.
If a rebuild is unavoidable:
# Check which index the alias points to
curl -s http://localhost:5180/_cat/aliases/speakers?v
# Delete the corrupt index
curl -X DELETE http://localhost:5180/speakers_v3
# Restart the backend - it will recreate indices on startup
./opentr.sh restart-backend
Prevention:
- Run OpenSearch on reliable storage (avoid network-mounted volumes for production)
- Monitor cluster health regularly
- Keep regular database backups (the database is the source of truth, not OpenSearch)
Runbook: Full Disk Space
Symptoms: Uploads fail. Container logs show No space left on device. Docker commands fail. Services crash and cannot restart.
Diagnosis:
- Check overall disk usage:
df -h /
- Check Docker disk usage:
docker system df
- Check large directories:
du -sh /var/lib/docker/*du -sh ./models/*du -sh ./backups/*
Resolution:
Step 1: Clean Docker resources
# Remove unused containers, networks, and dangling images
docker system prune -f
# Do NOT run `docker volume prune` here: it deletes every volume no running
# container uses, which includes the data volumes of a stopped OpenTranscribe
# stack (and of any other project on a shared Docker host).
# Remove old images
docker image prune -a -f --filter "until=168h"
Step 2: Clean temporary files
# Clean backend temp files
docker compose exec backend rm -rf /tmp/yt-dlp-*
# Check MinIO for orphaned files
# (files in storage but not referenced in the database)
Step 3: Archive old backups
# List backups sorted by size
ls -lhS backups/
# Remove backups older than 30 days (a dump and its .voiceprints.ndjson share the prefix;
# delete them together, never one without the other)
find backups/ -name "opentranscribe_backup_*" -mtime +30 -delete
Step 4: Move model cache to larger disk
# In .env, change MODEL_CACHE_DIR to a path on a larger disk
MODEL_CACHE_DIR=/mnt/large-disk/opentranscribe-models
# Fix permissions and restart
./scripts/fix-model-permissions.sh
./opentr.sh stop && ./opentr.sh start dev
Prevention:
- Monitor disk usage with alerts at 80% and 90% thresholds
- Set up automated backup rotation
- Use a separate volume for Docker storage and model cache
- Periodically run
docker system prune
Runbook: Service Won't Start
Symptoms: Running ./opentr.sh start dev fails. One or more containers exit immediately or enter a restart loop. The frontend or backend is unreachable.
Diagnosis:
- Check which services are failing:
./opentr.sh statusdocker compose ps -a
- Check logs for the failing service:
./opentr.sh logs backend./opentr.sh logs frontend./opentr.sh logs postgres
- Check for port conflicts:
# Check if ports are already in usess -tlnp | grep -E '5173|5174|5175|5179|5180'
- Verify the
.envfile exists and is valid:# Check for syntax errors (no spaces around =, no missing quotes)cat .env | grep -n "= \|^ \|^\t" - Check Docker daemon is running:
docker info
Resolution:
If a port is in use:
# Find what is using the port
lsof -i:5174
# If it's another OpenTranscribe stack (including a --fresh deployment), stop
# it properly instead of killing the PID directly:
./opentr.sh stop
# (or, for a --fresh deployment: ./opentr.sh stop --fresh <name>)
# Never `kill`/`kill -9` a container process directly -- a SIGKILL mid-task
# can wedge a CUDA context and require a machine restart to recover the GPU.
# Or change the port in .env
BACKEND_PORT=5184
If the backend fails on startup (migration error, missing env var):
# Check the specific error in logs
./opentr.sh logs backend | tail -50
# Common fix: ensure .env has all required variables
diff .env .env.example
If containers keep restarting:
# Check exit codes
docker compose ps -a
# Check for resource constraints
docker stats --no-stream
If Docker daemon issues:
sudo systemctl restart docker
./opentr.sh stop && ./opentr.sh start dev
Prevention:
- Keep
.envin sync with.env.examplewhen upgrading - Test configuration changes in development before production
- Use
./opentr.sh statusafter starting to verify all services are healthy
Runbook: Rolling back past self-service erasure
When: you are rolling the application back to a release before self-service account erasure,
or downgrading migration v455.
Why it matters: older code ignores the new ledger columns and treats a waiting self-service request as an ordinary deferred erasure, so it would erase the account early.
- Turn the feature off: Settings > Privacy & Redaction > policy tab, or
PUT /api/admin/self-erasurewith{"enabled": false}. New requests are refused (404). - List the open requests (ledger entries with
status = deferredanddeferred_reason = grace_period) and either let them run or ask the people to cancel from Settings > Profile & Security. - Only when none are open, downgrade. The downgrade refuses while any request is open, and deletes cancelled requests (their status does not exist in the old schema).
- Then roll the code back.
Worst-case execution lag after a waiting period ends is one reconciliation pass (daily by default).
Runbook: Failed Database Migration
Symptoms: Backend fails to start with Alembic errors. Logs show alembic.util.exc.CommandError, Can't locate revision, or Target database is not up to date.
Diagnosis:
- Check backend logs for the specific migration error:
./opentr.sh logs backend | grep -i "alembic\|migration\|revision"
- Check the current database revision:
./opentr.sh shell backendalembic current
- Check migration history:
alembic history --verbose
Resolution:
If the database has no Alembic stamp (fresh or legacy database):
The backend's migrations.py auto-detection handles this on startup. If it fails:
./opentr.sh shell backend
# Stamp the database at the appropriate version
alembic stamp <current_version>
# Then upgrade
alembic upgrade head
If a migration partially applied (some columns exist, some don't):
- Check what was applied:
./opentr.sh shell postgrespsql -U postgres -d opentranscribe -c "\d+ <table_name>"
- Manually complete the migration or stamp past it:
./opentr.sh shell backend# If the schema is correct but stamp is wrongalembic stamp <target_revision>
If you need to rollback a migration:
./opentr.sh shell backend
# Downgrade one revision
alembic downgrade -1
# Or downgrade to a specific revision
alembic downgrade <revision_id>
Nuclear option (development only -- destroys all data):
./opentr.sh reset dev
Prevention:
- Always use idempotent SQL in migrations (
IF NOT EXISTS,IF EXISTS) - Test migrations with
./opentr.sh reset devbefore deploying - Back up the database before applying new migrations in production:
./opentranscribe.sh backup
Runbook: High Memory Usage
Symptoms: System becomes sluggish. Docker containers are killed by the OOM killer. docker stats shows high memory consumption. Swap usage is high.
Diagnosis:
- Check per-container memory usage:
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"
- Identify the largest consumer:
docker stats --no-stream --format "{{.Name}}\t{{.MemUsage}}" | sort -k2 -h
- Check host system memory:
free -h
Resolution:
If OpenSearch is consuming too much memory:
OpenSearch defaults can be aggressive. Reduce the JVM heap:
# In docker-compose.yml or .env:
OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g # Default is 4g
1 GB is the measured floor for the default embedding model — below it a model reports
DEPLOY: COMPLETED and then fails to produce an embedding. See
Performance Tuning.
Restart OpenSearch after changing.
If the Celery worker is consuming too much:
This is usually due to model loading. Each model stays in memory:
# Restart the worker to free memory
docker compose restart celery-worker
If PostgreSQL is consuming too much:
Reduce shared buffers:
# In .env, then restart postgres:
PG_SHARED_BUFFERS=256MB # This is already the default; lower it only on very small hosts
Set container memory limits:
Add limits in docker-compose.yml:
services:
backend:
deploy:
resources:
limits:
memory: 2G
opensearch:
deploy:
resources:
limits:
memory: 2G
Prevention:
- Set memory limits on all containers
- Monitor memory usage trends over time
- Size your server appropriately (see the hardware table in Production Deployment)
Runbook: Credential Rotation
Symptoms: Planned maintenance -- you need to rotate credentials for security compliance or after a suspected compromise.
Diagnosis: Not applicable (proactive maintenance).
Resolution:
Rotate Database Password
- Back up the database first:
./opentranscribe.sh backup
- Update the password in PostgreSQL:
docker compose exec postgres psql -U postgres -c \"ALTER USER postgres WITH PASSWORD 'new_secure_password';"
- Update
.env(DATABASE_URLis empty by default and assembled fromPOSTGRES_*; only edit it if you set it explicitly):POSTGRES_PASSWORD=new_secure_password - Restart services that connect to the database:
docker compose restart backend celery-worker
Rotate MinIO Keys
- Update
.env(MinIO reads the root credentials at start-up):MINIO_ROOT_USER=NEW_ACCESS_KEYMINIO_ROOT_PASSWORD=NEW_SECRET_KEY - Restart the stack so MinIO and every service that talks to it pick up the new values:
This does not touch./opentr.sh stop && ./opentr.sh start dev # or: start prod
MINIO_KMS_SECRET_KEY, the at-rest encryption key; changing that one would make existing encrypted objects unreadable.
Rotate JWT Secret
- Update
.env:JWT_SECRET_KEY=$(openssl rand -hex 32) - Restart the backend:
docker compose restart backend
- Note: All existing user sessions will be invalidated. Users will need to log in again.
Rotate LLM API Keys
- Generate a new key from your LLM provider's dashboard
- Update it in the app: Settings > LLM Provider Configuration (provider keys are stored encrypted in the database)
- For a self-hosted vLLM server configured through
.env, updateVLLM_API_KEYthere and restart the backend andcelery-nlp-worker:docker compose restart backend celery-nlp-worker
Rotate Redis Password
- Update
.env:(The broker and result-backend URLs are built fromREDIS_PASSWORD=new_redis_passwordREDIS_PASSWORD; only editREDIS_URLorCELERY_BROKER_URLif you set them explicitly.) - Restart all services that use Redis:
docker compose restart redis backend celery-worker
Do not rotate ENCRYPTION_KEY
ENCRYPTION_KEY decrypts every secret stored in the database (provider API keys, MFA secrets, backup and watch-source credentials). Changing it makes them permanently undecryptable; there is no re-encryption step. Keep it, and keep a copy with your backups (see Backup Completeness Audit).
Prevention:
- Rotate credentials on a regular schedule (e.g., every 90 days)
- Use strong, randomly generated passwords (
openssl rand -hex 32) - Document credential rotation dates
- Never commit credentials to version control
Runbook: Model Re-Download
Symptoms: AI models are corrupted, producing garbled output, or you need to switch to a different model version. Or you want to force a fresh download after a failed partial download.
Diagnosis:
- Check current model cache:
ls -la ${MODEL_CACHE_DIR:-./models}/huggingface/hub/ls -la ${MODEL_CACHE_DIR:-./models}/torch/pyannote/
- Check for incomplete downloads (very small files, lock files):
find ${MODEL_CACHE_DIR:-./models} -name "*.lock" -o -name "*.incomplete"
Resolution:
Re-download a specific model type:
# Clear Whisper models
rm -rf ${MODEL_CACHE_DIR:-./models}/huggingface/hub/models--Systran--faster-whisper-*
# Clear PyAnnote speaker models (the in-process fallback engine)
rm -rf ${MODEL_CACHE_DIR:-./models}/torch/pyannote/
# Clear the diar-native speaker engine's exported models (DIAR_NATIVE_MODELS_DIR if set);
# the backend exports them again on its next start
rm -rf ${MODEL_CACHE_DIR:-./models}/diar-native/
# Clear sentence transformer models
rm -rf ${MODEL_CACHE_DIR:-./models}/sentence-transformers/
# Clear OpenSearch neural models
rm -rf ${MODEL_CACHE_DIR:-./models}/opensearch-ml/
Re-download all models:
# Stop services
./opentr.sh stop
# Clear the entire model cache
rm -rf ${MODEL_CACHE_DIR:-./models}/*
# Fix permissions
./scripts/fix-model-permissions.sh
# Restart -- models will download on first use
./opentr.sh start dev
Pre-download models for offline deployment:
bash scripts/download-models.sh models
Prevention:
- Verify model downloads completed successfully after first startup
- Use stable network connections for initial model downloads
- For air-gapped environments, pre-download models on an internet-connected machine and transfer via
rsync
Runbook: WebSocket Connection Issues
Symptoms: The UI does not update in real-time after uploading files. Progress bars do not animate. Transcription completes but the UI still shows "Processing". Browser console shows WebSocket errors.
Diagnosis:
- Check browser developer console (F12) for WebSocket errors:
- Look for
WebSocket connection to 'ws://...' failed - Look for
ERR_CONNECTION_REFUSEDor403 Forbidden
- Look for
- Verify the backend WebSocket endpoint is reachable:
# Test with wscat (install: npm install -g wscat)wscat -c ws://localhost:5174/api/ws
- Check backend logs for WebSocket errors:
./opentr.sh logs backend | grep -i "websocket\|ws\|upgrade"
- If using NGINX (production), check NGINX logs:
./opentr.sh logs nginx | grep -i "websocket\|upgrade"
Resolution:
If running in development (no NGINX):
- Verify the backend is running and healthy:
curl http://localhost:5174/health
- Restart the backend:
docker compose restart backend
- Hard-refresh the browser (
Ctrl+Shift+R)
If running in production (with NGINX):
- Verify NGINX is configured for WebSocket proxying. The configuration should include:
location /api/ws {proxy_pass http://backend:8080;proxy_http_version 1.1;proxy_set_header Upgrade $http_upgrade;proxy_set_header Connection "upgrade";proxy_set_header Host $host;proxy_read_timeout 86400;}
- Check that
proxy_read_timeoutis set high enough (WebSocket connections are long-lived) - Restart NGINX:
docker compose restart nginx
If WebSocket connects but no updates arrive:
- Check that the Celery worker is sending notifications:
./opentr.sh logs celery-worker | grep -i "notification\|websocket\|send"
- Check Redis pub/sub (used for WebSocket message passing):
docker compose exec redis redis-cli ping
Prevention:
- In production, always use the provided NGINX configuration which includes WebSocket support
- Set
proxy_read_timeoutto at least 86400 (24 hours) for WebSocket connections - Monitor WebSocket connection counts in production
- Test WebSocket functionality after any NGINX configuration changes