Skip to main content

Operational Runbooks

This page provides step-by-step procedures for diagnosing and resolving common production issues in OpenTranscribe.


Runbook: Stuck Transcription Tasks​

Symptoms: Files remain in "Processing" status indefinitely. The progress bar stops updating. No new files are being processed.

Diagnosis:

  1. Open the Flower dashboard at http://localhost:5175/flower
  2. Check the Active tab for tasks that have been running longer than expected (on a GPU, transcription runs far faster than real time; see Performance Tuning for measured figures)
  3. Check the backend logs for errors:
    ./opentr.sh logs celery-worker
  4. Query the database for stuck files:
    ./opentr.sh shell backend
    python -c "
    from app.db.session import SessionLocal
    from app.models.media import MediaFile, FileStatus
    db = SessionLocal()
    stuck = db.query(MediaFile).filter(
    MediaFile.status == FileStatus.PROCESSING
    ).all()
    for f in stuck:
    print(f'{f.id}: {f.filename} - {f.status} (updated: {f.updated_at})')
    "

Resolution:

Option A: Use the Admin Recovery Panel (recommended)

  1. Log in as an admin user
  2. Open Settings > File status > Recovery (the Task Health Monitor panel)
  3. The system automatically detects stuck tasks (files in processing state for longer than the configured timeout)
  4. Use Recover Stuck Tasks to reset them to a retryable state
  5. The built-in TaskRecoveryService will clean up partial transcript segments before retrying

Option B: Manual recovery via API

  1. Call the admin recovery endpoint:
    curl -X POST http://localhost:5174/api/tasks/recover-stuck-tasks \
    -H "Authorization: Bearer <admin-access-token>"

Option C: Cancel and retry individual files

  1. In the UI, click Cancel on the stuck file and wait for it to reach cancelled
  2. Click Retry to queue it again. A retry keeps the file's own upload options (model, speaker counts) and is otherwise planned from the owner's current transcription and speaker-detection settings

Prevention:

  • The TaskRecoveryService runs automatically on a schedule and handles most stuck task scenarios
  • A file waiting for a worker (for example while GPU workers are stopped or scaled to zero) stays in "Processing" and is not a stuck task: recovery only acts on a transcription whose worker stopped heartbeating or whose queue message is gone
  • Recovery thresholds are environment variables — see Task Recovery
  • Monitor the Flower dashboard regularly for long-running tasks

Runbook: File Refused Before Processing​

Symptoms: A file goes to error right after upload or retry, without any transcription progress. Its error message is a plain sentence about settings, for example "This deployment has no local speaker detection. Choose pyannote.ai or a provider that returns speakers." or "pyannote.ai speaker detection needs a pyannote.ai API key. Save your key first, then choose this option."

Cause: Before anything is queued, OpenTranscribe works out how the file will be processed from the owner's transcription and speaker-detection settings. A combination that cannot run on this deployment is refused at that point, so no worker is involved and there is nothing stuck to recover.

Resolution:

  1. Read the file's error message; it names what to change.
  2. Have the file owner fix the setting under Settings → Transcription or Settings → Speaker Identification → Speaker Detection (for example pick a cloud provider on a lite deployment, or save a pyannote.ai key).
  3. Retry the file.

On a lite (cloud-only) deployment the GPU worker celery-worker is scaled to zero, so its logs are empty. Cloud transcription and cloud speaker detection run in celery-cloud-asr-worker, and voiceprints in celery-cpu-worker:

./opentr.sh logs celery-cloud-asr-worker
./opentr.sh logs celery-cpu-worker

Runbook: Queue Backup / Slow Processing​

Symptoms: New uploads queue up but are not processed. Flower shows a growing number of tasks in the Reserved or Scheduled state. Processing takes much longer than usual.

Diagnosis:

  1. Check queue depth in Flower at http://localhost:5175/flower
  2. Check if the Celery worker is running:
    docker compose ps celery-worker
  3. Check worker logs for errors:
    ./opentr.sh logs celery-worker
  4. Check Redis connectivity (the message broker):
    docker compose exec redis redis-cli ping
    # Should respond: PONG
  5. Check GPU utilization (if applicable):
    docker compose exec celery-worker nvidia-smi

Resolution:

Clear stuck tasks first:

  1. Follow the "Stuck Transcription Tasks" runbook above to clear any blocked tasks

Scale workers if processing is just slow:

  1. For multi-GPU systems, enable GPU scaling:
    # In .env (GPU_SCALE_ENABLED is not the switch):
    GPU_SCALE_DEVICE_ID=2
    GPU_SCALE_WORKERS=4

    # The --gpu-scale flag on the next start is what actually enables scaling
    ./opentr.sh stop
    ./opentr.sh start dev --gpu-scale

Restart the worker if it is unhealthy:

  1. Restart only the Celery worker:
    docker compose restart celery-worker

Flush the queue as a last resort (WARNING: loses queued tasks):

  1. Purge all pending Celery tasks:
    docker compose exec celery-worker celery -A app.core.celery purge -f

Prevention:

  • Monitor queue depth via Flower
  • Scale GPU workers to match throughput needs
  • Set up alerts on queue depth thresholds

Runbook: GPU Out of Memory (OOM)​

Symptoms: Celery worker crashes or restarts unexpectedly. Logs show CUDA out of memory, RuntimeError: CUDA error: out of memory, or the container is killed by the OOM killer.

Diagnosis:

  1. Check worker logs for OOM errors:
    ./opentr.sh logs celery-worker | grep -i "out of memory\|OOM\|CUDA error"
  2. Check current GPU memory usage:
    docker compose exec celery-worker nvidia-smi
  3. Check if multiple workers are competing for GPU memory:
    docker compose ps | grep celery

Resolution:

Step 1: Reduce concurrency

Lower the number of parallel workers to reduce simultaneous GPU memory usage:

# In .env:
GPU_SCALE_WORKERS=2 # Reduce from 4 to 2

Restart the worker after changing this value. (GPU_SCALE_WORKERS only applies with --gpu-scale; on a single default worker, lower GPU_CONCURRENT_REQUESTS instead.)

Step 2: Use a smaller Whisper model

Switch from large-v3 (about 10GB VRAM) to large-v3-turbo (about 6GB VRAM), either in the admin UI or in .env (the UI wins):

# In .env:
WHISPER_MODEL=large-v3-turbo

Restart the worker. Note: large-v3-turbo does not support translation tasks.

Step 3: Check for GPU memory leaks

If memory usage grows over time without releasing:

# Restart the worker to free all GPU memory
docker compose restart celery-worker

# Monitor memory usage over time
watch -n 5 'docker compose exec celery-worker nvidia-smi'

Step 4: Pin the worker to a dedicated GPU (advanced)

Docker cannot cap GPU memory, but you can keep the worker off a card shared with other workloads (GPU_DEVICE_ID in .env, or --gpu-device N on ./opentr.sh start). The reservation in docker-compose.yml looks like:

deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
device_ids: ['0']

Prevention:

  • Match GPU_SCALE_WORKERS to your GPU's VRAM capacity
  • Use large-v3-turbo for GPUs with less than 10GB VRAM
  • Monitor GPU memory with nvidia-smi after deployments
  • Avoid running other GPU workloads on the same device

Runbook: Database Connection Issues​

Symptoms: Backend returns 500 errors. Logs show connection refused, too many connections, or connection pool exhausted. API requests hang or timeout.

Database user

The commands in this runbook use the default POSTGRES_USER=postgres and database opentranscribe. Substitute your own POSTGRES_USER if you changed it.

Diagnosis:

  1. Check if PostgreSQL is running:
    docker compose ps postgres
  2. Test database connectivity:
    docker compose exec postgres pg_isready -U postgres
  3. Check current connection count:
    docker compose exec postgres psql -U postgres -c \
    "SELECT count(*) FROM pg_stat_activity;"
  4. Check backend logs for connection errors:
    ./opentr.sh logs backend | grep -i "connection\|pool\|database"

Resolution:

If PostgreSQL is down:

docker compose restart postgres
# Wait for it to become healthy
docker compose exec postgres pg_isready -U postgres
# Then restart the backend to re-establish connections
docker compose restart backend

If connection pool is exhausted:

  1. Increase the pool size in .env:
    DB_POOL_SIZE=20 # API default is 20 (workers use a smaller pool, WORKER_DB_POOL_SIZE=2)
    DB_MAX_OVERFLOW=40 # API default is 40 (workers: WORKER_DB_MAX_OVERFLOW=3)
  2. Restart the backend:
    docker compose restart backend

If connections are leaking (count keeps growing):

  1. Kill idle connections:
    docker compose exec postgres psql -U postgres -c \
    "SELECT pg_terminate_backend(pid)
    FROM pg_stat_activity
    WHERE state = 'idle'
    AND query_start < now() - interval '10 minutes';"
  2. Restart the backend to get fresh connections:
    docker compose restart backend

Prevention:

  • Monitor active connection count
  • Set appropriate DB_POOL_SIZE / DB_MAX_OVERFLOW for your workload
  • Ensure the backend properly closes sessions (SQLAlchemy context managers)

Runbook: OpenSearch Index Corruption​

Symptoms: Search returns no results or incomplete results. Admin panel shows index health as "red" or "yellow". Logs show index_not_found_exception or shard allocation errors.

Diagnosis:

  1. Check cluster health:
    curl -s http://localhost:5180/_cluster/health | python3 -m json.tool
  2. Check index status:
    curl -s http://localhost:5180/_cat/indices?v
  3. Check for unassigned shards:
    curl -s http://localhost:5180/_cat/shards?v | grep UNASSIGNED
  4. Check the speaker alias:
    curl -s http://localhost:5180/_cat/aliases?v

Resolution:

For missing or corrupt transcription index:

  1. Trigger a full reindex from the settings UI:

    • Open Settings > Data & Storage > Search & Indexing
    • Check Index Health, and use Repair Indices for a red index
    • Click Re-index All Transcripts
    • This rebuilds the search index from the database (source of truth)
  2. Or via API:

    curl -X POST http://localhost:5174/api/search/reindex \
    -H "Authorization: Bearer <admin-access-token>"

For unassigned shards:

# Retry shard allocation
curl -X POST http://localhost:5180/_cluster/reroute?retry_failed=true

# If single-node, ensure replica count is 0
curl -X PUT http://localhost:5180/_settings -H 'Content-Type: application/json' -d '
{
"index.number_of_replicas": 0
}'

For corrupt speaker index:

The speaker index uses an alias-based architecture (speakers alias pointing to speakers_v3 or speakers_v4).

The speaker indices are the only copy of your voiceprints

Unlike the transcript indices, the speakers_v* indices cannot be rebuilt from PostgreSQL, which stores no embedding vectors. Deleting one destroys the speaker embeddings unless a *.voiceprints.ndjson backup artifact exists (see Backup & Restore). Take ./opentranscribe.sh backup first, then try Settings > Data & Storage > Search & Indexing > Repair Indices, which tries close/reopen and force merge before any rebuild. Only delete an index you have an artifact for.

If a rebuild is unavoidable:

# Check which index the alias points to
curl -s http://localhost:5180/_cat/aliases/speakers?v

# Delete the corrupt index
curl -X DELETE http://localhost:5180/speakers_v3

# Restart the backend - it will recreate indices on startup
./opentr.sh restart-backend

Prevention:

  • Run OpenSearch on reliable storage (avoid network-mounted volumes for production)
  • Monitor cluster health regularly
  • Keep regular database backups (the database is the source of truth, not OpenSearch)

Runbook: Full Disk Space​

Symptoms: Uploads fail. Container logs show No space left on device. Docker commands fail. Services crash and cannot restart.

Diagnosis:

  1. Check overall disk usage:
    df -h /
  2. Check Docker disk usage:
    docker system df
  3. Check large directories:
    du -sh /var/lib/docker/*
    du -sh ./models/*
    du -sh ./backups/*

Resolution:

Step 1: Clean Docker resources

# Remove unused containers, networks, and dangling images
docker system prune -f

# Do NOT run `docker volume prune` here: it deletes every volume no running
# container uses, which includes the data volumes of a stopped OpenTranscribe
# stack (and of any other project on a shared Docker host).

# Remove old images
docker image prune -a -f --filter "until=168h"

Step 2: Clean temporary files

# Clean backend temp files
docker compose exec backend rm -rf /tmp/yt-dlp-*

# Check MinIO for orphaned files
# (files in storage but not referenced in the database)

Step 3: Archive old backups

# List backups sorted by size
ls -lhS backups/

# Remove backups older than 30 days (a dump and its .voiceprints.ndjson share the prefix;
# delete them together, never one without the other)
find backups/ -name "opentranscribe_backup_*" -mtime +30 -delete

Step 4: Move model cache to larger disk

# In .env, change MODEL_CACHE_DIR to a path on a larger disk
MODEL_CACHE_DIR=/mnt/large-disk/opentranscribe-models

# Fix permissions and restart
./scripts/fix-model-permissions.sh
./opentr.sh stop && ./opentr.sh start dev

Prevention:

  • Monitor disk usage with alerts at 80% and 90% thresholds
  • Set up automated backup rotation
  • Use a separate volume for Docker storage and model cache
  • Periodically run docker system prune

Runbook: Service Won't Start​

Symptoms: Running ./opentr.sh start dev fails. One or more containers exit immediately or enter a restart loop. The frontend or backend is unreachable.

Diagnosis:

  1. Check which services are failing:
    ./opentr.sh status
    docker compose ps -a
  2. Check logs for the failing service:
    ./opentr.sh logs backend
    ./opentr.sh logs frontend
    ./opentr.sh logs postgres
  3. Check for port conflicts:
    # Check if ports are already in use
    ss -tlnp | grep -E '5173|5174|5175|5179|5180'
  4. Verify the .env file exists and is valid:
    # Check for syntax errors (no spaces around =, no missing quotes)
    cat .env | grep -n "= \|^ \|^\t"
  5. Check Docker daemon is running:
    docker info

Resolution:

If a port is in use:

# Find what is using the port
lsof -i:5174

# If it's another OpenTranscribe stack (including a --fresh deployment), stop
# it properly instead of killing the PID directly:
./opentr.sh stop
# (or, for a --fresh deployment: ./opentr.sh stop --fresh <name>)
# Never `kill`/`kill -9` a container process directly -- a SIGKILL mid-task
# can wedge a CUDA context and require a machine restart to recover the GPU.

# Or change the port in .env
BACKEND_PORT=5184

If the backend fails on startup (migration error, missing env var):

# Check the specific error in logs
./opentr.sh logs backend | tail -50

# Common fix: ensure .env has all required variables
diff .env .env.example

If containers keep restarting:

# Check exit codes
docker compose ps -a

# Check for resource constraints
docker stats --no-stream

If Docker daemon issues:

sudo systemctl restart docker
./opentr.sh stop && ./opentr.sh start dev

Prevention:

  • Keep .env in sync with .env.example when upgrading
  • Test configuration changes in development before production
  • Use ./opentr.sh status after starting to verify all services are healthy

Runbook: Rolling back past self-service erasure​

When: you are rolling the application back to a release before self-service account erasure, or downgrading migration v455.

Why it matters: older code ignores the new ledger columns and treats a waiting self-service request as an ordinary deferred erasure, so it would erase the account early.

  1. Turn the feature off: Settings > Privacy & Redaction > policy tab, or PUT /api/admin/self-erasure with {"enabled": false}. New requests are refused (404).
  2. List the open requests (ledger entries with status = deferred and deferred_reason = grace_period) and either let them run or ask the people to cancel from Settings > Profile & Security.
  3. Only when none are open, downgrade. The downgrade refuses while any request is open, and deletes cancelled requests (their status does not exist in the old schema).
  4. Then roll the code back.

Worst-case execution lag after a waiting period ends is one reconciliation pass (daily by default).

Runbook: Failed Database Migration​

Symptoms: Backend fails to start with Alembic errors. Logs show alembic.util.exc.CommandError, Can't locate revision, or Target database is not up to date.

Diagnosis:

  1. Check backend logs for the specific migration error:
    ./opentr.sh logs backend | grep -i "alembic\|migration\|revision"
  2. Check the current database revision:
    ./opentr.sh shell backend
    alembic current
  3. Check migration history:
    alembic history --verbose

Resolution:

If the database has no Alembic stamp (fresh or legacy database):

The backend's migrations.py auto-detection handles this on startup. If it fails:

./opentr.sh shell backend
# Stamp the database at the appropriate version
alembic stamp <current_version>
# Then upgrade
alembic upgrade head

If a migration partially applied (some columns exist, some don't):

  1. Check what was applied:
    ./opentr.sh shell postgres
    psql -U postgres -d opentranscribe -c "\d+ <table_name>"
  2. Manually complete the migration or stamp past it:
    ./opentr.sh shell backend
    # If the schema is correct but stamp is wrong
    alembic stamp <target_revision>

If you need to rollback a migration:

./opentr.sh shell backend
# Downgrade one revision
alembic downgrade -1

# Or downgrade to a specific revision
alembic downgrade <revision_id>

Nuclear option (development only -- destroys all data):

./opentr.sh reset dev

Prevention:

  • Always use idempotent SQL in migrations (IF NOT EXISTS, IF EXISTS)
  • Test migrations with ./opentr.sh reset dev before deploying
  • Back up the database before applying new migrations in production:
    ./opentranscribe.sh backup

Runbook: High Memory Usage​

Symptoms: System becomes sluggish. Docker containers are killed by the OOM killer. docker stats shows high memory consumption. Swap usage is high.

Diagnosis:

  1. Check per-container memory usage:
    docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"
  2. Identify the largest consumer:
    docker stats --no-stream --format "{{.Name}}\t{{.MemUsage}}" | sort -k2 -h
  3. Check host system memory:
    free -h

Resolution:

If OpenSearch is consuming too much memory:

OpenSearch defaults can be aggressive. Reduce the JVM heap:

# In docker-compose.yml or .env:
OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g # Default is 4g

1 GB is the measured floor for the default embedding model — below it a model reports DEPLOY: COMPLETED and then fails to produce an embedding. See Performance Tuning. Restart OpenSearch after changing.

If the Celery worker is consuming too much:

This is usually due to model loading. Each model stays in memory:

# Restart the worker to free memory
docker compose restart celery-worker

If PostgreSQL is consuming too much:

Reduce shared buffers:

# In .env, then restart postgres:
PG_SHARED_BUFFERS=256MB # This is already the default; lower it only on very small hosts

Set container memory limits:

Add limits in docker-compose.yml:

services:
backend:
deploy:
resources:
limits:
memory: 2G
opensearch:
deploy:
resources:
limits:
memory: 2G

Prevention:

  • Set memory limits on all containers
  • Monitor memory usage trends over time
  • Size your server appropriately (see the hardware table in Production Deployment)

Runbook: Credential Rotation​

Symptoms: Planned maintenance -- you need to rotate credentials for security compliance or after a suspected compromise.

Diagnosis: Not applicable (proactive maintenance).

Resolution:

Rotate Database Password​

  1. Back up the database first:
    ./opentranscribe.sh backup
  2. Update the password in PostgreSQL:
    docker compose exec postgres psql -U postgres -c \
    "ALTER USER postgres WITH PASSWORD 'new_secure_password';"
  3. Update .env (DATABASE_URL is empty by default and assembled from POSTGRES_*; only edit it if you set it explicitly):
    POSTGRES_PASSWORD=new_secure_password
  4. Restart services that connect to the database:
    docker compose restart backend celery-worker

Rotate MinIO Keys​

  1. Update .env (MinIO reads the root credentials at start-up):
    MINIO_ROOT_USER=NEW_ACCESS_KEY
    MINIO_ROOT_PASSWORD=NEW_SECRET_KEY
  2. Restart the stack so MinIO and every service that talks to it pick up the new values:
    ./opentr.sh stop && ./opentr.sh start dev # or: start prod
    This does not touch MINIO_KMS_SECRET_KEY, the at-rest encryption key; changing that one would make existing encrypted objects unreadable.

Rotate JWT Secret​

  1. Update .env:
    JWT_SECRET_KEY=$(openssl rand -hex 32)
  2. Restart the backend:
    docker compose restart backend
  3. Note: All existing user sessions will be invalidated. Users will need to log in again.

Rotate LLM API Keys​

  1. Generate a new key from your LLM provider's dashboard
  2. Update it in the app: Settings > LLM Provider Configuration (provider keys are stored encrypted in the database)
  3. For a self-hosted vLLM server configured through .env, update VLLM_API_KEY there and restart the backend and celery-nlp-worker:
    docker compose restart backend celery-nlp-worker

Rotate Redis Password​

  1. Update .env:
    REDIS_PASSWORD=new_redis_password
    (The broker and result-backend URLs are built from REDIS_PASSWORD; only edit REDIS_URL or CELERY_BROKER_URL if you set them explicitly.)
  2. Restart all services that use Redis:
    docker compose restart redis backend celery-worker

Do not rotate ENCRYPTION_KEY​

ENCRYPTION_KEY decrypts every secret stored in the database (provider API keys, MFA secrets, backup and watch-source credentials). Changing it makes them permanently undecryptable; there is no re-encryption step. Keep it, and keep a copy with your backups (see Backup Completeness Audit).

Prevention:

  • Rotate credentials on a regular schedule (e.g., every 90 days)
  • Use strong, randomly generated passwords (openssl rand -hex 32)
  • Document credential rotation dates
  • Never commit credentials to version control

Runbook: Model Re-Download​

Symptoms: AI models are corrupted, producing garbled output, or you need to switch to a different model version. Or you want to force a fresh download after a failed partial download.

Diagnosis:

  1. Check current model cache:
    ls -la ${MODEL_CACHE_DIR:-./models}/huggingface/hub/
    ls -la ${MODEL_CACHE_DIR:-./models}/torch/pyannote/
  2. Check for incomplete downloads (very small files, lock files):
    find ${MODEL_CACHE_DIR:-./models} -name "*.lock" -o -name "*.incomplete"

Resolution:

Re-download a specific model type:

# Clear Whisper models
rm -rf ${MODEL_CACHE_DIR:-./models}/huggingface/hub/models--Systran--faster-whisper-*

# Clear PyAnnote speaker models (the in-process fallback engine)
rm -rf ${MODEL_CACHE_DIR:-./models}/torch/pyannote/

# Clear the diar-native speaker engine's exported models (DIAR_NATIVE_MODELS_DIR if set);
# the backend exports them again on its next start
rm -rf ${MODEL_CACHE_DIR:-./models}/diar-native/

# Clear sentence transformer models
rm -rf ${MODEL_CACHE_DIR:-./models}/sentence-transformers/

# Clear OpenSearch neural models
rm -rf ${MODEL_CACHE_DIR:-./models}/opensearch-ml/

Re-download all models:

# Stop services
./opentr.sh stop

# Clear the entire model cache
rm -rf ${MODEL_CACHE_DIR:-./models}/*

# Fix permissions
./scripts/fix-model-permissions.sh

# Restart -- models will download on first use
./opentr.sh start dev

Pre-download models for offline deployment:

bash scripts/download-models.sh models

Prevention:

  • Verify model downloads completed successfully after first startup
  • Use stable network connections for initial model downloads
  • For air-gapped environments, pre-download models on an internet-connected machine and transfer via rsync

Runbook: WebSocket Connection Issues​

Symptoms: The UI does not update in real-time after uploading files. Progress bars do not animate. Transcription completes but the UI still shows "Processing". Browser console shows WebSocket errors.

Diagnosis:

  1. Check browser developer console (F12) for WebSocket errors:
    • Look for WebSocket connection to 'ws://...' failed
    • Look for ERR_CONNECTION_REFUSED or 403 Forbidden
  2. Verify the backend WebSocket endpoint is reachable:
    # Test with wscat (install: npm install -g wscat)
    wscat -c ws://localhost:5174/api/ws
  3. Check backend logs for WebSocket errors:
    ./opentr.sh logs backend | grep -i "websocket\|ws\|upgrade"
  4. If using NGINX (production), check NGINX logs:
    ./opentr.sh logs nginx | grep -i "websocket\|upgrade"

Resolution:

If running in development (no NGINX):

  1. Verify the backend is running and healthy:
    curl http://localhost:5174/health
  2. Restart the backend:
    docker compose restart backend
  3. Hard-refresh the browser (Ctrl+Shift+R)

If running in production (with NGINX):

  1. Verify NGINX is configured for WebSocket proxying. The configuration should include:
    location /api/ws {
    proxy_pass http://backend:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_read_timeout 86400;
    }
  2. Check that proxy_read_timeout is set high enough (WebSocket connections are long-lived)
  3. Restart NGINX:
    docker compose restart nginx

If WebSocket connects but no updates arrive:

  1. Check that the Celery worker is sending notifications:
    ./opentr.sh logs celery-worker | grep -i "notification\|websocket\|send"
  2. Check Redis pub/sub (used for WebSocket message passing):
    docker compose exec redis redis-cli ping

Prevention:

  • In production, always use the provided NGINX configuration which includes WebSocket support
  • Set proxy_read_timeout to at least 86400 (24 hours) for WebSocket connections
  • Monitor WebSocket connection counts in production
  • Test WebSocket functionality after any NGINX configuration changes