Symptoms

• 🚫 API returns 500 errors
• 📊 Health check shows database: down
• 📝 Logs show OperationalError: could not connect to server

Diagnostic Flowchart

flowchart TD
    A[Database Connection Error] --> B{Is PostgreSQL<br/>container running?}
    B -->|No| C[Start PostgreSQL]
    B -->|Yes| D{Can you ping<br/>postgres host?}
    D -->|No| E[Check Docker network]
    D -->|Yes| F{Are credentials<br/>correct?}
    F -->|No| G[Fix DATABASE_URL]
    F -->|Yes| H{Max connections<br/>exceeded?}
    H -->|Yes| I[Increase max_connections]
    H -->|No| J[Check PostgreSQL logs]

    C --> K([🔄 Restart & Retry])
    E --> K
    G --> K
    I --> K
    J --> K

Quick Fix Commands

Solutions

1. Verify PostgreSQL is running:

   Bash

   docker compose ps postgres
   docker compose logs postgres

2. Check connection parameters:

   Bash

   docker compose exec api python -c "from django.conf import settings; print(settings.DATABASE_URL)"

3. Test direct connection:

   Bash

   docker compose exec postgres psql -U adsb -d adsb -c "SELECT 1;"

4. Review connection timeout settings:

   Python

   # In settings.py, connection timeout is set to 10 seconds
   'OPTIONS': {
       'connect_timeout': 10,
   }

⚠️

Common Causes

• PostgreSQL container not fully initialized
• Incorrect DATABASE_URL environment variable
• Network isolation between containers
• PostgreSQL max connections exceeded

Symptoms

• 🔌 Socket.IO connections fail
• ⚙️ Celery workers cannot start
• 💾 Cache operations fail
• 📊 Health check shows cache: down

Quick Fix Commands

Solutions

1. Verify Redis is running:

   Bash

   docker compose ps redis
   docker compose logs redis

2. Test Redis connection:

   Bash

   docker compose exec redis redis-cli ping
   # Expected: PONG

3. Verify Redis memory:

   Bash

   docker compose exec redis redis-cli info memory

📝

Environment Variable

Bash

REDIS_URL=redis://redis:6379/0

Symptoms

• ✈️ No aircraft appearing in the interface
• 📊 Status shows adsb_online: false
• 📝 Logs show connection errors to ultrafeeder

Quick Fix Commands

Solutions

1. Verify ultrafeeder is accessible:

   Bash

   docker compose exec api curl -s http://ultrafeeder:80/data/aircraft.json | head -c 200

2. Check environment configuration:

   Bash

   ULTRAFEEDER_HOST=ultrafeeder
   ULTRAFEEDER_PORT=80

3. Review polling task logs:

   Bash

   docker compose logs -f celery-worker | grep -E "(poll_aircraft|ultrafeeder)"

Symptoms

• 🚫 API fails to start
• 📝 Logs show django.db.utils.ProgrammingError
• ❓ Missing tables or columns

Quick Fix Commands

Symptoms

• 🐢 Slow API responses
• 📈 High database CPU usage
• ⏱️ Query timeouts

Quick Fix Commands

SQL Queries

Check table sizes:

SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;

Monitor active queries:

SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC;

Symptoms

• ✏️ Write operations fail
• 📝 Logs show No space left on device

Quick Fix Commands

Cleanup Command

docker compose exec api python manage.py shell -c "
from skyspy.tasks.cleanup import run_all_cleanup_tasks
run_all_cleanup_tasks()
"

⚠️

Manual Truncation (Use with Caution)

SQL

-- Only if you're sure you want to delete data
DELETE FROM skyspy_aircraftsighting WHERE timestamp < NOW() - INTERVAL '7 days';

Symptoms

• ✈️ Aircraft disappear intermittently
• 🖥️ Browser console shows Socket.IO errors
• 👥 Users report real-time updates stopping

Diagnostic Flowchart

flowchart TD
    A[Socket.IO Dropping] --> B{Can you connect<br/>to Socket.IO endpoint?}
    B -->|No| C{Is API server<br/>running?}
    B -->|Yes| D{Connections<br/>stable locally?}

    C -->|No| E[Start API server]
    C -->|Yes| F[Check reverse proxy config]

    D -->|Yes| G[Check network/firewall]
    D -->|No| H{Redis channel<br/>layer working?}

    H -->|No| I[Check Redis connection]
    H -->|Yes| J[Check rate limits]

    E --> K([🔄 Retry])
    F --> K
    G --> K
    I --> K
    J --> K

Quick Fix Commands

Test Channel Layer

docker compose exec api python -c "
from channels.layers import get_channel_layer
import asyncio
layer = get_channel_layer()
print('Channel layer:', layer)
"

Browser Console Test

import { io } from 'socket.io-client';

const socket = io('http://localhost:8000/aircraft', {
  path: '/socket.io/',
  transports: ['websocket', 'polling']
});

socket.on('connect', () => console.log('✅ Connected'));
socket.on('connect_error', (e) => console.error('❌ Error:', e));
socket.on('disconnect', (reason) => console.log('🔌 Disconnected:', reason));

💡

Rate Limits

Python

WS_RATE_LIMITS = {
    'aircraft:update': 10,      # Max 10 Hz
    'aircraft:position': 5,     # Max 5 Hz
    'stats:update': 0.5,        # Max 0.5 Hz
}

📘

Auto-Reconnection

The frontend implements automatic reconnection with exponential backoff:

• Initial delay: 1 second
• Maximum delay: 30 seconds
• Heartbeat ping every 30 seconds with 10-second timeout

Symptoms

• 🔒 Socket.IO disconnects with authentication errors
• 📝 Logs show Socket.IO auth error

Error Code Reference

Code	Badge	Meaning	Solution
4001		Authentication required	Pass JWT token
4003		Token expired	Refresh token
4004		Invalid token	Re-authenticate

Solutions

1. Check AUTH_MODE setting:

   Bash

   # public: No auth required
   # private: Auth required for all
   # hybrid: Feature-based auth (default)
   AUTH_MODE=hybrid

2. Verify JWT token is being passed:

   JavaScript

   // Token should be passed in Socket.IO auth or query params
   const socket = io('http://host/aircraft', {
     path: '/socket.io/',
     auth: { token: '<jwt_token>' },
     query: { token: '<jwt_token>' }
   });

3. Check token expiration:

   Bash

   JWT_ACCESS_TOKEN_LIFETIME_MINUTES=60

Symptoms

• 💻 System becomes unresponsive
• ✈️ Aircraft updates lag
• 🌡️ Temperature warnings

Quick Fix Table

Solutions

1. Use RPi-optimized settings:

   Bash

   DJANGO_SETTINGS_MODULE=skyspy.settings_rpi

2. Reduce polling frequency:

   Bash

   POLLING_INTERVAL=3  # Default is 2 seconds

3. Limit Celery concurrency:

   YAML

   # In docker-compose.yml
   command: celery -A skyspy worker -l info --concurrency=2

4. Monitor resource usage:

   Bash

   docker stats

Symptoms

• 💀 OOMKilled containers
• 📊 Swap usage high
• 🔄 Services restarting

Quick Fix Table

Docker Compose Memory Limits

services:
  api:
    deploy:
      resources:
        limits:
          memory: 512M

💡

Bounded Caches

Python

# In services/cache.py - enabled by default
aircraft_info_cache = BoundedCache(maxsize=5000)
route_cache = BoundedCache(maxsize=1000)

Symptoms

• ⏱️ API requests taking > 1 second
• ⌛ Timeouts on complex queries

Quick Fix Table

Check Query Counts

docker compose exec api python manage.py shell -c "
from django.db import connection
# Make API request
print(len(connection.queries))
"

💡

Performance Tips

• Use select_related() and prefetch_related() in querysets
• Review N+1 query issues
• Enable query logging in development

Remove all containers and volumes:

docker compose down -v
docker compose up -d

⚠️

Warning

This deletes all data including aircraft history.

Backup:

docker compose exec postgres pg_dump -U adsb adsb > backup.sql

Restore:

cat backup.sql | docker compose exec -T postgres psql -U adsb adsb

docker compose exec api python manage.py changepassword admin

Or create a new superuser:

docker compose exec api python manage.py createsuperuser

Set AUTH_MODE=public in your environment:

AUTH_MODE=public

🚨

Security Warning

Never use public mode in production.

Check that:

1. ✅ Your JWT token is valid and not expired
2. ✅ The Authorization header format is correct: Authorization: Bearer <token>
3. ✅ AUTH_MODE is not set to private without providing credentials
4. ✅ API keys are enabled if using X-API-Key header

Use the RPi-optimized settings:

DJANGO_SETTINGS_MODULE=skyspy.settings_rpi
POLLING_INTERVAL=3

Also consider:

• ❌ Disabling transcription
• ⬇️ Reducing Celery worker concurrency
• ❌ Disabling auto photo downloads

Adjust data retention settings:

# In cleanup task settings
DATA_RETENTION_DAYS=7  # Default is 30

Or manually clean old data:

docker compose exec api python manage.py shell -c "
from skyspy.tasks.cleanup import cleanup_old_sightings
cleanup_old_sightings(days=7)
"

Common causes:

1. 🌐 CORS issues: Ensure CORS_ALLOWED_ORIGINS includes your frontend URL
2. 🔀 Reverse proxy: Configure Socket.IO proxying in nginx/traefik
3. 🔥 Firewall: Ensure Socket.IO port is accessible

For nginx, add:

location /socket.io/ {
    proxy_pass http://api:8000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
}

Using socket.io-client:

# Install socket.io-client
npm install -g socket.io-client

# Test in Node.js
node -e "const io = require('socket.io-client'); \
const socket = io('http://localhost:8000/aircraft', {path: '/socket.io/'}); \
socket.on('connect', () => console.log('Connected'));"

Using browser console:

import { io } from 'socket.io-client';

const socket = io('http://localhost:8000/aircraft', {
  path: '/socket.io/'
});

socket.on('connect', () => console.log('Connected'));
socket.on('aircraft:update', (data) => console.log('Update:', data));

View ACARS logs:

docker compose logs -f api | grep -i acars

SkysPy uses libacars for enhanced decoding. Check status:

curl -s http://localhost:8000/health | jq '.services.libacars'

Configure one of the weather APIs:

AVWX (free, unlimited basic requests):

AVWX_ENABLED=True
AVWX_API_KEY=your-key  # Optional but recommended

CheckWX (free tier: 3000/day):

CHECKWX_ENABLED=True
CHECKWX_API_KEY=your-key

Check photo cache settings:

PHOTO_CACHE_ENABLED=True
PHOTO_AUTO_DOWNLOAD=True

Check the cache status:

docker compose exec api python manage.py shell -c "
from skyspy.services.cache import photo_cache
print(photo_cache.get_stats())
"

💡

Photos are fetched from planespotters.net with rate limiting.

S3_ENABLED=True
S3_BUCKET=your-bucket
S3_REGION=us-east-1
S3_ACCESS_KEY=your-key
S3_SECRET_KEY=your-secret
S3_PREFIX=aircraft-photos

For S3-compatible storage (MinIO, Backblaze B2):

S3_ENDPOINT_URL=https://s3.example.com