System Monitoring and Health Checks¶

Target Audience: System administrators, DevOps engineers

This guide describes the health check endpoint and monitoring capabilities in Bifolk.

Health Check Endpoint¶

Bifolk provides a dedicated health check endpoint for container orchestration and monitoring tools.

Endpoint Details¶

Endpoint: GET /health/

Authentication: None (public endpoint)

Purpose: Container health monitoring, load balancer health checks, uptime monitoring

Response Format¶

Healthy System¶

When all checks pass, the endpoint returns:

{
  "status": "healthy",
  "database": "ok"
}

HTTP Status: 200 OK

Unhealthy System¶

When one or more checks fail, the endpoint returns:

{
  "status": "unhealthy",
  "database": "error",
  "error": "Database connectivity failed"
}

HTTP Status: 503 Service Unavailable

Caching¶

The response is cached for 10 seconds to reduce database load from frequent health checks. This is safe because health status doesn't change rapidly.

Health Checks Performed¶

The endpoint performs the following checks:

Database Connectivity: Executes a simple query (SELECT 1) to verify that the database connection is working

Use Cases¶

Docker Health Checks: Automatic container health monitoring
Kubernetes Probes: Liveness and readiness probes
Load Balancer Monitoring: Backend health verification
Uptime Monitoring Services: Automated availability checks
CI/CD Pipelines: Deployment verification

Example Usage¶

Basic Health Check¶

# Check health status
curl http://localhost:8000/health/

# Output (healthy):
# {"status": "healthy", "database": "ok"}

Check with HTTP Status Code¶

# Check health status and display HTTP status code
curl -w "\nHTTP Status: %{http_code}\n" http://localhost:8000/health/

# Output (healthy):
# {"status": "healthy", "database": "ok"}
# HTTP Status: 200

Check in Shell Script¶

#!/bin/bash
# Simple health check script

response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/health/)

if [ "$response" = "200" ]; then
    echo "✓ Application is healthy"
    exit 0
else
    echo "✗ Application is unhealthy (HTTP $response)"
    exit 1
fi

Docker Health Check¶

The Bifolk Docker image includes automatic health checking that uses the /health/ endpoint.

Configuration¶

The health check runs with the following parameters:

Interval: 30 seconds between checks
Timeout: 3 seconds per check
Retries: 3 consecutive failures before marking unhealthy
Start Period: 40 seconds grace period on container startup

View Health Status¶

Check Container Health¶

# List containers with health status
docker ps

# Output shows health status in STATUS column:
# CONTAINER ID   IMAGE           STATUS
# abc123def456   bifolk-app      Up 5 minutes (healthy)

Detailed Health Information¶

# View detailed health check information
docker inspect bifolk-app | grep -A 10 Health

# Or using jq for formatted output:
docker inspect bifolk-app | jq '.[0].State.Health'

View Health Check Logs¶

# Show last 5 health check results
docker inspect bifolk-app | jq '.[0].State.Health.Log[-5:]'

Health Status Meanings¶

Healthy (Containers with Status: "healthy") - All health checks are passing - Application is responding correctly - Database is accessible - Ready to serve traffic

Unhealthy (Containers with Status: "unhealthy") - Health check returned 503 status - Database connection failed - Check exceeded timeout (3 seconds) - 3 consecutive checks failed

Starting (Containers with Status: "starting") - Container recently started (within 40-second grace period) - Health checks are running but failures don't count yet - Allows time for application initialization

Troubleshooting¶

Health Check Fails Immediately¶

Symptoms: - Container marked unhealthy right after startup - Health check never succeeds

Possible Causes & Solutions:

Application didn't start

# Check application logs
docker logs bifolk-app

# Look for errors during startup
docker logs bifolk-app | grep -i error

Port not accessible

# Test from within container
docker exec bifolk-app curl http://localhost:8000/health/

# If this fails, application isn't listening on port 8000

Database not ready

# Check database container status
docker ps | grep postgres

# Verify database is healthy
docker inspect bifolk-db | grep Health -A 5

Intermittent Health Check Failures¶

Symptoms: - Health checks pass, then fail, then pass again - Container alternates between healthy and unhealthy

Possible Causes & Solutions:

Database connection pool exhaustion

# Check for database connection errors in logs
docker logs bifolk-app | grep -i "database\|connection"

# Review database connection settings in environment

High load causing timeouts

# Check system resources
docker stats bifolk-app

# Consider increasing health check timeout if needed

Network latency
Consider increasing the timeout in docker-compose.yml
Check for network issues between containers

Health Endpoint Returns 503¶

Symptoms: - Endpoint is accessible but returns unhealthy status - HTTP 503 Service Unavailable response

Troubleshooting Steps:

Check database connectivity

# Verify database is running
docker ps | grep db

# Test database connection
docker exec bifolk-app python manage.py dbshell

Review application logs

# Health check errors are logged with details
docker logs bifolk-app | grep "Health check failed"

Verify database credentials

# Check environment configuration
docker exec bifolk-app env | grep DB_

Test database from container

# For PostgreSQL
docker exec bifolk-app psql -U bifolk -d bifolk -c "SELECT 1"

# For SQLite
docker exec bifolk-app sqlite3 /bifolk/db/bifolk.db "SELECT 1"

Health Check Never Completes¶

Symptoms: - Container stuck in "starting" state - Health checks don't return within timeout

Solutions:

Increase timeout in docker-compose.yml:

healthcheck:
  timeout: 5s  # Increased from 3s

Increase start period for slow startup:

healthcheck:
  start_period: 60s  # Increased from 40s

Best Practices¶

For System Administrators¶

Monitor Health Check Status
Set up alerts for containers becoming unhealthy
Monitor health check log patterns
Graceful Handling
Unhealthy containers should be investigated, not automatically restarted
Check logs before taking action
Resource Planning
Health checks add minimal load due to caching
One database query every 10 seconds maximum

For Load Balancers¶

Configure Backend Health Checks
Use GET /health/ as the health check URL
Set appropriate check intervals (30 seconds recommended)
Configure proper timeout (3-5 seconds)
Response Codes
200: Backend is healthy, route traffic
503: Backend is unhealthy, remove from pool
Other codes: Consider unhealthy

For Monitoring Systems¶

Uptime Monitoring
Check /health/ endpoint every 1-5 minutes
Alert on 503 responses
Alert on timeouts or connection failures
Synthetic Monitoring
Verify endpoint from multiple geographic locations
Monitor response times
Track availability percentage

Advanced Configuration¶

Custom Health Check Intervals¶

To adjust health check timing, edit the appropriate docker-compose.yml file:

healthcheck:
  interval: 60s          # Check every 60 seconds (default: 30s)
  timeout: 5s            # Timeout after 5 seconds (default: 3s)
  retries: 5             # 5 failures before unhealthy (default: 3)
  start_period: 60s      # 60 second grace period (default: 40s)

Disable Health Checks (Not Recommended)¶

To disable health checks entirely:

healthcheck:
  disable: true

Warning: Disabling health checks removes automatic health monitoring and may hide application issues.

Access Logs¶

Bifolk writes HTTP access logs in Common Log Format (CLF) to help administrators monitor request traffic.

Log Format¶

Each request produces one log line with the following fields:

<user-id> <method> <path> <status> <size> <duration_ms>ms

Example:

a3f9c12b4e01 GET /hives/ 200 12453 42ms
- GET /health/ 200 45 3ms

Field	Description
User ID	12-character hex token (pseudonymized SHA-256 hash of the user's email). Anonymous requests show `-`.
Method	HTTP method (GET, POST, etc.)
Path	Request path
Status	HTTP response status code
Size	Response body size in bytes
Duration	Request processing time in milliseconds

User identifiers in access logs are pseudonymized: the raw email address is replaced by a 12-character hex token derived from a SHA-256 hash of the email. The same user always produces the same token, so log entries can be correlated across requests without exposing PII.

Note

The pseudonymized token is not reversible without the original email address. Log files do not contain email addresses and do not require the same level of DSAR handling as PII data stores.

Log File Location¶

Access logs are written via the Django logging framework. By default they appear in the container's stdout/stderr output. To persist them, configure your container runtime to collect Docker logs.

# View access log entries in the container log stream
docker logs bifolk-app | grep -E "^[a-f0-9-]+ (GET|POST|PUT|PATCH|DELETE|HEAD)"

Error Tracking (GlitchTip / Sentry)¶

Bifolk supports reporting unhandled exceptions to a Sentry-compatible error tracker. A self-hosted GlitchTip instance is the recommended choice.

Setup¶

Create a new project in GlitchTip (or Sentry) and copy the DSN from the project settings.

Set SENTRY_DSN in your bifolk.env:

SENTRY_DSN=https://<key>@glitchtip.example.com/<project-id>

For production, use the Docker secret variant instead:

SENTRY_DSN_FILE=/run/secrets/bifolk_sentry_dsn

Restart the container. The startup log will confirm activation:
```
Sentry error tracking: ENABLED
```

Performance Tracing¶

By default, performance tracing is disabled (SENTRY_TRACES_SAMPLE_RATE=0.0). To capture a sample of requests, set a value between 0.0 and 1.0:

SENTRY_TRACES_SAMPLE_RATE=0.1   # trace 10% of requests

Use a low sample rate in production to avoid overhead.

Privacy¶

The SDK is configured with send_default_pii=False. User IP addresses, email addresses, and session cookies are not sent to the error tracker.

Last Updated: 2026-03-22