THE FIELD GUIDE

Troubleshooting

Diagnose startup, network, cluster and alert delivery problems.

Project source · da59007Updated 2026-10-17

Quick Start Issues

Server Won't Start

Symptom: anubis serve exits immediately

Check:

BASH

# Check if port is already in use
# Windows
netstat -ano | findstr :8080

# Linux/macOS
lsof -i :8080

Solution:

BASH

# Use a different port
export ANUBIS_PORT=8444
./anubis serve

# Or kill the process using the port
kill <PID>

Config File Not Found

Symptom: failed to load config: open anubis.yaml: no such file or directory

Solution:

BASH

# Generate default config
./anubis init

# Or specify config path
export ANUBIS_CONFIG=/path/to/config.yaml
./anubis serve

Permission Denied

Symptom: permission denied when writing to data directory

Solution:

BASH

# Create data directory with correct permissions
mkdir -p /var/lib/anubis
chown $(whoami):$(whoami) /var/lib/anubis

# Or use a different data directory
export ANUBIS_DATA_DIR=$HOME/.anubis
./anubis serve

Configuration Problems

Invalid YAML Syntax

Symptom: yaml: line X: did not find expected key

Solution:

BASH

# Validate YAML syntax
python3 -c "import yaml; yaml.safe_load(open('anubis.yaml'))"

# Check for common issues:
# - Tabs vs spaces (use spaces)
# - Missing colons
# - Unclosed quotes

Missing Required Fields

Symptom: validation failed: field 'target' is required

Solution:

YAML

# Ensure all required fields are present
souls:
  - name: "My Monitor"
    type: "http"        # Required
    target: "https://example.com"  # Required
    weight: "60s"       # Required
    timeout: "10s"      # Required

Environment Variables Not Applied

Symptom: Config shows empty strings for expected values

Solution:

BASH

# Check environment variables are set
echo $ANUBIS_ENCRYPTION_KEY

# Restart after setting
export ANUBIS_ENCRYPTION_KEY="your-secret-key-here"
./anubis serve

Network & Connectivity

Health Checks Failing

Symptom: All souls showing as dead or degraded

Diagnosis:

BASH

# Check network connectivity from the server
curl -v https://your-target.com

# Check DNS resolution
nslookup your-target.com

# Check firewall rules
# Windows
netsh advfirewall show allprofiles

# Linux
iptables -L -n

Solutions:

DNS Issues: Use IP address instead of hostname
Firewall: Allow outbound connections on required ports

Proxy: Configure proxy in soul config:

YAML

http:
  proxy: "http://proxy.example.com:8080"

TLS Certificate Errors

Symptom: x509: certificate signed by unknown authority

Solutions:

YAML

# Option 1: Skip verification (not recommended for production)
souls:
  - name: internal-api
    type: http
    target: https://internal.example.com/health
    http:
      insecure_skip_verify: true

# Option 2: Install your internal CA into the OS trust store
# so default certificate verification can remain enabled.

Timeout Issues

Symptom: context deadline exceeded

Solutions:

YAML

# Increase timeout for slow endpoints
souls:
  - name: "Slow API"
    type: "http"
    target: "https://slow-api.com"
    timeout: "30s"  # Default is 10s

Storage Issues

Database Locked

Symptom: database is locked or file in use

Diagnosis:

BASH

# Check for running instances
ps aux | grep anubis

# Check file locks (Linux)
lsof /path/to/data/

Solution:

BASH

# Stop all anubis processes
pkill anubis

# Remove lock files
rm -f /path/to/data/*.lock

# Restart
./anubis serve

Disk Space Full

Symptom: no space left on device

Diagnosis:

BASH

# Check disk usage
df -h

# Check AnubisWatch data size
du -sh /path/to/data/*

Solution:

YAML

# Configure retention policy
storage:
  timeseries:
    retention:
      raw: "48h"
      minute: "720h"
      five: "2160h"
      hour: "8760h"
      day: "unlimited"

Data Corruption

Symptom: failed to decode JSON or unexpected errors

Solution:

BASH

# Backup current data
cp -r /path/to/data /path/to/data.backup

# List souls to identify corrupted files
./anubis judge

# Remove corrupted soul files
rm /path/to/data/default/souls/<corrupted-id>.json

# Restart
./anubis serve

Cluster Problems

Node Can't Join Cluster

Symptom: failed to join cluster: connection refused

Diagnosis:

BASH

# Check cluster secret matches
echo $ANUBIS_CLUSTER_SECRET

# Verify network connectivity between nodes
telnet <leader-ip> 7946

# Check Raft bind address
cat anubis.yaml | grep -A5 raft:

Solution:

YAML

necropolis:
  cluster_secret: "same-secret-on-all-nodes"
  raft:
    bind_addr: "0.0.0.0:7946"
    advertise_addr: "<public-ip>:7946"
    bootstrap: false  # Only true on initial leader

Leader Election Loops

Symptom: Constant leader changes, logs show repeated elections

Diagnosis:

BASH

# Check network stability between nodes
ping -c 10 <peer-ip>

# Check Raft logs
grep "election" /path/to/data/logs/*.log

Solution:

YAML

# Increase election timeout for unstable networks
necropolis:
  raft:
    election_timeout: "3s"     # Default: 1s
    heartbeat_timeout: "1s"    # Default: 500ms

Split Brain

Symptom: Two nodes both think they're leader

Solution:

BASH

# Identify healthy nodes
./anubis necropolis

# On minority partition nodes, stop and rejoin
./anubis banish <old-leader-id>

# Restart cluster with majority quorum
./anubis serve

Alert Delivery Failures

Slack Not Receiving Alerts

Symptom: Alerts logged but not appearing in Slack

Diagnosis:

BASH

# Test webhook URL directly
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test"}' \
  https://hooks.slack.com/services/YOUR/WEBHOOK/URL

Solution:

Verify webhook URL is correct
Check Slack app permissions
Ensure channel exists and bot is invited

Email Not Sending

Symptom: failed to send email: connection timeout

Solution:

YAML

channels:
  - name: "Email Alerts"
    type: "email"
    email:
      smtp_host: "smtp.example.com"
      smtp_port: 587
      starttls: true  # Required for port 587
      username: "alerts@example.com"
      password: "${SMTP_PASSWORD}"

PagerDuty Integration Issues

Symptom: Incidents not appearing in PagerDuty

Diagnosis:

BASH

# Test integration key
curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{"routing_key":"YOUR_KEY","event_action":"trigger","payload":{...}}'

Performance Issues

High Memory Usage

Symptom: Process using >1GB RAM

Diagnosis:

BASH

# Check memory profile
go tool pprof http://localhost:8444/debug/pprof/heap

# Check number of souls
./anubis judge | wc -l

Solution:

YAML

# Reduce judgment history
storage:
  max_judgments_per_soul: 1000

# Enable downsampling
storage:
  downsampling:
    enabled: true
    raw_retention: "24h"
    hourly_retention: "168h"
    daily_retention: "2160h"

Slow API Responses

Symptom: API calls taking >1 second

Diagnosis:

BASH

# Check API latency
time curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/api/v1/souls

Solution:

YAML

# Enable pagination
# Use ?limit=50 instead of fetching all

# Add indexes for frequently queried fields
storage:
  indexes:
    - field: "status"
    - field: "workspace_id"

Check Latency Spikes

Symptom: Judgments taking longer than expected

Solution:

YAML

# Limit concurrent checks
probe:
  max_concurrent_checks: 50  # Default: 100

# Increase check interval for non-critical souls
souls:
  - name: "Non-critical"
    weight: "300s"  # Check every 5 minutes

Common Error Messages

`context deadline exceeded`

Cause: Check or operation took longer than timeout

Fix:

YAML

# Increase timeout
timeout: "30s"

`connection refused`

Cause: Target service not reachable

Fix: Check network, firewall, target service status

`certificate verify failed`

Cause: TLS certificate issue

Fix: See TLS Certificate Errors

`rate limit exceeded`

Cause: Too many API requests

Fix: Wait and retry, or increase rate limit:

YAML

api:
  rate_limit: 200  # requests per minute

`no such file or directory`

Cause: Data directory or config file missing

Fix: Create directory or specify correct path

Debug Mode

Enable debug logging for detailed troubleshooting:

BASH

export ANUBIS_LOG_LEVEL=debug
./anubis serve

Debug logs show:

Every health check execution
Alert routing decisions
Raft consensus messages
Storage operations

Getting Help

If issues persist:

Check logs: /path/to/data/logs/
GitHub Issues: https://github.com/AnubisWatch/anubiswatch/issues
Documentation: docs/ directory
Health endpoint: curl http://localhost:8080/health

Last updated: 2026-04-05

Adapted from the AnubisWatch project. Examples use placeholder services and credentials.View project source Suggest a correction

Search the hall

Troubleshooting

Quick Start Issues

Server Won't Start

Config File Not Found

Permission Denied

Configuration Problems

Invalid YAML Syntax

Missing Required Fields

Environment Variables Not Applied

Network & Connectivity

Health Checks Failing

TLS Certificate Errors

Timeout Issues

Storage Issues

Database Locked

Disk Space Full

Data Corruption

Cluster Problems

Node Can't Join Cluster

Leader Election Loops

Split Brain

Alert Delivery Failures

Slack Not Receiving Alerts

Email Not Sending

PagerDuty Integration Issues

Performance Issues

High Memory Usage

Slow API Responses

Check Latency Spikes

Common Error Messages

`context deadline exceeded`

`connection refused`

`certificate verify failed`

`rate limit exceeded`

`no such file or directory`

Debug Mode

Getting Help

oxognet.