Troubleshooting

14. Troubleshooting

Cannot Log In

• Default credentials: admin / admin.
• Account locked: If login protection is enabled (Max Login Attempts > 0 in System Settings), wait for the lockout duration (default 15 minutes) or ask another admin to unlock the account. With the default settings, lockout triggers after 5 consecutive failures.
• Browser cookies: Ensure cookies are enabled. Osprey uses HTTP-only secure cookies for JWT auth. Third-party cookie blocking or privacy extensions can interfere.
• HTTPS certificate: Accept the self-signed certificate in your browser. Some browsers (especially Safari) block cookies on untrusted HTTPS origins.
• Mixed content: If you access Osprey over HTTP instead of HTTPS and secure_cookies: true is set in the config, the browser will reject the cookies. Either use HTTPS or set secure_cookies: false (not recommended for production).
• Clock skew: JWT tokens have a 15-minute lifetime by default. If the server clock is significantly ahead of the client, tokens may appear expired immediately. Ensure NTP is running on the server.

Single sign-on problems:

• "Single sign-on failed / could not be verified": usually clock skew between Osprey and the IdP (id_token validation) — verify NTP on both — or a stale login attempt (the sign-in state expires after 10 minutes; just retry).
• "Your account is not permitted to sign in (no matching role)": none of the user's IdP groups matches the provider's role mapping and the default role is Deny access. Fix the mapping (or the IdP group membership), not the user.
• "An account with this email already exists": a local account uses this email as its username and email-linking is off (default). An admin can enable Link SSO sign-ins to existing accounts by email in System Settings > Authentication — after reading the warning — or rename/remove the local account.
• The provider button is missing on the login page: the provider is disabled, or External URL is not set (check the provider's Test results and System Settings > Authentication).
• Redirect lands on an IdP error page: the Redirect URI registered at the IdP does not match the one shown in the provider form (it changes when External URL changes).
• Password form is gone: the local-login policy is admins_only (click "Sign in with username") or disabled (administrators: browse to /?local=1). Recovery without any working admin: osprey auth enable-local-login on the Osprey host.

No Topology Data

• Check collectors: Verify at least one collector is running (green status in sidebar). Open the sidebar and look for the collector status badges.
• GRE tunnels: Ensure the remote router has a matching GRE tunnel configured and the IGP is enabled on the tunnel interface. For OSPF, check show ip ospf neighbor. For IS-IS, check show clns neighbor or show isis adjacency. Verify IP connectivity between the Osprey server and the router's GRE endpoint (ping).
• SNMP discovery: Verify SNMP is reachable from the Osprey server: snmpwalk -v2c -c community target-ip 1.3.6.1.2.1.1.1. Check firewall rules for UDP 161. For SNMPv3, verify that engine ID, username, auth, and privacy settings match exactly.
• Collector manager logs: sudo journalctl -u osprey-collector-manager -f -- look for "starting collector" or error messages.
• Engine logs: sudo journalctl -u osprey-engine -f -- look for snapshot processing messages ("processing snapshot for area...").
• NATS connectivity: Verify NATS is running: sudo systemctl status nats-server. Check that both the engine and collector-manager can connect (look for "connected to NATS" in their logs).
• Hierarchy mismatch: If you deleted and re-created hierarchy entities (networks, areas), old collectors may be orphaned. Check Admin > Monitoring for stale collector configs.

No Traffic Data

• SNMP targets: Check Admin > Monitoring > SNMP Targets for target status. Active targets show a green status.
• Consecutive failures: Targets auto-disable after 10 consecutive failures (configurable in System Settings). Re-enable them manually by clicking the enable toggle.
• Credentials: Verify SNMP credentials are correct. For v3, auth protocol, auth password, privacy protocol, and privacy password must all match the device configuration exactly.
• Firewall: Ensure UDP 161 is open from the Osprey server to the managed devices. Also verify that SNMP ACLs on the device permit the Osprey server IP.
• Poller logs: sudo journalctl -u osprey-snmp-poller -f -- look for poll success/failure messages and error details.
• Utilization not showing on canvas: Verify that View > Color > By Utilization is selected. Utilization data takes one poll interval (default 5 minutes) to appear after targets are added.

No BGP Data

• BMP target status: Check the sidebar under the BGP protocol instance -- each target shows a status indicator (green = connected, grey = pending, red = error). Alternatively, check via GET /api/v1/bgp/targets. If pending, the router hasn't connected yet.
• Router BMP config: Verify the router is configured to send BMP to the correct IP and port (default TCP 11019). Check show bmp server or equivalent on the router.
• Firewall: Ensure TCP 11019 is open inbound to the Osprey server from the router's management IP.
• Connection filtering: The BMP server only accepts connections from IPs matching registered BMP targets (bmp_target.router_ip). The optional bmp.allowed_cidrs in osprey.yaml adds an additional CIDR allowlist filter on top of this. If bmp.allow_nat_fallback is enabled, connections from unknown IPs are accepted and correlated by BMP sysName instead.
• RIB mode: If the BMP target's RIB mode is set to none, peers are tracked but no routes are processed. Change to loc_rib or adj_rib_in_post to see route data.
• BMP server logs: sudo journalctl -u osprey-bmp-server -f -- look for "BMP Peer Up", "End-of-RIB", or error messages.
• Engine logs: sudo journalctl -u osprey-engine -f -- look for "BGP End-of-RIB sync complete" or "persisted BGP best-path deltas".
• Peers show but no routes: The initial RIB dump can take 30-60 seconds for a full internet table. Wait for the "End-of-RIB" log message. If the BMP target uses adj_rib_in_post, routes only appear after the router sends Route Monitoring messages.

Stale Devices Won't Disappear

Unreachable devices remain visible for the configured retention period (default 7 days). To change this:

• Admin > System Settings > Topology > Stale Device Retention: Reduce the hours (minimum 1 hour). Note that setting this below 24 hours risks removing devices during brief maintenance windows.
• Manual deletion: Right-click a stale or down device on the canvas and select Delete device (admin only). This permanently removes the device from the database.

Stale Devices Reappearing After Deletion

If you delete a device or hierarchy entity but it reappears, a running collector is likely re-discovering and re-creating it. Stop or delete the associated collector first, then delete the entity. When deleting a hierarchy entity (network, domain, protocol instance, area) via the API or sidebar, Osprey automatically disables associated collectors to prevent this.

Engine / Collector Manager / SNMP Poller Shows "Down"

The System Health popover shows heartbeat-based liveness for backend services. If a service shows "Down" or "Not responding":

• Verify the service is running: sudo systemctl status osprey-engine (or osprey-collector-manager, osprey-snmp-poller).
• Check the service logs for errors: sudo journalctl -u osprey-engine -n 50.
• Verify NATS is running -- heartbeats are published via NATS, so a NATS outage will cause all three services to appear down.
• After restarting a stopped service, its status recovers to "Healthy" within ~15 seconds (the heartbeat interval).
• The engine is treated as critical -- if it is unhealthy, the overall system status degrades to "Degraded" in the status bar.

WebSocket Disconnections

The bottom-right status bar shows WebSocket connection state. If it shows a red indicator:

• Check that nginx is properly proxying WebSocket upgrades. The default config includes proxy_set_header Upgrade $http_upgrade and proxy_set_header Connection "upgrade" with an 86400s read timeout.
• Verify the API service is running: sudo systemctl status osprey-api.
• If behind an external load balancer or reverse proxy, ensure it supports WebSocket upgrades and has a sufficiently long idle timeout (Osprey WebSocket connections are long-lived).
• Check for firewalls or corporate proxies that may be terminating long-lived connections.
• The UI automatically reconnects when the WebSocket drops. If you see frequent reconnections, check network stability between the browser and server.

UI Crash Recovery

If a rendering error occurs in the topology canvas, activity tray, or panel stack, Osprey isolates the failure to the affected zone. A fallback panel appears with the error message and a Retry button. The rest of the UI continues functioning normally. Clicking Retry re-renders the failed zone. Switching areas or navigating away also resets the error state automatically.

DNS Names Not Resolving

• PTR records: Osprey resolves reverse DNS (PTR) records for router IDs and interface IPs. Ensure PTR records exist in your DNS infrastructure for the relevant IP addresses.
• Trigger refresh: Use Tools > Refresh DNS to force re-resolution of all cached IPs. The engine clears its DNS cache and re-resolves asynchronously -- results appear within seconds.
• Display mode: Check Admin > System Settings > Display > Device Name Format -- if set to router_id, DNS names are not used for labels. Set it to dns or hostname instead.
• DNS server configuration: The Osprey engine uses the system resolver (/etc/resolv.conf). Verify the DNS servers configured there can resolve PTR records for your network IP ranges.

Database Issues

# Check PostgreSQL is running
sudo systemctl status postgresql

# Check database exists
sudo -u postgres psql -l | grep osprey

# Check if the osprey user can connect
sudo -u postgres psql -U osprey -d osprey -c "SELECT 1;"

# Run migrations manually (use the password from /etc/osprey/osprey.env)
/usr/bin/osprey migrate --db-url "postgres://osprey:PASSWORD@localhost:5432/osprey?sslmode=disable"

# Check migration state
sudo -u postgres psql osprey -c "SELECT version, dirty FROM schema_migrations;"

Dirty migration state: If a migration failed partway through, schema_migrations will show dirty=true. To fix:

1. Check which version is dirty: SELECT version, dirty FROM schema_migrations;
2. Manually inspect and fix the database state for that migration version.
3. Set dirty to false: UPDATE schema_migrations SET dirty=false;
4. Re-run migrations: /usr/bin/osprey migrate --db-url "..."

Disk space: PostgreSQL requires free disk space for WAL (write-ahead log) and temporary files. If the disk is full, PostgreSQL may stop accepting writes. Free space and restart: sudo systemctl restart postgresql.

Service Won't Start

# Check service logs for the specific error
sudo journalctl -u osprey-engine --no-pager -n 50
sudo journalctl -u osprey-api --no-pager -n 50
sudo journalctl -u osprey-collector-manager --no-pager -n 50
sudo journalctl -u osprey-snmp-poller --no-pager -n 50
sudo journalctl -u osprey-bmp-server --no-pager -n 50

Common error messages and solutions:

Error	Cause	Solution
connection refused (port 5432)	PostgreSQL not running	sudo systemctl start postgresql
connection refused (port 4222)	NATS not running	sudo systemctl start nats-server
permission denied	File permissions wrong	Check ownership: ls -la /etc/osprey/. The .env file should be root:osprey 0640.
address already in use	Another process on the port	Find it: sudo ss -tlnp | grep :8080 and stop the conflicting process.
migration dirty	A migration failed mid-way	See Database Issues above for dirty migration fix.
YAML parse error	Syntax error in config	Validate: python3 -c "import yaml; yaml.safe_load(open('/etc/osprey/osprey.yaml'))"
invalid JWT secret	Secret too short	Generate a new one: openssl rand -base64 32 and update /etc/osprey/osprey.env.
encryption_key not set warning	No encryption key configured	Generate: openssl rand -base64 32 and set OSPREY_ENCRYPTION_KEY in /etc/osprey/osprey.env. Restart services — existing plaintext credentials are encrypted automatically on startup.

NATS Issues

# Check NATS status
sudo systemctl status nats-server

# View NATS logs
sudo journalctl -u nats-server -f

# Verify NATS is listening
ss -tlnp | grep 4222

# Test NATS connectivity (if nats CLI is installed)
nats server ping

If NATS fails to start, check that the configuration file exists at /etc/osprey/nats.conf and that the JetStream data directory /var/lib/nats/jetstream exists and is writable.

nginx Issues

# Test nginx configuration
sudo nginx -t

# Check nginx status
sudo systemctl status nginx

# View nginx error log
sudo tail -50 /var/log/nginx/error.log

# Verify the Osprey site is enabled
ls -la /etc/nginx/sites-enabled/osprey

Common issues:

• 502 Bad Gateway: The API service is not running or not listening on port 8080. Check: sudo systemctl status osprey-api.
• SSL certificate errors: Regenerate the self-signed certificate: sudo openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /etc/osprey/certs/osprey.key -out /etc/osprey/certs/osprey.crt -subj "/CN=osprey" then sudo systemctl reload nginx.
• Port 80/443 conflict: Another web server (Apache, etc.) may be using the ports. Check: sudo ss -tlnp | grep -E ':80|:443'.

Performance

• Large topologies (1000+ devices): Use the geometric or grid layout instead of fcose. Force-directed layouts are CPU-intensive and can cause UI lag on large graphs. Disable View > Area Boundaries for better rendering performance.
• SNMP polling: Reduce poll frequency for non-critical devices in System Settings. The default 5-minute interval works well for most deployments. Consider increasing the PDU timeout for high-latency WAN devices.
• Event and snapshot retention: Reduce retention in System Settings if disk space is constrained. Events and snapshots both default to 90 days.
• Browser memory: Close or minimize the Link Detail Panel when not actively monitoring traffic (it triggers 5-second boosted polling). Hide unnecessary columns in report panels. Minimized panels use display: none and preserve state without active polling. For very large topologies, use filters (View > Filters) to reduce the number of rendered nodes.
• Database growth: The largest tables are typically topology_event and topology_snapshot. Monitor database size with: sudo -u postgres psql osprey -c "SELECT pg_size_pretty(pg_database_size('osprey'));". The retention settings in System Settings control automatic purging.
• SNMP poller concurrency: The SNMP poller walks all targets sequentially within each poll interval. If you have many targets and polls are taking longer than the interval, increase the interval or split targets across multiple poller instances.

Upgrading

When upgrading Osprey (installing a newer .deb package):

1. The postinst script automatically runs database migrations.
2. All five services are restarted.
3. Existing configuration in /etc/osprey/osprey.yaml and /etc/osprey/osprey.env is preserved (these are conffiles).
4. If a migration fails, the error is printed but the package install continues. Run migrations manually after fixing the issue.

# Upgrade
sudo apt install ./osprey_<new-version>_amd64.deb

# Verify all services are running after upgrade
systemctl status osprey.target

# If migrations failed, run manually
source /etc/osprey/osprey.env
/usr/bin/osprey migrate --db-url "postgres://osprey:${OSPREY_DB_PASSWORD}@localhost:5432/osprey?sslmode=disable"