Monitoring and alerts

With features.monitoring: true (default) the following is installed:

Part	Does
Prometheus	Collects metrics from the server, the containers, Traefik, the databases and the services' health checks
Grafana	Shows metrics and logs in ready-made dashboards
Loki	Collects the containers' logs
Alertmanager	Sends alerts by e-mail
Exporters and health checks	node-exporter, cAdvisor, Redis, PostgreSQL, Kafka (and PgBouncer if used), plus checks of every service's /health and of the TURN server

Alertmanager, exporters and alert rules can be switched off with monitoring.alerting: false.

Grafana

Open https://admin.example.se/grafana. With monitoring.grafana_public: true Grafana is also available at https://grafana.example.se (requires a DNS record).

The user name is admin. The password:

sudo grep grafana_password /opt/eurion/deploy/generated/group_vars/secrets.yml

Ready-made dashboards: system, containers, Traefik, services, Redis, PostgreSQL, Kafka, TURN, gateway, e-mail, MinIO, PgBouncer, video and logs. The home page is the gateway dashboard.

Logs in Grafana

Explore → Loki and the query {container_name=~"eurion-.*"} show the logs of every Eurion container in one place.

Alerts by e-mail

Alertmanager sends alerts to monitoring.alert_email, or to organization.admin_email if it is not set. It uses the same SMTP server as the rest of Eurion (smtp in the configuration).

No SMTP, no alerts

If smtp.host is empty, no alerts are sent at all, even though Alertmanager is running. See Outgoing e-mail.

The subject line looks like this:

[FIRING] Eurion (example.se): DiskSpaceLow
[RESOLVED] Eurion (example.se): DiskSpaceLow

An active alert is sent again every four hours, critical alerts every hour. You also get a message when the alert clears.

Alert rules

Alert	Level	Fires when
HighCPUUsage / CriticalCPUUsage	warning / critical	The server's CPU above 85 % for 5 min / above 95 % for 2 min
LowMemory	critical	Less than 10 % free memory for 5 min
DiskSpaceLow / DiskSpaceCritical	warning / critical	The root file system more than 85 % / 95 % full
ServiceDown	critical	A service's health check fails for 1 min
ServiceSlowResponse	warning	The health check takes more than 2 s for 5 min
CoturnDown	critical	The TURN server does not answer for 2 min
RedisDown, PostgresDown, KafkaBrokerDown	critical	Redis, PostgreSQL or Kafka down
RedisHighMemory	warning	Redis uses more than 85 % of its memory limit
PostgresHighConnections	warning	More than 80 connections to PostgreSQL
KafkaConsumerLag	warning	More than 10,000 unprocessed events in Kafka
TraefikHighErrorRate	critical	More than 5 % of responses are server errors
TraefikServiceDown	critical	A service returns errors on more than 25 % of requests
TraefikTLSCertExpiringSoon	warning	A certificate expires within 14 days
ContainerRestartLoop	critical	A container restarted more than 3 times in 15 min
ContainerOOM	critical	A container ran out of memory
ContainerHighCPU	warning	A container uses more than 90 % CPU
ContainerHighMemory	warning	A container is close to its memory limit (only containers with a limit, see resources)
MinIODiskUsageHigh, MinIODiskOffline, MinIOHighErrorRate	warning / critical	Problems with file storage
PgBouncerPoolSaturation, PgBouncerClientsWaiting	warning	The connection pool is nearly full (only with PgBouncer)
RedisExporterDown, PostgresExporterDown, KafkaExporterDown, TraefikMetricsDown	warning	A metrics source does not answer

Quick check without Grafana

sudo eurion-ctl status       # services and infrastructure
sudo eurion-ctl health       # stopped containers, restarts, disk, connections, backups

See Troubleshooting for what the output means.