Monitoring Configuration Guide

Last updated: February 8, 2026
Admin Tools

Monitoring Configuration Guide

This guide covers SMTP email setup, notification channel configuration, and optional integrations for Slack and PagerDuty.

Email Notification Setup (Required)

Overview

Email notifications are the primary alert delivery method. Alerts are sent via SMTP (Simple Mail Transfer Protocol) to configured recipients.

Supported SMTP Providers:

  • Gmail (recommended, free)
  • Outlook/Office 365
  • SendGrid
  • Mailgun
  • Self-hosted SMTP server

Gmail SMTP Setup (Recommended)

Step 1: Enable 2-Factor Authentication

⚠️ REQUIRED: Gmail requires 2FA to use App Passwords.

  1. Go to: https://myaccount.google.com/security
  2. Click "2-Step Verification"
  3. Follow setup wizard to enable 2FA
  4. Verify 2FA is active (shows "On" in security settings)

Step 2: Create App Password

What is an App Password?

A 16-character password generated by Google for applications that don't support 2FA login. This is NOT your regular Gmail password.

Steps:

  1. Go to: https://myaccount.google.com/apppasswords
  2. You may need to re-authenticate with your Gmail password
  3. Select app: "Mail"
  4. Select device: "Other (Custom name)"
  5. Enter custom name: "Sampo Alertmanager"
  6. Click "Generate"
  7. Copy the 16-character password (displayed with spaces, e.g., abcd efgh ijkl mnop)

Important:

  • Remove spaces when copying (e.g., abcdefghijklmnop)
  • Store securely (you won't be able to view it again)
  • If lost, delete old App Password and create new one

Step 3: Add SMTP Credentials to Server

On VPS (via SSH):

# Connect to VPS
ssh root@<VPS-IP>

# Navigate to deployment directory
cd /opt/sampo-alpha

# Edit environment file
nano .env.blueline-alpha

Add these lines (update with your values):

# Alertmanager Email Configuration
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USERNAME=your-email@gmail.com
SMTP_PASSWORD=abcdefghijklmnop  # Your 16-char App Password (no spaces)
ALERT_EMAIL_FROM=alerts@theblueline.com
ALERT_EMAIL_TO=admin@theblueline.com
ALERT_EMAIL_CRITICAL=admin@theblueline.com,ops@theblueline.com
ALERT_EMAIL_WARNING=admin@theblueline.com
ALERT_EMAIL_DATABASE=dba@theblueline.com,admin@theblueline.com
ALERT_EMAIL_INFRA=ops@theblueline.com,admin@theblueline.com

Save and exit: Ctrl+X, then Y, then Enter

Step 4: Restart Alertmanager

# Restart to load new environment variables
docker restart blueline-alpha-alertmanager

# Verify no SMTP errors in logs
docker logs blueline-alpha-alertmanager --tail 50 | grep -i "smtp\|email\|error"

Expected output: No errors related to SMTP configuration.

Step 5: Test Email Delivery

Trigger a test alert by stopping the API container:

# Stop API to trigger APIDown alert
docker stop blueline-alpha-api

# Wait 90 seconds for alert to fire (1m threshold + 10s group wait)
sleep 90

# Check if alert fired
curl http://localhost:9093/api/v2/alerts | jq '.[] | select(.labels.alertname=="APIDown")'

Expected: Email received at ALERT_EMAIL_CRITICAL address within 2 minutes.

Resolve alert:

# Restart API
docker start blueline-alpha-api

# Wait 2 minutes for resolution
sleep 120

Expected: Resolution email received within 3 minutes.

Email Configuration Reference

Environment Variables

| Variable | Description | Example | Required | | ---------------------- | --------------------------------------- | ------------------------------------------- | -------- | | SMTP_HOST | SMTP server hostname | smtp.gmail.com | Yes | | SMTP_PORT | SMTP server port (587 = TLS, 465 = SSL) | 587 | Yes | | SMTP_USERNAME | SMTP authentication username | your-email@gmail.com | Yes | | SMTP_PASSWORD | SMTP authentication password | abcdefghijklmnop | Yes | | ALERT_EMAIL_FROM | "From" address in alert emails | alerts@theblueline.com | Yes | | ALERT_EMAIL_TO | Default recipient (all alerts) | admin@theblueline.com | Yes | | ALERT_EMAIL_CRITICAL | Recipients for CRITICAL alerts | admin@theblueline.com,ops@theblueline.com | No | | ALERT_EMAIL_WARNING | Recipients for WARNING alerts | admin@theblueline.com | No | | ALERT_EMAIL_DATABASE | Recipients for database alerts | dba@theblueline.com,admin@theblueline.com | No | | ALERT_EMAIL_INFRA | Recipients for infrastructure alerts | ops@theblueline.com,admin@theblueline.com | No |

Multi-Recipient Format

Use commas to separate multiple email addresses (no spaces):

ALERT_EMAIL_CRITICAL=admin@example.com,ops@example.com,oncall@example.com

Alternative SMTP Providers

Outlook / Office 365

SMTP_HOST=smtp.office365.com
SMTP_PORT=587
SMTP_USERNAME=your-email@outlook.com
SMTP_PASSWORD=your-password

Note: Outlook does not require App Passwords unless 2FA is enabled.

SendGrid (Transactional Email Service)

Benefits: Higher deliverability, detailed analytics, 100 emails/day free

Setup:

  1. Create account: https://signup.sendgrid.com/
  2. Verify sender email address
  3. Create API Key:
    • Settings → API Keys → Create API Key
    • Name: "Sampo Alertmanager"
    • Permissions: "Full Access" or "Mail Send" only
  4. Copy API key (shown only once)

Configuration:

SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USERNAME=apikey  # Literally the word "apikey"
SMTP_PASSWORD=SG.abc123...  # Your SendGrid API key
ALERT_EMAIL_FROM=alerts@yourdomain.com  # Must be verified in SendGrid

Self-Hosted SMTP Server

SMTP_HOST=mail.yourdomain.com
SMTP_PORT=587  # Or 465 for SSL, 25 for no TLS (not recommended)
SMTP_USERNAME=alerts@yourdomain.com
SMTP_PASSWORD=your-password

TLS/SSL Notes:

  • Port 587: STARTTLS (recommended)
  • Port 465: SSL/TLS
  • Port 25: No encryption (not recommended for production)

Notification Receivers

How Alerts are Routed

Alerts are routed to different receivers based on severity and category:

Alert Fired → Alertmanager → Route Decision → Receiver → Email Sent

Receiver Types

| Receiver | Triggers | Recipients | Group Wait | | ---------------------- | ------------------------- | ---------------------- | ---------- | | default-email | All alerts (fallback) | ALERT_EMAIL_TO | 1m | | critical-email | severity=critical | ALERT_EMAIL_CRITICAL | 10s | | warning-email | severity=warning | ALERT_EMAIL_WARNING | 1m | | database-email | category=database | ALERT_EMAIL_DATABASE | 1m | | infrastructure-email | category=infrastructure | ALERT_EMAIL_INFRA | 1m |

Routing Example

Alert: HighMemoryUsage (severity: critical)

Route Decision:

  1. Check severity → critical
  2. Match receiver → critical-email
  3. Wait 10 seconds for grouping
  4. Send to recipients in ALERT_EMAIL_CRITICAL

Alert Grouping Configuration

What is Alert Grouping?

Multiple similar alerts are combined into a single email notification to prevent spam.

Example:

  • 5 containers restart simultaneously
  • Without grouping: 5 separate emails
  • With grouping: 1 email listing all 5 restarts

Grouping Rules

Alerts are grouped by:

  • alertname (e.g., all "HighMemoryUsage" alerts together)
  • deployment (e.g., all "blueline" alerts together)
  • severity (e.g., all "critical" alerts together)

Grouping Timing

| Parameter | Critical Alerts | Warning Alerts | Purpose | | ----------------- | --------------- | -------------- | ------------------------------------------ | | group_wait | 10 seconds | 1 minute | Wait before sending first notification | | group_interval | 5 minutes | 5 minutes | Wait before sending updates for same group | | repeat_interval | 30 minutes | 2 hours | Resend if alert still firing |

Example Timeline (Critical Alert):

00:00 - Alert fires
00:10 - Email sent (after 10s group_wait)
00:15 - Another instance fires → grouped with existing
00:15 - No new email (within group_interval)
00:40 - Email resent if still firing (30m repeat_interval)

Inhibition Rules

What are Inhibition Rules?

Rules that suppress lower-severity alerts when higher-severity alerts are already firing for the same deployment.

Current Inhibition Rules

Rule: Suppress WARNING if CRITICAL alert firing for same deployment

Example:

  • HighMemoryUsage (CRITICAL, >90%) fires → Email sent
  • MemoryUsageWarning (WARNING, >70%) also fires → Suppressed (no email)
  • HighMemoryUsage resolves → MemoryUsageWarning email sent (if still firing)

Benefit: Reduces alert fatigue by focusing on most urgent issues.

Advanced: Slack Integration (Optional)

Status: Ready to enable (currently commented out)

Prerequisites

  1. Slack workspace admin access
  2. Ability to create Incoming Webhooks

Setup Steps

1. Create Slack Incoming Webhook

  1. Go to: https://api.slack.com/messaging/webhooks
  2. Click "Create your Slack app"
  3. Choose "From scratch"
  4. App name: "Sampo Alerts"
  5. Select workspace
  6. Click "Incoming Webhooks" (left sidebar)
  7. Toggle "Activate Incoming Webhooks" to On
  8. Click "Add New Webhook to Workspace"
  9. Select channel (e.g., #sampo-alerts)
  10. Click "Allow"
  11. Copy Webhook URL (e.g., https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX)

2. Add Webhook to Server

Edit .env.blueline-alpha:

# Slack Configuration
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

3. Enable Slack Receiver

Edit deployments/blueline-alpha/alertmanager.yml:

Find (around line 60):

# Slack Integration (Optional)
# Uncomment to enable Slack notifications
# - name: 'slack-critical'
#   slack_configs:
#     - api_url: '${SLACK_WEBHOOK_URL}'

Change to:

# Slack Integration (Optional)
# Uncomment to enable Slack notifications
- name: 'slack-critical'
  slack_configs:
    - api_url: '${SLACK_WEBHOOK_URL}'
      channel: '#sampo-alerts-critical'
      title: '🚨 CRITICAL ALERT: {{ .GroupLabels.alertname }}'
      text: '{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}'

4. Update Routes

Find (around line 30):

# Critical alerts → email only
- match:
    severity: critical
  receiver: critical-email

Change to:

# Critical alerts → email + Slack
- match:
    severity: critical
  receiver: critical-email
  continue: true # Continue to next route
- match:
    severity: critical
  receiver: slack-critical

5. Restart Alertmanager

docker restart blueline-alpha-alertmanager

6. Test Slack Notification

Trigger test alert (see "Test Email Delivery" above).

Expected: Message posted to Slack channel.

Advanced: PagerDuty Integration (Optional)

Status: Ready to enable (currently commented out)

Prerequisites

  1. PagerDuty account (free tier available)
  2. Service created in PagerDuty

Setup Steps

1. Create PagerDuty Integration

  1. Log in to PagerDuty
  2. Go to: Services → Service Directory
  3. Click "+ New Service"
  4. Service name: "Sampo Production Alerts"
  5. Escalation Policy: Select or create policy
  6. Click "Next"
  7. Integration Type: "Events API V2"
  8. Click "Create Service"
  9. Copy Integration Key (Routing Key)
    • Format: abc123def456ghi789jkl012mno345pq

2. Add Integration Key to Server

Edit .env.blueline-alpha:

# PagerDuty Configuration
PAGERDUTY_ROUTING_KEY=abc123def456ghi789jkl012mno345pq

3. Enable PagerDuty Receiver

Edit deployments/blueline-alpha/alertmanager.yml:

Find (around line 85):

# PagerDuty Integration (Optional)
# - name: 'pagerduty-critical'
#   pagerduty_configs:
#     - routing_key: '${PAGERDUTY_ROUTING_KEY}'

Change to:

# PagerDuty Integration (Optional)
- name: 'pagerduty-critical'
  pagerduty_configs:
    - routing_key: '${PAGERDUTY_ROUTING_KEY}'
      severity: critical
      description:
        '{{ .GroupLabels.alertname }}: {{ .CommonAnnotations.description }}'

4. Update Routes

Add PagerDuty to critical alert route (same as Slack above):

# Critical alerts → email + PagerDuty
- match:
    severity: critical
  receiver: critical-email
  continue: true
- match:
    severity: critical
  receiver: pagerduty-critical

5. Restart Alertmanager

docker restart blueline-alpha-alertmanager

6. Test PagerDuty Integration

Trigger test alert.

Expected: Incident created in PagerDuty, on-call engineer paged.

Troubleshooting

Issue: No Emails Received

Diagnosis:

  1. Check Alertmanager logs:

    docker logs blueline-alpha-alertmanager --tail 100 | grep -i "smtp\|email\|error"

  2. Test SMTP connection:

    docker exec blueline-alpha-alertmanager \
  wget -O- --post-data='{}' http://localhost:9093/-/reload

  3. Check environment variables loaded:

    docker exec blueline-alpha-alertmanager env | grep SMTP

Common Causes:

  • Incorrect Gmail App Password (not regular password)
  • SMTP credentials not loaded (restart Alertmanager)
  • Port 587 blocked by firewall
  • Email address typo in ALERT_EMAIL_TO

Resolution:

# Verify credentials in .env file
cat .env.blueline-alpha | grep SMTP

# Restart Alertmanager
docker restart blueline-alpha-alertmanager

Issue: Emails Go to Spam

Causes:

  • No SPF/DKIM records for "From" domain
  • Gmail blocking automated emails

Solutions:

  1. Use proper "From" domain:

    ALERT_EMAIL_FROM=alerts@yourdomain.com  # NOT alerts@theblueline.com if you don't own it

  2. Add SPF record (if you own domain):

    TXT record: v=spf1 include:_spf.google.com ~all

  3. Use SendGrid for better deliverability (see Alternative SMTP Providers)

Issue: Alert Fired But No Email

Check if alert actually firing:

curl http://localhost:9093/api/v2/alerts | jq '.[] | select(.labels.alertname=="APIDown")'

Check Alertmanager routing:

docker exec blueline-alpha-alertmanager cat /etc/alertmanager/alertmanager.yml | grep -A 10 "route:"

Check for silences:

curl http://localhost:9093/api/v2/silences | jq '.[] | select(.status.state=="active")'

Configuration File Reference

alertmanager.yml Location

Path: deployments/blueline-alpha/alertmanager.yml

Edit (on local machine before deployment):

nano /path/to/sampo/deployments/blueline-alpha/alertmanager.yml

Validate changes:

# Use promtool to validate (if installed)
promtool check config deployments/blueline-alpha/alertmanager.yml

Deploy changes:

# Restart Alertmanager to reload config
docker restart blueline-alpha-alertmanager

Related Articles

Support

For configuration assistance:

  • Complete Reference: docs/operations/alerting-guide.md
  • SMTP Troubleshooting: See "Configuration" section in alerting guide
  • Technical Support: Contact DevOps team

Last Updated: 2026-02-08
Configuration Version: Week 3+ (Gmail SMTP recommended)

Was this article helpful?

Your feedback helps us improve our support content.

Still need assistance?

Our support team is ready to help you with more complex issues.

Contact Support