Get Notified When a Container Goes Down — stacker monitor

What You'll Set Up

A container-down alarm for your deployment using stacker monitor (new in Stacker 0.3.2). You declare an alert once in stacker.yml, and Stacker notifies you the moment any container stops running — and again when it recovers. It's edge-triggered (one alert per change, not per poll), cron-friendly, and needs no web UI. Every command and message below is from a live deployment.

• Configure the alarm with a monitoring.alerts block
• Choose a notification channel: terminal, webhook (ntfy/Slack), or a pipe
• Run it as a loop or a one-shot cron check
• Understand edge-triggering and the state file

Quick Answer

# stacker.yml
monitoring:
  status_panel: true
  alerts:
    interval: 30
    on_recovery: true
    target:
      terminal: true      # or a webhook / pipe — see below

stacker monitor --once    # one check (cron)   ·   stacker monitor   # loop

Step 1: Declare the Alarm in stacker.yml

Add an alerts block under monitoring. Three fields:

monitoring:
  status_panel: true
  alerts:
    interval: 30        # poll interval in seconds (default 60)
    on_recovery: true   # also notify when containers recover (default true)
    target:             # where to deliver the alert (required)
      terminal: true

Validate it like any config change:

stacker config validate      # ✓ Configuration is valid

Step 2: Choose a Notification Channel

The target is one of three shapes:

Terminal + desktop

The simplest — an OS notification, a terminal bell, and a line on stderr. Great for a laptop or a tmux pane you keep open:

target:
  terminal: true

Webhook (ntfy, Slack, anything HTTP)

POST the alert message to any URL. Perfect for phone push via ntfy, or a Slack incoming webhook:

target:
  url: "https://ntfy.example.com/alerts"
  method: POST          # optional, defaults to POST

Pipe (route through a Stacker pipe)

Hand the alert to a declared pipe for richer routing. (Accepted today; dispatch is a follow-up — use terminal or webhook for now.)

target:
  pipe: oncall-notify

Step 3: Run It

Two modes — a long-running loop, or a one-shot check you schedule yourself:

stacker monitor              # loops every `interval` seconds
stacker monitor --once       # a single check, then exits (cron-friendly)
stacker monitor --interval 15  # override the configured interval

For a hands-off setup, put the one-shot form in cron:

# check every minute
* * * * * cd /path/to/project && stacker monitor --once

…or run the loop under systemd / tmux on a box that's always on.

What It Looks Like When a Container Drops

Everything healthy — a quiet line, no alert:

$ stacker monitor --once
· 4 container(s), all healthy

Now stop a container to simulate a failure:

$ docker stop project-ntfy-1      # (on the server)
$ stacker monitor --once
● ⚠️ container problem: 1 not running (project-ntfy-1)
🔔 Stacker container alert: ⚠️ container problem: 1 not running (project-ntfy-1)

Run it again while still down — silence. That's edge-triggering: it already told you once.

$ stacker monitor --once
# (no alert — still down, nothing new)

Restart the container and the next check sends the recovery notice:

$ docker start project-ntfy-1
$ stacker monitor --once
● ✅ all containers recovered
🔔 Stacker container alert: ✅ all containers recovered

With a webhook target, that same message lands on your ntfy topic or Slack channel — verify it yourself:

curl "http://<server-ip>:8080/alerts/json?poll=1"
# {"event":"message","topic":"alerts",
#  "message":"⚠️ container problem: 1 not running (project-ntfy-1)"}

How Edge-Triggering Works

On each check, stacker monitor fetches live container health from the Status Panel agent, decides up (all running) or down (any not running), and compares it to the last known state stored in .stacker/monitor.state. It only notifies on a change:

• up → down → "container problem" alert
• down → up → "recovered" alert (unless on_recovery: false)
• same state → nothing

Because the state is a file, one-shot --once runs stay edge-triggered across separate cron invocations — you won't get a fresh alert every minute while a container is down. A brand-new watcher against an already-down stack alerts immediately (a sensible baseline).

Frequently Asked Questions

Is this different from stacker status --watch --notify?

Yes. status --watch --notify notifies on deployment lifecycle (a deploy reaching completed/paused/failed). stacker monitor watches ongoing container health after deploy.

Do I need the Status Panel agent?

Yes — set monitoring.status_panel: true so the agent is installed during deploy (or add it later with stacker agent install). The monitor reads live health from it.

Will cron spam me while a container stays down?

No — that's the whole point of edge-triggering plus the .stacker/monitor.state file. You get one alert on the drop and one on recovery.

Can I alert only, not on recovery?

Set on_recovery: false — you'll get the problem alerts but not the "recovered" notices.

How do I watch multiple projects?

Run stacker monitor --once from each project directory (each has its own stacker.yml and .stacker/monitor.state) — one cron line per project.

Key Takeaways

• monitoring.alerts + stacker monitor = a built-in container-down alarm (Stacker 0.3.2), no web UI
• Three channels: terminal, HTTP url (ntfy/Slack), or a pipe
• Edge-triggered via .stacker/monitor.state — one alert per change, cron-safe
• --once for cron, the bare command for a long-running loop
• Recovery notices are on by default; turn them off with on_recovery: false