mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-10-04 06:15:09 +00:00

Files

T

ChuckandClaude Opus 5.5 64c7289593 feat(display): systemd watchdog and heartbeat for a frozen render loop (#687 )

If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-09-30 11:15:31 -04:00

4.4 KiB

Raw Blame History

Systemd Service Files

This directory contains systemd service unit files for LEDMatrix services.

Service Files

ledmatrix.service - Main LED Matrix display service
- Runs the display controller (run.py)
- Starts automatically on boot
- Runs as root for hardware access
- Restarted by systemd's watchdog (WatchdogSec=120) when its render loop stops checking in, e.g. stuck inside a plugin; the loop writes a heartbeat to /run/ledmatrix/display-heartbeat.json (RuntimeDirectory=) that the web interface's health check reads. See src/display_watchdog.py
ledmatrix-web.service - Web interface service
- Runs the web interface conditionally based on config
- Starts automatically on boot if web_display_autostart is enabled
- Uses scripts/utils/start_web_conditionally.py
ledmatrix-update-verify.service / .path - Automatic update health check and rollback
- The path unit starts the service when the web interface creates data/auto_update_verify.request after a weekly automatic update
- Installed by the installers, or by the display service when automatic updates are turned on in the web UI (src/auto_update_setup.py)
- Restarts the services, checks they stay up, and otherwise resets to the previous commit and its dependencies
- Runs data/auto_update_verifier.py, a copy of scripts/utils/auto_update_verify.py taken before the update
ledmatrix-wifi-monitor.service - WiFi monitor daemon service
- Monitors WiFi/Ethernet connectivity
- Automatically enables/disables access point mode
- Uses scripts/utils/wifi_monitor_daemon.py
ledmatrix-dns-fix.service - DNS single-request fix (optional)
- Re-applies options single-request to the resolver on every boot, because whatever manages resolv.conf regenerates it and drops the option again
- Works around glibc's parallel A/AAAA lookup stalling ~5s per name on routers that answer only the A query, which makes any plugin calling an external API slow or (for Starlark apps, which have a render timeout) fail outright
- Uses scripts/utils/apply_dns_single_request.sh
- Install only if external API calls are timing out; it is not part of a normal install
ledmatrix-mqtt-bridge.service - Home Assistant MQTT bridge (optional)
- Exposes the display to Home Assistant over MQTT Discovery: force a mode, stop on-demand, toggle power, set brightness
- Uses integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py, which drives the web API rather than the display directly
- Needs integrations/mqtt_bridge/bridge_config.json; see that directory's README

Installation

These service files are installed by the installation scripts in scripts/install/:

install_service.sh installs ledmatrix.service, ledmatrix-web.service and the ledmatrix-update-verify service and path units, then enables and starts them
install_web_service.sh installs only ledmatrix-web.service and the ledmatrix-update-verify units
install_wifi_monitor.sh installs ledmatrix-wifi-monitor.service
install_dns_fix.sh installs ledmatrix-dns-fix.service (opt-in, not run by the normal installer)
install_mqtt_bridge.sh installs ledmatrix-mqtt-bridge.service (opt-in)

Manual Installation

Important: the unit files in this directory contain __PROJECT_ROOT_DIR__ placeholders that the install scripts replace with the actual project directory at install time. Do not copy them directly to /etc/systemd/system/ — the service will fail to start with WorkingDirectory=__PROJECT_ROOT_DIR__ errors.

Always install via the helper script:
sudo ./scripts/install/install_service.sh
If you really need to do it by hand, substitute the placeholder first:
PROJECT_ROOT="$(pwd)"
sed "s|__PROJECT_ROOT_DIR__|$PROJECT_ROOT|g" systemd/ledmatrix.service \
  | sudo tee /etc/systemd/system/ledmatrix.service > /dev/null
sudo systemctl daemon-reload
sudo systemctl enable ledmatrix.service
sudo systemctl start ledmatrix.service

Service Management

# Check status
sudo systemctl status ledmatrix.service

# Start/stop/restart
sudo systemctl start ledmatrix.service
sudo systemctl stop ledmatrix.service
sudo systemctl restart ledmatrix.service

# Enable/disable autostart
sudo systemctl enable ledmatrix.service
sudo systemctl disable ledmatrix.service

# View logs
journalctl -u ledmatrix.service -f

4.4 KiB Raw Blame History