feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)

If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-30 11:15:31 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent b09434a418
commit 64c7289593
20 changed files with 1635 additions and 12 deletions
+14
View File
@@ -301,6 +301,20 @@ def emulator_mode(monkeypatch):
return True
@pytest.fixture(autouse=True)
def _hermetic_display_watchdog(monkeypatch):
"""Keep the render loop's watchdog off the host.
Every test that runs DisplayController.run() arms the process-wide
watchdog, which would ping a real $NOTIFY_SOCKET and write a heartbeat
into /run/ledmatrix -- the live display's, when the suite runs as root on
a device. Each test gets a fresh instance that does neither.
"""
from src import display_watchdog
monkeypatch.setattr(display_watchdog, 'watchdog',
display_watchdog.RenderWatchdog(environ={}, heartbeat_dir=None))
@pytest.fixture(autouse=True)
def reset_logging():
"""Reset logging configuration before each test."""