mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-06 15:25:08 +00:00
feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)
If the render loop gets stuck inside a plugin's display(), ledmatrix.service stays active and the panel stays frozen. This adds a way to detect that. - src/display_watchdog.py (standard library only) sends sd_notify over $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the render thread counts: beats from other threads are ignored. - ledmatrix.service: WatchdogSec=120, NotifyAccess=main, RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog to 15 min for start-up, and load_plugin() does the same on the render thread. The loop arms after its first frame. - /api/v3/health adds checks.display_loop: running, stalled (no heartbeat for over 60s, which makes the status degraded) or not_reported. With web login on, a caller who is not logged in still gets only healthy/degraded, and a stall degrades that answer. - The update verifier requires a fresh heartbeat from the restarted display when the display it replaced was writing one. A frozen panel is rolled back. - Existing installs get the systemd watchdog only after install_service.sh is re-run. The heartbeat works right away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,7 @@ import time
|
||||
import threading
|
||||
from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING
|
||||
|
||||
from src import display_watchdog
|
||||
from src.common import render_gate
|
||||
from src.vegas_mode.config import VegasModeConfig
|
||||
from src.vegas_mode.plugin_adapter import PluginAdapter
|
||||
@@ -542,6 +543,9 @@ class VegasModeCoordinator:
|
||||
# the whole budget -- the render loop stalls for the size of the
|
||||
# correction. A forward jump inflates p99 and worst-frame instead.
|
||||
frame_started = time.monotonic()
|
||||
# An iteration runs for minutes (max_cycle_duration) without
|
||||
# returning to the display controller's loop.
|
||||
display_watchdog.beat()
|
||||
|
||||
# Check for STATIC mode plugin that should pause scroll
|
||||
static_plugin = self._check_static_plugin_trigger()
|
||||
@@ -923,6 +927,7 @@ class VegasModeCoordinator:
|
||||
|
||||
# Sleep in small increments to remain responsive
|
||||
time.sleep(0.1)
|
||||
display_watchdog.beat()
|
||||
|
||||
logger.info(
|
||||
"Static pause completed for %s after %.1fs",
|
||||
|
||||
Reference in New Issue
Block a user