feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)

If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/display_controller.py

@@ -34,6 +34,7 @@ from datetime import datetime
 from concurrent.futures import ThreadPoolExecutor, as_completed  # pylint: disable=no-name-in-module
 import pytz
 
+from src import display_watchdog
 from src.display_manager import DisplayManager
 from src.config_manager import ConfigManager
 from src.config_service import ConfigService
@@ -1052,6 +1053,8 @@ class DisplayController:
             the plugin's update() holds its lock (the panel keeps the last
             frame; that is not a failure).
         """
+        # Every frame of both per-screen render loops comes through here.
+        display_watchdog.watchdog.beat()
         plugin_id = getattr(plugin, 'plugin_id', None)
         with self._display_lock_or_skip(plugin_id) as can_display:
             if not can_display:
@@ -1148,6 +1151,9 @@ class DisplayController:
 
             sleep_time = min(tick_interval, remaining)
             time.sleep(sleep_time)
+            # A dwell can be a minute long (sixty seconds while scheduled
+            # off); the watchdog must hear from this thread throughout.
+            display_watchdog.watchdog.beat()
             self._tick_plugin_updates()
             self._service_pending_changes()
             if (self.current_display_mode != mode
@@ -2238,6 +2244,11 @@ class DisplayController:
                 "plugin is enabled via the web UI."
             )
 
+        # This thread is the one the systemd watchdog and the heartbeat
+        # vouch for: beats from any other thread are ignored, so a render
+        # thread stuck inside a plugin stops them.
+        display_watchdog.watchdog.bind_render_thread()
+
         try:
             # Initialize with cached data for fast startup - let background updates refresh naturally
             logger.info("Starting display with cached data (fast startup mode)")
@@ -2246,6 +2257,11 @@ class DisplayController:
             self._publish_current_mode_state()
             
             while True:
+                # Arms the watchdog after the first frame -- or after the
+                # first full pass, when there is nothing to draw -- and pings
+                # it from then on.
+                display_watchdog.watchdog.loop_pass()
+
                 # Apply plugin enable/disable edits saved via the web UI. The
                 # config-watcher thread only sets the flag; loading/unloading and
                 # rebuilding available_modes happens here on the render thread so
@@ -3625,6 +3641,9 @@ class DisplayController:
 
     def cleanup(self):
         """Clean up resources."""
+        # First: a clean stop is not a hang, and a heartbeat left behind
+        # would read as a frozen panel to the web interface.
+        display_watchdog.watchdog.stopping()
         # Stop the async update worker first so no in-flight update() call
         # is still touching display/cache-backed resources while they're
         # torn down below.