feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)

If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/plugin_system/plugin_manager.py

@@ -18,6 +18,7 @@ import types
 from pathlib import Path
 from typing import Dict, List, NamedTuple, Optional, Any, Tuple, Union
 import logging
+from src import display_watchdog
 from src.exceptions import PluginError, ConfigError
 from src.logging_config import get_logger
 from src.plugin_system.plugin_loader import PluginLoader
@@ -354,6 +355,19 @@ class PluginManager:
         return plugin_ids
 
     def load_plugin(self, plugin_id: str, force_enabled: bool = False) -> bool:
+        """Load a plugin by ID; see _load_plugin.
+
+        Loading can install the plugin's dependencies with pip -- minutes,
+        not seconds. When that happens on the display's render thread (a
+        plugin enabled from the web UI, or loaded for on-demand), its
+        systemd watchdog gets a longer limit for the duration. Start-up
+        loads, on a thread pool, are covered by the start-up allowance.
+        """
+        with display_watchdog.extended(display_watchdog.PLUGIN_LOAD_ALLOWANCE_SECONDS,
+                                       f'loading plugin {plugin_id}'):
+            return self._load_plugin(plugin_id, force_enabled)
+
+    def _load_plugin(self, plugin_id: str, force_enabled: bool = False) -> bool:
         """
         Load a plugin by ID.
         
@@ -1244,6 +1258,9 @@ class PluginManager:
                 # Kill-switch path: the original inline execution
                 # (blocks the caller until update() completes/times out)
                 self._execute_update_now(plugin_id, plugin_instance, current_time)
+                # Up to the executor's 30s each, one after another on the
+                # render thread: check in with its watchdog between them.
+                display_watchdog.beat()
             else:
                 self._enqueue_update(plugin_id, current_time)