feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)

If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

test/test_web_auth.py

@@ -503,6 +503,42 @@ class TestExemptions:
         assert set(r.get_json()['data']) == {'status'}
         assert 'checks' in admin.get('/api/v3/health').get_json()['data']
 
+    def test_a_stalled_render_loop_reaches_the_minimal_answer(
+            self, config_manager, api_v3_module, tmp_path, monkeypatch):
+        """The display's heartbeat (checks.display_loop) feeds the one word a
+        caller who is not logged in gets, without its detail."""
+        import time
+        from src import display_watchdog
+        from web_interface import display_preview
+        from web_interface.blueprints.api_v3 import misc
+        # Every other check healthy, so the heartbeat alone decides.
+        monkeypatch.setattr(misc, '_get_display_service_status', lambda: {'active': True})
+        monkeypatch.setattr(misc, '_discovered_plugin_manifests', lambda: {})
+        monkeypatch.setattr(api_v3_module.api_v3, 'plugin_catalog', object(), raising=False)
+        preview = tmp_path / 'preview.png'
+        preview.write_bytes(b'png')
+        monkeypatch.setattr(display_preview, 'SNAPSHOT_PATH', str(preview))
+        heartbeat = tmp_path / 'display-heartbeat.json'
+        monkeypatch.setattr(display_watchdog, 'HEARTBEAT_PATH', str(heartbeat))
+
+        def beat(age):
+            heartbeat.write_text(json.dumps({'pid': 1, 'mono': time.monotonic() - age,
+                                             'wall': time.time() - age}))
+
+        app = build(config_manager, api_v3_module)
+        admin = enable(app)
+        stranger = lan_client(app)
+
+        beat(age=2)
+        assert admin.get('/api/v3/health').get_json()['data']['checks']['display_loop']['status'] == 'running'
+        assert stranger.get('/api/v3/health').get_json()['data'] == {'status': 'healthy'}
+
+        beat(age=300)
+        full = admin.get('/api/v3/health').get_json()['data']
+        assert full['checks']['display_loop']['status'] == 'stalled'
+        assert full['status'] == 'degraded'
+        assert stranger.get('/api/v3/health').get_json()['data'] == {'status': 'degraded'}
+
 
 # --- The hash never leaves ------------------------------------------------------