Files
LEDMatrix/scripts/utils
ChuckandClaude Opus 5.5 64c7289593 feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)
If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:15:31 -04:00
..

Utility Scripts

This directory contains utility scripts for maintenance and system operations.

Scripts

  • clear_cache.py - Clears LEDMatrix cache data (specific keys or all cache)
  • start_web_conditionally.py - Conditionally starts the web interface based on config settings
  • wifi_monitor_daemon.py - Background daemon that monitors WiFi/Ethernet connection and manages access point mode
  • pixlet_config_editor.sh - Opens Pixlet's own config UI for one installed Starlark app
  • apply_dns_single_request.sh - Adds options single-request to the resolver (run by ledmatrix-dns-fix.service)
  • auto_update_verify.py - Health check after an automatic update, rolling back if it fails (the updater copies it to data/ before pulling and ledmatrix-update-verify.service runs that copy)

Usage

Clear Cache

python3 scripts/utils/clear_cache.py --list          # List cache keys
python3 scripts/utils/clear_cache.py --clear-all      # Clear all cache
python3 scripts/utils/clear_cache.py --clear <key>    # Clear specific key

Start Web Interface Conditionally

This script is typically called by the systemd service (ledmatrix-web.service) and checks the web_display_autostart setting in config/config.json before starting the web interface.

WiFi Monitor Daemon

This daemon is typically run as a systemd service (ledmatrix-wifi-monitor.service) and automatically manages WiFi access point mode based on network connectivity.

Pixlet Config Editor

Run it when you want Pixlet's own config form for a Starlark app -- live render preview, cascading dropdowns -- rather than the LEDMatrix one.

./scripts/utils/pixlet_config_editor.sh                 # list installed apps
./scripts/utils/pixlet_config_editor.sh penndot_signs   # edit, on localhost:8080

Deliberately not a service. It stops the display for the length of the session and pixlet serve listens with no authentication, so it should only be running while you are actually editing. It backs the config up first and restarts the display on exit, however it exits.

It binds loopback only, with no flag to change that: anything that can reach pixlet serve can rewrite the app's config, and a printed warning is not access control. To edit from another machine, forward the port -- SSH does the authenticating and nothing is left listening on the LAN:

ssh -L 8080:localhost:8080 pi@ledpi.local

Apply DNS Single-Request Fix

Installed and run by ledmatrix-dns-fix.service; see systemd/README.md. Safe to run by hand (sudo ./scripts/utils/apply_dns_single_request.sh) and idempotent.