Files
LEDMatrix/run.py
T
ChuckandClaude Opus 5.5 64c7289593 feat(display): systemd watchdog and heartbeat for a frozen render loop (#687)
If the render loop gets stuck inside a plugin's display(), ledmatrix.service
stays active and the panel stays frozen. This adds a way to detect that.

- src/display_watchdog.py (standard library only) sends sd_notify over
  $NOTIFY_SOCKET and writes /run/ledmatrix/display-heartbeat.json. Only the
  render thread counts: beats from other threads are ignored.
- ledmatrix.service: WatchdogSec=120, NotifyAccess=main,
  RuntimeDirectory=ledmatrix (0755), RestartSteps=4 and
  RestartMaxDelaySec=2min. It stays Type=simple. run.py widens the watchdog
  to 15 min for start-up, and load_plugin() does the same on the render
  thread. The loop arms after its first frame.
- /api/v3/health adds checks.display_loop: running, stalled (no heartbeat
  for over 60s, which makes the status degraded) or not_reported. With web
  login on, a caller who is not logged in still gets only healthy/degraded,
  and a stall degrades that answer.
- The update verifier requires a fresh heartbeat from the restarted display
  when the display it replaced was writing one. A frozen panel is rolled
  back.
- Existing installs get the systemd watchdog only after install_service.sh
  is re-run. The heartbeat works right away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:15:31 -04:00

62 lines
2.6 KiB
Python
Executable File

#!/usr/bin/env python3
import logging
import sys
import os
import argparse
# Prevent Python from creating __pycache__ directories in plugin dirs.
# The root service loads plugins via importlib, and root-owned __pycache__
# files block the web service (non-root) from updating/uninstalling plugins.
sys.dont_write_bytecode = True
# Add project directory to Python path (needed before importing src modules)
project_dir = os.path.dirname(os.path.abspath(__file__))
if project_dir not in sys.path:
sys.path.insert(0, project_dir)
# Under systemd the watchdog clock is already running, and start-up (plugin
# loads, initial updates) takes far longer than the render loop's limit. Widen
# it before anything slow is imported; the render loop narrows it again once
# its first frame is on the panel. A no-op outside systemd. Standard library
# only -- see src/display_watchdog.py.
from src import display_watchdog
display_watchdog.watchdog.begin_startup()
# Parse command-line arguments BEFORE any imports
parser = argparse.ArgumentParser(description='LEDMatrix Display Controller')
parser.add_argument('-e', '--emulator', action='store_true',
help='Run in emulator mode (uses pygame/RGBMatrixEmulator instead of hardware)')
parser.add_argument('-d', '--debug', action='store_true',
help='Enable debug logging and verbose output')
args = parser.parse_args()
# Set emulator mode if requested (must be done BEFORE any imports that check EMULATOR env var)
if args.emulator:
os.environ["EMULATOR"] = "true"
print("=" * 60)
print("LEDMatrix Emulator Mode Enabled")
print("=" * 60)
print("Using pygame/RGBMatrixEmulator for display")
print("Press ESC to exit\n")
# Debug output (only in debug mode or emulator mode)
debug_mode = args.debug or args.emulator or os.environ.get('LEDMATRIX_DEBUG', '').lower() == 'true'
if debug_mode:
print(f"DEBUG: Project directory: {project_dir}", flush=True)
print(f"DEBUG: Python path[0]: {sys.path[0]}", flush=True)
print(f"DEBUG: Current working directory: {os.getcwd()}", flush=True)
print(f"DEBUG: EMULATOR mode: {os.environ.get('EMULATOR', 'false')}", flush=True)
# Configure logging before importing any other modules
# Use centralized logging configuration
from src.logging_config import setup_logging
log_level = logging.DEBUG if debug_mode else logging.INFO
format_type = 'readable' # Use 'json' for structured logging in production
setup_logging(level=log_level, format_type=format_type, include_location=debug_mode)
# Now import the display controller
from src.display_controller import main
if __name__ == "__main__":
main()