Compare commits

...
Author SHA1 Message Date
ChuckandClaude Opus 5 e6249dcc7e fix(service): survive corrupt health cache and clean exits
Three independent failure modes that each end with a dark panel and no
automatic recovery.

1. PluginHealthTracker._load_health_state returned the cached value
   verbatim. If that value is not a dict, every caller raises
   AttributeError: 'list' object has no attribute 'get' — during
   DisplayController.__init__, so the process dies before the display
   loop starts. systemd restarts it, the same bad entry is read back
   from disk, and it dies again: an unattended restart loop that
   survives reboots because the cause is persisted. Observed in the
   field with plugin_health:<id> holding an unrelated plugin's list
   payload. Now non-dict entries are discarded with a warning and the
   defaults are rebuilt.

2. ledmatrix.service used Restart=on-failure, so any exit with status 0
   left the unit stopped and the panel dark indefinitely — systemd
   treats it as success and never brings it back. Restart=always.

3. ledmatrix-wifi-monitor.service used StandardOutput=syslog, which
   systemd has marked obsolete; it warns and rewrites it to journal on
   every load.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 19:04:57 -04:00
3 changed files with 19 additions and 4 deletions
+12 -1
View File
@@ -64,9 +64,20 @@ class PluginHealthTracker:
cache_key, max_age=None, memory_ttl=0 if force_reload else None cache_key, max_age=None, memory_ttl=0 if force_reload else None
) )
if cached: if isinstance(cached, dict) and cached:
return cached return cached
# A cache entry that is not a dict means the persisted state was written
# by something other than _save_health_state (a key collision, a partial
# write, a restored backup). Returning it verbatim makes every caller
# blow up on .get(), which takes the display down in a restart loop that
# survives reboots because the bad entry is on disk. Discard and rebuild.
if cached is not None and not isinstance(cached, dict):
self.logger.warning(
f"Discarding malformed health state for {plugin_id}: expected "
f"dict, got {type(cached).__name__}. Falling back to defaults."
)
# Default state # Default state
return { return {
'consecutive_failures': 0, 'consecutive_failures': 0,
+2 -2
View File
@@ -10,8 +10,8 @@ WorkingDirectory=__PROJECT_ROOT_DIR__
ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/scripts/utils/wifi_monitor_daemon.py --interval 30 ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/scripts/utils/wifi_monitor_daemon.py --interval 30
Restart=on-failure Restart=on-failure
RestartSec=10 RestartSec=10
StandardOutput=syslog StandardOutput=journal
StandardError=syslog StandardError=journal
SyslogIdentifier=ledmatrix-wifi-monitor SyslogIdentifier=ledmatrix-wifi-monitor
[Install] [Install]
+5 -1
View File
@@ -9,7 +9,11 @@ User=root
WorkingDirectory=__PROJECT_ROOT_DIR__ WorkingDirectory=__PROJECT_ROOT_DIR__
Environment=PYTHONDONTWRITEBYTECODE=1 Environment=PYTHONDONTWRITEBYTECODE=1
ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py
Restart=on-failure # Restart=always, not on-failure: run.py exiting 0 (a clean shutdown path taken
# for a reason that no longer applies, e.g. a config reload) would otherwise leave
# the service stopped and the panel dark indefinitely, with systemd considering
# that a successful outcome and never bringing it back.
Restart=always
RestartSec=10 RestartSec=10
StandardOutput=journal StandardOutput=journal
StandardError=journal StandardError=journal