2 Commits

Include renames

Author	SHA1	Message	Date
3 people	0c5b9c57d3	fix: keep low-memory boards reachable under load (#464) ... * fix(service): survive corrupt health cache and clean exits Three independent failure modes that each end with a dark panel and no automatic recovery. 1. PluginHealthTracker._load_health_state returned the cached value verbatim. If that value is not a dict, every caller raises AttributeError: 'list' object has no attribute 'get' — during DisplayController.__init__, so the process dies before the display loop starts. systemd restarts it, the same bad entry is read back from disk, and it dies again: an unattended restart loop that survives reboots because the cause is persisted. Observed in the field with plugin_health:<id> holding an unrelated plugin's list payload. Now non-dict entries are discarded with a warning and the defaults are rebuilt. 2. ledmatrix.service used Restart=on-failure, so any exit with status 0 left the unit stopped and the panel dark indefinitely — systemd treats it as success and never brings it back. Restart=always. 3. ledmatrix-wifi-monitor.service used StandardOutput=syslog, which systemd has marked obsolete; it warns and rewrites it to journal on every load. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(memory): size the cache to the board and stop reinstalling deps On a 1GB Pi 3B+ the display process settles around 600MB RSS of 905MB total. When the remaining headroom runs out the failure is not a clean crash: fork() starts returning ENOMEM, so sshd accepts connections and closes them before its banner, timer jobs stop running, and the panel goes dark, while already-resident processes keep serving normally. The board looks healthy from outside and cannot be logged into. Only a power cycle clears it. Three contributing causes: - MemoryCache had a fixed 1000-entry ceiling. Entries are parsed API payloads of tens of KB, so one ceiling cannot serve both a 512MB Zero 2 W and an 8GB Pi 5. Now scaled from MemTotal (150 entries at <=1GB, 1500 at >=8GB), overridable with LEDMATRIX_CACHE_MAX_ENTRIES. - requirements_are_satisfied() returned False for any requirement with extras, so a plugin depending on python-socketio[client] re-ran pip on every single start: ~8s, a network dependency, and a 100-200MB spike at the least convenient moment. During a restart loop it repeats for each restart. Extras are now resolved one level deep against installed metadata, keeping the conservative "anything unverifiable falls through to pip" contract. - ledmatrix.service had no memory ceiling. MemoryMax=85% expressed as a percentage so one unit file suits every board. Note this needs the memory cgroup controller, which Pi firmware disables by default; first_time_install.sh now adds cgroup_enable=memory to cmdline.txt, and the unit file documents how to verify it took effect. first_time_install.sh also enables persistent journald storage (capped at 64M). Default storage is volatile, so every reboot destroys the logs that would explain why the board rebooted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: guidance for 512MB and 1GB boards Documents the memory ceiling on small boards and, more usefully, what running into it actually looks like: sshd accepting connections and closing them before the banner, the web UI still responding normally, clean ping, a dark panel, and a wrong clock after the next boot. None of those read as "out of memory", which makes the failure hard to identify from the symptoms. Cross-referenced from SSH_UNAVAILABLE_AFTER_INSTALL.md, since "I can't SSH in any more" is how most people will first meet this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: address review findings on the low-memory work Nine CodeRabbit findings, five in code. **Health state (the one that matters).** The non-dict guard did not cover a dict missing fields the callers index directly, which is the shape actually seen in the wild: a record carrying only circuit_state produced `plugin clock-simple operation failed: 'circuit_state'` about fifty times a minute with the panel frozen. The record is now completed against the defaults per field rather than trusted or discarded wholesale. Per field matters: a first pass rejected any incomplete record outright, which reset a tripped breaker and real failure counts to healthy because one optional field was absent -- an existing test caught it. Values of the wrong type (a counter persisted as a string, an unknown circuit_state) fall back individually, valid neighbours survive, and newer fields the schema has grown since (degraded, degraded_reason) are carried through untouched. **Cache ceiling.** MemoryCache.set() accepted entries without bound between cleanup sweeps, which run every 300s by default, so a burst could take the cache far past max_size -- the unbounded growth the limit exists to stop. Eviction now runs under the same lock on every write, sharing one helper with the periodic sweep so the two cannot drift. **Installer, cgroups.** Only cgroup_enable=memory was checked, so a board carrying that without cgroup_memory=1 reported success and got no change, leaving MemoryMax= inert. Each parameter is now checked and appended independently; verified against all four combinations, single line preserved. **Installer, journald.** Persistence was inferred from /var/log/journal being non-empty, which proves neither Storage=persistent nor a size cap -- the directory survives a switch back to volatile. The effective configuration is read instead (systemd-analyze cat-config, falling back to the conf files), and an explicitly configured SystemMaxUse is preserved rather than overwritten. Verified across volatile, persistent-without-cap, persistent-with-user-cap, cap-without-storage, and commented-only configs. **Dependency extras.** _extras_are_satisfied stopped at one level, so a gated dependency that itself requests an extra (requests[socks]) passed on the base distribution's version while the extra's own dependency was missing, and pip was skipped. It now recurses, with a visited (distribution, extras) set so a cycle terminates. Docs: both kernel command-line paths documented (the installer falls back to /boot/cmdline.txt), daemon-reload and restart added after the systemd override example, memory exhaustion added to the SSH summary with its power-cycle-only recovery, and a language on the fenced block for MD040. Tests: five for the health-state repair including the exact wild shape and that record_failure/record_success no longer raise against it, and one for the cache ceiling. Both mutation-checked. Full suite 2927 passed, with the one pre-existing tmpfs failure that also fails on main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW * fix: harden the health-state repair and confirm journald took effect Second review round; all three findings were valid and two were bugs in the repair added last commit. The repair could raise out of itself. An unhashable circuit_state (a list or dict on disk) hit `value in {...}` and raised TypeError -- from the code whose whole job is to stop a malformed record crashing the caller. It now requires a str before the membership test. bool is a subclass of int, so True passed the timestamp check and then compared as 1.0: enough to expire a cooldown the instant the breaker opened, while False would stop the elapsed check firing at all. Timestamps now exclude bool explicitly. The regression test for the original crash was seeded with a record that *contained* circuit_state, so it passed against the old raw-return behaviour too -- the counters are read with .get(), so circuit_state is the only field whose absence used to raise. Reseeded to omit it, and it now fails against raw-return as intended. journald: drop-ins apply in lexical order, so a local file sorting after ledmatrix-persistent.conf still wins and writing ours proves nothing. The effective Storage is re-read afterwards and a warning naming the diagnostic command is printed if persistence is still not active, rather than reporting a success that was not verified. Full suite 2934 passed, same single pre-existing tmpfs failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-19 12:28:22 -04:00
3 people	3b93024993	feat: activate dormant plugin health/metrics subsystem and surface it in the web UI (#388) ... * feat(plugin-system): activate dormant plugin health & metrics subsystem PluginManager shipped a fully-built health tracker, resource monitor and circuit breaker that were never instantiated (health_tracker/resource_monitor were left as None), so the circuit breaker never engaged and the existing health/metrics API routes always returned "not available". - DisplayController now wires a PluginHealthTracker and PluginResourceMonitor onto the plugin manager, enabling the circuit breaker (a repeatedly-failing plugin's update() is skipped after consecutive failures, then retried after a cooldown) and per-plugin execution-time metrics. Both persist to the shared cache. - load_plugin() now validates each plugin's config against its JSON schema in a strictly warn/degrade-only way: a violation logs a warning and flags the plugin degraded in the health tracker, but never changes whether the plugin loads or its pass/fail behaviour. Adds PluginHealthTracker.set_degraded(), which never touches the circuit breaker. - ResourceMonitor CPU/memory sampling now reuses a cached psutil.Process and reads cpu_percent(interval=None), so monitoring no longer blocks ~100ms per call on the display loop's update path. - Fix DiskCache.get() raising TypeError for max_age=None ("never expires"), which silently discarded persisted plugin health/metrics on read and thus broke cross-process and post-restart surfacing. - Fix two dead PluginManager helpers that called non-existent tracker methods. Tests: new test_resource_monitor, test_plugin_health, test_plugin_manager_schema_soft; extended test_cache_manager and test_display_controller. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq * feat(web-ui): surface plugin health, metrics and load state With the health/metrics subsystem now active in the display service, expose it in the web UI (which runs as a separate process from the display loop): - Wire a health tracker / resource monitor backed by the shared on-disk cache into the web process so /api/v3/plugins/health and /plugins/metrics read the data the display service persists. - Build those route responses per installed plugin id (the tracker's in-memory view is empty in a fresh web process) so cross-process data is included. - Add state + error_info to /plugins/installed entries so the UI can show why a plugin isn't running instead of just loaded:false. - Add a "Plugin Health" panel to the Tools page (circuit status, avg/max update time, update count, last error) plus PluginAPI.getPluginMetrics(). Tests: route-level tests for the health/metrics endpoints in test_web_api. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq * fix(plugin-metrics): refresh cross-process health/metrics reads; type hints Addresses CodeRabbit review on #388: - Major: the web process's health/resource trackers cached the first persisted read in an in-memory dict (and the CacheManager memory tier held max_age=None entries indefinitely), so a long-lived web process showed the first snapshot and never reflected the display service's later updates. Add an opt-in force_reload path (get_health_summary/get_health_state/_load_health_state and get_metrics_summary/get_metrics) that bypasses the in-memory copy and, via a new memory_ttl passthrough on CacheManager.get, the cache manager's memory tier — so each /plugins/health and /plugins/metrics poll reads fresh persisted state. Default behaviour (force_reload=False) is unchanged for the display process and existing callers. - Minor: DiskCache.get type hint is now Optional[int] with the None ("never expires") semantics documented, matching MemoryCache.get. Tests: new force_reload staleness cases in test_plugin_health and test_resource_monitor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-09 07:54:18 -04:00