mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-08-01 16:58:06 +00:00
3b93024993
* feat(plugin-system): activate dormant plugin health & metrics subsystem
PluginManager shipped a fully-built health tracker, resource monitor and
circuit breaker that were never instantiated (health_tracker/resource_monitor
were left as None), so the circuit breaker never engaged and the existing
health/metrics API routes always returned "not available".
- DisplayController now wires a PluginHealthTracker and PluginResourceMonitor
onto the plugin manager, enabling the circuit breaker (a repeatedly-failing
plugin's update() is skipped after consecutive failures, then retried after
a cooldown) and per-plugin execution-time metrics. Both persist to the
shared cache.
- load_plugin() now validates each plugin's config against its JSON schema in
a strictly warn/degrade-only way: a violation logs a warning and flags the
plugin degraded in the health tracker, but never changes whether the plugin
loads or its pass/fail behaviour. Adds PluginHealthTracker.set_degraded(),
which never touches the circuit breaker.
- ResourceMonitor CPU/memory sampling now reuses a cached psutil.Process and
reads cpu_percent(interval=None), so monitoring no longer blocks ~100ms per
call on the display loop's update path.
- Fix DiskCache.get() raising TypeError for max_age=None ("never expires"),
which silently discarded persisted plugin health/metrics on read and thus
broke cross-process and post-restart surfacing.
- Fix two dead PluginManager helpers that called non-existent tracker methods.
Tests: new test_resource_monitor, test_plugin_health,
test_plugin_manager_schema_soft; extended test_cache_manager and
test_display_controller.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq
* feat(web-ui): surface plugin health, metrics and load state
With the health/metrics subsystem now active in the display service, expose it
in the web UI (which runs as a separate process from the display loop):
- Wire a health tracker / resource monitor backed by the shared on-disk cache
into the web process so /api/v3/plugins/health and /plugins/metrics read the
data the display service persists.
- Build those route responses per installed plugin id (the tracker's in-memory
view is empty in a fresh web process) so cross-process data is included.
- Add state + error_info to /plugins/installed entries so the UI can show why a
plugin isn't running instead of just loaded:false.
- Add a "Plugin Health" panel to the Tools page (circuit status, avg/max update
time, update count, last error) plus PluginAPI.getPluginMetrics().
Tests: route-level tests for the health/metrics endpoints in test_web_api.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq
* fix(plugin-metrics): refresh cross-process health/metrics reads; type hints
Addresses CodeRabbit review on #388:
- Major: the web process's health/resource trackers cached the first persisted
read in an in-memory dict (and the CacheManager memory tier held max_age=None
entries indefinitely), so a long-lived web process showed the first snapshot
and never reflected the display service's later updates. Add an opt-in
force_reload path (get_health_summary/get_health_state/_load_health_state and
get_metrics_summary/get_metrics) that bypasses the in-memory copy and, via a
new memory_ttl passthrough on CacheManager.get, the cache manager's memory
tier — so each /plugins/health and /plugins/metrics poll reads fresh persisted
state. Default behaviour (force_reload=False) is unchanged for the display
process and existing callers.
- Minor: DiskCache.get type hint is now Optional[int] with the None ("never
expires") semantics documented, matching MemoryCache.get.
Tests: new force_reload staleness cases in test_plugin_health and
test_resource_monitor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq
---------
Co-authored-by: Claude <noreply@anthropic.com>
94 lines
3.3 KiB
Python
94 lines
3.3 KiB
Python
"""
|
|
Tests for src/plugin_system/plugin_health.py
|
|
|
|
Focus on the additive ``set_degraded`` mechanism used by the warn-only schema
|
|
validation path: it must surface a degraded reason without touching the circuit
|
|
breaker or causing the plugin to be skipped.
|
|
"""
|
|
|
|
from unittest.mock import MagicMock
|
|
|
|
from src.plugin_system.plugin_health import PluginHealthTracker, CircuitState
|
|
|
|
|
|
def _cache():
|
|
cache = MagicMock()
|
|
cache.get.return_value = None
|
|
return cache
|
|
|
|
|
|
def test_set_degraded_marks_and_surfaces_reason():
|
|
tracker = PluginHealthTracker(_cache())
|
|
tracker.set_degraded("p", "bad config")
|
|
summary = tracker.get_health_summary("p")
|
|
assert summary["degraded"] is True
|
|
assert summary["degraded_reason"] == "bad config"
|
|
|
|
|
|
def test_set_degraded_none_clears():
|
|
tracker = PluginHealthTracker(_cache())
|
|
tracker.set_degraded("p", "bad config")
|
|
tracker.set_degraded("p", None)
|
|
summary = tracker.get_health_summary("p")
|
|
assert summary["degraded"] is False
|
|
assert summary["degraded_reason"] is None
|
|
|
|
|
|
def test_set_degraded_does_not_affect_circuit_breaker():
|
|
tracker = PluginHealthTracker(_cache())
|
|
tracker.set_degraded("p", "bad config")
|
|
summary = tracker.get_health_summary("p")
|
|
# Degraded is a *separate* signal from circuit health: the plugin is not
|
|
# counted as failing, the circuit stays closed, and it is not skipped.
|
|
assert summary["circuit_state"] == CircuitState.CLOSED.value
|
|
assert summary["consecutive_failures"] == 0
|
|
assert summary["is_healthy"] is True
|
|
assert tracker.should_skip_plugin("p") is False
|
|
|
|
|
|
def test_set_degraded_skips_redundant_cache_write():
|
|
cache = _cache()
|
|
tracker = PluginHealthTracker(cache)
|
|
tracker.set_degraded("p", "x")
|
|
writes_after_first = cache.set.call_count
|
|
assert writes_after_first >= 1
|
|
tracker.set_degraded("p", "x") # unchanged → no extra write
|
|
assert cache.set.call_count == writes_after_first
|
|
|
|
|
|
def test_default_summary_has_degraded_fields():
|
|
tracker = PluginHealthTracker(_cache())
|
|
summary = tracker.get_health_summary("never-seen")
|
|
assert summary["degraded"] is False
|
|
assert summary["degraded_reason"] is None
|
|
|
|
|
|
def test_force_reload_refreshes_stale_in_memory_snapshot():
|
|
"""A long-lived reader (e.g. the web process) must not be pinned to the
|
|
first snapshot: force_reload re-reads persisted state and bypasses the
|
|
cache manager's memory tier so cross-process updates are visible."""
|
|
cache = _cache()
|
|
tracker = PluginHealthTracker(cache)
|
|
|
|
# First read snapshots an empty (healthy) state into the in-memory copy.
|
|
assert tracker.get_health_summary("p")["consecutive_failures"] == 0
|
|
|
|
# The display service later persists a failing/open state.
|
|
cache.get.return_value = {
|
|
"consecutive_failures": 5,
|
|
"circuit_state": "open",
|
|
"total_failures": 5,
|
|
"total_successes": 0,
|
|
}
|
|
|
|
# A plain read is still pinned to the stale snapshot...
|
|
assert tracker.get_health_summary("p")["consecutive_failures"] == 0
|
|
|
|
# ...but force_reload observes the new persisted state.
|
|
fresh = tracker.get_health_summary("p", force_reload=True)
|
|
assert fresh["consecutive_failures"] == 5
|
|
assert fresh["circuit_state"] == "open"
|
|
|
|
# and it asked the cache to bypass the in-memory tier (memory_ttl=0).
|
|
assert any(c.kwargs.get("memory_ttl") == 0 for c in cache.get.call_args_list)
|