Files
LEDMatrix/test/test_resource_monitor.py
T
3b93024993 feat: activate dormant plugin health/metrics subsystem and surface it in the web UI (#388)
* feat(plugin-system): activate dormant plugin health & metrics subsystem

PluginManager shipped a fully-built health tracker, resource monitor and
circuit breaker that were never instantiated (health_tracker/resource_monitor
were left as None), so the circuit breaker never engaged and the existing
health/metrics API routes always returned "not available".

- DisplayController now wires a PluginHealthTracker and PluginResourceMonitor
  onto the plugin manager, enabling the circuit breaker (a repeatedly-failing
  plugin's update() is skipped after consecutive failures, then retried after
  a cooldown) and per-plugin execution-time metrics. Both persist to the
  shared cache.
- load_plugin() now validates each plugin's config against its JSON schema in
  a strictly warn/degrade-only way: a violation logs a warning and flags the
  plugin degraded in the health tracker, but never changes whether the plugin
  loads or its pass/fail behaviour. Adds PluginHealthTracker.set_degraded(),
  which never touches the circuit breaker.
- ResourceMonitor CPU/memory sampling now reuses a cached psutil.Process and
  reads cpu_percent(interval=None), so monitoring no longer blocks ~100ms per
  call on the display loop's update path.
- Fix DiskCache.get() raising TypeError for max_age=None ("never expires"),
  which silently discarded persisted plugin health/metrics on read and thus
  broke cross-process and post-restart surfacing.
- Fix two dead PluginManager helpers that called non-existent tracker methods.

Tests: new test_resource_monitor, test_plugin_health,
test_plugin_manager_schema_soft; extended test_cache_manager and
test_display_controller.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* feat(web-ui): surface plugin health, metrics and load state

With the health/metrics subsystem now active in the display service, expose it
in the web UI (which runs as a separate process from the display loop):

- Wire a health tracker / resource monitor backed by the shared on-disk cache
  into the web process so /api/v3/plugins/health and /plugins/metrics read the
  data the display service persists.
- Build those route responses per installed plugin id (the tracker's in-memory
  view is empty in a fresh web process) so cross-process data is included.
- Add state + error_info to /plugins/installed entries so the UI can show why a
  plugin isn't running instead of just loaded:false.
- Add a "Plugin Health" panel to the Tools page (circuit status, avg/max update
  time, update count, last error) plus PluginAPI.getPluginMetrics().

Tests: route-level tests for the health/metrics endpoints in test_web_api.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* fix(plugin-metrics): refresh cross-process health/metrics reads; type hints

Addresses CodeRabbit review on #388:

- Major: the web process's health/resource trackers cached the first persisted
  read in an in-memory dict (and the CacheManager memory tier held max_age=None
  entries indefinitely), so a long-lived web process showed the first snapshot
  and never reflected the display service's later updates. Add an opt-in
  force_reload path (get_health_summary/get_health_state/_load_health_state and
  get_metrics_summary/get_metrics) that bypasses the in-memory copy and, via a
  new memory_ttl passthrough on CacheManager.get, the cache manager's memory
  tier — so each /plugins/health and /plugins/metrics poll reads fresh persisted
  state. Default behaviour (force_reload=False) is unchanged for the display
  process and existing callers.
- Minor: DiskCache.get type hint is now Optional[int] with the None ("never
  expires") semantics documented, matching MemoryCache.get.

Tests: new force_reload staleness cases in test_plugin_health and
test_resource_monitor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-09 07:54:18 -04:00

130 lines
4.8 KiB
Python

"""
Tests for src/plugin_system/resource_monitor.py
Focus areas:
- Execution-time metrics are captured regardless of psutil availability.
- CPU sampling is non-blocking (regression guard for the previous
``cpu_percent(interval=0.1)`` call that blocked 100 ms per monitored call).
- Resource limits are enforced.
"""
import time
import pytest
from unittest.mock import MagicMock
from src.plugin_system.resource_monitor import (
PluginResourceMonitor,
ResourceLimits,
ResourceLimitExceeded,
PSUTIL_AVAILABLE,
)
def _cache():
cache = MagicMock()
cache.get.return_value = None
return cache
class TestExecutionTimeMetrics:
def test_monitor_call_returns_value_and_records_call(self):
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
result = mon.monitor_call("p", lambda: 42)
assert result == 42
metrics = mon.get_metrics("p")
assert metrics.call_count == 1
assert metrics.total_execution_time >= 0.0
def test_avg_and_max_execution_time(self):
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
mon.monitor_call("p", lambda: time.sleep(0.01))
mon.monitor_call("p", lambda: None)
summary = mon.get_metrics_summary("p")
assert summary["call_count"] == 2
assert summary["max_execution_time"] >= summary["avg_execution_time"] >= 0.0
def test_exception_propagates_but_is_still_timed(self):
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
def boom():
raise ValueError("nope")
with pytest.raises(ValueError):
mon.monitor_call("p", boom)
# Execution time is still recorded even when the call raised.
assert mon.get_metrics("p").execution_time >= 0.0
class TestNonBlockingCpu:
def test_cpu_sampling_is_fast_when_disabled(self):
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
start = time.time()
for _ in range(50):
mon._get_process_cpu_percent()
# The old implementation blocked ~0.1s/call (~5s for 50). Non-blocking
# must complete near-instantly.
assert time.time() - start < 0.5
assert mon._get_process_cpu_percent() == 0.0
@pytest.mark.skipif(not PSUTIL_AVAILABLE, reason="psutil not installed")
def test_cpu_sampling_is_fast_with_psutil(self):
mon = PluginResourceMonitor(_cache(), enable_monitoring=True)
assert mon._process is not None
start = time.time()
for _ in range(30):
mon._get_process_cpu_percent()
# 30 blocking 0.1s samples would be ~3s; non-blocking must be well under.
assert time.time() - start < 0.5
def test_monitor_call_does_not_block_on_cpu_sampling(self):
mon = PluginResourceMonitor(_cache()) # enable depends on psutil
start = time.time()
for _ in range(25):
mon.monitor_call("p", lambda: None)
# 25 * 0.1s = 2.5s under the old blocking bug; must be far faster now.
assert time.time() - start < 1.0
class TestResourceLimits:
def test_execution_time_limit_raises(self):
mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
mon.set_limits("p", ResourceLimits(max_execution_time=0.001))
with pytest.raises(ResourceLimitExceeded):
mon.monitor_call("p", lambda: time.sleep(0.02))
def test_reset_metrics_clears_counts(self):
cache = _cache()
mon = PluginResourceMonitor(cache, enable_monitoring=False)
mon.monitor_call("p", lambda: None)
assert mon.get_metrics("p").call_count == 1
mon.reset_metrics("p")
assert mon.get_metrics("p").call_count == 0
class TestForceReload:
def test_force_reload_refreshes_stale_snapshot(self):
"""A read-only consumer must see the writer process's latest persisted
metrics rather than a pinned first snapshot."""
cache = MagicMock()
persisted = {"value": None} # only the metrics key returns data
def cache_get(key, max_age=None, memory_ttl=None):
return persisted["value"] if key.startswith("plugin_metrics:") else None
cache.get.side_effect = cache_get
mon = PluginResourceMonitor(cache, enable_monitoring=False)
# First read snapshots empty metrics.
assert mon.get_metrics_summary("p")["call_count"] == 0
# The display service later persists real metrics.
persisted["value"] = {"call_count": 7, "total_execution_time": 1.4}
# Plain read stays stale...
assert mon.get_metrics_summary("p")["call_count"] == 0
# ...force_reload picks up the persisted values and bypasses memory.
fresh = mon.get_metrics_summary("p", force_reload=True)
assert fresh["call_count"] == 7
assert any(c.kwargs.get("memory_ttl") == 0 for c in cache.get.call_args_list)