feat: activate dormant plugin health/metrics subsystem and surface it in the web UI (#388)

* feat(plugin-system): activate dormant plugin health & metrics subsystem

PluginManager shipped a fully-built health tracker, resource monitor and
circuit breaker that were never instantiated (health_tracker/resource_monitor
were left as None), so the circuit breaker never engaged and the existing
health/metrics API routes always returned "not available".

- DisplayController now wires a PluginHealthTracker and PluginResourceMonitor
  onto the plugin manager, enabling the circuit breaker (a repeatedly-failing
  plugin's update() is skipped after consecutive failures, then retried after
  a cooldown) and per-plugin execution-time metrics. Both persist to the
  shared cache.
- load_plugin() now validates each plugin's config against its JSON schema in
  a strictly warn/degrade-only way: a violation logs a warning and flags the
  plugin degraded in the health tracker, but never changes whether the plugin
  loads or its pass/fail behaviour. Adds PluginHealthTracker.set_degraded(),
  which never touches the circuit breaker.
- ResourceMonitor CPU/memory sampling now reuses a cached psutil.Process and
  reads cpu_percent(interval=None), so monitoring no longer blocks ~100ms per
  call on the display loop's update path.
- Fix DiskCache.get() raising TypeError for max_age=None ("never expires"),
  which silently discarded persisted plugin health/metrics on read and thus
  broke cross-process and post-restart surfacing.
- Fix two dead PluginManager helpers that called non-existent tracker methods.

Tests: new test_resource_monitor, test_plugin_health,
test_plugin_manager_schema_soft; extended test_cache_manager and
test_display_controller.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* feat(web-ui): surface plugin health, metrics and load state

With the health/metrics subsystem now active in the display service, expose it
in the web UI (which runs as a separate process from the display loop):

- Wire a health tracker / resource monitor backed by the shared on-disk cache
  into the web process so /api/v3/plugins/health and /plugins/metrics read the
  data the display service persists.
- Build those route responses per installed plugin id (the tracker's in-memory
  view is empty in a fresh web process) so cross-process data is included.
- Add state + error_info to /plugins/installed entries so the UI can show why a
  plugin isn't running instead of just loaded:false.
- Add a "Plugin Health" panel to the Tools page (circuit status, avg/max update
  time, update count, last error) plus PluginAPI.getPluginMetrics().

Tests: route-level tests for the health/metrics endpoints in test_web_api.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* fix(plugin-metrics): refresh cross-process health/metrics reads; type hints

Addresses CodeRabbit review on #388:

- Major: the web process's health/resource trackers cached the first persisted
  read in an in-memory dict (and the CacheManager memory tier held max_age=None
  entries indefinitely), so a long-lived web process showed the first snapshot
  and never reflected the display service's later updates. Add an opt-in
  force_reload path (get_health_summary/get_health_state/_load_health_state and
  get_metrics_summary/get_metrics) that bypasses the in-memory copy and, via a
  new memory_ttl passthrough on CacheManager.get, the cache manager's memory
  tier — so each /plugins/health and /plugins/metrics poll reads fresh persisted
  state. Default behaviour (force_reload=False) is unchanged for the display
  process and existing callers.
- Minor: DiskCache.get type hint is now Optional[int] with the None ("never
  expires") semantics documented, matching MemoryCache.get.

Tests: new force_reload staleness cases in test_plugin_health and
test_resource_monitor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

---------

Co-authored-by: Claude <noreply@anthropic.com>

test/test_cache_manager.py

@@ -279,10 +279,23 @@ class TestDiskCache:
         """Test getting expired cache entry."""
         cache = DiskCache(cache_dir=str(tmp_path))
         cache.set("test_key", {"data": "value"})
-        
+
         # Get with max_age=0 to force expiration
         result = cache.get("test_key", max_age=0)
         assert result is None
+
+    def test_get_max_age_none_never_expires(self, tmp_path):
+        """max_age=None must return persisted records regardless of age.
+
+        Regression: the age comparison raised TypeError for max_age=None,
+        which was swallowed and treated as a miss — silently breaking
+        long-lived state (plugin health/metrics) read across processes.
+        """
+        cache = DiskCache(cache_dir=str(tmp_path))
+        cache.set("test_key", {"data": "value", "timestamp": 0})  # epoch → very old
+        result = cache.get("test_key", max_age=None)
+        assert result is not None
+        assert result["data"] == "value"
     
     def test_get_nonexistent(self, tmp_path):
         """Test getting non-existent key."""

test/test_display_controller.py

@@ -391,3 +391,16 @@ class TestDisplayControllerSchedule:
 
             controller._check_schedule()
             assert controller.is_display_active is False
+
+
+class TestPluginHealthWiring:
+    """Phase 1: DisplayController activates the dormant plugin health/metrics
+    subsystem by wiring real tracker/monitor instances onto the plugin manager."""
+
+    def test_health_tracker_and_resource_monitor_wired(self, test_display_controller):
+        from src.plugin_system.plugin_health import PluginHealthTracker
+        from src.plugin_system.resource_monitor import PluginResourceMonitor
+
+        pm = test_display_controller.plugin_manager
+        assert isinstance(pm.health_tracker, PluginHealthTracker)
+        assert isinstance(pm.resource_monitor, PluginResourceMonitor)

test/test_plugin_health.py

@@ -0,0 +1,93 @@
+"""
+Tests for src/plugin_system/plugin_health.py
+
+Focus on the additive ``set_degraded`` mechanism used by the warn-only schema
+validation path: it must surface a degraded reason without touching the circuit
+breaker or causing the plugin to be skipped.
+"""
+
+from unittest.mock import MagicMock
+
+from src.plugin_system.plugin_health import PluginHealthTracker, CircuitState
+
+
+def _cache():
+    cache = MagicMock()
+    cache.get.return_value = None
+    return cache
+
+
+def test_set_degraded_marks_and_surfaces_reason():
+    tracker = PluginHealthTracker(_cache())
+    tracker.set_degraded("p", "bad config")
+    summary = tracker.get_health_summary("p")
+    assert summary["degraded"] is True
+    assert summary["degraded_reason"] == "bad config"
+
+
+def test_set_degraded_none_clears():
+    tracker = PluginHealthTracker(_cache())
+    tracker.set_degraded("p", "bad config")
+    tracker.set_degraded("p", None)
+    summary = tracker.get_health_summary("p")
+    assert summary["degraded"] is False
+    assert summary["degraded_reason"] is None
+
+
+def test_set_degraded_does_not_affect_circuit_breaker():
+    tracker = PluginHealthTracker(_cache())
+    tracker.set_degraded("p", "bad config")
+    summary = tracker.get_health_summary("p")
+    # Degraded is a *separate* signal from circuit health: the plugin is not
+    # counted as failing, the circuit stays closed, and it is not skipped.
+    assert summary["circuit_state"] == CircuitState.CLOSED.value
+    assert summary["consecutive_failures"] == 0
+    assert summary["is_healthy"] is True
+    assert tracker.should_skip_plugin("p") is False
+
+
+def test_set_degraded_skips_redundant_cache_write():
+    cache = _cache()
+    tracker = PluginHealthTracker(cache)
+    tracker.set_degraded("p", "x")
+    writes_after_first = cache.set.call_count
+    assert writes_after_first >= 1
+    tracker.set_degraded("p", "x")  # unchanged → no extra write
+    assert cache.set.call_count == writes_after_first
+
+
+def test_default_summary_has_degraded_fields():
+    tracker = PluginHealthTracker(_cache())
+    summary = tracker.get_health_summary("never-seen")
+    assert summary["degraded"] is False
+    assert summary["degraded_reason"] is None
+
+
+def test_force_reload_refreshes_stale_in_memory_snapshot():
+    """A long-lived reader (e.g. the web process) must not be pinned to the
+    first snapshot: force_reload re-reads persisted state and bypasses the
+    cache manager's memory tier so cross-process updates are visible."""
+    cache = _cache()
+    tracker = PluginHealthTracker(cache)
+
+    # First read snapshots an empty (healthy) state into the in-memory copy.
+    assert tracker.get_health_summary("p")["consecutive_failures"] == 0
+
+    # The display service later persists a failing/open state.
+    cache.get.return_value = {
+        "consecutive_failures": 5,
+        "circuit_state": "open",
+        "total_failures": 5,
+        "total_successes": 0,
+    }
+
+    # A plain read is still pinned to the stale snapshot...
+    assert tracker.get_health_summary("p")["consecutive_failures"] == 0
+
+    # ...but force_reload observes the new persisted state.
+    fresh = tracker.get_health_summary("p", force_reload=True)
+    assert fresh["consecutive_failures"] == 5
+    assert fresh["circuit_state"] == "open"
+
+    # and it asked the cache to bypass the in-memory tier (memory_ttl=0).
+    assert any(c.kwargs.get("memory_ttl") == 0 for c in cache.get.call_args_list)

test/test_plugin_manager_schema_soft.py

@@ -0,0 +1,79 @@
+"""
+Tests for PluginManager._validate_config_schema_soft (Phase 1, warn-only schema
+validation).
+
+Contract:
+- A schema violation logs a warning and marks the plugin degraded in the health
+  tracker, but never raises and never changes load pass/fail behaviour.
+- A valid config (or no schema) clears any stale degraded flag.
+- The method is safe when no health tracker is wired.
+"""
+
+import tempfile
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import pytest
+
+from src.plugin_system.plugin_manager import PluginManager
+
+
+@pytest.fixture
+def pm():
+    with tempfile.TemporaryDirectory() as tmp:
+        manager = PluginManager(plugins_dir=str(Path(tmp) / "plugins"))
+        manager.schema_manager = MagicMock()
+        yield manager
+
+
+def test_invalid_config_marks_degraded_without_raising(pm):
+    pm.health_tracker = MagicMock()
+    pm.schema_manager.load_schema.return_value = {"type": "object"}
+    pm.schema_manager.validate_config_against_schema.return_value = (
+        False,
+        ["Missing required field: 'api_key'"],
+    )
+
+    pm._validate_config_schema_soft("youtube-stats", {})
+
+    pm.health_tracker.set_degraded.assert_called_once()
+    plugin_id, reason = pm.health_tracker.set_degraded.call_args[0]
+    assert plugin_id == "youtube-stats"
+    assert "api_key" in reason
+
+
+def test_valid_config_clears_degraded(pm):
+    pm.health_tracker = MagicMock()
+    pm.schema_manager.load_schema.return_value = {"type": "object"}
+    pm.schema_manager.validate_config_against_schema.return_value = (True, [])
+
+    pm._validate_config_schema_soft("p", {"api_key": "x"})
+
+    pm.health_tracker.set_degraded.assert_called_once_with("p", None)
+
+
+def test_no_schema_clears_degraded(pm):
+    pm.health_tracker = MagicMock()
+    pm.schema_manager.load_schema.return_value = None
+
+    pm._validate_config_schema_soft("p", {})
+
+    pm.health_tracker.set_degraded.assert_called_once_with("p", None)
+
+
+def test_validation_exception_is_swallowed(pm):
+    pm.health_tracker = MagicMock()
+    pm.schema_manager.load_schema.return_value = {"type": "object"}
+    pm.schema_manager.validate_config_against_schema.side_effect = RuntimeError("boom")
+
+    # Must not raise — the validation machinery failing must never break loading.
+    pm._validate_config_schema_soft("p", {})
+
+
+def test_safe_without_health_tracker(pm):
+    pm.health_tracker = None
+    pm.schema_manager.load_schema.return_value = {"type": "object"}
+    pm.schema_manager.validate_config_against_schema.return_value = (False, ["err"])
+
+    # Must not raise even though there is no tracker to record against.
+    pm._validate_config_schema_soft("p", {})

test/test_resource_monitor.py

@@ -0,0 +1,129 @@
+"""
+Tests for src/plugin_system/resource_monitor.py
+
+Focus areas:
+- Execution-time metrics are captured regardless of psutil availability.
+- CPU sampling is non-blocking (regression guard for the previous
+  ``cpu_percent(interval=0.1)`` call that blocked 100 ms per monitored call).
+- Resource limits are enforced.
+"""
+
+import time
+
+import pytest
+from unittest.mock import MagicMock
+
+from src.plugin_system.resource_monitor import (
+    PluginResourceMonitor,
+    ResourceLimits,
+    ResourceLimitExceeded,
+    PSUTIL_AVAILABLE,
+)
+
+
+def _cache():
+    cache = MagicMock()
+    cache.get.return_value = None
+    return cache
+
+
+class TestExecutionTimeMetrics:
+    def test_monitor_call_returns_value_and_records_call(self):
+        mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
+        result = mon.monitor_call("p", lambda: 42)
+        assert result == 42
+        metrics = mon.get_metrics("p")
+        assert metrics.call_count == 1
+        assert metrics.total_execution_time >= 0.0
+
+    def test_avg_and_max_execution_time(self):
+        mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
+        mon.monitor_call("p", lambda: time.sleep(0.01))
+        mon.monitor_call("p", lambda: None)
+        summary = mon.get_metrics_summary("p")
+        assert summary["call_count"] == 2
+        assert summary["max_execution_time"] >= summary["avg_execution_time"] >= 0.0
+
+    def test_exception_propagates_but_is_still_timed(self):
+        mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
+
+        def boom():
+            raise ValueError("nope")
+
+        with pytest.raises(ValueError):
+            mon.monitor_call("p", boom)
+        # Execution time is still recorded even when the call raised.
+        assert mon.get_metrics("p").execution_time >= 0.0
+
+
+class TestNonBlockingCpu:
+    def test_cpu_sampling_is_fast_when_disabled(self):
+        mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
+        start = time.time()
+        for _ in range(50):
+            mon._get_process_cpu_percent()
+        # The old implementation blocked ~0.1s/call (~5s for 50). Non-blocking
+        # must complete near-instantly.
+        assert time.time() - start < 0.5
+        assert mon._get_process_cpu_percent() == 0.0
+
+    @pytest.mark.skipif(not PSUTIL_AVAILABLE, reason="psutil not installed")
+    def test_cpu_sampling_is_fast_with_psutil(self):
+        mon = PluginResourceMonitor(_cache(), enable_monitoring=True)
+        assert mon._process is not None
+        start = time.time()
+        for _ in range(30):
+            mon._get_process_cpu_percent()
+        # 30 blocking 0.1s samples would be ~3s; non-blocking must be well under.
+        assert time.time() - start < 0.5
+
+    def test_monitor_call_does_not_block_on_cpu_sampling(self):
+        mon = PluginResourceMonitor(_cache())  # enable depends on psutil
+        start = time.time()
+        for _ in range(25):
+            mon.monitor_call("p", lambda: None)
+        # 25 * 0.1s = 2.5s under the old blocking bug; must be far faster now.
+        assert time.time() - start < 1.0
+
+
+class TestResourceLimits:
+    def test_execution_time_limit_raises(self):
+        mon = PluginResourceMonitor(_cache(), enable_monitoring=False)
+        mon.set_limits("p", ResourceLimits(max_execution_time=0.001))
+        with pytest.raises(ResourceLimitExceeded):
+            mon.monitor_call("p", lambda: time.sleep(0.02))
+
+    def test_reset_metrics_clears_counts(self):
+        cache = _cache()
+        mon = PluginResourceMonitor(cache, enable_monitoring=False)
+        mon.monitor_call("p", lambda: None)
+        assert mon.get_metrics("p").call_count == 1
+        mon.reset_metrics("p")
+        assert mon.get_metrics("p").call_count == 0
+
+
+class TestForceReload:
+    def test_force_reload_refreshes_stale_snapshot(self):
+        """A read-only consumer must see the writer process's latest persisted
+        metrics rather than a pinned first snapshot."""
+        cache = MagicMock()
+        persisted = {"value": None}  # only the metrics key returns data
+
+        def cache_get(key, max_age=None, memory_ttl=None):
+            return persisted["value"] if key.startswith("plugin_metrics:") else None
+
+        cache.get.side_effect = cache_get
+        mon = PluginResourceMonitor(cache, enable_monitoring=False)
+
+        # First read snapshots empty metrics.
+        assert mon.get_metrics_summary("p")["call_count"] == 0
+
+        # The display service later persists real metrics.
+        persisted["value"] = {"call_count": 7, "total_execution_time": 1.4}
+
+        # Plain read stays stale...
+        assert mon.get_metrics_summary("p")["call_count"] == 0
+        # ...force_reload picks up the persisted values and bypasses memory.
+        fresh = mon.get_metrics_summary("p", force_reload=True)
+        assert fresh["call_count"] == 7
+        assert any(c.kwargs.get("memory_ttl") == 0 for c in cache.get.call_args_list)

test/test_web_api.py

@@ -759,3 +759,54 @@ class TestDottedKeyNormalization:
         teams = soccer_cfg.get('leagues', {}).get('eng.1', {}).get('favorite_teams')
         assert isinstance(teams, list), f"Expected list, got: {type(teams)}"
         assert teams == [], f"Expected empty default list, got: {teams}"
+
+
+class TestPluginHealthRoutes:
+    """Phase 1: /plugins/health and /plugins/metrics build per-installed-id so
+    they surface cross-process data persisted by the display service."""
+
+    def test_health_route_builds_per_installed_id(self, client, mock_plugin_manager):
+        from web_interface.blueprints.api_v3 import api_v3
+        from src.plugin_system.plugin_health import PluginHealthTracker
+
+        cache = MagicMock()
+        cache.get.return_value = None
+        api_v3.plugin_manager = mock_plugin_manager
+        mock_plugin_manager.plugin_manifests = {'p1': {}, 'p2': {}}
+        mock_plugin_manager.health_tracker = PluginHealthTracker(cache)
+
+        resp = client.get('/api/v3/plugins/health')
+        assert resp.status_code == 200
+        data = resp.get_json()['data']
+        assert set(data.keys()) == {'p1', 'p2'}
+        assert data['p1']['is_healthy'] is True
+        assert data['p1']['degraded'] is False
+
+    def test_health_route_reports_not_available_without_tracker(self, client, mock_plugin_manager):
+        from web_interface.blueprints.api_v3 import api_v3
+        api_v3.plugin_manager = mock_plugin_manager
+        mock_plugin_manager.health_tracker = None
+
+        resp = client.get('/api/v3/plugins/health')
+        assert resp.status_code == 200
+        body = resp.get_json()
+        assert body['data'] == {}
+        assert 'not available' in body['message'].lower()
+
+    def test_metrics_route_builds_per_installed_id(self, client, mock_plugin_manager):
+        from web_interface.blueprints.api_v3 import api_v3
+        from src.plugin_system.resource_monitor import PluginResourceMonitor
+
+        cache = MagicMock()
+        cache.get.return_value = None
+        api_v3.plugin_manager = mock_plugin_manager
+        mock_plugin_manager.plugin_manifests = {'p1': {}}
+        mock_plugin_manager.resource_monitor = PluginResourceMonitor(
+            cache, enable_monitoring=False
+        )
+
+        resp = client.get('/api/v3/plugins/metrics')
+        assert resp.status_code == 200
+        data = resp.get_json()['data']
+        assert 'p1' in data
+        assert data['p1']['call_count'] == 0