feat: activate dormant plugin health/metrics subsystem and surface it in the web UI (#388)

* feat(plugin-system): activate dormant plugin health & metrics subsystem

PluginManager shipped a fully-built health tracker, resource monitor and
circuit breaker that were never instantiated (health_tracker/resource_monitor
were left as None), so the circuit breaker never engaged and the existing
health/metrics API routes always returned "not available".

- DisplayController now wires a PluginHealthTracker and PluginResourceMonitor
  onto the plugin manager, enabling the circuit breaker (a repeatedly-failing
  plugin's update() is skipped after consecutive failures, then retried after
  a cooldown) and per-plugin execution-time metrics. Both persist to the
  shared cache.
- load_plugin() now validates each plugin's config against its JSON schema in
  a strictly warn/degrade-only way: a violation logs a warning and flags the
  plugin degraded in the health tracker, but never changes whether the plugin
  loads or its pass/fail behaviour. Adds PluginHealthTracker.set_degraded(),
  which never touches the circuit breaker.
- ResourceMonitor CPU/memory sampling now reuses a cached psutil.Process and
  reads cpu_percent(interval=None), so monitoring no longer blocks ~100ms per
  call on the display loop's update path.
- Fix DiskCache.get() raising TypeError for max_age=None ("never expires"),
  which silently discarded persisted plugin health/metrics on read and thus
  broke cross-process and post-restart surfacing.
- Fix two dead PluginManager helpers that called non-existent tracker methods.

Tests: new test_resource_monitor, test_plugin_health,
test_plugin_manager_schema_soft; extended test_cache_manager and
test_display_controller.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* feat(web-ui): surface plugin health, metrics and load state

With the health/metrics subsystem now active in the display service, expose it
in the web UI (which runs as a separate process from the display loop):

- Wire a health tracker / resource monitor backed by the shared on-disk cache
  into the web process so /api/v3/plugins/health and /plugins/metrics read the
  data the display service persists.
- Build those route responses per installed plugin id (the tracker's in-memory
  view is empty in a fresh web process) so cross-process data is included.
- Add state + error_info to /plugins/installed entries so the UI can show why a
  plugin isn't running instead of just loaded:false.
- Add a "Plugin Health" panel to the Tools page (circuit status, avg/max update
  time, update count, last error) plus PluginAPI.getPluginMetrics().

Tests: route-level tests for the health/metrics endpoints in test_web_api.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* fix(plugin-metrics): refresh cross-process health/metrics reads; type hints

Addresses CodeRabbit review on #388:

- Major: the web process's health/resource trackers cached the first persisted
  read in an in-memory dict (and the CacheManager memory tier held max_age=None
  entries indefinitely), so a long-lived web process showed the first snapshot
  and never reflected the display service's later updates. Add an opt-in
  force_reload path (get_health_summary/get_health_state/_load_health_state and
  get_metrics_summary/get_metrics) that bypasses the in-memory copy and, via a
  new memory_ttl passthrough on CacheManager.get, the cache manager's memory
  tier — so each /plugins/health and /plugins/metrics poll reads fresh persisted
  state. Default behaviour (force_reload=False) is unchanged for the display
  process and existing callers.
- Minor: DiskCache.get type hint is now Optional[int] with the None ("never
  expires") semantics documented, matching MemoryCache.get.

Tests: new force_reload staleness cases in test_plugin_health and
test_resource_monitor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

---------

Co-authored-by: Claude <noreply@anthropic.com>

src/plugin_system/plugin_health.py

@@ -52,11 +52,18 @@ class PluginHealthTracker:
         """Get cache key for plugin health data."""
         return f"plugin_health:{plugin_id}"
     
-    def _load_health_state(self, plugin_id: str) -> Dict[str, Any]:
-        """Load health state from cache or return defaults."""
+    def _load_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]:
+        """Load health state from cache or return defaults.
+
+        ``force_reload=True`` bypasses the cache manager's in-memory tier so a
+        read-only consumer (e.g. the web process) observes the writer process's
+        latest persisted state instead of a stale first snapshot.
+        """
         cache_key = self._get_health_key(plugin_id)
-        cached = self.cache_manager.get(cache_key, max_age=None)
-        
+        cached = self.cache_manager.get(
+            cache_key, max_age=None, memory_ttl=0 if force_reload else None
+        )
+
         if cached:
             return cached
         
@@ -79,10 +86,17 @@ class PluginHealthTracker:
         self.cache_manager.set(cache_key, state)  # Persist indefinitely
         self._health_state[plugin_id] = state
     
-    def get_health_state(self, plugin_id: str) -> Dict[str, Any]:
-        """Get current health state for a plugin."""
-        if plugin_id not in self._health_state:
-            self._health_state[plugin_id] = self._load_health_state(plugin_id)
+    def get_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]:
+        """Get current health state for a plugin.
+
+        ``force_reload=True`` re-reads the persisted state from the cache,
+        bypassing the in-memory copy — needed by cross-process readers that
+        would otherwise be pinned to the first snapshot they loaded.
+        """
+        if force_reload or plugin_id not in self._health_state:
+            self._health_state[plugin_id] = self._load_health_state(
+                plugin_id, force_reload=force_reload
+            )
         return self._health_state[plugin_id]
     
     def record_success(self, plugin_id: str) -> None:
@@ -139,6 +153,28 @@ class PluginHealthTracker:
         
         self._save_health_state(plugin_id, state)
     
+    def set_degraded(self, plugin_id: str, reason: Optional[str]) -> None:
+        """Flag (or clear) a plugin as degraded without touching the circuit breaker.
+
+        Used for non-fatal issues — e.g. a config that no longer satisfies the
+        plugin's schema — that should be surfaced to the user but must NOT cause
+        the plugin to be skipped or counted as a runtime failure. Passing
+        ``reason=None`` clears the flag. The write is skipped when nothing
+        actually changes, so calling this on every load is cheap.
+
+        Args:
+            plugin_id: Plugin identifier
+            reason: Human-readable reason string, or None to clear the flag
+        """
+        state = self.get_health_state(plugin_id)
+        new_degraded = bool(reason)
+        new_reason = reason if reason else None
+        if state.get('degraded', False) == new_degraded and state.get('degraded_reason') == new_reason:
+            return  # No change — avoid a redundant cache write
+        state['degraded'] = new_degraded
+        state['degraded_reason'] = new_reason
+        self._save_health_state(plugin_id, state)
+
     def should_skip_plugin(self, plugin_id: str) -> bool:
         """
         Check if plugin should be skipped due to circuit breaker.
@@ -181,9 +217,13 @@ class PluginHealthTracker:
         
         return False
     
-    def get_health_summary(self, plugin_id: str) -> Dict[str, Any]:
-        """Get health summary for a plugin."""
-        state = self.get_health_state(plugin_id)
+    def get_health_summary(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]:
+        """Get health summary for a plugin.
+
+        ``force_reload=True`` refreshes from the persisted cache first so
+        cross-process readers reflect the writer's latest state.
+        """
+        state = self.get_health_state(plugin_id, force_reload=force_reload)
         
         total_calls = state.get('total_successes', 0) + state.get('total_failures', 0)
         success_rate = 0.0
@@ -201,6 +241,8 @@ class PluginHealthTracker:
             'last_failure_time': state.get('last_failure_time'),
             'last_error': state.get('last_error'),
             'is_healthy': state.get('circuit_state') == CircuitState.CLOSED.value,
+            'degraded': state.get('degraded', False),
+            'degraded_reason': state.get('degraded_reason'),
             'circuit_opened_time': state.get('circuit_opened_time'),
             'half_open_start_time': state.get('half_open_start_time')
         }