feat: activate dormant plugin health/metrics subsystem and surface it in the web UI (#388)

* feat(plugin-system): activate dormant plugin health & metrics subsystem

PluginManager shipped a fully-built health tracker, resource monitor and
circuit breaker that were never instantiated (health_tracker/resource_monitor
were left as None), so the circuit breaker never engaged and the existing
health/metrics API routes always returned "not available".

- DisplayController now wires a PluginHealthTracker and PluginResourceMonitor
  onto the plugin manager, enabling the circuit breaker (a repeatedly-failing
  plugin's update() is skipped after consecutive failures, then retried after
  a cooldown) and per-plugin execution-time metrics. Both persist to the
  shared cache.
- load_plugin() now validates each plugin's config against its JSON schema in
  a strictly warn/degrade-only way: a violation logs a warning and flags the
  plugin degraded in the health tracker, but never changes whether the plugin
  loads or its pass/fail behaviour. Adds PluginHealthTracker.set_degraded(),
  which never touches the circuit breaker.
- ResourceMonitor CPU/memory sampling now reuses a cached psutil.Process and
  reads cpu_percent(interval=None), so monitoring no longer blocks ~100ms per
  call on the display loop's update path.
- Fix DiskCache.get() raising TypeError for max_age=None ("never expires"),
  which silently discarded persisted plugin health/metrics on read and thus
  broke cross-process and post-restart surfacing.
- Fix two dead PluginManager helpers that called non-existent tracker methods.

Tests: new test_resource_monitor, test_plugin_health,
test_plugin_manager_schema_soft; extended test_cache_manager and
test_display_controller.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* feat(web-ui): surface plugin health, metrics and load state

With the health/metrics subsystem now active in the display service, expose it
in the web UI (which runs as a separate process from the display loop):

- Wire a health tracker / resource monitor backed by the shared on-disk cache
  into the web process so /api/v3/plugins/health and /plugins/metrics read the
  data the display service persists.
- Build those route responses per installed plugin id (the tracker's in-memory
  view is empty in a fresh web process) so cross-process data is included.
- Add state + error_info to /plugins/installed entries so the UI can show why a
  plugin isn't running instead of just loaded:false.
- Add a "Plugin Health" panel to the Tools page (circuit status, avg/max update
  time, update count, last error) plus PluginAPI.getPluginMetrics().

Tests: route-level tests for the health/metrics endpoints in test_web_api.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

* fix(plugin-metrics): refresh cross-process health/metrics reads; type hints

Addresses CodeRabbit review on #388:

- Major: the web process's health/resource trackers cached the first persisted
  read in an in-memory dict (and the CacheManager memory tier held max_age=None
  entries indefinitely), so a long-lived web process showed the first snapshot
  and never reflected the display service's later updates. Add an opt-in
  force_reload path (get_health_summary/get_health_state/_load_health_state and
  get_metrics_summary/get_metrics) that bypasses the in-memory copy and, via a
  new memory_ttl passthrough on CacheManager.get, the cache manager's memory
  tier — so each /plugins/health and /plugins/metrics poll reads fresh persisted
  state. Default behaviour (force_reload=False) is unchanged for the display
  process and existing callers.
- Minor: DiskCache.get type hint is now Optional[int] with the None ("never
  expires") semantics documented, matching MemoryCache.get.

Tests: new force_reload staleness cases in test_plugin_health and
test_resource_monitor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq

---------

Co-authored-by: Claude <noreply@anthropic.com>

web_interface/blueprints/api_v3.py

@@ -2073,6 +2073,18 @@ def get_installed_plugins():
                 return None
 
         def _build_plugin_entry_inner(plugin_info, plugin_id):
+            # Capture runtime state (state machine + error context) before the
+            # manifest merge below can shadow the 'state' key. get_all_plugin_info
+            # attaches this via PluginStateManager.get_state_info(); surfacing it
+            # lets the UI show *why* a plugin isn't running instead of just
+            # 'loaded: false'.
+            state_info = plugin_info.get('state')
+            plugin_state = None
+            plugin_error_info = None
+            if isinstance(state_info, dict):
+                plugin_state = state_info.get('state')
+                plugin_error_info = state_info.get('error_info')
+
             # Re-read manifest from disk to ensure we have the latest metadata
             manifest_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id / "manifest.json"
             if manifest_path.exists():
@@ -2154,6 +2166,8 @@ def get_installed_plugins():
                 'enabled': enabled,
                 'verified': verified,
                 'loaded': plugin_info.get('loaded', False),
+                'state': plugin_state,
+                'error_info': plugin_error_info,
                 'last_updated': last_updated,
                 'last_commit': last_commit,
                 'last_commit_message': last_commit_message,
@@ -2173,6 +2187,31 @@ def get_installed_plugins():
         logger.error('Error in get_installed_plugins', exc_info=True)
         return jsonify({'status': 'error', 'message': 'An error occurred; see logs for details'}), 500
 
+def _installed_plugin_ids():
+    """Best-effort list of installed plugin IDs for the web process.
+
+    Health/metrics state is written by the separate display service to the
+    shared on-disk cache, so the tracker's in-memory set is empty here. We
+    enumerate the installed plugins and read each one's persisted summary by ID
+    instead of relying on the tracker's in-memory `get_all_*` view.
+    """
+    pm = api_v3.plugin_manager
+    manifests = getattr(pm, 'plugin_manifests', None)
+    if not manifests:
+        # Only pay for a discovery scan when we haven't discovered anything yet;
+        # subsequent polls reuse the already-populated manifest map.
+        try:
+            pm.discover_plugins()
+        except Exception:
+            logger.debug('discover_plugins failed while listing plugin ids', exc_info=True)
+        manifests = getattr(pm, 'plugin_manifests', None)
+    try:
+        return list(manifests.keys()) if manifests else []
+    except Exception:
+        logger.debug('listing plugin_manifests failed while building plugin ids', exc_info=True)
+        return []
+
+
 @api_v3.route('/plugins/health', methods=['GET'])
 def get_plugin_health():
     """Get health metrics for all plugins"""
@@ -2188,8 +2227,23 @@ def get_plugin_health():
                 'message': 'Health tracking not available'
             })
 
-        # Get health summaries for all plugins
-        health_summaries = api_v3.plugin_manager.health_tracker.get_all_health_summaries()
+        tracker = api_v3.plugin_manager.health_tracker
+        # Build per-plugin summaries by ID so persisted (cross-process) health
+        # is included, then fold in any in-memory-only entries.
+        health_summaries = {}
+        for pid in _installed_plugin_ids():
+            try:
+                # force_reload: this process only reads; bypass the in-memory
+                # snapshot so each poll reflects the display service's latest
+                # persisted state.
+                health_summaries[pid] = tracker.get_health_summary(pid, force_reload=True)
+            except Exception:
+                logger.debug('Could not read health summary for %s', pid, exc_info=True)
+        try:
+            for pid, summary in tracker.get_all_health_summaries().items():
+                health_summaries.setdefault(pid, summary)
+        except Exception:
+            logger.debug('get_all_health_summaries failed', exc_info=True)
 
         return jsonify({
             'status': 'success',
@@ -2264,8 +2318,22 @@ def get_plugin_metrics():
                 'message': 'Resource monitoring not available'
             })
 
-        # Get metrics summaries for all plugins
-        metrics_summaries = api_v3.plugin_manager.resource_monitor.get_all_metrics_summaries()
+        monitor = api_v3.plugin_manager.resource_monitor
+        # Build per-plugin summaries by ID so persisted (cross-process) metrics
+        # are included, then fold in any in-memory-only entries.
+        metrics_summaries = {}
+        for pid in _installed_plugin_ids():
+            try:
+                # force_reload: read-only path — bypass the in-memory snapshot so
+                # each poll reflects the display service's latest persisted metrics.
+                metrics_summaries[pid] = monitor.get_metrics_summary(pid, force_reload=True)
+            except Exception:
+                logger.debug('Could not read metrics summary for %s', pid, exc_info=True)
+        try:
+            for pid, summary in monitor.get_all_metrics_summaries().items():
+                metrics_summaries.setdefault(pid, summary)
+        except Exception:
+            logger.debug('get_all_metrics_summaries failed', exc_info=True)
 
         return jsonify({
             'status': 'success',