mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-08-01 08:48:05 +00:00
feat: activate dormant plugin health/metrics subsystem and surface it in the web UI (#388)
* feat(plugin-system): activate dormant plugin health & metrics subsystem
PluginManager shipped a fully-built health tracker, resource monitor and
circuit breaker that were never instantiated (health_tracker/resource_monitor
were left as None), so the circuit breaker never engaged and the existing
health/metrics API routes always returned "not available".
- DisplayController now wires a PluginHealthTracker and PluginResourceMonitor
onto the plugin manager, enabling the circuit breaker (a repeatedly-failing
plugin's update() is skipped after consecutive failures, then retried after
a cooldown) and per-plugin execution-time metrics. Both persist to the
shared cache.
- load_plugin() now validates each plugin's config against its JSON schema in
a strictly warn/degrade-only way: a violation logs a warning and flags the
plugin degraded in the health tracker, but never changes whether the plugin
loads or its pass/fail behaviour. Adds PluginHealthTracker.set_degraded(),
which never touches the circuit breaker.
- ResourceMonitor CPU/memory sampling now reuses a cached psutil.Process and
reads cpu_percent(interval=None), so monitoring no longer blocks ~100ms per
call on the display loop's update path.
- Fix DiskCache.get() raising TypeError for max_age=None ("never expires"),
which silently discarded persisted plugin health/metrics on read and thus
broke cross-process and post-restart surfacing.
- Fix two dead PluginManager helpers that called non-existent tracker methods.
Tests: new test_resource_monitor, test_plugin_health,
test_plugin_manager_schema_soft; extended test_cache_manager and
test_display_controller.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq
* feat(web-ui): surface plugin health, metrics and load state
With the health/metrics subsystem now active in the display service, expose it
in the web UI (which runs as a separate process from the display loop):
- Wire a health tracker / resource monitor backed by the shared on-disk cache
into the web process so /api/v3/plugins/health and /plugins/metrics read the
data the display service persists.
- Build those route responses per installed plugin id (the tracker's in-memory
view is empty in a fresh web process) so cross-process data is included.
- Add state + error_info to /plugins/installed entries so the UI can show why a
plugin isn't running instead of just loaded:false.
- Add a "Plugin Health" panel to the Tools page (circuit status, avg/max update
time, update count, last error) plus PluginAPI.getPluginMetrics().
Tests: route-level tests for the health/metrics endpoints in test_web_api.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq
* fix(plugin-metrics): refresh cross-process health/metrics reads; type hints
Addresses CodeRabbit review on #388:
- Major: the web process's health/resource trackers cached the first persisted
read in an in-memory dict (and the CacheManager memory tier held max_age=None
entries indefinitely), so a long-lived web process showed the first snapshot
and never reflected the display service's later updates. Add an opt-in
force_reload path (get_health_summary/get_health_state/_load_health_state and
get_metrics_summary/get_metrics) that bypasses the in-memory copy and, via a
new memory_ttl passthrough on CacheManager.get, the cache manager's memory
tier — so each /plugins/health and /plugins/metrics poll reads fresh persisted
state. Default behaviour (force_reload=False) is unchanged for the display
process and existing callers.
- Minor: DiskCache.get type hint is now Optional[int] with the None ("never
expires") semantics documented, matching MemoryCache.get.
Tests: new force_reload staleness cases in test_plugin_health and
test_resource_monitor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTav268UXv44ub9K11LYq
---------
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Chuck
@@ -160,6 +160,22 @@ api_v3.health_monitor = health_monitor
|
||||
from src.cache_manager import CacheManager
|
||||
api_v3.cache_manager = CacheManager()
|
||||
|
||||
# Wire plugin health/metrics for the web process. The display service records
|
||||
# health and execution-time metrics to the shared on-disk cache; giving the web
|
||||
# process its own tracker/monitor backed by that same cache lets the health API
|
||||
# routes (/api/v3/plugins/health, /plugins/metrics) read that persisted data.
|
||||
# Guarded so any init failure degrades to "not available" rather than breaking
|
||||
# the web server.
|
||||
try:
|
||||
from src.plugin_system.plugin_health import PluginHealthTracker
|
||||
from src.plugin_system.resource_monitor import PluginResourceMonitor
|
||||
plugin_manager.health_tracker = PluginHealthTracker(api_v3.cache_manager)
|
||||
plugin_manager.resource_monitor = PluginResourceMonitor(api_v3.cache_manager)
|
||||
except Exception as _hm_err: # pragma: no cover - defensive startup guard
|
||||
logging.getLogger(__name__).warning(
|
||||
"Could not enable plugin health/metrics for web UI: %s", _hm_err
|
||||
)
|
||||
|
||||
app.register_blueprint(pages_v3, url_prefix='/v3')
|
||||
app.register_blueprint(api_v3, url_prefix='/api/v3')
|
||||
|
||||
|
||||
@@ -2073,6 +2073,18 @@ def get_installed_plugins():
|
||||
return None
|
||||
|
||||
def _build_plugin_entry_inner(plugin_info, plugin_id):
|
||||
# Capture runtime state (state machine + error context) before the
|
||||
# manifest merge below can shadow the 'state' key. get_all_plugin_info
|
||||
# attaches this via PluginStateManager.get_state_info(); surfacing it
|
||||
# lets the UI show *why* a plugin isn't running instead of just
|
||||
# 'loaded: false'.
|
||||
state_info = plugin_info.get('state')
|
||||
plugin_state = None
|
||||
plugin_error_info = None
|
||||
if isinstance(state_info, dict):
|
||||
plugin_state = state_info.get('state')
|
||||
plugin_error_info = state_info.get('error_info')
|
||||
|
||||
# Re-read manifest from disk to ensure we have the latest metadata
|
||||
manifest_path = Path(api_v3.plugin_manager.plugins_dir) / plugin_id / "manifest.json"
|
||||
if manifest_path.exists():
|
||||
@@ -2154,6 +2166,8 @@ def get_installed_plugins():
|
||||
'enabled': enabled,
|
||||
'verified': verified,
|
||||
'loaded': plugin_info.get('loaded', False),
|
||||
'state': plugin_state,
|
||||
'error_info': plugin_error_info,
|
||||
'last_updated': last_updated,
|
||||
'last_commit': last_commit,
|
||||
'last_commit_message': last_commit_message,
|
||||
@@ -2173,6 +2187,31 @@ def get_installed_plugins():
|
||||
logger.error('Error in get_installed_plugins', exc_info=True)
|
||||
return jsonify({'status': 'error', 'message': 'An error occurred; see logs for details'}), 500
|
||||
|
||||
def _installed_plugin_ids():
|
||||
"""Best-effort list of installed plugin IDs for the web process.
|
||||
|
||||
Health/metrics state is written by the separate display service to the
|
||||
shared on-disk cache, so the tracker's in-memory set is empty here. We
|
||||
enumerate the installed plugins and read each one's persisted summary by ID
|
||||
instead of relying on the tracker's in-memory `get_all_*` view.
|
||||
"""
|
||||
pm = api_v3.plugin_manager
|
||||
manifests = getattr(pm, 'plugin_manifests', None)
|
||||
if not manifests:
|
||||
# Only pay for a discovery scan when we haven't discovered anything yet;
|
||||
# subsequent polls reuse the already-populated manifest map.
|
||||
try:
|
||||
pm.discover_plugins()
|
||||
except Exception:
|
||||
logger.debug('discover_plugins failed while listing plugin ids', exc_info=True)
|
||||
manifests = getattr(pm, 'plugin_manifests', None)
|
||||
try:
|
||||
return list(manifests.keys()) if manifests else []
|
||||
except Exception:
|
||||
logger.debug('listing plugin_manifests failed while building plugin ids', exc_info=True)
|
||||
return []
|
||||
|
||||
|
||||
@api_v3.route('/plugins/health', methods=['GET'])
|
||||
def get_plugin_health():
|
||||
"""Get health metrics for all plugins"""
|
||||
@@ -2188,8 +2227,23 @@ def get_plugin_health():
|
||||
'message': 'Health tracking not available'
|
||||
})
|
||||
|
||||
# Get health summaries for all plugins
|
||||
health_summaries = api_v3.plugin_manager.health_tracker.get_all_health_summaries()
|
||||
tracker = api_v3.plugin_manager.health_tracker
|
||||
# Build per-plugin summaries by ID so persisted (cross-process) health
|
||||
# is included, then fold in any in-memory-only entries.
|
||||
health_summaries = {}
|
||||
for pid in _installed_plugin_ids():
|
||||
try:
|
||||
# force_reload: this process only reads; bypass the in-memory
|
||||
# snapshot so each poll reflects the display service's latest
|
||||
# persisted state.
|
||||
health_summaries[pid] = tracker.get_health_summary(pid, force_reload=True)
|
||||
except Exception:
|
||||
logger.debug('Could not read health summary for %s', pid, exc_info=True)
|
||||
try:
|
||||
for pid, summary in tracker.get_all_health_summaries().items():
|
||||
health_summaries.setdefault(pid, summary)
|
||||
except Exception:
|
||||
logger.debug('get_all_health_summaries failed', exc_info=True)
|
||||
|
||||
return jsonify({
|
||||
'status': 'success',
|
||||
@@ -2264,8 +2318,22 @@ def get_plugin_metrics():
|
||||
'message': 'Resource monitoring not available'
|
||||
})
|
||||
|
||||
# Get metrics summaries for all plugins
|
||||
metrics_summaries = api_v3.plugin_manager.resource_monitor.get_all_metrics_summaries()
|
||||
monitor = api_v3.plugin_manager.resource_monitor
|
||||
# Build per-plugin summaries by ID so persisted (cross-process) metrics
|
||||
# are included, then fold in any in-memory-only entries.
|
||||
metrics_summaries = {}
|
||||
for pid in _installed_plugin_ids():
|
||||
try:
|
||||
# force_reload: read-only path — bypass the in-memory snapshot so
|
||||
# each poll reflects the display service's latest persisted metrics.
|
||||
metrics_summaries[pid] = monitor.get_metrics_summary(pid, force_reload=True)
|
||||
except Exception:
|
||||
logger.debug('Could not read metrics summary for %s', pid, exc_info=True)
|
||||
try:
|
||||
for pid, summary in monitor.get_all_metrics_summaries().items():
|
||||
metrics_summaries.setdefault(pid, summary)
|
||||
except Exception:
|
||||
logger.debug('get_all_metrics_summaries failed', exc_info=True)
|
||||
|
||||
return jsonify({
|
||||
'status': 'success',
|
||||
|
||||
@@ -340,11 +340,25 @@ const PluginAPI = {
|
||||
* @returns {Promise<Object>} Health data
|
||||
*/
|
||||
async getPluginHealth(pluginId = null) {
|
||||
const endpoint = pluginId
|
||||
const endpoint = pluginId
|
||||
? `/plugins/health/${pluginId}`
|
||||
: '/plugins/health';
|
||||
const response = await this.request(endpoint);
|
||||
return response.data || {};
|
||||
},
|
||||
|
||||
/**
|
||||
* Get plugin resource metrics (execution time, memory, cpu).
|
||||
*
|
||||
* @param {string} pluginId - Optional plugin identifier (null for all)
|
||||
* @returns {Promise<Object>} Metrics data keyed by plugin id
|
||||
*/
|
||||
async getPluginMetrics(pluginId = null) {
|
||||
const endpoint = pluginId
|
||||
? `/plugins/metrics/${pluginId}`
|
||||
: '/plugins/metrics';
|
||||
const response = await this.request(endpoint);
|
||||
return response.data || {};
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -157,6 +157,38 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Plugin Health -->
|
||||
<div class="bg-white rounded-lg shadow p-6">
|
||||
<div class="border-b border-gray-200 pb-4 mb-6 flex items-start justify-between gap-4">
|
||||
<div>
|
||||
<h2 class="text-lg font-semibold text-gray-900">Plugin Health</h2>
|
||||
<p class="mt-1 text-sm text-gray-600">Circuit-breaker status and per-plugin update timings recorded by the display service. A plugin whose <code class="bg-gray-100 px-1 rounded">update()</code> keeps failing is paused ("Circuit open") and retried automatically after a cooldown.</p>
|
||||
</div>
|
||||
<button id="btn-plugin-health-refresh" onclick="refreshPluginHealth(true)"
|
||||
class="shrink-0 inline-flex items-center px-3 py-2 border border-gray-300 text-sm font-medium rounded-md text-gray-700 bg-white hover:bg-gray-50">
|
||||
<i class="fas fa-sync-alt mr-2"></i>Refresh
|
||||
</button>
|
||||
</div>
|
||||
<div id="plugin-health-message" class="hidden mb-4 text-sm text-gray-500"></div>
|
||||
<div class="overflow-x-auto">
|
||||
<table class="min-w-full divide-y divide-gray-200">
|
||||
<thead class="bg-gray-50">
|
||||
<tr>
|
||||
<th class="px-4 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Plugin</th>
|
||||
<th class="px-4 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Status</th>
|
||||
<th class="px-4 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">Avg update</th>
|
||||
<th class="px-4 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">Max update</th>
|
||||
<th class="px-4 py-3 text-right text-xs font-medium text-gray-500 uppercase tracking-wider">Updates</th>
|
||||
<th class="px-4 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Last error</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="plugin-health-tbody" class="bg-white divide-y divide-gray-200">
|
||||
<tr><td colspan="6" class="px-4 py-8 text-center text-gray-500">Loading…</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<script>
|
||||
@@ -397,7 +429,80 @@
|
||||
});
|
||||
}
|
||||
|
||||
// ── plugin health panel ──────────────────────────────────────────────────
|
||||
function phEscape(s) {
|
||||
return String(s).replace(/[&<>"']/g, function (c) {
|
||||
return { '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' }[c];
|
||||
});
|
||||
}
|
||||
function phFmtSecs(v) {
|
||||
if (typeof v !== 'number' || !isFinite(v)) return '—';
|
||||
return v.toFixed(3) + 's';
|
||||
}
|
||||
function phStatus(h) {
|
||||
if (!h) return { label: 'Unknown', cls: 'warning' };
|
||||
if (h.circuit_state === 'open') return { label: 'Circuit open', cls: 'error' };
|
||||
if (h.circuit_state === 'half_open') return { label: 'Recovering', cls: 'warning' };
|
||||
if (h.degraded) return { label: 'Degraded', cls: 'warning' };
|
||||
if (h.is_healthy) return { label: 'Healthy', cls: 'success' };
|
||||
return { label: 'Unknown', cls: 'warning' };
|
||||
}
|
||||
async function refreshPluginHealth(force) {
|
||||
const tbody = document.getElementById('plugin-health-tbody');
|
||||
const msg = document.getElementById('plugin-health-message');
|
||||
if (!tbody || !window.PluginAPI) return;
|
||||
try {
|
||||
if (force && PluginAPI.clearCache) PluginAPI.clearCache();
|
||||
const results = await Promise.all([
|
||||
PluginAPI.getPluginHealth(),
|
||||
PluginAPI.getPluginMetrics()
|
||||
]);
|
||||
const health = results[0] || {};
|
||||
const metrics = results[1] || {};
|
||||
const ids = Array.from(new Set(Object.keys(health).concat(Object.keys(metrics)))).sort();
|
||||
if (!ids.length) {
|
||||
if (msg) {
|
||||
msg.textContent = 'No plugin health data yet — it appears once the display service has run plugins.';
|
||||
msg.classList.remove('hidden');
|
||||
}
|
||||
tbody.innerHTML = '<tr><td colspan="6" class="px-4 py-8 text-center text-gray-500">No data</td></tr>';
|
||||
return;
|
||||
}
|
||||
if (msg) msg.classList.add('hidden');
|
||||
let rows = '';
|
||||
ids.forEach(function (id) {
|
||||
const h = health[id] || {};
|
||||
const m = metrics[id] || {};
|
||||
const st = phStatus(h);
|
||||
const lastErr = h.degraded_reason || h.last_error || '';
|
||||
const calls = (typeof m.call_count === 'number') ? m.call_count : '—';
|
||||
const errCell = lastErr
|
||||
? '<span title="' + phEscape(lastErr) + '">' + phEscape(lastErr) + '</span>'
|
||||
: '<span class="text-gray-400">—</span>';
|
||||
rows += '<tr>' +
|
||||
'<td class="px-4 py-3 whitespace-nowrap text-sm font-medium text-gray-900">' + phEscape(id) + '</td>' +
|
||||
'<td class="px-4 py-3 whitespace-nowrap"><span class="status-indicator ' + st.cls + '">' + st.label + '</span></td>' +
|
||||
'<td class="px-4 py-3 whitespace-nowrap text-right text-sm text-gray-600">' + phFmtSecs(m.avg_execution_time) + '</td>' +
|
||||
'<td class="px-4 py-3 whitespace-nowrap text-right text-sm text-gray-600">' + phFmtSecs(m.max_execution_time) + '</td>' +
|
||||
'<td class="px-4 py-3 whitespace-nowrap text-right text-sm text-gray-600">' + calls + '</td>' +
|
||||
'<td class="px-4 py-3 text-sm text-red-600 max-w-xs truncate">' + errCell + '</td>' +
|
||||
'</tr>';
|
||||
});
|
||||
tbody.innerHTML = rows;
|
||||
} catch (e) {
|
||||
const emsg = (e && e.message) ? e.message : String(e);
|
||||
tbody.innerHTML = '<tr><td colspan="6" class="px-4 py-6 text-center text-red-500">Failed to load plugin health: ' + phEscape(emsg) + '</td></tr>';
|
||||
}
|
||||
}
|
||||
window.refreshPluginHealth = refreshPluginHealth;
|
||||
|
||||
// Load on first render; HTMX will have already swapped us in by this point.
|
||||
loadGitInfo();
|
||||
refreshPluginHealth(false);
|
||||
// Refresh periodically. Guard against duplicate timers if this partial is
|
||||
// re-swapped in by HTMX; the handler re-resolves DOM nodes by id each tick.
|
||||
if (!window._pluginHealthTimer) {
|
||||
window._pluginHealthTimer = setInterval(function () { refreshPluginHealth(true); }, 15000);
|
||||
}
|
||||
})();
|
||||
</script>
|
||||
|
||||
Reference in New Issue
Block a user