mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-05 06:45:09 +00:00
fix(errors): serve /api/v3/errors/* from the display service; add a Plugin errors panel (#614)
* fix(errors): serve /api/v3/errors/* from the display service's aggregator The error aggregator is a per-process singleton and only the display service runs plugins, so only its aggregator records anything. The routes read the web process's own, empty one and always reported no errors. The display service now publishes a bounded snapshot of its aggregator to the shared cache (plugin_error_snapshot) from a daemon thread: at most once every 10 s and only when something changed, never raising into the caller. The routes read it and keep their response shapes, adding snapshot_available, generated_at and clear_pending; exception text has credentials redacted. POST /errors/clear writes a clear request (plugin_error_clear_request) that the display applies on its next 5 s tick via the new clear_before(), which keeps errors recorded after the cutoff and rebuilds the counts. Until the snapshot acknowledges the request, reads hide everything before the cutoff, so a snapshot written just before the click cannot bring errors back. Adds "all": true; cleared_count is null when only the display can know it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(web): show plugin errors in the Logs tab A compact panel under the log viewer: per-plugin error counts, repeating errors (type, count, affected plugins, a sample message, last seen) and a Clear button, with empty states for "no errors" and "display service hasn't reported yet". Polls every 15 s while the tab is active; all text goes through escapeHtml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: describe where plugin error reports come from and how clear works Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(errors): redact the published snapshot before clipping it Keeping only a traceback's tail (or clipping a message) could cut an `api_key=` marker off while keeping the secret after it, and the web side's redaction would then have nothing to match. The display now redacts every free-text field of the snapshot first. The patterns move to a Flask-free src/redaction.py so the display service can use them; redact_text in the web error handler uses the same function, unchanged in behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
@@ -10,10 +10,11 @@ from web_interface.blueprints.api_v3 import (
|
||||
_MQTT_BRIDGE_DIR, _SUDO, _coerce_mqtt_bridge_value,
|
||||
_get_display_service_status, _mqtt_bridge_service_state,
|
||||
_read_mqtt_bridge_config, api_v3, contextlib, describe_exception,
|
||||
error_response, get_error_aggregator, json, jsonify, logger, os, request,
|
||||
error_response, json, jsonify, logger, os, redact_text, request,
|
||||
subprocess, success_response, tempfile,
|
||||
)
|
||||
from src.common.path_safety import safe_path_component
|
||||
from src import error_aggregator as _errors
|
||||
import web_interface.blueprints.api_v3 as _pkg
|
||||
# Read through the module rather than bound by value: tests patch these
|
||||
# as module attributes, and a value binding would not see the patch.
|
||||
@@ -326,17 +327,62 @@ def delete_cache_file():
|
||||
except Exception as e:
|
||||
logger.error('Error in delete_cache_file', exc_info=True)
|
||||
return jsonify({'status': 'error', 'message': 'An error occurred; see logs for details', 'details': describe_exception(e)}), 500
|
||||
def _errors_cache():
|
||||
"""The shared cache the display service publishes its errors to."""
|
||||
if not api_v3.cache_manager:
|
||||
from src.cache_manager import CacheManager
|
||||
api_v3.cache_manager = CacheManager()
|
||||
return api_v3.cache_manager
|
||||
|
||||
|
||||
def _redact_error_text(text, keep_lines=False):
|
||||
"""Credentials out of plugin exception text, which can quote a URL with
|
||||
an API key in it. Stack traces keep their line breaks and indentation."""
|
||||
if not isinstance(text, str):
|
||||
return text
|
||||
if not keep_lines:
|
||||
return redact_text(text, max_length=len(text) + 1)
|
||||
return '\n'.join(
|
||||
line[:len(line) - len(line.lstrip())] + redact_text(line, max_length=len(line) + 1)
|
||||
for line in text.splitlines()
|
||||
)
|
||||
|
||||
|
||||
def _redact_error_record(record):
|
||||
if not isinstance(record, dict):
|
||||
return record
|
||||
record = dict(record)
|
||||
record['message'] = _redact_error_text(record.get('message'))
|
||||
record['stack_trace'] = _redact_error_text(record.get('stack_trace'), keep_lines=True)
|
||||
if isinstance(record.get('context'), dict):
|
||||
record['context'] = {k: _redact_error_text(v) for k, v in record['context'].items()}
|
||||
return record
|
||||
|
||||
|
||||
def _read_errors():
|
||||
snapshot, clear_request = _errors.read_error_report(_errors_cache())
|
||||
return snapshot, clear_request
|
||||
|
||||
|
||||
@api_v3.route('/errors/summary', methods=['GET'])
|
||||
def get_error_summary():
|
||||
"""
|
||||
Get summary of all errors for monitoring and debugging.
|
||||
|
||||
Returns error counts, detected patterns, and recent errors.
|
||||
Returns error counts, detected patterns, and recent errors, as last
|
||||
reported by the display service (which runs the plugins, so it is the
|
||||
only process that records their errors). ``snapshot_available`` is false
|
||||
until it has reported; ``generated_at`` says when it did.
|
||||
"""
|
||||
try:
|
||||
aggregator = get_error_aggregator()
|
||||
summary = aggregator.get_error_summary()
|
||||
return success_response(data=summary, message="Error summary retrieved")
|
||||
summary = _errors.error_summary_from_report(*_read_errors())
|
||||
summary['recent_errors'] = [_redact_error_record(r) for r in summary['recent_errors']]
|
||||
for pattern in summary['active_patterns'].values():
|
||||
if isinstance(pattern, dict) and isinstance(pattern.get('sample_messages'), list):
|
||||
pattern['sample_messages'] = [_redact_error_text(m) for m in pattern['sample_messages']]
|
||||
message = ("Error summary retrieved" if summary['snapshot_available']
|
||||
else "The display service has not reported any errors yet")
|
||||
return success_response(data=summary, message=message)
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting error summary: {e}", exc_info=True)
|
||||
return error_response(
|
||||
@@ -352,11 +398,13 @@ def get_plugin_errors(plugin_id):
|
||||
Args:
|
||||
plugin_id: Plugin identifier
|
||||
|
||||
Returns health status and error statistics for the plugin.
|
||||
Returns health status and error statistics for the plugin, from the
|
||||
display service's last report (see get_error_summary). A plugin with no
|
||||
recorded errors is "healthy".
|
||||
"""
|
||||
try:
|
||||
aggregator = get_error_aggregator()
|
||||
health = aggregator.get_plugin_health(plugin_id)
|
||||
health = _errors.plugin_health_from_report(*_read_errors(), plugin_id)
|
||||
health['last_error'] = _redact_error_record(health['last_error'])
|
||||
return success_response(data=health, message="Plugin health retrieved")
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting plugin health for {plugin_id}: {e}", exc_info=True)
|
||||
@@ -372,42 +420,61 @@ def clear_old_errors():
|
||||
|
||||
Request body (optional):
|
||||
max_age_hours: Maximum age in hours (default: 24, max: 8760 = 1 year)
|
||||
all: true clears every error recorded so far (max_age_hours ignored)
|
||||
|
||||
The errors live in the display service, so this records a clear request
|
||||
that it applies within a few seconds. Reads hide the cleared errors from
|
||||
the moment the request is recorded.
|
||||
"""
|
||||
try:
|
||||
data = request.get_json(silent=True) or {}
|
||||
clear_all = _coerce_to_bool(data.get('all'))
|
||||
raw_max_age = data.get('max_age_hours', 24)
|
||||
|
||||
# Validate and coerce max_age_hours
|
||||
max_age_hours = None
|
||||
if not clear_all:
|
||||
try:
|
||||
max_age_hours = int(raw_max_age)
|
||||
if max_age_hours < 1:
|
||||
return error_response(
|
||||
error_code=ErrorCode.INVALID_INPUT,
|
||||
message="max_age_hours must be at least 1",
|
||||
context={'provided_value': raw_max_age},
|
||||
status_code=400
|
||||
)
|
||||
if max_age_hours > 8760: # 1 year max
|
||||
return error_response(
|
||||
error_code=ErrorCode.INVALID_INPUT,
|
||||
message="max_age_hours cannot exceed 8760 (1 year)",
|
||||
context={'provided_value': raw_max_age},
|
||||
status_code=400
|
||||
)
|
||||
except (ValueError, TypeError, OverflowError):
|
||||
return error_response(
|
||||
error_code=ErrorCode.INVALID_INPUT,
|
||||
message="max_age_hours must be a valid integer",
|
||||
context={'provided_value': str(raw_max_age)},
|
||||
status_code=400
|
||||
)
|
||||
|
||||
now = _pkg.time.time()
|
||||
cutoff = now if clear_all else now - max_age_hours * 3600
|
||||
try:
|
||||
max_age_hours = int(raw_max_age)
|
||||
if max_age_hours < 1:
|
||||
return error_response(
|
||||
error_code=ErrorCode.INVALID_INPUT,
|
||||
message="max_age_hours must be at least 1",
|
||||
context={'provided_value': raw_max_age},
|
||||
status_code=400
|
||||
)
|
||||
if max_age_hours > 8760: # 1 year max
|
||||
return error_response(
|
||||
error_code=ErrorCode.INVALID_INPUT,
|
||||
message="max_age_hours cannot exceed 8760 (1 year)",
|
||||
context={'provided_value': raw_max_age},
|
||||
status_code=400
|
||||
)
|
||||
except (ValueError, TypeError, OverflowError):
|
||||
result = _errors.request_error_clear(_errors_cache(), cutoff)
|
||||
except OSError as e:
|
||||
logger.error("Could not record an error clear request: %s", e)
|
||||
return error_response(
|
||||
error_code=ErrorCode.INVALID_INPUT,
|
||||
message="max_age_hours must be a valid integer",
|
||||
context={'provided_value': str(raw_max_age)},
|
||||
status_code=400
|
||||
error_code=ErrorCode.SYSTEM_ERROR,
|
||||
message="Could not record the clear request in the shared cache",
|
||||
status_code=500
|
||||
)
|
||||
|
||||
aggregator = get_error_aggregator()
|
||||
cleared_count = aggregator.clear_old_records(max_age_hours=max_age_hours)
|
||||
|
||||
scope = "all errors" if clear_all else f"errors older than {max_age_hours} hours"
|
||||
return success_response(
|
||||
data={'cleared_count': cleared_count},
|
||||
message=f"Cleared {cleared_count} error records older than {max_age_hours} hours"
|
||||
data=result,
|
||||
message=(f"Clear of {scope} requested; the display service applies it "
|
||||
f"within about {int(_errors.SNAPSHOT_TICK_INTERVAL)} seconds")
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error clearing old errors: {e}", exc_info=True)
|
||||
|
||||
@@ -104,6 +104,25 @@
|
||||
<div class="w-2 h-2 bg-green-500 rounded-full"></div>
|
||||
<span>Connected to log stream</span>
|
||||
</div>
|
||||
|
||||
<!-- Plugin errors, as recorded by the display service -->
|
||||
<div id="plugin-errors-panel" class="mt-6 border-t border-gray-200 pt-4">
|
||||
<div class="flex flex-wrap items-center justify-between gap-2 mb-3">
|
||||
<div>
|
||||
<h3 class="text-base font-semibold text-gray-900">Plugin errors</h3>
|
||||
<p id="plugin-errors-meta" class="text-xs text-gray-600">Recorded by the display service</p>
|
||||
</div>
|
||||
<div class="flex items-center gap-2">
|
||||
<button id="plugin-errors-refresh-btn" type="button" class="btn bg-gray-600 hover:bg-gray-700 text-white px-3 py-1 rounded text-sm">
|
||||
<i class="fas fa-sync-alt mr-1"></i>Refresh
|
||||
</button>
|
||||
<button id="plugin-errors-clear-btn" type="button" class="btn bg-red-600 hover:bg-red-700 text-white px-3 py-1 rounded text-sm">
|
||||
<i class="fas fa-eraser mr-1"></i>Clear
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="plugin-errors-body" class="text-sm text-gray-600" aria-live="polite">Loading...</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
@@ -218,10 +237,18 @@ window._filteredLogs = [];
|
||||
// Current-plugin poll and the realtime log stream run only while the Logs
|
||||
// tab is active and the page is visible (both restart with an immediate
|
||||
// refresh). Re-registering on partial reload replaces the old one.
|
||||
const pluginErrorsRefreshBtn = document.getElementById('plugin-errors-refresh-btn');
|
||||
if (pluginErrorsRefreshBtn) pluginErrorsRefreshBtn.onclick = refreshPluginErrors;
|
||||
const pluginErrorsClearBtn = document.getElementById('plugin-errors-clear-btn');
|
||||
if (pluginErrorsClearBtn) pluginErrorsClearBtn.onclick = clearPluginErrors;
|
||||
|
||||
function startLogsActivity() {
|
||||
refreshCurrentPluginStatus();
|
||||
if (window._currentPluginPollTimer) clearInterval(window._currentPluginPollTimer);
|
||||
window._currentPluginPollTimer = setInterval(refreshCurrentPluginStatus, 5000);
|
||||
refreshPluginErrors();
|
||||
if (window._pluginErrorsPollTimer) clearInterval(window._pluginErrorsPollTimer);
|
||||
window._pluginErrorsPollTimer = setInterval(refreshPluginErrors, 15000);
|
||||
if (window._isRealtime && !window._logsEventSource) setupRealtimeLogs();
|
||||
}
|
||||
function stopLogsActivity() {
|
||||
@@ -229,6 +256,10 @@ window._filteredLogs = [];
|
||||
clearInterval(window._currentPluginPollTimer);
|
||||
window._currentPluginPollTimer = null;
|
||||
}
|
||||
if (window._pluginErrorsPollTimer) {
|
||||
clearInterval(window._pluginErrorsPollTimer);
|
||||
window._pluginErrorsPollTimer = null;
|
||||
}
|
||||
stopRealtimeLogs();
|
||||
}
|
||||
if (window.LEDVisibility) {
|
||||
@@ -787,6 +818,120 @@ function refreshCurrentPluginStatus() {
|
||||
});
|
||||
}
|
||||
|
||||
// Plugin errors panel. The display service records them and publishes a
|
||||
// snapshot every few seconds; /api/v3/errors/summary serves that snapshot.
|
||||
// Plugin ids and messages come from plugins: everything goes through escapeHtml.
|
||||
function formatErrorTime(iso) {
|
||||
return iso ? String(iso).replace('T', ' ').slice(0, 19) : '';
|
||||
}
|
||||
|
||||
function renderPluginErrors(summary) {
|
||||
const body = document.getElementById('plugin-errors-body');
|
||||
const meta = document.getElementById('plugin-errors-meta');
|
||||
if (!body) return;
|
||||
|
||||
if (meta) {
|
||||
let text = 'Recorded by the display service';
|
||||
if (summary.generated_at) text += ' · updated ' + formatErrorTime(summary.generated_at);
|
||||
if (summary.clear_pending) text += ' · clearing…';
|
||||
meta.textContent = text;
|
||||
}
|
||||
|
||||
if (!summary.snapshot_available) {
|
||||
body.innerHTML = '<p class="text-gray-600"><i class="fas fa-hourglass-half mr-1"></i>' +
|
||||
'Display service hasn’t reported yet. Errors appear here once it is running.</p>';
|
||||
return;
|
||||
}
|
||||
const plugins = Object.entries(summary.plugin_error_counts || {}).map(([id, types]) => {
|
||||
const total = Object.values(types || {}).reduce((sum, n) => sum + (Number(n) || 0), 0);
|
||||
return { id, types: types || {}, total };
|
||||
}).sort((a, b) => b.total - a.total);
|
||||
const patterns = Object.values(summary.active_patterns || {});
|
||||
|
||||
if (!Number(summary.total_errors) && plugins.length === 0 && patterns.length === 0) {
|
||||
body.innerHTML = '<p class="text-gray-600"><i class="fas fa-check-circle text-green-600 mr-1"></i>' +
|
||||
'No plugin errors recorded</p>';
|
||||
return;
|
||||
}
|
||||
|
||||
let html = '';
|
||||
if (plugins.length) {
|
||||
html += '<div class="overflow-x-auto"><table class="min-w-full text-sm">' +
|
||||
'<thead><tr class="text-left text-xs uppercase text-gray-500 border-b border-gray-200">' +
|
||||
'<th class="py-1 pr-4 font-medium">Plugin</th><th class="py-1 pr-4 font-medium">Errors</th>' +
|
||||
'<th class="py-1 font-medium">Types</th></tr></thead><tbody>';
|
||||
plugins.forEach(p => {
|
||||
const types = Object.entries(p.types)
|
||||
.map(([type, n]) => `${escapeHtml(type)} ×${escapeHtml(String(Number(n) || 0))}`)
|
||||
.join(', ');
|
||||
html += '<tr class="border-b border-gray-100">' +
|
||||
`<td class="py-1 pr-4"><code class="bg-gray-100 px-1 rounded">${escapeHtml(p.id)}</code></td>` +
|
||||
`<td class="py-1 pr-4 font-semibold text-gray-900">${escapeHtml(String(p.total))}</td>` +
|
||||
`<td class="py-1 text-gray-700 break-words">${types}</td></tr>`;
|
||||
});
|
||||
html += '</tbody></table></div>';
|
||||
}
|
||||
|
||||
if (patterns.length) {
|
||||
const badge = {
|
||||
critical: 'bg-red-100 text-red-800',
|
||||
error: 'bg-amber-100 text-amber-800',
|
||||
warning: 'bg-yellow-100 text-yellow-800'
|
||||
};
|
||||
html += '<h4 class="mt-4 mb-2 text-sm font-semibold text-gray-900">Repeating errors</h4><ul class="list-none pl-0 space-y-2">';
|
||||
patterns.forEach(p => {
|
||||
const severity = String(p.severity || 'warning');
|
||||
const affected = (p.affected_plugins || []).map(id => escapeHtml(id)).join(', ') || 'unknown';
|
||||
const sample = (p.sample_messages || [])[0];
|
||||
html += '<li class="border border-gray-200 rounded-lg px-3 py-2">' +
|
||||
'<div class="flex flex-wrap items-center gap-2">' +
|
||||
`<span class="px-2 py-0.5 rounded text-xs font-semibold ${badge[severity] || badge.warning}">${escapeHtml(severity)}</span>` +
|
||||
`<code class="font-semibold text-gray-900">${escapeHtml(p.error_type || '')}</code>` +
|
||||
`<span class="text-gray-700">×${escapeHtml(String(Number(p.count) || 0))}</span>` +
|
||||
`<span class="text-xs text-gray-500 ml-auto">last seen ${escapeHtml(formatErrorTime(p.last_seen))}</span></div>` +
|
||||
`<div class="mt-1 text-xs text-gray-600">Plugins: ${affected}</div>` +
|
||||
(sample ? `<div class="mt-1 text-xs font-mono text-gray-700 break-words">${escapeHtml(sample)}</div>` : '') +
|
||||
'</li>';
|
||||
});
|
||||
html += '</ul>';
|
||||
}
|
||||
body.innerHTML = html;
|
||||
}
|
||||
|
||||
function refreshPluginErrors() {
|
||||
fetch('/api/v3/errors/summary')
|
||||
.then(response => response.json())
|
||||
.then(data => {
|
||||
if (data.status !== 'success' || !data.data) throw new Error(data.message || 'Request failed');
|
||||
renderPluginErrors(data.data);
|
||||
})
|
||||
.catch(() => {
|
||||
const body = document.getElementById('plugin-errors-body');
|
||||
if (body) body.textContent = 'Could not load plugin errors.';
|
||||
});
|
||||
}
|
||||
|
||||
function clearPluginErrors() {
|
||||
if (!confirm('Clear all recorded plugin errors?')) return;
|
||||
fetch('/api/v3/errors/clear', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ all: true })
|
||||
})
|
||||
.then(response => response.json())
|
||||
.then(data => {
|
||||
if (typeof showNotification !== 'undefined') {
|
||||
showNotification(data.status === 'success' ? 'Plugin errors cleared'
|
||||
: (data.message || 'Could not clear plugin errors'),
|
||||
data.status === 'success' ? 'success' : 'error');
|
||||
}
|
||||
refreshPluginErrors();
|
||||
})
|
||||
.catch(() => {
|
||||
if (typeof showNotification !== 'undefined') showNotification('Could not clear plugin errors', 'error');
|
||||
});
|
||||
}
|
||||
|
||||
// Cleanup on page unload
|
||||
window.addEventListener('beforeunload', function() {
|
||||
if (window._logsEventSource) {
|
||||
@@ -796,5 +941,9 @@ window.addEventListener('beforeunload', function() {
|
||||
clearInterval(window._currentPluginPollTimer);
|
||||
window._currentPluginPollTimer = null;
|
||||
}
|
||||
if (window._pluginErrorsPollTimer) {
|
||||
clearInterval(window._pluginErrorsPollTimer);
|
||||
window._pluginErrorsPollTimer = null;
|
||||
}
|
||||
});
|
||||
</script>
|
||||
|
||||
Reference in New Issue
Block a user