mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-09 08:36:37 +00:00
fix(plugins): a busy-lock update skip is report-only, never a breaker failure
When the update worker gives up waiting for a plugin's lock (PLUGIN_LOCK_TIMEOUT, 5s) the skip was recorded as a hang, so three in a row opened the circuit breaker. Vegas prefetch holds a plugin's lock for its whole content render, which on a slow Pi can outlast 5s, so a healthy plugin could be pulled from rotation. The skip is now report-only: still logged (rate-limited) and still left as PluginBusyError state error info with last-update stamped, but in health it is counted as a busy skip (busy_skip_count / last_busy_skip, via the new PluginHealthTracker.record_busy_skip) and never touches the failure streak, last_error or the breaker. Only real hangs -- display() or update() running past the executor timeout -- still count toward the breaker. Busy-skip persistence shares the slow-call throttle (first one saved at once so the web process sees it, then at most once a minute per plugin). Tests prove repeated busy skips never open the breaker and repeated real hangs still do, with busy skips interleaved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+9
-5
@@ -36,17 +36,21 @@ accepts both, but the store flags the old spelling as deprecated
|
||||
display() that never returned (or a first frame still running after the
|
||||
executor's 30s timeout) parked the worker for good, so scores, weather and
|
||||
clocks all froze while the panel kept scrolling. The worker now waits at
|
||||
most 5s (the bound `unload_plugin()` already uses), skips the busy plugin
|
||||
and records the skip as a hang, so a plugin that keeps hanging opens its
|
||||
circuit breaker and drops out of updates and rotation until the cooldown.
|
||||
The other plugins keep updating.
|
||||
most 5s (the bound `unload_plugin()` already uses) and skips that update;
|
||||
the other plugins keep updating. The skip is logged (at most once a minute
|
||||
per plugin) and counted in plugin health as a busy skip (`busy_skip_count`,
|
||||
`last_busy_skip`), but it is not a failure and never opens the circuit
|
||||
breaker: Vegas mode holds a plugin's lock for its whole content render,
|
||||
which on a slow Pi can outlast 5s, and a healthy plugin must not be pulled
|
||||
from rotation for that.
|
||||
- display() calls are timed on every frame. One taking 2s or more is logged
|
||||
(at most once a minute per plugin) and counted in plugin health
|
||||
(`slow_call_count`, `last_slow_call`); one that runs past the executor's
|
||||
timeout counts as a hang (`hang_count`, `last_hang`) and as a failure to
|
||||
the circuit breaker. A first frame that times out is no longer recorded as
|
||||
a success, and an update() still running after its timeout is recorded as
|
||||
a hang instead of leaving the plugin silently stuck.
|
||||
a hang instead of leaving the plugin silently stuck. Only these real hangs
|
||||
count toward the breaker.
|
||||
- A plugin's `on_config_change()` no longer runs while its update() is
|
||||
running on the worker thread. It now runs under the plugin's lock; if the
|
||||
lock stays busy past the same 5s bound the change is handed to the update
|
||||
|
||||
Reference in New Issue
Block a user