When the update worker gives up waiting for a plugin's lock
(PLUGIN_LOCK_TIMEOUT, 5s) the skip was recorded as a hang, so three in a
row opened the circuit breaker. Vegas prefetch holds a plugin's lock for
its whole content render, which on a slow Pi can outlast 5s, so a healthy
plugin could be pulled from rotation.
The skip is now report-only: still logged (rate-limited) and still left as
PluginBusyError state error info with last-update stamped, but in health it
is counted as a busy skip (busy_skip_count / last_busy_skip, via the new
PluginHealthTracker.record_busy_skip) and never touches the failure streak,
last_error or the breaker. Only real hangs -- display() or update() running
past the executor timeout -- still count toward the breaker.
Busy-skip persistence shares the slow-call throttle (first one saved at
once so the web process sees it, then at most once a minute per plugin).
Tests prove repeated busy skips never open the breaker and repeated real
hangs still do, with busy skips interleaved.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The single update worker took each plugin's lock with a blocking acquire(),
and the render thread holds that lock while it runs the plugin's display().
A display() that never returned -- or a first frame still running on the
executor's lingering thread after its 30s timeout -- parked the worker for
good, and no plugin updated again.
- The worker waits at most PLUGIN_LOCK_TIMEOUT (5s, the bound unload_plugin
already uses), skips the busy plugin and records it through the normal
update-failure path as a hang (PluginBusyError), so repeats open its
circuit breaker. Log lines about it are rate-limited per plugin.
- display() is timed on every frame (two monotonic reads). Calls of 2s or
more are logged once a minute and counted in plugin health
(slow_call_count, last_slow_call); calls past the executor timeout, and a
first frame still running at it, are recorded as hangs (hang_count,
last_hang) and no longer as successes. An update() still running after
its timeout is recorded as a hang too.
- on_config_change() runs under the plugin's lock via
PluginManager.apply_config_change(); if the lock stays busy the latest
change is deferred to the update worker, applied as soon as the lock
frees and before the plugin's next update() at the latest. The plugin API
is unchanged. Which thread runs each hook is documented in
PluginManager.__init__.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
