fix(plugins): one hung plugin no longer stops every plugin from updating (#677)

The update worker no longer blocks forever on a plugin whose display()
never returns. It waits at most PLUGIN_LOCK_TIMEOUT (5s) for a plugin's
lock, then skips that plugin's update (a report-only "busy skip" in
health) and keeps updating every other plugin. display() frames are timed
(slow calls logged and counted; calls past the executor timeout recorded as
hangs), a hung update() is recorded, and on_config_change() now runs under
the plugin lock or is deferred to the worker. The plugin-facing API is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-30 09:10:41 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent e3c85cece6
commit c0d97e4867
10 changed files with 1077 additions and 28 deletions
+26
View File
@@ -101,6 +101,32 @@ accepts both, but the store flags the old spelling as deprecated
- A stop request now clears an on-demand error. After a failed request,
`/display/on-demand/status` kept reporting `status: error` for up to two
minutes even after a stop.
- One hung plugin no longer stops every plugin from updating. The single
update worker waited on each plugin's lock with no time limit, and the
render thread holds that lock while it runs the plugin's display(); a
display() that never returned (or a first frame still running after the
executor's 30s timeout) parked the worker for good, so scores, weather and
clocks all froze while the panel kept scrolling. The worker now waits at
most 5s (the bound `unload_plugin()` already uses) and skips that update;
the other plugins keep updating. The skip is logged (at most once a minute
per plugin) and counted in plugin health as a busy skip (`busy_skip_count`,
`last_busy_skip`), but it is not a failure and never opens the circuit
breaker: Vegas mode holds a plugin's lock for its whole content render,
which on a slow Pi can outlast 5s, and a healthy plugin must not be pulled
from rotation for that.
- display() calls are timed on every frame. One taking 2s or more is logged
(at most once a minute per plugin) and counted in plugin health
(`slow_call_count`, `last_slow_call`); one that runs past the executor's
timeout counts as a hang (`hang_count`, `last_hang`) and as a failure to
the circuit breaker. A first frame that times out is no longer recorded as
a success, and an update() still running after its timeout is recorded as
a hang instead of leaving the plugin silently stuck. Only these real hangs
count toward the breaker.
- A plugin's `on_config_change()` no longer runs while its update() is
running on the worker thread. It now runs under the plugin's lock; if the
lock stays busy past the same 5s bound the change is handed to the update
worker, which applies the latest one as soon as the lock frees, and before
the plugin's next update() at the latest. The plugin API is unchanged.
## 3.7.0