fix(plugins): sub-package reload, symlinked dev plugins, BaseException in update(), config callbacks outside the lock (#741)

* fix(plugins): drop a plugin's package modules when it unloads

A plugin that keeps helpers in a package (providers/feed.py, imported as
`from providers.feed import ...`) leaves dotted entries in sys.modules.
PluginLoader only tracked bare names: `providers` was namespaced and
dropped on unload, `providers.feed` stayed. A reload after a store update
imported a fresh `providers`, then got the old `feed` back from the module
cache, so the new manager.py ran against the old helpers until the display
restarted. A load that failed part-way left them behind the same way.
Elections (providers/), flights (enrichment/) and olympics (data/,
renderers/) ship packages.

The loader now records the dotted modules whose file (or, for a namespace
package, every __path__ entry) lies inside the plugin directory. They keep
their names while the plugin runs, as before, and unregister_plugin_modules()
drops them, only while sys.modules still holds that plugin's module. The
failed-load cleanup in load_module() drops them too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): remove a symlinked dev plugin as a link

PluginStoreManager._safe_remove_directory, behind uninstall and behind
discarding the set-aside copy after an install or update, handed a
symlinked dev plugin (scripts/dev/dev_plugin_setup.sh) to shutil.rmtree,
which refuses a symlink. The chmod fallback then walked through the link
and set every directory and file in the linked checkout to 0700, and the
sudo stage refused the resolved path as outside the plugins directory. The
removal failed, the link stayed, and the developer's checkout lost its
group/other permissions. A dangling link read as already removed, because
exists() follows it, and was left behind.

A symlink is now unlinked before any other stage runs, and before the
exists() check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): load a dev plugin linked in under a different name

contained_plugin_dir(), the containment check before a plugin's
dependencies are installed, resolved the plugin directory and looked for
the resolved folder's name among the plugins directory's entries. A dev
plugin symlinked in under its id by a name its checkout does not share --
`dev_plugin_setup.sh link-github foo <url>` clones ledmatrix-foo, the
repository naming convention, and links it as plugins/foo -- has no such
entry, so install_dependencies() returned False and the load failed with
"Dependency installation failed", even with no requirements.txt.

When the path sits directly in the plugins directory, the entry it names
(the link) is looked up first; anything else is resolved and matched by
name as before. The answer is still always rebuilt from a name os.scandir()
returned for the plugins directory, so a path outside it is still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(plugins): release a plugin whose update() raises a BaseException

On the async update worker, the wrapped update() finished its bookkeeping
(_finish: release the plugin lock, drop the pending slot, state back to
ENABLED) only for an Exception. asyncio.CancelledError and SystemExit
derive from BaseException, so one raised from update() skipped _finish:
the plugin kept its lock and stayed RUNNING for the life of the process,
never rescheduled, with every display() skipped as busy. PluginExecutor
caught only Exception as well, so its thread died with the call never
marked complete and an immediate failure was logged and recorded as a
timeout.

_target_update now runs _finish for any BaseException and re-raises it,
and the executor's thread stores it like any other exception, so it is
reported as the operation's failure (PluginError) on both the async and
the synchronous path. _finish and _record_update_failure take a
BaseException.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(config): notify config subscribers outside the service lock

ConfigService._load_config ran every subscriber while holding _lock. The
display's per-plugin subscriber calls PluginManager.apply_config_change,
which waits up to PLUGIN_LOCK_TIMEOUT (5 s) for a plugin busy in update().
A save that enables or disables a plugin also flags a reconcile, which the
render thread runs: its get_config(), and the unsubscribe() of a plugin it
disables, both take _lock, so the panel froze behind every slow callback,
up to 5 s per busy plugin.

The config is now swapped under _lock and the subscribers are called after
it is released, from a copy of the subscriber lists. A separate
_notify_lock is held across a whole reload (read, swap, notify), so one
reload's notifications still finish before the next one's start. Each
callback is checked against the live lists just before it runs, and
unsubscribe() waits only for a call of that same callback already in
progress (unless it is that callback's own thread), so a callback it
removed is not running and will not run once it returns, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-10-03 22:18:42 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 04f0d8d134
commit d18e4d3c9d
12 changed files with 730 additions and 51 deletions
+27
View File
@@ -13,6 +13,7 @@ The invariants that keep this change safe:
inline path exactly.
"""
import asyncio
import os
import sys
import threading
@@ -209,6 +210,32 @@ class TestFailurePaths:
assert pm.get_plugin_lock(plugin_id).acquire(blocking=False) is True
pm.get_plugin_lock(plugin_id).release()
@pytest.mark.parametrize("raised", [asyncio.CancelledError, SystemExit])
def test_update_raising_a_base_exception_still_releases_the_plugin(self, pm, raised):
"""asyncio.CancelledError and SystemExit derive from BaseException,
not Exception. Raised from update() on the worker, one skipped the
bookkeeping entirely: the plugin kept its lock and stayed RUNNING for
the life of the process -- never updated again, and every display()
skipped as busy."""
class CancellingPlugin(SlowPlugin):
def update(self):
self.update_calls += 1
raise raised()
plugin_id = _install(pm, CancellingPlugin())
pm.run_scheduled_updates()
deadline = time.monotonic() + 3
while pm.plugins[plugin_id].update_calls == 0 and time.monotonic() < deadline:
time.sleep(0.05)
time.sleep(0.2)
assert pm.get_plugin_lock(plugin_id).acquire(blocking=False) is True
pm.get_plugin_lock(plugin_id).release()
assert pm.state_manager.can_execute(plugin_id) is True
assert pm.plugin_last_update.get(plugin_id, 0) > 0
error = pm.state_manager.get_error_info(plugin_id)
assert error is not None and error["error_type"] == raised.__name__
def test_unloaded_while_queued_is_harmless(self, pm):
"""Exercise the public unload_plugin() lifecycle rather than
deleting pm.plugins directly: queue the target's update behind a
+199
View File
@@ -0,0 +1,199 @@
"""ConfigService notifies subscribers outside its lock, in order.
Subscribers ran while _load_config held the service's lock. The display's
per-plugin subscriber is PluginManager.apply_config_change, which waits up to
PLUGIN_LOCK_TIMEOUT (5 s) for a busy plugin. The same save that toggles a
plugin's ``enabled`` flags a reconcile, and the render thread runs it: its
get_config() -- and the unsubscribe() of a plugin it disables -- waited behind
every slow callback, freezing the panel for up to 5 s per busy plugin.
What callers could rely on before still holds: one reload's notifications
finish before the next reload's start, and a callback unsubscribe() removed is
not running, and will not run, once unsubscribe() returns.
"""
import itertools
import json
import os
import threading
import time
import pytest
from src.config_manager import ConfigManager
from src.config_service import ConfigService
SLOW = 2.0 # how long a blocked callback waits before giving up
@pytest.fixture
def service(tmp_path):
config_path = tmp_path / "config.json"
config_path.write_text(json.dumps({"display": {"brightness": 50},
"weather": {"enabled": True}}),
encoding="utf-8")
manager = ConfigManager(str(config_path), str(tmp_path / "config_secrets.json"))
manager.template_path = str(tmp_path / "no-template.json")
svc = ConfigService(manager, enable_hot_reload=False)
yield svc, config_path
svc.shutdown()
_saves = itertools.count(1)
def _save(config_path, **sections):
config = json.loads(config_path.read_text(encoding="utf-8"))
config.update(sections)
config_path.write_text(json.dumps(config), encoding="utf-8")
# ConfigManager re-reads only when (mtime, size) moves. Two quick saves of
# the same size can share an mtime tick (about 16 ms on Windows), so step
# it forward explicitly.
st = config_path.stat()
os.utime(config_path, ns=(st.st_atime_ns, st.st_mtime_ns + next(_saves) * 50_000_000))
def _reload_in_background(svc):
thread = threading.Thread(target=svc._load_config, daemon=True)
thread.start()
return thread
def test_get_config_does_not_wait_for_a_slow_subscriber(service):
svc, config_path = service
entered, release = threading.Event(), threading.Event()
def slow(_old, _new):
entered.set()
release.wait(SLOW)
svc.subscribe(slow, plugin_id="weather")
_save(config_path, weather={"enabled": False})
reload = _reload_in_background(svc)
assert entered.wait(SLOW)
start = time.monotonic()
config = svc.get_config()
waited = time.monotonic() - start
release.set()
reload.join(SLOW)
assert waited < 0.5
# Swapped before anyone was told: a subscriber that reads it sees the new one.
assert config["weather"]["enabled"] is False
def test_unsubscribing_another_callback_does_not_wait(service):
svc, config_path = service
entered, release = threading.Event(), threading.Event()
def slow(_old, _new):
entered.set()
release.wait(SLOW)
def other(_old, _new):
pass
svc.subscribe(slow, plugin_id="weather")
svc.subscribe(other, plugin_id="clock")
_save(config_path, weather={"enabled": False})
reload = _reload_in_background(svc)
assert entered.wait(SLOW)
start = time.monotonic()
svc.unsubscribe(other, plugin_id="clock")
waited = time.monotonic() - start
release.set()
reload.join(SLOW)
assert waited < 0.5
def test_a_callback_unsubscribed_mid_notification_is_not_called(service):
svc, config_path = service
entered, release = threading.Event(), threading.Event()
called = []
def slow_global(_old, _new): # global subscribers are notified first
entered.set()
release.wait(SLOW)
def weather(_old, _new):
called.append("weather")
svc.subscribe(slow_global)
svc.subscribe(weather, plugin_id="weather")
_save(config_path, weather={"enabled": False})
reload = _reload_in_background(svc)
assert entered.wait(SLOW)
svc.unsubscribe(weather, plugin_id="weather")
release.set()
reload.join(SLOW)
assert called == []
def test_unsubscribe_waits_for_its_own_callback_to_return(service):
svc, config_path = service
entered, release = threading.Event(), threading.Event()
returned = threading.Event()
def slow(_old, _new):
entered.set()
release.wait(SLOW)
returned.set()
svc.subscribe(slow, plugin_id="weather")
_save(config_path, weather={"enabled": False})
reload = _reload_in_background(svc)
assert entered.wait(SLOW)
threading.Timer(0.2, release.set).start()
svc.unsubscribe(slow, plugin_id="weather")
assert returned.is_set()
reload.join(SLOW)
def test_a_callback_may_read_config_and_unsubscribe_itself(service):
svc, config_path = service
seen = []
def once(_old, _new):
seen.append(svc.get_config()["weather"]["enabled"])
svc.unsubscribe(once, plugin_id="weather")
svc.subscribe(once, plugin_id="weather")
_save(config_path, weather={"enabled": False})
reload = _reload_in_background(svc)
reload.join(SLOW)
assert not reload.is_alive()
assert seen == [False]
def test_two_reloads_notify_in_order(service):
svc, config_path = service
entered, release = threading.Event(), threading.Event()
seen = []
def record(old, new):
seen.append((old["brightness"], new["brightness"]))
if len(seen) == 1:
entered.set()
release.wait(SLOW)
svc.subscribe(record, plugin_id="display")
_save(config_path, display={"brightness": 60})
first = _reload_in_background(svc)
assert entered.wait(SLOW)
_save(config_path, display={"brightness": 100})
second = _reload_in_background(svc)
time.sleep(0.2)
release.set()
first.join(SLOW)
second.join(SLOW)
assert seen == [(50, 60), (60, 100)]
@@ -58,3 +58,83 @@ def test_a_reloaded_plugin_still_gets_its_own_bare_module(plugins):
assert reloaded.WHO == "alpha"
assert sys.path.index(str(plugins["alpha"])) < sys.path.index(str(plugins["beta"]))
assert sys.path.count(str(plugins["alpha"])) == 1
# -- sub-packages ------------------------------------------------------------
#
# A plugin that keeps helpers in a package (``providers/feed.py``, imported as
# ``from providers.feed import ...``) leaves dotted entries in sys.modules.
# Only the bare ``providers`` used to be tracked, so ``providers.feed`` outlived
# the plugin: a reload after a store update re-ran the new manager.py against
# the old feed.py, until the display restarted. Elections (providers/),
# flights (enrichment/) and olympics (data/, renderers/) ship packages.
@pytest.fixture
def package_plugin(tmp_path):
before_path = list(sys.path)
before_modules = set(sys.modules)
plugin_dir = tmp_path / "pkgdemo"
(plugin_dir / "providers").mkdir(parents=True)
(plugin_dir / "providers" / "__init__.py").write_text("", encoding="utf-8")
(plugin_dir / "providers" / "feed.py").write_text("VERSION = 'v1'\n", encoding="utf-8")
(plugin_dir / "manager.py").write_text(
"from providers.feed import VERSION\n", encoding="utf-8")
yield plugin_dir
sys.path[:] = before_path
for key in set(sys.modules) - before_modules:
sys.modules.pop(key, None)
def test_a_reloaded_plugin_runs_its_updated_subpackage_module(package_plugin):
loader = PluginLoader()
assert loader.load_module("pkgdemo", package_plugin, "manager.py").VERSION == "v1"
_unload(loader, "pkgdemo")
# The store update: a different size, so no cached bytecode can match.
(package_plugin / "providers" / "feed.py").write_text(
"VERSION = 'v2 from the update'\n", encoding="utf-8")
reloaded = loader.load_module("pkgdemo", package_plugin, "manager.py")
assert reloaded.VERSION == "v2 from the update"
def test_unload_drops_the_plugins_subpackage_modules(package_plugin):
loader = PluginLoader()
loader.load_module("pkgdemo", package_plugin, "manager.py")
# Still importable while the plugin runs, as before.
assert "providers.feed" in sys.modules
_unload(loader, "pkgdemo")
assert not [k for k in sys.modules if k.startswith("providers")]
def test_a_failed_load_leaves_no_subpackage_module_behind(package_plugin):
(package_plugin / "manager.py").write_text(
"from providers.feed import VERSION\nraise RuntimeError('broken')\n",
encoding="utf-8")
loader = PluginLoader()
with pytest.raises(RuntimeError):
loader.load_module("pkgdemo", package_plugin, "manager.py")
assert not [k for k in sys.modules if k.startswith("providers")]
def test_unload_leaves_packages_from_outside_the_plugin_alone(package_plugin, tmp_path):
# A library the plugin imports is not the plugin's to drop.
lib_root = tmp_path / "site"
(lib_root / "extlib").mkdir(parents=True)
(lib_root / "extlib" / "__init__.py").write_text("", encoding="utf-8")
(lib_root / "extlib" / "sub.py").write_text("X = 1\n", encoding="utf-8")
sys.path.append(str(lib_root))
(package_plugin / "manager.py").write_text(
"import extlib.sub\nfrom providers.feed import VERSION\n", encoding="utf-8")
loader = PluginLoader()
loader.load_module("pkgdemo", package_plugin, "manager.py")
_unload(loader, "pkgdemo")
assert "extlib.sub" in sys.modules
assert "extlib" in sys.modules
+81
View File
@@ -0,0 +1,81 @@
"""A dev plugin linked in under a name its checkout does not share still loads.
``scripts/dev/dev_plugin_setup.sh`` links a checkout into the plugins
directory under the plugin's id: ``link-github foo <url>`` clones
``ledmatrix-foo`` (the repository naming convention) and links it as
``plugins/foo``. ``contained_plugin_dir`` resolved the link and looked for the
*target's* folder name, ``ledmatrix-foo``, among the plugins directory's
entries. There is none, so ``install_dependencies`` refused the plugin as
outside the plugins directory and the load failed with "Dependency
installation failed" -- even with no requirements.txt at all.
The containment it exists for still holds: the answer is always rebuilt from
an entry enumerated under the plugins directory.
Skipped where this process cannot create a symlink (Windows without the
privilege).
"""
import os
from unittest.mock import MagicMock, patch
import pytest
from src.plugin_system.plugin_loader import PluginLoader, contained_plugin_dir
def _symlink_or_skip(target, link):
try:
os.symlink(target, link, target_is_directory=True)
except (OSError, NotImplementedError) as e:
pytest.skip(f"cannot create a symlink here: {e}")
@pytest.fixture
def linked(tmp_path):
checkout = tmp_path / "dev-plugins" / "ledmatrix-foo"
checkout.mkdir(parents=True)
plugins_dir = tmp_path / "plugins"
plugins_dir.mkdir()
link = plugins_dir / "foo"
_symlink_or_skip(checkout, link)
return plugins_dir, link, checkout
def test_a_link_resolves_to_its_own_entry_in_the_plugins_dir(linked):
plugins_dir, link, _checkout = linked
assert contained_plugin_dir(link, plugins_dir) == os.path.join(
os.path.realpath(plugins_dir), "foo")
def test_a_linked_plugin_without_requirements_needs_no_install(linked):
plugins_dir, link, _checkout = linked
with patch("subprocess.run") as pip:
assert PluginLoader().install_dependencies(link, "foo", plugins_dir=plugins_dir) is True
pip.assert_not_called()
@patch("src.plugin_system.plugin_loader.requirements_are_satisfied", return_value=False)
def test_a_linked_plugins_requirements_are_installed_through_the_link(_satisfied, linked):
plugins_dir, link, checkout = linked
(checkout / "requirements.txt").write_text("package1==1.0.0\n", encoding="utf-8")
with patch("subprocess.run", return_value=MagicMock(returncode=0, stderr="")) as pip:
assert PluginLoader().install_dependencies(link, "foo", plugins_dir=plugins_dir) is True
argv = pip.call_args[0][0]
assert argv[argv.index("-r") + 1] == os.path.join(
os.path.realpath(plugins_dir), "foo", "requirements.txt")
def test_a_link_outside_the_plugins_dir_is_still_refused(linked, tmp_path):
plugins_dir, _link, checkout = linked
elsewhere = tmp_path / "elsewhere"
elsewhere.mkdir()
stray = elsewhere / "bar"
_symlink_or_skip(checkout, stray)
assert contained_plugin_dir(stray, plugins_dir) is None
assert contained_plugin_dir(plugins_dir / ".." / "elsewhere" / "bar", plugins_dir) is None
+18
View File
@@ -189,6 +189,24 @@ class TestPluginExecutor:
assert result is False
def test_a_base_exception_is_a_failure_not_a_timeout(self):
"""asyncio.CancelledError derives from BaseException. Uncaught on
the executor's thread it ended the thread with the call never marked
complete, so a call that failed at once was reported, and recorded,
as timing out."""
import asyncio
import pytest
from src.exceptions import PluginError
from src.plugin_system.plugin_executor import PluginExecutor
executor = PluginExecutor(default_timeout=5.0)
def cancelled():
raise asyncio.CancelledError()
with pytest.raises(PluginError) as raised:
executor.execute_with_timeout(cancelled, plugin_id="test_plugin")
assert isinstance(raised.value.__cause__, asyncio.CancelledError)
class TestPluginHealth:
"""Test plugin health monitoring."""
+90
View File
@@ -0,0 +1,90 @@
"""Removing a dev plugin linked into the plugins directory removes the link.
``scripts/dev/dev_plugin_setup.sh`` symlinks a checkout into the plugins
directory. ``PluginStoreManager._safe_remove_directory`` -- behind uninstall,
and behind discarding the set-aside copy after an install or update -- handed
the link to ``shutil.rmtree``, which refuses a symlink. Its fallback then
walked through the link and chmodded every directory and file of the linked
checkout to 0700, and the sudo stage refused a path outside the plugins
directory. So the uninstall failed, the link stayed, and the developer's
checkout lost its group/other permissions and gained execute bits.
Skipped where this process cannot create a symlink (Windows without the
privilege).
"""
import json
import os
from unittest.mock import MagicMock
import pytest
from src.plugin_system.store_manager import PluginStoreManager
PLUGIN_ID = "linked-demo"
def _symlink_or_skip(target, link):
try:
os.symlink(target, link, target_is_directory=True)
except (OSError, NotImplementedError) as e:
pytest.skip(f"cannot create a symlink here: {e}")
@pytest.fixture
def linked(tmp_path):
checkout = tmp_path / "dev-plugins" / PLUGIN_ID
checkout.mkdir(parents=True)
(checkout / "manifest.json").write_text(
json.dumps({"id": PLUGIN_ID, "name": "Linked", "class_name": "P",
"display_modes": ["linked"], "version": "1.0.0"}),
encoding="utf-8")
(checkout / "manager.py").write_text("X = 1\n", encoding="utf-8")
plugins_dir = tmp_path / "plugins"
plugins_dir.mkdir()
link = plugins_dir / PLUGIN_ID
_symlink_or_skip(checkout, link)
store = PluginStoreManager(plugins_dir=str(plugins_dir))
store.logger = MagicMock()
return store, link, checkout
def test_removing_a_linked_plugin_removes_only_the_link(linked):
store, link, checkout = linked
assert store._safe_remove_directory(link) is True
assert not os.path.lexists(link)
assert (checkout / "manager.py").read_text(encoding="utf-8") == "X = 1\n"
@pytest.mark.skipif(os.name != "posix", reason="POSIX permission bits")
def test_removing_a_linked_plugin_leaves_the_checkouts_permissions(linked):
store, link, checkout = linked
os.chmod(checkout, 0o755)
os.chmod(checkout / "manager.py", 0o644)
store._safe_remove_directory(link)
assert checkout.stat().st_mode & 0o777 == 0o755
assert (checkout / "manager.py").stat().st_mode & 0o777 == 0o644
def test_uninstalling_a_linked_plugin_removes_the_link(linked):
store, link, checkout = linked
assert store.uninstall_plugin(PLUGIN_ID) is True
assert not os.path.lexists(link)
assert (checkout / "manifest.json").exists()
def test_a_dangling_link_is_removed_too(linked):
store, link, checkout = linked
for child in checkout.iterdir():
child.unlink()
checkout.rmdir()
assert store._safe_remove_directory(link) is True
assert not os.path.lexists(link)