fix(display): retry a plugin that is enabled but failed to load (#495)

* fix(display): retry a plugin that is enabled but failed to load

A plugin whose validate_config() returns False is treated as a hard load
failure. The API then reports enabled=true, loaded=false, error=null: the
plugin is simply absent, with nothing saying why. hockey-scoreboard sat in
that state on a live rig for four days.

The recovery path existed but could not be reached. _reconcile_enabled_plugins
computes to_add = desired - current, and a plugin that failed to load is never
in current, so it stays in to_add and would be retried. But the reconcile is
queued by _enabled_set_changed(), which compares only top-level `enabled`
flags -- and the edit that actually fixes such a plugin (enabling a league,
filling in an API key) is nested inside the plugin's own config section. No
top-level flag changes, so no reconcile is queued, and the save that should
have fixed it does nothing. Only toggling some unrelated plugin -- which does
change a top-level flag -- queues the global reconcile that recovers it.

Add a second gate: queue a reconcile when a discovered plugin is enabled in
config but absent from the running set.

It is deliberately narrow rather than "reconcile on any config change".
Reconcile calls discover_plugins(), a ~39-manifest filesystem scan, and it
runs on the render thread; doing that on every config save would trade this
bug for a frame hitch. Gating on plugin_manifests also keeps non-plugin
sections that carry their own `enabled` flag (schedule, display) from
queueing a reconcile they can never satisfy. In the steady state -- every
enabled plugin loaded -- the new check is False and costs nothing.

The same valid-but-unconfigured => hard-fail shape still exists in
text-display, youtube-stats, birdnet-go, ledmatrix-flights and
mqtt-notifications; this makes all of them recoverable without a restart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW

* fix(display): snapshot the plugin mappings under their locks

Addresses the review finding on the cross-thread reads.

_enabled_plugin_not_running runs on the config-watcher thread and read two
mappings the render thread mutates. Catching RuntimeError was not a fix: it
turned a torn read into a coin flip between an unnecessary discovery scan and
a missed retry, which is the bug this PR exists to remove.

Both reads are now snapshots taken under the lock that guards their writes:

- plugin_manifests via a new PluginManager.discovered_plugin_ids(), which
  copies the ids while holding the existing _discovery_lock. Discovery
  rebuilds that mapping entry by entry, so an unsynchronised reader can see
  it half-populated.
- plugin_display_modes under a new controller lock, taken at the only two
  sites that mutate it (_register_loaded_plugin / _unregister_plugin).

The locks are never nested -- each snapshot is taken and released before the
next -- so this cannot deadlock against discovery, which holds _discovery_lock
while it rebuilds.

No cost on the per-frame path. Both mutation sites run during reconcile, which
is rare, and every hot-path read of plugin_display_modes is on the render
thread itself, same thread as the writes, so those stay lock-free.

Tests: the accessor returns a snapshot rather than a live view, and actually
takes the discovery lock (proved from a second thread, since an RLock is
reentrant on the owning one) so a later refactor cannot quietly drop it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW

* fix(display): consume the reconcile request before serving it

Addresses the second review finding: a lost update on
_pending_plugin_reconcile.

The flag was cleared after a successful reconcile. Reconcile has already read
its config by that point, so a config change arriving mid-flight set a flag
that the trailing clear then erased -- a request that was never served, and
the newest config never reconciled. That is the same "my save did nothing"
symptom this PR exists to remove, so leaving it would have undercut the fix.

Consume the request before running it instead, and re-arm only on a retryable
failure. A change that lands during reconcile now stays set and is picked up
on the next pass.

The per-frame read stays lock-free. It is a fast path that can only produce a
false negative -- the watcher setting the flag just after it is read is seen
on the next iteration -- never a false positive that loses a request. The lock
is taken only when a reconcile is actually pending or a config change arrives.

Extracted _service_pending_reconcile() so the sequence is testable rather than
buried in run()'s loop; the review asked for a regression test that invokes
the subscriber during reconciliation, which is not reachable otherwise.

Tests: 4 new, covering a request racing in mid-reconcile, the quiet success,
the retryable-failure re-arm, and not reconciling when nothing is pending.
Two of them fail against the previous clear-after-success semantics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

src/display_controller.py

@@ -181,6 +181,16 @@ class DisplayController:
         self.plugin_modes = {}  # mode -> plugin_instance mapping for plugin-first dispatch
         self.mode_to_plugin_id: Dict[str, str] = {}
         self.plugin_display_modes: Dict[str, List[str]] = {}
+        # plugin_display_modes is mutated only by _register_loaded_plugin /
+        # _unregister_plugin on the render thread, but the config-watcher
+        # thread reads it in _enabled_plugin_not_running. Both mutation sites
+        # run during reconcile (rare), so this lock never touches the per-frame
+        # path -- the hot-path reads are same-thread as the writes.
+        self._plugin_modes_lock = threading.Lock()
+        # Guards the consume-and-clear of _pending_plugin_reconcile. Only taken
+        # when a reconcile is actually pending or a config change arrives, both
+        # rare -- the per-frame path just reads the bool.
+        self._reconcile_flag_lock = threading.Lock()
         # Per-plugin config-change callbacks, kept so we can unsubscribe a
         # plugin when it is disabled live.
         self._plugin_config_callbacks: Dict[str, Callable] = {}
@@ -463,8 +473,10 @@ class DisplayController:
             self._refresh_config_cache(new_config)
             # If a plugin was enabled/disabled, flag a reconcile for the main
             # loop to apply (loading/unloading off the watcher thread is unsafe).
-            if self._enabled_set_changed(old_config, new_config):
-                self._pending_plugin_reconcile = True
+            if (self._enabled_set_changed(old_config, new_config)
+                    or self._enabled_plugin_not_running(new_config)):
+                with self._reconcile_flag_lock:
+                    self._pending_plugin_reconcile = True
 
         self.config_service.subscribe(_controller_config_change)
 
@@ -1749,11 +1761,12 @@ class DisplayController:
                 # rebuilding available_modes happens here on the render thread so
                 # it can't race with rendering. Deferred while on-demand is active
                 # (the flag stays set) so we don't fight its temporary-enable.
+                # The lock-free read is a fast path only; it can be a false
+                # negative (the watcher setting the flag just after it is read
+                # is seen next iteration), never a false positive that loses a
+                # request.
                 if self._pending_plugin_reconcile and not self.on_demand_active:
-                    # Only clear the flag on success -- a retryable failure
-                    # (e.g. discovery) leaves it set so the request isn't lost.
-                    if self._reconcile_enabled_plugins():
-                        self._pending_plugin_reconcile = False
+                    self._service_pending_reconcile()
 
                 if not self.available_modes:
                     # Nothing to render yet. Re-check _pending_plugin_reconcile
@@ -2813,7 +2826,8 @@ class DisplayController:
             logger.debug("Using manifest display_modes for %s: %s", plugin_id, display_modes)
         if not (isinstance(display_modes, list) and display_modes):
             display_modes = [plugin_id]
-        self.plugin_display_modes[plugin_id] = list(display_modes)
+        with self._plugin_modes_lock:
+            self.plugin_display_modes[plugin_id] = list(display_modes)
 
         # Subscribe to config changes for per-plugin hot-reload. Bind plugin_id
         # and instance as defaults so each plugin's callback targets its own
@@ -2847,7 +2861,8 @@ class DisplayController:
     def _unregister_plugin(self, plugin_id: str) -> None:
         """Remove a plugin's modes, config subscription and instance, then
         unload it. Used by live disable hot-reload."""
-        modes = self.plugin_display_modes.pop(plugin_id, [])
+        with self._plugin_modes_lock:
+            modes = self.plugin_display_modes.pop(plugin_id, [])
         for mode in modes:
             if mode in self.available_modes:
                 self.available_modes.remove(mode)
@@ -2892,6 +2907,67 @@ class DisplayController:
             }
         return enabled_map(old_config) != enabled_map(new_config)
 
+    def _service_pending_reconcile(self) -> None:
+        """Consume a pending reconcile request and run it.
+
+        The request is consumed BEFORE reconciling, not cleared after. Clearing
+        after would drop any config change that lands while reconcile is
+        running: reconcile has already read its config by then, so the clear
+        erases a request it never served and the newest config never
+        reconciles -- the same "your save did nothing" failure this whole path
+        exists to prevent. Consuming first means such a request stays set and
+        is picked up on the next pass.
+
+        A retryable failure (e.g. discovery) re-arms the flag.
+        """
+        with self._reconcile_flag_lock:
+            pending = self._pending_plugin_reconcile
+            self._pending_plugin_reconcile = False
+        if pending and not self._reconcile_enabled_plugins():
+            with self._reconcile_flag_lock:
+                self._pending_plugin_reconcile = True
+
+    def _enabled_plugin_not_running(self, new_config: Dict[str, Any]) -> bool:
+        """True when a discovered plugin is enabled in config but not running.
+
+        ``_enabled_set_changed`` compares only top-level ``enabled`` flags, which
+        misses the case that strands a plugin: one whose ``validate_config()``
+        returned False is absent from the running set, and the edit that fixes it
+        (enabling a league, filling in an API key) lives *nested* inside that
+        plugin's own section. No top-level flag changes, so no reconcile is
+        queued, and the save that should have fixed it appears to do nothing --
+        only toggling some unrelated plugin recovers it. hockey-scoreboard sat
+        enabled-but-absent on a live rig for four days this way.
+
+        Deliberately narrow: it fires only for ids the plugin manager has
+        actually discovered, so non-plugin sections that carry their own
+        ``enabled`` flag (``schedule``, ``display``, ...) don't queue a reconcile
+        on every save. In the steady state -- everything enabled is loaded --
+        this is False and costs nothing. That matters because reconcile runs
+        ``discover_plugins()`` on the render thread, where a needless
+        filesystem scan per config save would show up as a frame hitch.
+
+        Runs on the config-watcher thread, so both mappings it reads are
+        snapshotted under the lock that guards their writes.
+        """
+        if self.plugin_manager is None:
+            return False
+        # Two snapshots, each taken under its own lock and never nested, so a
+        # half-written mapping is never observed and this can't deadlock
+        # against discovery (which holds the discovery lock while rebuilding).
+        try:
+            known = self.plugin_manager.discovered_plugin_ids()
+        except AttributeError:
+            # Older manager without the accessor: fall back to a plain read.
+            known = set(getattr(self.plugin_manager, 'plugin_manifests', ()) or ())
+        with self._plugin_modes_lock:
+            running = set(self.plugin_display_modes)
+        for key, value in new_config.items():
+            if (key in known and isinstance(value, dict)
+                    and value.get('enabled', False) and key not in running):
+                return True
+        return False
+
     def _reconcile_enabled_plugins(self) -> bool:
         """Load/unload plugins so the running set matches the enabled set in
         config. Runs on the main display thread (never the config-watcher

src/plugin_system/plugin_manager.py

@@ -631,6 +631,17 @@ class PluginManager:
         
         return self.load_plugin(plugin_id)
     
+    def discovered_plugin_ids(self) -> set:
+        """Snapshot of the discovered plugin ids, taken under the discovery lock.
+
+        Callers on other threads (the config watcher) must not iterate
+        ``plugin_manifests`` directly: discovery rebuilds it entry by entry, so
+        an unsynchronised reader can see a half-populated mapping or raise
+        "dictionary changed size during iteration".
+        """
+        with self._discovery_lock:
+            return set(self.plugin_manifests)
+
     def get_plugin(self, plugin_id: str) -> Optional[Any]:
         """
         Get a loaded plugin instance by ID.