Merge origin/main into claude/plugin-hang-containment

Conflict with #678 in the plugin config-change callback: keep #678's
override (a plugin loaded for on-demand sees enabled: True) and apply the
result through the locked apply_config_change. The #678 test now asserts
on that call, which delivers the override to on_config_change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -19,6 +19,101 @@ accepts both, but the store flags the old spelling as deprecated
 
 ## Unreleased
 
+### Fixes
+
+- On-demand no longer restarts a running display. `POST
+  /display/on-demand/start` treated `start_service` (on by default, and what
+  "Preview on display", the on-demand dialog and the MQTT bridge all send) as
+  "restart": it stopped the service, waited 1.5s and started it again, so
+  every request reloaded every plugin and left the panel blank for seconds.
+  The running display already reads the request within a quarter of a second,
+  mid-screen and mid-Vegas included, so the route now only starts the service
+  when it is not running. `POST /display/on-demand/stop` reads
+  `stop_service` as a boolean, so `"false"` no longer stops the service.
+- On-demand works for a disabled plugin. The display only loads enabled
+  plugins, so "Preview on display" on a disabled plugin's config page (which
+  says the plugin will be enabled for the preview) failed with
+  `invalid-mode`. The display now loads the plugin live for the session,
+  without writing `enabled` to `config.json`, and unloads it when on-demand
+  is stopped, expires or moves to another plugin. A plugin that fails to
+  load reports on-demand status `error` with `load-failed`. A session
+  restored after a restart unloads its disabled plugin the same way; it used
+  to stay loaded until the next restart.
+- A stop request now clears an on-demand error. After a failed request,
+  `/display/on-demand/status` kept reporting `status: error` for up to two
+  minutes even after a stop.
+- One hung plugin no longer stops every plugin from updating. The single
+  update worker waited on each plugin's lock with no time limit, and the
+  render thread holds that lock while it runs the plugin's display(); a
+  display() that never returned (or a first frame still running after the
+  executor's 30s timeout) parked the worker for good, so scores, weather and
+  clocks all froze while the panel kept scrolling. The worker now waits at
+  most 5s (the bound `unload_plugin()` already uses) and skips that update;
+  the other plugins keep updating. The skip is logged (at most once a minute
+  per plugin) and counted in plugin health as a busy skip (`busy_skip_count`,
+  `last_busy_skip`), but it is not a failure and never opens the circuit
+  breaker: Vegas mode holds a plugin's lock for its whole content render,
+  which on a slow Pi can outlast 5s, and a healthy plugin must not be pulled
+  from rotation for that.
+- display() calls are timed on every frame. One taking 2s or more is logged
+  (at most once a minute per plugin) and counted in plugin health
+  (`slow_call_count`, `last_slow_call`); one that runs past the executor's
+  timeout counts as a hang (`hang_count`, `last_hang`) and as a failure to
+  the circuit breaker. A first frame that times out is no longer recorded as
+  a success, and an update() still running after its timeout is recorded as
+  a hang instead of leaving the plugin silently stuck. Only these real hangs
+  count toward the breaker.
+- A plugin's `on_config_change()` no longer runs while its update() is
+  running on the worker thread. It now runs under the plugin's lock; if the
+  lock stays busy past the same 5s bound the change is handed to the update
+  worker, which applies the latest one as soon as the lock frees, and before
+  the plugin's next update() at the latest. The plugin API is unchanged.
+
+## 3.7.0
+
+Sports consolidation stage 3 (#672). No behaviour change: nothing in core
+uses these yet, and the scoreboards adopt them when they floor on 3.7.0.
+
+### New modules
+
+A plugin may import these via `src.*` (floor on 3.7.0). All three hold code
+the scoreboard plugins carry as identical copies, moved without behaviour
+change under the plugins' own method names; each docstring lists what the
+host class must provide. The plugins delete their copies when they floor on
+3.7.0.
+
+- `src/common/sports_celebration.py` — `SportsCelebrationMixin`, the
+  score/win celebration takeover drawn by afl, football, hockey, nrl and
+  soccer (`_draw_celebration_layout` and the palette, backdrop, scenery,
+  confetti and crest steps behind it), plus its colour helpers as free
+  functions: `logo_palette`, `lift_color`, `cap_luminance`, `mix_color`,
+  `scale_color`, `dim_rgba`, `rgb_luminance`, `rgb_saturation`,
+  `color_distance`. Only the drawing: when to celebrate, the phrase and the
+  scenery stay in each plugin.
+- `src/common/sports_fetch.py` — `SportsFetchMixin`, four `SportsCore`
+  methods identical in all nine scoreboards: `_fetch_season_directly`,
+  `_background_fetches_espn_ranges`, `_needs_previous_day` and
+  `_wants_live_odds` (with `_LOOKBACK_CUTOFF_HOUR` and
+  `_LIVE_ODDS_LOOKAHEAD`).
+- `src/common/sports_card_wrappers.py` — `SportsCardWrappersMixin`, the
+  seventeen `sports_card` delegations the eight scoreboard game renderers
+  carry (`_vs_text`, `_element_color`, `_format_game_date`, ...): the methods
+  `SportsGameRendererMixin` expects its host to provide.
+
+## 3.6.2
+
+A fix to `src.common.favorite_team_check` (#670).
+
+### Fixes
+
+- The favourite-team check no longer says the Europa League season has
+  finished between matchdays. Its scoreboard keeps showing the last matchday,
+  and its calendar is a "list" of rounds rather than match days, so neither
+  3.6.1 rule applied. When every event is past, a round in a list calendar
+  that has not started yet (outside an offseason phase) now draws no
+  conclusion. PLL, the World Cup and AFL, whose seasons are over, are still
+  reported as finished: no round of theirs is still to start. (#670)
+
 ## 3.6.1
 
 A fix to `src.common.favorite_team_check` (#667). Plugins that drop their

docs/ARCHITECTURE.md

@@ -52,8 +52,10 @@ each other. They share three things:
 | Preview viewer marker | `/tmp/led_matrix_preview_viewer` | web, while a preview is open | display: writes full-rate snapshots only while it is fresh |
 | Hardware init status | `/tmp/led_matrix_hw_status.json` | display | web: `/api/v3/hardware/status` |
 
-The on-demand start route also restarts `ledmatrix.service` by default so the
-request takes effect straight away.
+The on-demand start route starts `ledmatrix.service` when it is not running
+(`start_service`, on by default) but never restarts a running one: the display
+reads the mailbox every `ON_DEMAND_POLL_INTERVAL` (0.25s), from its dwell
+sleep, its render loops and Vegas's interrupt check as well as the main loop.
 
 ## Display loop
 
@@ -82,7 +84,11 @@ then normal rotation.
 - **On-demand.** A request from the web interface pins one plugin (or mode)
   for a duration. `_activate_on_demand()` / `_clear_on_demand()`; the
   session is saved under `display_on_demand_config` so it survives a
-  restart. It also keeps the display on during scheduled off hours.
+  restart. It also keeps the display on during scheduled off hours. A
+  request for a plugin that is disabled in config loads it live
+  (`_load_plugin_for_on_demand()`, `load_plugin(force_enabled=True)`)
+  without writing `config.json`; the main loop unloads it once on-demand
+  moves off it (`_release_on_demand_plugins()`).
 - **Live priority.** `_check_live_priority()` looks for a plugin whose
   `has_live_priority()` and `has_live_content()` are both true and switches
   to it, rotating between several live games.
@@ -99,7 +105,8 @@ then normal rotation.
   changes. The controller refreshes its cached settings; enabling or
   disabling a plugin queues `_reconcile_enabled_plugins()`, which loads or
   unloads it on the display thread; each plugin gets `on_config_change()`
-  for its own section. Set `LEDMATRIX_HOT_RELOAD=false` to turn this off.
+  for its own section, under its plugin lock
+  (`PluginManager.apply_config_change()`). Set `LEDMATRIX_HOT_RELOAD=false` to turn this off.
   Matrix hardware settings are only read at start-up.
 - **Vegas mode.** [`src/vegas_mode/`](../src/vegas_mode/): the display loop
   calls `VegasModeCoordinator.run_iteration()`

docs/PLUGIN_API_REFERENCE.md

@@ -151,6 +151,12 @@ Clean up resources when plugin is unloaded. Override to close connections, stop
 
 Called after plugin configuration is updated via web API.
 
+In the display service it runs on the config watcher thread while holding
+the plugin's lock, so it never overlaps your `update()` or `display()`. If
+the plugin stays busy for more than 5 seconds, the change is applied later
+from the update thread: as soon as the plugin is free, and before its next
+`update()` at the latest.
+
 #### `on_enable() -> None`
 
 Called when plugin is enabled.

docs/REST_API_REFERENCE.md

@@ -390,7 +390,7 @@ Request a specific plugin to display on-demand.
 - `mode` (string, optional): Display mode name (plugin_id inferred if not provided)
 - `duration` (number, optional): Duration in seconds (0 = until stopped)
 - `pinned` (boolean, optional): Pin display (pause rotation)
-- `start_service` (boolean, optional): (Re)start the display service so it picks the request up (default: true)
+- `start_service` (boolean, optional): Start the display service if it is not running (default: true). A running service is never restarted: it picks the request up within about a quarter of a second. When false and the service is stopped, the route returns 400.
 
 **Response**:
 ```json

docs/SPORTS_UNIFICATION.md

@@ -83,6 +83,9 @@ more. Shared sports code lives in `src/common`:
 | `espn_dates.py` | 3.5.0 | ESPN date-range and `limit` workarounds |
 | `favorite_team_check.py` | 3.6.0 | `FavoriteTeamCheck` — logs why a favourite team code shows nothing |
 | `sports_timezone.py` | 3.6.0 | Which timezone start times are drawn in (`resolve_timezone_name`) |
+| `sports_celebration.py` | 3.7.0 | `SportsCelebrationMixin` — draws the score/win takeover; colour helpers |
+| `sports_fetch.py` | 3.7.0 | `SportsFetchMixin` — season fetch, live lookback and live-odds decisions |
+| `sports_card_wrappers.py` | 3.7.0 | `SportsCardWrappersMixin` — the game renderer's `sports_card` delegations |
 
 Each is described in [src/common/README.md](../src/common/README.md).
 
@@ -168,6 +171,15 @@ Mix it in **before** the mode class — `class SoccerLive(CelebrationMixin,
 SportsLive)` — so the celebration `display()` runs first and falls through to
 the scorebug via `super()`.
 
+What shipped is narrower. `src/common/sports_celebration.py`
+(`SportsCelebrationMixin`) holds only the drawing, which is identical in the
+five scoreboards that celebrate (afl, football, hockey, nrl, soccer — hockey
+grew celebrations after this was written). Arming a celebration stays in each
+plugin: the trigger bodies differ (nrl matches favourites by team id, football
+folds a touchdown's extra point into one celebration and picks scenery by
+points), and so does `display()`. The seams above were not needed to move the
+drawing, so none was added.
+
 **Rotation strategies.** The three "dialects" turned out to be one algorithm
 (Smooth Weighted Round-Robin) in two shapes: an incremental picker holding state
 across calls (afl/nrl/soccer) and a precomputed per-cycle list

mypy-clean.txt

@@ -32,6 +32,9 @@ src/common/render_gate.py
 src/common/scroll_config.py
 src/common/snapshot_policy.py
 src/common/sports_card.py
+src/common/sports_card_wrappers.py
+src/common/sports_celebration.py
+src/common/sports_fetch.py
 src/common/sports_scroll.py
 src/common/sports_timezone.py
 src/config_service.py

src/__init__.py

@@ -4,5 +4,5 @@ LEDMatrix Display System
 Core source package for the LED Matrix Display project.
 """
 
-__version__ = "3.6.1"
+__version__ = "3.7.0"
 

src/common/README.md

@@ -38,6 +38,9 @@ Rules for the package:
 | [`scroll_helper`](#scroll_helper) | Pre-rendered horizontal scrolling | Yes | — |
 | [`snapshot_policy`](#snapshot_policy) | When to write the web preview frame | No, core-internal | n/a |
 | [`sports_card`](#sports_card) | Scoreboard card settings, colours, fonts, dates | Yes (scoreboards) | 3.3.0 |
+| [`sports_card_wrappers`](#sports_card_wrappers) | The game renderer's `sports_card` delegations | Yes (scoreboards) | 3.7.0 |
+| [`sports_celebration`](#sports_celebration) | Draw a scoreboard's score/win celebration | Yes (scoreboards) | 3.7.0 |
+| [`sports_fetch`](#sports_fetch) | Scoreboard season fetch, lookback and live-odds decisions | Yes (scoreboards) | 3.7.0 |
 | [`sports_game_renderer`](#sports_game_renderer) | Scoreboard scroll/Vegas card geometry | Yes (scoreboards) | 3.3.0 |
 | [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 |
 | [`sports_scroll`](#sports_scroll) | Scoreboard scroll-display orchestration | Yes (scoreboards) | 3.2.0 |
@@ -46,7 +49,7 @@ Rules for the package:
 | [`sync_manager`](#sync_manager) | Leader/follower sync between two displays | No, core-internal | n/a |
 | [`text_helper`](#text_helper) | Outlined text, wrapping, measurement | Yes | — |
 
-The four `sports_*` mixin and card modules hold code the scoreboard plugins
+The `sports_*` mixin and card modules hold code the scoreboard plugins
 used to carry as identical copies. Each module docstring lists what a host
 class must provide. The plan behind them is in
 [docs/SPORTS_UNIFICATION.md](../../docs/SPORTS_UNIFICATION.md).
@@ -216,6 +219,33 @@ dates (`format_game_date()`, `format_game_time()`, `card_tzinfo()`) and font
 sizes (`schema_font_size()`, `resolve_font_size()`). A plugin keeps its own
 method and delegates the body.
 
+### sports_card_wrappers
+
+[`sports_card_wrappers.py`](sports_card_wrappers.py).
+`SportsCardWrappersMixin`: the one-line methods a scoreboard's game renderer
+uses to call `sports_card` with its own `config` and `logger`
+(`_vs_text()`, `_element_color()`, `_format_game_date()`, ... seventeen in
+all), under their existing names. They are what `sports_game_renderer`'s
+mixin expects its host to provide. No `__init__` and no state.
+
+### sports_celebration
+
+[`sports_celebration.py`](sports_celebration.py). `SportsCelebrationMixin`
+draws the full-screen takeover a scoreboard shows when a team scores or wins
+(`_draw_celebration_layout(celebration)`): a backdrop in the scoring team's
+colours read off its crest, scenery, confetti, the headline and the score.
+The colour helpers are free functions (`logo_palette()`, `lift_color()`,
+`mix_color()`, ...). Deciding *when* to celebrate stays in the plugin, which
+builds the celebration dict the docstring describes.
+
+### sports_fetch
+
+[`sports_fetch.py`](sports_fetch.py). `SportsFetchMixin`: the `SportsCore`
+methods that decide which requests a scoreboard makes --
+`_fetch_season_directly()` (a season, in chunks ESPN accepts),
+`_background_fetches_espn_ranges()`, `_needs_previous_day()` (the live
+lookback) and `_wants_live_odds()` (odds only for games near the screen).
+
 ### sports_game_renderer
 
 [`sports_game_renderer.py`](sports_game_renderer.py).

src/common/favorite_team_check.py

@@ -218,6 +218,8 @@ class FavoriteTeamCheck:
                 return None  # Nothing published either way; draw no conclusion.
             if cls._moved_to_later_phase(payload):
                 return None  # e.g. postseason under way; see the method.
+            if cls._later_round_scheduled(payload, now):
+                return None  # e.g. Europa League between matchdays.
             return ("the season has finished and the next one's fixtures are "
                     "not published yet")
 
@@ -259,6 +261,44 @@ class FavoriteTeamCheck:
         known = [t for t in event_types if isinstance(t, int)]
         return bool(known) and all(t < league_type for t in known)
 
+    @classmethod
+    def _later_round_scheduled(cls, payload, now: datetime) -> bool:
+        """
+        Whether a "list" calendar has a round that has not started yet.
+
+        Competitions with a list calendar (the UEFA club competitions, the
+        World Cup, AFL, NFL) give each phase its rounds as ``entries`` with
+        start and end dates. Between matchdays the Europa League scoreboard
+        keeps showing the last one: on 2026-09-29 every event was from 17
+        September, the next matchday was only days away, and the rounds from
+        the knockout play-offs to the final were all still to come. A round
+        that starts later means the season is not over, even though the
+        date of the next fixture is not known.
+
+        Only a round's *start* counts. End dates are padded well past the
+        last game -- the World Cup's final round ran to 1 August for a 19 July
+        final -- so a future end date is also true of a finished season.
+        Rounds in an offseason phase (the college football All-Star week)
+        are not games for the favourites and do not count either.
+        """
+        league = (payload.get('leagues') or [{}])[0] or {}
+        for phase in league.get('calendar') or []:
+            if not isinstance(phase, dict) or cls._is_offseason(phase.get('label')):
+                continue
+            for entry in phase.get('entries') or []:
+                if not isinstance(entry, dict) or cls._is_offseason(entry.get('label')):
+                    continue
+                start = cls._parse_date(entry.get('startDate'))
+                if start and start > now:
+                    return True
+        return False
+
+    @staticmethod
+    def _is_offseason(label) -> bool:
+        """'Off Season', 'Offseason', 'Off-season' ..."""
+        return isinstance(label, str) and 'offseason' in re.sub(
+            r'[^a-z]', '', label.lower())
+
     @staticmethod
     def _parse_date(raw) -> Optional[datetime]:
         if not raw or not isinstance(raw, str):

src/common/sports_card_wrappers.py

@@ -0,0 +1,136 @@
+"""The ``sports_card`` delegations every scoreboard's game renderer carries.
+
+After the card helpers moved to ``sports_card`` (3.3.0), each of the eight
+scoreboards with a ``game_renderer.py`` -- afl, baseball, basketball,
+football, hockey, lacrosse, nrl and soccer -- kept one-line methods that
+forward to them with its own ``config`` and ``logger``. Seventeen are
+identical in all eight (executable AST, docstrings stripped) or in all but
+football, and were copied here from ledmatrix-plugins ``30455671``
+(origin/main, 2026-09-29) under their existing names. Football's own
+``_format_game_date`` and ``_upcoming_center_mode`` (they follow the
+switch-mode settings when it draws the full-screen scorebug) stay in football
+and override these.
+
+``_schema_font_size`` and ``_resolve_font_size`` look the same in every copy
+but are not moved: they read ``_SCHEMA_PATH``, a module global that is each
+plugin's own ``config_schema.json``.
+
+These are the methods ``SportsGameRendererMixin`` (``sports_game_renderer``)
+lists among what its host must provide, so a renderer that inherits both no
+longer has to write them. Like that mixin this has no ``__init__`` and no
+state. It is a separate module rather than more methods there for the reason
+``sports_helpers`` gives: a missing module fails at load, where the version
+checks see it; a missing method fails mid-render.
+
+WHAT A HOST MUST PROVIDE
+------------------------
+Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
+test in ``test/test_sports_card_wrappers.py`` fails if a read is added
+without being listed here.
+
+- ``config`` and ``logger``.
+- ``fonts``, read with ``getattr`` -- ``_font_color``.
+- ``_FONT_NAME_ALIASES`` and ``_FONT_PIXEL_GRID`` class attributes --
+  ``_crisp_size``, which passes them to ``sports_card.crisp_size`` so a
+  renderer that declares extra faces keeps them.
+
+Add it as a base of the plugin's renderer, e.g.
+``class GameRenderer(SportsCardWrappersMixin, SportsGameRendererMixin)``.
+The two define no name in common; a method on the plugin's own class still
+wins over either.
+"""
+
+import logging
+from typing import Any, ClassVar, Dict, Optional, Tuple
+
+from src.common import sports_card as _card
+
+
+class SportsCardWrappersMixin:
+    """The game renderer's ``sports_card`` delegations. See module docstring."""
+
+    # The host contract, declared for type checking only: these create no
+    # attributes, so the host's own values are what the methods read.
+    config: Dict[str, Any]
+    logger: logging.Logger
+    _FONT_NAME_ALIASES: ClassVar[Dict[str, str]]
+    _FONT_PIXEL_GRID: ClassVar[Dict[str, Any]]
+
+    # ---- fonts ---------------------------------------------------------
+
+    @classmethod
+    def _crisp_size(cls, font_file, desired):
+        """``sports_card.crisp_size`` with this renderer's font tables."""
+        return _card.crisp_size(font_file, desired,
+                                cls._FONT_NAME_ALIASES, cls._FONT_PIXEL_GRID)
+
+    def _unshare_element_fonts(self, fonts):
+        """``sports_card.unshare_element_fonts``."""
+        return _card.unshare_element_fonts(self.logger, fonts)
+
+    def _font_color(self, font, default: Tuple[int, int, int] = (255, 255, 255)):
+        """``sports_card.font_color`` for one of ``self.fonts``."""
+        return _card.font_color(self.config, getattr(self, "fonts", None), font, default)
+
+    # ---- colours and favourites ---------------------------------------
+
+    @staticmethod
+    def _coerce_rgb(value, fallback):
+        """``sports_card.coerce_rgb``."""
+        return _card.coerce_rgb(value, fallback)
+
+    @staticmethod
+    def _side_is_favorite(game: Dict[str, Any], side: str, favorites: set) -> bool:
+        """``sports_card.side_is_favorite``."""
+        return _card.side_is_favorite(game, side, favorites)
+
+    @staticmethod
+    def _side_score(game: Dict[str, Any], side: str) -> Optional[int]:
+        """``sports_card.side_score``."""
+        return _card.side_score(game, side)
+
+    def _favorite_result(self, game: Dict[str, Any]) -> Optional[str]:
+        """``sports_card.favorite_result``."""
+        return _card.favorite_result(self.config, game)
+
+    def _score_color_for(self, game: Dict[str, Any], game_type: str, default=None):
+        """``sports_card.score_color_for``."""
+        return _card.score_color_for(self.config, self.logger, game, game_type, default)
+
+    def _recent_score_color(self, game: Dict[str, Any], default):
+        """``sports_card.recent_score_color``."""
+        return _card.recent_score_color(self.config, self.logger, game, default)
+
+    def _element_color(self, element: str, default: Tuple[int, int, int] = (255, 255, 255)):
+        """``sports_card.element_color``."""
+        return _card.element_color(self.config, element, default)
+
+    # ---- card options, dates and times --------------------------------
+
+    def _scroll_card_option(self, key: str, default: Any = None) -> Any:
+        """``sports_card.scroll_card_option``."""
+        return _card.scroll_card_option(self.config, key, default)
+
+    def _upcoming_center_mode(self) -> str:
+        """``sports_card.upcoming_center_mode``."""
+        return _card.upcoming_center_mode(self.config)
+
+    def _vs_text(self) -> str:
+        """``sports_card.vs_text``."""
+        return _card.vs_text(self.config)
+
+    def _format_game_date(self, date_text: str, game: Optional[Dict] = None) -> str:
+        """``sports_card.format_game_date``."""
+        return _card.format_game_date(self.config, self.logger, date_text, game)
+
+    def _weekday_for(self, game: Optional[Dict]) -> str:
+        """``sports_card.weekday_for``."""
+        return _card.weekday_for(self.config, self.logger, game)
+
+    def _card_tzinfo(self):
+        """``sports_card.card_tzinfo``."""
+        return _card.card_tzinfo(self.config, self.logger)
+
+    def _format_game_time(self, time_text: str) -> str:
+        """``sports_card.format_game_time``."""
+        return _card.format_game_time(self.config, time_text)

src/common/sports_celebration.py

@@ -0,0 +1,780 @@
+"""How the scoreboards draw a score or win celebration.
+
+Five scoreboards -- afl, football, hockey, nrl and soccer -- take over the
+panel when a team scores or wins: a backdrop in the scoring team's colours
+read off its crest, scenery for the kind of score, confetti, the headline and
+the score with the scoring side's digits breathing. The drawing is identical
+in all five ``sports.py`` copies (executable AST, docstrings stripped), and
+so are the colour helpers it uses; they were copied here from
+ledmatrix-plugins ``30455671`` (origin/main, 2026-09-29).
+
+Only the drawing moved. What *arms* a celebration stays in each plugin,
+because it differs: which scores count (``_check_for_goal`` /
+``_check_for_score``, and nrl matches favourites by team id), the phrase and
+the scenery (``_start_celebration``), and when a win fires
+(``_check_for_win``). So does ``display()``, which decides whether the
+takeover or the scorebug is on screen. A plugin hands this mixin a
+celebration dict and it draws it.
+
+The colour helpers are public free functions here (``logo_palette``,
+``lift_color``, ``mix_color``, ...); in the plugins they were the same
+functions with a leading underscore.
+
+THE CELEBRATION DICT
+--------------------
+Built by the plugin's ``_start_celebration``. Read here: ``game`` (a
+view-model dict; ``<side>_id``, ``<side>_abbr``, ``<side>_logo_path`` and
+``<side>_logo_url`` for the crests, ``id`` for the confetti seed),
+``scored_side`` (``"away"`` or ``"home"``), ``away_score``, ``home_score``,
+``phrase``, ``started_at`` (a ``time.time()`` value) and ``motif``
+(``"score"``, ``"kick"``, ``"touchdown"``, ``"net"`` or ``"win"``; anything
+else draws the ``"score"`` diagonals). The drawing caches what it derives in
+the same dict, under ``_palette``, ``_backdrop``, ``_confetti`` and
+``_crests``, so each is worked out once per celebration.
+
+WHAT A HOST MUST PROVIDE
+------------------------
+Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
+test in ``test/test_sports_celebration.py`` fails if a read is added without
+being listed here. All five scoreboards' ``SportsLive`` provide them.
+
+- ``display_manager`` -- ``image`` is replaced with the frame, then
+  ``update_display()``; ``clear()`` on ``force_clear``. Its ``matrix``
+  width and height are used when it has a matrix, else ``display_width`` /
+  ``display_height``.
+- ``fonts`` -- ``"time"`` and ``"status"`` for the headline (the first that
+  fits), ``"score"`` for the score.
+- ``logger``.
+- ``_load_and_resize_logo(team_id, abbr, logo_path, logo_url)`` -- a crest
+  as an RGBA image, or ``None``.
+- ``_draw_text_with_outline(draw, text, position, font, fill=...)`` -- on
+  ``SportsCoreSharedMixin``.
+- ``celebration_duration``, ``celebration_team_colors`` and
+  ``celebration_confetti``, read with ``getattr`` (defaults 8, on, on).
+
+Mix it in ahead of the mode classes, e.g.
+``class SportsLive(SportsCelebrationMixin, SportsLiveSharedMixin,
+SportsCore)``. It defines nothing any of them define, so the order only
+matters for a plugin that keeps its own copy of one of these methods: a
+method on the plugin's class always wins over the mixin's.
+"""
+
+import colorsys
+import logging
+import math
+import random
+import time
+from typing import Any, Callable, ClassVar, Dict, List, Optional, Sequence, Tuple
+
+from PIL import Image, ImageDraw
+
+#: A colour as the helpers return it: three 0-255 channels.
+Color = Tuple[int, ...]
+#: ``deep``, ``glow``, ``headline`` and ``accent``; see ``logo_palette``.
+Palette = Dict[str, Color]
+#: One confetti flake: column, start height, fall speed, sway phase, size
+#: in pixels, colour.
+Flake = Tuple[float, float, float, float, int, Color]
+
+# ----------------------------------------------------------------------
+# Colour helpers for the score/win celebration
+#
+# Module level rather than methods: they are pure, which is what makes the
+# palette testable without standing up a live manager, and they are shared by
+# the takeover's backdrop, confetti and text.
+# ----------------------------------------------------------------------
+
+#: The crest is sampled at this resolution. Big enough that a secondary
+#: colour survives (a helmet stripe, a trim), small enough that the whole
+#: sample is ~1600 pixels of pure-Python work, once per team.
+_PALETTE_SAMPLE_PX = 40
+#: Above this, a colour carries team identity; below it, it is a grey.
+_PALETTE_VIVID_SATURATION = 0.22
+#: Ignore pixels this dark -- crest outlines, drop shadows, anti-aliasing.
+_PALETTE_MIN_CHANNEL = 24
+#: How far apart two bins must be to count as a second, different colour.
+_PALETTE_DISTINCT_DISTANCE = 90.0
+#: Never bleed a lifted colour below this saturation; past it a hue stops
+#: being the team's colour and starts being a pastel.
+_PALETTE_MIN_SATURATION = 0.42
+#: Lift a headline colour until it is at least this luminous. Chosen so
+#: midnight navy reaches a blue that reads at 6px on a panel without
+#: becoming a different colour.
+_PALETTE_HEADLINE_LUMINANCE = 112.0
+#: A crest colour this luminous already reads on a panel, so it is preferred
+#: over a darker one that would have to be lifted to get there. Lifting is a
+#: compromise -- Green Bay's dark green only reaches legibility as a teal --
+#: and most teams whose primary is dark carry a bright second colour that is
+#: just as much theirs. This is what picks the Packers' gold over that teal.
+_PALETTE_LEGIBLE_LUMINANCE = 90.0
+#: ...but only from a colour the crest actually means. The pixels where a
+#: bright edge is anti-aliased into a dark fill are luminous too, and there is
+#: always a band of them: Kansas City's white-on-red outline leaves a pink at
+#: luminance 90 that would otherwise be preferred over the red itself. A blend
+#: is a mix, so it is markedly less saturated than either colour it sits
+#: between -- that pink is 0.48 where the red is 0.96 and the Packers' gold,
+#: which this must keep, is 0.89.
+_PALETTE_LEGIBLE_SATURATION = 0.65
+#: And it has to be a band of the crest, not a speck of one.
+_PALETTE_LEGIBLE_AREA = 0.02
+#: Cap the backdrop's luminance so the headline stays legible over it,
+#: and the scenery's so it stays behind the headline. Both are luminance and
+#: not HSV value on purpose: a silver crest -- the Raiders, or the grey
+#: placeholder a failed logo download leaves behind -- has a value of ~0.95,
+#: and capping that at 0.34 still yields a light grey card that white text
+#: then vanishes into. Scaling the channels down is also hue-exact, which is
+#: what lets this be the plain arithmetic that lifting a colour cannot be.
+_PALETTE_BACKDROP_LUMINANCE = 34.0
+_PALETTE_SCENERY_LUMINANCE = 70.0
+
+
+def rgb_luminance(color: Sequence[float]) -> float:
+    """Rec. 709 relative luminance, 0-255."""
+    return 0.2126 * color[0] + 0.7152 * color[1] + 0.0722 * color[2]
+
+
+def rgb_saturation(color: Sequence[float]) -> float:
+    """HSV saturation, 0-1."""
+    high = max(color)
+    return (high - min(color)) / high if high else 0.0
+
+
+def color_distance(a: Sequence[float], b: Sequence[float]) -> float:
+    """Euclidean distance between two colours in RGB."""
+    return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))
+
+
+def mix_color(a: Sequence[float], b: Sequence[float], t: float) -> Color:
+    """Blend ``a`` towards ``b``; t=0 is all a, t=1 is all b."""
+    t = min(max(t, 0.0), 1.0)
+    return tuple(int(round(a[i] + (b[i] - a[i]) * t)) for i in range(3))
+
+
+def scale_color(color: Sequence[float], factor: float) -> Color:
+    """Scale a colour's brightness, clamped to the panel's range."""
+    return tuple(min(255, max(0, int(round(c * factor)))) for c in color)
+
+
+def lift_color(color: Sequence[float], min_luminance: float = _PALETTE_HEADLINE_LUMINANCE,
+               cap_saturation: float = 0.92) -> Color:
+    """Raise a colour's brightness until it reads on a panel, keeping its hue.
+
+    Scaling the channels directly is what the obvious version of this does,
+    and it shifts hue badly on exactly the colours that need lifting: it turns
+    Baltimore's navy-purple into magenta. Working in HSV and raising only the
+    value leaves the hue where the team put it.
+    """
+    if rgb_luminance(color) >= min_luminance:
+        return tuple(int(c) for c in color)
+    hue, saturation, value = colorsys.rgb_to_hsv(*[c / 255.0 for c in color])
+    if saturation < 0.12:
+        # A grey or a silver has no hue to preserve; just make it bright.
+        lifted = colorsys.hsv_to_rgb(hue, saturation, max(value, 0.85))
+        return tuple(int(round(c * 255)) for c in lifted)
+    saturation = min(saturation, cap_saturation)
+
+    def _rgb(s: float, v: float) -> Color:
+        return tuple(int(round(c * 255)) for c in colorsys.hsv_to_rgb(hue, s, v))
+
+    out = _rgb(saturation, value)
+    while value < 1.0 and rgb_luminance(out) < min_luminance:
+        value = min(1.0, value + 0.05)
+        out = _rgb(saturation, value)
+    # Blue carries almost no luminance -- pure blue sits at 18 of 255 -- so a
+    # navy or a deep purple runs out of value long before it is legible.
+    # Bleeding saturation out of it is the only way up, and it keeps the hue
+    # (Baltimore stays purple, just a lighter one) where giving up would
+    # leave the headline unreadable. Floored so it never washes out to white.
+    while saturation > _PALETTE_MIN_SATURATION and rgb_luminance(out) < min_luminance:
+        saturation = max(_PALETTE_MIN_SATURATION, saturation - 0.05)
+        out = _rgb(saturation, value)
+    return out
+
+
+def cap_luminance(color: Sequence[float], max_luminance: float) -> Color:
+    """Darken a colour until it is no brighter than ``max_luminance``.
+
+    A straight channel scale, which is exactly hue-preserving on the way down
+    -- unlike lifting, where clamping at 255 is what bends the hue.
+    """
+    luminance = rgb_luminance(color)
+    if luminance <= max_luminance or luminance <= 0:
+        return tuple(int(c) for c in color)
+    return scale_color(color, max_luminance / luminance)
+
+
+def dim_rgba(image: Image.Image, factor: float) -> Image.Image:
+    """Scale an RGBA image's colour channels, leaving its alpha alone.
+
+    ImageEnhance.Brightness would scale the alpha band too, which fades the
+    crest out instead of dimming it and leaves its anti-aliased edge looking
+    chewed against the backdrop.
+    """
+    red, green, blue, alpha = image.split()
+    lut = [min(255, int(i * factor)) for i in range(256)]
+    return Image.merge(
+        "RGBA", (red.point(lut), green.point(lut), blue.point(lut), alpha)
+    )
+
+
+_Buckets = Dict[Tuple[int, int, int], List[int]]
+
+
+def _palette_buckets(logo: Image.Image) -> Tuple[_Buckets, _Buckets]:
+    """Bucket a crest's opaque pixels into coarse colour bins.
+
+    Returns ``(vivid, neutral)``; each maps a 3-bit-per-channel key to
+    ``[r_sum, g_sum, b_sum, count]``. Neutral holds the greys, silvers and
+    whites that carry no identity on their own but are all a monochrome crest
+    -- the Raiders' silver on black -- has to offer.
+    """
+    sample = logo.convert("RGBA")
+    sample.thumbnail((_PALETTE_SAMPLE_PX, _PALETTE_SAMPLE_PX), Image.Resampling.BOX)
+    vivid: _Buckets = {}
+    neutral: _Buckets = {}
+    # tobytes() rather than getdata(): same pixels, no per-pixel Python
+    # object, and getdata() is deprecated from Pillow 14.
+    raw = sample.tobytes()
+    for i in range(0, len(raw) - 3, 4):
+        red, green, blue, alpha = raw[i], raw[i + 1], raw[i + 2], raw[i + 3]
+        if alpha < 160:
+            continue
+        high, low = max(red, green, blue), min(red, green, blue)
+        if high < _PALETTE_MIN_CHANNEL:
+            continue
+        target = vivid if (high - low) / high >= _PALETTE_VIVID_SATURATION else neutral
+        acc = target.setdefault((red >> 5, green >> 5, blue >> 5), [0, 0, 0, 0])
+        acc[0] += red
+        acc[1] += green
+        acc[2] += blue
+        acc[3] += 1
+    return vivid, neutral
+
+
+def _bucket_mean(acc: List[int]) -> Color:
+    count = acc[3]
+    return (acc[0] // count, acc[1] // count, acc[2] // count)
+
+
+def _bucket_headline_score(acc: List[int]) -> float:
+    """How well a colour bin would serve as 6px of text on a panel.
+
+    Area alone picks the biggest block of colour, which on a lot of crests is
+    a dark navy fill -- correct as a backdrop, invisible as text. Weighting
+    area by saturation and by luminance picks the colour the team is loud in:
+    Chicago's orange over its navy, Baltimore's gold over its purple.
+    """
+    color = _bucket_mean(acc)
+    return (
+        acc[3]
+        * (0.30 + 0.70 * rgb_saturation(color))
+        * (0.20 + 0.80 * min(1.0, rgb_luminance(color) / 120.0))
+    )
+
+
+def logo_palette(logo: Image.Image) -> Optional[Palette]:
+    """Pick a celebration palette out of a team crest, or None.
+
+    Two rankings, because a crest's largest colour and its most legible one
+    are usually not the same and the takeover needs both:
+
+    * ``deep`` -- the largest vivid area, darkened into the background wash.
+      This is what the team reads as at a glance: Chicago navy, Dallas navy,
+      Baltimore purple.
+    * ``headline`` -- the vivid area that best survives being shrunk to text,
+      then lifted until it is legible: Chicago orange, Baltimore gold.
+    * ``accent`` -- the next vivid colour far enough away from the headline to
+      be told apart, for confetti. Falls back to the headline.
+
+    A crest with no vivid pixels at all falls back to its brightest neutral,
+    which for the Raiders' silver-on-black is exactly the right answer.
+    """
+    try:
+        vivid, neutral = _palette_buckets(logo)
+    except Exception:  # noqa: BLE001 - a crest is never worth the takeover
+        return None
+
+    pool = list(vivid.values())
+    if not pool and neutral:
+        pool = [
+            max(
+                neutral.values(),
+                key=lambda acc: acc[3]
+                * (0.2 + 0.8 * min(1.0, rgb_luminance(_bucket_mean(acc)) / 160.0)),
+            )
+        ]
+    if not pool:
+        return None
+
+    deep_base = _bucket_mean(max(pool, key=lambda acc: acc[3]))
+    ranked = sorted(pool, key=_bucket_headline_score, reverse=True)
+    headline_base = _bucket_mean(ranked[0])
+    vivid_pixels = sum(acc[3] for acc in pool)
+    for acc in ranked:
+        candidate = _bucket_mean(acc)
+        if (
+            rgb_luminance(candidate) >= _PALETTE_LEGIBLE_LUMINANCE
+            and rgb_saturation(candidate) >= _PALETTE_LEGIBLE_SATURATION
+            and acc[3] >= max(3, vivid_pixels * _PALETTE_LEGIBLE_AREA)
+        ):
+            headline_base = candidate
+            break
+    headline = lift_color(headline_base)
+
+    accent = headline
+    for acc in ranked[1:]:
+        candidate = _bucket_mean(acc)
+        if color_distance(candidate, headline_base) > _PALETTE_DISTINCT_DISTANCE:
+            accent = lift_color(candidate)
+            break
+
+    deep = cap_luminance(deep_base, _PALETTE_BACKDROP_LUMINANCE)
+    return {
+        "deep": deep,
+        # Scenery is the backdrop carried a little way towards the headline:
+        # tied to the team's colours, and guaranteed to be visible even when
+        # the backdrop is nearly black.
+        "glow": cap_luminance(
+            mix_color(deep, headline, 0.22), _PALETTE_SCENERY_LUMINANCE
+        ),
+        "headline": headline,
+        "accent": accent,
+    }
+
+
+class SportsCelebrationMixin:
+    """Draws a score/win celebration takeover. See the module docstring."""
+
+    # The host contract, declared for type checking only: these create no
+    # attributes, so the host's own values are what the methods read.
+    display_manager: Any
+    display_width: int
+    display_height: int
+    fonts: Dict[str, Any]
+    logger: logging.Logger
+    _load_and_resize_logo: Callable[..., Optional[Image.Image]]
+    _draw_text_with_outline: Callable[..., None]
+
+    def _fit_font(self, draw, text: str, max_width: int, fonts: list):
+        """Return the first font whose rendered ``text`` fits ``max_width``,
+        falling back to the last (smallest) font."""
+        for font in fonts:
+            if draw.textlength(text, font=font) <= max_width - 2:
+                return font
+        return fonts[-1]
+
+    # ------------------------------------------------------------------
+    # Celebration palette
+    #
+    # The takeover is drawn in the scoring team's own colours, taken from the
+    # pixels of its crest.
+    #
+    # ESPN does serve team.color / team.alternateColor, but only inside
+    # _extract_game_details_common -- a function each scoreboard lineage
+    # keeps its own copy of -- so reading it there would drag every one of
+    # them into a celebration change. The crest is already downloaded,
+    # decoded and sitting in the logo cache by the time a celebration draws,
+    # so the colours come from it instead: no extra request, no per-league
+    # colour table to maintain, and it works for any team ESPN can name --
+    # including the FCS opponents no table would list.
+    #
+    # Where a crest's colour differs from the club's published one it
+    # tends to differ usefully: a published primary is often a near-black
+    # navy, or an actual #000000, where what the crest carries is the colour
+    # that reads on an LED panel. Measured across all 32 clubs in
+    # football-scoreboard.
+    # ------------------------------------------------------------------
+
+    #: Used when the crest yields nothing (no logo on disk yet, or the grey
+    #: placeholder a failed download leaves) or team colours are switched
+    #: off -- the navy and amber the celebration wore before it had a palette.
+    _DEFAULT_CELEBRATION_PALETTE: ClassVar[Palette] = {
+        "deep": (10, 10, 40),
+        "glow": (30, 30, 86),
+        "headline": (255, 208, 56),
+        "accent": (255, 255, 255),
+    }
+
+    def _celebration_palette(self, celebration: Dict) -> Palette:
+        """The scoring team's colours, derived once per celebration."""
+        cached: Optional[Palette] = celebration.get("_palette")
+        if cached is not None:
+            return cached
+
+        palette = dict(self._DEFAULT_CELEBRATION_PALETTE)
+        if getattr(self, "celebration_team_colors", True):
+            try:
+                game = celebration["game"]
+                side = celebration.get("scored_side") or "home"
+                logo = self._load_and_resize_logo(
+                    game.get("%s_id" % side),
+                    game.get("%s_abbr" % side),
+                    game.get("%s_logo_path" % side),
+                    game.get("%s_logo_url" % side),
+                )
+                derived = logo_palette(logo) if logo is not None else None
+                if derived:
+                    palette = derived
+            except Exception as e:  # noqa: BLE001 - never lose a takeover to a crest
+                self.logger.debug(f"Celebration palette fell back to the default: {e}")
+
+        celebration["_palette"] = palette
+        return palette
+
+    # ------------------------------------------------------------------
+    # Celebration choreography
+    #
+    # Every frame is a finished card. The beats below shift the emphasis --
+    # an opening colour hit, confetti, a breathing score -- but none of them
+    # leaves the panel mid-wipe, because on a switch-mode board the core
+    # drives this plugin at 1 FPS (display_controller reserves its high-FPS
+    # loop for plugins that scroll or declare needs_high_fps), so any single
+    # frame may be the only one a viewer ever sees of it.
+    # ------------------------------------------------------------------
+
+    #: Fraction of the celebration spent on the opening colour hit.
+    _CELEBRATION_IMPACT: ClassVar[float] = 0.11
+    #: Fraction of it after which the takeover eases back down.
+    _CELEBRATION_SETTLE: ClassVar[float] = 0.80
+    #: Seconds per breath of the scoring side's digits. Deliberately a
+    #: continuous sine rather than an on/off toggle: the 4 Hz flash this
+    #: replaced was sampled once a second on a switch-mode board, which
+    #: aliases into a colour that changes at random. A ramp degrades into a
+    #: slow glow instead, and still reads as a pulse at 125 FPS.
+    _CELEBRATION_BREATH_SECONDS: ClassVar[float] = 1.7
+
+    def _celebration_backdrop(
+        self,
+        celebration: Dict,
+        width: int,
+        height: int,
+        palette: Palette,
+    ) -> Image.Image:
+        """The static half of the takeover: a team-colour gradient with the
+        scenery for this kind of score painted into it.
+
+        Built once per celebration per panel size and copied per frame, so the
+        per-pixel work never lands on the render path.
+        """
+        cached: Optional[Tuple[Tuple[int, int], Image.Image]] = celebration.get("_backdrop")
+        if cached is not None and cached[0] == (width, height):
+            return cached[1]
+
+        # One column, then stretched: filling the panel pixel by pixel would
+        # be `width` times the work for the same image.
+        column = Image.new("RGB", (1, max(height, 1)))
+        pixels: Any = column.load()
+        for y in range(height):
+            k = y / max(height - 1, 1)
+            pixels[0, y] = mix_color(palette["deep"], (0, 0, 0), 0.18 + 0.82 * k)
+        backdrop = column.resize((width, height)).convert("RGBA")
+
+        try:
+            self._draw_celebration_motif(
+                ImageDraw.Draw(backdrop),
+                celebration.get("motif") or "score",
+                width,
+                height,
+                palette,
+            )
+        except Exception as e:  # noqa: BLE001 - scenery is never worth a blank panel
+            self.logger.debug(f"Celebration motif skipped: {e}")
+
+        celebration["_backdrop"] = ((width, height), backdrop)
+        return backdrop
+
+    def _draw_celebration_motif(
+        self,
+        draw,
+        motif: str,
+        width: int,
+        height: int,
+        palette: Palette,
+    ) -> None:
+        """Paint the scenery for one kind of score, dim enough to stay behind
+        the headline and the score instead of competing with them."""
+        glow = palette["glow"]
+        if motif == "kick":
+            # The uprights a field goal or an extra point went through,
+            # spread wide enough to frame the score rather than sit beside it.
+            half = max(8, min(width // 3, height))
+            mid = width // 2
+            crossbar = int(height * 0.60)
+            draw.line([(mid - half, int(height * 0.08)), (mid - half, crossbar)], fill=glow)
+            draw.line([(mid + half, int(height * 0.08)), (mid + half, crossbar)], fill=glow)
+            draw.line([(mid - half, crossbar), (mid + half, crossbar)], fill=glow)
+            draw.line([(mid, crossbar), (mid, height - 1)], fill=glow)
+        elif motif == "touchdown":
+            # The goal line, with its hash marks.
+            line_y = int(height * 0.36)
+            draw.line([(0, line_y), (width, line_y)], fill=glow)
+            for x in range(3, width, 9):
+                draw.line([(x, line_y - 2), (x, line_y + 2)], fill=glow)
+        elif motif == "net":
+            # The goal a puck just went into: frame, posts and mesh, sized to
+            # frame the score the way the uprights do.
+            half = max(7, min(width // 4, height))
+            mid = width // 2
+            top = int(height * 0.34)
+            draw.rectangle([(mid - half, top), (mid + half, height - 1)], outline=glow)
+            step = max(3, (half * 2) // 6)
+            for x in range(mid - half + step, mid + half, step):
+                draw.line([(x, top + 1), (x, height - 2)], fill=glow)
+            for y in range(top + step, height - 1, step):
+                draw.line([(mid - half + 1, y), (mid + half - 1, y)], fill=glow)
+        elif motif == "win":
+            # A sunburst behind the winner.
+            cx, cy = width // 2, height // 2
+            reach = max(width, height)
+            for i in range(10):
+                angle = (math.pi * 2 * i / 10) + math.pi / 20
+                draw.line(
+                    [
+                        (cx, cy),
+                        (cx + math.cos(angle) * reach, cy + math.sin(angle) * reach),
+                    ],
+                    fill=glow,
+                )
+        else:
+            for x in range(-height, width + height, 11):
+                draw.line([(x, height), (x + height, 0)], fill=glow)
+
+    def _celebration_confetti(
+        self,
+        celebration: Dict,
+        width: int,
+        height: int,
+        palette: Palette,
+    ) -> List[Flake]:
+        """Seed the confetti once per celebration.
+
+        Seeded from the game rather than the clock, so the same score always
+        produces the same fall -- which is what lets a golden screen lock the
+        effect down instead of having to tolerate it.
+        """
+        cached: Optional[Tuple[Tuple[int, int], List[Flake]]] = celebration.get("_confetti")
+        if cached is not None and cached[0] == (width, height):
+            return cached[1]
+
+        # Sparse on purpose. At one flake per 170 square pixels a 128x32
+        # panel carried 24 single-pixel specks over the headline and the
+        # score, which reads as a dead-pixel problem rather than as confetti.
+        count = max(6, min(22, (width * height) // 260))
+        seed = "%s/%s" % (
+            (celebration.get("game") or {}).get("id", "?"),
+            celebration.get("phrase", ""),
+        )
+        rng = random.Random(seed)  # nosec B311 - confetti, not security
+        # Team colours, plus a pale tint of the headline rather than a flat
+        # white, so the fall still belongs to the team that scored.
+        colors = [
+            palette["headline"],
+            palette["accent"],
+            mix_color(palette["headline"], (255, 255, 255), 0.55),
+        ]
+        flakes = [
+            (
+                float(rng.randrange(max(width, 1))),  # column
+                rng.uniform(0.0, float(height)),      # start height
+                rng.uniform(0.40, 1.15),              # fall speed
+                rng.uniform(0.0, math.pi * 2),        # sway phase
+                2 if rng.random() < 0.6 else 1,       # size in pixels
+                colors[rng.randrange(len(colors))],
+            )
+            for _ in range(count)
+        ]
+        celebration["_confetti"] = ((width, height), flakes)
+        return flakes
+
+    def _draw_celebration_confetti(
+        self,
+        draw,
+        celebration: Dict,
+        width: int,
+        height: int,
+        palette: Palette,
+        elapsed: float,
+        progress: float,
+    ) -> None:
+        """Draw the confetti for this instant, thinning it out as the
+        celebration eases back towards the scorebug."""
+        flakes = self._celebration_confetti(celebration, width, height, palette)
+        fade = 1.0
+        if progress > self._CELEBRATION_SETTLE:
+            fade = max(
+                0.0,
+                1.0
+                - (progress - self._CELEBRATION_SETTLE)
+                / (1.0 - self._CELEBRATION_SETTLE),
+            )
+        if fade <= 0.02:
+            return
+        alpha = int(235 * fade)
+        for column, start, speed, phase, size, color in flakes:
+            y = (start + speed * elapsed * height * 0.42) % (height + 4) - 2
+            x = column + math.sin(elapsed * 2.1 + phase) * 2.4
+            draw.rectangle(
+                [(int(x), int(y)), (int(x) + size - 1, int(y) + size - 1)],
+                fill=tuple(color) + (alpha,),
+            )
+
+    def _celebration_crests(
+        self, celebration: Dict, height: int
+    ) -> Dict[str, Optional[Image.Image]]:
+        """The two crests for the takeover, with the side that did not score
+        dimmed so the scoring team reads at a glance."""
+        cached: Optional[Tuple[int, Dict[str, Optional[Image.Image]]]] = celebration.get("_crests")
+        if cached is not None and cached[0] == height:
+            return cached[1]
+
+        game = celebration["game"]
+        scored = celebration.get("scored_side")
+        crests: Dict[str, Optional[Image.Image]] = {}
+        for side in ("away", "home"):
+            logo = None
+            try:
+                logo = self._load_and_resize_logo(
+                    game.get("%s_id" % side),
+                    game.get("%s_abbr" % side),
+                    game.get("%s_logo_path" % side),
+                    game.get("%s_logo_url" % side),
+                )
+            except Exception as e:  # noqa: BLE001 - a crest is never worth the panel
+                self.logger.debug(f"Celebration logo load failed: {e}")
+            if logo is not None and side != scored:
+                logo = dim_rgba(logo, 0.40)
+            crests[side] = logo
+
+        celebration["_crests"] = (height, crests)
+        return crests
+
+    def _draw_celebration_layout(self, celebration: Dict, force_clear: bool = False) -> None:
+        """Render the full-screen goal/win takeover."""
+        if force_clear:
+            self.display_manager.clear()
+
+        display_width = (
+            self.display_manager.matrix.width
+            if hasattr(self.display_manager, "matrix") and self.display_manager.matrix
+            else self.display_width
+        )
+        display_height = (
+            self.display_manager.matrix.height
+            if hasattr(self.display_manager, "matrix") and self.display_manager.matrix
+            else self.display_height
+        )
+
+        elapsed = max(0.0, time.time() - celebration["started_at"])
+        # getattr throughout the render path: the golden-screen tests build a
+        # live manager through __new__ and set only what they draw with, and a
+        # celebration must never be lost to a missing knob.
+        duration = max(float(getattr(self, "celebration_duration", 8) or 8), 0.5)
+        progress = min(elapsed / duration, 1.0)
+        palette = self._celebration_palette(celebration)
+
+        main_img = self._celebration_backdrop(
+            celebration, display_width, display_height, palette
+        ).copy()
+
+        # Crests at the edges, bleeding off as the scorebug's do.
+        crests = self._celebration_crests(celebration, display_height)
+        center_y = display_height // 2
+        home_logo, away_logo = crests.get("home"), crests.get("away")
+        if home_logo is not None:
+            main_img.paste(
+                home_logo,
+                (display_width - home_logo.width + 2, center_y - home_logo.height // 2),
+                home_logo,
+            )
+        if away_logo is not None:
+            main_img.paste(away_logo, (-2, center_y - away_logo.height // 2), away_logo)
+
+        # The opening hit: the team's headline colour washes the panel and
+        # decays out of it. Held below opaque so the outlined text drawn on
+        # top still reads in whichever frame happens to catch it.
+        impact = max(0.0, 1.0 - progress / self._CELEBRATION_IMPACT)
+        if impact > 0.0:
+            # Scaled by how colourful the team is. A saturated crest gets the
+            # full hit; a silver one -- the Raiders, or the grey placeholder a
+            # failed logo download leaves -- would otherwise wash the whole
+            # panel out to the same flat grey as its own headline colour.
+            punch = 0.45 + 0.55 * rgb_saturation(palette["headline"])
+            alpha = int(140 * punch * (impact ** 1.5))
+            if alpha > 0:
+                main_img = Image.alpha_composite(
+                    main_img,
+                    Image.new(
+                        "RGBA",
+                        (display_width, display_height),
+                        tuple(palette["headline"]) + (alpha,),
+                    ),
+                )
+
+        overlay = Image.new("RGBA", (display_width, display_height), (0, 0, 0, 0))
+        draw = ImageDraw.Draw(overlay)
+
+        if getattr(self, "celebration_confetti", True):
+            try:
+                self._draw_celebration_confetti(
+                    draw,
+                    celebration,
+                    display_width,
+                    display_height,
+                    palette,
+                    elapsed,
+                    progress,
+                )
+            except Exception as e:  # noqa: BLE001
+                self.logger.debug(f"Celebration confetti skipped: {e}")
+
+        # Headline across the top, shrunk to fit the panel width, struck
+        # white on the opening hit and settling into the team's colour.
+        phrase = celebration["phrase"]
+        phrase_font = self._fit_font(
+            draw, phrase, display_width, [self.fonts["time"], self.fonts["status"]]
+        )
+        phrase_width = draw.textlength(phrase, font=phrase_font)
+        # Eased in by colour rather than by position. Sliding it down into
+        # place put the first frame at y=-3 with its top row cut off, and on a
+        # 1 FPS board that clipped frame can be the only one anyone sees.
+        self._draw_text_with_outline(
+            draw,
+            phrase,
+            ((display_width - phrase_width) // 2, 1),
+            phrase_font,
+            fill=mix_color(palette["headline"], (255, 255, 255), impact),
+        )
+
+        # Score centred low, the scoring side's digits breathing in the team's
+        # headline colour so the change reads at a glance.
+        away_text = str(celebration["away_score"])
+        home_text = str(celebration["home_score"])
+        score_font = self.fonts["score"]
+        segments = [
+            (away_text, celebration["scored_side"] == "away"),
+            ("-", False),
+            (home_text, celebration["scored_side"] == "home"),
+        ]
+        total_width = sum(draw.textlength(seg, font=score_font) for seg, _ in segments)
+        breath = 0.72 + 0.28 * (
+            0.5
+            + 0.5 * math.sin(2 * math.pi * elapsed / self._CELEBRATION_BREATH_SECONDS)
+        )
+        highlight = scale_color(palette["headline"], breath)
+        x = (display_width - total_width) // 2
+        # display_height - 14 was sized for the old fixed 8px score. #338
+        # scales the score with the panel (16px at 48 and 64 tall), which put
+        # the bottom of the digits off the panel. Lift it by the measured ink
+        # (+1 for the outline stroke) only when it would clip, so panels where
+        # it always fitted render exactly as before.
+        score_text = "".join(seg for seg, _ in segments)
+        ink_bottom = draw.textbbox((0, 0), score_text, font=score_font)[3]
+        y = min(display_height - 14, display_height - ink_bottom - 2)
+        for seg, is_highlight in segments:
+            color = highlight if is_highlight else (216, 216, 216)
+            self._draw_text_with_outline(draw, seg, (int(x), y), score_font, fill=color)
+            x += draw.textlength(seg, font=score_font)
+
+        main_img = Image.alpha_composite(main_img, overlay).convert("RGB")
+        self.display_manager.image = main_img
+        self.display_manager.update_display()

src/common/sports_fetch.py

@@ -0,0 +1,188 @@
+"""Which requests a scoreboard makes: season fetches, the lookback, live odds.
+
+Four ``SportsCore`` methods are identical (executable AST, docstrings
+stripped) in all nine scoreboards' ``sports.py`` -- afl, baseball,
+basketball, football, hockey, lacrosse, nrl, soccer and ufc -- and were
+copied here from ledmatrix-plugins ``30455671`` (origin/main, 2026-09-29)
+under their existing names:
+
+- ``_background_fetches_espn_ranges`` -- whether the core's background
+  service can fetch an ESPN date range, or the plugin must;
+- ``_fetch_season_directly`` -- fetch and cache a season in chunks ESPN
+  accepts, on the calling thread;
+- ``_needs_previous_day`` (with ``_LOOKBACK_CUTOFF_HOUR``) -- whether the
+  live fetch still has to ask for yesterday;
+- ``_wants_live_odds`` (with ``_LIVE_ODDS_LOOKAHEAD``) -- whether a live
+  game is close enough to the screen to be worth an odds request.
+
+Three other ``SportsCore`` methods are as identical and stay in the plugins,
+for the reasons ``sports_shared`` gives: ``_get_timezone`` binds each
+plugin's own ``resolve_timezone`` shim, and ``_extract_game_details`` /
+``_fetch_data`` are the abstract sport-specific contract. So does
+``SportsUpcoming.__init__``: the mixins in ``src/common`` hold no
+constructor, so the plugins' constructor signature stays theirs.
+
+A new module rather than more methods on ``sports_shared``, for the reason
+``sports_helpers`` gives: a missing module fails at load, where the version
+checks see it; a missing method fails mid-update.
+
+WHAT A HOST MUST PROVIDE
+------------------------
+Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
+test in ``test/test_sports_fetch.py`` fails if a read is added without being
+listed here.
+
+- ``session``, ``headers``, ``cache_manager`` and ``logger`` --
+  ``_fetch_season_directly``.
+- ``_games_lock`` -- ``_wants_live_odds``, which also reads ``live_games``,
+  ``current_game_index`` and ``_rotation_schedule`` with ``getattr``
+  (only ``SportsLive`` has them).
+- ``live_games``, read with ``getattr`` -- ``_needs_previous_day``.
+- ``background_service``, read with ``getattr`` --
+  ``_background_fetches_espn_ranges``.
+
+Add it as a base of the plugin's ``SportsCore``, e.g.
+``class SportsCore(SportsFetchMixin, SportsCoreSharedMixin,
+SportsHelpersMixin, ABC)``. It defines nothing those define; a method or
+constant on the plugin's own class still wins over the mixin's.
+"""
+
+import logging
+import threading
+from datetime import datetime, timedelta
+from typing import Any, ClassVar, Dict, Optional
+
+from src.common.espn_dates import ESPN_MAX_LIMIT, fetch_espn_scoreboard
+
+
+class SportsFetchMixin:
+    """Season fetch, lookback and live-odds decisions. See module docstring."""
+
+    # The host contract, declared for type checking only: these create no
+    # attributes, so the host's own values are what the methods read.
+    session: Any
+    headers: Dict[str, str]
+    cache_manager: Any
+    logger: logging.Logger
+    _games_lock: threading.RLock
+
+    #: How many games past the one on screen keep their odds warm. One is
+    #: enough for the line to be ready when the rotation advances; more just
+    #: re-creates the whole-slate fetch this replaced.
+    _LIVE_ODDS_LOOKAHEAD: ClassVar[int] = 1
+
+    def _wants_live_odds(self, game: Dict) -> bool:
+        """Whether a live game is near enough the front of the rotation to be
+        worth an odds request.
+
+        Odds used to be fetched for *every* live game in the league on every
+        update. The renderer only ever draws ``current_game``, and a full
+        rotation of a big slate takes minutes while ``live_odds_update_interval``
+        is 60s -- so all but one of those requests expired before the game they
+        belonged to came round.
+
+        Measured 2026-09-19 over a full college-football slate: 11,978 odds
+        requests in 13h on one rig, 54% of all its ESPN traffic, across only
+        ~140 distinct games. The eager loop also cost up to 2s of ``update()``
+        per live game, because ``_fetch_odds`` waits on its worker thread.
+
+        Mirrors the narrowing already applied to the upcoming path and to
+        ``_attach_odds_to_rotated_games``: only games about to be on screen are
+        asked about. ``get_odds`` still caches per game, so a game re-entering
+        the window inside its TTL costs a cache lookup, not a request.
+
+        The rotation state read here is the previous cycle's -- the new list is
+        still being built -- which is exactly the question being asked: is this
+        game at or near the position currently on the panel?
+        """
+        # Read defensively: this predicate lives on SportsCore so it sits
+        # beside _fetch_odds, but live_games/_rotation_schedule belong to
+        # SportsLive, which is the only caller.
+        with self._games_lock:
+            games = list(getattr(self, "live_games", ()) or ())
+            index = getattr(self, "current_game_index", 0)
+            schedule = list(getattr(self, "_rotation_schedule", ()) or ())
+        if not games:
+            # Cold start: nothing is on screen yet, so let the games seen on
+            # this first pass through rather than render a blank line for a
+            # whole cycle. Bounded -- the next pass has a rotation to narrow by.
+            return True
+        order = schedule or [g.get("id") for g in games]
+        if not order:
+            return True
+        start = index if 0 <= index < len(order) else 0
+        wanted = {
+            order[(start + offset) % len(order)]
+            for offset in range(self._LIVE_ODDS_LOOKAHEAD + 1)
+        }
+        return game.get("id") in wanted
+
+    #: Hour of the Eastern day past which last night's games are assumed over.
+    #:
+    #: The live fetch asks ESPN for a two-day window so a game that started
+    #: yesterday and is still running is not lost. ESPN rejects date *ranges*,
+    #: so that window is split into one request per day -- doubling every live
+    #: poll. Measured 2026-09-19: 1,858 requests per rig spent on yesterday's
+    #: date, which after breakfast holds nothing but final games.
+    #:
+    #: No sport on these boards runs six hours past midnight, and one that
+    #: somehow did is still covered: a game already being tracked keeps its own
+    #: day in the window regardless of the hour.
+    _LOOKBACK_CUTOFF_HOUR: ClassVar[int] = 6
+
+    def _needs_previous_day(self, now: datetime) -> bool:
+        """Whether the previous Eastern day can still hold a live game."""
+        if now.hour < self._LOOKBACK_CUTOFF_HOUR:
+            return True
+        previous = (now - timedelta(days=1)).strftime("%Y%m%d")
+        for game in (getattr(self, "live_games", None) or []):
+            start: Any = game.get("start_time_utc") if hasattr(game, "get") else None
+            try:
+                if start.astimezone(now.tzinfo).strftime("%Y%m%d") == previous:
+                    return True
+            except (AttributeError, ValueError, OSError, OverflowError):
+                continue
+        return False
+
+    def _background_fetches_espn_ranges(self) -> bool:
+        """Can the core's background service fetch an ESPN date range?
+
+        Cores from before the 2026-09-15 fix send a season range to ESPN as-is,
+        which now answers 400 for every sport. On those cores the managers fetch
+        the season themselves with _fetch_season_directly instead.
+        """
+        service = getattr(self, "background_service", None)
+        return bool(getattr(service, "handles_espn_date_ranges", False))
+
+    def _fetch_season_directly(
+        self,
+        url: str,
+        datestring: str,
+        cache_key: str,
+        label: str,
+        ttl: Optional[int] = None,
+    ) -> Optional[Dict]:
+        """Fetch a season schedule on this thread, in chunks ESPN accepts, and cache it.
+
+        ``label`` names the schedule in log lines, e.g. ``"2026 season"``.
+        """
+        try:
+            data = fetch_espn_scoreboard(
+                self.session,
+                url,
+                params={"dates": datestring, "limit": ESPN_MAX_LIMIT},
+                headers=self.headers,
+                timeout=30,
+                logger=self.logger,
+            )
+        except Exception as e:
+            self.logger.error(f"Failed to fetch {label} schedule: {e}")
+            return None
+        if ttl is None:
+            self.cache_manager.set(cache_key, data)
+        else:
+            self.cache_manager.set(cache_key, data, ttl=ttl)
+        self.logger.info(
+            f"Fetched {label} schedule: {len(data.get('events', []))} events"
+        )
+        return data

src/display_controller.py

@@ -29,7 +29,7 @@ import threading
 import types
 from collections import deque
 from contextlib import contextmanager
-from typing import Dict, Any, List, Optional, Callable, Tuple
+from typing import Dict, Any, List, Optional, Callable, Set, Tuple
 from datetime import datetime
 from concurrent.futures import ThreadPoolExecutor, as_completed  # pylint: disable=no-name-in-module
 import pytz
@@ -266,6 +266,10 @@ class DisplayController:
         self.on_demand_last_error: Optional[str] = None
         self.on_demand_last_event: Optional[str] = None
         self.on_demand_schedule_override = False
+        # Plugins that are disabled in config and loaded only because an
+        # on-demand request named them. The main loop unloads each one once
+        # on-demand has moved off it (_release_on_demand_plugins).
+        self._on_demand_loaded_plugins: Set[str] = set()
         self.rotation_resume_index: Optional[int] = None
         # Saved rotation position when a live-priority plugin preempts the
         # rotation, so it resumes where it left off (not after the live plugin)
@@ -369,7 +373,11 @@ class DisplayController:
                 """Load a single plugin and return result."""
                 plugin_load_start = time.time()
                 try:
-                    if self.plugin_manager.load_plugin(plugin_id):
+                    if plugin_id in self._on_demand_loaded_plugins:
+                        loaded = self.plugin_manager.load_plugin(plugin_id, force_enabled=True)
+                    else:
+                        loaded = self.plugin_manager.load_plugin(plugin_id)
+                    if loaded:
                         plugin_load_time = time.time() - plugin_load_start
                         return {
                             'success': True,
@@ -1021,17 +1029,28 @@ class DisplayController:
             accepts_display_mode: Whether display() takes ``display_mode``.
             force_clear: Passed through to display().
 
+        Each call is timed (two monotonic reads) and handed to
+        PluginManager.note_display_duration, which logs and records slow
+        calls and counts one that ran past the executor's timeout as a hang.
+
         Returns:
             display()'s result, or True when the frame was skipped because
             the plugin's update() holds its lock (the panel keeps the last
             frame; that is not a failure).
         """
-        with self._display_lock_or_skip(getattr(plugin, 'plugin_id', None)) as can_display:
+        plugin_id = getattr(plugin, 'plugin_id', None)
+        with self._display_lock_or_skip(plugin_id) as can_display:
             if not can_display:
                 return True
-            if accepts_display_mode:
-                return plugin.display(display_mode=mode, force_clear=force_clear)
-            return plugin.display(force_clear=force_clear)
+            started = time.monotonic()
+            try:
+                if accepts_display_mode:
+                    return plugin.display(display_mode=mode, force_clear=force_clear)
+                return plugin.display(force_clear=force_clear)
+            finally:
+                note = getattr(self.plugin_manager, 'note_display_duration', None)
+                if note is not None and plugin_id:
+                    note(plugin_id, time.monotonic() - started)
 
     def _health_tracker(self):
         """The plugin circuit breaker, or None when it is not enabled."""
@@ -1475,8 +1494,13 @@ class DisplayController:
         On-demand still resumes on its saved mode; this only widens what gets
         loaded, so normal rotation has somewhere to return to when it ends.
         A plugin that is disabled in config but named by the on-demand request
-        is still enabled and added, since otherwise the mode being resumed
-        would have nothing behind it.
+        is still loaded, since otherwise the mode being resumed would have
+        nothing behind it. It is tracked as loaded for on-demand only, the
+        same as one loaded live by _activate_on_demand, so it is unloaded
+        when the session ends instead of staying loaded until the next
+        restart. Its config section is not touched: setting ``enabled`` in
+        self.config wrote into the dict config_manager caches and returns to
+        every later load_config() in this process.
         """
         enabled_plugins = [p for p in discovered_plugins
                            if self.config.get(p, {}).get('enabled', False)]
@@ -1491,11 +1515,10 @@ class DisplayController:
             logger.warning("Falling back to normal mode (all enabled plugins)")
             return enabled_plugins
 
-        if not self.config.get(on_demand_plugin_id, {}).get('enabled', False):
-            logger.info("Temporarily enabling plugin '%s' for on-demand mode", on_demand_plugin_id)
-            self.config.setdefault(on_demand_plugin_id, {})['enabled'] = True
-            if on_demand_plugin_id not in enabled_plugins:
-                enabled_plugins.append(on_demand_plugin_id)
+        if on_demand_plugin_id not in enabled_plugins:
+            logger.info("Loading disabled plugin '%s' for on-demand mode only", on_demand_plugin_id)
+            self._on_demand_loaded_plugins.add(on_demand_plugin_id)
+            enabled_plugins.append(on_demand_plugin_id)
 
         # Restore on-demand state from the cached request so it resumes.
         self.on_demand_active = True
@@ -1591,6 +1614,11 @@ class DisplayController:
                 logger.debug("Stop request %s received but on-demand is not active", request_id)
                 # Still update request_id to acknowledge the request
                 self.on_demand_request_id = request_id
+                if self.on_demand_status == 'error':
+                    # A failed request left status 'error' published, and
+                    # without this the status route kept reporting it until
+                    # the state aged out (120s) or another request came in.
+                    self._clear_on_demand(reason='requested-stop')
             # Stop requests are deliberately exempt from the request_id/
             # processed_id guards above, so that a second click stops a mode
             # that a race left running. Consuming the mailbox is therefore the
@@ -1757,10 +1785,136 @@ class DisplayController:
                    plugin_id, ordered_modes, self.on_demand_mode_index, 
                    ordered_modes[self.on_demand_mode_index] if ordered_modes else 'N/A')
 
+    def _load_plugin_for_on_demand(self, plugin_id: str) -> bool:
+        """Load an installed plugin that isn't running so on-demand can show it.
+
+        This process only loads the plugins enabled in config, so a request
+        for a disabled one -- the config page's "Preview on display" button
+        offers it on every plugin -- failed with "invalid-mode" while the UI
+        said the plugin would be enabled for the session. Nothing did that
+        short of a restart, and restarts no longer happen on a request.
+
+        Loads through the same path as a live enable (load_plugin, then
+        _register_loaded_plugin), with force_enabled so the instance runs
+        enabled while config.json keeps saying disabled. The plugin is
+        recorded in _on_demand_loaded_plugins, and the main loop unloads it
+        once on-demand moves off it (_release_on_demand_plugins).
+
+        Returns False after publishing an error when the load fails. A
+        plugin that isn't installed returns True without loading anything:
+        the mode checks that follow report it as they always have.
+        """
+        if self.plugin_manager is None:
+            return True
+        try:
+            known = self.plugin_manager.discovered_plugin_ids()
+        except AttributeError:
+            known = set(getattr(self.plugin_manager, 'plugin_manifests', ()) or ())
+        if plugin_id not in known:
+            # Installed after this process scanned: the web process checked
+            # its own, fresher list before posting the request.
+            try:
+                known = set(self.plugin_manager.discover_plugins())
+            except Exception:  # pylint: disable=broad-except
+                logger.exception("On-demand: plugin discovery failed")
+                known = set()
+            if plugin_id not in known:
+                return True
+
+        logger.info("On-demand: loading disabled plugin '%s' for this session only", plugin_id)
+        self._on_demand_loaded_plugins.add(plugin_id)
+        try:
+            loaded = self.plugin_manager.load_plugin(plugin_id, force_enabled=True)
+            if loaded:
+                modes = self._register_loaded_plugin(plugin_id)
+                logger.info("On-demand: loaded plugin '%s' (modes: %s)", plugin_id, modes)
+        except Exception:  # pylint: disable=broad-except
+            logger.exception("On-demand: error loading plugin '%s'", plugin_id)
+            loaded = False
+        if not loaded:
+            # Stays in _on_demand_loaded_plugins so the main loop removes
+            # whatever part of it did get registered.
+            logger.error("On-demand: could not load plugin '%s'", plugin_id)
+            self._set_on_demand_error("load-failed")
+            return False
+        return True
+
+    def _release_on_demand_plugins(self) -> None:
+        """Unload plugins loaded only for on-demand that it has moved off.
+
+        Runs from the main loop, right after its own on-demand poll, not
+        where on-demand ends: a stop, an expiry or the next request is often
+        read from inside a render loop or a dwell sleep, where the plugin
+        being released may still be on the stack mid-display(). Unloading
+        goes through _unregister_plugin, as a live disable does, and nothing
+        is written to config.json.
+
+        A plugin the user enabled in the meantime stays loaded and takes its
+        place in the rotation, which is what the reconcile that the enable
+        queued would have done.
+        """
+        if self.plugin_manager is None:  # plugin system failed after startup restore
+            self._on_demand_loaded_plugins.clear()
+            return
+        keep = self.on_demand_plugin_id if self.on_demand_active else None
+        releasable = [p for p in self._on_demand_loaded_plugins if p != keep]
+        if not releasable:
+            return
+        try:
+            config = self.config_service.get_config()
+        except Exception as e:  # pylint: disable=broad-except
+            logger.warning("On-demand release: falling back to cached config: %s", e)
+            config = self.config
+        previous_mode = self.current_display_mode
+        for plugin_id in releasable:
+            self._on_demand_loaded_plugins.discard(plugin_id)
+            section = config.get(plugin_id)
+            if isinstance(section, dict) and section.get('enabled', False):
+                logger.info("On-demand: keeping plugin '%s' loaded; it was enabled "
+                            "while on-demand showed it", plugin_id)
+                continue
+            if (plugin_id in self.plugin_display_modes
+                    or self.plugin_manager.get_plugin(plugin_id) is not None):
+                logger.info("On-demand: unloading plugin '%s'; it is disabled in config",
+                            plugin_id)
+                self._unregister_plugin(plugin_id)
+        if not self.on_demand_active:
+            # Only outside a session: rotation_resume_index points into
+            # available_modes until the session ends.
+            self._apply_plugin_rotation_order()
+        self._resync_mode_index_after_change(previous_mode)
+        if self.current_display_mode != previous_mode:
+            self.force_change = True
+
+    def _rotation_index_outside_on_demand(self, start: int) -> Optional[int]:
+        """First index from `start` (wrapping) whose mode is not owned by a
+        plugin loaded only for on-demand, or None if every mode is.
+
+        Ending a session must not resume the rotation onto the plugin that
+        is about to be unloaded. A live load appends that plugin's modes
+        after the saved resume index, but a session restored after a
+        restart has no saved index and its plugin was ordered in with the
+        rest -- the rotation resumed onto it, and a stop read during its own
+        screen changed nothing on the panel until that screen ended.
+        """
+        if not self._on_demand_loaded_plugins:
+            return start
+        on_demand_only = {mode for plugin_id in self._on_demand_loaded_plugins
+                          for mode in self.plugin_display_modes.get(plugin_id, [])}
+        count = len(self.available_modes)
+        for step in range(count):
+            index = (start + step) % count
+            if self.available_modes[index] not in on_demand_only:
+                return index
+        return None
+
     def _activate_on_demand(self, request: Dict[str, Any]) -> None:
         """Activate on-demand mode for a specific plugin display."""
         plugin_id = request.get('plugin_id')
         mode = request.get('mode')
+        if (plugin_id and plugin_id not in self.plugin_display_modes
+                and not self._load_plugin_for_on_demand(plugin_id)):
+            return
         resolved_mode = self._resolve_mode_for_plugin(plugin_id, mode)
 
         if not resolved_mode:
@@ -1866,6 +2020,15 @@ class DisplayController:
                 self.on_demand_last_event = 'stop-request-ignored'  # Already idle
                 self._publish_on_demand_state()
             return
+        if not self.on_demand_active and self.on_demand_status == 'error':
+            # _set_on_demand_error already ended any session and dropped
+            # rotation_resume_index; the full clear below would only move
+            # the rotation and force a redraw. Just drop the error.
+            self.on_demand_status = 'idle'
+            self.on_demand_last_error = None
+            self.on_demand_last_event = reason or 'cleared'
+            self._publish_on_demand_state()
+            return
 
         self._reset_on_demand_fields()
         self.on_demand_status = 'idle'
@@ -1875,17 +2038,27 @@ class DisplayController:
         # Clear on-demand configuration from cache
         self.cache_manager.clear_cache('display_on_demand_config')
 
-        if self.rotation_resume_index is not None and self.available_modes:
-            self.current_mode_index = self.rotation_resume_index % len(self.available_modes)
-            self.current_display_mode = self.available_modes[self.current_mode_index]
-            logger.info("Resuming rotation from saved index %d: mode '%s'", 
-                       self.rotation_resume_index, self.current_display_mode)
-        elif self.available_modes:
-            # Default to first mode if no resume index
-            self.current_mode_index = self.current_mode_index % len(self.available_modes)
-            self.current_display_mode = self.available_modes[self.current_mode_index]
-            logger.info("Resuming rotation to mode '%s' (index %d)", 
-                       self.current_display_mode, self.current_mode_index)
+        if self.available_modes:
+            saved = self.rotation_resume_index
+            # Default to the current index if no resume index
+            start = saved if saved is not None else self.current_mode_index
+            index = self._rotation_index_outside_on_demand(start % len(self.available_modes))
+            if index is None:
+                # Every mode belongs to a plugin loaded only for on-demand,
+                # which the main loop is about to unload; it then idles.
+                self.current_mode_index = 0
+                self.current_display_mode = None
+                logger.info("No enabled mode to resume rotation to")
+            elif saved is not None:
+                self.current_mode_index = index
+                self.current_display_mode = self.available_modes[index]
+                logger.info("Resuming rotation from saved index %d: mode '%s'",
+                            saved, self.current_display_mode)
+            else:
+                self.current_mode_index = index
+                self.current_display_mode = self.available_modes[index]
+                logger.info("Resuming rotation to mode '%s' (index %d)",
+                            self.current_display_mode, self.current_mode_index)
         else:
             logger.warning("No available modes to resume rotation to")
 
@@ -2082,6 +2255,14 @@ class DisplayController:
                 # Handle on-demand commands before rendering
                 self._poll_on_demand_requests()
                 self._check_on_demand_expiration()
+                # Unload plugins loaded only to show them on-demand once it
+                # has moved off them. Here, where no display() is on the
+                # stack; one ended from inside a screen is caught here on
+                # the next pass.
+                if self._on_demand_loaded_plugins:
+                    self._release_on_demand_plugins()
+                    if not self.available_modes:
+                        continue  # it was all there was; idle as above
                 self._tick_plugin_updates()
                 
                 # Clean up expired WiFi status messages
@@ -2328,6 +2509,7 @@ class DisplayController:
                         pm = self.plugin_manager
                         display_lock = pm.get_plugin_lock(plugin_id) if pm else None
                         can_display = display_lock is None or display_lock.acquire(blocking=False)
+                        display_hung = False
 
                         if display_lock is None:
                             # Only when plugin loading failed part-way.
@@ -2347,7 +2529,7 @@ class DisplayController:
                             # thread actually finishes it, rather than
                             # here when this dispatch merely returns.
                             release_guard = threading.Lock()
-                            released = {'done': False}
+                            released = {'done': False, 'started': False}
 
                             def _release_display_lock():
                                 with release_guard:
@@ -2358,6 +2540,7 @@ class DisplayController:
 
                             if _accepts_display_mode:
                                 def _display_target(display_mode=None, force_clear=False):
+                                    released['started'] = True
                                     try:
                                         return manager_to_display.display(
                                             display_mode=display_mode, force_clear=force_clear)
@@ -2365,11 +2548,13 @@ class DisplayController:
                                         _release_display_lock()
                             else:
                                 def _display_target(force_clear=False):
+                                    released['started'] = True
                                     try:
                                         return manager_to_display.display(force_clear=force_clear)
                                     finally:
                                         _release_display_lock()
 
+                            dispatch_start = time.monotonic()
                             try:
                                 result = pm.plugin_executor.execute_display(
                                     types.SimpleNamespace(display=_display_target),
@@ -2392,6 +2577,18 @@ class DisplayController:
                                 _release_display_lock()
                                 raise
 
+                            dispatch_seconds = time.monotonic() - dispatch_start
+                            if released['started'] and not released['done']:
+                                # The executor gave up waiting and display()
+                                # is still running on its thread, holding
+                                # the lock. A hang, not a success: recorded
+                                # so repeats open the circuit breaker, and
+                                # the update worker's bounded wait skips it.
+                                display_hung = True
+                                pm.record_display_hang(plugin_id, dispatch_seconds)
+                            else:
+                                pm.note_display_duration(plugin_id, dispatch_seconds)
+
                         logger.debug(f"display() returned: {result} (type: {type(result)})")
                         if isinstance(result, bool):
                             display_result = result
@@ -2405,7 +2602,7 @@ class DisplayController:
                         # be lost when display() finally does run.
                         if can_display:
                             health_tracker = self._health_tracker()
-                            if health_tracker is not None:
+                            if health_tracker is not None and not display_hung:
                                 health_tracker.record_success(plugin_id)
                             self.force_change = False
                     except Exception as exc:  # pylint: disable=broad-except
@@ -3099,8 +3296,23 @@ class DisplayController:
                     prepared = prepare(_pid, new_config) if callable(prepare) else None
                     if isinstance(prepared, dict):
                         new_config = prepared
-                    _plugin.on_config_change(new_config)
-                    logger.debug("Plugin %s notified of config change", _pid)
+                    if _pid in self._on_demand_loaded_plugins:
+                        # Saved while on-demand shows it: config.json still
+                        # says disabled, and on_config_change would switch
+                        # the instance off mid-session.
+                        new_config = {**new_config, 'enabled': True}
+                    # Runs on ConfigService's watcher thread. Under the
+                    # plugin's lock, so it cannot interleave with update()
+                    # on the worker or display() on the render thread; a
+                    # lock held past the bound defers it to the worker.
+                    apply = getattr(self.plugin_manager, 'apply_config_change', None)
+                    if callable(apply):
+                        applied = apply(_pid, new_config, plugin_instance=_plugin)
+                    else:
+                        _plugin.on_config_change(new_config)
+                        applied = True
+                    logger.debug("Plugin %s notified of config change%s", _pid,
+                                 "" if applied else " (deferred: plugin busy)")
                 except Exception as e:
                     logger.error("Error in plugin %s config change handler: %s", _pid, e, exc_info=True)
 

src/plugin_system/plugin_executor.py

@@ -19,8 +19,25 @@ class PluginTimeoutError(Exception):
     """Raised when a plugin operation times out."""
 
 
+class PluginBusyError(PluginTimeoutError):
+    """A plugin's lock stayed held past its bound.
+
+    Not raised; recorded. The lock is held by the plugin's own display(),
+    update(), on_config_change() or a Vegas content render -- slow, or hung
+    -- so the caller skipped the plugin rather than wait on it. Report-only:
+    it is kept as the plugin's state error info and counted as a busy skip in
+    health, never as a failure, so it cannot open the circuit breaker.
+    """
+
+
 class PluginExecutor:
     """Handles plugin execution with timeout and error isolation."""
+
+    #: A display() call at least this long is logged and counted as slow.
+    #: A frame is milliseconds; two seconds is a plugin doing I/O in display().
+    SLOW_DISPLAY_SECONDS = 2.0
+    #: An update() call at least this long is logged as slow.
+    SLOW_UPDATE_SECONDS = 5.0
     
     def __init__(
         self,
@@ -117,15 +134,15 @@ class PluginExecutor:
             True if update succeeded, False otherwise
         """
         try:
-            start_time = time.time()
+            start_time = time.monotonic()
             self.execute_with_timeout(
                 lambda: plugin.update(),
                 timeout=timeout,
                 plugin_id=plugin_id
             )
-            duration = time.time() - start_time
+            duration = time.monotonic() - start_time
             
-            if duration > 5.0:  # Warn if update takes more than 5 seconds
+            if duration > self.SLOW_UPDATE_SECONDS:
                 self.logger.warning(
                     "Plugin %s update() took %.2fs (consider optimizing)",
                     plugin_id,
@@ -175,7 +192,7 @@ class PluginExecutor:
             True if display succeeded, False otherwise
         """
         try:
-            start_time = time.time()
+            start_time = time.monotonic()
             
             # Does display() take a display_mode keyword? The caller usually
             # knows and caches the answer, so prefer what it passed.
@@ -206,9 +223,9 @@ class PluginExecutor:
                     plugin_id=plugin_id
                 )
             
-            duration = time.time() - start_time
+            duration = time.monotonic() - start_time
             
-            if duration > 2.0:  # Warn if display takes more than 2 seconds
+            if duration > self.SLOW_DISPLAY_SECONDS:
                 self.logger.warning(
                     "Plugin %s display() took %.2fs (consider optimizing)",
                     plugin_id,

src/plugin_system/plugin_health.py

@@ -254,6 +254,104 @@ class PluginHealthTracker:
         
         self._save_health_state(plugin_id, state)
     
+    def record_hang(self, plugin_id: str, operation: str, seconds: float,
+                    error: Optional[Exception] = None) -> None:
+        """Record a display() or update() call that ran past its limit.
+
+        Counts as a failure, so the ordinary circuit breaker handles a plugin
+        that keeps hanging: after ``failure_threshold`` in a row it is skipped
+        by both the update scheduler and the display rotation until the
+        cooldown ends. The hang itself is kept alongside (``hang_count``,
+        ``last_hang``) so the health API can tell "hung" from "raised".
+
+        Not for an update skipped because the plugin's lock stayed held: the
+        holder may be a healthy but long render (Vegas prefetch). That is
+        :meth:`record_busy_skip`, which never touches the breaker.
+
+        Args:
+            plugin_id: Plugin identifier
+            operation: What hung: ``"display"`` or ``"update"``.
+            seconds: How long it had been running when this was recorded.
+            error: The error to store as ``last_error``; one is built from
+                the other arguments when omitted.
+        """
+        state = self.get_health_state(plugin_id)
+        count = state.get('hang_count')
+        state['hang_count'] = (count if isinstance(count, int) and not isinstance(count, bool)
+                               else 0) + 1
+        state['last_hang'] = {
+            'operation': operation,
+            'seconds': round(float(seconds), 3),
+            'time': time.time(),
+        }
+        if error is None:
+            error = TimeoutError(f"{operation} still running after {seconds:.1f}s")
+        # record_failure saves the record, hang fields included.
+        self.record_failure(plugin_id, error)
+
+    #: Minimum seconds between persisting a plugin's slow-call or busy-skip
+    #: counters. The in-memory record is updated every time; a plugin that is
+    #: slow on every frame must not become an SD-card write per frame.
+    SLOW_CALL_PERSIST_INTERVAL = 60.0
+
+    def record_slow_call(self, plugin_id: str, operation: str, seconds: float) -> None:
+        """Note a call that finished, but slowly. Reporting only.
+
+        Unlike :meth:`record_hang` this never touches the circuit breaker: a
+        slow display() still drew its frame.
+        """
+        state = self.get_health_state(plugin_id)
+        count = state.get('slow_call_count')
+        state['slow_call_count'] = (count if isinstance(count, int) and not isinstance(count, bool)
+                                    else 0) + 1
+        now = time.time()
+        state['last_slow_call'] = {
+            'operation': operation,
+            'seconds': round(float(seconds), 3),
+            'time': now,
+        }
+        self._save_reporting_throttled('slow', plugin_id, state, now)
+
+    def record_busy_skip(self, plugin_id: str, operation: str, seconds: float) -> None:
+        """Note a call skipped because the plugin's lock stayed held. Reporting only.
+
+        The update worker gives up on a plugin's lock after
+        ``PluginManager.PLUGIN_LOCK_TIMEOUT``. Whatever held it may be healthy
+        -- Vegas prefetch holds the lock for a plugin's whole content render,
+        which on a slow Pi can take longer than that -- so like
+        :meth:`record_slow_call` this never touches the circuit breaker, the
+        failure streak or ``last_error``. A real hang is recorded by
+        :meth:`record_hang` where it is measured.
+
+        Args:
+            plugin_id: Plugin identifier
+            operation: What was skipped, e.g. ``"update lock wait"``.
+            seconds: How long the lock was waited on.
+        """
+        state = self.get_health_state(plugin_id)
+        count = state.get('busy_skip_count')
+        state['busy_skip_count'] = (count if isinstance(count, int) and not isinstance(count, bool)
+                                    else 0) + 1
+        now = time.time()
+        state['last_busy_skip'] = {
+            'operation': operation,
+            'seconds': round(float(seconds), 3),
+            'time': now,
+        }
+        self._save_reporting_throttled('busy', plugin_id, state, now)
+
+    def _save_reporting_throttled(self, kind: str, plugin_id: str,
+                                  state: Dict[str, Any], now: float) -> None:
+        """Persist a reporting-only change at most once per
+        SLOW_CALL_PERSIST_INTERVAL per plugin and ``kind``. The first one is
+        saved at once, so the web process (which reads the persisted record)
+        sees it; repeats in between stay in memory until the next save."""
+        saved_at = self.__dict__.setdefault('_reporting_saved_at', {})
+        last = saved_at.get((kind, plugin_id))
+        if last is None or now - last >= self.SLOW_CALL_PERSIST_INTERVAL:
+            saved_at[(kind, plugin_id)] = now
+            self._save_health_state(plugin_id, state)
+
     def set_degraded(self, plugin_id: str, reason: Optional[str]) -> None:
         """Flag (or clear) a plugin as degraded without touching the circuit breaker.
 
@@ -345,7 +443,13 @@ class PluginHealthTracker:
             'degraded': state.get('degraded', False),
             'degraded_reason': state.get('degraded_reason'),
             'circuit_opened_time': state.get('circuit_opened_time'),
-            'half_open_start_time': state.get('half_open_start_time')
+            'half_open_start_time': state.get('half_open_start_time'),
+            'hang_count': state.get('hang_count', 0),
+            'last_hang': state.get('last_hang'),
+            'slow_call_count': state.get('slow_call_count', 0),
+            'last_slow_call': state.get('last_slow_call'),
+            'busy_skip_count': state.get('busy_skip_count', 0),
+            'last_busy_skip': state.get('last_busy_skip'),
         }
     
     def get_all_health_summaries(self) -> Dict[str, Dict[str, Any]]:

src/plugin_system/plugin_manager.py

@@ -16,12 +16,14 @@ import time
 import threading
 import types
 from pathlib import Path
-from typing import Dict, List, Optional, Any, Tuple
+from typing import Dict, List, NamedTuple, Optional, Any, Tuple, Union
 import logging
 from src.exceptions import PluginError, ConfigError
 from src.logging_config import get_logger
 from src.plugin_system.plugin_loader import PluginLoader
-from src.plugin_system.plugin_executor import PluginExecutor
+from src.plugin_system.plugin_executor import (
+    PluginBusyError, PluginExecutor, PluginTimeoutError,
+)
 from src.plugin_system.plugin_state import PluginStateManager, PluginState
 from src.plugin_system.schema_manager import (
     CORE_VEGAS_TUNING_KEYS, SchemaManager, normalize_legacy_booleans,
@@ -36,6 +38,16 @@ from src.common.permission_utils import (
 )
 
 
+class _DeferredConfigChange(NamedTuple):
+    """Update-queue item: apply the config change parked for ``plugin_id``.
+
+    Queued by apply_config_change() when the plugin's lock was busy; the
+    change itself waits in ``PluginManager._deferred_config_changes`` so only
+    the latest one is ever applied.
+    """
+    plugin_id: str
+
+
 class PluginManager:
     """
     Manages plugin discovery, loading, and lifecycle.
@@ -56,6 +68,19 @@ class PluginManager:
     # How long unload_plugin() waits for an in-flight update() to finish
     # before tearing the instance down anyway.
     UNLOAD_LOCK_TIMEOUT = 5.0
+
+    # How long the update worker and apply_config_change() wait for a
+    # plugin's lock -- the same bound unload already uses for the same lock.
+    # A display() frame holds it for milliseconds, so this only runs out when
+    # the holder is hung or pathologically slow. The worker then skips that
+    # plugin (recorded as a hang, so repeats open its circuit breaker)
+    # instead of stalling every other plugin's update behind it.
+    PLUGIN_LOCK_TIMEOUT = UNLOAD_LOCK_TIMEOUT
+
+    # Minimum seconds between repeats of the same hang/slow-call warning for
+    # one plugin. A hung plugin is re-detected every interval; a slow
+    # display() can be re-detected every frame.
+    HANG_LOG_INTERVAL = 60.0
     
     def __init__(self, plugins_dir: str = "plugins", 
                  config_manager: Optional[Any] = None, 
@@ -122,7 +147,33 @@ class PluginManager:
         # post-timeout window.
         # Kill switch: plugin_system.synchronous_updates: true restores the
         # inline path.
-        self._update_queue: "queue.Queue[Optional[Tuple[str, float]]]" = queue.Queue()
+        #
+        # Which thread runs each plugin hook, and what it holds:
+        #   __init__, on_enable   the loading thread (main thread at startup,
+        #                         the render thread on a live enable).
+        #   update()              plugin-update-worker, under the plugin lock,
+        #                         via PluginExecutor (whose daemon thread runs
+        #                         the call; if it outlives the executor's
+        #                         timeout it keeps the lock until it returns).
+        #                         Exceptions: the startup pass
+        #                         (DisplayController._run_initial_updates, main
+        #                         thread, before the display loop starts) and
+        #                         the synchronous_updates kill switch (render
+        #                         thread) run it without the lock.
+        #   display()             the render thread, under a try-lock: a busy
+        #                         lock skips the frame. The first frame of a
+        #                         screen goes through PluginExecutor. Vegas
+        #                         mode's adapter and coordinator take the lock
+        #                         with a bounded wait.
+        #   on_config_change()    ConfigService's watcher thread, under the
+        #                         plugin lock via apply_config_change(); if the
+        #                         lock stays busy it is deferred to the update
+        #                         worker, which applies it under the lock.
+        #   cleanup(), on_disable()  whoever calls unload_plugin(), under the
+        #                         lock with UNLOAD_LOCK_TIMEOUT.
+        # No wait on a plugin lock is unbounded, so one hung plugin can only
+        # cost the worker PLUGIN_LOCK_TIMEOUT per attempt.
+        self._update_queue: "queue.Queue[Union[None, Tuple[str, float], _DeferredConfigChange]]" = queue.Queue()
         self._pending_updates: set = set()
         self._pending_lock = threading.Lock()
         # Serializes the "is this plugin eligible?" -> "claim it (RUNNING)"
@@ -145,6 +196,12 @@ class PluginManager:
         # run_scheduled_updates_with_changes().
         self._completed_updates: set = set()
         self._completed_updates_lock = threading.Lock()
+        # Config changes that found the plugin's lock busy, latest per plugin,
+        # with the instance they were meant for. See apply_config_change().
+        self._deferred_config_changes: Dict[str, Tuple[Any, Dict[str, Any]]] = {}
+        self._deferred_config_lock = threading.Lock()
+        # key -> (monotonic time last logged, repeats suppressed since)
+        self._rate_limited_warnings: Dict[str, Tuple[float, int]] = {}
         self._synchronous_updates = False
         if self.config_manager is not None:
             try:
@@ -296,7 +353,7 @@ class PluginManager:
 
         return plugin_ids
 
-    def load_plugin(self, plugin_id: str) -> bool:
+    def load_plugin(self, plugin_id: str, force_enabled: bool = False) -> bool:
         """
         Load a plugin by ID.
         
@@ -310,6 +367,10 @@ class PluginManager:
         
         Args:
             plugin_id: Plugin identifier
+            force_enabled: Run the plugin enabled even though config.json has
+                it disabled. On-demand uses this to show a disabled plugin
+                (DisplayController._load_plugin_for_on_demand). Only the
+                instance's config says enabled; config.json is not written.
             
         Returns:
             True if loaded successfully, False otherwise
@@ -376,6 +437,12 @@ class PluginManager:
             # (prepare_plugin_config). In memory only: config.json is written
             # by saves, never by loading a plugin.
             config = self.prepare_plugin_config(plugin_id, config, schema=schema)
+            if force_enabled:
+                # A copy: prepare_plugin_config can hand back the section from
+                # config_manager's cached config, and setting the flag there
+                # would read as enabled to everything else in this process.
+                config = dict(config)
+                config['enabled'] = True
             
             # Use PluginLoader to load plugin
             plugin_instance, _module = self.plugin_loader.load_plugin(
@@ -676,6 +743,8 @@ class PluginManager:
             
             # Remove from active plugins
             del self.plugins[plugin_id]
+            with self._deferred_config_lock:
+                self._deferred_config_changes.pop(plugin_id, None)
             with self._plugin_last_update_lock:
                 self.plugin_last_update.pop(plugin_id, None)
             self._update_interval_cache.pop(plugin_id, None)
@@ -1012,6 +1081,8 @@ class PluginManager:
         self,
         plugin_id: str,
         exc: Optional[Exception] = None,
+        log: bool = True,
+        count_failure: bool = True,
     ) -> None:
         """Apply the standard failure-recovery path for a plugin update.
 
@@ -1025,6 +1096,11 @@ class PluginManager:
             exc: The exception that caused the failure, if any.  When None a
                  synthetic ExecutionFailure exception is constructed from the
                  timeout/executor-error path.
+            log: Log the generic failure line. Callers that already logged
+                 something more specific (rate-limited) pass False.
+            count_failure: Record the failure in plugin health, where it
+                 counts toward the circuit breaker. A busy skip passes False:
+                 it records itself as a busy skip, reporting only.
         """
         failure_time = time.time()
         if exc is not None:
@@ -1040,13 +1116,91 @@ class PluginManager:
             'timestamp': failure_time,
             'recoverable': True,
         }
-        self.logger.warning("Plugin %s update() failed; will retry after interval", plugin_id)
+        if log:
+            self.logger.warning("Plugin %s update() failed; will retry after interval", plugin_id)
         with self._plugin_last_update_lock:
             self.plugin_last_update[plugin_id] = failure_time
         self.state_manager.set_state_with_error(plugin_id, PluginState.ENABLED, error_info)
-        if self.health_tracker:
+        if count_failure and self.health_tracker:
             self.health_tracker.record_failure(plugin_id, err)
 
+    def _warn_rate_limited(self, key: str, message: str, *args: Any) -> None:
+        """Log a warning at most once per HANG_LOG_INTERVAL for ``key``.
+
+        Repeats in between are counted and the count is appended to the next
+        one that is logged, so the journal shows the problem continuing
+        without a line per frame or per scheduler tick.
+        """
+        # setdefault: tests build bare managers with PluginManager.__new__.
+        seen = self.__dict__.setdefault('_rate_limited_warnings', {})
+        now = time.monotonic()
+        last, suppressed = seen.get(key, (None, 0))
+        if last is not None and now - last < self.HANG_LOG_INTERVAL:
+            seen[key] = (last, suppressed + 1)
+            return
+        seen[key] = (now, 0)
+        if suppressed:
+            message += " (%d more since the last warning)"
+            args = args + (suppressed,)
+        self.logger.warning(message, *args)
+
+    def _record_hang(self, plugin_id: str, operation: str, seconds: float,
+                     err: Exception) -> None:
+        """Record a hang in plugin health: a failure to the circuit breaker.
+
+        PluginHealthTracker.record_hang also counts the hang separately. Never
+        raises: this runs on the update worker and the render thread.
+        """
+        tracker = self.health_tracker
+        if tracker is None:
+            return
+        try:
+            tracker.record_hang(plugin_id, operation, seconds, err)
+        except Exception as e:  # pylint: disable=broad-except
+            self.logger.debug("Could not record hang for %s: %s", plugin_id, e)
+
+    def note_display_duration(self, plugin_id: str, seconds: float) -> None:
+        """Account for one display() call that took ``seconds``.
+
+        Called by the render loop for every frame, so the common case is one
+        comparison. At or above PluginExecutor.SLOW_DISPLAY_SECONDS the call
+        is logged (rate-limited) and counted as slow in plugin health; at or
+        above the executor's timeout -- the limit the first frame of a screen
+        is already held to -- it is recorded as a hang, which the circuit
+        breaker counts as a failure.
+        """
+        if seconds < PluginExecutor.SLOW_DISPLAY_SECONDS:
+            return
+        if seconds >= self.plugin_executor.default_timeout:
+            self.record_display_hang(plugin_id, seconds)
+            return
+        self._warn_rate_limited(
+            "slow-display:" + plugin_id,
+            "Plugin %s display() took %.2fs; a frame should take milliseconds "
+            "(is it fetching or loading files in display()?)", plugin_id, seconds)
+        tracker = self.health_tracker
+        record_slow = getattr(tracker, 'record_slow_call', None) if tracker is not None else None
+        if callable(record_slow):
+            try:
+                record_slow(plugin_id, 'display', seconds)
+            except Exception as e:  # pylint: disable=broad-except
+                self.logger.debug("Could not record slow display for %s: %s", plugin_id, e)
+
+    def record_display_hang(self, plugin_id: str, seconds: float) -> None:
+        """Record a display() call that ran ``seconds``, past its limit.
+
+        Either it has since returned (note_display_duration) or it is still
+        running on the executor's lingering thread, holding the plugin's lock
+        (the render loop's first-frame dispatch).
+        """
+        self._warn_rate_limited(
+            "hung-display:" + plugin_id,
+            "Plugin %s display() ran for at least %.1fs (limit %.0fs); recorded "
+            "as a hang -- repeated hangs open its circuit breaker",
+            plugin_id, seconds, self.plugin_executor.default_timeout)
+        self._record_hang(plugin_id, 'display', seconds, PluginTimeoutError(
+            f"Plugin {plugin_id} display() ran for at least {seconds:.1f}s"))
+
     def run_scheduled_updates(self, current_time: Optional[float] = None) -> None:
         """
         Trigger plugin updates based on their defined update intervals.
@@ -1132,11 +1286,13 @@ class PluginManager:
         self.state_manager.set_state(plugin_id, PluginState.ENABLED)
 
     def get_plugin_lock(self, plugin_id: str) -> threading.Lock:
-        """Per-plugin lock keeping update() and display() mutually exclusive.
+        """Per-plugin lock keeping update(), display() and on_config_change()
+        mutually exclusive.
 
         The update worker holds it for the duration of a plugin's update();
         the display side acquires it non-blocking and skips that frame's
-        display() call when the plugin is mid-update.
+        display() call when the plugin is mid-update. Every other waiter uses
+        a bounded acquire (see the thread notes in __init__).
         """
         with self._plugin_locks_guard:
             lock = self._plugin_locks.get(plugin_id)
@@ -1202,14 +1358,28 @@ class PluginManager:
         real update() call genuinely finishes (see _execute_update_now),
         which can be after this dispatch returns if PluginExecutor's own
         timeout elapses first.
+
+        The lock wait is bounded by PLUGIN_LOCK_TIMEOUT. Whatever holds it
+        past that -- a hung display() on the render thread, a lingering
+        executor thread, or a long but healthy Vegas content render -- costs
+        this worker that long once per attempt, and the plugin's update is
+        skipped and reported as a busy skip (_skip_busy_update), which never
+        counts toward the circuit breaker; the other plugins' queued updates
+        carry on.
         """
         while True:
             item = self._update_queue.get()
             if item is None:  # shutdown sentinel
                 return
+            if isinstance(item, _DeferredConfigChange):
+                self._apply_deferred_config_change(item.plugin_id)
+                continue
             plugin_id, scheduled_time = item
             lock = self.get_plugin_lock(plugin_id)
-            lock.acquire()
+            wait_start = time.monotonic()
+            if not lock.acquire(timeout=self.PLUGIN_LOCK_TIMEOUT):
+                self._skip_busy_update(plugin_id, time.monotonic() - wait_start)
+                continue
             plugin_instance = self.plugins.get(plugin_id)
             if plugin_instance is None:  # unloaded while queued; its
                 # lifecycle state was already cleared by unload_plugin —
@@ -1218,6 +1388,9 @@ class PluginManager:
                 with self._pending_lock:
                     self._pending_updates.discard(plugin_id)
                 continue
+            # A config change that found the lock busy goes in first, so
+            # this update() runs against the settings the user saved.
+            self._apply_deferred_config_locked(plugin_id, plugin_instance)
             try:
                 self._execute_update_now(plugin_id, plugin_instance,
                                          scheduled_time, lock=lock)
@@ -1228,6 +1401,142 @@ class PluginManager:
                 self.logger.exception("update worker: unexpected error for %s",
                                       plugin_id)
 
+    def _skip_busy_update(self, plugin_id: str, waited: float) -> None:
+        """Give up on a queued update whose plugin lock stayed held.
+
+        Same bookkeeping as a failed update() -- pending slot dropped before
+        the state returns to ENABLED with PluginBusyError error info,
+        last-update stamped so the retry waits a full interval -- but
+        report-only in health: counted as a busy skip (``busy_skip_count`` /
+        ``last_busy_skip``), never as a failure or a hang. The lock holder
+        may be perfectly healthy: Vegas prefetch holds a plugin's lock for its
+        whole content render, which on a slow Pi can outlast
+        PLUGIN_LOCK_TIMEOUT, and counting that would pull a healthy plugin
+        from rotation. Real hangs -- display() or update() past the executor
+        timeout -- are recorded where they are measured and still open the
+        breaker.
+        """
+        with self._pending_lock:
+            self._pending_updates.discard(plugin_id)
+        if plugin_id not in self.plugins:
+            # Unloaded while we waited: its lifecycle state is already
+            # cleared; recording anything would resurrect it as ENABLED.
+            return
+        self._warn_rate_limited(
+            "busy-update:" + plugin_id,
+            "Plugin %s update skipped: its lock was still held after %.1fs "
+            "(a display(), Vegas render or update() of it is still running); "
+            "retrying next interval, not counted as a failure", plugin_id, waited)
+        self._record_update_failure(
+            plugin_id,
+            exc=PluginBusyError(
+                f"Plugin {plugin_id} busy: its lock was held for over {waited:.1f}s "
+                "by a slow or hung display()/update(); update skipped"),
+            log=False,
+            count_failure=False)
+        tracker = self.health_tracker
+        record_busy = getattr(tracker, 'record_busy_skip', None) if tracker is not None else None
+        if callable(record_busy):
+            try:
+                record_busy(plugin_id, 'update lock wait', waited)
+            except Exception as e:  # pylint: disable=broad-except
+                self.logger.debug("Could not record busy skip for %s: %s", plugin_id, e)
+
+    def apply_config_change(self, plugin_id: str, new_config: Dict[str, Any],
+                            plugin_instance: Optional[Any] = None) -> bool:
+        """Call ``on_config_change(new_config)`` without racing update()/display().
+
+        Runs on the calling thread -- ConfigService's watcher, for the display
+        service -- holding the plugin's lock, waited on for at most
+        PLUGIN_LOCK_TIMEOUT. If the lock is still busy (an update() mid-fetch
+        can outlast that) the change is parked and handed to the update
+        worker, which applies it under the same lock once it is free, and at
+        the latest just before the plugin's next update(). A later change for
+        the same plugin replaces a parked one.
+
+        Exceptions from on_config_change propagate on the immediate path,
+        as they did when the caller invoked it directly.
+
+        Args:
+            plugin_id: Plugin identifier.
+            new_config: The prepared config to hand the plugin.
+            plugin_instance: The instance to notify; defaults to the loaded one.
+
+        Returns:
+            True if on_config_change ran now, False if it was deferred or there
+            is no loaded plugin to notify.
+        """
+        if plugin_instance is None:
+            plugin_instance = self.plugins.get(plugin_id)
+        if plugin_instance is None or not hasattr(plugin_instance, 'on_config_change'):
+            return False
+        lock = self.get_plugin_lock(plugin_id)
+        if lock.acquire(timeout=self.PLUGIN_LOCK_TIMEOUT):
+            try:
+                with self._deferred_config_lock:
+                    # This change supersedes any older one still parked.
+                    self._deferred_config_changes.pop(plugin_id, None)
+                plugin_instance.on_config_change(new_config)
+            finally:
+                lock.release()
+            return True
+
+        with self._deferred_config_lock:
+            self._deferred_config_changes[plugin_id] = (plugin_instance, new_config)
+        self._warn_rate_limited(
+            "busy-config:" + plugin_id,
+            "Plugin %s is busy (lock held for over %.1fs); its config change "
+            "will be applied by the update worker once it is free",
+            plugin_id, self.PLUGIN_LOCK_TIMEOUT)
+        try:
+            self._ensure_update_worker()
+            self._update_queue.put(_DeferredConfigChange(plugin_id))
+        except Exception as exc:  # pylint: disable=broad-except
+            # No worker (thread start refused): still parked, so the next
+            # update() of this plugin applies it.
+            self.logger.error(
+                "Could not queue the config change for plugin %s (%s: %s); it "
+                "will be applied before its next update()",
+                plugin_id, type(exc).__name__, exc)
+        return False
+
+    def _apply_deferred_config_change(self, plugin_id: str) -> None:
+        """Worker side of a parked config change: take the lock, apply it."""
+        with self._deferred_config_lock:
+            if plugin_id not in self._deferred_config_changes:
+                return  # applied or superseded meanwhile
+        lock = self.get_plugin_lock(plugin_id)
+        wait_start = time.monotonic()
+        if not lock.acquire(timeout=self.PLUGIN_LOCK_TIMEOUT):
+            self._warn_rate_limited(
+                "busy-config:" + plugin_id,
+                "Plugin %s still busy after %.1fs; its config change stays "
+                "parked until its next update()",
+                plugin_id, time.monotonic() - wait_start)
+            return
+        try:
+            self._apply_deferred_config_locked(plugin_id, self.plugins.get(plugin_id))
+        finally:
+            lock.release()
+
+    def _apply_deferred_config_locked(self, plugin_id: str,
+                                      current_instance: Optional[Any]) -> None:
+        """Apply the parked config change for plugin_id; caller holds its lock."""
+        with self._deferred_config_lock:
+            entry = self._deferred_config_changes.pop(plugin_id, None)
+        if entry is None:
+            return
+        instance, new_config = entry
+        if current_instance is None or instance is not current_instance:
+            # Unloaded, or reloaded as a new instance built from the current
+            # config: nothing left to tell.
+            return
+        try:
+            instance.on_config_change(new_config)
+            self.logger.info("Applied deferred config change for plugin %s", plugin_id)
+        except Exception:  # pylint: disable=broad-except
+            self.logger.exception("Error in plugin %s config change handler", plugin_id)
+
     def stop_update_worker(self, timeout: float = 5.0) -> None:
         """Signal the worker to exit (used by cleanup; thread is a daemon)."""
         if self._update_worker is not None and self._update_worker.is_alive():
@@ -1338,14 +1647,29 @@ class PluginManager:
             else:
                 _finish(True)
 
+        started = time.monotonic()
         try:
-            self.plugin_executor.execute_update(
+            success = self.plugin_executor.execute_update(
                 types.SimpleNamespace(update=_target_update), plugin_id)
         except Exception as exc:  # pragma: no cover - defensive; execute_update
             # catches everything internally, but guarantee _finish still
             # runs (releasing the lock) if something unexpected slips through.
             self.logger.exception("Unexpected error dispatching update for %s: %s", plugin_id, exc)
             _finish(False, exc=exc)
+            return
+        if not success and not finished['done']:
+            # The executor stopped waiting but update() is still running: it
+            # keeps the lock and the RUNNING state until it returns (then
+            # _finish records the outcome). Say so now, rather than leave the
+            # plugin silently stuck; record_success on a late return clears it.
+            elapsed = time.monotonic() - started
+            self._warn_rate_limited(
+                "hung-update:" + plugin_id,
+                "Plugin %s update() still running after %.1fs; it keeps its "
+                "lock until it returns, and is not rescheduled until then",
+                plugin_id, elapsed)
+            self._record_hang(plugin_id, 'update', elapsed, PluginTimeoutError(
+                f"Plugin {plugin_id} update() still running after {elapsed:.1f}s"))
 
     def run_scheduled_updates_with_changes(self, current_time: Optional[float] = None) -> List[str]:
         """

test/conftest.py

@@ -16,7 +16,54 @@ if str(project_root) not in sys.path:
     sys.path.insert(0, str(project_root))
 
 
+class _DisarmStartupReconciliation:
+    """Import hook: every ``web_interface.app`` this process builds starts disarmed.
+
+    app.py wires itself to the checkout's real config/config.json and
+    plugin-repos/ at import, and its before_request hook launches startup
+    reconciliation on the first request any test sends. Reconciliation
+    reinstalls every configured plugin missing on disk from the live store,
+    so a full run downloaded basketball-scoreboard, calendar,
+    football-scoreboard, leaderboard and ledmatrix-stocks into the real
+    plugin-repos/ (not gitignored), minutes in, from a daemon thread no test
+    waits on. Setting ``_reconciliation_started`` is the app's own run-once
+    latch; doing it as the module finishes executing covers fixtures that
+    import the app lazily and send a request at once, and ``importlib.reload``.
+    StateReconciliation itself stays fully testable.
+    """
+
+    _MODULE = "web_interface.app"
+
+    def find_spec(self, fullname, path, target=None):
+        if fullname != self._MODULE:
+            return None
+        import importlib.machinery
+        spec = importlib.machinery.PathFinder.find_spec(fullname, path, target)
+        if spec is None or spec.loader is None:
+            return spec
+        exec_module = spec.loader.exec_module
+
+        def exec_disarmed(module):
+            exec_module(module)
+            module._reconciliation_started = True
+
+        spec.loader.exec_module = exec_disarmed
+        return spec
+
+
+_DISARM_HOOK = _DisarmStartupReconciliation()
+
+
 def pytest_configure(config):
+    sys.meta_path.insert(0, _DISARM_HOOK)
+    app_module = sys.modules.get(_DisarmStartupReconciliation._MODULE)
+    if app_module is not None:
+        app_module._reconciliation_started = True
+
+    _point_emulator_at_raw_adapter(config)
+
+
+def _point_emulator_at_raw_adapter(config):
     """Point the emulator at a per-process config that binds no socket.
 
     Six test modules set EMULATOR=true and build a real DisplayManager. The
@@ -63,7 +110,9 @@ def pytest_configure(config):
 
 
 def pytest_unconfigure(config):
-    """Remove the throwaway emulator config written by pytest_configure."""
+    """Undo pytest_configure: the import hook and the throwaway emulator config."""
+    if _DISARM_HOOK in sys.meta_path:
+        sys.meta_path.remove(_DISARM_HOOK)
     tmp_dir = getattr(config, "_ledmatrix_emulator_tmp", None)
     if tmp_dir is not None:
         import shutil

test/test_api_v3_on_demand_restart.py

@@ -1,25 +1,27 @@
-"""Regression test: POST /display/on-demand/start restarting a running
-service must not import a name that does not exist.
+"""POST /display/on-demand/start and /stop must not restart a running display.
 
-display.py has `import web_interface.blueprints.api_v3 as _pkg` and reads
-mutable, test-patched attributes back through it (`_pkg.time.time()`,
-`_pkg._get_starlark_plugin()`, ...) rather than binding them by value, per
-the package's own docstring. One spot went further and wrote a genuine
-`import` *statement* against that alias --
+The start route used to treat ``start_service`` (default True, and what both
+the web UI and the MQTT bridge send) as "restart": with the service running it
+ran ``systemctl stop``, slept 1.5s and started it again. Every on-demand or
+"Preview on display" click therefore cold-restarted the display process --
+every plugin reloaded, the panel blank for seconds -- to deliver a request the
+running process polls for every ON_DEMAND_POLL_INTERVAL anyway (see
+test_on_demand_mailbox.py and test_display_pending_changes.py for the display
+side: the mailbox is read mid-dwell, mid-screen and mid-Vegas-iteration).
 
-    import _pkg.time as time_module
+The restart did not buy anything either: a freshly started display restores
+only the on-demand session it saved itself (``display_on_demand_config``), so
+the new request reached it through the same mailbox, one cold start later.
 
--- but `_pkg` is a local name bound by `import ... as _pkg` in this module,
-not a real top-level package, so `import _pkg.time` is not something Python
-can resolve; it raises ModuleNotFoundError. That line only runs when the
-display service is already running and the caller also asked to (re)start
-it, so this endpoint failed on exactly the restart path -- the one where a
-cache write recording the new on-demand request had already happened.
+This file previously pinned that restart path (it guarded a broken
+``import _pkg.time`` inside it). The path is gone; these tests pin its
+replacement: a running service is left alone, a stopped one is started (only
+when start_service is set), and the request lands in the mailbox either way.
 
-The route wraps its body in `except Exception`, so the failure reached the
-caller as a handled 500 with a generic message, not an unhandled crash --
-but a 500 all the same on a request that should have restarted the service
-and reported success.
+The service helpers are patched where they run. display.py binds
+_get_display_service_status by value, while _ensure_display_service_running
+(in the package __init__) looks it up in its own module, so both are patched;
+_run_systemctl_command is the one place a systemctl command is issued.
 """
 
 import sys
@@ -32,60 +34,137 @@ sys.path.insert(0, str(Path(__file__).parent.parent))
 
 from test._api_v3_test_helpers import api_v3_client, api_v3_module  # noqa: F401,E402
 
-URL = "/api/v3/display/on-demand/start"
+START_URL = "/api/v3/display/on-demand/start"
+STOP_URL = "/api/v3/display/on-demand/stop"
+MAILBOX = "display_on_demand_request"
 
 
 @pytest.fixture
-def restart_path(api_v3_module):
-    """Force the `service_was_running and start_service` branch.
+def service(api_v3_module):
+    """A display service whose state the test sets; records systemctl calls.
 
-    plugin_manager and config_manager are set to None so the route takes
-    the simplest path to that branch rather than tripping over unrelated
-    MagicMock plumbing. The cache is the blueprint's cache_manager, which
-    api_v3_module already set to a MagicMock. _get_display_service_status,
-    _stop_display_service and _ensure_display_service_running are bound by
-    value in display.py (see its own docstring), so they are patched on
-    that submodule rather than on the package.
+    plugin_manager and config_manager are None so the route skips plugin
+    resolution (not what is under test here). The cache is the blueprint's
+    MagicMock cache_manager, so mailbox writes are visible as set() calls.
     """
     api_v3_module.api_v3.plugin_manager = None
     api_v3_module.api_v3.config_manager = None
+    state = {"active": True}
 
-    with patch("web_interface.blueprints.api_v3.display._get_display_service_status") as get_status, \
-         patch("web_interface.blueprints.api_v3.display._stop_display_service") as stop_service, \
-         patch("web_interface.blueprints.api_v3.display._ensure_display_service_running") as ensure_running:
-        # Active before the request: service_was_running becomes True.
-        get_status.return_value = {"active": True}
-        ensure_running.return_value = {"active": True}
+    def status():
+        return {"active": state["active"]}
+
+    def systemctl(args):
+        if args[-2:] == ["start", "ledmatrix.service"]:
+            state["active"] = True
+        elif args[-2:] == ["stop", "ledmatrix.service"]:
+            state["active"] = False
+        return {"returncode": 0, "stdout": "", "stderr": ""}
+
+    with patch("web_interface.blueprints.api_v3._get_display_service_status",
+               side_effect=status), \
+         patch("web_interface.blueprints.api_v3.display._get_display_service_status",
+               side_effect=status), \
+         patch("web_interface.blueprints.api_v3._run_systemctl_command",
+               side_effect=systemctl) as run_systemctl, \
+         patch("web_interface.blueprints.api_v3.display._stop_display_service") as stop_service:
         yield {
-            "get_status": get_status,
+            "state": state,
+            "systemctl": run_systemctl,
             "stop_service": stop_service,
-            "ensure_running": ensure_running,
+            "cache": api_v3_module.api_v3.cache_manager,
         }
 
 
-class TestRestartingARunningService:
-    def test_it_does_not_500(self, api_v3_client, restart_path):
-        response = api_v3_client.post(
-            URL, json={"plugin_id": "weather", "start_service": True})
-        body = response.get_json()
-        assert response.status_code == 200, body
-        assert body["status"] == "success", body
+def _mailbox_writes(cache):
+    return [c.args[1] for c in cache.set.call_args_list if c.args and c.args[0] == MAILBOX]
 
-    def test_the_service_is_actually_stopped_and_restarted(
-            self, api_v3_client, restart_path):
-        api_v3_client.post(
-            URL, json={"plugin_id": "weather", "start_service": True})
-        restart_path["stop_service"].assert_called_once()
-        restart_path["ensure_running"].assert_called_once()
 
-    def test_a_service_that_was_not_running_is_not_stopped_first(
-            self, api_v3_client, restart_path):
-        # The buggy import sits inside `if service_was_running and
-        # start_service`, so it only ever fired on the restart path --
-        # this is the other side of that branch, unaffected either way,
-        # kept here so the branch condition itself stays covered.
-        restart_path["get_status"].return_value = {"active": False}
-        response = api_v3_client.post(
-            URL, json={"plugin_id": "weather", "start_service": True})
+def _systemctl_verbs(run_systemctl):
+    return [c.args[0][-2] for c in run_systemctl.call_args_list]
+
+
+class TestStartWhileTheServiceIsRunning:
+    @pytest.mark.parametrize("body", [
+        {"plugin_id": "weather"},                           # "Preview on display", MQTT
+        {"plugin_id": "weather", "start_service": True},    # on-demand modal, box ticked
+        {"plugin_id": "weather", "start_service": "true"},
+    ])
+    def test_the_service_is_not_stopped_or_restarted(self, api_v3_client, service, body):
+        response = api_v3_client.post(START_URL, json=body)
         assert response.status_code == 200, response.get_json()
-        restart_path["stop_service"].assert_not_called()
+        assert response.get_json()["status"] == "success"
+        service["stop_service"].assert_not_called()
+        assert _systemctl_verbs(service["systemctl"]) == [], (
+            "a running display service was sent a systemctl command")
+
+    def test_the_request_is_posted_for_the_running_display(self, api_v3_client, service):
+        response = api_v3_client.post(
+            START_URL, json={"plugin_id": "weather", "mode": "weather_current",
+                             "duration": 60, "pinned": True})
+        data = response.get_json()["data"]
+        writes = _mailbox_writes(service["cache"])
+        assert len(writes) == 1
+        assert writes[0]["action"] == "start"
+        assert writes[0]["request_id"] == data["request_id"]
+        assert writes[0]["plugin_id"] == "weather"
+        assert writes[0]["mode"] == "weather_current"
+        assert writes[0]["duration"] == 60
+        assert writes[0]["pinned"] is True
+
+    def test_the_response_reports_the_service_was_not_started(self, api_v3_client, service):
+        data = api_v3_client.post(START_URL, json={"plugin_id": "weather"}).get_json()["data"]
+        assert data["service"]["active"] is True
+        assert data["service"]["started"] is False
+
+    def test_it_answers_without_the_old_restart_pause(self, api_v3_client, service):
+        # The restart slept 1.5s; nothing here should sleep at all.
+        with patch("time.sleep") as sleep:
+            api_v3_client.post(START_URL, json={"plugin_id": "weather"})
+        sleep.assert_not_called()
+
+
+class TestStartWhileTheServiceIsStopped:
+    def test_start_service_starts_it_once_and_never_stops_it(self, api_v3_client, service):
+        service["state"]["active"] = False
+        response = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
+        assert response.status_code == 200, response.get_json()
+        assert _systemctl_verbs(service["systemctl"]) == ["start"]
+        service["stop_service"].assert_not_called()
+        # Written before the start, so the new process finds it on its first poll.
+        assert len(_mailbox_writes(service["cache"])) == 1
+
+    def test_without_start_service_it_is_left_stopped(self, api_v3_client, service):
+        service["state"]["active"] = False
+        response = api_v3_client.post(
+            START_URL, json={"plugin_id": "weather", "start_service": "false"})
+        assert response.status_code == 400
+        assert _systemctl_verbs(service["systemctl"]) == []
+
+    def test_a_start_that_fails_is_reported(self, api_v3_client, service):
+        service["state"]["active"] = False
+        service["systemctl"].side_effect = lambda args: {
+            "returncode": 1, "stdout": "", "stderr": "denied"}
+        response = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
+        assert response.status_code == 500
+        assert response.get_json()["status"] == "error"
+
+
+class TestStop:
+    def test_stop_posts_a_stop_request_and_leaves_the_service_running(
+            self, api_v3_client, service):
+        response = api_v3_client.post(STOP_URL, json={})
+        assert response.status_code == 200, response.get_json()
+        writes = _mailbox_writes(service["cache"])
+        assert [w["action"] for w in writes] == ["stop"]
+        service["stop_service"].assert_not_called()
+        assert _systemctl_verbs(service["systemctl"]) == []
+
+    def test_a_string_false_stop_service_does_not_stop_it(self, api_v3_client, service):
+        # bool("false") is True: the flag was read raw and stopped the service.
+        api_v3_client.post(STOP_URL, json={"stop_service": "false"})
+        service["stop_service"].assert_not_called()
+
+    def test_stop_service_true_still_stops_it(self, api_v3_client, service):
+        api_v3_client.post(STOP_URL, json={"stop_service": True})
+        service["stop_service"].assert_called_once()

test/test_favorite_team_check.py

@@ -439,3 +439,101 @@ class ScheduleNoteMatchdayTests(unittest.TestCase):
         # without games, so a future entry there is not a next fixture.
         note = self.note(self.payload([-9], [11], whitelist=False))
         self.assertIn("season has finished", note)
+
+
+class ScheduleNoteListCalendarTests(unittest.TestCase):
+    """A round still to start in a "list" calendar is not a finished season.
+
+    Shapes captured from ESPN on 2026-09-29, with dates kept relative to that
+    day. The Europa League scoreboard still showed the 17 September matchday
+    and its calendar is a ``"list"`` of rounds, not match days, so the check
+    said the season had finished -- with the knockout rounds, and the next
+    league-phase matchday, still to come. PLL, the World Cup and AFL really had
+    finished and must still say so, although each has a season or round
+    ``endDate`` in the future.
+    """
+
+    note = ScheduleNoteTests.note
+
+    @staticmethod
+    def iso(days):
+        return (datetime.now(timezone.utc) + timedelta(days=days)).strftime(
+            "%Y-%m-%dT%H:%MZ")
+
+    @classmethod
+    def list_league(cls, event_days, rounds, league_type=14540,
+                    event_type=14540, phase_label="UEFA Europa League",
+                    extra_phases=()):
+        """``rounds`` is ``[(label, start_day, end_day), ...]`` for one phase."""
+        return {
+            "events": [{"date": cls.iso(d), "season": {"type": event_type}}
+                       for d in event_days],
+            "leagues": [{
+                "season": {"type": {"type": league_type}},
+                "calendarType": "list",
+                "calendarIsWhitelist": True,
+                "calendar": [{
+                    "label": phase_label,
+                    "startDate": cls.iso(-90), "endDate": cls.iso(275),
+                    "entries": [{"label": label, "startDate": cls.iso(start),
+                                 "endDate": cls.iso(end)}
+                                for label, start, end in rounds],
+                }] + list(extra_phases),
+            }],
+        }
+
+    def test_europa_between_matchdays_is_not_finished(self):
+        note = self.note(self.list_league([-12], [
+            ("League Phase", -31, 123),
+            ("Knockout Round Playoffs", 123, 151),
+            ("Rd of 16", 151, 172),
+            ("Quarterfinals", 172, 200),
+            ("Semifinals", 200, 221),
+            ("Final", 222, 275),
+        ]))
+        self.assertIsNone(note)
+
+    def test_world_cup_after_the_final_is_still_finished(self):
+        # The competition runs to 31 December, and the last round ended 12
+        # days after the final; no round is still to start.
+        note = self.note(self.list_league([-72], [
+            ("Group", -110, -93),
+            ("Semifinals", -77, -72),
+            ("Final", -72, -59),
+        ], league_type=13803, event_type=13803, phase_label="FIFA World Cup"))
+        self.assertIn("season has finished", note)
+
+    def test_afl_after_the_grand_final_is_still_finished(self):
+        # The Grand Final round had started but had not ended yet.
+        note = self.note(self.list_league([-3], [
+            ("Preliminary Finals", -13, -6),
+            ("Grand Final", -6, 1),
+        ], league_type=3, event_type=3, phase_label="Postseason"))
+        self.assertIn("season has finished", note)
+
+    def test_an_offseason_round_does_not_count(self):
+        # College football's "Off Season" phase holds the All-Star week.
+        offseason = {"label": "Off Season", "startDate": self.iso(2),
+                     "endDate": self.iso(6),
+                     "entries": [{"label": "All-Star", "startDate": self.iso(2),
+                                  "endDate": self.iso(6)}]}
+        note = self.note(self.list_league(
+            [-3], [("CFP", -40, 1)], league_type=3, event_type=3,
+            phase_label="Postseason", extra_phases=[offseason]))
+        self.assertIn("season has finished", note)
+
+    def test_pll_with_a_season_end_date_in_the_future_is_still_finished(self):
+        # A "day" whitelist whose last match day is past; the season's own
+        # endDate (1 January) is ignored.
+        note = self.note({
+            "events": [{"date": self.iso(-9), "season": {"type": 2}}],
+            "leagues": [{
+                "season": {"type": {"type": 2}, "startDate": self.iso(-271),
+                           "endDate": self.iso(94)},
+                "calendarType": "day",
+                "calendarIsWhitelist": True,
+                "calendarEndDate": self.iso(94),
+                "calendar": [self.iso(-30), self.iso(-22), self.iso(-9)],
+            }],
+        })
+        self.assertIn("season has finished", note)

test/test_on_demand_disabled_plugin.py

@@ -0,0 +1,426 @@
+"""On-demand for a plugin that is installed but disabled in config.
+
+The display process only loads enabled plugins, so a request for a disabled
+one -- "Preview on display" offers it on every plugin's config page, with a
+note that the plugin will be enabled for the preview -- failed with
+"invalid-mode". Nothing loaded it short of a restart, and the on-demand
+route no longer restarts the service.
+
+The display now loads such a plugin live for the session (force_enabled, so
+config.json keeps saying disabled) and the main loop unloads it once
+on-demand moves off it: a stop, an expiry, or a request for another plugin.
+
+Also here: a stop sent after a failed request clears the error instead of
+leaving status 'error' published until the state ages out.
+"""
+
+import time
+from unittest.mock import MagicMock
+
+import pytest
+
+from src.plugin_system.plugin_manager import PluginManager
+from src.plugin_system.plugin_state import PluginState
+
+
+def _make_plugin(modes):
+    plugin = MagicMock()
+    plugin.modes = list(modes)
+    return plugin
+
+
+@pytest.fixture
+def controller(test_display_controller):
+    """An idle controller running 'clock', with 'preview-me' installed but disabled."""
+    c = test_display_controller
+    clock = _make_plugin(['clock'])
+    preview = _make_plugin(['preview_a', 'preview_b'])
+    instances = {'clock': clock}
+    catalogue = {'clock': clock, 'preview-me': preview}
+
+    def load_plugin(plugin_id, force_enabled=False):
+        instances[plugin_id] = catalogue[plugin_id]
+        return True
+
+    def unload_plugin(plugin_id):
+        return instances.pop(plugin_id, None) is not None
+
+    pm = c.plugin_manager
+    pm.discovered_plugin_ids.return_value = set(catalogue)
+    pm.discover_plugins.return_value = list(catalogue)
+    pm.plugin_manifests = {}
+    pm.load_plugin = MagicMock(side_effect=load_plugin)
+    pm.unload_plugin = MagicMock(side_effect=unload_plugin)
+    pm.get_plugin.side_effect = instances.get
+
+    config = {'clock': {'enabled': True}, 'preview-me': {'enabled': False}}
+    c.config_service.get_config = lambda: config
+    c.config_manager.save_config = MagicMock()
+    c.cache_manager.set = MagicMock()
+    c.cache_manager.clear_cache = MagicMock()
+
+    c._register_loaded_plugin('clock')
+    c.current_mode_index = 0
+    c.current_display_mode = 'clock'
+    c.test_config = config
+    c.test_instances = instances
+    return c
+
+
+def _start(c, plugin_id='preview-me', mode=None, **extra):
+    request = {'request_id': 'r-' + plugin_id, 'action': 'start',
+               'plugin_id': plugin_id, 'mode': mode or plugin_id}
+    request.update(extra)
+    c._activate_on_demand(request)
+
+
+class TestLoadingForOnDemand:
+    def test_a_disabled_plugin_is_loaded_and_shown(self, controller):
+        _start(controller)
+
+        controller.plugin_manager.load_plugin.assert_called_once_with(
+            'preview-me', force_enabled=True)
+        assert controller.on_demand_active is True
+        assert controller.on_demand_status == 'active'
+        assert controller.on_demand_plugin_id == 'preview-me'
+        assert controller.current_display_mode == 'preview_a'
+        assert controller.plugin_display_modes['preview-me'] == ['preview_a', 'preview_b']
+
+    def test_config_json_is_not_written(self, controller):
+        _start(controller)
+
+        controller.config_manager.save_config.assert_not_called()
+        assert controller.test_config['preview-me'] == {'enabled': False}
+
+    def test_a_requested_mode_is_honoured(self, controller):
+        _start(controller, mode='preview_b')
+        assert controller.current_display_mode == 'preview_b'
+
+    def test_an_enabled_plugin_is_not_reloaded(self, controller):
+        _start(controller, plugin_id='clock')
+
+        controller.plugin_manager.load_plugin.assert_not_called()
+        assert controller.on_demand_active is True
+        assert controller._on_demand_loaded_plugins == set()
+
+    def test_a_plugin_that_is_not_installed_is_not_loaded(self, controller):
+        _start(controller, plugin_id='uninstalled')
+
+        controller.plugin_manager.load_plugin.assert_not_called()
+        assert controller.on_demand_status == 'error'
+        assert controller.on_demand_last_error == 'invalid-mode'
+
+    def test_a_plugin_installed_after_startup_is_found_by_rescanning(self, controller):
+        controller.plugin_manager.discovered_plugin_ids.return_value = {'clock'}
+
+        _start(controller)
+
+        controller.plugin_manager.discover_plugins.assert_called()
+        assert controller.on_demand_active is True
+
+
+class TestLoadFailures:
+    def test_a_failed_load_reports_load_failed(self, controller):
+        controller.plugin_manager.load_plugin = MagicMock(return_value=False)
+
+        _start(controller)
+
+        assert controller.on_demand_active is False
+        assert controller.on_demand_status == 'error'
+        assert controller.on_demand_last_error == 'load-failed'
+        assert 'preview_a' not in controller.available_modes
+        published = controller.cache_manager.set.call_args_list[-1]
+        assert published.args[0] == 'display_on_demand_state'
+        assert published.args[1]['status'] == 'error'
+        assert published.args[1]['error'] == 'load-failed'
+
+    def test_a_load_that_raises_reports_load_failed(self, controller):
+        controller.plugin_manager.load_plugin = MagicMock(side_effect=ImportError('no module'))
+
+        _start(controller)
+
+        assert controller.on_demand_status == 'error'
+        assert controller.on_demand_last_error == 'load-failed'
+
+    def test_a_failed_load_leaves_the_rotation_alone(self, controller):
+        controller.plugin_manager.load_plugin = MagicMock(return_value=False)
+
+        _start(controller)
+        controller._release_on_demand_plugins()
+
+        assert controller.available_modes == ['clock']
+        assert controller.current_display_mode == 'clock'
+        assert controller._on_demand_loaded_plugins == set()
+        controller.plugin_manager.unload_plugin.assert_not_called()
+
+    def test_a_plugin_that_loads_but_has_no_modes_is_unloaded_again(self, controller):
+        """Registered, then the activation fails: the release removes it."""
+        controller._on_demand_modes_for_plugin = MagicMock(return_value=[])
+
+        _start(controller)
+        assert controller.on_demand_last_error == 'no-modes'
+        controller._release_on_demand_plugins()
+
+        controller.plugin_manager.unload_plugin.assert_called_once_with('preview-me')
+        assert controller.available_modes == ['clock']
+
+
+class TestReleasingThePlugin:
+    def test_it_stays_loaded_while_on_demand_shows_it(self, controller):
+        _start(controller)
+        controller._release_on_demand_plugins()
+
+        controller.plugin_manager.unload_plugin.assert_not_called()
+        assert 'preview_a' in controller.plugin_modes
+
+    def test_a_stop_unloads_it_and_resumes_the_rotation(self, controller):
+        _start(controller)
+        controller._clear_on_demand(reason='requested-stop')
+        # Deferred to the main loop: the stop may be read mid-display().
+        controller.plugin_manager.unload_plugin.assert_not_called()
+
+        controller._release_on_demand_plugins()
+
+        controller.plugin_manager.unload_plugin.assert_called_once_with('preview-me')
+        assert controller.available_modes == ['clock']
+        assert 'preview-me' not in controller.plugin_display_modes
+        assert 'preview_a' not in controller.plugin_modes
+        assert controller.current_display_mode == 'clock'
+        assert controller._on_demand_loaded_plugins == set()
+        assert controller.test_config['preview-me'] == {'enabled': False}
+
+    def test_expiry_unloads_it(self, controller):
+        _start(controller, duration=30)
+        controller.on_demand_expires_at = time.time() - 1
+        controller._check_on_demand_expiration()
+        controller._release_on_demand_plugins()
+
+        assert controller.on_demand_last_event == 'expired'
+        controller.plugin_manager.unload_plugin.assert_called_once_with('preview-me')
+
+    def test_a_request_for_another_plugin_unloads_it(self, controller):
+        _start(controller)
+        _start(controller, plugin_id='clock')
+        controller._release_on_demand_plugins()
+
+        controller.plugin_manager.unload_plugin.assert_called_once_with('preview-me')
+        assert controller.on_demand_active is True
+        assert controller.on_demand_plugin_id == 'clock'
+        assert controller.current_display_mode == 'clock'
+
+    def test_a_failed_request_that_ends_the_session_unloads_it(self, controller):
+        _start(controller)
+        _start(controller, plugin_id='uninstalled')
+        controller._release_on_demand_plugins()
+
+        controller.plugin_manager.unload_plugin.assert_called_once_with('preview-me')
+        assert controller.current_display_mode == 'clock'
+        assert controller.force_change is True
+
+    def test_a_plugin_enabled_during_the_session_stays_loaded(self, controller):
+        _start(controller)
+        controller.test_config['preview-me'] = {'enabled': True}
+        controller._clear_on_demand(reason='requested-stop')
+        controller._release_on_demand_plugins()
+
+        controller.plugin_manager.unload_plugin.assert_not_called()
+        assert 'preview_a' in controller.available_modes
+        assert controller._on_demand_loaded_plugins == set()
+
+    def test_the_main_loop_releases_right_after_its_own_poll(self, controller):
+        """A stop read by the main loop unloads before the next screen, not
+        one screen later. That poll runs with no display() on the stack."""
+        import inspect
+        source = inspect.getsource(type(controller).run)
+        poll = source.index('self._check_on_demand_expiration()')
+        release = source.index('self._release_on_demand_plugins()')
+        render = source.index('self._tick_plugin_updates()')
+        assert poll < release < render
+
+    def test_a_reconcile_that_runs_first_unloads_it_the_same_way(self, controller):
+        """A reconcile queued during the session runs at the top of the loop,
+        before the release: it removes the plugin itself (not in the enabled
+        set) and the release is then a no-op."""
+        _start(controller)
+        controller._clear_on_demand(reason='requested-stop')
+
+        controller._reconcile_enabled_plugins()
+        controller._release_on_demand_plugins()
+
+        controller.plugin_manager.unload_plugin.assert_called_once_with('preview-me')
+        assert controller.available_modes == ['clock']
+        assert controller._on_demand_loaded_plugins == set()
+
+    def test_a_config_save_mid_session_keeps_the_instance_enabled(self, controller):
+        """on_config_change would otherwise read enabled: false and switch it off."""
+        controller.config_service.subscribe = MagicMock()
+        _start(controller)
+        callback = controller._plugin_config_callbacks['preview-me']
+        controller.plugin_manager.prepare_plugin_config = None
+
+        callback({}, {'enabled': False, 'color': 'red'})
+
+        # The change goes through the manager's locked apply_config_change
+        # (which calls on_config_change under the plugin's lock).
+        plugin = controller.plugin_modes['preview_a']
+        controller.plugin_manager.apply_config_change.assert_called_once_with(
+            'preview-me', {'enabled': True, 'color': 'red'}, plugin_instance=plugin)
+
+
+class TestRestoredSession:
+    """A restart during a session for a disabled plugin restores it the same way."""
+
+    def test_the_plugin_is_tracked_and_config_is_left_alone(self, test_display_controller):
+        c = test_display_controller
+        c.config.update({'clock': {'enabled': True}, 'disabled-one': {'enabled': False}})
+
+        selected = c._select_startup_plugins(
+            ['clock', 'disabled-one'], {'plugin_id': 'disabled-one', 'mode': 'x'})
+
+        assert 'disabled-one' in selected
+        assert c._on_demand_loaded_plugins == {'disabled-one'}
+        assert c.config['disabled-one']['enabled'] is False
+
+
+class TestResumingAfterTheSession:
+    """Ending a session never resumes the rotation onto the plugin that is
+    about to be unloaded."""
+
+    def _restored_session(self, c, other_modes=('clock',)):
+        """As after a restart: no saved resume index, and the plugin's modes
+        ordered in ahead of the rest (load order is not deterministic)."""
+        c._on_demand_loaded_plugins.add('preview-me')
+        c.plugin_manager.load_plugin('preview-me', force_enabled=True)
+        c._register_loaded_plugin('preview-me')
+        c.available_modes = ['preview_a', 'preview_b'] + list(other_modes)
+        c.on_demand_active = True
+        c.on_demand_status = 'active'
+        c.on_demand_plugin_id = 'preview-me'
+        c.on_demand_modes = ['preview_a', 'preview_b']
+        c.rotation_resume_index = None
+        c.current_mode_index = 0
+        c.current_display_mode = 'preview_a'
+
+    def test_a_restored_session_resumes_on_an_enabled_mode(self, controller):
+        self._restored_session(controller)
+
+        controller._clear_on_demand(reason='requested-stop')
+
+        assert controller.current_display_mode == 'clock'
+        controller._release_on_demand_plugins()
+        assert controller.available_modes == ['clock']
+        assert controller.current_display_mode == 'clock'
+
+    def test_with_nothing_else_enabled_the_display_goes_idle(self, controller):
+        controller._unregister_plugin('clock')
+        self._restored_session(controller, other_modes=())
+
+        controller._clear_on_demand(reason='requested-stop')
+        assert controller.current_display_mode is None
+
+        controller._release_on_demand_plugins()
+        assert controller.available_modes == []
+        assert controller.current_display_mode is None
+
+    def test_a_saved_resume_index_is_still_used(self, controller):
+        c = controller
+        c.available_modes = ['clock', 'other']
+        c.plugin_modes['other'] = MagicMock()
+        c.current_mode_index = 1
+        c.current_display_mode = 'other'
+
+        _start(c)
+        c._clear_on_demand(reason='requested-stop')
+
+        assert c.current_display_mode == 'other'
+
+
+class TestStopClearsAnError:
+    def _post_stop(self, c):
+        stop = {'request_id': 'S1', 'action': 'stop'}
+        c._last_on_demand_poll = None
+        c.cache_manager.get = MagicMock(
+            side_effect=lambda key, *a, **kw:
+                stop if key == 'display_on_demand_request' else None)
+        c.cache_manager.delete = MagicMock()
+        c._poll_on_demand_requests()
+
+    def test_a_stop_after_a_failed_request_clears_the_error(self, controller):
+        _start(controller, plugin_id='uninstalled')
+        assert controller.on_demand_status == 'error'
+
+        self._post_stop(controller)
+
+        assert controller.on_demand_status == 'idle'
+        assert controller.on_demand_last_error is None
+        state = controller.cache_manager.set.call_args_list[-1].args[1]
+        assert state['status'] == 'idle'
+        assert state['error'] is None
+
+    def test_clearing_the_error_leaves_the_rotation_alone(self, controller):
+        _start(controller, plugin_id='uninstalled')
+        controller.force_change = False
+
+        self._post_stop(controller)
+
+        assert controller.current_display_mode == 'clock'
+        assert controller.force_change is False
+
+    def test_a_stop_while_idle_is_still_just_acknowledged(self, controller):
+        controller._clear_on_demand = MagicMock()
+
+        self._post_stop(controller)
+
+        assert controller.on_demand_status == 'idle'
+        assert controller.on_demand_request_id == 'S1'
+        controller._clear_on_demand.assert_not_called()
+
+
+class TestForceEnabledLoad:
+    """PluginManager.load_plugin(force_enabled=True) runs the plugin enabled
+    without touching the config it read."""
+
+    class _Plugin:
+        def __init__(self, config):
+            self.config = config
+            self.enabled_calls = 0
+
+        def on_enable(self):
+            self.enabled_calls += 1
+
+    @pytest.fixture
+    def pm(self, tmp_path):
+        plugins_dir = tmp_path / 'plugins'
+        (plugins_dir / 'demo').mkdir(parents=True)
+        manager = PluginManager(plugins_dir=str(plugins_dir))
+        manager.plugin_manifests['demo'] = {'id': 'demo', 'name': 'Demo'}
+        manager.schema_manager = MagicMock()
+        manager.schema_manager.get_schema_path.return_value = None
+        # Hand the section back as-is, as the fallback path can: the copy in
+        # load_plugin is what keeps the cached config clean.
+        manager.schema_manager.prepare_plugin_config.side_effect = (
+            lambda pid, cfg, schema=None, changed_paths=None: cfg)
+        manager.plugin_loader = MagicMock()
+        manager.plugin_loader.find_plugin_directory.return_value = plugins_dir / 'demo'
+        manager.plugin_loader.load_plugin.side_effect = (
+            lambda **kw: (self._Plugin(kw['config']), None))
+        manager.config_manager = MagicMock()
+        manager.cached_config = {'demo': {'enabled': False, 'color': 'red'}}
+        manager.config_manager.load_config.return_value = manager.cached_config
+        return manager
+
+    def test_a_disabled_plugin_loads_disabled_by_default(self, pm):
+        assert pm.load_plugin('demo') is True
+        assert pm.plugins['demo'].enabled_calls == 0
+        assert pm.state_manager.get_state('demo') == PluginState.DISABLED
+
+    def test_force_enabled_runs_it_enabled(self, pm):
+        assert pm.load_plugin('demo', force_enabled=True) is True
+        plugin = pm.plugins['demo']
+        assert plugin.config == {'enabled': True, 'color': 'red'}
+        assert plugin.enabled_calls == 1
+        assert pm.state_manager.get_state('demo') == PluginState.ENABLED
+
+    def test_force_enabled_does_not_touch_the_cached_config(self, pm):
+        pm.load_plugin('demo', force_enabled=True)
+        assert pm.cached_config['demo'] == {'enabled': False, 'color': 'red'}

test/test_on_demand_pinning_and_restart.py

@@ -164,12 +164,14 @@ class TestRestartDoesNotStarveTheOtherPlugins:
         assert controller.on_demand_mode == 'app_a'
         assert controller.on_demand_pinned is True
 
-    def test_a_disabled_on_demand_plugin_is_enabled_and_loaded(self, controller):
-        """Otherwise the mode being resumed has nothing behind it."""
+    def test_a_disabled_on_demand_plugin_is_still_loaded(self, controller):
+        """Otherwise the mode being resumed has nothing behind it. It loads
+        for on-demand only; its config section is left disabled."""
         selected = controller._select_startup_plugins(
             self.DISCOVERED, {'plugin_id': 'disabled-one', 'mode': 'x'})
         assert 'disabled-one' in selected
-        assert controller.config['disabled-one']['enabled'] is True
+        assert controller._on_demand_loaded_plugins == {'disabled-one'}
+        assert controller.config['disabled-one']['enabled'] is False
 
     def test_an_unknown_on_demand_plugin_falls_back_to_normal(self, controller):
         selected = controller._select_startup_plugins(

test/test_plugin_config_preparation.py

@@ -84,6 +84,19 @@ def plugin_manager(plugin_dir, tmp_path):
     return manager
 
 
+@pytest.fixture
+def delivering_manager(tmp_path):
+    """A real PluginManager whose apply_config_change() the mocked one uses.
+
+    The hot-reload callback hands on_config_change to the plugin manager, so
+    it runs under the plugin's lock; delegate to the real method so these
+    tests still see the plugin called.
+    """
+    manager = PluginManager(plugins_dir=str(tmp_path / "plugin-repos"))
+    yield manager
+    manager.stop_update_worker()
+
+
 class TestPluginManagerPreparation:
     def test_legacy_boolean_and_defaults(self, plugin_manager):
         prepared = plugin_manager.prepare_plugin_config(
@@ -107,11 +120,13 @@ class TestPluginManagerPreparation:
 
 
 class TestHotReload:
-    def test_on_config_change_gets_the_prepared_config(self, test_display_controller, plugin_manager):
+    def test_on_config_change_gets_the_prepared_config(self, test_display_controller, plugin_manager,
+                                                       delivering_manager):
         controller = test_display_controller
         plugin = MagicMock()
         plugin.modes = ["demo"]
         pm = controller.plugin_manager
+        pm.apply_config_change.side_effect = delivering_manager.apply_config_change
         pm.discover_plugins.return_value = ["demo"]
         pm.load_plugin.return_value = True
         pm.plugin_manifests = {}
@@ -128,11 +143,13 @@ class TestHotReload:
             "enabled": True, "max_duration_seconds": 300}
         assert new_config["nhl"]["show_records"] is True
 
-    def test_raw_section_still_delivered_without_a_preparer(self, test_display_controller):
+    def test_raw_section_still_delivered_without_a_preparer(self, test_display_controller,
+                                                            delivering_manager):
         controller = test_display_controller
         plugin = MagicMock()
         plugin.modes = ["demo"]
         pm = controller.plugin_manager
+        pm.apply_config_change.side_effect = delivering_manager.apply_config_change
         pm.discover_plugins.return_value = ["demo"]
         pm.load_plugin.return_value = True
         pm.plugin_manifests = {}

test/test_plugin_hang_containment.py

@@ -0,0 +1,524 @@
+"""One hung plugin must not stop every other plugin from updating.
+
+The update worker is a single thread. It took each plugin's lock with a
+blocking ``acquire()``, and the render thread holds that same lock while it
+runs the plugin's display(). A display() that never returned -- or a first
+frame that outlived PluginExecutor's timeout and kept running on its lingering
+thread -- parked the worker on that acquire for good, and from then on no
+plugin updated at all: scores, weather and clocks all froze while the panel
+kept scrolling stale data.
+
+Now:
+- the worker waits PLUGIN_LOCK_TIMEOUT at most and skips the busy plugin. The
+  skip is report-only (a busy skip in health): the holder may be a healthy
+  Vegas prefetch render, so it never counts toward the circuit breaker;
+- display() calls are timed, slow ones recorded, overlong ones counted as
+  hangs; update() past the executor timeout is a hang too. Only hangs open
+  the breaker;
+- on_config_change runs under the plugin's lock, deferred to the worker if the
+  lock stays busy, so it never interleaves with update().
+
+Timeouts here are fractions of a second so the suite stays fast.
+"""
+
+import copy
+import threading
+import time
+import types
+from unittest.mock import MagicMock
+
+import pytest
+
+from src.plugin_system.plugin_health import CircuitState, PluginHealthTracker
+from src.plugin_system.plugin_manager import PluginManager
+from src.plugin_system.plugin_state import PluginState
+
+
+class _Cache:
+    """Serialising stand-in for CacheManager; counts writes."""
+
+    def __init__(self):
+        self.store = {}
+        self.writes = 0
+
+    def set(self, key, data, ttl=None, **kwargs):
+        self.writes += 1
+        self.store[key] = copy.deepcopy(data)
+
+    def get(self, key, max_age=None, memory_ttl=None, **kwargs):
+        return copy.deepcopy(self.store.get(key))
+
+
+class _Plugin:
+    """Counts update() calls; display() can be made to block on an Event."""
+
+    def __init__(self, plugin_id, update_seconds=0.0):
+        self.plugin_id = plugin_id
+        self.enabled = True
+        self.update_seconds = update_seconds
+        self.update_calls = 0
+        self.in_update = False
+        self.display_gate = None  # threading.Event: display() blocks on it
+        self.config_changes = []
+        self.overlap = False
+        self.events = []
+
+    def update(self):
+        self.in_update = True
+        self.update_calls += 1
+        self.events.append('update')
+        time.sleep(self.update_seconds)
+        self.in_update = False
+
+    def display(self, force_clear=False):
+        if self.display_gate is not None:
+            self.display_gate.wait(timeout=10)
+        return True
+
+    def on_config_change(self, new_config):
+        if self.in_update:
+            self.overlap = True
+        self.events.append('config')
+        self.config_changes.append(new_config)
+
+
+@pytest.fixture
+def pm(tmp_path):
+    manager = PluginManager(plugins_dir=str(tmp_path), config_manager=None,
+                            display_manager=None, cache_manager=None)
+    manager.PLUGIN_LOCK_TIMEOUT = 0.15
+    yield manager
+    manager.stop_update_worker()
+
+
+@pytest.fixture
+def tracker(pm):
+    t = PluginHealthTracker(cache_manager=_Cache())
+    pm.health_tracker = t
+    return t
+
+
+def _install(pm, plugin, interval=0.01):
+    pm.plugins[plugin.plugin_id] = plugin
+    pm._update_interval_cache[plugin.plugin_id] = interval
+    pm.state_manager.set_state(plugin.plugin_id, PluginState.ENABLED)
+    return plugin.plugin_id
+
+
+def _hang_display(pm, plugin):
+    """Run the plugin's display() the way the render loop does, on its own
+    thread, holding the plugin's lock while display() blocks on its gate."""
+    plugin.display_gate = threading.Event()
+    entered = threading.Event()
+
+    def render():
+        lock = pm.get_plugin_lock(plugin.plugin_id)
+        with lock:
+            entered.set()
+            plugin.display()
+
+    thread = threading.Thread(target=render, daemon=True)
+    thread.start()
+    assert entered.wait(timeout=2)
+    return plugin.display_gate, thread
+
+
+def _wait_for(predicate, timeout=3.0):
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        if predicate():
+            return True
+        time.sleep(0.01)
+    return predicate()
+
+
+class TestHungDisplayDoesNotStallTheWorker:
+    def test_other_plugins_keep_updating_on_schedule(self, pm):
+        hung = _Plugin('hung')
+        healthy = _Plugin('healthy')
+        _install(pm, hung)
+        _install(pm, healthy)
+        gate, render_thread = _hang_display(pm, hung)
+        try:
+            # 'hung' is dict-ordered first, so its item is dequeued ahead of
+            # 'healthy' every round. With a blocking acquire the worker parks
+            # on it forever and 'healthy' never updates.
+            for _ in range(8):
+                pm.run_scheduled_updates()
+                time.sleep(0.05)
+            assert _wait_for(lambda: healthy.update_calls >= 2), (
+                f"healthy plugin updated {healthy.update_calls} time(s) while "
+                "another plugin's display() was hung")
+            assert hung.update_calls == 0
+            assert pm._update_worker.is_alive()
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+
+    def test_hung_plugin_is_retried_once_its_display_returns(self, pm):
+        plugin = _Plugin('hung')
+        _install(pm, plugin)
+        gate, render_thread = _hang_display(pm, plugin)
+        pm.run_scheduled_updates()
+        # Skipped, and handed back rather than left RUNNING.
+        assert _wait_for(lambda: pm.state_manager.can_execute('hung'))
+        assert 'hung' not in pm._pending_updates
+        gate.set()
+        render_thread.join(timeout=2)
+        pm.plugin_last_update.pop('hung', None)  # due again now
+        pm.run_scheduled_updates()
+        assert _wait_for(lambda: plugin.update_calls == 1)
+
+
+class TestLockTimeoutIsReportOnly:
+    def test_skip_lands_in_health_and_state_as_a_busy_skip(self, pm, tracker):
+        plugin = _Plugin('hung')
+        _install(pm, plugin)
+        gate, render_thread = _hang_display(pm, plugin)
+        try:
+            pm.run_scheduled_updates()
+            assert _wait_for(lambda: tracker.get_health_summary('hung')['busy_skip_count'] == 1)
+            summary = tracker.get_health_summary('hung')
+            assert summary['last_busy_skip']['operation'] == 'update lock wait'
+            assert summary['last_busy_skip']['seconds'] >= pm.PLUGIN_LOCK_TIMEOUT - 0.01
+            # Reporting only: no failure, no hang, no error, breaker closed.
+            assert summary['consecutive_failures'] == 0
+            assert summary['total_failures'] == 0
+            assert summary['hang_count'] == 0
+            assert summary['last_error'] is None
+            assert summary['circuit_state'] == CircuitState.CLOSED.value
+            # Still visible in the plugin's state for the web UI.
+            error_info = pm.state_manager.get_error_info('hung')
+            assert error_info['error_type'] == 'PluginBusyError'
+            # Stamped like any failed update, so the retry waits an interval.
+            assert pm.plugin_last_update.get('hung', 0) > 0
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+
+    def test_repeated_busy_skips_never_open_the_circuit_breaker(self, pm, tracker):
+        """A Vegas prefetch render can hold a healthy plugin's lock past the
+        bound on every update; that must not pull it from rotation."""
+        plugin = _Plugin('busy')
+        _install(pm, plugin)
+        gate, render_thread = _hang_display(pm, plugin)
+        skips = tracker.failure_threshold * 3
+        try:
+            for attempt in range(1, skips + 1):
+                pm.run_scheduled_updates()
+                assert _wait_for(
+                    lambda: pm.state_manager.can_execute('busy')
+                    and tracker.get_health_summary('busy')['busy_skip_count'] == attempt)
+                time.sleep(0.02)  # past the 0.01s interval
+            summary = tracker.get_health_summary('busy')
+            assert summary['busy_skip_count'] == skips
+            assert summary['consecutive_failures'] == 0
+            assert summary['hang_count'] == 0
+            assert summary['circuit_state'] == CircuitState.CLOSED.value
+            assert tracker.should_skip_plugin('busy') is False
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+        # Once the lock frees the plugin updates normally.
+        pm.plugin_last_update.pop('busy', None)
+        pm.run_scheduled_updates()
+        assert _wait_for(lambda: plugin.update_calls == 1)
+
+    def test_busy_skips_do_not_add_to_a_real_hang_streak(self, pm, tracker):
+        plugin = _Plugin('p')
+        _install(pm, plugin)
+        tracker.record_hang('p', 'display', 31.0)
+        gate, render_thread = _hang_display(pm, plugin)
+        try:
+            for attempt in range(1, tracker.failure_threshold + 2):
+                pm.run_scheduled_updates()
+                assert _wait_for(
+                    lambda: pm.state_manager.can_execute('p')
+                    and tracker.get_health_summary('p')['busy_skip_count'] == attempt)
+                time.sleep(0.02)
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+        summary = tracker.get_health_summary('p')
+        assert summary['consecutive_failures'] == 1
+        assert summary['hang_count'] == 1
+        assert summary['circuit_state'] == CircuitState.CLOSED.value
+
+    def test_repeated_real_hangs_still_open_the_circuit_breaker(self, pm, tracker):
+        """display() past the executor timeout is a real hang: the breaker
+        opens after failure_threshold of them, busy skips in between or not."""
+        plugin = _Plugin('hung')
+        _install(pm, plugin)
+        gate, render_thread = _hang_display(pm, plugin)
+        too_long = pm.plugin_executor.default_timeout + 1
+        try:
+            for attempt in range(1, tracker.failure_threshold + 1):
+                pm.run_scheduled_updates()
+                assert _wait_for(
+                    lambda: pm.state_manager.can_execute('hung')
+                    and tracker.get_health_summary('hung')['busy_skip_count'] == attempt)
+                assert tracker.get_health_summary('hung')['circuit_state'] ==                     CircuitState.CLOSED.value
+                pm.note_display_duration('hung', too_long)
+                time.sleep(0.02)  # past the 0.01s interval
+            summary = tracker.get_health_summary('hung')
+            assert summary['hang_count'] == tracker.failure_threshold
+            assert summary['busy_skip_count'] == tracker.failure_threshold
+            assert summary['circuit_state'] == CircuitState.OPEN.value
+            assert tracker.should_skip_plugin('hung') is True
+            # Circuit open: the scheduler no longer queues it at all.
+            pm.run_scheduled_updates()
+            assert 'hung' not in pm._pending_updates
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+
+    def test_skip_is_logged_once_per_interval(self, pm):
+        pm.logger = MagicMock()
+        plugin = _Plugin('hung')
+        _install(pm, plugin)
+        gate, render_thread = _hang_display(pm, plugin)
+        try:
+            for _ in range(3):
+                pm.run_scheduled_updates()
+                assert _wait_for(lambda: pm.state_manager.can_execute('hung'))
+                time.sleep(0.02)
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+        busy_warnings = [c for c in pm.logger.warning.call_args_list
+                         if 'update skipped' in c.args[0]]
+        assert len(busy_warnings) == 1
+
+    def test_unloaded_while_waiting_is_not_resurrected(self, pm, tracker):
+        plugin = _Plugin('hung')
+        _install(pm, plugin)
+        gate, render_thread = _hang_display(pm, plugin)
+        try:
+            pm.PLUGIN_LOCK_TIMEOUT = 0.3
+            pm.UNLOAD_LOCK_TIMEOUT = 0.05
+            pm.run_scheduled_updates()
+            time.sleep(0.05)  # worker is now waiting on the lock
+            assert pm.unload_plugin('hung') is True
+            assert _wait_for(lambda: 'hung' not in pm._pending_updates)
+            time.sleep(0.35)
+            assert pm.state_manager.get_state('hung') == PluginState.UNLOADED
+            summary = tracker.get_health_summary('hung')
+            assert summary['hang_count'] == 0
+            assert summary['busy_skip_count'] == 0
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+
+
+class TestUpdateOutlivingTheExecutorTimeout:
+    def test_recorded_as_a_hang_then_cleared_when_it_returns(self, pm, tracker):
+        plugin = _Plugin('slow', update_seconds=0.4)
+        _install(pm, plugin)
+        pm.plugin_executor.default_timeout = 0.05
+        pm.run_scheduled_updates()
+        assert _wait_for(lambda: tracker.get_health_summary('slow')['hang_count'] == 1)
+        summary = tracker.get_health_summary('slow')
+        assert summary['last_hang']['operation'] == 'update'
+        assert summary['consecutive_failures'] == 1
+        # The real update() then returns: success resets the streak.
+        assert _wait_for(lambda: pm.state_manager.can_execute('slow'))
+        assert _wait_for(lambda: tracker.get_health_summary('slow')['consecutive_failures'] == 0)
+
+
+class TestDisplayTiming:
+    def test_fast_frames_record_nothing(self, pm, tracker):
+        pm.logger = MagicMock()
+        for _ in range(100):
+            pm.note_display_duration('p', 0.004)
+        assert tracker.get_health_summary('p')['slow_call_count'] == 0
+        pm.logger.warning.assert_not_called()
+
+    def test_slow_display_is_recorded_and_warned_once(self, pm, tracker):
+        pm.logger = MagicMock()
+        for _ in range(5):
+            pm.note_display_duration('p', 2.5)
+        summary = tracker.get_health_summary('p')
+        assert summary['slow_call_count'] == 5
+        assert summary['last_slow_call']['operation'] == 'display'
+        assert summary['last_slow_call']['seconds'] == 2.5
+        # Slow is not failing: the circuit breaker is untouched.
+        assert summary['consecutive_failures'] == 0
+        assert summary['circuit_state'] == CircuitState.CLOSED.value
+        assert pm.logger.warning.call_count == 1
+
+    def test_suppressed_repeats_are_counted_in_the_next_warning(self, pm):
+        pm.logger = MagicMock()
+        for _ in range(4):
+            pm.note_display_duration('p', 2.5)
+        pm.HANG_LOG_INTERVAL = 0.0
+        pm.note_display_duration('p', 2.5)
+        assert pm.logger.warning.call_count == 2
+        last = pm.logger.warning.call_args
+        assert '3 more since the last warning' in (last.args[0] % last.args[1:])
+
+    def test_display_past_the_executor_timeout_is_a_hang(self, pm, tracker):
+        pm.plugin_executor.default_timeout = 3.0
+        for _ in range(tracker.failure_threshold):
+            pm.note_display_duration('p', 3.5)
+        summary = tracker.get_health_summary('p')
+        assert summary['hang_count'] == tracker.failure_threshold
+        assert summary['last_hang']['operation'] == 'display'
+        assert summary['circuit_state'] == CircuitState.OPEN.value
+
+    def test_render_loop_times_each_frame(self, monkeypatch, emulator_mode):
+        """DisplayController._display_once hands every frame's duration on."""
+        from src import display_controller as dc_module
+        ticks = iter([100.0, 103.25])
+        monkeypatch.setattr(dc_module, 'time', types.SimpleNamespace(
+            monotonic=lambda: next(ticks)))
+        controller = dc_module.DisplayController.__new__(dc_module.DisplayController)
+        controller.plugin_manager = MagicMock()
+        lock = threading.Lock()
+        controller.plugin_manager.get_plugin_lock.return_value = lock
+        plugin = _Plugin('ticker')
+
+        assert controller._display_once(plugin, 'ticker', False) is True
+
+        controller.plugin_manager.note_display_duration.assert_called_once_with('ticker', 3.25)
+        assert not lock.locked()
+
+    def test_skipped_frame_is_not_timed(self, emulator_mode):
+        from src import display_controller as dc_module
+        controller = dc_module.DisplayController.__new__(dc_module.DisplayController)
+        controller.plugin_manager = MagicMock()
+        lock = threading.Lock()
+        lock.acquire()  # update() in flight
+        controller.plugin_manager.get_plugin_lock.return_value = lock
+        try:
+            assert controller._display_once(_Plugin('ticker'), 'ticker', False) is True
+        finally:
+            lock.release()
+        controller.plugin_manager.note_display_duration.assert_not_called()
+
+
+class TestConfigChangeIsSerialised:
+    def test_does_not_run_concurrently_with_update(self, pm):
+        pm.PLUGIN_LOCK_TIMEOUT = 2.0
+        plugin = _Plugin('p', update_seconds=0.3)
+        _install(pm, plugin)
+        pm.run_scheduled_updates()
+        assert _wait_for(lambda: plugin.in_update)
+
+        applied = pm.apply_config_change('p', {'enabled': True, 'n': 1})
+
+        assert applied is True
+        assert plugin.overlap is False
+        assert plugin.events == ['update', 'config']
+        assert plugin.config_changes == [{'enabled': True, 'n': 1}]
+
+    def test_runs_holding_the_plugin_lock(self, pm):
+        plugin = _Plugin('p')
+        _install(pm, plugin)
+        held = {}
+        plugin.on_config_change = lambda cfg: held.setdefault(
+            'locked', pm.get_plugin_lock('p').locked())
+        assert pm.apply_config_change('p', {}) is True
+        assert held == {'locked': True}
+        assert not pm.get_plugin_lock('p').locked()
+
+    def test_busy_lock_defers_the_latest_change_to_before_the_next_update(self, pm):
+        plugin = _Plugin('p')
+        _install(pm, plugin)
+        gate, render_thread = _hang_display(pm, plugin)
+        try:
+            start = time.monotonic()
+            assert pm.apply_config_change('p', {'n': 1}) is False
+            assert time.monotonic() - start < 1.0, "the wait must be bounded"
+            assert pm.apply_config_change('p', {'n': 2}) is False
+            time.sleep(0.4)  # the worker's own bounded retry also finds it busy
+            assert plugin.config_changes == []
+        finally:
+            gate.set()
+            render_thread.join(timeout=2)
+
+        pm.run_scheduled_updates()
+        assert _wait_for(lambda: plugin.update_calls == 1)
+        # Only the latest parked change, applied before the update ran.
+        assert plugin.config_changes == [{'n': 2}]
+        assert plugin.events == ['config', 'update']
+        assert plugin.overlap is False
+
+    def test_deferred_change_applied_by_worker_once_lock_frees(self, pm):
+        """No update due to piggyback on: the worker's own queued retry
+        applies it. Timeline: the watcher gives up at 0.2s, the worker waits
+        from 0.2s to 0.4s, the holder lets go at 0.3s."""
+        pm.PLUGIN_LOCK_TIMEOUT = 0.2
+        plugin = _Plugin('p')
+        _install(pm, plugin, interval=3600)
+        pm.plugin_last_update['p'] = time.time()  # not due
+        lock = pm.get_plugin_lock('p')
+        lock.acquire()
+        releaser = threading.Timer(0.3, lock.release)
+        releaser.start()
+        try:
+            assert pm.apply_config_change('p', {'n': 1}) is False
+            assert _wait_for(lambda: plugin.config_changes == [{'n': 1}])
+        finally:
+            releaser.join(timeout=2)
+        assert plugin.update_calls == 0
+        assert 'p' not in pm._deferred_config_changes
+
+    def test_stale_deferred_change_is_dropped_after_reload(self, pm):
+        plugin = _Plugin('p')
+        _install(pm, plugin)
+        pm._deferred_config_changes['p'] = (plugin, {'n': 1})
+        replacement = _Plugin('p')
+        pm.plugins['p'] = replacement
+        pm._apply_deferred_config_change('p')
+        assert plugin.config_changes == []
+        assert replacement.config_changes == []
+        assert 'p' not in pm._deferred_config_changes
+
+    def test_exceptions_still_reach_the_caller(self, pm):
+        plugin = _Plugin('p')
+        _install(pm, plugin)
+
+        def boom(cfg):
+            raise ValueError("bad config")
+
+        plugin.on_config_change = boom
+        with pytest.raises(ValueError):
+            pm.apply_config_change('p', {})
+        assert not pm.get_plugin_lock('p').locked()
+
+
+class TestHealthRecords:
+    def test_slow_call_persistence_is_rate_limited(self):
+        cache = _Cache()
+        t = PluginHealthTracker(cache_manager=cache)
+        t.record_slow_call('p', 'display', 2.5)
+        writes = cache.writes
+        for _ in range(50):
+            t.record_slow_call('p', 'display', 2.5)
+        assert cache.writes == writes
+        assert t.get_health_summary('p')['slow_call_count'] == 51
+
+    def test_busy_skip_persistence_is_rate_limited_and_breaker_free(self):
+        cache = _Cache()
+        t = PluginHealthTracker(cache_manager=cache)
+        t.record_busy_skip('p', 'update lock wait', 5.0)
+        writes = cache.writes
+        # The first one is saved at once, so the web process sees it.
+        assert PluginHealthTracker(cache_manager=cache).get_health_summary(
+            'p')['busy_skip_count'] == 1
+        for _ in range(50):
+            t.record_busy_skip('p', 'update lock wait', 5.0)
+        assert cache.writes == writes
+        summary = t.get_health_summary('p')
+        assert summary['busy_skip_count'] == 51
+        assert summary['consecutive_failures'] == 0
+        assert summary['circuit_state'] == CircuitState.CLOSED.value
+        assert t.should_skip_plugin('p') is False
+
+    def test_hang_survives_a_restart(self):
+        cache = _Cache()
+        PluginHealthTracker(cache_manager=cache).record_hang('p', 'display', 31.0)
+        summary = PluginHealthTracker(cache_manager=cache).get_health_summary('p')
+        assert summary['hang_count'] == 1
+        assert summary['last_hang']['operation'] == 'display'
+        assert summary['consecutive_failures'] == 1

test/test_sports_card_wrappers.py

@@ -0,0 +1,152 @@
+"""src.common.sports_card_wrappers: each delegation, and the host contract.
+
+Every method here forwards to the ``sports_card`` function it names with the
+host's ``config`` and ``logger``. The tests pin what each returns for a
+configured card, so a delegation that passes the wrong thing -- an empty
+config, the other side, a dropped default -- fails here rather than as a
+wrong colour on a panel.
+"""
+
+import ast
+import logging
+from pathlib import Path
+from zoneinfo import ZoneInfo
+
+from PIL import ImageFont
+
+from src.common import sports_card, sports_card_wrappers, sports_game_renderer
+from src.common.sports_card_wrappers import SportsCardWrappersMixin
+from src.common.sports_game_renderer import SportsGameRendererMixin
+
+CONFIG = {
+    "timezone": "America/Chicago",
+    "favorite_teams": ["BOS"],
+    "scroll_card": {"vs_text": "@", "upcoming_center": "date_time",
+                    "date_format": "weekday", "time_format": "24h"},
+    "customization": {
+        "score_text": {"text_color": [9, 9, 9]},
+        "favorite_result_colors": {"enabled": True, "win_color": [0, 200, 0]},
+    },
+}
+
+GAME = {"home_abbr": "BOS", "away_abbr": "NYY", "home_score": "3", "away_score": "1",
+        "start_time_utc": "2026-09-19T23:00:00Z"}
+
+
+class Host(SportsCardWrappersMixin):
+    """The documented contract, and not one attribute more."""
+
+    _FONT_NAME_ALIASES = dict(sports_card.FONT_NAME_ALIASES)
+    _FONT_PIXEL_GRID = dict(sports_card.FONT_PIXEL_GRID)
+
+    def __init__(self, config=CONFIG):
+        self.config = config
+        self.logger = logging.getLogger("test.sports_card_wrappers")
+        self.fonts = {"score": ImageFont.load_default()}
+
+
+class TestDelegations:
+    def test_card_options(self):
+        host = Host()
+        assert host._scroll_card_option("vs_text", "VS") == "@"
+        assert host._scroll_card_option("missing", "fallback") == "fallback"
+        assert host._vs_text() == "@"
+        assert host._upcoming_center_mode() == "date_time"
+
+    def test_dates_and_times(self):
+        host = Host()
+        assert host._card_tzinfo() == ZoneInfo("America/Chicago")
+        assert host._weekday_for(GAME) == "Sat"          # 18:00 in Chicago
+        assert host._format_game_time("7:05 PM") == "19:05"
+        assert host._format_game_date("9/19", GAME) == sports_card.format_game_date(
+            CONFIG, host.logger, "9/19", GAME)
+        assert host._format_game_date("9/19", GAME) != "9/19"
+
+    def test_colours(self):
+        host = Host()
+        assert tuple(host._element_color("score_text")) == (9, 9, 9)
+        assert tuple(host._element_color("missing_element", (1, 2, 3))) == (1, 2, 3)
+        assert tuple(host._font_color(host.fonts["score"])) == (9, 9, 9)
+        assert host._coerce_rgb([300, -1, "7"], (1, 2, 3)) == (255, 0, 7)
+
+    def test_favourites(self):
+        host = Host()
+        assert host._side_is_favorite(GAME, "home", {"BOS"}) is True
+        assert host._side_is_favorite(GAME, "away", {"BOS"}) is False
+        assert host._side_score(GAME, "home") == 3
+        assert host._favorite_result(GAME) == "win"
+        assert host._recent_score_color(GAME, (1, 1, 1)) == (0, 200, 0)
+        assert host._score_color_for(GAME, "recent") == (0, 200, 0)
+        assert tuple(host._score_color_for(GAME, "live")) == (9, 9, 9)
+
+    def test_fonts(self):
+        host = Host()
+        font = host.fonts["score"]
+        unshared = host._unshare_element_fonts({"score": font, "time": font})
+        assert set(unshared) == {"score", "time"}
+
+    def test_crisp_size_uses_the_hosts_own_tables(self):
+        class NoTables(Host):
+            _FONT_NAME_ALIASES = {}
+            _FONT_PIXEL_GRID = {}
+
+        assert Host._crisp_size("PressStart2P-Regular.ttf", 9) == 8   # snapped
+        assert NoTables._crisp_size("PressStart2P-Regular.ttf", 9) == 9  # unknown face
+
+
+class TestComposition:
+    def test_it_supplies_what_the_geometry_mixin_needs(self):
+        # sports_game_renderer's docstring lists these as host-provided.
+        for name in ("_scroll_card_option", "_upcoming_center_mode", "_vs_text",
+                     "_element_color", "_format_game_date", "_format_game_time"):
+            assert f"``{name}``" in sports_game_renderer.__doc__
+            assert name in SportsCardWrappersMixin.__dict__
+
+    def test_the_two_mixins_share_no_names(self):
+        ours = {n for n in SportsCardWrappersMixin.__dict__ if not n.startswith("__")}
+        theirs = {n for n in SportsGameRendererMixin.__dict__ if not n.startswith("__")}
+        assert ours & theirs == set()
+
+    def test_a_renderers_own_method_wins(self):
+        class Renderer(SportsCardWrappersMixin, SportsGameRendererMixin):
+            def _vs_text(self):
+                return "v"
+
+            def __init__(self):
+                self.config, self.logger = CONFIG, logging.getLogger("t")
+
+        assert Renderer()._vs_text() == "v"
+        assert Renderer()._upcoming_center_mode() == "date_time"
+
+
+# ---------------------------------------------------------------------------
+# Host contract
+# ---------------------------------------------------------------------------
+
+def _self_reads():
+    """Every ``self.X`` / ``cls.X`` / ``getattr(self, "X")`` the mixin reads."""
+    tree = ast.parse(Path(sports_card_wrappers.__file__).read_text(encoding="utf-8"))
+    cls = next(n for n in tree.body
+               if isinstance(n, ast.ClassDef) and n.name == "SportsCardWrappersMixin")
+    names = set()
+    for node in ast.walk(cls):
+        if (isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load)
+                and isinstance(node.value, ast.Name) and node.value.id in ("self", "cls")):
+            names.add(node.attr)
+        if (isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
+                and node.func.id == "getattr" and len(node.args) >= 2
+                and isinstance(node.args[0], ast.Name) and node.args[0].id == "self"
+                and isinstance(node.args[1], ast.Constant)):
+            names.add(node.args[1].value)
+    return names
+
+
+class TestHostContract:
+    def test_every_host_read_is_documented(self):
+        needed = _self_reads() - set(dir(SportsCardWrappersMixin))
+        undocumented = sorted(n for n in needed if f"``{n}``" not in sports_card_wrappers.__doc__)
+        assert undocumented == [], f"read but not in the host contract: {undocumented}"
+
+    def test_the_mixin_creates_no_attributes_of_its_own(self):
+        for name in ("config", "logger", "fonts", "_FONT_NAME_ALIASES", "_FONT_PIXEL_GRID"):
+            assert not hasattr(SportsCardWrappersMixin, name)

test/test_sports_celebration.py

@@ -0,0 +1,344 @@
+"""src.common.sports_celebration: the palette, the takeover, and the host contract.
+
+Ported from the scoreboards' own celebration tests (football's
+test_score_celebration.py, hockey's and soccer's test_goal_celebration.py),
+which drive the same code through a plugin's SportsLive. Here the host is a
+stub carrying exactly the documented contract, and the crests are drawn by the
+test, so every input is fixed. Pixel-exact goldens of every plugin's takeover
+live in ledmatrix-plugins (scripts/test_celebration_renders.py).
+"""
+
+import ast
+import logging
+from pathlib import Path
+from unittest import mock
+
+import pytest
+from PIL import Image, ImageChops, ImageDraw, ImageFont
+
+from src.common import sports_celebration
+from src.common.sports_celebration import (
+    SportsCelebrationMixin,
+    cap_luminance,
+    lift_color,
+    logo_palette,
+    mix_color,
+    rgb_luminance,
+    rgb_saturation,
+    scale_color,
+)
+
+FONTS = Path(__file__).resolve().parents[1] / "assets" / "fonts"
+SIZES = [(64, 32), (128, 32), (64, 64), (96, 48),
+         (128, 64), (256, 32), (128, 96), (256, 128)]
+
+
+def crest(body, band, size=64):
+    """A shield in ``body`` with a horizontal band in ``band``."""
+    img = Image.new("RGBA", (size, size), (0, 0, 0, 0))
+    draw = ImageDraw.Draw(img)
+    s = size / 64
+    draw.polygon([(6 * s, 4 * s), (58 * s, 4 * s), (58 * s, 34 * s),
+                  (32 * s, 60 * s), (6 * s, 34 * s)], fill=body + (255,))
+    draw.rectangle([(6 * s, 22 * s), (58 * s, 32 * s)], fill=band + (255,))
+    return img
+
+
+CRESTS = {
+    "RED": crest((200, 16, 46), (255, 255, 255)),
+    "NAV": crest((12, 35, 64), (255, 184, 28)),
+    "SIL": crest((165, 172, 175), (0, 0, 0)),
+}
+
+
+class _DisplayManager:
+    def __init__(self, width, height):
+        self.width, self.height = width, height
+        self.image = Image.new("RGB", (width, height))
+        self.updates = 0
+
+    def clear(self):
+        self.image = Image.new("RGB", (self.width, self.height))
+
+    def update_display(self):
+        self.updates += 1
+
+
+class Host(SportsCelebrationMixin):
+    """The documented contract, and not one attribute more."""
+
+    def __init__(self, width=128, height=32, **knobs):
+        self.display_manager = _DisplayManager(width, height)
+        self.display_width, self.display_height = width, height
+        press = str(FONTS / "PressStart2P-Regular.ttf")
+        self.fonts = {
+            "time": ImageFont.truetype(press, 8),
+            "status": ImageFont.truetype(str(FONTS / "4x6-font.ttf"), 6),
+            "score": ImageFont.truetype(press, 16 if height >= 48 else 10),
+        }
+        self.logger = logging.getLogger("test.sports_celebration")
+        for name, value in knobs.items():
+            setattr(self, name, value)
+
+    def _load_and_resize_logo(self, team_id, abbr, logo_path, logo_url):
+        logo = CRESTS.get(abbr)
+        if logo is None:
+            return None
+        logo = logo.copy()
+        logo.thumbnail((self.display_height, self.display_height), Image.Resampling.LANCZOS)
+        return logo
+
+    def _draw_text_with_outline(self, draw, text, position, font,
+                                fill=(255, 255, 255), outline_color=(0, 0, 0)):
+        x, y = position
+        for dx, dy in ((-1, 0), (1, 0), (0, -1), (0, 1)):
+            draw.text((x + dx, y + dy), text, font=font, fill=outline_color)
+        draw.text((x, y), text, font=font, fill=fill)
+
+
+def celebration(scorer="RED", other="NAV", side="away", kind="score", motif="score"):
+    away, home = (scorer, other) if side == "away" else (other, scorer)
+    return {
+        "kind": kind, "motif": motif,
+        "game": {"id": "401", "away_abbr": away, "home_abbr": home},
+        "scored_side": side, "team_abbr": scorer,
+        "away_score": 3, "home_score": 2, "started_at": 1000.0,
+        "phrase": f"{scorer} WINS!" if kind == "win" else f"{scorer} SCORES!",
+    }
+
+
+def render(host=None, elapsed=2.0, **kwargs):
+    host = host or Host()
+    with mock.patch("time.time", return_value=1000.0 + elapsed):
+        host._draw_celebration_layout(celebration(**kwargs), force_clear=True)
+    return host.display_manager.image
+
+
+def brightest(img, box=None):
+    region = img.crop(box) if box else img
+    return max(region.convert("RGB").getextrema()[i][1] for i in range(3))
+
+
+# ---------------------------------------------------------------------------
+# Colour helpers
+# ---------------------------------------------------------------------------
+
+class TestColourHelpers:
+    def test_mix_is_clamped_to_the_two_ends(self):
+        assert mix_color((0, 0, 0), (200, 100, 50), 0.5) == (100, 50, 25)
+        assert mix_color((0, 0, 0), (200, 100, 50), 2) == (200, 100, 50)
+        assert mix_color((0, 0, 0), (200, 100, 50), -1) == (0, 0, 0)
+
+    def test_scale_is_clamped_to_the_panel(self):
+        assert scale_color((200, 100, 0), 2) == (255, 200, 0)
+
+    def test_saturation_of_black_is_zero(self):
+        assert rgb_saturation((0, 0, 0)) == 0.0
+
+    def test_lifting_a_colour_keeps_its_hue(self):
+        # Scaling channels turns Baltimore's navy-purple magenta; HSV does not.
+        lifted = lift_color((39, 15, 98))
+        assert rgb_luminance(lifted) >= 100
+        assert lifted[2] > lifted[0] > lifted[1]
+
+    def test_a_colour_that_already_reads_is_left_alone(self):
+        assert lift_color((255, 208, 56)) == (255, 208, 56)
+
+    def test_a_grey_is_just_made_bright(self):
+        lifted = lift_color((40, 40, 40))
+        assert lifted[0] == lifted[1] == lifted[2] and rgb_luminance(lifted) > 200
+
+    def test_capping_keeps_the_hue_and_the_cap(self):
+        capped = cap_luminance((248, 61, 1), 34)
+        assert capped[0] > capped[1] > capped[2]
+        assert rgb_luminance(capped) <= 35
+
+
+class TestLogoPalette:
+    def test_a_saturated_crest_is_its_own_headline(self):
+        palette = logo_palette(CRESTS["RED"])
+        r, g, b = palette["headline"]
+        assert r > 150 and r > 2 * g and r > 2 * b
+        assert rgb_luminance(palette["deep"]) <= 36
+
+    def test_a_legible_band_beats_lifting_a_dark_body(self):
+        palette = logo_palette(CRESTS["NAV"])
+        r, g, b = palette["headline"]
+        assert r > 150 and g > 110 and b < 110, f"{palette['headline']} is not the gold band"
+        assert palette["deep"][2] >= palette["deep"][0], "the backdrop lost the navy"
+
+    def test_a_crest_with_no_colour_falls_back_to_its_brightest_grey(self):
+        palette = logo_palette(CRESTS["SIL"])
+        assert palette is not None
+        assert rgb_saturation(palette["headline"]) < 0.12
+
+    def test_nothing_opaque_is_no_palette(self):
+        assert logo_palette(Image.new("RGBA", (16, 16))) is None
+
+    def test_an_unreadable_crest_is_no_palette(self):
+        assert logo_palette(object()) is None
+
+    def test_every_colour_has_three_channels(self):
+        palette = logo_palette(CRESTS["RED"])
+        assert set(palette) == {"deep", "glow", "headline", "accent"}
+        assert all(len(c) == 3 for c in palette.values())
+
+
+# ---------------------------------------------------------------------------
+# The celebration's palette
+# ---------------------------------------------------------------------------
+
+class TestCelebrationPalette:
+    def test_read_off_the_scoring_side(self):
+        away = Host()._celebration_palette(celebration(side="away"))
+        home = Host()._celebration_palette(celebration(scorer="NAV", other="RED", side="home"))
+        assert away == logo_palette(Host()._load_and_resize_logo(None, "RED", None, None))
+        assert home["headline"] != away["headline"]
+
+    def test_worked_out_once_per_celebration(self):
+        host, c = Host(), celebration()
+        first = host._celebration_palette(c)
+        host._load_and_resize_logo = mock.Mock(side_effect=AssertionError("reloaded"))
+        assert host._celebration_palette(c) is first
+
+    @pytest.mark.parametrize("loader", [lambda *a: None, mock.Mock(side_effect=OSError("bad png"))])
+    def test_no_usable_crest_falls_back(self, loader):
+        host = Host()
+        host._load_and_resize_logo = loader
+        assert host._celebration_palette(celebration()) == Host._DEFAULT_CELEBRATION_PALETTE
+
+    def test_team_colours_off_is_the_default(self):
+        host = Host(celebration_team_colors=False)
+        assert host._celebration_palette(celebration()) == Host._DEFAULT_CELEBRATION_PALETTE
+
+
+# ---------------------------------------------------------------------------
+# The takeover
+# ---------------------------------------------------------------------------
+
+class TestTakeover:
+    def test_frame_is_presented(self):
+        host = Host()
+        render(host)
+        assert host.display_manager.updates == 1
+        assert host.display_manager.image.size == (128, 32)
+
+    def test_same_inputs_same_frame(self):
+        assert render().tobytes() == render().tobytes()
+
+    def test_the_highlight_follows_the_scoring_side(self):
+        away = render(side="away", scorer="RED", other="RED")
+        home = render(side="home", scorer="RED", other="RED")
+        assert ImageChops.difference(away, home).getbbox() is not None
+
+    def test_each_motif_paints_its_own_scenery(self):
+        shots = {m: render(Host(celebration_confetti=False), motif=m).tobytes()
+                 for m in ("score", "kick", "touchdown", "net", "win")}
+        assert len(set(shots.values())) == len(shots)
+
+    def test_an_unknown_motif_draws_the_score_scenery(self):
+        host = Host(celebration_confetti=False)
+        assert render(host, motif="bogus").tobytes() == render(Host(celebration_confetti=False),
+                                                                motif="score").tobytes()
+
+    @pytest.mark.parametrize("knob", ["celebration_team_colors", "celebration_confetti"])
+    def test_switches_change_the_frame(self, knob):
+        assert render(Host(**{knob: False})).tobytes() != render(Host()).tobytes()
+
+    def test_the_matrix_size_wins_over_the_configured_one(self):
+        host = Host(width=64, height=32)
+        host.display_manager.matrix = type("M", (), {"width": 128, "height": 32})()
+        assert render(host).size == (128, 32)
+
+    @pytest.mark.parametrize("width,height", SIZES)
+    def test_every_frame_is_a_finished_card(self, width, height):
+        # A switch-mode board samples once a second: any frame may be the only
+        # one seen, so each carries the headline and nothing is blank.
+        for elapsed in [0.0] + [i + 0.5 for i in range(8)]:
+            img = render(Host(width, height), elapsed=elapsed)
+            assert brightest(img) > 40
+            assert brightest(img, (0, 0, width, max(2, height // 4))) > 60
+
+    def test_the_score_stays_on_a_tall_panel(self):
+        # 16px digits at 48 tall used to run off the bottom row. The goal
+        # line keeps the scenery off that row, so only the score could be.
+        img = render(Host(192, 48, celebration_confetti=False), motif="touchdown")
+        assert brightest(img, (48, 47, 144, 48)) < 10
+        assert brightest(img, (48, 24, 144, 47)) > 10
+
+    def test_the_scoring_side_breathes_rather_than_toggling(self):
+        frames = {render(Host(celebration_confetti=False), elapsed=t).tobytes()
+                  for t in (0.9, 1.9, 2.9, 3.9, 4.9, 5.9)}
+        assert len(frames) > 2
+
+    def test_confetti_is_seeded_from_the_game_not_the_clock(self):
+        host, c = Host(), celebration()
+        palette = host._celebration_palette(c)
+        flakes = host._celebration_confetti(c, 128, 32, palette)
+        again = Host()._celebration_confetti(celebration(), 128, 32, palette)
+        assert flakes == again and 6 <= len(flakes) <= 22
+
+    def test_confetti_is_gone_by_the_end(self):
+        host, c = Host(), celebration()
+        palette = host._celebration_palette(c)
+        overlay = Image.new("RGBA", (128, 32), (0, 0, 0, 0))
+        host._draw_celebration_confetti(ImageDraw.Draw(overlay), c, 128, 32, palette, 8.0, 1.0)
+        assert overlay.getbbox() is None
+
+    def test_the_side_that_did_not_score_is_dimmed(self):
+        crests = Host()._celebration_crests(celebration(scorer="RED", other="RED"), 32)
+        assert brightest(crests["home"]) < brightest(crests["away"])
+
+    def test_a_crest_that_fails_to_load_is_left_out(self):
+        host = Host()
+        host._load_and_resize_logo = mock.Mock(side_effect=OSError("bad png"))
+        assert host._celebration_crests(celebration(), 32) == {"away": None, "home": None}
+        assert brightest(render(host)) > 40
+
+    def test_fit_font_falls_back_to_the_smallest(self):
+        host = Host()
+        draw = ImageDraw.Draw(Image.new("RGB", (8, 8)))
+        fonts = [host.fonts["time"], host.fonts["status"]]
+        assert host._fit_font(draw, "A", 128, fonts) is fonts[0]
+        assert host._fit_font(draw, "A VERY LONG HEADLINE", 8, fonts) is fonts[-1]
+
+
+# ---------------------------------------------------------------------------
+# Host contract
+# ---------------------------------------------------------------------------
+
+def _self_reads():
+    """Every ``self.X`` / ``getattr(self, "X")`` the mixin reads, by parsing it."""
+    tree = ast.parse(Path(sports_celebration.__file__).read_text(encoding="utf-8"))
+    cls = next(n for n in tree.body
+               if isinstance(n, ast.ClassDef) and n.name == "SportsCelebrationMixin")
+    names = set()
+    for node in ast.walk(cls):
+        if (isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load)
+                and isinstance(node.value, ast.Name) and node.value.id == "self"):
+            names.add(node.attr)
+        if (isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
+                and node.func.id == "getattr" and len(node.args) >= 2
+                and isinstance(node.args[0], ast.Name) and node.args[0].id == "self"
+                and isinstance(node.args[1], ast.Constant)):
+            names.add(node.args[1].value)
+    return names
+
+
+class TestHostContract:
+    def test_every_host_read_is_documented(self):
+        needed = _self_reads() - set(dir(SportsCelebrationMixin))
+        undocumented = sorted(n for n in needed if f"``{n}" not in sports_celebration.__doc__)
+        assert undocumented == [], f"read but not in the host contract: {undocumented}"
+
+    def test_the_stub_host_is_enough(self):
+        # Host above sets the contract and nothing else; it drew every test.
+        needed = _self_reads() - set(dir(SportsCelebrationMixin))
+        host = Host(celebration_duration=8, celebration_team_colors=True,
+                    celebration_confetti=True)
+        assert all(hasattr(host, n) for n in needed)
+
+    def test_the_mixin_creates_no_attributes_of_its_own(self):
+        # The annotations are for type checking; the host's values must win.
+        for name in ("display_manager", "fonts", "logger", "_load_and_resize_logo"):
+            assert not hasattr(SportsCelebrationMixin, name)

test/test_sports_fetch.py

@@ -0,0 +1,193 @@
+"""src.common.sports_fetch: behaviour and host contract.
+
+Ported from the scoreboards' tests of the same methods (football's
+test_live_odds_follow_the_rotation.py, test_lookback_only_when_it_can_matter.py
+and test_espn_date_ranges.py) against a stub host carrying exactly the
+documented contract.
+"""
+
+import ast
+import logging
+import threading
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+import pytest
+
+from src.common import espn_dates, sports_fetch
+from src.common.sports_fetch import SportsFetchMixin
+
+ET = timezone(timedelta(hours=-5))
+
+
+class _Response:
+    status_code = 200
+    content = None
+
+    def __init__(self, data):
+        self._data = data
+
+    def json(self):
+        return self._data
+
+    def raise_for_status(self):
+        pass
+
+
+class _Session:
+    def __init__(self, data=None, error=None):
+        self.data, self.error, self.calls = data, error, []
+
+    def get(self, url, params=None, headers=None, timeout=None):
+        self.calls.append((url, dict(params or {}), headers, timeout))
+        if self.error:
+            raise self.error
+        return _Response(self.data)
+
+
+class _Cache:
+    def __init__(self):
+        self.sets = []
+
+    def set(self, key, data, **kwargs):
+        self.sets.append((key, data, kwargs))
+
+
+class Host(SportsFetchMixin):
+    """The documented contract, and not one attribute more."""
+
+    def __init__(self, session=None):
+        self.session = session or _Session(data={"events": []})
+        self.headers = {"User-Agent": "test"}
+        self.cache_manager = _Cache()
+        self.logger = logging.getLogger("test.sports_fetch")
+        self._games_lock = threading.RLock()
+
+
+class TestWantsLiveOdds:
+    def test_cold_start_asks_for_every_game(self):
+        assert Host()._wants_live_odds({"id": "a"}) is True
+
+    def test_only_the_game_on_screen_and_the_next(self):
+        host = Host()
+        host.live_games = [{"id": i} for i in "abcd"]
+        host.current_game_index = 1
+        assert [host._wants_live_odds({"id": i}) for i in "abcd"] == [False, True, True, False]
+
+    def test_the_rotation_schedule_is_followed_and_wraps(self):
+        host = Host()
+        host.live_games = [{"id": i} for i in "abcd"]
+        host._rotation_schedule = ["d", "c", "b", "a"]
+        host.current_game_index = 3
+        assert [host._wants_live_odds({"id": i}) for i in "abcd"] == [True, False, False, True]
+
+    def test_an_index_past_the_end_starts_at_the_front(self):
+        host = Host()
+        host.live_games = [{"id": i} for i in "abc"]
+        host.current_game_index = 9
+        assert [host._wants_live_odds({"id": i}) for i in "abc"] == [True, True, False]
+
+    def test_the_lookahead_is_a_class_setting(self):
+        class Wider(Host):
+            _LIVE_ODDS_LOOKAHEAD = 2
+
+        host = Wider()
+        host.live_games = [{"id": i} for i in "abcd"]
+        host.current_game_index = 0
+        assert [host._wants_live_odds({"id": i}) for i in "abcd"] == [True, True, True, False]
+
+
+class TestNeedsPreviousDay:
+    def test_before_the_cutoff_yesterday_is_kept(self):
+        assert Host()._needs_previous_day(datetime(2026, 1, 15, 5, 59, tzinfo=ET)) is True
+
+    def test_after_it_with_nothing_live_it_is_dropped(self):
+        assert Host()._needs_previous_day(datetime(2026, 1, 15, 6, 0, tzinfo=ET)) is False
+
+    def test_a_live_game_from_yesterday_keeps_it(self):
+        host = Host()
+        host.live_games = [{"start_time_utc": datetime(2026, 1, 15, 3, 0, tzinfo=timezone.utc)}]
+        assert host._needs_previous_day(datetime(2026, 1, 15, 12, 0, tzinfo=ET)) is True
+
+    def test_todays_live_game_does_not(self):
+        host = Host()
+        host.live_games = [{"start_time_utc": datetime(2026, 1, 15, 18, 0, tzinfo=timezone.utc)}]
+        assert host._needs_previous_day(datetime(2026, 1, 15, 12, 0, tzinfo=ET)) is False
+
+    @pytest.mark.parametrize("game", [{}, {"start_time_utc": "2026-01-14"}, "not a game"])
+    def test_unusable_start_times_are_skipped(self, game):
+        host = Host()
+        host.live_games = [game]
+        assert host._needs_previous_day(datetime(2026, 1, 15, 12, 0, tzinfo=ET)) is False
+
+
+class TestBackgroundFetchesEspnRanges:
+    def test_no_service(self):
+        assert Host()._background_fetches_espn_ranges() is False
+
+    @pytest.mark.parametrize("flag,expected", [(True, True), (False, False), (None, False)])
+    def test_follows_the_service(self, flag, expected):
+        host = Host()
+        host.background_service = type("S", (), {"handles_espn_date_ranges": flag})()
+        assert host._background_fetches_espn_ranges() is expected
+
+    def test_an_old_service_without_the_flag(self):
+        host = Host()
+        host.background_service = object()
+        assert host._background_fetches_espn_ranges() is False
+
+
+class TestFetchSeasonDirectly:
+    def test_fetches_caches_and_returns(self):
+        host = Host(_Session(data={"events": [1, 2]}))
+        data = host._fetch_season_directly("http://espn/sb", "20260115", "k", "2026 season")
+        assert data == {"events": [1, 2]}
+        assert host.cache_manager.sets == [("k", data, {})]
+        assert host.session.calls == [
+            ("http://espn/sb", {"dates": "20260115", "limit": espn_dates.ESPN_MAX_LIMIT},
+             {"User-Agent": "test"}, 30)]
+
+    def test_a_ttl_reaches_the_cache(self):
+        host = Host()
+        host._fetch_season_directly("http://espn/sb", "20260115", "k", "x", ttl=60)
+        assert host.cache_manager.sets[0][2] == {"ttl": 60}
+
+    def test_a_failure_returns_none_and_caches_nothing(self, caplog):
+        host = Host(_Session(error=OSError("down")))
+        with caplog.at_level(logging.ERROR):
+            assert host._fetch_season_directly("http://espn/sb", "20260115", "k", "2026 season") is None
+        assert host.cache_manager.sets == []
+        assert "Failed to fetch 2026 season schedule" in caplog.text
+
+
+# ---------------------------------------------------------------------------
+# Host contract
+# ---------------------------------------------------------------------------
+
+def _self_reads():
+    """Every ``self.X`` / ``getattr(self, "X")`` the mixin reads, by parsing it."""
+    tree = ast.parse(Path(sports_fetch.__file__).read_text(encoding="utf-8"))
+    cls = next(n for n in tree.body
+               if isinstance(n, ast.ClassDef) and n.name == "SportsFetchMixin")
+    names = set()
+    for node in ast.walk(cls):
+        if (isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load)
+                and isinstance(node.value, ast.Name) and node.value.id == "self"):
+            names.add(node.attr)
+        if (isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
+                and node.func.id == "getattr" and len(node.args) >= 2
+                and isinstance(node.args[0], ast.Name) and node.args[0].id == "self"
+                and isinstance(node.args[1], ast.Constant)):
+            names.add(node.args[1].value)
+    return names
+
+
+class TestHostContract:
+    def test_every_host_read_is_documented(self):
+        needed = _self_reads() - set(dir(SportsFetchMixin))
+        undocumented = sorted(n for n in needed if f"``{n}``" not in sports_fetch.__doc__)
+        assert undocumented == [], f"read but not in the host contract: {undocumented}"
+
+    def test_the_mixin_creates_no_attributes_of_its_own(self):
+        for name in ("session", "headers", "cache_manager", "logger", "_games_lock"):
+            assert not hasattr(SportsFetchMixin, name)

test/test_sports_stage3_parity.py

@@ -0,0 +1,148 @@
+"""The stage 3 sports modules still match every plugin copy that remains.
+
+``sports_celebration``, ``sports_fetch`` and ``sports_card_wrappers`` were
+copied from the scoreboard plugins, which delete their copies once they floor
+on the release that ships these. Until each has, a copy that changes on its
+own is a fix one side has and the other lacks. Point LEDMATRIX_PLUGINS at a
+ledmatrix-plugins checkout and every body here is compared, as an AST with
+docstrings and type annotations removed and public names folded to the
+plugins' private spelling, against every plugin copy. A copy that is gone
+counts as adopted. Without the variable this skips: core CI has no plugins
+checkout.
+"""
+
+import ast
+import os
+from pathlib import Path
+
+import pytest
+
+from src.common import sports_card_wrappers, sports_celebration, sports_fetch
+
+#: module -> (its mixin, plugin file, plugin class, carriers,
+#: {plugin: names it deliberately overrides}).
+MODULES = {
+    sports_celebration: ("SportsCelebrationMixin", "sports.py", "SportsLive",
+                         ("afl", "football", "hockey", "nrl", "soccer"), {}),
+    sports_fetch: ("SportsFetchMixin", "sports.py", "SportsCore",
+                   ("afl", "baseball", "basketball", "football", "hockey",
+                    "lacrosse", "nrl", "soccer", "ufc"), {}),
+    sports_card_wrappers: ("SportsCardWrappersMixin", "game_renderer.py", "GameRenderer",
+                           ("afl", "baseball", "basketball", "football", "hockey",
+                            "lacrosse", "nrl", "soccer"),
+                           {"football": {"_format_game_date", "_upcoming_center_mode"}}),
+}
+
+#: Type aliases the modules declare for annotations; nothing to compare.
+TYPE_ALIASES = {"Color", "Palette", "Flake", "_Buckets"}
+
+#: Public here, private in the plugins.
+RENAMES = {name: "_" + name for name in (
+    "rgb_luminance", "rgb_saturation", "color_distance", "mix_color",
+    "scale_color", "lift_color", "cap_luminance", "dim_rgba", "logo_palette")}
+
+
+def _plugins_root():
+    raw = os.environ.get("LEDMATRIX_PLUGINS")
+    if not raw:
+        pytest.skip("set LEDMATRIX_PLUGINS to a ledmatrix-plugins checkout to "
+                    "compare these modules against the plugin copies")
+    root = Path(raw)
+    if (root / "plugins").is_dir():
+        root = root / "plugins"
+    if not (root / "football-scoreboard" / "sports.py").is_file():
+        pytest.skip(f"LEDMATRIX_PLUGINS={raw} has no football-scoreboard/sports.py")
+    return root
+
+
+class _Normalise(ast.NodeTransformer):
+    """Drop docstrings and annotations; fold public names to private ones."""
+
+    def visit_Name(self, node):
+        node.id = RENAMES.get(node.id, node.id)
+        return node
+
+    def visit_arg(self, node):
+        node.annotation = None
+        return node
+
+    def visit_AnnAssign(self, node):
+        return self.visit(ast.Assign(targets=[node.target], value=node.value, lineno=0))
+
+    def visit_FunctionDef(self, node):
+        node.name = RENAMES.get(node.name, node.name)
+        node.returns = None
+        body = node.body
+        if (body and isinstance(body[0], ast.Expr)
+                and isinstance(body[0].value, ast.Constant)
+                and isinstance(body[0].value.value, str)):
+            node.body = body[1:] or [ast.Pass()]
+        self.generic_visit(node)
+        return node
+
+
+def _dump(node):
+    node = ast.parse(ast.unparse(node)).body[0]  # detach and copy
+    return ast.dump(_Normalise().visit(node))
+
+
+def _definitions(tree, class_name):
+    """Module-level functions and assignments, plus ``class_name``'s members."""
+    found = {}
+
+    def add(node, owner):
+        if isinstance(node, ast.FunctionDef):
+            found[(owner, node.name)] = node
+        elif isinstance(node, (ast.Assign, ast.AnnAssign)):
+            target = node.targets[0] if isinstance(node, ast.Assign) else node.target
+            if isinstance(target, ast.Name) and node.value is not None:
+                found[(owner, target.id)] = node
+
+    for node in tree.body:
+        add(node, "module")
+        if isinstance(node, ast.ClassDef) and node.name == class_name:
+            for item in node.body:
+                add(item, "class")
+    return found
+
+
+def _promoted(module, mixin):
+    """What the module moved: its functions and ``_PALETTE_*``-style constants,
+    and its mixin's methods and constants (not the host-contract annotations)."""
+    tree = ast.parse(Path(module.__file__).read_text(encoding="utf-8"))
+    ours = {}
+    for (owner, name), node in _definitions(tree, mixin).items():
+        if name in TYPE_ALIASES:
+            continue
+        ours[(owner, RENAMES.get(name, name))] = node
+    return ours
+
+
+CASES = [(module.__name__.rsplit(".", 1)[1], key)
+         for module, (mixin, *_rest) in MODULES.items()
+         for key in sorted(_promoted(module, mixin))]
+
+
+@pytest.mark.parametrize("module_name,key", CASES, ids=lambda v: str(v))
+def test_every_remaining_plugin_copy_matches(module_name, key):
+    root = _plugins_root()
+    module = next(m for m in MODULES if m.__name__.endswith("." + module_name))
+    mixin, filename, class_name, carriers, overrides = MODULES[module]
+    ours = _dump(_promoted(module, mixin)[key])
+    drifted, missing = [], []
+    for sport in carriers:
+        source = (root / f"{sport}-scoreboard" / filename).read_text(encoding="utf-8")
+        theirs = _definitions(ast.parse(source), class_name).get(key)
+        if key[1] in overrides.get(sport, ()):
+            continue
+        if theirs is None:
+            # Gone is fine once the plugin uses the module; otherwise the
+            # finder is not seeing its copy.
+            if module.__name__ not in source:
+                missing.append(sport)
+        elif _dump(theirs) != ours:
+            drifted.append(sport)
+    assert missing == [], f"{key[1]} not found in: {missing}"
+    assert drifted == [], (
+        f"{key[1]} in {module_name} differs from the copy in: {drifted}. "
+        f"Port the change to both, or stop treating it as shared.")

test/web_interface/test_startup_reconciliation_disarmed.py

@@ -0,0 +1,23 @@
+"""
+No test may start the real app's startup reconciliation.
+
+web_interface/app.py reads the checkout's real config/config.json and
+plugin-repos/ at import, and its first request launches a reconciliation
+thread that reinstalls every configured-but-missing plugin from the live
+store. A full suite run on a dev checkout used to leave whole plugins
+untracked in plugin-repos/ that way. test/conftest.py disarms the run-once
+latch on every import of the module; this pins that.
+"""
+
+from unittest.mock import MagicMock, patch
+
+
+def test_a_request_to_the_imported_app_launches_no_reconciliation():
+    import web_interface.app as web_app
+
+    assert web_app._reconciliation_started is True
+
+    with patch.object(web_app, "threading", MagicMock()) as threading_mock:
+        web_app.app.test_client().get("/favicon.ico")
+
+    threading_mock.Thread.assert_not_called()

web_interface/blueprints/api_v3/display.py

@@ -180,9 +180,9 @@ def start_on_demand_display():
             if not resolved_plugin:
                 return jsonify({'status': 'error', 'message': f'Mode {resolved_mode} not found'}), 404
 
-    # Note: On-demand can work with disabled plugins - the display controller
-    # will temporarily enable them during initialization if needed
-    # We don't block the request here, but log it for debugging
+    # On-demand works with disabled plugins: the running display loads one
+    # for the session and unloads it afterwards, leaving config.json alone
+    # (DisplayController._load_plugin_for_on_demand). Logged for debugging.
     if api_v3.config_manager and resolved_plugin:
         config = api_v3.config_manager.load_config()
         plugin_config = config.get(resolved_plugin, {})
@@ -192,8 +192,9 @@ def start_on_demand_display():
                 resolved_plugin,
             )
 
-    # Set the on-demand request in cache FIRST (before starting service)
-    # This ensures the request is available when the service starts/restarts
+    # Post the request to the mailbox the display process polls
+    # (DisplayController._poll_on_demand_requests). Written before any
+    # service start, so a freshly started display finds it on its first poll.
     cache = _cache_manager()
     request_id = data.get('request_id') or str(uuid.uuid4())
     request_payload = {
@@ -207,18 +208,7 @@ def start_on_demand_display():
     }
     cache.set('display_on_demand_request', request_payload)
 
-    # Check if display service is running (or will be started)
     service_status = _get_display_service_status()
-    service_was_running = service_status.get('active', False)
-        
-    # Stop the display service first to ensure clean state when we will restart it
-    if service_was_running and start_service:
-        import time as time_module
-        logger.debug("Stopping display service before starting on-demand mode")
-        _stop_display_service()
-        # Wait a brief moment for the service to fully stop
-        time_module.sleep(1.5)
-        logger.debug("Display service stopped, now starting with on-demand request")
 
     if not service_status.get('active') and not start_service:
         return jsonify({
@@ -227,6 +217,18 @@ def start_on_demand_display():
             'service_status': service_status
         }), 400
 
+    # start_service means "start it if it is not running", as the UI's
+    # checkbox says; _ensure_display_service_running leaves a running service
+    # alone. This used to stop a running service, sleep 1.5s and start it
+    # again, so every on-demand or "Preview on display" click -- and every
+    # MQTT on-demand command, which posts here with the default -- cold-
+    # restarted the display process: every plugin reloaded and the panel was
+    # blank for seconds. The restart bought nothing. The running process
+    # reads this mailbox every ON_DEMAND_POLL_INTERVAL (0.25s), from its
+    # dwell sleep, its render loops and Vegas's interrupt check as well as
+    # the main loop, and a restarted one got the request the same way: the
+    # startup path only restores a session the display itself saved
+    # (display_on_demand_config), so it loaded nothing it would not have had.
     service_result = None
     if start_service:
         service_result = _ensure_display_service_running()
@@ -237,9 +239,6 @@ def start_on_demand_display():
                 'message': 'Failed to start display service. Please check service logs or start it manually.',
                 'service_result': service_result
             }), 500
-            
-        # Service was restarted (or started fresh) with on-demand request in cache
-        # The display controller will read the request during initialization or when it polls
 
     response_data = {
         'request_id': request_id,
@@ -254,10 +253,12 @@ def start_on_demand_display():
 def stop_on_demand_display():
     """Request the display controller to stop on-demand mode."""
     data = request.get_json(silent=True) or {}
-    stop_service = data.get('stop_service', False)
+    # _coerce_to_bool: bool("false") is True, which stopped the service.
+    stop_service = _coerce_to_bool(data.get('stop_service', False))
 
-    # Set the stop request in cache FIRST
-    # The display controller will poll this and restart without the on-demand filter
+    # The running display reads the stop from the mailbox within
+    # ON_DEMAND_POLL_INTERVAL and resumes normal rotation in place
+    # (_clear_on_demand); nothing is restarted.
     cache = _cache_manager()
     request_id = data.get('request_id') or str(uuid.uuid4())
     request_payload = {
@@ -266,10 +267,7 @@ def stop_on_demand_display():
         'timestamp': _pkg.time.time()
     }
     cache.set('display_on_demand_request', request_payload)
-        
-    # Note: The display controller's _clear_on_demand() will handle the restart
-    # to restore normal operation with all plugins
-        
+
     service_result = None
     if stop_service:
         service_result = _stop_display_service()