Merge origin/main into fix/auto-update-not-a-plugin

Resolve the CHANGELOG Unreleased conflict with #587 and #588 by keeping all
three entries.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -19,6 +19,60 @@ accepts both, but the store flags the old spelling as deprecated
 
 ## Unreleased
 
+New module a plugin may import via `src.*` (floor on the release that ships
+this):
+
+- `src/common/sports_helpers.py` — the helpers the scoreboards' `sports.py`
+  carry byte-identical copies of: `clamp_window`, `clamp_seconds`,
+  `logo_needs_refresh`, `spread_weighted_order` (+ `MIN_WINDOW_DAYS`,
+  `MAX_WINDOW_DAYS`), and `SportsHelpersMixin` with `_mode_customization`,
+  `_setting_int`, `_reset_dwell_on_reentry`, `_next_switch_index`,
+  `_spread_weighted_order`, `_odds_color`, `_upcoming_date_and_time_text` under
+  the plugins' names and signatures, plus the `_favorite_key` override point.
+  Constructor-free; keeps lazy state on its host (see the module docstring,
+  which also gives the host contract).
+  A new module rather than more methods on `sports_shared`: a plugin that
+  deletes a copy and leans on an older module having grown the method fails at
+  runtime with `AttributeError`, which no load-time check sees, while a missing
+  module fails at load. Nothing in core uses it yet.
+- `test/test_common_is_hardware_free.py` — `src/common` must import without
+  `rgbmatrix` and never import `src.base_classes`, `src.display_manager` or
+  `src.plugin_system` at module level.
+
+Web interface:
+
+- The plugin settings form honours `"x-display": "hidden"` in config schemas:
+  the property gets no control at any depth (top level, nested objects, array
+  rows, Advanced Settings), and saving the form never changes its stored value.
+  JSON API saves are unaffected. Lets plugins keep deprecated or internal keys
+  declared, e.g. countdown's row `id` and weather's `api_key` / `radar_zoom`.
+  See `docs/widget-guide.md`.
+- The Plugin Config Warning no longer lists core settings as plugins that are
+  "in config but not installed" (seen as `auto_update` on 3.4.0, where the
+  advice would have deleted the weekly-update setting). Core top-level config
+  keys now live in one list, `src/core_config_keys.py`, which reconciliation
+  uses and tests pin to `config.template.json` and the settings save endpoint.
+  A stored warning is also dropped once its entry is no longer a plugin in
+  config, so an old verdict clears without a restart.
+- **Check & Update All** no longer sends installed Starlark apps
+  (`starlark:<app_id>` entries in `/plugins/installed`) to the plugin updater,
+  which answered each with a 500 "plugin not found". `POST /plugins/update`
+  now answers a `starlark:` id with a 400 saying it is a Starlark app. A
+  request that gets no HTTP answer (e.g. the web service restarting mid-run) is
+  re-sent with backoff instead of being counted as failed and skipped — that is
+  how a disabled plugin with an update waiting was silently left out.
+
+Plugin system:
+
+- A plugin no longer starts with a schema warning and a degraded flag because
+  config.json still holds a boolean where its schema now has an object with an
+  `enabled` property (news' `global.dynamic_duration: true`). The loader reads
+  the boolean as `{"enabled": <bool>}` before merging schema defaults and
+  validating, the same rule the settings form already applies
+  (`legacy_bool_as_object` in `src/plugin_system/schema_manager.py`). Nothing
+  is written at load; the next save of that plugin's settings stores the object.
+  Other type mismatches still warn.
+
 ## 3.4.0
 
 Plugin-facing changes since 3.3.0 (tag `v3.3.1`) not covered further down:

docs/SPORTS_UNIFICATION.md

@@ -80,11 +80,30 @@ src/base_classes/sports/
 src/common/
   sports_scroll.py       SportsScrollDisplay / …Manager — scroll orchestration
                          (content building stays in the plugins)
+  sports_helpers.py      clamp/logo/rotation free functions + SportsHelpersMixin
+                         (unreleased) — the helpers byte-identical in the
+                         plugins' sports.py, and the _favorite_key seam
 ```
 
 `from src.base_classes.sports import SportsCore` keeps working — the package
 `__init__` re-exports, so the conversion is invisible to every existing importer.
 
+### Converging on `src/common`
+
+The scoreboards do not build on `src/base_classes`; their own `sports.py` copies
+have moved past it. So shared code now lands in hardware-free `src/common`
+modules taken from the plugin copies, each a **new module** rather than growth
+on an existing one: a plugin that deletes a method copy and relies on an older
+module having gained it fails at runtime with an `AttributeError`, while a
+missing module fails at load, where the version checks can see it.
+`sports_helpers.py` is the first (it holds `_favorite_key`, the override point
+listed below, for later phases); its parity test compares every body against
+the plugin copies when `LEDMATRIX_PLUGINS` points at a checkout, and
+`test/test_common_is_hardware_free.py` keeps `src/common` free of
+`rgbmatrix`, `src.base_classes` and `src.plugin_system`. How a plugin adopts a
+module and drops its copy is documented in the plugins repo's
+`docs/plugin-development/08-shared-sports-code.md`.
+
 ## Override points (the plugin-facing seam)
 
 The base class calls these; plugins implement or override them. This table is the

docs/widget-guide.md

@@ -281,6 +281,42 @@ Guidelines:
   their own collapsible sections) and is safely ignored by older cores, so
   adding it never breaks compatibility.
 
+## Hiding Fields From the Form (`x-display: "hidden"`)
+
+Add `"x-display": "hidden"` to a property that must stay in the schema but
+should not appear as a control: a deprecated key kept so existing configs keep
+validating, or an internal value such as an auto-generated row id.
+
+```json
+{
+  "properties": {
+    "radar_zoom": {
+      "type": "integer",
+      "default": 6,
+      "title": "Radar Zoom Level (deprecated)",
+      "x-display": "hidden"
+    }
+  }
+}
+```
+
+What the core does with it:
+
+- **Not rendered** at any depth: top-level fields, children of an object
+  section, and properties of array-of-object items (never a table column, even
+  if `x-columns` names it, and never in the row editor). A hidden field flagged
+  `x-advanced` is not listed or counted in Advanced Settings, and an object
+  whose children are all hidden draws no empty section. Hidden fields don't
+  show up in the settings search either, since it indexes the rendered form.
+- **Stored value preserved on save.** Saving the form never changes a hidden
+  value. The unchecked-checkbox rule ignores a hidden boolean. Array rows carry
+  a hidden property's stored value through the form, so the value survives the
+  row being posted back; a new row gets no value (the plugin fills it in).
+- **The API is unaffected.** A JSON save to `POST /api/v3/plugins/config` can
+  still set a hidden field.
+
+Older cores ignore the flag and render the field as a normal control.
+
 ## Creating Custom Widgets
 
 ### Step 1: Create Widget File

src/common/sports_helpers.py

@@ -0,0 +1,408 @@
+"""Sports helpers every scoreboard's ``sports.py`` carries an identical copy of.
+
+Ten helpers are byte-identical (executable AST, docstrings stripped) in the
+scoreboard plugins' ``sports.py`` and have no equivalent elsewhere in
+``src/common``. The bodies below were copied from those plugin copies -- not
+from ``src/base_classes`` -- at ledmatrix-plugins ``f09bff2`` (origin/main,
+2026-09-14):
+
+- In all nine (afl, baseball, basketball, football, hockey, lacrosse, nrl,
+  soccer, ufc): ``_clamp_window``, ``_clamp_seconds``, ``_logo_needs_refresh``
+  (with ``_MIN_WINDOW_DAYS`` / ``_MAX_WINDOW_DAYS``), and the ``SportsCore``
+  methods ``_mode_customization``, ``_setting_int``,
+  ``_reset_dwell_on_reentry`` (with ``_DWELL_REENTRY_GAP_SECONDS``),
+  ``_next_switch_index`` and ``_spread_weighted_order``.
+- In the eight that share a lineage (all but ufc, which has neither):
+  ``_odds_color`` and ``_upcoming_date_and_time_text``.
+
+The module-level helpers become public free functions here (``clamp_window``,
+``clamp_seconds``, ``logo_needs_refresh``, ``spread_weighted_order``,
+``MIN_WINDOW_DAYS``, ``MAX_WINDOW_DAYS``); the methods keep the plugins' exact
+names and signatures on ``SportsHelpersMixin`` so adopting it is deleting the
+copies and adding one base. ``test/test_sports_helpers.py`` compares every body
+here against every plugin copy when a plugins checkout is available
+(``LEDMATRIX_PLUGINS``), so drift in either direction fails a test.
+
+WHY A NEW MODULE, NOT MORE METHODS ON sports_shared
+---------------------------------------------------
+A plugin that deletes its copy of a method and relies on an *existing* module
+having grown it fails at runtime on an older core: ``AttributeError`` mid-render,
+usually swallowed into a blank board. The plugin loader cannot see that, and
+neither can the plugins' ``scripts/check_min_core_version.py``, which checks
+modules. A plugin importing a module that does not exist yet fails at load,
+where both can. Plugins import this and floor ``ledmatrix_min_version`` on the
+first core release that ships it (see ``CHANGELOG.md``).
+
+``_favorite_key`` is the one method not taken from the plugins: it is the
+override point from ``src/base_classes/sports/core.py``, carried here so later
+phases (shared celebrations and game selection) have a hardware-free home for
+the seam. No plugin defines it today and nothing in this module calls it.
+
+WHAT A HOST MUST PROVIDE
+------------------------
+Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
+test in ``test/test_sports_helpers.py`` fails if a read is added without being
+listed here.
+
+- ``config`` (dict) -- ``_mode_customization``.
+- ``mode_config`` (dict) and ``logger`` -- ``_setting_int``. ``league`` is
+  read with ``getattr`` for the warning text only.
+- ``games_list`` and ``current_game_index`` -- ``_next_switch_index``; plus
+  ``_is_favorite_game`` (called with a game), which stays per-plugin and is
+  only called when
+  ``favorite_rotation_boost`` is above 1. ``favorite_rotation_boost`` itself
+  defaults to 1 on the mixin.
+- ``last_game_switch`` -- ``_reset_dwell_on_reentry``, read with ``getattr``
+  and written back.
+- ``_card_option``, ``_format_game_date`` and ``_format_game_time`` --
+  ``_upcoming_date_and_time_text``. All three are on
+  ``SportsCoreSharedMixin`` (``src/common/sports_shared.py``).
+- ``_element_color`` -- ``_odds_color``, looked up with ``getattr`` and falling
+  back to green, so a host without it still works. Also on
+  ``SportsCoreSharedMixin``.
+- ``SKIN_MODE`` -- optional, read with ``getattr`` by ``_mode_customization``;
+  without it the mode overrides are simply not merged.
+
+State the mixin keeps on the host, created on first use (all read with
+``getattr``, so nothing needs initialising): ``_last_display_call_monotonic``,
+``_switch_order``, ``_switch_order_key``, ``_switch_position``.
+
+Class-level defaults: ``_DWELL_REENTRY_GAP_SECONDS = 5.0`` and
+``favorite_rotation_boost = 1``, matching the plugins. A host's own value
+shadows them.
+
+MRO
+---
+No ``__init__`` and no bare ``super()``, so base order does not change what
+runs. By convention list it after ``SportsCoreSharedMixin``::
+
+    class SportsCore(SportsCoreSharedMixin, SportsHelpersMixin, ABC):
+
+HARDWARE-FREE
+-------------
+Nothing here may import ``src.display_manager``, ``src.base_classes``,
+``src.plugin_system`` or anything else that reaches ``rgbmatrix`` at module
+level; ``test/test_common_is_hardware_free.py`` enforces that for all of
+``src/common``. ``logo_needs_refresh`` imports ``src.logo_downloader`` lazily,
+exactly as the plugin copy does.
+"""
+
+from __future__ import annotations
+
+import time
+from typing import Any, ClassVar, Dict, List, Optional, Tuple
+
+__all__ = [
+    "MIN_WINDOW_DAYS",
+    "MAX_WINDOW_DAYS",
+    "clamp_window",
+    "clamp_seconds",
+    "logo_needs_refresh",
+    "spread_weighted_order",
+    "SportsHelpersMixin",
+]
+
+MIN_WINDOW_DAYS = 1
+MAX_WINDOW_DAYS = 60
+
+
+def clamp_window(value: Any, fallback: int) -> int:
+    """Days for one side of the schedule window, or the default if unusable."""
+    try:
+        days = int(value)
+    except (TypeError, ValueError, OverflowError):
+        # OverflowError: json parses a bare Infinity and int(inf) raises,
+        # which crashed manager init from a hand-edited config.
+        return fallback
+    return max(MIN_WINDOW_DAYS, min(MAX_WINDOW_DAYS, days))
+
+
+def clamp_seconds(value: Any, fallback: int, low: int = 5,
+                  high: int = 86400) -> int:
+    """An interval in seconds, or the fallback when the value is unusable."""
+    try:
+        seconds = int(value)
+    except (TypeError, ValueError, OverflowError):
+        # OverflowError: json parses bare Infinity by default and int(inf)
+        # raises -- the same gap clamp_window above already covers.
+        return fallback
+    return max(low, min(high, seconds))
+
+
+def logo_needs_refresh(logo_file) -> bool:
+    """True if this file is a placeholder stale enough to retry the real logo.
+
+    A failed logo download is cached as a placeholder wearing the real logo's
+    filename, so "the file exists" is not proof the logo was ever fetched.
+    Without this check one transient failure leaves a team a grey box forever.
+
+    Returns False on a core that predates placeholder marking, which keeps the
+    previous behaviour rather than breaking the load.
+    """
+    # Imported from the core by its full path, never as a bare name: a
+    # deferred bare-name import can bind another plugin's vendored
+    # logo_downloader once the core isolates top-level plugin modules.
+    try:
+        from src.logo_downloader import (
+            PLACEHOLDER_RETRY_SECONDS,
+            is_placeholder_logo,
+            placeholder_age_seconds,
+        )
+    except ImportError:
+        return False
+
+    try:
+        if not is_placeholder_logo(logo_file):
+            return False
+        age = placeholder_age_seconds(logo_file)
+        return age is None or age >= PLACEHOLDER_RETRY_SECONDS
+    except Exception:
+        return False
+
+
+def spread_weighted_order(weights: List[int]) -> List[int]:
+    """Indices into ``weights``, each repeated by its weight and spread out.
+
+    Each index keeps its own slot and places its extra turns at even
+    fractions of the rotation after it, wrapping round. That keeps the
+    list's schedule order for everything else and spaces a favourite's
+    repeats evenly *around the loop* -- the live rotation's smooth
+    weighted round-robin schedules a boosted game first and last, so a
+    rotation that wraps shows it back to back. Equal weights come back in
+    plain order, so a boost that applies to no card changes nothing.
+
+    Repeats are kept apart only where the ratio leaves room: once one
+    weight exceeds all the others combined, no cyclic order can separate
+    its turns ([3, 1, 1] gives [0, 1, 0, 2, 0]). Each index still gets
+    exactly its weight in turns -- the configured ratio wins over spacing.
+    """
+    count = len(weights)
+    slots = []
+    for index, weight in enumerate(weights):
+        for turn in range(weight):
+            slots.append(((index + turn * count / weight) % count, turn > 0, index))
+    return [index for _, _, index in sorted(slots)]
+
+
+class SportsHelpersMixin:
+    """The ``SportsCore`` methods identical across the scoreboards' sports.py.
+
+    Constructor-free; keeps lazy state on its host (see the module docstring
+    for that state, the host contract and base-order guidance).
+    """
+
+    #: Longest gap between two display() calls that still counts as one
+    #: on-screen stint. Frames arrive many times a second while a mode is on
+    #: the panel; between mode blocks the gap is the length of every other
+    #: mode's block -- a minute or more. Anything past a few seconds can only
+    #: be a block boundary, or the very first frame after startup.
+    _DWELL_REENTRY_GAP_SECONDS: ClassVar[float] = 5.0
+
+    #: Turns a favourite's card gets per turn of any other card in switch
+    #: mode. The plugins set it per instance from config; this default keeps
+    #: a host that does not on the plain rotation.
+    favorite_rotation_boost: int = 1
+
+    def _favorite_key(self, game: Dict, side: str) -> Optional[str]:
+        """Override point: which view-model field identifies a team when
+        matching against ``favorite_teams``.
+
+        ``side`` is ``"home"`` or ``"away"``. The default is the team
+        abbreviation -- what eight of the nine scoreboards match on, and what
+        users type into their favorites list.
+
+        NRL needs the team **id** instead, because NRL abbreviations are not
+        unique: "NEW" is both Newcastle Knights and New Zealand Warriors,
+        "CAN" both Canberra Raiders and Canterbury Bulldogs. It is a seam
+        rather than a branch so core never has to learn the string "nrl"::
+
+            def _favorite_key(self, game, side):
+                return str(game.get(f"{side}_id"))
+
+        An override that stringifies should note that a missing id becomes the
+        literal ``"None"``, which would spuriously match a favorites list
+        containing that string. The default returns ``None`` for a missing
+        abbreviation, which never matches.
+
+        Carried from ``src/base_classes/sports/core.py`` for later phases;
+        nothing in this module calls it yet.
+        """
+        return game.get(f"{side}_abbr")
+
+    def _mode_customization(self) -> dict:
+        """``customization`` with this mode's overrides merged over it.
+
+        SportsUpcoming / SportsRecent / SportsLive are separate instances
+        with their own SKIN_MODE, so merging once here makes every
+        per-element lookup mode-aware without changing one of them.
+
+        ``None`` in a mode block means "inherit", which is what lets a user
+        restyle one element on live cards and leave everything else
+        following the settings above. It has to stay distinct from 0: a mode
+        y_offset of 0 means "sit at the base position", not "no preference".
+        """
+        customization = self.config.get('customization', {})
+        if not isinstance(customization, dict):
+            return {}
+        mode = getattr(self, 'SKIN_MODE', None)
+        if not mode:
+            return customization
+        modes = customization.get('modes')
+        block = modes.get(mode) if isinstance(modes, dict) else None
+        if not isinstance(block, dict):
+            return customization
+
+        merged = dict(customization)
+        for element, override in block.items():
+            if element == 'layout' or not isinstance(override, dict):
+                continue
+            base = merged.get(element)
+            base = dict(base) if isinstance(base, dict) else {}
+            base.update({k: v for k, v in override.items() if v is not None})
+            merged[element] = base
+
+        mode_layout = block.get('layout')
+        if isinstance(mode_layout, dict):
+            base_layout = merged.get('layout')
+            new_layout = dict(base_layout) if isinstance(base_layout, dict) else {}
+            for element, axes in mode_layout.items():
+                if not isinstance(axes, dict):
+                    continue
+                current = new_layout.get(element)
+                current = dict(current) if isinstance(current, dict) else {}
+                current.update({k: v for k, v in axes.items() if v is not None})
+                new_layout[element] = current
+            merged['layout'] = new_layout
+        return merged
+
+    def _setting_int(self, key: str, default: int, low: int, high: int) -> int:
+        """A count from config, clamped to the range its schema declares.
+
+        The schema constrains these, but config.json can be hand-edited or
+        written by an older tool, and a string or a negative here does not
+        raise where anyone would see it -- it raises inside update()'s own
+        try/except, which shows up as a mode that silently renders nothing.
+        Same shape as the favorite_live_boost clamp in the plugins.
+        """
+        try:
+            return max(low, min(high, int(self.mode_config.get(key, default))))
+        except (TypeError, ValueError, OverflowError):
+            # OverflowError: json parses a bare Infinity, and int(inf) raises
+            # -- from __init__, outside any try/except, so the manager would
+            # fail to construct instead of falling back.
+            self.logger.warning(
+                "%s: ignoring unusable %s=%r, using %s",
+                getattr(self, "league", "?"), key,
+                self.mode_config.get(key), default,
+            )
+            return default
+
+    def _reset_dwell_on_reentry(self) -> bool:
+        """Give the current card a full turn when this mode (re)takes the panel.
+
+        The dwell clock (last_game_switch) keeps running while the mode is off
+        screen, so on re-entry it was always long expired and the first
+        display() call advanced immediately: the card cut off by the end of
+        the previous block was skipped instead of shown -- measured at one in
+        five card transitions on a 30s block of 15s cards -- and after a
+        service restart the clock started at manager construction, seconds
+        before the first frame, shaving that much off the first card. Both are
+        the same defect: the dwell clock counting time the viewer never saw.
+
+        Returns True when the dwell was reset, so the caller forces a redraw.
+        The one-frame card at the end of a block (the advance that races the
+        controller's mode switch) still renders -- this reset is what turns it
+        into the card that opens the next block with a full turn, instead of
+        one the rotation skipped.
+        """
+        # getattr, and zero treated as "never displayed": the managers are
+        # constructed in several places -- the plugin tests among them -- not
+        # all of which set every attribute, and a freshly booted Pi can reach
+        # the first frame while time.monotonic() itself is still under the
+        # gap threshold, which would make `now - 0.0` look like one stint.
+        last = getattr(self, "_last_display_call_monotonic", 0.0)
+        now = time.monotonic()
+        self._last_display_call_monotonic = now
+        if last > 0.0 and now - last < self._DWELL_REENTRY_GAP_SECONDS:
+            return False
+        if getattr(self, "last_game_switch", 0) <= 0:
+            # Zero is the live screen's "no game shown yet" sentinel with its
+            # own handling; overwriting it here would hide the first game's
+            # arrival from that logic.
+            return False
+        self.last_game_switch = time.time()
+        return True
+
+    _spread_weighted_order = staticmethod(spread_weighted_order)
+
+    def _next_switch_index(self) -> int:
+        """The games_list index switch mode shows next.
+
+        favorite_rotation_boost gives a favourite's card that many turns for
+        every one turn another card gets, spread through the rotation and kept
+        apart wherever the other cards leave room (a boost above the number of
+        other cards makes some repeats adjacent; the ratio is kept either way).
+        games_list itself stays one entry per game -- the
+        cycle-duration count, the scroll strip and the other-games re-cut all
+        read it -- so the weighting is an order walked over it instead.
+
+        The order is rebuilt whenever the list's games change, and the walk
+        resyncs from current_game_index whenever the two disagree: update()
+        and the other-games rotation both set the index directly when they
+        swap a list in, and the card on screen is where the walk resumes.
+
+        Called with _games_lock held and games_list non-empty.
+        """
+        count = len(self.games_list)
+        boost = getattr(self, "favorite_rotation_boost", 1)
+        if boost <= 1 or count < 2:
+            return (self.current_game_index + 1) % count
+        key = (boost, tuple(g.get("id") for g in self.games_list))
+        if getattr(self, "_switch_order_key", None) != key:
+            self._switch_order = self._spread_weighted_order(
+                [boost if self._is_favorite_game(g) else 1 for g in self.games_list]
+            )
+            self._switch_order_key = key
+            self._switch_position = -1
+        order = self._switch_order
+        position = getattr(self, "_switch_position", -1)
+        if not 0 <= position < len(order) or order[position] != self.current_game_index:
+            position = (order.index(self.current_game_index)
+                        if self.current_game_index in order else -1)
+        position = (position + 1) % len(order)
+        self._switch_position = position
+        return order[position]
+
+    def _odds_color(self) -> Tuple[int, int, int]:
+        """Colour for the odds text; the green it always drew unless configured.
+
+        Guarded with getattr because not every class that reaches
+        _draw_dynamic_odds carries the element-colour helper -- the plugins'
+        own test harnesses build minimal manager objects, and a bare
+        AttributeError here is swallowed by the surrounding except, which
+        drops the odds off the card instead of failing loudly.
+        """
+        getter = getattr(self, "_element_color", None)
+        if getter is None:
+            return (0, 255, 0)
+        try:
+            return getter("odds_text", (0, 255, 0))  # pylint: disable=not-callable
+        except Exception:
+            return (0, 255, 0)
+
+    def _upcoming_date_and_time_text(self, game_date: str, game_time: str,
+                                     game: Optional[Dict] = None) -> Tuple[str, str]:
+        """The formatted (date, time) pair, blanked by switch_show_date/_time.
+
+        Deliberately not the shared show_date/show_time: those governed only
+        the scroll and Vegas cards before this display read the block, so a
+        config that had turned them off there would silently blank a scorebug
+        that has always drawn both lines. The switch keys default to True for
+        the same reason switch_upcoming_center defaults to "date_time" -- an
+        untouched panel keeps rendering exactly what it rendered before.
+        """
+        date_text = (self._format_game_date(game_date, game)
+                     if self._card_option("switch_show_date", True) else "")
+        time_text = (self._format_game_time(game_time)
+                     if self._card_option("switch_show_time", True) else "")
+        return date_text, time_text

src/core_config_keys.py

@@ -0,0 +1,43 @@
+"""Top-level config.json keys that belong to the LEDMatrix core, not to plugins.
+
+config.json mixes two kinds of top-level section: core settings (``display``,
+``schedule``, ``auto_update``, ...) and one section per plugin, keyed by plugin
+id. Anything that needs to tell them apart -- plugin-state reconciliation above
+all -- must use this list, not a private copy of it.
+
+A private copy is what went wrong: #581 added a top-level ``auto_update``
+section, reconciliation's own list did not know about it, and every device with
+the new setting was told "In config but not installed: auto_update. Reinstall
+via the Plugin Store, or remove these entries from config.json" -- advice that
+deletes a real core setting.
+
+When you add a top-level core setting, add its key here. The tests in
+``test/test_core_config_keys.py`` fail if a top-level key in
+``config/config.template.json``, or one written by the general-settings save
+endpoint, is missing from this list.
+"""
+
+CORE_CONFIG_KEYS = frozenset({
+    # config/config.template.json
+    'web_display_autostart',
+    'auto_update',
+    'schedule',
+    'dim_schedule',
+    'timezone',
+    'target_fps',
+    'location',
+    'display',
+    'sync',
+    'plugin_system',
+    # Older or optional core sections still found in existing config files.
+    'logging',
+    'network',
+    'system',
+    'vegas_scroll_speed',
+    'vegas_separator_width',
+    'vegas_target_fps',
+    'vegas_buffer_ahead',
+    'vegas_plugin_order',
+    'vegas_excluded_plugins',
+    'vegas_scroll_enabled',
+})

src/plugin_system/plugin_manager.py

@@ -22,7 +22,7 @@ from src.logging_config import get_logger
 from src.plugin_system.plugin_loader import PluginLoader
 from src.plugin_system.plugin_executor import PluginExecutor
 from src.plugin_system.plugin_state import PluginStateManager, PluginState
-from src.plugin_system.schema_manager import SchemaManager
+from src.plugin_system.schema_manager import SchemaManager, normalize_legacy_booleans
 from src.common.permission_utils import (
     ensure_directory_permissions,
     get_plugin_dir_mode
@@ -351,6 +351,7 @@ class PluginManager:
                 config = {}
             
             # Check if plugin has a config schema
+            schema = None
             schema_path = self.schema_manager.get_schema_path(plugin_id)
             if schema_path is None:
                 # Schema file doesn't exist
@@ -368,6 +369,24 @@ class PluginManager:
                         f"The schema may be invalid. Please verify the schema file at: {schema_path}"
                     )
 
+            # A plugin that turned an on/off boolean into an {enabled, ...}
+            # object still finds the boolean in config.json until the user saves
+            # its settings form, which carries it over (plugin_config.html).
+            # Read it the same way here, before the defaults fill in the rest
+            # of the object and before schema validation, so the plugin doesn't
+            # start with a schema warning and a degraded flag. In memory only:
+            # config.json is written by saves, never by loading a plugin.
+            if schema:
+                upgraded: List[str] = []
+                config = normalize_legacy_booleans(config, schema, upgraded)
+                if upgraded:
+                    self.logger.info(
+                        "Plugin %s: reading legacy boolean setting %s as "
+                        "{\"enabled\": ...}; saving the plugin's settings "
+                        "stores the new shape",
+                        plugin_id, ", ".join(upgraded),
+                    )
+
             # Merge config with schema defaults to ensure all defaults are applied
             try:
                 defaults = self.schema_manager.generate_default_config(plugin_id, use_cache=True)

src/plugin_system/schema_manager.py

@@ -14,6 +14,85 @@ import jsonschema
 from jsonschema import Draft7Validator, ValidationError
 
 
+def _renders_as_object(prop: Dict[str, Any]) -> bool:
+    """``field_type == 'object'`` as ``plugin_config.html`` computes it.
+
+    The template takes a type list's *first* entry, so ``["object", "null"]``
+    is an object and ``["boolean", "object"]`` is a checkbox.
+    """
+    field_type = prop.get('type')
+    if isinstance(field_type, list):
+        field_type = field_type[0] if field_type else None
+    return field_type == 'object'
+
+
+def legacy_bool_as_object(value: Any, prop: Any) -> Any:
+    """Read a boolean stored where the schema now has an ``{enabled, ...}`` object.
+
+    Plugins turn an on/off switch into a settings object (news'
+    ``global.dynamic_duration: true`` became ``{enabled, min_duration_seconds,
+    ...}``), but config.json keeps the boolean until the user next saves that
+    plugin's form. The boolean was the switch the object's ``enabled`` now
+    holds, so it becomes ``{"enabled": value}`` and schema defaults fill the rest.
+
+    Returns ``value`` itself when the rule does not apply: anything that is not
+    a real ``bool`` (``1``, ``"true"``, ``None``) or a schema property that is not
+    an object with an ``enabled`` child. Those stay as they are, so validation
+    still reports a genuine mismatch.
+
+    This is the rule ``render_nested_section`` in
+    ``web_interface/templates/v3/partials/plugin_config.html`` applies when it
+    draws the form. ``test/test_legacy_boolean_config.py`` renders that macro
+    against this function, so change both together.
+    """
+    if not isinstance(value, bool) or not isinstance(prop, dict):
+        return value
+    properties = prop.get('properties')
+    if (_renders_as_object(prop) and isinstance(properties, dict)
+            and 'enabled' in properties):
+        return {'enabled': value}
+    return value
+
+
+def normalize_legacy_booleans(config: Any, schema: Any,
+                              changed_paths: Optional[List[str]] = None,
+                              _prefix: str = '') -> Any:
+    """Apply :func:`legacy_bool_as_object` at every depth of a plugin config.
+
+    Walks the config along the schema's ``properties`` the way the settings
+    form does: into nested objects, not into array items (the form hands arrays
+    to widgets and never applies the rule there).
+
+    Never mutates ``config``. Returns the same object when nothing changed, and
+    otherwise copies only the dicts on the path to each upgraded value. When
+    ``changed_paths`` is given, the dotted path of each upgraded value is
+    appended to it.
+    """
+    if not isinstance(config, dict) or not isinstance(schema, dict):
+        return config
+    properties = schema.get('properties')
+    if not isinstance(properties, dict):
+        return config
+
+    result = config
+    for key, value in config.items():
+        prop = properties.get(key)
+        if not isinstance(prop, dict) or not _renders_as_object(prop):
+            continue
+        path = f"{_prefix}.{key}" if _prefix else key
+        new_value = legacy_bool_as_object(value, prop)
+        if new_value is not value:
+            if changed_paths is not None:
+                changed_paths.append(path)
+        elif isinstance(value, dict):
+            new_value = normalize_legacy_booleans(value, prop, changed_paths, path)
+        if new_value is not value:
+            if result is config:
+                result = dict(config)
+            result[key] = new_value
+    return result
+
+
 class SchemaManager:
     """
     Manages plugin configuration schemas with caching and validation.

src/plugin_system/state_reconciliation.py

@@ -14,6 +14,7 @@ from dataclasses import dataclass
 from enum import Enum
 from pathlib import Path
 
+from src.core_config_keys import CORE_CONFIG_KEYS
 from src.plugin_system.state_manager import PluginStateManager
 from src.logging_config import get_logger
 
@@ -72,9 +73,16 @@ def secrets_top_level_keys(config_manager) -> Set[str]:
     return set(secrets) if isinstance(secrets, dict) else set()
 
 
-def ignored_config_keys(config_manager) -> Set[str]:
-    """Config keys that are not plugin ids: system keys plus secrets keys."""
-    return set(StateReconciliation._SYSTEM_CONFIG_KEYS) | secrets_top_level_keys(config_manager)
+def ignored_config_keys(config_manager, installed_ids=frozenset()) -> Set[str]:
+    """Config keys that are not plugin ids: core keys plus secrets keys.
+
+    A secrets key that is also an installed plugin's id is not ignored: plugin
+    secrets are namespaced by plugin id, so that key is the plugin's own entry.
+    Ignoring it made every installed plugin with secrets read as "installed but
+    missing from config" on each run.
+    """
+    secrets_only = secrets_top_level_keys(config_manager) - set(installed_ids)
+    return set(StateReconciliation._SYSTEM_CONFIG_KEYS) | secrets_only
 
 
 def config_plugin_ids(config: Dict[str, Any], ignored_keys: Set[str]) -> Set[str]:
@@ -132,6 +140,13 @@ def still_unresolved(entries: List[Dict[str, Any]],
     plugins were all present in config kept being told, for hours, that four of
     them were missing and should be removed from config.json.
 
+    An "in config but not on disk" finding is stale once the plugin is
+    installed, and equally once the id is no longer a plugin entry in config --
+    whether the user removed it or it was never a plugin (a core key such as
+    ``auto_update`` that an older build misread). Without the second check a
+    verdict written by such a build kept telling users to delete a real core
+    setting until the next full reconciliation run.
+
     Entry kinds this cannot re-check are kept, so filtering only ever removes
     findings that are provably stale.
     """
@@ -143,7 +158,7 @@ def still_unresolved(entries: List[Dict[str, Any]],
             if plugin_id not in config_keys:
                 live.append(entry)
         elif kind == InconsistencyType.PLUGIN_MISSING_ON_DISK.value:
-            if plugin_id not in installed_ids:
+            if plugin_id in config_keys and plugin_id not in installed_ids:
                 live.append(entry)
         else:
             live.append(entry)
@@ -280,39 +295,22 @@ class StateReconciliation:
                 message=f"Reconciliation failed: {str(e)}"
             )
     
-    # Top-level config keys that are NOT plugins.
-    # Includes both config.json structural keys and config_secrets.json top-level
-    # keys (load_config() deep-merges secrets in, so secrets keys appear here too).
-    _SYSTEM_CONFIG_KEYS = frozenset({
-        'web_display_autostart', 'timezone', 'location', 'display',
-        'plugin_system', 'vegas_scroll_speed', 'vegas_separator_width',
-        'vegas_target_fps', 'vegas_buffer_ahead', 'vegas_plugin_order',
-        'vegas_excluded_plugins', 'vegas_scroll_enabled', 'logging',
-        'dim_schedule', 'network', 'system', 'schedule',
-        # Multi-display sync config (config.json structural key)
-        'sync',
-        # Secrets file top-level keys (merged in by load_config)
-        'github', 'youtube',
-    })
-
-    def _secrets_top_level_keys(self) -> Set[str]:
-        """Top-level keys that load_config() merges in from the secrets file.
-
-        load_config() merges config_secrets.json into the config it returns, so
-        those keys sit alongside plugin ids. _SYSTEM_CONFIG_KEYS named them
-        individually ('github', 'youtube'), which broke the moment anything else
-        was written there: a 'data' key became a phantom plugin, permanently
-        reported as "in config but not on disk". Reading the file keeps this
-        correct no matter what it holds.
-        """
-        return secrets_top_level_keys(self.config_manager)
+    # Top-level config keys that are NOT plugins. The core keys come from the
+    # shared list in src/core_config_keys.py -- a private copy here missed
+    # #581's 'auto_update' and reported it as a plugin missing from disk.
+    # 'github'/'youtube' are the historical secrets-file keys. The secrets file
+    # itself is read at run time too (ignored_config_keys): load_config() merges
+    # it in, and naming its keys one by one let a 'data' key become a phantom
+    # plugin permanently reported as "in config but not on disk".
+    _SYSTEM_CONFIG_KEYS = CORE_CONFIG_KEYS | frozenset({'github', 'youtube'})
 
     def _get_config_state(self) -> Dict[str, Dict[str, Any]]:
         """Get plugin state from config file."""
         state = {}
         try:
             config = self.config_manager.load_config()
-            ignored = self._SYSTEM_CONFIG_KEYS | self._secrets_top_level_keys()
+            ignored = ignored_config_keys(self.config_manager,
+                                          disk_plugin_ids(self.plugins_dir))
             for plugin_id in config_plugin_ids(config, ignored):
                 plugin_config = config[plugin_id]
                 state[plugin_id] = {
@@ -389,7 +387,20 @@ class StateReconciliation:
     ) -> List[Inconsistency]:
         """Check consistency for a single plugin."""
         inconsistencies = []
-        
+
+        if plugin_id in CORE_CONFIG_KEYS:
+            # A plugin whose id is a core setting's key ('display', 'sync',
+            # ...) can never have a config section of its own: that section
+            # is the core setting. Every check below would misfire -- "not in
+            # config" forever, and an auto-fix that writes {'enabled': False}
+            # where a core setting belongs -- so report it in the log instead.
+            if disk_state.get(plugin_id, {}).get('exists_on_disk'):
+                self.logger.warning(
+                    "Plugin id %r is reserved for a core config setting; "
+                    "the plugin cannot be configured and is skipped by "
+                    "reconciliation. Rename the plugin.", plugin_id)
+            return inconsistencies
+
         config = config_state.get(plugin_id, {})
         disk = disk_state.get(plugin_id, {})
         state_mgr = state_manager_state.get(plugin_id, {})
@@ -468,6 +479,12 @@ class StateReconciliation:
         """Attempt to fix an inconsistency."""
         try:
             if inconsistency.inconsistency_type == InconsistencyType.PLUGIN_MISSING_IN_CONFIG:
+                if inconsistency.plugin_id in CORE_CONFIG_KEYS:
+                    # Never create or replace a core setting with a plugin stub.
+                    self.logger.warning(
+                        "Refusing to add plugin entry %r: that key is a core "
+                        "config setting", inconsistency.plugin_id)
+                    return False
                 config = self.config_manager.load_config()
                 if inconsistency.plugin_id in config:
                     # Detection said "not in config" but it is there -- the

test/js/README.md

@@ -35,6 +35,7 @@ nothing is listening, so it stays useful in a bare checkout.
 | Suite | Needs a server | Covers |
 |---|---|---|
 | `unit/test_list_filter.js` | no | `ListFilter` search/filter/sort/count/sticky, and the installed-plugins config **extracted verbatim** from `plugins_manager.js` so the test can't drift from it |
+| `unit/test_update_all.js` | no | `PluginInstallManager.updateAll` from `plugins/install_manager.js`: Check & Update All sends only plugin ids (never `starlark:` app entries), and re-sends a request that got no HTTP answer (web service restarting) instead of skipping that plugin. Also run by `test/web_interface/test_update_all_plugins.py` so CI covers it |
 | `unit/test_render_cards.js` | no | `renderInstalledCards` markup, both empty states, and HTML-escaping of hostile plugin metadata |
 | `unit/test_style_editor_element_keys.js` | no | `elementKeys()`/`styleRows()`/`positionRows()` from `widgets/style-editor.js`: every `customization.layout` entry gets exactly one row -- paired with its style element through core's `x-layout-key` (so `score` belongs to `score_text`, not a second row), or a position row of its own, leaves included -- since the widget claims the whole `layout` block from the generic fallback renderer |
 | `unit/test_style_editor_layout_leaf_columns.js` | no | `columnsFor()` from `widgets/style-editor.js`: a layout-only key whose own value is a leaf (no x/y sub-object, e.g. a `show_logo` toggle) gets a self-keyed column instead of a blank, uneditable row |

test/js/run_all.js

@@ -17,7 +17,8 @@ const BASE = process.env.BASE || 'http://localhost:5000';
 const UNIT = ['unit/test_list_filter.js', 'unit/test_render_cards.js',
               'unit/test_html_escaping.js', 'unit/test_style_editor_element_keys.js',
               'unit/test_style_editor_layout_leaf_columns.js',
-              'unit/test_style_editor_layout_leaf_collision.js'];
+              'unit/test_style_editor_layout_leaf_collision.js',
+              'unit/test_update_all.js'];
 const DOM = ['dom/test_installed_dom.js', 'dom/test_store_dom.js', 'dom/test_no_double_fetch.js',
              'dom/test_tools_sections.js'];
 

test/js/unit/test_update_all.js

@@ -0,0 +1,151 @@
+// "Check & Update All" -- which ids it sends to POST /api/v3/plugins/update.
+//
+// Pins two bugs seen on a real device (core 3.4.0):
+//
+//  1. /plugins/installed lists installed Starlark apps as virtual
+//     `starlark:<app_id>` entries. Update-all sent those to the plugin
+//     updater, which answered 500 "plugin not found" for each one.
+//
+//  2. A web-service restart landed mid-run. The request in flight died with
+//     the old process and the next one (stock-news) was refused while the
+//     service was coming back; both were recorded as failures and never sent
+//     again, so stock-news -- installed, disabled, with an update waiting --
+//     was silently not updated.
+//
+// Runs the shipped install_manager.js (it exports itself under node) against
+// a fake PluginAPI. The installed list is the device's, trimmed.
+
+const path = require('path');
+const V3 = path.resolve(__dirname, '../../../web_interface/static/v3');
+
+let pass = 0, fail = 0;
+const ok = (label, cond, extra) => cond
+  ? (pass++, console.log('  ok   ' + label))
+  : (fail++, console.log('  FAIL ' + label + (extra !== undefined ? '  ' + JSON.stringify(extra) : '')));
+
+global.window = {};
+const Manager = require(path.join(V3, 'js/plugins/install_manager.js'));
+
+const INSTALLED = [
+  { id: 'ledmatrix-flights', enabled: true, version: '1.14.0' },
+  { id: 'stock-news', enabled: false, version: '2.6.2', latest_version: '2.8.0', update_available: true },
+  { id: 'static-image', enabled: false, version: '1.1.3' },
+  { id: 'starlark-apps', enabled: false, version: '1.0.0' },   // a real plugin -- keep it
+  { id: 'pomodoro-timer', enabled: false, version: '1.3.6' },
+  { id: 'starlark:analogtime', enabled: false, version: 'starlark', is_starlark_app: true },
+  { id: 'starlark:analogclock', enabled: true, version: 'starlark', is_starlark_app: true },
+];
+const EXPECTED = ['ledmatrix-flights', 'stock-news', 'static-image', 'starlark-apps', 'pomodoro-timer'];
+
+const netErr = () => ({ error_code: 'NETWORK_ERROR', message: 'Failed to fetch' });
+
+function fakeApi(behaviour = {}) {
+  const calls = [];
+  return {
+    calls,
+    updatePlugin: async (id) => {
+      calls.push(id);
+      const b = behaviour[id];
+      if (typeof b === 'function') return b(calls.filter(c => c === id).length);
+      return { status: 'success', message: `Plugin ${id} updated successfully` };
+    },
+  };
+}
+
+function setup(api, { stateList, windowList } = {}) {
+  global.window = {
+    PluginAPI: api,
+    installedPlugins: windowList,
+    PluginStateManager: stateList === undefined ? undefined : {
+      installedPlugins: stateList,
+      loadInstalledPlugins: async () => stateList,
+    },
+  };
+}
+
+const noSleep = { sleep: async () => {} };
+
+(async () => {
+  console.log('\nselection');
+  ok('starlark app entries are not updatable',
+     !Manager.isUpdatablePlugin(INSTALLED[5]) && !Manager.isUpdatablePlugin(INSTALLED[6]));
+  ok('a starlark: id without the flag is still excluded',
+     !Manager.isUpdatablePlugin({ id: 'starlark:foo' }));
+  ok('the starlark-apps plugin itself is updatable', Manager.isUpdatablePlugin(INSTALLED[3]));
+  ok('a disabled plugin with an update is updatable', Manager.isUpdatablePlugin(INSTALLED[1]));
+  ok('entries without a usable id are skipped',
+     !Manager.isUpdatablePlugin({}) && !Manager.isUpdatablePlugin(null) && !Manager.isUpdatablePlugin({ id: '' }));
+  const sel = Manager.updatablePlugins(INSTALLED).map(p => p.id);
+  ok('updatablePlugins keeps list order and drops only starlark apps',
+     JSON.stringify(sel) === JSON.stringify(EXPECTED), sel);
+
+  console.log('\nupdateAll sends only plugin ids');
+  {
+    const api = fakeApi();
+    setup(api, { windowList: INSTALLED });
+    const progress = [];
+    const results = await Manager.updateAll((i, n, id) => progress.push([i, n, id]), noSleep);
+    ok('POSTs exactly the non-starlark ids, in order',
+       JSON.stringify(api.calls) === JSON.stringify(EXPECTED), api.calls);
+    ok('no starlark: id reached the plugin updater', !api.calls.some(id => id.startsWith('starlark:')));
+    ok('one result per plugin sent', results.length === EXPECTED.length, results.length);
+    ok('progress total counts only what is sent',
+       progress.length === EXPECTED.length && progress.every(([, n]) => n === EXPECTED.length), progress);
+  }
+  {
+    const api = fakeApi();
+    setup(api, { stateList: INSTALLED, windowList: [] });
+    await Manager.updateAll(null, noSleep);
+    ok('the PluginStateManager list is filtered the same way',
+       JSON.stringify(api.calls) === JSON.stringify(EXPECTED), api.calls);
+  }
+  {
+    const api = fakeApi();
+    setup(api, { windowList: INSTALLED.filter(p => p.is_starlark_app) });
+    const results = await Manager.updateAll(null, noSleep);
+    ok('a list of only starlark apps sends nothing', api.calls.length === 0 && results.length === 0);
+  }
+
+  console.log('\nweb service restarting mid-run (the stock-news case)');
+  {
+    // ledmatrix-flights is in flight when the service stops (reset); stock-news
+    // is refused twice while it comes back; then everything answers.
+    const api = fakeApi({
+      'ledmatrix-flights': (n) => { if (n === 1) throw netErr(); return { status: 'success', message: 'ok' }; },
+      'stock-news': (n) => { if (n <= 2) throw netErr(); return { status: 'success', message: 'Plugin stock-news updated successfully' }; },
+    });
+    setup(api, { windowList: INSTALLED });
+    const slept = [];
+    const results = await Manager.updateAll(null, { sleep: async ms => { slept.push(ms); }, retryDelaysMs: [5, 10, 20] });
+    const byId = Object.fromEntries(results.map(r => [r.pluginId, r]));
+    ok('stock-news is sent again until the server answers',
+       api.calls.filter(id => id === 'stock-news').length === 3, api.calls);
+    ok('stock-news ends up updated, not skipped', byId['stock-news'] && byId['stock-news'].success === true, byId['stock-news']);
+    ok('the request lost with the old process is re-sent too',
+       byId['ledmatrix-flights'] && byId['ledmatrix-flights'].success === true);
+    ok('it backs off between attempts', JSON.stringify(slept) === JSON.stringify([5, 5, 10]), slept);
+    ok('every plugin still gets exactly one result',
+       JSON.stringify(results.map(r => r.pluginId)) === JSON.stringify(EXPECTED), results.map(r => r.pluginId));
+  }
+  {
+    const api = fakeApi({ 'stock-news': () => { throw netErr(); } });
+    setup(api, { windowList: INSTALLED });
+    const results = await Manager.updateAll(null, { sleep: async () => {}, retryDelaysMs: [1, 1] });
+    const r = results.find(x => x.pluginId === 'stock-news');
+    ok('a server that never comes back is retried a bounded number of times',
+       api.calls.filter(id => id === 'stock-news').length === 3, api.calls);
+    ok('...then reported as a failure, and the run continues',
+       r && r.success === false && api.calls[api.calls.length - 1] === 'pomodoro-timer');
+  }
+  {
+    const api = fakeApi({ 'static-image': () => { throw { error_code: 'PLUGIN_UPDATE_FAILED', message: 'check logs' }; } });
+    setup(api, { windowList: INSTALLED });
+    const results = await Manager.updateAll(null, { sleep: async () => { throw new Error('must not sleep'); } });
+    ok('an HTTP error answer is not retried',
+       api.calls.filter(id => id === 'static-image').length === 1, api.calls);
+    ok('...and is reported as a failure', results.find(x => x.pluginId === 'static-image').success === false);
+  }
+
+  console.log(`\n${pass} passed, ${fail} failed`);
+  process.exit(fail ? 1 : 0);
+})().catch(e => { console.error(e); process.exit(1); });

test/test_common_is_hardware_free.py

@@ -0,0 +1,163 @@
+"""src/common must stay importable without display hardware.
+
+Plugins import src.common.sports_* in place of their bundled copies. If any of
+those modules reaches src.display_manager (and through it rgbmatrix) or the
+src.base_classes package (whose core.py imports DisplayManager), a scoreboard
+adopting it acquires a hardware dependency it never had, and the headless
+tooling -- the web preview, check_plugin.py, these tests on a laptop -- stops
+being able to load it.
+
+Two checks, because each misses what the other catches:
+
+1. A real import in a fresh interpreter with rgbmatrix made unimportable.
+   Catches indirect imports (common -> X -> display_manager) that no text scan
+   of src/common would see.
+2. An AST scan of src/common/*.py for module-level imports of the hardware and
+   plugin-system packages. Catches a direct import even where the target
+   happens to import cleanly on this machine.
+"""
+
+import ast
+import subprocess
+import sys
+import textwrap
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+COMMON = REPO_ROOT / "src" / "common"
+
+FORBIDDEN = ("src.base_classes", "src.display_manager", "src.plugin_system")
+
+#: Existing module-level violations, by file name, each with the reason it is
+#: tolerated. Empty when this test was added (core 3.4.0 + sports_helpers):
+#: no src/common module imported any of FORBIDDEN at module level. Add to this
+#: only with a comment saying why, never to get a new import past the test.
+ALLOWLIST = {}
+
+
+def test_sports_modules_import_without_hardware():
+    modules = sorted(p.stem for p in COMMON.glob("sports_*.py"))
+    assert "sports_helpers" in modules
+    script = textwrap.dedent(f"""
+        import importlib, json, sys
+        sys.modules["rgbmatrix"] = None   # any import of it now raises
+        importlib.import_module("src.common")
+        for name in {modules!r}:
+            importlib.import_module("src.common." + name)
+        loaded = sorted(m for m in sys.modules
+                        if m.startswith({FORBIDDEN!r}))
+        print(json.dumps(loaded))
+    """)
+    result = subprocess.run(
+        [sys.executable, "-c", script], cwd=str(REPO_ROOT),
+        capture_output=True, text=True, timeout=120)
+    assert result.returncode == 0, (
+        f"importing src.common without rgbmatrix failed:\n{result.stderr}")
+    loaded = result.stdout.strip().splitlines()[-1]
+    assert loaded == "[]", f"src.common pulled in hardware modules: {loaded}"
+
+
+def _module_level_imports(tree):
+    """Imports executed at import time: top level, including inside if/try
+    blocks, but not inside functions or classes (those are deferred)."""
+    stack = list(tree.body)
+    while stack:
+        node = stack.pop()
+        if isinstance(node, (ast.Import, ast.ImportFrom)):
+            yield node
+        elif isinstance(node, (ast.If, ast.Try, ast.With)):
+            stack.extend(node.body)
+            stack.extend(getattr(node, "orelse", []))
+            stack.extend(getattr(node, "finalbody", []))
+            for handler in getattr(node, "handlers", []):
+                stack.extend(handler.body)
+
+
+def _targets(node, package="src.common"):
+    """Absolute dotted names an import can load. Relative imports resolve
+    against ``package``, the scanned module's package."""
+    if isinstance(node, ast.Import):
+        return [alias.name for alias in node.names]
+    base = node.module
+    if node.level:
+        parts = package.split(".")
+        if node.level > len(parts):
+            return []   # beyond the top-level package; Python rejects it too
+        anchor = ".".join(parts[:len(parts) - node.level + 1])
+        base = f"{anchor}.{node.module}" if node.module else anchor
+    if base is None:
+        return []
+    # `from pkg import name` may load the submodule pkg.name.
+    return [base] + [f"{base}.{alias.name}" for alias in node.names
+                     if alias.name != "*"]
+
+
+def _violations():
+    found = {}
+    for path in sorted(COMMON.glob("*.py")):
+        tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
+        for node in _module_level_imports(tree):
+            for target in _targets(node, "src.common"):
+                if target == FORBIDDEN or any(
+                        target == f or target.startswith(f + ".") for f in FORBIDDEN):
+                    found.setdefault(path.name, []).append(f"line {node.lineno}: {target}")
+    return found
+
+
+def test_no_module_level_hardware_or_plugin_system_imports():
+    found = {name: hits for name, hits in _violations().items() if name not in ALLOWLIST}
+    assert found == {}, (
+        "src/common must not import these at module level (defer the import "
+        f"into the function that needs it): {found}")
+
+
+def test_allowlist_has_no_stale_entries():
+    stale = sorted(set(ALLOWLIST) - set(_violations()))
+    assert stale == [], f"allowlisted but no longer violating, remove: {stale}"
+
+
+def test_the_scan_sees_a_direct_import(tmp_path):
+    # Guard the guard: the walker must find imports nested in try/if.
+    tree = ast.parse(textwrap.dedent("""
+        try:
+            from src.display_manager import DisplayManager
+        except ImportError:
+            pass
+        if True:
+            import src.base_classes.sports
+        from src import plugin_system
+        def later():
+            from src.display_manager import DisplayManager
+    """))
+    targets = {t for n in _module_level_imports(tree) for t in _targets(n)}
+    assert {"src.base_classes.sports", "src.display_manager",
+            "src.plugin_system"} <= targets
+
+
+def test_the_scan_resolves_relative_imports():
+    # A module in src/common reaches src.plugin_system through `..`.
+    tree = ast.parse(textwrap.dedent("""
+        from .. import plugin_system
+        from ..plugin_system import plugin_manager
+        from . import sports_shared
+    """))
+    targets = {t for n in _module_level_imports(tree)
+               for t in _targets(n, "src.common")}
+    assert {"src.plugin_system", "src.plugin_system.plugin_manager",
+            "src.common.sports_shared"} <= targets
+    # The package decides where `..` points.
+    node = ast.parse("from .. import plugin_system").body[0]
+    assert _targets(node, "src.common") == ["src", "src.plugin_system"]
+    assert _targets(node, "a.b") == ["a", "a.plugin_system"]
+
+
+def test_the_scan_flags_relative_imports_in_src_common(monkeypatch, tmp_path):
+    # End to end through _violations(): both relative forms are reported.
+    (tmp_path / "bad.py").write_text(
+        "from .. import plugin_system\nfrom ..plugin_system import x\n",
+        encoding="utf-8")
+    monkeypatch.setitem(globals(), "COMMON", tmp_path)
+    found = {name: sorted(hits) for name, hits in _violations().items()}
+    assert found == {"bad.py": ["line 1: src.plugin_system",
+                                "line 2: src.plugin_system",
+                                "line 2: src.plugin_system.x"]}

test/test_core_config_keys.py

@@ -0,0 +1,270 @@
+"""Core settings in config.json must never be reported as orphaned plugins.
+
+Observed on a v3.4.0 device: the web UI showed "Plugin Config Warning -- In
+config but not installed: auto_update. Reinstall via the Plugin Store, or
+remove these entries from config.json." ``auto_update`` is the top-level core
+section #581 added for weekly automatic updates. Reconciliation kept its own
+private list of non-plugin keys, which #581 did not know to extend, so any
+top-level dict it did not list was treated as a plugin id. Following the
+warning's advice deletes a real core setting.
+
+These tests pin the shared list in src/core_config_keys.py to the sources that
+actually define top-level keys, so the next new core setting cannot regress it.
+"""
+
+import json
+import re
+import sys
+from pathlib import Path
+from unittest.mock import MagicMock, Mock
+
+import pytest
+from flask import Flask
+
+REPO = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO))
+
+from src.core_config_keys import CORE_CONFIG_KEYS  # noqa: E402
+from src.plugin_system.state_reconciliation import (  # noqa: E402
+    InconsistencyType,
+    StateReconciliation,
+)
+
+ON_DISK = InconsistencyType.PLUGIN_MISSING_ON_DISK
+IN_CONFIG = InconsistencyType.PLUGIN_MISSING_IN_CONFIG
+
+TEMPLATE = json.loads((REPO / "config" / "config.template.json").read_text(encoding="utf-8"))
+BUNDLED_PLUGINS = {p.parent.name for p in (REPO / "plugin-repos").glob("*/manifest.json")}
+
+# Top-level keys of the hdpi device's config.json (v3.4.0 fddb0e06), keys only.
+DEVICE_CONFIG_KEYS = [
+    'web_display_autostart', 'schedule', 'dim_schedule', 'timezone', 'target_fps',
+    'location', 'display', 'sync', 'plugin_system', 'web-ui-info', 'starlark-apps',
+    'football-scoreboard', 'hockey-scoreboard', 'lacrosse-scoreboard', 'afl-scoreboard',
+    'nrl-scoreboard', 'soccer-scoreboard', 'baseball-scoreboard', 'ledmatrix-flights',
+    'birdnet-go', 'geochron', 'incoming-packages', 'jellyfin-now-playing', 'tide-display',
+    'ledmatrix-leaderboard', 'ledmatrix-music', 'odds-ticker', 'stock-news',
+    'basketball-scoreboard', 'ledmatrix-stocks', 'ledmatrix-weather', 'news',
+    'clock-simple', 'christmas-countdown', '7-segment-clock', 'calendar', 'on-air',
+    'static-image', 'ledmatrix-dresden-departures', 'olympics', 'ledmatrix-elections',
+    'f1-scoreboard', 'hello-world', 'tidbyt-baseball-scoreboard', 'youtube-stats',
+    'mqtt-notifications', 'march-madness', 'pga-tour-leaderboard', 'nfl-draft',
+    'pomodoro-timer', 'countdown', 'gif-player', 'of-the-day', 'text-display',
+    'masters-tournament', 'cricket-scoreboard', 'ufc-scoreboard', 'auto_update',
+]
+DEVICE_SECRETS_KEYS = ['github', 'incoming-packages', 'jellyfin-now-playing']
+DEVICE_INSTALLED = [k for k in DEVICE_CONFIG_KEYS if k not in CORE_CONFIG_KEYS] + [
+    'ledmatrix-flights.standalone-backup-migrating']
+
+
+def _device_config():
+    """Device-shaped config: real key set, placeholder values."""
+    cfg = {}
+    for key in DEVICE_CONFIG_KEYS:
+        template_value = TEMPLATE.get(key)
+        cfg[key] = template_value if template_value is not None else {"enabled": True}
+    return cfg
+
+
+class _ConfigManager:
+    def __init__(self, config, secrets_path=None):
+        self._config = config
+        self._secrets_path = secrets_path
+        self.saved = []
+
+    def load_config(self):
+        return dict(self._config)
+
+    def save_config(self, config):
+        self.saved.append(dict(config))
+        self._config = dict(config)
+
+    def get_secrets_path(self):
+        return self._secrets_path
+
+
+def _install(plugins_dir, plugin_id):
+    d = plugins_dir / plugin_id
+    d.mkdir(parents=True, exist_ok=True)
+    (d / "manifest.json").write_text(json.dumps({"id": plugin_id, "version": "1.0.0"}),
+                                     encoding="utf-8")
+
+
+def _reconcile(tmp_path, config, installed=(), secrets=None):
+    plugins_dir = tmp_path / "plugin-repos"
+    plugins_dir.mkdir(exist_ok=True)
+    for pid in installed:
+        _install(plugins_dir, pid)
+    secrets_path = tmp_path / "config_secrets.json"
+    secrets_path.write_text(json.dumps(secrets or {}), encoding="utf-8")
+    state_manager = Mock()
+    state_manager.get_all_states.return_value = {}
+    plugin_manager = Mock()
+    plugin_manager.plugin_manifests = {}
+    reconciler = StateReconciliation(
+        state_manager=state_manager,
+        config_manager=_ConfigManager(config, str(secrets_path)),
+        plugin_manager=plugin_manager,
+        plugins_dir=plugins_dir,
+    )
+    return reconciler, reconciler.reconcile_state()
+
+
+def _ids(result, kind):
+    return sorted(i.plugin_id for i in result.inconsistencies_found
+                  if i.inconsistency_type == kind)
+
+
+class TestTheListCoversEveryCoreKey:
+    def test_every_template_top_level_key_is_core_or_a_bundled_plugin(self):
+        unaccounted = [k for k in TEMPLATE
+                       if k not in CORE_CONFIG_KEYS and k not in BUNDLED_PLUGINS]
+        assert unaccounted == [], (
+            "config.template.json has top-level keys that are neither in "
+            "src/core_config_keys.py nor a plugin under plugin-repos/; "
+            "reconciliation will report them as plugins missing from disk")
+
+    def test_the_template_keys_this_bug_was_about_are_covered(self):
+        for key in ("auto_update", "sync", "schedule", "dim_schedule", "location",
+                    "display", "plugin_system", "timezone", "target_fps",
+                    "web_display_autostart"):
+            assert key in CORE_CONFIG_KEYS, key
+
+    def test_keys_written_by_the_general_settings_save_are_core(self):
+        """A new setting can be added to the save endpoint without touching
+        the template; catch that too."""
+        source = (REPO / "web_interface" / "blueprints" / "api_v3" / "config.py").read_text(
+            encoding="utf-8")
+        written = set(re.findall(r"current_config\[['\"]([A-Za-z0-9_-]+)['\"]\]\s*=", source))
+        assert "auto_update" in written, "pattern no longer finds the writes; update the test"
+        assert written - CORE_CONFIG_KEYS == set()
+
+    def test_no_core_key_is_also_a_bundled_plugin(self):
+        assert CORE_CONFIG_KEYS & BUNDLED_PLUGINS == set()
+
+
+class TestReconciliationIgnoresCoreKeys:
+    def test_auto_update_is_not_reported(self, tmp_path):
+        _, result = _reconcile(tmp_path, {"auto_update": {"enabled": False}})
+        assert "auto_update" not in [i.plugin_id for i in result.inconsistencies_found]
+
+    def test_the_whole_template_yields_only_its_bundled_plugins(self, tmp_path):
+        template_plugins = {k for k in TEMPLATE if k in BUNDLED_PLUGINS}
+        reconciler, result = _reconcile(tmp_path, dict(TEMPLATE), installed=template_plugins)
+        assert result.inconsistencies_found == []
+        assert set(reconciler._get_config_state()) == template_plugins
+
+    @pytest.mark.parametrize("key", sorted(CORE_CONFIG_KEYS))
+    def test_no_core_key_is_ever_a_plugin_missing_from_disk(self, tmp_path, key):
+        _, result = _reconcile(tmp_path, {key: {"enabled": True}})
+        assert _ids(result, ON_DISK) == []
+
+    def test_a_genuinely_orphaned_plugin_is_still_reported(self, tmp_path):
+        config = {"auto_update": {"enabled": False}, "display": {"hardware": {}},
+                  "ghost-plugin": {"enabled": True}, "odds-ticker": {"enabled": True}}
+        _, result = _reconcile(tmp_path, config, installed=["odds-ticker"])
+        assert _ids(result, ON_DISK) == ["ghost-plugin"]
+        assert [i.plugin_id for i in result.inconsistencies_manual] == ["ghost-plugin"]
+
+    def test_the_device_config_reports_nothing(self, tmp_path):
+        _, result = _reconcile(
+            tmp_path, _device_config(), installed=DEVICE_INSTALLED,
+            secrets={k: {"api_key": "x"} for k in DEVICE_SECRETS_KEYS})
+        assert result.inconsistencies_found == []
+        assert result.reconciliation_successful
+
+
+class TestPluginSecretsDoNotHideThePlugin:
+    """Plugin secrets are namespaced by plugin id in config_secrets.json. The
+    device had 'incoming-packages' and 'jellyfin-now-playing' there; ignoring
+    every secrets key reported both installed plugins as missing from config on
+    every run (auto-"fixed" as a no-op, so it never reached the banner)."""
+
+    def test_installed_plugin_with_secrets_is_a_plugin(self, tmp_path):
+        config = {"incoming-packages": {"enabled": True, "api_key": "x"}}
+        reconciler, result = _reconcile(tmp_path, config, installed=["incoming-packages"],
+                                        secrets={"incoming-packages": {"api_key": "x"}})
+        assert result.inconsistencies_found == []
+        assert "incoming-packages" in reconciler._get_config_state()
+
+    def test_a_secrets_only_key_is_still_not_a_plugin(self, tmp_path):
+        # The #557 phantom: a non-plugin 'data' key in the secrets file.
+        _, result = _reconcile(tmp_path, {"data": {"mode": "nfl_recent"}},
+                               secrets={"data": {"mode": "nfl_recent"}})
+        assert result.inconsistencies_found == []
+
+
+class TestAPluginNamedLikeACoreKey:
+    """A plugin whose id collides with a core key has no config section of its
+    own. Reconciliation must neither nag about it nor write a stub over (or
+    in place of) the core setting."""
+
+    def test_installed_plugin_named_like_a_core_key_is_not_reported(self, tmp_path):
+        config = {"sync": {"role": "standalone"}}
+        reconciler, result = _reconcile(tmp_path, config, installed=["sync"])
+        assert result.inconsistencies_found == []
+        assert reconciler.config_manager.saved == []
+        assert reconciler.config_manager.load_config() == config
+
+    def test_core_key_absent_from_config_is_not_stubbed(self, tmp_path):
+        reconciler, result = _reconcile(tmp_path, {}, installed=["display"])
+        assert result.inconsistencies_found == []
+        assert reconciler.config_manager.saved == []
+
+    def test_fix_refuses_to_write_a_core_key(self, tmp_path):
+        reconciler, _ = _reconcile(tmp_path, {})
+        inc = Mock(inconsistency_type=IN_CONFIG, plugin_id="auto_update")
+        assert reconciler._fix_inconsistency(inc) is False
+        assert reconciler.config_manager.saved == []
+
+
+class TestTheStatusEndpoint:
+    @pytest.fixture
+    def get_status(self, tmp_path, monkeypatch):
+        import tempfile
+        from web_interface.blueprints.api_v3 import api_v3
+
+        monkeypatch.setattr(tempfile, "gettempdir", lambda: str(tmp_path))
+
+        def _get(verdict, config, installed=(), secrets=None):
+            (tmp_path / "ledmatrix_reconciliation.json").write_text(
+                json.dumps(verdict), encoding="utf-8")
+            plugins_dir = tmp_path / "plugin-repos"
+            plugins_dir.mkdir(exist_ok=True)
+            for pid in installed:
+                _install(plugins_dir, pid)
+            secrets_path = tmp_path / "config_secrets.json"
+            secrets_path.write_text(json.dumps(secrets or {}), encoding="utf-8")
+            cm = MagicMock()
+            cm.load_config.return_value = config
+            cm.get_secrets_path.return_value = str(secrets_path)
+            pm = MagicMock()
+            pm.plugins_dir = str(plugins_dir)
+            monkeypatch.setattr(api_v3, "config_manager", cm, raising=False)
+            monkeypatch.setattr(api_v3, "plugin_manager", pm, raising=False)
+            app = Flask(__name__)
+            app.config["TESTING"] = True
+            app.register_blueprint(api_v3, url_prefix="/api/v3")
+            resp = app.test_client().get("/api/v3/plugins/reconciliation-status")
+            return resp.get_json()["data"]
+
+        return _get
+
+    def test_the_verdict_the_device_served_now_clears(self, get_status):
+        """The exact stored verdict from the device, written by the old build,
+        is dropped against the device's config -- no restart needed."""
+        verdict = {"done": True, "fixed_count": 3, "successful": False, "unresolved": [
+            {"description": "Plugin auto_update in config but not on disk",
+             "plugin_id": "auto_update", "type": ON_DISK.value}]}
+        data = get_status(verdict, _device_config(), installed=DEVICE_INSTALLED,
+                          secrets={k: {} for k in DEVICE_SECRETS_KEYS})
+        assert data["unresolved"] == []
+
+    def test_a_real_orphan_still_reaches_the_banner(self, get_status):
+        verdict = {"done": True, "unresolved": [
+            {"plugin_id": "auto_update", "type": ON_DISK.value},
+            {"plugin_id": "ghost-plugin", "type": ON_DISK.value}]}
+        config = _device_config()
+        config["ghost-plugin"] = {"enabled": True}
+        data = get_status(verdict, config, installed=DEVICE_INSTALLED)
+        assert [e["plugin_id"] for e in data["unresolved"]] == ["ghost-plugin"]

test/test_legacy_boolean_config.py

@@ -0,0 +1,394 @@
+"""A boolean the schema has since turned into an ``{enabled, ...}`` object.
+
+The news plugin's ``global.dynamic_duration`` went from ``true`` to an object
+with ``enabled``, ``min_duration_seconds``, ... An install that has not saved
+the news settings since still holds ``true``, and on every start the display
+service logged::
+
+    Plugin news config does not match its schema (loading anyway):
+    Field 'global.dynamic_duration': Expected type object, got bool
+
+and flagged news degraded. The settings form already reads such a boolean as
+``{"enabled": <bool>}`` (``render_nested_section`` in plugin_config.html); the
+loader did not. Both now follow ``legacy_bool_as_object``, and the parity tests
+below render the real macro against it so the two cannot drift.
+"""
+
+import json
+import sys
+from html.parser import HTMLParser
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+PROJECT_ROOT = Path(__file__).parent.parent
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+from src.plugin_system.plugin_manager import PluginManager
+from src.plugin_system.schema_manager import (
+    legacy_bool_as_object,
+    normalize_legacy_booleans,
+)
+
+
+DYNAMIC_DURATION = {
+    "type": "object",
+    "properties": {
+        "enabled": {"type": "boolean", "default": True},
+        "min_duration_seconds": {"type": "integer", "default": 30,
+                                 "minimum": 10, "maximum": 300},
+        "max_duration_seconds": {"type": "integer", "default": 300,
+                                 "minimum": 30, "maximum": 600},
+        "buffer_ratio": {"type": "number", "default": 0.1,
+                         "minimum": 0.01, "maximum": 1.0},
+    },
+    "additionalProperties": False,
+}
+
+# The shape of news' schema, cut down to what the tests need.
+NEWS_SCHEMA = {
+    "type": "object",
+    "properties": {
+        "enabled": {"type": "boolean", "default": True},
+        "global": {
+            "type": "object",
+            "properties": {
+                "font_size": {"type": "integer", "default": 12},
+                "dynamic_duration": DYNAMIC_DURATION,
+                "display": {
+                    "type": "object",
+                    "properties": {
+                        "scroll_speed": {"type": "number", "default": 1.0},
+                    },
+                    "additionalProperties": False,
+                },
+                "headline_paging": {
+                    "type": "object",
+                    "properties": {
+                        "enabled": {"type": "boolean", "default": False},
+                        "page_hold_seconds": {"type": "number", "default": 2.0},
+                    },
+                    "additionalProperties": False,
+                },
+            },
+            "additionalProperties": False,
+        },
+        "feeds": {
+            "type": "object",
+            "properties": {
+                "custom_feeds": {
+                    "type": "array",
+                    "items": {
+                        "type": "object",
+                        "properties": {
+                            "name": {"type": "string"},
+                            "enabled": {"type": "boolean", "default": True},
+                            "schedule": {
+                                "type": "object",
+                                "properties": {"enabled": {"type": "boolean"}},
+                            },
+                        },
+                    },
+                },
+            },
+        },
+    },
+    "additionalProperties": False,
+}
+
+
+# --------------------------------------------------------------------------
+# The helper
+# --------------------------------------------------------------------------
+
+class TestLegacyBoolAsObject:
+    def test_true_under_an_enabled_object_becomes_its_switch(self):
+        assert legacy_bool_as_object(True, DYNAMIC_DURATION) == {"enabled": True}
+
+    def test_false_stays_off(self):
+        """Upgrading must never turn a feature the user switched off back on."""
+        assert legacy_bool_as_object(False, DYNAMIC_DURATION) == {"enabled": False}
+
+    def test_nullable_object_type_list_counts_as_object(self):
+        prop = dict(DYNAMIC_DURATION, type=["object", "null"])
+        assert legacy_bool_as_object(True, prop) == {"enabled": True}
+
+    @pytest.mark.parametrize("value", [1, 0, "true", "false", None, [], 2.5])
+    def test_a_value_that_is_not_a_real_bool_is_untouched(self, value):
+        assert legacy_bool_as_object(value, DYNAMIC_DURATION) is value
+
+    def test_an_object_without_enabled_is_untouched(self):
+        prop = {"type": "object", "properties": {"speed": {"type": "number"}}}
+        assert legacy_bool_as_object(True, prop) is True
+
+    def test_a_boolean_or_object_union_is_untouched(self):
+        """The form draws ``["boolean", "object"]`` as a checkbox, and the
+        schema accepts the boolean, so there is nothing to upgrade."""
+        prop = dict(DYNAMIC_DURATION, type=["boolean", "object"])
+        assert legacy_bool_as_object(True, prop) is True
+
+    def test_a_non_object_schema_is_untouched(self):
+        assert legacy_bool_as_object(True, {"type": "boolean"}) is True
+        assert legacy_bool_as_object(True, None) is True
+
+
+class TestNormalizeLegacyBooleans:
+    def test_top_level(self):
+        schema = {"type": "object",
+                  "properties": {"dynamic_duration": DYNAMIC_DURATION}}
+        out = normalize_legacy_booleans({"dynamic_duration": True}, schema)
+        assert out == {"dynamic_duration": {"enabled": True}}
+
+    def test_nested(self):
+        cfg = {"global": {"font_size": 12, "dynamic_duration": True}}
+        out = normalize_legacy_booleans(cfg, NEWS_SCHEMA)
+        assert out == {"global": {"font_size": 12,
+                                  "dynamic_duration": {"enabled": True}}}
+
+    def test_several_depths_report_their_paths(self):
+        schema = {"type": "object", "properties": {
+            "dynamic_duration": DYNAMIC_DURATION,
+            "outer": {"type": "object", "properties": {
+                "inner": {"type": "object", "properties": {
+                    "mode": DYNAMIC_DURATION}}}},
+        }}
+        cfg = {"dynamic_duration": False, "outer": {"inner": {"mode": True}}}
+        changed = []
+        out = normalize_legacy_booleans(cfg, schema, changed)
+        assert out == {"dynamic_duration": {"enabled": False},
+                       "outer": {"inner": {"mode": {"enabled": True}}}}
+        assert changed == ["dynamic_duration", "outer.inner.mode"]
+
+    def test_array_items_are_left_alone_as_the_form_does(self):
+        """The form hands arrays to widgets and never upgrades inside them."""
+        cfg = {"feeds": {"custom_feeds": [{"name": "a", "schedule": True}]}}
+        out = normalize_legacy_booleans(cfg, NEWS_SCHEMA)
+        assert out is cfg
+
+    def test_already_an_object_is_untouched(self):
+        cfg = {"global": {"dynamic_duration": {"enabled": False,
+                                               "min_duration_seconds": 45}}}
+        assert normalize_legacy_booleans(cfg, NEWS_SCHEMA) is cfg
+
+    def test_other_mismatches_are_untouched(self):
+        cfg = {"global": {"font_size": True, "display": True,
+                          "dynamic_duration": "yes"}}
+        changed = []
+        assert normalize_legacy_booleans(cfg, NEWS_SCHEMA, changed) is cfg
+        assert changed == []
+
+    def test_keys_the_schema_does_not_declare_are_untouched(self):
+        cfg = {"global": {"mystery": True}, "vegas_width_pct": 50}
+        assert normalize_legacy_booleans(cfg, NEWS_SCHEMA) is cfg
+
+    def test_the_stored_config_is_not_mutated(self):
+        """load_config() may hand back a cached dict; it must stay as stored."""
+        cfg = {"enabled": True, "global": {"dynamic_duration": True}}
+        before = json.loads(json.dumps(cfg))
+        out = normalize_legacy_booleans(cfg, NEWS_SCHEMA)
+        assert cfg == before
+        assert out is not cfg and out["global"] is not cfg["global"]
+
+    @pytest.mark.parametrize("config,schema", [
+        (None, NEWS_SCHEMA), ([], NEWS_SCHEMA), ({"a": True}, None),
+        ({"a": True}, {"type": "object"}),
+    ])
+    def test_tolerates_odd_inputs(self, config, schema):
+        assert normalize_legacy_booleans(config, schema) is config
+
+
+# --------------------------------------------------------------------------
+# The load path
+# --------------------------------------------------------------------------
+
+PLUGIN_ID = "news-shaped"
+
+
+@pytest.fixture
+def load(tmp_path):
+    """Run the real ``PluginManager.load_plugin`` for a plugin with
+    ``NEWS_SCHEMA`` and the given stored section; the plugin class is stubbed.
+
+    Returns ``(loaded, config the plugin received, logger, health_tracker)``.
+    """
+    plugins_dir = tmp_path / "plugins"
+    plugin_dir = plugins_dir / PLUGIN_ID
+    plugin_dir.mkdir(parents=True)
+    manifest = {"id": PLUGIN_ID, "name": "News-shaped", "version": "1.0.0",
+                "entry_point": "manager.py", "class_name": "Plugin"}
+    (plugin_dir / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
+    (plugin_dir / "config_schema.json").write_text(json.dumps(NEWS_SCHEMA),
+                                                   encoding="utf-8")
+
+    def _load(stored_section):
+        config_manager = MagicMock()
+        config_manager.load_config.return_value = {
+            PLUGIN_ID: json.loads(json.dumps(stored_section))}
+        with patch('src.common.permission_utils.ensure_directory_permissions'):
+            manager = PluginManager(plugins_dir=str(plugins_dir),
+                                    config_manager=config_manager,
+                                    display_manager=MagicMock(),
+                                    cache_manager=MagicMock())
+        manager.logger = MagicMock()
+        manager.health_tracker = MagicMock()
+        manager.plugin_manifests[PLUGIN_ID] = manifest
+        manager.plugin_loader.find_plugin_directory = MagicMock(return_value=plugin_dir)
+
+        received = {}
+
+        def fake_load_plugin(**kwargs):
+            received.update(kwargs["config"])
+            instance = MagicMock()
+            instance.validate_config.return_value = True
+            return instance, MagicMock()
+
+        manager.plugin_loader.load_plugin = MagicMock(side_effect=fake_load_plugin)
+        loaded = manager.load_plugin(PLUGIN_ID)
+        return loaded, received, manager.logger, manager.health_tracker
+
+    return _load
+
+
+def _schema_warnings(logger):
+    return [c for c in logger.warning.call_args_list
+            if "does not match its schema" in str(c.args[0])]
+
+
+def _degraded_reason(health_tracker):
+    calls = [c.args for c in health_tracker.set_degraded.call_args_list
+             if c.args[0] == PLUGIN_ID]
+    assert calls, "schema validation never ran"
+    return calls[-1][1]
+
+
+class TestLoadPath:
+    def test_news_shaped_legacy_config_loads_without_warning_or_degraded(self, load):
+        stored = {"enabled": True,
+                  "global": {"font_size": 12, "dynamic_duration": True,
+                             "headline_paging": {"enabled": False}}}
+        loaded, received, logger, tracker = load(stored)
+
+        assert loaded is True
+        assert _schema_warnings(logger) == []
+        assert _degraded_reason(tracker) is None
+        # The plugin gets the object, with the schema defaults filled in.
+        assert received["global"]["dynamic_duration"] == {
+            "enabled": True, "min_duration_seconds": 30,
+            "max_duration_seconds": 300, "buffer_ratio": 0.1}
+        # And the upgrade is logged once, below warning level.
+        assert any("global.dynamic_duration" in str(c.args)
+                   for c in logger.info.call_args_list)
+
+    def test_a_legacy_false_loads_switched_off(self, load):
+        loaded, received, logger, tracker = load(
+            {"enabled": True, "global": {"dynamic_duration": False}})
+        assert loaded is True
+        assert received["global"]["dynamic_duration"]["enabled"] is False
+        assert _schema_warnings(logger) == []
+        assert _degraded_reason(tracker) is None
+
+    def test_a_real_mismatch_still_warns_and_degrades(self, load):
+        loaded, received, logger, tracker = load(
+            {"enabled": True, "global": {"dynamic_duration": True,
+                                         "font_size": "big"}})
+        assert loaded is True  # warn-only, as before
+        warnings = _schema_warnings(logger)
+        assert len(warnings) == 1
+        assert "font_size" in str(warnings[0].args)
+        assert "dynamic_duration" not in str(warnings[0].args)
+        reason = _degraded_reason(tracker)
+        assert reason and "font_size" in reason
+
+    def test_a_boolean_under_an_object_without_enabled_still_warns(self, load):
+        loaded, received, logger, tracker = load(
+            {"enabled": True, "global": {"display": True}})
+        assert loaded is True
+        assert received["global"]["display"] is True
+        warnings = _schema_warnings(logger)
+        assert len(warnings) == 1 and "global.display" in str(warnings[0].args)
+        assert _degraded_reason(tracker)
+
+
+# --------------------------------------------------------------------------
+# Parity with the settings form
+# --------------------------------------------------------------------------
+
+PARITY_SCHEMA = {
+    "type": "object",
+    "properties": {
+        "enabled": {"type": "boolean", "default": True},
+        "global": {"type": "object", "properties": {
+            "dynamic_duration": DYNAMIC_DURATION,
+            "outer": {"type": "object", "properties": {
+                "inner": {"type": "object", "properties": {
+                    "enabled": {"type": "boolean", "default": True},
+                    "level": {"type": "integer", "default": 1},
+                }},
+            }},
+            "union": {"type": ["boolean", "object"], "properties": {
+                "enabled": {"type": "boolean"}}},
+        }},
+    },
+}
+
+
+class _Checkboxes(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.checked = {}
+
+    def handle_starttag(self, tag, attrs):
+        a = dict(attrs)
+        if tag == "input" and a.get("type") == "checkbox" and a.get("name"):
+            self.checked[a["name"]] = "checked" in a
+
+
+def _form_checkboxes(stored):
+    from jinja2 import Environment, FileSystemLoader, select_autoescape
+
+    env = Environment(
+        loader=FileSystemLoader(str(PROJECT_ROOT / "web_interface" / "templates")),
+        autoescape=select_autoescape(["html"]),
+    )
+    plugin = {"id": "demo", "name": "Demo", "description": "", "enabled": True,
+              "author": "me", "version": "1.0.0"}
+    html = env.get_template("v3/partials/plugin_config.html").render(
+        plugin=plugin, schema=PARITY_SCHEMA, config=json.loads(json.dumps(stored)))
+    parser = _Checkboxes()
+    parser.feed(html)
+    return parser.checked
+
+
+def _loader_enabled(normalized, *path):
+    node = normalized
+    for key in path:
+        node = node.get(key) if isinstance(node, dict) else None
+    return isinstance(node, dict) and node.get("enabled") is True
+
+
+@pytest.mark.parametrize("value", [True, False, 1, "true", None])
+def test_form_and_loader_agree_on_what_a_stored_value_means(value):
+    """Where the loader reads a stored value as ``{"enabled": true}``, the form
+    draws the object's ``enabled`` checkbox ticked, and nowhere else."""
+    stored = {"enabled": True,
+              "global": {"dynamic_duration": value, "outer": {"inner": value}}}
+    normalized = normalize_legacy_booleans(stored, PARITY_SCHEMA)
+    boxes = _form_checkboxes(stored)
+
+    for path in (("global", "dynamic_duration"), ("global", "outer", "inner")):
+        name = ".".join(path) + ".enabled"
+        assert name in boxes, f"form drew no {name} checkbox"
+        assert boxes[name] is _loader_enabled(normalized, *path), name
+
+
+@pytest.mark.parametrize("value", [True, False])
+def test_form_and_loader_agree_a_boolean_union_stays_a_boolean(value):
+    stored = {"enabled": True, "global": {"union": value}}
+    normalized = normalize_legacy_booleans(stored, PARITY_SCHEMA)
+    boxes = _form_checkboxes(stored)
+
+    assert normalized is stored
+    assert boxes.get("global.union") is value
+    assert "global.union.enabled" not in boxes

test/test_reconciliation_phantom_plugins.py

@@ -142,8 +142,14 @@ class TestStaleFindingsAreDropped:
         assert still_unresolved(entries, set(), {"odds-ticker"}) == []
 
     def test_genuinely_missing_on_disk_is_kept(self):
-        entries = [{"plugin_id": "data", "type": self.ON_DISK}]
-        assert still_unresolved(entries, set(), {"odds-ticker"}) == entries
+        entries = [{"plugin_id": "ghost-plugin", "type": self.ON_DISK}]
+        assert still_unresolved(entries, {"ghost-plugin"}, {"odds-ticker"}) == entries
+
+    def test_missing_on_disk_no_longer_in_config_is_dropped(self):
+        # Removed from config.json (or never a plugin entry at all, like a
+        # core key an older build misread): nothing left to reinstall.
+        entries = [{"plugin_id": "ghost-plugin", "type": self.ON_DISK}]
+        assert still_unresolved(entries, set(), {"odds-ticker"}) == []
 
     def test_unrecheckable_kinds_are_kept(self):
         # Filtering must only ever remove what it can prove stale.
@@ -164,7 +170,6 @@ class TestStaleFindingsAreDropped:
         live = still_unresolved(entries, installed, installed)
 
         # All four "installed but not in config" findings were false and clear.
-        # 'data' legitimately is not installed, so this filter keeps it; it stops
-        # being reported because _get_config_state() no longer invents it from the
-        # secrets file, which takes effect on the next reconciliation run.
-        assert [e["plugin_id"] for e in live] == ["data"]
+        # 'data' is not a plugin entry in config (it came from the secrets
+        # file), so its "not on disk" finding is provably stale and clears too.
+        assert live == []

test/test_reconciliation_status_endpoint.py

@@ -94,9 +94,16 @@ class TestStaleFindingsAreDroppedByTheEndpoint:
 
     def test_a_finding_that_still_holds_is_kept(self, client):
         c = client({"done": True, "unresolved": [
-            {"plugin_id": "ghost-plugin", "type": ON_DISK}]})
+            {"plugin_id": "ghost-plugin", "type": ON_DISK}]},
+            config={"ghost-plugin": {"enabled": True}})
         assert [e["plugin_id"] for e in _get(c)["unresolved"]] == ["ghost-plugin"]
 
+    def test_a_plugin_removed_from_config_is_no_longer_reported(self, client):
+        c = client({"done": True, "unresolved": [
+            {"plugin_id": "ghost-plugin", "type": ON_DISK}]},
+            config={"other-plugin": {"enabled": True}})
+        assert _get(c)["unresolved"] == []
+
     def test_the_reported_device_verdict_clears(self, client):
         """The five findings the live device served, against its real state."""
         installed = ["football-scoreboard", "ledmatrix-weather",
@@ -157,11 +164,13 @@ class TestTheFilterUsesTheReconcilersOwnRules:
         # finding on the strength of something nothing can read.
         c = client({"done": True, "unresolved": [
             {"plugin_id": "broken-plugin", "type": ON_DISK}]},
+            config={"broken-plugin": {"enabled": True}},
             corrupt=["broken-plugin"])
         assert [e["plugin_id"] for e in _get(c)["unresolved"]] == ["broken-plugin"]
 
     def test_a_standalone_backup_dir_does_not_count_as_installed(self, client):
         c = client({"done": True, "unresolved": [
             {"plugin_id": "weather.standalone-backup-20260101", "type": ON_DISK}]},
+            config={"weather.standalone-backup-20260101": {"enabled": True}},
             installed=["weather.standalone-backup-20260101"])
         assert len(_get(c)["unresolved"]) == 1

test/test_sports_helpers.py

@@ -0,0 +1,603 @@
+"""src.common.sports_helpers: behaviour, host contract, and parity with the plugins.
+
+The bodies were copied from the scoreboard plugins' sports.py, so the plugins'
+own tests already cover them there. They are ported here for two reasons: core
+now owns a copy that can drift on its own, and a plugin that deletes its copy
+loses those tests with it.
+
+The parity class is what keeps "byte-identical" true after this lands. Point
+LEDMATRIX_PLUGINS at a ledmatrix-plugins checkout and every promoted body is
+compared, as a docstring-stripped AST, against every plugin copy that carries
+it. Without the variable it skips rather than fails, since core CI has no
+plugins checkout; ledmatrix-plugins CI runs the same comparison against core
+(scripts/check_sports_helpers_parity.py, ledmatrix-plugins#495).
+"""
+
+import ast
+import logging
+import math
+import os
+import time
+from pathlib import Path
+
+import pytest
+from PIL import Image
+from PIL.PngImagePlugin import PngInfo
+
+from src.common import sports_helpers
+from src.common.sports_helpers import (
+    MAX_WINDOW_DAYS,
+    MIN_WINDOW_DAYS,
+    SportsHelpersMixin,
+    clamp_seconds,
+    clamp_window,
+    logo_needs_refresh,
+    spread_weighted_order,
+)
+from src.common.sports_shared import SportsCoreSharedMixin
+
+
+# ---------------------------------------------------------------------------
+# Free functions
+# ---------------------------------------------------------------------------
+
+class TestClampWindow:
+    def test_within_range_is_kept(self):
+        assert clamp_window(14, 7) == 14
+        assert clamp_window("30", 7) == 30
+
+    def test_clamped_to_bounds(self):
+        assert (MIN_WINDOW_DAYS, MAX_WINDOW_DAYS) == (1, 60)
+        assert clamp_window(0, 7) == 1
+        assert clamp_window(-5, 7) == 1
+        assert clamp_window(365, 7) == 60
+
+    @pytest.mark.parametrize("bad", [None, "abc", [], {}, float("nan")])
+    def test_unusable_falls_back(self, bad):
+        assert clamp_window(bad, 7) == 7
+
+    @pytest.mark.parametrize("inf", [float("inf"), float("-inf")])
+    def test_infinity_falls_back_instead_of_raising(self, inf):
+        # json.loads accepts a bare Infinity; int(inf) raises OverflowError.
+        with pytest.raises(OverflowError):
+            int(inf)
+        assert clamp_window(inf, 7) == 7
+
+
+class TestClampSeconds:
+    def test_defaults_bound_five_seconds_to_a_day(self):
+        assert clamp_seconds(1, 60) == 5
+        assert clamp_seconds(10 ** 9, 60) == 86400
+        assert clamp_seconds(300, 60) == 300
+
+    def test_custom_bounds(self):
+        assert clamp_seconds(1, 60, low=30, high=900) == 30
+        assert clamp_seconds(5000, 60, low=30, high=900) == 900
+
+    @pytest.mark.parametrize("bad", [None, "x", float("nan"), float("inf")])
+    def test_unusable_falls_back(self, bad):
+        assert clamp_seconds(bad, 60) == 60
+
+
+def _png(path, size=(64, 64), color=(100, 100, 100, 255), marker=None):
+    img = Image.new("RGBA", size, color)
+    info = None
+    if marker is not None:
+        info = PngInfo()
+        info.add_text("ledmatrix_placeholder", marker)
+    img.save(path, "PNG", pnginfo=info)
+    return path
+
+
+class TestLogoNeedsRefresh:
+    def test_stale_placeholder_is_retried(self, tmp_path):
+        from src.logo_downloader import PLACEHOLDER_RETRY_SECONDS
+        stamp = str(time.time() - PLACEHOLDER_RETRY_SECONDS - 60)
+        assert logo_needs_refresh(_png(tmp_path / "A.png", marker=stamp)) is True
+
+    def test_fresh_placeholder_is_trusted(self, tmp_path):
+        assert logo_needs_refresh(_png(tmp_path / "A.png", marker=str(time.time()))) is False
+
+    def test_real_logo_is_never_refreshed(self, tmp_path):
+        assert logo_needs_refresh(_png(tmp_path / "A.png", size=(32, 32),
+                                       color=(200, 0, 0, 255))) is False
+
+    def test_missing_file_is_not_a_placeholder(self, tmp_path):
+        assert logo_needs_refresh(tmp_path / "nope.png") is False
+
+    def test_core_without_placeholder_marking_keeps_old_behaviour(self, monkeypatch, tmp_path):
+        # A None entry in sys.modules makes the import raise ImportError, the
+        # same thing an older core without these names does.
+        monkeypatch.setitem(__import__("sys").modules, "src.logo_downloader", None)
+        stamp = str(time.time() - 10 ** 7)
+        assert logo_needs_refresh(_png(tmp_path / "A.png", marker=stamp)) is False
+
+    def test_the_import_is_deferred(self):
+        tree = ast.parse(Path(sports_helpers.__file__).read_text(encoding="utf-8"))
+        top = [n for n in tree.body if isinstance(n, (ast.Import, ast.ImportFrom))]
+        assert not any(isinstance(n, ast.ImportFrom) and n.module == "src.logo_downloader"
+                       for n in top)
+
+
+def _back_to_back(cycle):
+    return any(cycle[i] == cycle[(i + 1) % len(cycle)] for i in range(len(cycle)))
+
+
+class TestSpreadWeightedOrder:
+    def test_equal_weights_are_plain_order(self):
+        assert spread_weighted_order([1, 1, 1]) == [0, 1, 2]
+        assert spread_weighted_order([]) == []
+
+    def test_a_favourite_at_the_end_wraps_its_extra_turn(self):
+        order = spread_weighted_order([1, 1, 1, 2])
+        assert sorted(order) == [0, 1, 2, 3, 3]
+        assert not _back_to_back(order)
+
+    @pytest.mark.parametrize("weights,expected", [
+        ([3, 1, 1], [0, 1, 0, 2, 0]),
+        ([1, 3, 1], [0, 1, 1, 2, 1]),
+        ([5, 1], None),
+        ([2, 1], None),
+    ])
+    def test_ratio_wins_over_spacing(self, weights, expected):
+        order = spread_weighted_order(weights)
+        assert [order.count(i) for i in range(len(weights))] == weights
+        if expected is not None:
+            assert order == expected
+
+    def test_mixin_exposes_it_as_a_staticmethod(self):
+        assert SportsHelpersMixin._spread_weighted_order([1, 2]) == spread_weighted_order([1, 2])
+        assert isinstance(SportsHelpersMixin.__dict__["_spread_weighted_order"], staticmethod)
+
+
+# ---------------------------------------------------------------------------
+# Mixin methods, against stand-in hosts carrying only what each reads
+# ---------------------------------------------------------------------------
+
+class _Dwell(SportsHelpersMixin):
+    def __init__(self, last_game_switch):
+        self._last_display_call_monotonic = 0.0
+        self.last_game_switch = last_game_switch
+
+
+class TestResetDwellOnReentry:
+    def test_first_frame_resets(self):
+        mgr = _Dwell(time.time() - 6.0)
+        assert mgr._reset_dwell_on_reentry() is True
+        assert time.time() - mgr.last_game_switch < 1.0
+
+    def test_frames_inside_one_stint_do_not(self):
+        mgr = _Dwell(time.time() - 6.0)
+        mgr._reset_dwell_on_reentry()
+        stamp = mgr.last_game_switch
+        assert [mgr._reset_dwell_on_reentry() for _ in range(3)] == [False] * 3
+        assert mgr.last_game_switch == stamp
+
+    def test_returning_after_a_long_gap_resets(self):
+        mgr = _Dwell(time.time() - 150.0)
+        mgr._last_display_call_monotonic = time.monotonic() - 150.0
+        assert mgr._reset_dwell_on_reentry() is True
+
+    def test_just_under_the_threshold_is_the_same_stint(self):
+        mgr = _Dwell(time.time() - 150.0)
+        mgr._last_display_call_monotonic = time.monotonic() - (
+            SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS - 1.0)
+        stamp = mgr.last_game_switch
+        assert mgr._reset_dwell_on_reentry() is False
+        assert mgr.last_game_switch == stamp
+
+    def test_live_zero_sentinel_survives(self):
+        mgr = _Dwell(0)
+        assert mgr._reset_dwell_on_reentry() is False
+        assert mgr.last_game_switch == 0
+
+    def test_host_without_any_state(self):
+        mgr = SportsHelpersMixin()
+        assert mgr._reset_dwell_on_reentry() is False  # no last_game_switch
+        assert mgr._last_display_call_monotonic > 0
+
+
+def _game(gid, home="X", away="Y"):
+    return {"id": gid, "home_abbr": home, "away_abbr": away}
+
+
+class _Switch(SportsHelpersMixin):
+    def __init__(self, games, boost=None, favorites=("UF",)):
+        self.games_list = games
+        self.favorite_teams = list(favorites)
+        if boost is not None:
+            self.favorite_rotation_boost = boost
+        self.current_game_index = 0
+
+    def _is_favorite_game(self, g):
+        return g["home_abbr"] in self.favorite_teams or g["away_abbr"] in self.favorite_teams
+
+    def walk(self, steps):
+        shown = []
+        for _ in range(steps):
+            self.current_game_index = self._next_switch_index()
+            shown.append(self.games_list[self.current_game_index]["id"])
+        return shown
+
+
+class TestNextSwitchIndex:
+    def test_no_boost_attribute_is_the_plain_rotation(self):
+        s = _Switch([_game("a"), _game("b", "UF"), _game("c")])
+        assert s.walk(6) == ["b", "c", "a", "b", "c", "a"]
+
+    def test_boost_without_a_favourite_changes_nothing(self):
+        assert _Switch([_game("a"), _game("b"), _game("c")], boost=3).walk(3) == ["b", "c", "a"]
+
+    def test_boost_two_spreads_the_favourite(self):
+        s = _Switch([_game("a"), _game("uf", "UF"), _game("c"), _game("d")], boost=2)
+        cycle = s.walk(5)
+        assert sorted(cycle) == ["a", "c", "d", "uf", "uf"]
+        assert not _back_to_back(cycle)
+        assert s.walk(5) == cycle
+
+    def test_boost_three_two_favourites(self):
+        s = _Switch([_game("a", "UF"), _game("b"), _game("c", "FSU"), _game("d"), _game("e")],
+                    boost=3, favorites=("UF", "FSU"))
+        cycle = s.walk(9)
+        assert cycle.count("a") == 3 and cycle.count("c") == 3
+
+    def test_index_reset_by_update_is_followed(self):
+        s = _Switch([_game("a"), _game("uf", "UF"), _game("c"), _game("d")], boost=2)
+        s.walk(3)
+        s.current_game_index = 0
+        assert s.walk(1)[0] == "uf"
+
+    def test_single_card_stays_put(self):
+        assert _Switch([_game("uf", "UF")], boost=5).walk(3) == ["uf", "uf", "uf"]
+
+
+class _Settings(SportsHelpersMixin):
+    def __init__(self, mode_config):
+        self.mode_config = mode_config
+        self.logger = logging.getLogger("test.sports_helpers")
+        self.league = "nfl"
+
+
+class TestSettingInt:
+    def test_reads_and_clamps(self):
+        assert _Settings({"n": 3})._setting_int("n", 1, 1, 5) == 3
+        assert _Settings({"n": "4"})._setting_int("n", 1, 1, 5) == 4
+        assert _Settings({"n": 99})._setting_int("n", 1, 1, 5) == 5
+        assert _Settings({"n": -2})._setting_int("n", 1, 1, 5) == 1
+        assert _Settings({})._setting_int("n", 2, 1, 5) == 2
+
+    @pytest.mark.parametrize("bad", ["abc", None, [], float("inf"), float("nan")])
+    def test_unusable_falls_back_with_a_warning(self, bad, caplog):
+        with caplog.at_level(logging.WARNING, logger="test.sports_helpers"):
+            assert _Settings({"n": bad})._setting_int("n", 2, 1, 5) == 2
+        assert "ignoring unusable n" in caplog.text
+
+
+class _Custom(SportsHelpersMixin):
+    def __init__(self, config, mode=None):
+        self.config = config
+        if mode is not None:
+            self.SKIN_MODE = mode
+
+
+class TestModeCustomization:
+    def test_no_mode_returns_block_as_is(self):
+        cust = {"score": {"font_size": 10}}
+        assert _Custom({"customization": cust})._mode_customization() is cust
+
+    def test_non_dict_customization(self):
+        assert _Custom({"customization": "junk"}, "live")._mode_customization() == {}
+        assert _Custom({}, "live")._mode_customization() == {}
+
+    def test_mode_overrides_merge_and_none_inherits(self):
+        cfg = {"customization": {
+            "score": {"font_size": 10, "text_color": [1, 1, 1]},
+            "layout": {"score": {"x_offset": 2, "y_offset": 3}},
+            "modes": {"live": {
+                "score": {"font_size": 12, "text_color": None},
+                "status": "not-a-dict",
+                "layout": {"score": {"y_offset": 0, "x_offset": None}, "bad": 5},
+            }},
+        }}
+        merged = _Custom(cfg, "live")._mode_customization()
+        assert merged["score"] == {"font_size": 12, "text_color": [1, 1, 1]}
+        assert merged["layout"]["score"] == {"x_offset": 2, "y_offset": 0}
+        assert "bad" not in merged["layout"]
+        # The stored config is not mutated.
+        assert cfg["customization"]["score"]["font_size"] == 10
+        assert cfg["customization"]["layout"]["score"]["y_offset"] == 3
+
+    def test_other_mode_is_untouched(self):
+        cust = {"modes": {"live": {"score": {"font_size": 12}}}}
+        assert _Custom({"customization": cust}, "recent")._mode_customization() is cust
+
+
+class TestOddsColor:
+    def test_without_element_color_is_green(self):
+        assert SportsHelpersMixin()._odds_color() == (0, 255, 0)
+
+    def test_raising_getter_is_green(self):
+        class Host(SportsHelpersMixin):
+            def _element_color(self, element, default):
+                raise RuntimeError("boom")
+        assert Host()._odds_color() == (0, 255, 0)
+
+    def test_asks_for_odds_text_with_green_default(self):
+        seen = []
+
+        class Host(SportsHelpersMixin):
+            def _element_color(self, element, default):
+                seen.append((element, default))
+                return (9, 9, 9)
+        assert Host()._odds_color() == (9, 9, 9)
+        assert seen == [("odds_text", (0, 255, 0))]
+
+
+class _Scorebug(SportsHelpersMixin):
+    def __init__(self, scroll_card):
+        self.config = {"scroll_card": scroll_card}
+
+    def _card_option(self, key, default=None):
+        return SportsCoreSharedMixin._card_option(self, key, default)
+
+    def _format_game_date(self, game_date, game=None):
+        return "DATE"
+
+    def _format_game_time(self, game_time):
+        return "TIME"
+
+
+class TestUpcomingDateAndTimeText:
+    def _text(self, block):
+        return _Scorebug(block)._upcoming_date_and_time_text("2026-10-07", "19:00")
+
+    def test_defaults_draw_both(self):
+        assert self._text({}) == ("DATE", "TIME")
+
+    def test_scroll_card_toggles_do_not_reach_the_scorebug(self):
+        assert self._text({"show_date": False, "show_time": False}) == ("DATE", "TIME")
+
+    def test_switch_toggles(self):
+        assert self._text({"switch_show_date": False}) == ("", "TIME")
+        assert self._text({"switch_show_time": False}) == ("DATE", "")
+
+
+class TestFavoriteKey:
+    def test_defaults_to_abbreviation(self):
+        assert SportsHelpersMixin()._favorite_key({"home_abbr": "UF"}, "home") == "UF"
+
+    def test_missing_abbreviation_never_matches(self):
+        assert SportsHelpersMixin()._favorite_key({}, "away") is None
+
+
+# ---------------------------------------------------------------------------
+# Host contract
+# ---------------------------------------------------------------------------
+
+def _self_reads():
+    """Every ``self.X`` / ``getattr(self, "X")`` the mixin reads, by parsing it."""
+    tree = ast.parse(Path(sports_helpers.__file__).read_text(encoding="utf-8"))
+    cls = next(n for n in tree.body
+               if isinstance(n, ast.ClassDef) and n.name == "SportsHelpersMixin")
+    names = set()
+    for node in ast.walk(cls):
+        if (isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load)
+                and isinstance(node.value, ast.Name) and node.value.id == "self"):
+            names.add(node.attr)
+        if (isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
+                and node.func.id == "getattr" and len(node.args) >= 2
+                and isinstance(node.args[0], ast.Name) and node.args[0].id == "self"
+                and isinstance(node.args[1], ast.Constant)):
+            names.add(node.args[1].value)
+    return names
+
+
+class _Host(SportsCoreSharedMixin, SportsHelpersMixin):
+    """The smallest thing a scoreboard's SportsCore has to be for these to run."""
+
+    def __init__(self, config, mode_config=None):
+        self.config = config
+        self.mode_config = mode_config or {}
+        self.logger = logging.getLogger("test.sports_helpers.host")
+        self.games_list = []
+        self.current_game_index = 0
+        self.last_game_switch = 0
+
+    def _is_favorite_game(self, game):
+        return game.get("home_abbr") == "UF"
+
+
+class TestHostContract:
+    def test_every_host_read_is_documented(self):
+        defined = set(dir(SportsHelpersMixin))
+        # Written by the mixin itself before (or instead of) being read.
+        own_state = {"_last_display_call_monotonic", "_switch_order",
+                     "_switch_order_key", "_switch_position"}
+        needed = _self_reads() - defined - own_state
+        doc = sports_helpers.__doc__
+        undocumented = sorted(n for n in needed | own_state if f"``{n}``" not in doc)
+        assert undocumented == [], (
+            f"self attributes read but not in the module docstring's host "
+            f"contract: {undocumented}")
+
+    def test_every_constant_read_has_a_default(self):
+        # SKIN_MODE is the one exception: the host's mode name, read with
+        # getattr(self, 'SKIN_MODE', None) where absence means "no mode". A
+        # default here would say nothing the getattr does not already.
+        missing = sorted(n for n in _self_reads() - {"SKIN_MODE"}
+                         if n.upper() == n and not hasattr(SportsHelpersMixin, n))
+        assert missing == []
+
+    def test_defaults_match_what_the_plugins_ship(self):
+        assert SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS == 5.0
+        assert SportsHelpersMixin.favorite_rotation_boost == 1
+
+    def test_stateless_and_constructor_free(self):
+        assert "__init__" not in SportsHelpersMixin.__dict__
+        assert SportsHelpersMixin().__dict__ == {}
+
+    def test_base_order_does_not_matter(self):
+        class Reversed(SportsHelpersMixin, SportsCoreSharedMixin):
+            __init__ = _Host.__init__
+            _is_favorite_game = _Host._is_favorite_game
+
+        for cls in (_Host, Reversed):
+            h = cls({"scroll_card": {"switch_show_time": False, "time_format": "24h"}})
+            assert h._upcoming_date_and_time_text("10/7", "7:00 PM") == ("10/7", "")
+
+    def test_shared_mixin_supplies_the_card_helpers(self):
+        h = _Host({"scroll_card": {"switch_date_format": "abbrev", "time_format": "24h"}})
+        assert h._upcoming_date_and_time_text("10/7", "7:05 PM") == ("Oct 7", "19:05")
+
+    def test_odds_color_resolves_through_element_style(self):
+        h = _Host({"customization": {"odds_text": {"text_color": [1, 2, 3]}}})
+        assert tuple(h._odds_color()) == (1, 2, 3)
+        assert tuple(_Host({})._odds_color()) == (0, 255, 0)
+
+    def test_rotation_settings_and_dwell_on_one_host(self):
+        h = _Host({}, mode_config={"favorite_rotation_boost": "2"})
+        h.favorite_rotation_boost = h._setting_int("favorite_rotation_boost", 1, 1, 5)
+        h.games_list = [_game("a"), _game("uf", "UF"), _game("c")]
+        seen = []
+        for _ in range(4):
+            h.current_game_index = h._next_switch_index()
+            seen.append(h.games_list[h.current_game_index]["id"])
+        assert sorted(seen) == ["a", "c", "uf", "uf"]
+        h.last_game_switch = time.time() - 100
+        assert h._reset_dwell_on_reentry() is True
+
+
+# ---------------------------------------------------------------------------
+# Parity with the plugin copies
+# ---------------------------------------------------------------------------
+
+SCOREBOARDS = ("afl", "baseball", "basketball", "football", "hockey",
+               "lacrosse", "nrl", "soccer", "ufc")
+
+#: core name -> (where it lives in the plugins' sports.py, plugin name,
+#: plugins expected to carry it).
+_ALL = SCOREBOARDS
+_NOT_UFC = tuple(p for p in SCOREBOARDS if p != "ufc")
+PROMOTED = {
+    "clamp_window": ("module", "_clamp_window", _ALL),
+    "clamp_seconds": ("module", "_clamp_seconds", _ALL),
+    "logo_needs_refresh": ("module", "_logo_needs_refresh", _ALL),
+    "spread_weighted_order": ("SportsCore", "_spread_weighted_order", _ALL),
+    "_mode_customization": ("SportsCore", "_mode_customization", _ALL),
+    "_setting_int": ("SportsCore", "_setting_int", _ALL),
+    "_reset_dwell_on_reentry": ("SportsCore", "_reset_dwell_on_reentry", _ALL),
+    "_next_switch_index": ("SportsCore", "_next_switch_index", _ALL),
+    "_odds_color": ("SportsCore", "_odds_color", _NOT_UFC),
+    "_upcoming_date_and_time_text": ("SportsCore", "_upcoming_date_and_time_text", _NOT_UFC),
+}
+
+#: Public names here that are private in the plugins.
+_RENAMES = {
+    "clamp_window": "_clamp_window",
+    "clamp_seconds": "_clamp_seconds",
+    "logo_needs_refresh": "_logo_needs_refresh",
+    "spread_weighted_order": "_spread_weighted_order",
+    "MIN_WINDOW_DAYS": "_MIN_WINDOW_DAYS",
+    "MAX_WINDOW_DAYS": "_MAX_WINDOW_DAYS",
+}
+
+
+def _plugins_root():
+    raw = os.environ.get("LEDMATRIX_PLUGINS")
+    if not raw:
+        pytest.skip("set LEDMATRIX_PLUGINS to a ledmatrix-plugins checkout to "
+                    "compare sports_helpers against the plugin copies; in CI "
+                    "this parity check runs in ledmatrix-plugins "
+                    "scripts/check_sports_helpers_parity.py (#495)")
+    root = Path(raw)
+    if (root / "plugins").is_dir():
+        root = root / "plugins"
+    if not (root / "football-scoreboard" / "sports.py").is_file():
+        pytest.skip(f"LEDMATRIX_PLUGINS={raw} has no football-scoreboard/sports.py")
+    return root
+
+
+class _Normalise(ast.NodeTransformer):
+    def visit_Name(self, node):
+        node.id = _RENAMES.get(node.id, node.id)
+        return node
+
+    def _func(self, node):
+        node.name = _RENAMES.get(node.name, node.name)
+        node.decorator_list = []
+        body = node.body
+        if (body and isinstance(body[0], ast.Expr)
+                and isinstance(body[0].value, ast.Constant)
+                and isinstance(body[0].value.value, str)):
+            node.body = body[1:] or [ast.Pass()]
+        self.generic_visit(node)
+        return node
+
+    visit_FunctionDef = _func
+
+
+def _dump(node):
+    node = ast.parse(ast.unparse(node)).body[0]   # detach and copy
+    return ast.dump(_Normalise().visit(node))
+
+
+def _definitions(tree):
+    module, core = {}, {}
+    for node in tree.body:
+        if isinstance(node, ast.FunctionDef):
+            module[node.name] = node
+        elif isinstance(node, ast.Assign) and isinstance(node.targets[0], ast.Name):
+            module[node.targets[0].id] = node
+        elif isinstance(node, ast.ClassDef) and node.name == "SportsCore":
+            for item in node.body:
+                if isinstance(item, ast.FunctionDef):
+                    core[item.name] = item
+                elif isinstance(item, (ast.Assign, ast.AnnAssign)):
+                    target = item.targets[0] if isinstance(item, ast.Assign) else item.target
+                    if isinstance(target, ast.Name):
+                        core[target.id] = item
+    return {"module": module, "SportsCore": core}
+
+
+def _core_definitions():
+    tree = ast.parse(Path(sports_helpers.__file__).read_text(encoding="utf-8"))
+    out = {}
+    for node in tree.body:
+        if isinstance(node, ast.FunctionDef):
+            out[node.name] = node
+        elif isinstance(node, ast.ClassDef) and node.name == "SportsHelpersMixin":
+            for item in node.body:
+                if isinstance(item, ast.FunctionDef):
+                    out[item.name] = item
+    return out
+
+
+class TestParityWithPlugins:
+    @pytest.mark.parametrize("name", sorted(PROMOTED))
+    def test_body_matches_every_plugin_copy(self, name):
+        root = _plugins_root()
+        where, plugin_name, carriers = PROMOTED[name]
+        ours = _dump(_core_definitions()[name])
+        drifted, missing = [], []
+        for sport in carriers:
+            defs = _definitions(ast.parse(
+                (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8")))
+            theirs = defs[where].get(plugin_name)
+            if theirs is None:
+                missing.append(sport)
+            elif _dump(theirs) != ours:
+                drifted.append(sport)
+        assert missing == [], f"{plugin_name} no longer in: {missing}"
+        assert drifted == [], (
+            f"{plugin_name} differs from src/common/sports_helpers.py in: {drifted}. "
+            f"Port the change to both, or stop treating it as shared.")
+
+    @pytest.mark.parametrize("sport", SCOREBOARDS)
+    def test_constants_match(self, sport):
+        root = _plugins_root()
+        defs = _definitions(ast.parse(
+            (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8")))
+        assert ast.literal_eval(defs["module"]["_MIN_WINDOW_DAYS"].value) == MIN_WINDOW_DAYS
+        assert ast.literal_eval(defs["module"]["_MAX_WINDOW_DAYS"].value) == MAX_WINDOW_DAYS
+        gap = defs["SportsCore"]["_DWELL_REENTRY_GAP_SECONDS"].value
+        assert math.isclose(ast.literal_eval(gap), SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS)

test/web_interface/test_update_all_plugins.py

@@ -0,0 +1,104 @@
+"""'Check & Update All' must only send ids POST /plugins/update handles.
+
+Seen on a device running core 3.4.0: update-all posted every entry from
+/plugins/installed, including the virtual `starlark:<app_id>` entries that
+list installed Starlark apps, and the route answered each with a 500
+"Plugin update failed: plugin not found". A web-service restart during the
+same run also cost stock-news its update: its request was refused while the
+service came back, and update-all never sent it again.
+
+The id selection and retry live in install_manager.js and are covered by
+test/js/unit/test_update_all.js, which this module runs so CI sees it. The
+route contract is tested here directly.
+"""
+
+import shutil
+import subprocess
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+REPO = Path(__file__).resolve().parents[2]
+JS_SUITE = REPO / 'test' / 'js' / 'unit' / 'test_update_all.js'
+
+
+@pytest.fixture
+def client():
+    from web_interface.app import app
+    app.config['TESTING'] = True
+    with app.test_client() as c:
+        yield c
+
+
+@pytest.fixture
+def store(tmp_path):
+    """A store manager that records calls; no git, no network."""
+    from web_interface.blueprints.api_v3 import api_v3
+    sm = MagicMock()
+    sm.plugins_dir = str(tmp_path)
+    sm._get_local_git_info.return_value = None
+    sm.get_plugin_info.return_value = None
+    sm.update_plugin.return_value = True
+    with patch.object(api_v3, 'plugin_store_manager', sm, create=True), \
+         patch.object(api_v3, 'plugin_manager', None, create=True), \
+         patch.object(api_v3, 'schema_manager', None, create=True), \
+         patch.object(api_v3, 'plugin_state_manager', None, create=True), \
+         patch.object(api_v3, 'operation_history', None, create=True):
+        yield sm
+
+
+class TestUpdateRouteRejectsStarlarkIds:
+
+    @pytest.mark.parametrize('app_id', ['starlark:analogtime', 'starlark:analogclock'])
+    def test_a_starlark_id_is_a_400_not_a_500(self, client, store, app_id):
+        resp = client.post('/api/v3/plugins/update', json={'plugin_id': app_id})
+        assert resp.status_code == 400, resp.get_json()
+        body = resp.get_json()
+        assert body['status'] == 'error'
+        assert body['error_code'] == 'INVALID_INPUT'
+        assert 'Starlark app' in body['message'], body
+        assert 'not found' not in body['message'], \
+            "the app is installed; 'not found' is the misleading message this replaces"
+
+    def test_the_store_manager_is_never_asked(self, client, store):
+        client.post('/api/v3/plugins/update', json={'plugin_id': 'starlark:analogtime'})
+        store.update_plugin.assert_not_called()
+
+    def test_form_encoded_starlark_id_is_rejected_the_same_way(self, client, store):
+        resp = client.post('/api/v3/plugins/update', data={'plugin_id': 'starlark:analogtime'})
+        assert resp.status_code == 400
+
+    def test_a_plugin_id_still_reaches_the_updater(self, client, store, tmp_path):
+        # The guard is on the 'starlark:' prefix only: the starlark-apps plugin
+        # and a disabled plugin with an update are still updated.
+        for pid in ('stock-news', 'starlark-apps'):
+            (tmp_path / pid).mkdir()
+            (tmp_path / pid / 'manifest.json').write_text('{"id": "%s"}' % pid, encoding='utf-8')
+            resp = client.post('/api/v3/plugins/update', json={'plugin_id': pid})
+            assert resp.status_code == 200, (pid, resp.get_json())
+        assert [c.args[0] for c in store.update_plugin.call_args_list] == ['stock-news', 'starlark-apps']
+
+
+class TestInstalledListContract:
+    """What update-all filters on is what /plugins/installed publishes."""
+
+    def test_starlark_entries_are_flagged_and_prefixed(self, client):
+        apps = {'apps': {'analogtime': {'name': 'Analog Time', 'enabled': False}}}
+        with patch('web_interface.blueprints.api_v3._get_starlark_plugin', return_value=None), \
+             patch('web_interface.blueprints.api_v3._read_starlark_manifest', return_value=apps):
+            resp = client.get('/api/v3/plugins/installed')
+        plugins = resp.get_json()['data']['plugins']
+        entry = next(p for p in plugins if p['id'] == 'starlark:analogtime')
+        assert entry['is_starlark_app'] is True
+        assert not any(p.get('is_starlark_app') for p in plugins
+                       if not p['id'].startswith('starlark:')), \
+            "a real plugin carries the Starlark flag and would be dropped from update-all"
+
+
+@pytest.mark.skipif(shutil.which('node') is None, reason='node is not installed')
+def test_update_all_js_selection_and_retry():
+    node = shutil.which('node')
+    result = subprocess.run([node, str(JS_SUITE)], capture_output=True, text=True,
+                            timeout=120, cwd=str(JS_SUITE.parent))
+    assert result.returncode == 0, result.stdout + result.stderr

test/web_interface/test_x_display_hidden.py

@@ -0,0 +1,577 @@
+"""``"x-display": "hidden"`` keeps a setting declared but off the form.
+
+Plugins keep deprecated keys in their schema so configs that still carry them
+keep validating (weather's ``api_key`` and ``radar_zoom``, olympics'
+notification keys), and some keys are internal (countdown's auto-generated
+row ``id``). Drawn as live controls they invite edits that do nothing, or
+worse, change an internal id.
+
+The contract pinned here:
+
+* nothing hidden is drawn -- top level, nested, in Advanced Settings, or as an
+  array-table column -- and visible siblings still are;
+* an object of nothing but hidden children draws no empty section;
+* a hidden top-level key is not reported in ``__rendered_section`` and does
+  not count toward the Advanced Settings total;
+* saving the rendered form never changes a hidden value: plain and nested ones
+  are simply not posted (the save deep-merges over the stored section), and a
+  hidden array-row property is carried through JSON-encoded, because a posted
+  row replaces the stored item wholesale;
+* an unchecked *visible* checkbox beside them is still cleared;
+* a JSON API save is unaffected.
+"""
+
+import json
+import re
+import sys
+from html.parser import HTMLParser
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import pytest
+from flask import Flask
+from werkzeug.datastructures import MultiDict
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+HIDDEN = {"x-display": "hidden"}
+
+SCHEMA = {
+    "type": "object",
+    "properties": {
+        "enabled": {"type": "boolean", "default": True},
+        "city": {"type": "string", "title": "City", "default": "Dallas"},
+        "legacy_key": {"type": "string", "title": "Legacy Key", **HIDDEN},
+        "legacy_flag": {"type": "boolean", "title": "Legacy Flag", **HIDDEN},
+        "timeout": {"type": "integer", "title": "Timeout", "default": 10,
+                    "x-advanced": True},
+        "old_zoom": {"type": "integer", "title": "Old Zoom", "default": 6,
+                     "minimum": 4, "maximum": 8, "x-advanced": True, **HIDDEN},
+        "display": {
+            "type": "object",
+            "title": "Display",
+            "properties": {
+                "show_clock": {"type": "boolean", "title": "Show Clock",
+                               "default": True},
+                "old_flag": {"type": "boolean", "title": "Old Flag", **HIDDEN},
+                "old_mode": {"type": "string", "title": "Old Mode", **HIDDEN},
+            },
+        },
+        # Every child hidden: no empty section.
+        "internal": {
+            "type": "object",
+            "title": "Internal Stuff",
+            "properties": {
+                "token_seen": {"type": "boolean", **HIDDEN},
+                "revision": {"type": "integer", **HIDDEN},
+            },
+        },
+        "entries": {
+            "type": "array",
+            "title": "Entries",
+            "x-widget": "array-table",
+            # x-columns naming a hidden property must not draw it either.
+            "x-columns": ["enabled", "name", "id"],
+            "items": {
+                "type": "object",
+                "properties": {
+                    "id": {"type": "string", "title": "Row ID", **HIDDEN},
+                    "enabled": {"type": "boolean", "default": True},
+                    "name": {"type": "string"},
+                    "pinned": {"type": "boolean", "title": "Pinned", **HIDDEN},
+                    "weight": {"type": "number", "title": "Weight", **HIDDEN},
+                    "layout": {
+                        "type": "object",
+                        "properties": {
+                            "x": {"type": "integer", "default": 0},
+                            "legacy_x": {"type": "integer", **HIDDEN},
+                        },
+                    },
+                },
+                "required": ["name"],
+            },
+        },
+    },
+}
+
+STORED = {
+    "enabled": True,
+    "city": "Dallas",
+    "legacy_key": "keep-me",
+    "legacy_flag": True,
+    "timeout": 10,
+    "old_zoom": 5,
+    "display": {"show_clock": True, "old_flag": True, "old_mode": "classic"},
+    "internal": {"token_seen": True, "revision": 7},
+    "entries": [
+        # An id that looks like a number must come back a string.
+        {"id": "1", "enabled": True, "name": "First", "pinned": True,
+         "weight": 2.5, "layout": {"x": 3, "legacy_x": 9}},
+        {"id": "cd_ab12cd34ef56", "enabled": False, "name": "Second",
+         "pinned": False},
+    ],
+}
+
+HIDDEN_LABELS = ("Legacy Key", "Legacy Flag", "Old Zoom", "Old Flag",
+                 "Old Mode", "Internal Stuff", "Row ID", "Pinned", "Weight")
+
+
+class _FormFields(HTMLParser):
+    """Collect what a browser would submit from the rendered form."""
+
+    def __init__(self):
+        super().__init__()
+        self.pairs = []
+        self._select = None
+        self._first = None
+        self._chosen = None
+
+    def handle_starttag(self, tag, attrs):
+        a = dict(attrs)
+        name = a.get("name")
+        if tag == "input" and name and "disabled" not in a:
+            if a.get("type") in ("checkbox", "radio") and "checked" not in a:
+                return
+            if a.get("type") in ("button", "submit", "file"):
+                return
+            self.pairs.append((name, a.get("value", "")))
+        elif tag == "select" and name:
+            self._select, self._first, self._chosen = name, None, None
+        elif tag == "option" and self._select:
+            if self._first is None:
+                self._first = a.get("value", "")
+            if "selected" in a:
+                self._chosen = a.get("value", "")
+
+    def handle_endtag(self, tag):
+        if tag == "select" and self._select:
+            chosen = self._chosen if self._chosen is not None else self._first
+            self.pairs.append((self._select, chosen or ""))
+            self._select = None
+
+
+def _render(schema, config, plugin_id="demo"):
+    from jinja2 import Environment, FileSystemLoader, select_autoescape
+
+    env = Environment(
+        loader=FileSystemLoader(str(PROJECT_ROOT / "web_interface" / "templates")),
+        autoescape=select_autoescape(["html"]),
+    )
+    plugin = {"id": plugin_id, "name": "Demo", "description": "",
+              "enabled": True, "author": "me", "version": "1.0.0"}
+    return env.get_template("v3/partials/plugin_config.html").render(
+        plugin=plugin, schema=schema, config=json.loads(json.dumps(config)))
+
+
+def _form_pairs(html):
+    parser = _FormFields()
+    parser.feed(html)
+    return parser.pairs
+
+
+def _names(html):
+    return {name for name, _ in _form_pairs(html)}
+
+
+def _visible_names(html):
+    """Names of controls a user can see (anything but type=hidden)."""
+    names = set()
+    for tag in re.findall(r"<(?:input|select|textarea)\b[^>]*>", html):
+        m = re.search(r'\bname="([^"]+)"', tag)
+        if m and 'type="hidden"' not in tag:
+            names.add(m.group(1))
+    return names
+
+
+def _make_post(tmp_path, schema, stored, plugin_id="demo"):
+    from src.plugin_system.schema_manager import SchemaManager
+    from web_interface.blueprints import api_v3 as a
+
+    originals = {k: getattr(a.api_v3, k, None)
+                 for k in ("config_manager", "schema_manager", "plugin_manager")}
+
+    pdir = tmp_path / "plugins" / plugin_id
+    pdir.mkdir(parents=True)
+    (pdir / "config_schema.json").write_text(json.dumps(schema), encoding="utf-8")
+
+    store = {plugin_id: json.loads(json.dumps(stored))}
+    cm = MagicMock()
+    cm.load_config.side_effect = lambda: json.loads(json.dumps(store))
+    cm.get_raw_file_content.return_value = {}
+    cm.get_config_path.return_value = str(tmp_path / "config.json")
+
+    def _save(cfg, **_kw):
+        store.clear()
+        store.update(cfg)
+        return type("R", (), {"status": type("S", (), {"value": "success"})(),
+                              "message": None})()
+
+    cm.save_config_atomic.side_effect = _save
+
+    a.api_v3.config_manager = cm
+    a.api_v3.schema_manager = SchemaManager(plugins_dir=tmp_path / "plugins",
+                                            project_root=tmp_path)
+    pm = MagicMock()
+    pm.plugins = {}
+    pm.get_plugin.return_value = None
+    a.api_v3.plugin_manager = pm
+
+    app = Flask(__name__)
+    app.config["TESTING"] = True
+    app.register_blueprint(a.api_v3, url_prefix="/api/v3")
+    client = app.test_client()
+
+    def _post(pairs):
+        resp = client.post(f"/api/v3/plugins/config?plugin_id={plugin_id}",
+                           data=MultiDict(pairs))
+        return resp, store.get(plugin_id, {})
+
+    def _post_json(config):
+        resp = client.post("/api/v3/plugins/config",
+                           json={"plugin_id": plugin_id, "config": config})
+        return resp, store.get(plugin_id, {})
+
+    def _restore():
+        for k, v in originals.items():
+            setattr(a.api_v3, k, v)
+
+    return _post, _post_json, _restore
+
+
+@pytest.fixture
+def post(tmp_path):
+    _post, _post_json, restore = _make_post(tmp_path, SCHEMA, STORED)
+    _post.json = _post_json
+    try:
+        yield _post
+    finally:
+        restore()
+
+
+# --------------------------------------------------------------------------
+# Rendering
+# --------------------------------------------------------------------------
+
+class TestRendering:
+    html = None
+
+    @classmethod
+    def setup_class(cls):
+        cls.html = _render(SCHEMA, STORED)
+
+    def test_no_hidden_field_gets_a_visible_control(self):
+        visible = _visible_names(self.html)
+        for name in ("legacy_key", "legacy_flag", "old_zoom",
+                     "display.old_flag", "display.old_mode",
+                     "internal.token_seen", "internal.revision"):
+            assert name not in visible, name
+        assert not [n for n in visible if re.fullmatch(
+            r"entries\.\d+\.(id|pinned|weight|layout\.legacy_x)", n)], visible
+
+    def test_plain_and_nested_hidden_fields_are_not_posted_at_all(self):
+        names = _names(self.html)
+        for name in ("legacy_key", "legacy_flag", "old_zoom",
+                     "display.old_flag", "display.old_mode",
+                     "internal.token_seen", "internal.revision"):
+            assert name not in names, name
+
+    def test_no_hidden_label_is_drawn(self):
+        for label in HIDDEN_LABELS:
+            assert f">{label}<" not in self.html, label
+            assert f"{label}\n" not in self.html, label
+
+    def test_visible_siblings_still_render(self):
+        visible = _visible_names(self.html)
+        assert {"city", "timeout", "display.show_clock",
+                "entries.0.name", "entries.0.enabled"} <= visible
+
+    def test_rendered_sections_exclude_hidden_top_level_keys(self):
+        sections = re.findall(r'name="__rendered_section" value="([^"]+)"',
+                              self.html)
+        assert set(sections) == {"city", "timeout", "display", "entries"}
+
+    def test_an_object_of_only_hidden_children_draws_no_section(self):
+        assert "Internal Stuff" not in self.html
+        assert "section-internal" not in self.html
+
+    def test_the_advanced_count_excludes_hidden_fields(self):
+        assert re.findall(r"Advanced Settings \((\d+)\)", self.html) == ["1"]
+
+    def test_advanced_settings_disappear_when_all_advanced_are_hidden(self):
+        schema = json.loads(json.dumps(SCHEMA))
+        del schema["properties"]["timeout"]
+        html = _render(schema, STORED)
+        assert "Advanced Settings" not in html
+
+    def test_hidden_array_properties_are_not_columns(self):
+        headers = re.findall(r"<th[^>]*>([^<]+)</th>", self.html)
+        assert "Row ID" not in headers and "Pinned" not in headers
+        cols = re.search(r"data-display-columns='([^']*)'", self.html)
+        assert json.loads(cols.group(1)) == ["enabled", "name"]
+
+    def test_hidden_array_properties_stay_out_of_the_row_editor(self):
+        """The editor draws a field for every data-nested-prop and every key
+        of the advanced cell's data-prop-schema."""
+        assert 'data-nested-prop="pinned"' not in self.html
+        assert 'data-nested-prop="layout.legacy_x"' not in self.html
+        assert 'data-nested-prop="layout.x"' in self.html
+        for m in re.finditer(r"class=\"array-table-advanced-data\"\s+"
+                             r"data-prop-schema='([^']*)'", self.html):
+            import html as _h
+            adv = json.loads(_h.unescape(m.group(1)))
+            assert "pinned" not in adv and "weight" not in adv and "id" not in adv
+
+    def test_stored_hidden_row_values_are_carried_json_encoded(self):
+        pairs = dict(_form_pairs(self.html))
+        assert pairs["entries.0.id"] == '"1"'
+        assert pairs["entries.0.pinned"] == "true"
+        assert pairs["entries.0.weight"] == "2.5"
+        assert pairs["entries.0.layout.legacy_x"] == "9"
+        assert pairs["entries.1.id"] == '"cd_ab12cd34ef56"'
+        # Absent stored values are not invented.
+        assert "entries.1.weight" not in pairs
+        assert "entries.1.layout.legacy_x" not in pairs
+
+
+def test_a_schema_without_the_flag_renders_as_before():
+    """Regression guard: the flag's absence changes nothing."""
+    schema = json.loads(json.dumps(SCHEMA))
+
+    def strip(node):
+        if isinstance(node, dict):
+            node.pop("x-display", None)
+            for v in node.values():
+                strip(v)
+    strip(schema)
+    html = _render(schema, STORED)
+    visible = _visible_names(html)
+    assert {"legacy_key", "legacy_flag", "old_zoom", "display.old_flag",
+            "internal.token_seen", "entries.0.id"} <= visible
+    # Not a column, so it lives in the row editor's advanced cell as before.
+    assert 'data-nested-prop="pinned"' in html
+    assert re.findall(r"Advanced Settings \((\d+)\)", html) == ["2"]
+
+
+# --------------------------------------------------------------------------
+# Saving
+# --------------------------------------------------------------------------
+
+class TestSavingTheRenderedForm:
+    def test_the_untouched_form_changes_no_hidden_value(self, post):
+        resp, cfg = post(_form_pairs(_render(SCHEMA, STORED)))
+        assert resp.status_code == 200, resp.get_json()
+        assert cfg["legacy_key"] == "keep-me"
+        assert cfg["legacy_flag"] is True
+        assert cfg["old_zoom"] == 5
+        assert cfg["display"]["old_flag"] is True
+        assert cfg["display"]["old_mode"] == "classic"
+        assert cfg["internal"] == {"token_seen": True, "revision": 7}
+        first, second = cfg["entries"]
+        assert first["id"] == "1" and isinstance(first["id"], str)
+        assert first["pinned"] is True
+        assert first["weight"] == 2.5
+        assert first["layout"]["legacy_x"] == 9
+        assert second["id"] == "cd_ab12cd34ef56"
+        assert second["pinned"] is False
+
+    def test_visible_edits_apply_and_an_unchecked_visible_box_is_cleared(self, post):
+        pairs = [(k, v) for k, v in _form_pairs(_render(SCHEMA, STORED))
+                 if k != "display.show_clock"]
+        pairs = [(k, "Austin" if k == "city" else v) for k, v in pairs]
+        resp, cfg = post(pairs)
+        assert resp.status_code == 200, resp.get_json()
+        assert cfg["city"] == "Austin"
+        # The existing unchecked-checkbox behaviour, in the same section as
+        # hidden booleans that must not be touched.
+        assert cfg["display"]["show_clock"] is False
+        assert cfg["display"]["old_flag"] is True
+        assert cfg["legacy_flag"] is True
+        assert cfg["entries"][0]["pinned"] is True
+
+    def test_a_deleted_row_takes_its_hidden_values_with_it(self, post):
+        """Row 0 removed; the browser renumbers row 1 to index 0."""
+        pairs = []
+        for k, v in _form_pairs(_render(SCHEMA, STORED)):
+            m = re.match(r"entries\.(\d+)\.(.*)", k)
+            if m:
+                if m.group(1) == "0":
+                    continue
+                k = f"entries.0.{m.group(2)}"
+            pairs.append((k, v))
+        resp, cfg = post(pairs)
+        assert resp.status_code == 200, resp.get_json()
+        assert [e["id"] for e in cfg["entries"]] == ["cd_ab12cd34ef56"]
+
+    def test_a_new_row_gets_no_invented_hidden_values(self, post):
+        pairs = _form_pairs(_render(SCHEMA, STORED)) + [
+            ("entries.2.enabled", "true"), ("entries.2.name", "Third")]
+        resp, cfg = post(pairs)
+        assert resp.status_code == 200, resp.get_json()
+        assert cfg["entries"][2]["name"] == "Third"
+        assert "id" not in cfg["entries"][2]
+        assert "pinned" not in cfg["entries"][2]
+
+    def test_a_json_api_save_still_writes_hidden_fields(self, post):
+        """Hidden is a form concern: API callers can still set these."""
+        config = json.loads(json.dumps(STORED))
+        config["legacy_flag"] = False
+        config["display"]["old_mode"] = "modern"
+        config["entries"][0]["id"] = "renamed"
+        resp, cfg = post.json(config)
+        assert resp.status_code == 200, resp.get_json()
+        assert cfg["legacy_flag"] is False
+        assert cfg["display"]["old_mode"] == "modern"
+        assert cfg["entries"][0]["id"] == "renamed"
+
+
+class TestMissingBooleans:
+    """_set_missing_booleans_to_false, directly, at every depth."""
+
+    def test_hidden_booleans_are_out_of_scope_everywhere(self):
+        from web_interface.blueprints.api_v3 import _set_missing_booleans_to_false
+
+        cfg = json.loads(json.dumps(STORED))
+        form_keys = {"city", "entries.0.name", "entries.1.name"}
+        _set_missing_booleans_to_false(
+            cfg, SCHEMA["properties"], form_keys,
+            sections={"city", "display", "entries", "internal", "legacy_flag"})
+
+        assert cfg["legacy_flag"] is True
+        assert cfg["display"]["old_flag"] is True
+        assert cfg["internal"]["token_seen"] is True
+        assert cfg["entries"][0]["pinned"] is True
+        # Visible booleans in the same places are still cleared.
+        assert cfg["display"]["show_clock"] is False
+        assert cfg["entries"][0]["enabled"] is False
+
+    def test_hidden_array_item_property_lookup(self):
+        from web_interface.blueprints.api_v3 import _hidden_array_item_property
+
+        assert _hidden_array_item_property(SCHEMA, "entries.0.id")
+        assert _hidden_array_item_property(SCHEMA, "entries.12.layout.legacy_x")
+        assert _hidden_array_item_property(SCHEMA, "entries.0.name") is None
+        assert _hidden_array_item_property(SCHEMA, "entries.0.layout.x") is None
+        # Only array rows qualify: plain hidden keys are never posted.
+        assert _hidden_array_item_property(SCHEMA, "legacy_key") is None
+        assert _hidden_array_item_property(SCHEMA, "display.old_flag") is None
+
+
+# --------------------------------------------------------------------------
+# Real plugin schemas (snippets copied from ledmatrix-plugins)
+# --------------------------------------------------------------------------
+
+# plugins/countdown/config_schema.json, trimmed to the countdowns table.
+COUNTDOWN_SCHEMA = {
+    "type": "object",
+    "properties": {
+        "enabled": {"type": "boolean", "default": True},
+        "countdowns": {
+            "type": "array",
+            "x-widget": "array-table",
+            "x-columns": ["enabled", "name", "target_date", "target_time", "mode"],
+            "default": [],
+            "items": {
+                "type": "object",
+                "properties": {
+                    "id": {"type": "string", "title": "ID",
+                           "description": "Unique identifier (auto-generated)",
+                           "x-display": "hidden"},
+                    "enabled": {"type": "boolean", "title": "Enabled", "default": True},
+                    "name": {"type": "string", "title": "Name",
+                             "minLength": 1, "maxLength": 30},
+                    "target_date": {"type": "string", "title": "Target Date",
+                                    "x-widget": "date-picker",
+                                    "pattern": "^\\d{4}-\\d{2}-\\d{2}$"},
+                    "target_time": {"type": "string", "title": "Target Time",
+                                    "x-widget": "time-picker", "default": "00:00"},
+                    "mode": {"type": "string", "title": "Mode",
+                             "enum": ["until", "since"], "default": "until"},
+                    "display_order": {"x-advanced": True, "type": "integer",
+                                      "title": "Order", "default": 0, "minimum": 0},
+                },
+                "required": ["name", "target_date"],
+            },
+        },
+        "show_expired": {"x-advanced": True, "type": "boolean",
+                         "title": "Show Expired", "default": False},
+    },
+    "required": ["enabled"],
+}
+
+COUNTDOWN_STORED = {
+    "enabled": True,
+    "show_expired": True,
+    "countdowns": [
+        {"id": "cd_0123456789ab", "enabled": True, "name": "Vacation",
+         "target_date": "2026-12-20", "target_time": "00:00", "mode": "until",
+         "display_order": 0},
+        {"id": "42", "enabled": True, "name": "Anniversary",
+         "target_date": "2020-06-01", "target_time": "00:00", "mode": "since",
+         "display_order": 1},
+    ],
+}
+
+# plugins/ledmatrix-weather/config_schema.json, trimmed.
+WEATHER_SCHEMA = {
+    "type": "object",
+    "properties": {
+        "enabled": {"type": "boolean", "default": False},
+        "location_city": {"type": "string", "default": "Dallas", "title": "City"},
+        "update_interval": {"x-advanced": True, "type": "integer", "default": 1800,
+                            "minimum": 300},
+        "api_key": {"x-advanced": True, "x-display": "hidden", "type": "string",
+                    "title": "API Key (deprecated)"},
+        "radar_range_miles": {"type": "integer", "default": 75, "minimum": 10,
+                              "maximum": 500, "title": "Radar Range (miles)"},
+        "radar_zoom": {"x-advanced": True, "x-display": "hidden", "type": "integer",
+                       "default": 6, "minimum": 4, "maximum": 8,
+                       "title": "Radar Zoom Level (deprecated)"},
+        "radar_show_nowcast": {"type": "boolean", "default": True,
+                               "title": "Show Forecast Frames"},
+    },
+}
+
+WEATHER_STORED = {"enabled": True, "location_city": "Tampa", "update_interval": 900,
+                  "api_key": "legacy-owm-key", "radar_range_miles": 75,
+                  "radar_zoom": 7, "radar_show_nowcast": True}
+
+
+def test_countdown_ids_survive_a_form_save(tmp_path):
+    html = _render(COUNTDOWN_SCHEMA, COUNTDOWN_STORED, "countdown")
+    assert ">ID<" not in html
+    assert "countdowns.0.id" not in _visible_names(html)
+
+    _post, _json, restore = _make_post(tmp_path, COUNTDOWN_SCHEMA,
+                                       COUNTDOWN_STORED, "countdown")
+    try:
+        # Edit a visible cell too.
+        pairs = [(k, "Beach" if k == "countdowns.0.name" else v)
+                 for k, v in _form_pairs(html)]
+        resp, cfg = _post(pairs)
+    finally:
+        restore()
+    assert resp.status_code == 200, resp.get_json()
+    assert [c["id"] for c in cfg["countdowns"]] == ["cd_0123456789ab", "42"]
+    assert cfg["countdowns"][0]["name"] == "Beach"
+
+
+def test_weather_deprecated_keys_are_hidden_and_kept(tmp_path):
+    html = _render(WEATHER_SCHEMA, WEATHER_STORED, "weather")
+    assert "api_key" not in html and "radar_zoom" not in html
+    assert "Radar Zoom Level" not in html and "API Key" not in html
+    assert re.findall(r"Advanced Settings \((\d+)\)", html) == ["1"]
+    assert "radar_range_miles" in _visible_names(html)
+
+    _post, _json, restore = _make_post(tmp_path, WEATHER_SCHEMA,
+                                       WEATHER_STORED, "weather")
+    try:
+        pairs = [(k, v) for k, v in _form_pairs(html)
+                 if k != "radar_show_nowcast"]
+        resp, cfg = _post(pairs)
+    finally:
+        restore()
+    assert resp.status_code == 200, resp.get_json()
+    assert cfg["api_key"] == "legacy-owm-key"
+    assert cfg["radar_zoom"] == 7
+    assert cfg["radar_show_nowcast"] is False

web_interface/blueprints/api_v3/__init__.py

@@ -1039,6 +1039,51 @@ def _set_nested_value(config, key_path, value):
 _RENDERED_SECTION_FIELD = '__rendered_section'
 
 
+def _is_hidden_prop(prop_schema):
+    """Whether a schema property is declared ``"x-display": "hidden"``.
+
+    Mirrors ``prop_is_hidden`` in ``plugin_config.html``: an object whose every
+    child is hidden counts as hidden too. A hidden property stays declared (so
+    stored configs carrying it keep validating) but the form draws no control
+    for it, which puts it outside everything the save infers from the form.
+    """
+    if not isinstance(prop_schema, dict):
+        return False
+    if prop_schema.get('x-display') == 'hidden':
+        return True
+    children = prop_schema.get('properties')
+    if isinstance(children, dict) and children:
+        return all(_is_hidden_prop(child) for child in children.values())
+    return False
+
+
+def _hidden_array_item_property(schema, key_path):
+    """The hidden schema property a posted array-row key names, else None.
+
+    Only paths through an array index (``countdowns.0.id``) qualify: that is
+    the one place the rendered form posts a hidden value, because a posted row
+    replaces the stored item wholesale. The value is JSON-encoded there so its
+    exact type survives the round trip.
+    """
+    parts = key_path.split('.')
+    digit_at = next((i for i, p in enumerate(parts) if p.isdigit()), None)
+    if digit_at is None or digit_at == 0 or digit_at == len(parts) - 1:
+        return None
+    array_prop = _get_schema_property(schema, '.'.join(parts[:digit_at]))
+    if not _schema_type_is(array_prop, 'array'):
+        return None
+    items = array_prop.get('items')
+    if not isinstance(items, dict) or not isinstance(items.get('properties'), dict):
+        return None
+    rest = '.'.join(parts[digit_at + 1:])
+    item_schema = {'properties': items['properties']}
+    prop = _get_schema_property(item_schema, rest)
+    if prop is None:
+        # Deeper array inside the item (rare): keep walking.
+        return _hidden_array_item_property(item_schema, rest)
+    return prop if _is_hidden_prop(prop) else None
+
+
 def _boolean_is_in_scope(full_path, prefix, sections, submitted_parents):
     """Whether a missing checkbox at ``full_path`` may be forced to False.
 
@@ -1112,6 +1157,12 @@ def _set_missing_booleans_to_false(plugin_config, schema_props, form_keys, prefi
         if not isinstance(prop_schema, dict):
             continue
 
+        # A hidden property (or an object of nothing but hidden ones) was never
+        # drawn, so its absence from the form is not an unchecked box. Leave it
+        # -- and everything under it -- exactly as stored.
+        if _is_hidden_prop(prop_schema):
+            continue
+
         full_path = f"{prefix}.{prop_name}" if prefix else prop_name
         prop_type = prop_schema.get('type')
 

web_interface/blueprints/api_v3/plugins.py

@@ -8,7 +8,7 @@ from web_interface.blueprints.api_v3 import (
     _CALENDAR_LIST_MAX_PAGES, _RENDERED_SECTION_FIELD, _SKIP_FIELD, _coerce_to_bool,
     _do_transactional_uninstall, _enhance_schema_with_core_properties,
     _filter_config_by_schema, _get_plugin_version, _get_schema_property,
-    _installed_plugin_ids, _is_plugin_update_available,
+    _hidden_array_item_property, _installed_plugin_ids, _is_plugin_update_available,
     _parse_form_value_with_schema, _prune_credential_backups,
     _run_calendar_registration, _schema_allows_null, _schema_type_is,
     _set_missing_booleans_to_false,
@@ -735,10 +735,10 @@ def _drop_stale_reconciliation_findings(unresolved):
         )
 
         cm = api_v3.config_manager
-        config_keys = config_plugin_ids(cm.load_config() or {},
-                                       ignored_config_keys(cm))
         plugins_dir = getattr(api_v3.plugin_manager, 'plugins_dir', None)
         installed = disk_plugin_ids(plugins_dir) if plugins_dir else set()
+        config_keys = config_plugin_ids(cm.load_config() or {},
+                                       ignored_config_keys(cm, installed))
         return still_unresolved(unresolved, config_keys, installed)
     except Exception:
         logger.debug("[Reconciliation] Could not re-check stored findings", exc_info=True)
@@ -943,6 +943,19 @@ def update_plugin():
                 )
             data = {'plugin_id': plugin_id}
 
+        # /plugins/installed lists installed Starlark apps as virtual
+        # 'starlark:<app_id>' entries. They are not plugin directories, so the
+        # store manager can only fail to find them -- which used to surface as
+        # a 500 "plugin not found" for an app that is installed and working.
+        raw_id = data.get('plugin_id')
+        if isinstance(raw_id, str) and raw_id.startswith('starlark:'):
+            return error_response(
+                ErrorCode.INVALID_INPUT,
+                f'{raw_id} is a Starlark app, not a plugin; Starlark apps are '
+                'not updated through the plugin updater',
+                status_code=400
+            )
+
         if not api_v3.plugin_store_manager:
             return error_response(
                 ErrorCode.SYSTEM_ERROR,
@@ -1848,7 +1861,21 @@ def save_plugin_config():
                     # (to avoid overwriting the combined array with a single value)
                     if key not in indexed_base_paths:
                         # Parse value using schema to determine correct type
-                        parsed_value = _parse_form_value_with_schema(value, key, schema)
+                        parsed_value = _SKIP_FIELD
+                        decoded = False
+                        # A hidden property inside an array row is carried
+                        # through the form JSON-encoded (a posted row replaces
+                        # the stored item, so it must be posted at all). Decode
+                        # it exactly: the generic parse would turn an id "1"
+                        # into the integer 1 and fail validation.
+                        if _hidden_array_item_property(schema, key) is not None:
+                            try:
+                                parsed_value = json.loads(value)
+                                decoded = True
+                            except (TypeError, ValueError):
+                                pass
+                        if not decoded:
+                            parsed_value = _parse_form_value_with_schema(value, key, schema)
                         # Debug logging for array fields
                         if schema:
                             prop = _get_schema_property(schema, key)

web_interface/static/v3/js/app-shell.js

@@ -2421,7 +2421,10 @@
 
         async function updateAllPlugins() {
             try {
-                const plugins = Array.isArray(window.installedPlugins) ? window.installedPlugins : [];
+                // Starlark apps are listed as virtual 'starlark:<id>' entries;
+                // /plugins/update does not handle them (see install_manager.js).
+                const plugins = (Array.isArray(window.installedPlugins) ? window.installedPlugins : [])
+                    .filter(p => p && typeof p.id === 'string' && !p.is_starlark_app && !p.id.startsWith('starlark:'));
 
                 if (!plugins.length) {
                     showNotification('No installed plugins to update.', 'warning');

web_interface/static/v3/js/plugins/install_manager.js

@@ -78,33 +78,87 @@ const PluginInstallManager = {
         }
     },
     
+    /**
+     * Whether POST /plugins/update can update this installed-list entry.
+     *
+     * /plugins/installed also lists installed Starlark apps as virtual
+     * `starlark:<app_id>` entries (flagged `is_starlark_app`) so they can be
+     * seen and toggled with everything else. They are not plugin directories:
+     * the plugin updater has nothing to update for them, and there is no
+     * Starlark update route (reinstalling from the repository would reset the
+     * app's saved settings), so they are left out of update-all.
+     *
+     * @param {Object} plugin - Entry from /plugins/installed
+     * @returns {boolean}
+     */
+    isUpdatablePlugin(plugin) {
+        if (!plugin || typeof plugin.id !== 'string' || !plugin.id) return false;
+        return !plugin.is_starlark_app && !plugin.id.startsWith('starlark:');
+    },
+
+    /**
+     * The entries update-all sends to POST /plugins/update, in list order.
+     *
+     * @param {Array} plugins - Entries from /plugins/installed
+     * @returns {Array}
+     */
+    updatablePlugins(plugins) {
+        return (Array.isArray(plugins) ? plugins : []).filter(p => this.isUpdatablePlugin(p));
+    },
+
+    /**
+     * Backoff (ms) before re-sending an update whose request never got an
+     * answer. Covers a web-service restart (~3s on a Pi) with room to spare.
+     */
+    NETWORK_RETRY_DELAYS_MS: [1000, 2000, 4000, 8000, 15000],
+
     /**
      * Update all plugins.
      *
      * @param {Function} onProgress - Optional callback(index, total, pluginId) for progress updates
-     * @returns {Promise<Array>} Update results
+     * @param {Object} options - Optional { sleep(ms), retryDelaysMs } (tests inject these)
+     * @returns {Promise<Array>} Update results, one per plugin sent
      */
-    async updateAll(onProgress) {
+    async updateAll(onProgress, options = {}) {
         // Prefer PluginStateManager if populated, fall back to window.installedPlugins
         // (plugins_manager.js populates window.installedPlugins independently)
         const stateManagerPlugins = window.PluginStateManager && window.PluginStateManager.installedPlugins;
-        const plugins = (stateManagerPlugins && stateManagerPlugins.length > 0)
+        const listed = (stateManagerPlugins && stateManagerPlugins.length > 0)
             ? stateManagerPlugins
             : (window.installedPlugins || []);
+        // Snapshot: the list can be replaced while this loop is awaiting.
+        const plugins = this.updatablePlugins(listed);
 
         if (!plugins.length) {
             return [];
         }
+        const sleep = options.sleep || (ms => new Promise(resolve => setTimeout(resolve, ms)));
+        const retryDelays = options.retryDelaysMs || this.NETWORK_RETRY_DELAYS_MS;
         const results = [];
 
         for (let i = 0; i < plugins.length; i++) {
             const plugin = plugins[i];
             if (onProgress) onProgress(i + 1, plugins.length, plugin.id);
-            try {
-                const result = await window.PluginAPI.updatePlugin(plugin.id);
-                results.push({ pluginId: plugin.id, success: true, result });
-            } catch (error) {
-                results.push({ pluginId: plugin.id, success: false, error });
+            // Each plugin gets its own pass over the backoff schedule.
+            const pendingDelays = retryDelays.slice();
+            for (;;) {
+                try {
+                    const result = await window.PluginAPI.updatePlugin(plugin.id);
+                    results.push({ pluginId: plugin.id, success: true, result });
+                    break;
+                } catch (error) {
+                    // No HTTP answer at all (connection refused/reset, e.g. the
+                    // web service restarting mid-run): the server never saw or
+                    // never finished this plugin, so send it again once it is
+                    // back rather than skipping it. An HTTP error response is
+                    // the server's answer and is not retried.
+                    if (error && error.error_code === 'NETWORK_ERROR' && pendingDelays.length > 0) {
+                        await sleep(pendingDelays.shift());
+                        continue;
+                    }
+                    results.push({ pluginId: plugin.id, success: false, error });
+                    break;
+                }
             }
         }
 

web_interface/static/v3/js/widgets/array-table.js

@@ -111,6 +111,37 @@
 
     // ─── Helpers ────────────────────────────────────────────────────────────
 
+    /**
+     * "x-display": "hidden" — the property stays in the schema but gets no
+     * control. An object whose every child is hidden counts as hidden too.
+     * Mirrors prop_is_hidden in plugin_config.html and _is_hidden_prop on the
+     * server.
+     */
+    function isHiddenProp(schema) {
+        if (!schema || typeof schema !== 'object') return false;
+        if (schema['x-display'] === 'hidden') return true;
+        const children = schema.properties;
+        if (children && typeof children === 'object') {
+            const values = Object.values(children);
+            return values.length > 0 && values.every(isHiddenProp);
+        }
+        return false;
+    }
+
+    /**
+     * A hidden input carrying a hidden property's stored value, JSON-encoded so
+     * the save restores its exact type. A posted row replaces the stored item,
+     * so a value that isn't posted is lost.
+     */
+    function hiddenValueInput(name, dataKey, value) {
+        const input = document.createElement('input');
+        input.type  = 'hidden';
+        input.name  = name;
+        input.value = JSON.stringify(value);
+        input.dataset.hiddenProp = dataKey;
+        return input;
+    }
+
     function safeSetHTML(target, html) {
         target.textContent = '';
         // createContextualFragment parses html relative to the document context
@@ -343,6 +374,16 @@
             if (propType === 'object' && propSchema.properties) {
                 const nestedVal = (item && item[propName]) || {};
                 Object.entries(propSchema.properties).forEach(([subName, subSchema]) => {
+                    if (isHiddenProp(subSchema)) {
+                        // No data-nested-prop, so the row editor never shows it.
+                        const stored = Object.entries(nestedVal).find(([name]) => name === subName);
+                        if (stored && stored[1] !== undefined && stored[1] !== null) {
+                            cell.appendChild(hiddenValueInput(
+                                `${fullKey}.${index}.${propName}.${subName}`,
+                                `${propName}.${subName}`, stored[1]));
+                        }
+                        return;
+                    }
                     const subType = Array.isArray(subSchema.type)
                         ? subSchema.type.find(t => t !== 'null') || 'string'
                         : (subSchema.type || 'string');
@@ -386,7 +427,16 @@
         row.className = 'array-table-row';
         row.setAttribute('data-index', index);
 
+        // "x-display": "hidden" item properties: never a column (even if
+        // x-columns names one), never in the advanced cell or row editor.
+        const hiddenKeys = new Set(
+            Object.entries(fullItemProperties)
+                .filter(([, propSchema]) => isHiddenProp(propSchema))
+                .map(([propName]) => propName)
+        );
+
         // Visible column cells
+        displayColumns = displayColumns.filter(colName => !hiddenKeys.has(colName));
         displayColumns.forEach(colName => {
             const colDef   = itemProperties[colName] || {};
             const colType  = Array.isArray(colDef.type) ? colDef.type.find(t => t !== 'null') || 'string' : (colDef.type || 'string');
@@ -396,10 +446,11 @@
             row.appendChild(createCell(fullKey, index, colName, colDef, colValue, pluginId));
         });
 
-        // Determine non-displayed properties (these go into the advanced cell + edit modal)
+        // Determine non-displayed properties (these go into the advanced cell + edit modal).
+        // Hidden ones ("x-display": "hidden") get no column and no editor field.
         const nonDisplayed = {};
         Object.keys(fullItemProperties).forEach(k => {
-            if (!displayColumns.includes(k) && k !== 'id') {
+            if (!displayColumns.includes(k) && k !== 'id' && !hiddenKeys.has(k)) {
                 nonDisplayed[k] = fullItemProperties[k];
             }
         });
@@ -430,6 +481,14 @@
             actionsCell.appendChild(editBtn);
         }
 
+        // Hidden properties: carry stored values only. A new row has none, so
+        // nothing is invented (countdown fills in its own id).
+        Object.entries(item).forEach(([propName, stored]) => {
+            if (hiddenKeys.has(propName) && stored !== undefined && stored !== null) {
+                actionsCell.appendChild(hiddenValueInput(`${fullKey}.${index}.${propName}`, propName, stored));
+            }
+        });
+
         row.appendChild(actionsCell);
 
         // Hidden advanced data cell
@@ -494,7 +553,8 @@
                 grid.className = 'grid grid-cols-2 gap-3';
 
                 Object.entries(propSchema.properties).forEach(([subName, subSchema]) => {
-                    const subType    = Array.isArray(subSchema.type) ? subSchema.type.find(t => t !== 'null') || 'string' : (subSchema.type || 'string');
+                    if (isHiddenProp(subSchema)) return;
+                    const subType    =Array.isArray(subSchema.type) ? subSchema.type.find(t => t !== 'null') || 'string' : (subSchema.type || 'string');
                     const subLabel   = subSchema.title || subName.replace(/_/g, ' ').replace(/\b\w/g, c => c.toUpperCase());
                     const subDesc    = subSchema.description || '';
                     const nestedPath = `${propName}.${subName}`;

web_interface/static/v3/plugins_manager.js

@@ -2409,6 +2409,19 @@ function getSchemaProperty(schema, path) {
     return null;
 }
 
+// "x-display": "hidden" (or an object of nothing but hidden children).
+// Mirrors prop_is_hidden in plugin_config.html.
+function isHiddenSchemaProp(schema) {
+    if (!schema || typeof schema !== 'object') return false;
+    if (schema['x-display'] === 'hidden') return true;
+    const children = schema.properties;
+    if (children && typeof children === 'object') {
+        const values = Object.values(children);
+        return values.length > 0 && values.every(isHiddenSchemaProp);
+    }
+    return false;
+}
+
 // Helper function to render a single item in an array of objects
 function renderArrayObjectItem(fieldId, fullKey, itemProperties, itemValue, index, itemsSchema) {
     const item = itemValue || {};
@@ -2422,6 +2435,9 @@ function renderArrayObjectItem(fieldId, fullKey, itemProperties, itemValue, inde
     const propertyOrder = itemsSchema['x-propertyOrder'] || Object.keys(itemProperties);
     propertyOrder.forEach(propKey => {
         if (!itemProperties[propKey]) return;
+        // "x-display": "hidden": no control. The stored value survives through
+        // data-item-data, which updateArrayObjectData overlays edits onto.
+        if (isHiddenSchemaProp(itemProperties[propKey])) return;
 
         const propSchema = itemProperties[propKey];
         const propValue = item[propKey] !== undefined ? item[propKey] : propSchema.default;
@@ -5478,6 +5494,7 @@ if (typeof window !== 'undefined') {
             itemHtml = `<div class="border border-gray-300 rounded-lg p-4 bg-gray-50 array-object-item" data-index="${newIndex}">`;
             Object.keys(itemsSchema.properties || {}).forEach(propKey => {
                 const propSchema = itemsSchema.properties[propKey];
+                if (propSchema && propSchema['x-display'] === 'hidden') return;
                 const propValue = newItem[propKey] !== undefined ? newItem[propKey] : propSchema.default;
                 const propLabel = propSchema.title || propKey.replace(/_/g, ' ').replace(/\b\w/g, l => l.toUpperCase());
                 itemHtml += `<div class="mb-3"><label class="block text-sm font-medium text-gray-700 mb-1">${escapeHtml(propLabel)}</label>`;

web_interface/templates/v3/partials/plugin_config.html

@@ -6,7 +6,29 @@
 {# ===== MACROS FOR FORM FIELD GENERATION ===== #}
 
 {# Render a single form field based on schema type #}
+{# "x-display": "hidden" keeps a property declared -- so stored configs that
+   still carry it keep validating -- without drawing a control for it
+   (deprecated keys, internal ids). An object whose every child is hidden
+   counts as hidden too, so it doesn't render as an empty section. Outputs
+   "1" when hidden, nothing otherwise; test it with `|trim`.
+   Nothing hidden is posted for a plain or nested field: the save deep-merges
+   the form over the stored section, so the stored value stays. Array rows are
+   the exception (a posted list replaces the stored one), see array-table. #}
+{% macro prop_is_hidden(prop) -%}
+    {%- if prop is mapping -%}
+        {%- if prop.get('x-display') == 'hidden' -%}1
+        {%- elif prop.get('properties') is mapping and prop.get('properties') -%}
+            {%- set _hid = namespace(all=true) -%}
+            {%- for _child in prop.get('properties').values() -%}
+                {%- if not prop_is_hidden(_child)|trim -%}{%- set _hid.all = false -%}{%- endif -%}
+            {%- endfor -%}
+            {%- if _hid.all -%}1{%- endif -%}
+        {%- endif -%}
+    {%- endif -%}
+{%- endmacro %}
+
 {% macro render_field(key, prop, value, prefix='', plugin_id='') %}
+    {% if not prop_is_hidden(prop)|trim %}
     {% set full_key = (prefix ~ '.' ~ key) if prefix else key %}
     {% set field_id = (plugin_id ~ '-' ~ full_key)|replace('.', '-')|replace('_', '-') %}
     {% set label = prop.title if prop.title else key|replace('_', ' ')|title %}
@@ -660,14 +682,20 @@
                     {% set array_value = value if value is not none and value is iterable and value is not string else (prop.default if prop.default is defined and prop.default is iterable and prop.default is not string else []) %}
 
                     {# Use x-columns if specified, otherwise auto-detect first 4 simple properties #}
+                    {# Hidden item properties never become columns (see prop_is_hidden). #}
                     {% set x_columns = prop.get('x-columns') %}
                     {% if x_columns %}
-                        {% set display_columns = x_columns %}
+                        {% set display_columns = [] %}
+                        {% for col_name in x_columns %}
+                            {% if not prop_is_hidden(item_properties.get(col_name))|trim %}
+                                {% set _ = display_columns.append(col_name) %}
+                            {% endif %}
+                        {% endfor %}
                     {% else %}
                         {% set display_columns = [] %}
                         {% for col_name in item_properties.keys() %}
                             {% set col_def = item_properties[col_name] %}
-                            {% if col_def.get('type') not in ['object', 'array'] and display_columns|length < 4 %}
+                            {% if col_def.get('type') not in ['object', 'array'] and display_columns|length < 4 and not prop_is_hidden(col_def)|trim %}
                                 {% set _ = display_columns.append(col_name) %}
                             {% endif %}
                         {% endfor %}
@@ -814,8 +842,16 @@
 
                                     {# Actions cell: delete + optional edit button for advanced props #}
                                     {% set has_advanced = namespace(value=false) %}
-                                    {% for k in item_properties.keys() %}{% if k not in display_columns and k != 'id' %}{% set has_advanced.value = true %}{% endif %}{% endfor %}
+                                    {% for k, v in item_properties.items() %}{% if k not in display_columns and k != 'id' and not prop_is_hidden(v)|trim %}{% set has_advanced.value = true %}{% endif %}{% endfor %}
                                     <td class="px-3 py-3 whitespace-nowrap text-center" style="min-width:90px;vertical-align:middle">
+                                        {# Hidden item properties (e.g. countdown's auto-generated id) get
+                                           no control, but a posted row replaces the stored item wholesale,
+                                           so carry the stored value through. JSON-encoded so the save can
+                                           restore its exact type (an id "1" must stay a string); absent
+                                           values are not invented. #}
+                                        {% for k, v in item_properties.items() %}{% if prop_is_hidden(v)|trim and k in item and item[k] is not none %}
+                                        <input type="hidden" name="{{ full_key }}.{{ item_index }}.{{ k }}" data-hidden-prop="{{ k }}" value='{{ item[k]|tojson }}'>
+                                        {% endif %}{% endfor %}
                                         <button type="button"
                                                 onclick="removeArrayTableRow(this)"
                                                 aria-label="Remove row {{ item_index + 1 }}"
@@ -836,13 +872,23 @@
                                     {# Hidden cell: flat hidden inputs for non-displayed props (layout, style, etc.) #}
                                     {% if has_advanced.value %}
                                     {% set adv_schema = namespace(d={}) %}
-                                    {% for k, v in item_properties.items() %}{% if k not in display_columns and k != 'id' %}{% set _ = adv_schema.d.update({k: v}) %}{% endif %}{% endfor %}
+                                    {% for k, v in item_properties.items() %}{% if k not in display_columns and k != 'id' and not prop_is_hidden(v)|trim %}{% set _ = adv_schema.d.update({k: v}) %}{% endif %}{% endfor %}
                                     <td style="display:none" class="array-table-advanced-data"
                                         data-prop-schema='{{ adv_schema.d|tojson }}'>
                                         {% for prop_name, prop_schema in adv_schema.d.items() %}
                                             {% set prop_type = prop_schema.get('type', 'string') %}
                                             {% if prop_type == 'object' and prop_schema.get('properties') %}
                                                 {% for sub_name, sub_schema in prop_schema.get('properties', {}).items() %}
+                                                    {% set _sub_obj = item.get(prop_name) if item.get(prop_name) is mapping else {} %}
+                                                    {% if prop_is_hidden(sub_schema)|trim %}
+                                                    {# Kept out of the row editor (no data-nested-prop); stored value only. #}
+                                                    {% if sub_name in _sub_obj and _sub_obj[sub_name] is not none %}
+                                                    <input type="hidden"
+                                                           name="{{ full_key }}.{{ item_index }}.{{ prop_name }}.{{ sub_name }}"
+                                                           data-hidden-prop="{{ prop_name }}.{{ sub_name }}"
+                                                           value='{{ _sub_obj[sub_name]|tojson }}'>
+                                                    {% endif %}
+                                                    {% else %}
                                                     {% set sub_val = item.get(prop_name, {}).get(sub_name) %}
                                                     {% set sub_default = sub_schema.get('default') %}
                                                     {% set final_val = sub_val if sub_val is not none else sub_default %}
@@ -852,6 +898,7 @@
                                                            data-prop-type="{{ sub_schema.get('type', 'string') }}"
                                                            data-prop-schema='{{ sub_schema|tojson }}'
                                                            value="{{ final_val if final_val is not none else '' }}">
+                                                    {% endif %}
                                                 {% endfor %}
                                             {% else %}
                                                 {% set prop_val = item.get(prop_name) %}
@@ -1104,6 +1151,7 @@
             {% endif %}
         </div>
     {% endif %}
+    {% endif %}{# not prop_is_hidden #}
 {% endmacro %}
 
 {# Render a nested/collapsible section for object types #}
@@ -1118,7 +1166,11 @@
        TypeError that takes down the whole config page. A legacy boolean was
        the on/off switch the object's `enabled` now holds, so carry it over --
        rendering {} would draw that checkbox unchecked and the next save would
-       quietly turn the feature off. The save then writes the object. #}
+       quietly turn the feature off. The save then writes the object.
+       The plugin loader applies the same rule before validating
+       (legacy_bool_as_object in src/plugin_system/schema_manager.py), and
+       test/test_legacy_boolean_config.py renders this macro against it:
+       change the two together. #}
     {% if value is mapping %}
         {% set nested_value = value %}
     {% elif value is boolean and 'enabled' in (prop.properties or {}) %}
@@ -1142,7 +1194,7 @@
         <div id="{{ section_id }}" class="nested-content bg-gray-50 px-4 py-4 space-y-3 hidden" style="display: none;">
             {% set property_order = prop['x-propertyOrder'] if 'x-propertyOrder' in prop else prop.properties.keys()|list %}
             {% for nested_key in property_order %}
-                {% if nested_key in prop.properties %}
+                {% if nested_key in prop.properties and not prop_is_hidden(prop.properties[nested_key])|trim %}
                     {% set nested_prop = prop.properties[nested_key] %}
                     {% set nested_val = nested_value[nested_key] if nested_key in nested_value else none %}
                     {# Tagged so a widget that takes over part of a block can
@@ -1295,7 +1347,10 @@
                            them. Schemas without the flag render exactly as before. #}
                         {% set tiers = namespace(basic=[], advanced=[]) %}
                         {% for key in property_order %}
-                            {% if key in schema.properties and key != 'enabled' %}
+                            {# A hidden key is neither drawn nor reported in
+                               __rendered_section, so the save never reads its
+                               absence as an unchecked box. #}
+                            {% if key in schema.properties and key != 'enabled' and not prop_is_hidden(schema.properties[key])|trim %}
                                 {% set prop = schema.properties[key] %}
                                 {% set is_object = prop.type is defined and 'object' in prop.type %}
                                 {% if prop.get('x-advanced') and not is_object %}