diff --git a/CHANGELOG.md b/CHANGELOG.md index db5ea778..ab90b8e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,81 @@ release that ships it. accepts both, but the store flags the old spelling as deprecated (`store_manager.py`) and only the new one is in `schema/manifest_schema.json`. +## 3.3.0 + +**The release the sports scoreboards floor on to delete their bundled copies.** +3.2.0 shipped the unified sports library and made `ledmatrix_min_version` +enforceable; this ships the last three shared modules and completes the store +gate, so a scoreboard can now floor here and carry no fallback at all. + +New modules a plugin may import via `src.*` and floor on 3.3.0 for: + +- `src/common/sports_card.py` — settings, colour, font and date helpers for a + scoreboard's `game_renderer.py`. Free functions taking `config`/`fonts` + explicitly, so nothing about the caller's class is assumed. +- `src/common/sports_game_renderer.py` — `SportsGameRendererMixin`: scroll/Vegas + card geometry (centre gap, logo slot and cache key, layout offsets, the + upcoming-card date and time layout). No `__init__` and no state, so adoption + is one line on the class statement. +- `src/common/sports_shared.py` — `SportsCoreSharedMixin`, + `SportsLiveSharedMixin`, `SportsRecentSharedMixin`: the `sports.py` bodies + byte-identical in all eight lineage-sharing scoreboards. + +All three sit under `src/common/` rather than `src/base_classes/sports/`, +deliberately: importing that package pulls `core.py` → `DisplayManager` → +`rgbmatrix`, and these are pure logic. Plugins importing them must not acquire a +hardware dependency. + +Three notes for anyone adopting `sports_shared`: + +- `SportsRecentSharedMixin` defines `__init__`. Its bare `super()` binds to the + mixin, so it reaches the host only when the mixin is listed **first** in the + bases. Reversing that order silently skips the host constructor. +- Methods that resolve the plugin's `config_schema.json` use `_plugin_dir()`, + which walks the MRO rather than reading `__file__` — `__file__` is now + `src/common/`. It walks because `SportsCore` is an ABC: a subclass built with + `type(name, bases, ns)` reports `__module__` as `"abc"`. +- `_get_timezone`, `_extract_game_details` and `_fetch_data` are byte-identical + across the eight but stay in the plugins. The first binds a per-plugin + timezone module whose contents differ; the other two are the abstract stubs + that define the sport. + +**The store's compatibility gate is now on every registry-managed route.** 3.2.0 +gated `install_plugin`. This release gates the git-pull update path and +`install_from_url`, so a plugin whose floor the core cannot meet is refused +after download with no partial directory left behind. + +### Fixed + +- **ESPN 403s.** `site.api` began rejecting the User-Agent strings this repo + sent on 2026-08-04; every shared-data-source scoreboard returned + `403 Forbidden`. Requests now send an identifying token with a project URL — + browser-style strings and bare custom tokens are both refused. +- **Low-memory boards becoming unreachable under load** while still answering + pings and serving the web UI. Fetched payloads are released after delivery + rather than pinned on the completed request for up to an hour, malloc arenas + are capped, and log volume and SD writes are reduced. Available memory is now + reported in Tools diagnostics. +- **A failed logo download pinning a team to a grey box**: the placeholder was + written under the real logo's filename, so later attempts found a file and + reported success without retrying. +- **A plugin enabled but never loaded is retried** rather than staying absent + with `error = null`. +- `ttl` now controls cache expiry; abandoned cache writes no longer leave temp + files; one cache-cleanup thread per directory rather than per manager. +- Odds are fetched for the games displayed, not the whole schedule window, and a + stalled ESPN no longer stalls the whole plugin update. +- Array-item secrets are no longer wiped or logged. +- A restored backup matches the device it was taken from. +- The web UI reports the real error instead of "unknown", rejects non-finite + JSON numbers, and stops checkbox groups posting back hidden options. + +### Added + +- `display.hardware.orientation` for panels mounted upside down. +- Vegas keeps live content in the ticker rather than being preempted by it. +- Schemas can label enum dropdown options. + ## 3.2.0 **The first release shipping the unified sports library.** This is the version diff --git a/README.md b/README.md index 681176ef..2d61b614 100644 --- a/README.md +++ b/README.md @@ -161,9 +161,11 @@ The system supports live, recent, and upcoming game information for multiple spo ### LED Matrix Panels (2x in a horizontal chain is recommended) - [Adafruit 64×32](https://www.adafruit.com/product/2278) – designed for 128×32 but works with dynamic scaling on many displays (pixel pitch is user preference) +**Warning: Lately the Waveshare Panels have had different variations - only some are compatible with this project. I hope to identify what is different to fix it but so far there is a decent chance you get a mis-matched set of panels if you don't buy them all at once! ** - [Waveshare 64×32](https://amzn.to/3Kw55jK) - Does not require E addressable pad -- [Waveshare 96×48](https://amzn.to/4bydNcv) – higher resolution, requires soldering the **E addressable pad** on the [Adafruit RGB Bonnet](https://www.adafruit.com/product/3211) to “8” **OR** toggling the DIP switch on the Adafruit Triple LED Matrix Bonnet *(no soldering required!)* - > Amazon Affiliate Link – ChuckBuilds receives a small commission on purchases +- [Waveshare 96×48](https://amzn.to/4bydNcv) – higher resolution, requires soldering the **E addressable pad** on the [Adafruit RGB Bonnet](https://www.adafruit.com/product/3211) to “8” **OR** toggling the DIP switch on the Adafruit Triple LED Matrix Bonnet *(no soldering required!)* +- There are some Panels on Aliexpress that have worked fine for me, shop around! I think Adafruit is probably the "safest" but they do have some limitation on resolution and layout. + > Amazon Affiliate Links – ChuckBuilds receives a small commission on purchases ### Power Supply - [5V 4A DC Power Supply](https://www.adafruit.com/product/658) (good for 2 -3 displays, depending on brightness and pixel density, you'll need higher amperage for more) diff --git a/assets/sports/mlb_logos/mlb.png b/assets/sports/mlb_logos/mlb.png deleted file mode 100644 index 7196811c..00000000 Binary files a/assets/sports/mlb_logos/mlb.png and /dev/null differ diff --git a/assets/sports/nba_logos/nba.png b/assets/sports/nba_logos/nba.png deleted file mode 100644 index 6738f818..00000000 Binary files a/assets/sports/nba_logos/nba.png and /dev/null differ diff --git a/assets/sports/nfl_logos/nfl.png b/assets/sports/nfl_logos/nfl.png deleted file mode 100644 index 49f71d91..00000000 Binary files a/assets/sports/nfl_logos/nfl.png and /dev/null differ diff --git a/assets/sports/nhl_logos/nhl.png b/assets/sports/nhl_logos/nhl.png deleted file mode 100644 index eaebd594..00000000 Binary files a/assets/sports/nhl_logos/nhl.png and /dev/null differ diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index 4e731278..d783e15b 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -27,7 +27,7 @@ These are independent concerns. Conflating them is what produces god classes. | Plugin loads on a core that predates a module | Guarded import with a bundled fallback (`try: from src.X import Y / except ModuleNotFoundError: from y import Y`) | | Plugin loads on a core that predates a *method* | Capability probing — `hasattr(SportsCore, "_detect_stale_games")` — never a version comparison. The loader's compat check is advisory-only (it logs and continues), so probing is the real protection. | | Core changes never break a plugin's rendering | The **view-model contract**: `_extract_game_details_common` returns a dict whose `GUARANTEED_KEYS` are frozen by `test/test_skin_system.py::TestViewModelContract`. Keys may be added, never renamed or removed. | -| A plugin can drop its bundled copy safely | The **sunset rule**: its manifest must floor `ledmatrix_min_version` at the first core release shipping the module (recorded in `CHANGELOG.md`) — *necessary but not sufficient*. Nothing enforces that floor today, so the copy also waits for the B6 gate below. | +| A plugin can drop its bundled copy safely | The **sunset rule**: its manifest must floor `ledmatrix_min_version` at the first core release shipping the module (recorded in `CHANGELOG.md`) — *necessary but not sufficient*. The store enforces that floor on every registry-managed install and on both supported update paths (sideloading via `install_from_url` is not gated), but a floor cannot reach a user who never updates, so the copy also waits for the B6 gate below. | The core API is **additive-only**. A method the plugins call is never removed or given a new required parameter; new behavior arrives as new methods with @@ -204,7 +204,7 @@ legacy compatibility rather than the mechanism. B0–B3 are merged and shipping in core 3.2.0. Everything that remains is **rollout**, and it splits into three phases with very different risk profiles. The original plan folded the last two together; they are separated here because -one of them is safe by construction and the other is not. +one of them cannot break a user on an old core and the other can. | Phase | Scope | Status | Gate | |---|---|---|---| @@ -212,9 +212,9 @@ one of them is safe by construction and the other is not. | **B1** | Promote the nine universal methods; convert `sports.py` → package | ✅ | Characterization suite green; no behavior change intended | | **B2** | `CelebrationMixin` + rotation strategies as opt-in capabilities | ✅ | Non-adopters have zero new code in their MRO; strategies checked against verbatim plugin transcriptions | | **B3** | Upstream the scroll **orchestration** layer as `src/common/sports_scroll.py`, reading `global_config['target_fps']` natively | ✅ | Content building stays per-sport | -| **B4** | Ship 3.2.0 *and* make version reporting trustworthy | ⏳ **next** | Tag, release, and `src.__version__` agree; compatibility gate merged | -| **B5** | Adoption — guarded core imports: three pilots, then the remaining six. **Bundled copies stay.** | after B4 | Per plugin: harness + goldens byte-identical, then a device soak | -| **B6** | Sunset — delete the bundled copies | **blocked** | B4's gate shipped *and* in users' hands (see below) | +| **B4** | Ship 3.2.0 *and* make version reporting trustworthy | ✅ | Released 2026-08-03; tag, release and `src.__version__` agree; compatibility gate merged (#428, #431, #433) | +| **B5** | Adoption — guarded core imports, all eight. **Bundled copies stay.** | ✅ | All eight adopted; harness byte-identical; see the B5 retrospective below — four shipped broken and were repaired in plugins #251 | +| **B6** | Sunset — delete the bundled copies | ✅ | Ran 2026-09-01, all eight. Floors at 3.2.0; the store refuses on all three routes in (#431/#433, #508, #510). See "B6 — what actually happened" | ### B4 — what "ship 3.2.0" actually requires @@ -234,6 +234,20 @@ a floor can be trusted against, and today it is not: compares the plugin's manifest version against the registry's `latest_version` and nothing else. + *Fixed, in two parts.* `install_plugin` gained the gate in #431/#433, which + covers every registry-managed install and, through `_reinstall_with_rollback`, + the update path that re-downloads. + `update_plugin`'s git branch pulls in place and re-downloads nothing, so it + stayed ungated until `_gate_pulled_commit` closed it — checked after the pull + (the registry carries no floor field, so the incoming floor is unknowable + before it) and undone with `git reset --hard` to the pre-pull commit. That + route is rare in practice, since monorepo plugins install as archives; it was + closed because the sunset rule in the plugins repo's + `08-shared-sports-code.md` states as **condition 3** that the core enforces + the floor "at install/update time", and B6 rests on that being true rather + than merely written down. `install_from_url` — sideloading a plugin from a + URL — is still ungated. + So B4 is: tag and release 3.2.0; make the tag, the release, and `__version__` agree, and keep them agreeing; reconsider the `< 2.0.0` skip; migrate manifests from `ledmatrix_min` to `ledmatrix_min_version`; and add the install/update @@ -262,13 +276,21 @@ default is the more restrictive. `compatible_versions`. No manifest still carries it, so there is nothing to migrate there.) -### B5 — adoption is safe by construction +### B5 — the *fallback* is safe by construction; the modern path is not + +The heading matters, because the unqualified version of this claim is false and +this document proves it two sections down: four of the eight adopted plugins +shipped with scroll mode broken on a 3.2.0 core. What is safe by construction is +narrower than "adoption". A plugin adopting core imports keeps its bundled copy and reaches it through the -guarded import (see the Upgradability table above). On a core that ships the -module the plugin uses core code; on one that doesn't it falls back and behaves -exactly as it does today. There is no version of this step that breaks a user, -which is why it does not wait for B6's gate. +guarded import (see the Upgradability table above). On a core that doesn't ship +the module the plugin falls back and behaves exactly as it does today. That +fallback compatibility — and only that — is safe by construction. On a core that +*does* ship the module, correctness is not automatic — object-level and scroll-mode validation +(building both classes and comparing, per the retrospective below) is required +to prove full behavior. There is no version of this step that breaks a user *on +an old core*, which is why it does not wait for B6's gate. The hockey scroll-display pilot is **already validated**: adopted against a core carrying 3.2.0, `scroll_display.py` went from 691 to 289 lines and all 16 harness @@ -297,7 +319,8 @@ been in users' hands long enough that the population running a core without it is small.** The bundled copies cost disk space; deleting them early costs scoreboards, silently. That trade is not close. -Before the first sunset, add a **compatibility regression test**. It has to +Before the first sunset, add a **compatibility regression test**. **Built:** +core `test/test_sports_sunset_matrix.py` (#505). It has to cover four cases, not one — B5's safety claim and B6's failure mode are different propositions and only the second is obvious: @@ -321,35 +344,144 @@ gate rather than trusting the failure to be noticed. The same suite should exercise the install/update gate, since it is the other half of the guarantee. +### B6 — what actually happened + +**Ran 2026-09-01, across all eight scoreboards.** Held from 2026-08-05 to +2026-09-01 on the argument below, which is kept because the reasoning applies to +the next module, not because it is still in force. + +**The hold, and why it lifted.** The stated gate was evidence of 3.2.0 uptake — +"a few months of it being the default download, or store-side install data". +That evidence never arrived and could not: the core updates by +`git pull --rebase`, so release-asset counts cannot measure uptake, and no +store-side telemetry exists. What changed instead is that the *risk* the gate +protected against was closed directly. The store now refuses a plugin whose +floor exceeds the running core on **all three** routes in: + +| route | gated by | +|---|---| +| `install_plugin` — every path that re-downloads, `_reinstall_with_rollback` included | #431, #433 | +| `update_plugin`'s git branch — pulls in place, re-downloads nothing | #508 | +| `install_from_url` — sideloading | #510 | + +With all three closed a pre-3.2.0 user cannot receive a sunset plugin at all; +they keep the version they already run. The population the hold existed to +protect is protected by refusal rather than by a bundled copy — which is what +the copy was standing in for. + +**What shipped.** Eight plugins, ~5,800 lines of frozen fallback deleted. Each: +copy removed, guarded import collapsed to a plain one, floor raised to 3.2.0, +`test_core_fallback.py` rewritten as `test_core_scroll.py` asserting the sunset +rather than the fallback. `scripts/check_scroll_adoption.py` gained +`sunset_violations` and a `SUNSET_PLUGINS` set naming all eight, so a +resurrected copy or a returned guard fails CI. + +Delivered as plugins #346 (hockey, later folded into #351), #349 (football), +#350 (baseball), #351 (the remaining six). + +**Two things found by doing it, both worth carrying forward:** + +- **Only one fallback held orchestration logic the core lacked.** baseball's + `_configure_scroll_helper` reinterpreted `scroll_speed` as pixels-per-*frame* + when `speed × delay` fell outside the 0.1–5.0 window — measured, 10–20× + faster than configured for a speed between 1.0 and 5.0. Standardised onto the + core's behaviour (honour the documented px/sec, clamp) rather than preserved. + Every other difference across the eight was a docstring, an unreachable + `scroll_helper is None` guard, or an equivalent diagnostic. +- **Two tests had been leaning on the guard without anyone knowing.** + `soccer/test_live_screens.py` stubbed `src` in a way that shadowed the core, + so its guarded import fell back and the test had been exercising the *frozen + copy* rather than the shipping class since B5. Before sunsetting anything else + that carries a guarded core import, grep for tests that stub `src`. + +**The floor-raising traps still apply** to any future sunset: four plugins +declare their floor top-level, where editing `versions[0]` is a silent no-op, +and the floor has three live spellings (`min_ledmatrix_version`, +`requires.min_ledmatrix_version`, `versions[].ledmatrix_min_version`, plus +deprecated `ledmatrix_min`). See +`src/plugin_system/compatibility.py:declared_min_version` for the resolution +order any floor-raising tool must reproduce — and note the name is **inverted** +between the top level and `versions[]`. + +**Still not adopted, deliberately:** `data_sources.py`, `game_renderer.py` and +`base_odds_manager.py`. The standing decision held them until B6 closed; it now +has, so they can be reconsidered — with B5's lesson applied, which is to build +the object and diff rendered output rather than trust a static check. + +### B5 retrospective — what the adoption actually cost + +Recorded because it is the evidence behind the two decisions above, and because +"the adoption went fine" is not what happened. + +**Four of the eight shipped with scroll mode broken** on a 3.2.0 core, and were +repaired in plugins-repo #251. The restructure lifted the content methods +verbatim but left the state they read off `self` behind: separator-icon +constants (hockey, basketball, lacrosse) and the game-renderer cache (afl). +hockey/basketball/lacrosse could not construct the scroll display at all; afl +raised inside `prepare_scroll_content`, which the core base *catches*, so its +only symptom was scroll mode silently drawing nothing. + +Three things are worth carrying forward: + +- **The bundled fallback did not protect anyone from this.** The break was on + the modern path, which the fallback never touches. Carrying the second copy + bought nothing against the actual defect while creating the divergence that + produced it. That is an argument *for* B6, not against it. +- **Every gate was green.** The safety harness renders the scoreboard screens, + not scroll mode; `test_core_fallback.py` checked that methods existed and that + their *globals* resolved, and `self.NHL_SEPARATOR_ICON` is an attribute read, + invisible to an AST scan for `Name` loads. The fix was to stop reasoning about + source and **build the object**: construct both classes on both paths, compare + the separator icons they end up with, and assert the adopted class ends up + with every instance attribute the bundled one sets. +- **Test what the change touches, not what is convenient to render.** Scroll + mode had no coverage because the harness could not reach it. A comparison + harness that renders the same games through both paths and diffs the pixels + needs no per-sport knowledge of the right answer, only that adopting core code + did not change it. + +**The ledger.** Before adoption, eight duplicated copies totalled 5,685 lines. +After adoption plus the frozen legacy copies it was 10,610; removing the dead +inline duplication (plugins #252) brought it to roughly 8,620. B6 would take it +to about 3,300 including the shared core module — some 2,400 fewer than before +this project started. **Until B6 runs, the adoption is net negative on disk**, +and its one delivered user-visible gain is that adopted plugins honour the +global `target_fps` instead of hardcoding ~100 FPS. + +### Decision: stop adopting further modules until B6 closes + +`data_sources.py` (9 copies), `game_renderer.py` (8) and `base_odds_manager.py` +are the obvious next candidates. **Do not adopt them yet.** Each adoption adds +carrying cost — a second copy to keep in step — against a payoff that is +contingent on B6, and B6 is gated on an installed base we cannot currently +measure. Consolidate what is already committed; revisit when B6 does. + ## What's next -In order. Each step is independently useful and independently revertible. +Steps 1–5 of the original plan are **done**: 3.2.0 is tagged and published with +a version number CI now asserts (#428), the compatibility gate is in +`install_plugin` and reads `compatible_versions` as well as the floor +(#431, #433), the newest manifest entry is required to use `ledmatrix_min_version` +(plugins #244), and all eight plugins have adopted the scroll orchestration +(plugins #245–#249, repaired in #251, tidied in #252). -1. **Tag and publish v3.2.0.** The code is already on `main` (`21825cbf`). - Nothing else blocks this, and it is what makes `ledmatrix_min_version: - "3.2.0"` refer to something real. -2. **Make the version number honest.** Have the release process assert that the - tag, the GitHub release, and `src.__version__` agree — a check in CI is - cheaper than the confusion of the last two releases. Then revisit the - `< 2.0.0` skip in `_warn_if_incompatible`, which currently silences the - warning for the users who most need it. -3. **Add the compatibility gate** to `StoreManager.install_plugin` and - `.update_plugin`: refuse a plugin whose declared floor exceeds - `src.__version__`, and surface the reason in the store UI rather than only - the log. This is the single change that turns the floor from documentation - into a guarantee, and B6 depends on it. -4. **Migrate the manifests** to `ledmatrix_min_version`, and reconcile them with - `compatible_versions` (see above — that field is the required, canonical one, - and the gate does not read it yet). Currently 28 plugins spell the floor both - ways across their `versions[]` entries, 12 use only the old spelling, and 2 - only the new. Scope the sweep to the nine sports plugins if a 42-plugin - version-bump wave isn't worth it — but the `compatible_versions` half has to - cover every manifest the gate can refuse, or define explicit legacy handling, - before the gate is allowed to block anything. -5. **Run B5 adoption** — hockey, soccer, football, then the remaining six. - Bundled copies stay. Byte-identical harness output per plugin, then a soak. -6. **Only then plan B6**, with the compatibility regression test described above - in CI first. +What actually remains, smallest first: + +1. **Soak the adoptions on hardware.** football and hockey have been run on a + live rig through real games; baseball was watched through one earlier. The + rest are proven by harness, unit tests and pixel comparison. Out-of-season + sports cannot be soaked until their season starts. When you do, **check the + rig's `*_display_mode` first** — a board in `switch` mode will happily load a + sunset plugin and tell you nothing about the scroll code the sunset changed. +2. **Cut 3.3.0.** Not required by B6 — its floors are 3.2.0, which is released — + but `calendar` 1.2.3 floors at 3.3.0 for the device-authorization endpoints + that landed after 3.2.0, so it is un-installable until the release exists. +3. **Reconsider the held modules** (`data_sources.py`, `game_renderer.py`, + `base_odds_manager.py`) now that the sunset has closed. `game_renderer.py` is + the largest single duplication left: ~11,500 lines across eight plugins, with + ~36,500 more in the eight `sports.py`. Note that core already ships + `src/base_classes/sports/` (~143KB, promoted in B1/B2) that **no plugin + imports** — check whether it has drifted before treating it as the target. ## How to keep this project healthy diff --git a/first_time_install.sh b/first_time_install.sh index c7ec7cf1..198a8f2e 100755 --- a/first_time_install.sh +++ b/first_time_install.sh @@ -1740,12 +1740,15 @@ journald_effective() { systemd-analyze cat-config systemd/journald.conf >/dev/null 2>&1; then systemd-analyze cat-config systemd/journald.conf 2>/dev/null else - cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null + cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null || true fi } journald_conf="$(journald_effective)" -journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')" -journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')" +# Both settings are optional, and stock images ship them commented out. grep +# exits 1 on no match, which pipefail turns fatal under set -e — an absent +# setting must read as empty, not abort the install. +journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)" +journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)" if [ "$journald_storage" = "persistent" ] && [ -n "$journald_cap" ]; then echo "Persistent journald storage already configured (SystemMaxUse=$journald_cap)" @@ -1771,7 +1774,7 @@ else # after ledmatrix-persistent.conf (zz-local.conf and friends) still wins. # Writing the file is not evidence it took effect -- re-read and say so # plainly rather than reporting success we cannot confirm. - journald_now="$(journald_effective | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')" + journald_now="$(journald_effective | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)" if [ "$journald_now" = "persistent" ]; then echo " Persistent journald storage active" else diff --git a/scripts/render_plugin.py b/scripts/render_plugin.py index 39dee61f..6db85c1b 100644 --- a/scripts/render_plugin.py +++ b/scripts/render_plugin.py @@ -52,6 +52,10 @@ def main() -> int: parser.add_argument('--height', type=int, default=32, help='Display height (default: 32)') parser.add_argument('--skip-update', action='store_true', help='Skip calling update() (render display only)') + parser.add_argument('--display-mode', default=None, + help='Display mode to render, for plugins that declare ' + 'more than one in their manifest (e.g. nrl_live). ' + 'Omitted, the plugin picks its own default.') args = parser.parse_args() @@ -141,8 +145,26 @@ def main() -> int: except Exception as e: logger.warning("update() raised: %s — continuing to display()", e) + # A plugin that declares several display modes usually renders nothing + # useful without being told which one to draw: the scoreboards keep their + # state on per-mode sub-managers and their no-argument path returns False. + # Only pass the argument when asked for, so the many plugins whose display() + # takes no display_mode keep working untouched. try: - plugin_instance.display(force_clear=True) + if args.display_mode: + try: + plugin_instance.display(display_mode=args.display_mode, + force_clear=True) + except TypeError as error: + if ("unexpected keyword argument" not in str(error) + or "display_mode" not in str(error)): + raise + logger.warning( + "%s.display() does not accept display_mode; rendering its " + "default screen instead", args.plugin) + plugin_instance.display(force_clear=True) + else: + plugin_instance.display(force_clear=True) logger.debug("display() completed") except Exception as e: logger.error("Error in display(): %s", e) diff --git a/src/__init__.py b/src/__init__.py index f8fa0daa..c73f8102 100644 --- a/src/__init__.py +++ b/src/__init__.py @@ -4,5 +4,5 @@ LEDMatrix Display System Core source package for the LED Matrix Display project. """ -__version__ = "3.2.0" +__version__ = "3.3.0" diff --git a/src/background_data_service.py b/src/background_data_service.py index 355c93c4..fa20783c 100644 --- a/src/background_data_service.py +++ b/src/background_data_service.py @@ -14,11 +14,12 @@ Key Features: - Memory-efficient data storage """ +import itertools import time import logging import threading import requests -from typing import Dict, Any, Optional, Callable +from typing import Dict, Any, Optional, Callable, List from dataclasses import dataclass, field from enum import Enum import queue @@ -50,14 +51,33 @@ class FetchRequest: max_retries: int = 3 priority: int = 1 # Higher number = higher priority callback: Optional[Callable] = None + # Callbacks from submitters that JOINED this fetch instead of starting a + # duplicate one. The primary `callback` above belongs to whoever created + # the request; these belong to everyone who asked for the same cache_key + # while it was still in flight. + extra_callbacks: List[Callable] = field(default_factory=list) created_at: float = field(default_factory=time.time) status: FetchStatus = FetchStatus.PENDING + # Set once the worker has decided this response will be cached, while it + # still holds the lock. From that point cancelling is refused: the write + # is already authorised, and abandoning it here would put the payload in + # the cache with the callbacks suppressed -- joiners waiting forever for a + # fetch that did, in fact, succeed. + commit_claimed: bool = False result: Optional[Any] = None error: Optional[str] = None @dataclass class FetchResult: - """Result of a background fetch operation.""" + """Result of a background fetch operation. + + ``data`` survives on the stored result only for requests submitted without + a ``callback``, where polling ``get_result()`` is the sole way to collect + it. When a callback was given, the payload has already been delivered and + the service releases it -- see :meth:`BackgroundDataService._release_payload`. + Either way the data remains in the cache under the request's ``cache_key``, + which is where consumers read it from. + """ request_id: str success: bool data: Optional[Any] = None @@ -66,6 +86,10 @@ class FetchResult: fetch_time: float = 0.0 retry_count: int = 0 completed_at: float = field(default_factory=time.time) # Timestamp when request completed + # The request's final status, recorded so a finished request can still be + # reported accurately. Without it a caller can only be told COMPLETED or + # FAILED, which turns "you cancelled this" into "this errored". + final_status: Optional[FetchStatus] = None class BackgroundDataService: """ @@ -90,6 +114,20 @@ class BackgroundDataService: # Thread management self.executor = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix="BackgroundData") + # cache_key -> request_id for fetches currently in flight. Submitting + # the same key twice used to start two identical fetches: request_id + # carries a millisecond timestamp, so every submit looked new, and + # active_requests is keyed by it rather than by what is being fetched. + # On a real board the season-schedule key is requested by both the + # Recent and the Upcoming manager, which miss the cache in the same + # millisecond and each download and parse the same payload. + self._inflight_by_cache_key: Dict[str, str] = {} + # request_id was sport_year_milliseconds, which is not unique: two + # submits inside the same millisecond produced the SAME id, so one + # silently replaced the other in active_requests and completed_requests. + # Rare before, but dedupe hands this id back to every joiner as their + # handle for get_result(), so it has to be unique. A counter is enough. + self._request_seq = itertools.count() self.active_requests: Dict[str, FetchRequest] = {} self.completed_requests: Dict[str, FetchResult] = {} self.request_queue = queue.PriorityQueue() @@ -177,7 +215,9 @@ class BackgroundDataService: if cache_key is None: cache_key = self.get_sport_cache_key(sport) - request_id = f"{sport}_{year}_{int(time.time() * 1000)}" + with self._lock: + request_id = (f"{sport}_{year}_{int(time.time() * 1000)}" + f"_{next(self._request_seq)}") # Check cache first cached_data = self.cache_manager.get(cache_key) @@ -191,14 +231,19 @@ class BackgroundDataService: cached=True, fetch_time=0.0 ) + # Filed before the callback runs, as it always was: a callback + # that queries get_result()/is_request_complete() for its own + # request must still find it. Releasing afterwards mutates the + # same object the dict holds. self.completed_requests[request_id] = result - + if callback: try: callback(result) except Exception as e: logger.error(f"Error in callback for request {request_id}: {e}") - + self._release_payload(result) + logger.debug(f"Cache hit for {sport} {year} data") return request_id @@ -218,7 +263,29 @@ class BackgroundDataService: ) with self._lock: + existing_id = self._inflight_by_cache_key.get(cache_key) + existing = self.active_requests.get(existing_id) if existing_id else None + if existing_id and existing is None: + # Stranded index entry: the request it names is gone. Drop it and + # fetch normally. Looking the request up rather than trusting the + # id is what stops a stale entry wedging a key forever. + del self._inflight_by_cache_key[cache_key] + if existing is not None: + # Someone is already fetching this key. Ride along rather than + # duplicating the download, the parse and the resident copy. + if callback: + existing.extra_callbacks.append(callback) + self.stats['deduplicated_requests'] = ( + self.stats.get('deduplicated_requests', 0) + 1 + ) + logger.info( + "Joined in-flight fetch %s for %s (cache_key=%s) instead of " + "starting a duplicate", existing_id, sport, cache_key + ) + return existing_id + self.active_requests[request_id] = request + self._inflight_by_cache_key[cache_key] = request_id self.stats['total_requests'] += 1 self.stats['cache_misses'] += 1 @@ -243,7 +310,31 @@ class BackgroundDataService: try: with self._lock: - request.status = FetchStatus.IN_PROGRESS + # A request cancelled while it sat in the executor queue must + # stay cancelled. Overwriting the status here undid the cancel + # outright: the worker went on to download, cache and call back + # for work the caller had already withdrawn. + if request.status == FetchStatus.CANCELLED: + cancelled_before_start = True + else: + cancelled_before_start = False + request.status = FetchStatus.IN_PROGRESS + if cancelled_before_start: + logger.info( + "Request %s was cancelled before its worker started; " + "skipping the fetch entirely", request.id + ) + # Assign before returning: the finally block stores `result` + # in completed_requests, so building a fresh one here would + # file the untouched placeholder instead of this outcome. + result = FetchResult( + request_id=request.id, + success=False, + error="cancelled", + fetch_time=time.time() - start_time, + retry_count=request.retry_count + ) + return result logger.info(f"Starting background fetch for {request.sport} {request.year}") @@ -269,6 +360,37 @@ class BackgroundDataService: # Log data validation logger.debug(f"Validated {len(events)} events for {request.sport} {request.year}") + # A cancelled request must not commit anything. Cancelling + # releases the cache_key, so a replacement fetch for the same key + # may already be in flight or finished -- writing this response to + # the cache now would overwrite fresher data with the response + # nobody wanted. The worker has no way to abort the HTTP call, so + # this is where the work gets discarded. + with self._lock: + cancelled = request.status == FetchStatus.CANCELLED + if not cancelled: + # Claim the commit in the same critical section that read + # the status, so a cancel cannot slip in between the check + # and the cache write below. The write itself stays outside + # the lock: it serialises a multi-megabyte payload to the + # SD card, and holding the service lock across that would + # stall every submit, status query and cancel behind it. + request.commit_claimed = True + if cancelled: + logger.info( + "Discarding response for cancelled request %s; %s may " + "already belong to a replacement fetch", + request.id, request.cache_key + ) + result = FetchResult( + request_id=request.id, + success=False, + error="cancelled", + fetch_time=time.time() - start_time, + retry_count=request.retry_count + ) + return result + # Cache the data self.cache_manager.set(request.cache_key, data) @@ -294,7 +416,12 @@ class BackgroundDataService: logger.error(f"Failed to fetch {request.sport} {request.year} data: {error_msg}") with self._lock: - request.status = FetchStatus.FAILED + # Don't relabel a cancelled request. The callback gate in the + # finally block only suppresses CANCELLED, so promoting it to + # FAILED here delivered an error callback for a fetch nobody + # was waiting on any more. + if request.status != FetchStatus.CANCELLED: + request.status = FetchStatus.FAILED request.error = error_msg result = FetchResult( @@ -308,9 +435,26 @@ class BackgroundDataService: finally: # Store result and clean up with self._lock: + result.final_status = request.status self.completed_requests[request.id] = result if request.id in self.active_requests: del self.active_requests[request.id] + # Stop accepting joiners and take the callback list in the same + # critical section. A submitter that arrives after this point + # finds no in-flight entry and either hits the cache (written + # above, before the result was built) or starts a fresh fetch -- + # what it must never do is join a fetch whose callbacks have + # already run and then never be called. + if self._inflight_by_cache_key.get(request.cache_key) == request.id: + del self._inflight_by_cache_key[request.cache_key] + # A cancelled request delivers nothing: its joiners were told + # about a fetch that has been abandoned, and a replacement will + # call them via its own request. + if request.status == FetchStatus.CANCELLED: + callbacks = [] + else: + callbacks = ([request.callback] if request.callback else []) + callbacks.extend(request.extra_callbacks) # Update statistics if result.success: @@ -327,14 +471,53 @@ class BackgroundDataService: # Periodic cleanup after storing result self._cleanup_completed_requests() - # Call callback if provided - if request.callback: + # Call every callback: the original submitter's and any that joined + # this fetch. One raising must not stop the others being delivered. + for cb in callbacks: try: - request.callback(result) + cb(result) except Exception as e: logger.error(f"Error in callback for request {request.id}: {e}") - + + # Released AFTER the loop, not inside it. Every callback here holds + # the same FetchResult, so releasing per-delivery handed the first + # one the data and every joiner `result.data is None` -- which is + # not a quiet degradation: they read `result.data.get('events')` and + # raise AttributeError, which this very loop catches and logs, so + # the symptom was one ERROR line and a manager that silently never + # got its schedule. Deduplication is the normal case, not a corner: + # a sport's recent, upcoming and live managers all ride one season + # fetch. + # + # Guarded on `callbacks`, because a request submitted without one + # has no other way to collect its payload than polling get_result(). + # The old per-delivery release got that right by accident: an empty + # list never entered the loop body. + if callbacks: + self._release_payload(result) + request.result = None + return result + + @staticmethod + def _release_payload(result: FetchResult) -> None: + """Drop a delivered payload, keeping the result's status and timings. + + Only called once EVERY callback has been handed the data -- callers + that joined an in-flight fetch share this object, so releasing between + deliveries strips the payload out from under the ones still queued. + Consumers read fetched data back from the cache under ``cache_key``; + the copy carried + here was pinning a parsed season schedule -- 946 games for NCAA + football, roughly a tenth of total RAM on a 1GB Pi -- in memory until + the hourly sweep. + + The cache-hit path matters most: it runs once per update interval per + sport, mints a fresh request_id each time, and a memory-tier miss + re-parses the payload from disk. Those were genuinely separate copies + accumulating toward the 500-entry cap, not shared references. + """ + result.data = None def _make_request_with_retry(self, request: FetchRequest) -> requests.Response: """ @@ -422,6 +605,8 @@ class BackgroundDataService: return self.active_requests[request_id].status elif request_id in self.completed_requests: result = self.completed_requests[request_id] + if result.final_status is not None: + return result.final_status return FetchStatus.COMPLETED if result.success else FetchStatus.FAILED return None @@ -438,8 +623,22 @@ class BackgroundDataService: with self._lock: if request_id in self.active_requests: request = self.active_requests[request_id] + if request.commit_claimed: + # Too late: the worker holds an authorised commit. Report + # the failure rather than half-cancelling a request whose + # data is about to land in the cache. + logger.debug( + "Not cancelling %s: its response is already being " + "committed", request_id + ) + return False request.status = FetchStatus.CANCELLED del self.active_requests[request_id] + # Cancelling is the other way a request leaves active_requests, + # so the in-flight index has to be released here too or the key + # stays pointed at a request that no longer exists. + if self._inflight_by_cache_key.get(request.cache_key) == request_id: + del self._inflight_by_cache_key[request.cache_key] logger.info(f"Cancelled request {request_id}") return True return False diff --git a/src/common/logo_helper.py b/src/common/logo_helper.py index 743e7b7b..a64ca340 100644 --- a/src/common/logo_helper.py +++ b/src/common/logo_helper.py @@ -143,8 +143,10 @@ class LogoHelper: """ logo_path = Path(logo_path) - # Try to load existing logo first - if logo_path.exists(): + # Try to load existing logo first. A placeholder written by a previous + # failed download does not count: it wears the real logo's filename, so + # trusting the file's existence is what left teams as grey boxes. + if logo_path.exists() and not self._is_stale_placeholder(logo_path): return self.load_logo(team_abbr, logo_path, max_width, max_height) # Download if URL provided and file doesn't exist @@ -152,13 +154,61 @@ class LogoHelper: try: self.logger.info(f"Downloading logo for {team_abbr} from {logo_url}") self._download_logo(logo_url, logo_path) + # The file on disk just changed. Any cached image for it is the + # placeholder we came here to replace, and load_logo() answers + # from the cache before touching the disk -- so without this the + # real logo would not appear until the process restarted. + self._invalidate_cached_logo(team_abbr, logo_path) return self.load_logo(team_abbr, logo_path, max_width, max_height) except Exception as e: self.logger.error(f"Failed to download logo for {team_abbr}: {e}") + # The retry failed, so restart the back-off. The stale + # placeholder is still on disk with its old timestamp, and + # leaving it there means the next call retries immediately -- + # a download attempt per call, which is what the back-off + # exists to prevent. + self._refresh_stale_placeholder(logo_path) # Create placeholder if all else fails return self._create_placeholder_logo(team_abbr, max_width, max_height) + def _invalidate_cached_logo(self, team_abbr: str, logo_path: Path) -> None: + """Drop every cached size of one logo after its file changed on disk.""" + prefix = f"{team_abbr}_{logo_path}_" + for key in [k for k in self._logo_cache if k.startswith(prefix)]: + self._logo_cache.pop(key, None) + if key in self._cache_order: + self._cache_order.remove(key) + + @staticmethod + def _refresh_stale_placeholder(logo_path: Path) -> None: + """Restart the retry back-off after a failed download attempt.""" + try: + from src.logo_downloader import refresh_placeholder_timestamp + except ImportError: + return + refresh_placeholder_timestamp(logo_path) + + @staticmethod + def _is_stale_placeholder(logo_path: Path) -> bool: + """True if the file is a placeholder old enough to be worth retrying. + + Imported lazily so this module keeps working against a core build whose + logo_downloader predates placeholder marking. + """ + try: + from src.logo_downloader import ( + PLACEHOLDER_RETRY_SECONDS, + is_placeholder_logo, + placeholder_age_seconds, + ) + except ImportError: + return False + if not is_placeholder_logo(logo_path): + return False + age = placeholder_age_seconds(logo_path) + return age is None or age >= PLACEHOLDER_RETRY_SECONDS + def get_logo_variations(self, team_abbr: str) -> List[str]: """ Get possible filename variations for a team abbreviation. diff --git a/src/common/scroll_helper.py b/src/common/scroll_helper.py index 88c6d498..6e2f4218 100644 --- a/src/common/scroll_helper.py +++ b/src/common/scroll_helper.py @@ -328,6 +328,10 @@ class ScrollHelper: elapsed_time = current_time - (self.scroll_start_time or current_time) # The image already includes display_width padding, so we only need total_scroll_width required_total_distance = self.total_scroll_width + # Progress telemetry, emitted every few seconds for the whole of + # every scroll. It says how far along a marquee is, which is what + # you turn debug on to watch and not something an operator needs + # in the journal on a device that scrolls all day. self.logger.debug( "Scroll progress: elapsed=%.2fs, target=%.2fs, total_scrolled=%.0f/%d px (%.1f%%)", elapsed_time, diff --git a/src/common/sports_card.py b/src/common/sports_card.py new file mode 100644 index 00000000..6b8bd7ea --- /dev/null +++ b/src/common/sports_card.py @@ -0,0 +1,470 @@ +"""Card-drawing helpers shared by every sports scoreboard plugin. + +The eight scoreboards each carried byte-identical copies of the functions +below: the colour pickers, the settings lookup, the date and time formatting, +the favourite-team rules and the font-size grid snapping. One fix had to be +made eight times, and a new scoreboard started by copying them a ninth. + +Everything here is a **free function taking explicit arguments**, not a base +class. Adoption is therefore per-function and reversible: a plugin keeps its +method and delegates the body, so the call sites and the override points are +untouched. That is also why `config`, `logger` and `fonts` are parameters +rather than attributes -- the helper never reaches back into the caller. + +The bodies are the plugins' own code, moved rather than rewritten. The one +deliberate difference is `crisp_size`, which takes the seven-plugin guard +(`not desired`) instead of football's: they agree on every real input, and +the extra guard only stops a None size raising TypeError. +""" + +import logging +from datetime import datetime, timezone +from typing import Any, Dict, Optional, Tuple +from zoneinfo import ZoneInfo + +logger = logging.getLogger(__name__) + +__all__ = [ + "ELEMENT_FOR_FONT", "FAVORITE_RESULT_COLOR_DEFAULTS", "FONT_NAME_ALIASES", + "FONT_PIXEL_GRID", "MONTH_ABBR", "WEEKDAY_ABBR", + "scroll_card_option", "element_color", "font_color", "coerce_rgb", + "score_color_for", "recent_score_color", "favorite_teams_for", + "side_is_favorite", "side_score", "favorite_result", + "card_tzinfo", "weekday_for", "format_game_date", "format_game_time", + "vs_text", "upcoming_center_mode", "crisp_size", "schema_font_size", + "resolve_font_size", "unshare_element_fonts", +] + +#: Which customization element owns each font key, for colour resolution. +ELEMENT_FOR_FONT: Dict[str, str] = { + "score": "score_text", + "time": "period_text", + "team": "team_name", + "status": "status_text", + "detail": "detail_text", + "rank": "rank_text", +} + +#: Fallback colours when favourite_result_colors is on but a slot is unset. +FAVORITE_RESULT_COLOR_DEFAULTS: Dict[str, Tuple[int, int, int]] = { + "win": (0, 255, 0), + "loss": (255, 0, 0), + "tie": (255, 200, 0), +} + +#: Family aliases the web UI may write, mapped to the shipped filename. +FONT_NAME_ALIASES: Dict[str, str] = { + "press_start": "PressStart2P-Regular.ttf", + "four_by_six": "4x6-font.ttf", +} + +#: Pixel grid each face renders crisply on. Off-grid sizes anti-alias, which +#: on an LED matrix is a dim lamp rather than a soft edge. +FONT_PIXEL_GRID: Dict[str, int] = { + "PressStart2P-Regular.ttf": 8, + "4x6-font.ttf": 7, +} + +MONTH_ABBR = ("Jan", "Feb", "Mar", "Apr", "May", "Jun", + "Jul", "Aug", "Sep", "Oct", "Nov", "Dec") +WEEKDAY_ABBR = ("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun") + + +# --------------------------------------------------------------------------- +# Settings lookup +# --------------------------------------------------------------------------- + +def scroll_card_option(config: Optional[Dict[str, Any]], key: str, + default: Any = None) -> Any: + """Read one key from the scroll_card config block.""" + block = (config or {}).get("scroll_card") + if isinstance(block, dict) and block.get(key) is not None: + return block.get(key) + return default + + +def vs_text(config: Optional[Dict[str, Any]]) -> str: + """Separator drawn between the teams -- "VS", "@", "at", anything.""" + return str(scroll_card_option(config, "vs_text", "VS")) + + +def upcoming_center_mode(config: Optional[Dict[str, Any]]) -> str: + """Middle of an upcoming card: 'vs', 'date_time' or 'none'.""" + mode = str(scroll_card_option(config, "upcoming_center", "vs") or "vs").lower() + return mode if mode in ("vs", "date_time", "none") else "vs" + + +# --------------------------------------------------------------------------- +# Colour +# --------------------------------------------------------------------------- + +def element_color(config: Optional[Dict[str, Any]], element: str, + default: Tuple[int, int, int] = (255, 255, 255)): + """Per-element text colour from customization..text_color.""" + try: + cfg = (config or {}).get("customization", {}).get(element, {}) + value = cfg.get("text_color") + if isinstance(value, (list, tuple)) and len(value) == 3: + return tuple(max(0, min(255, int(c))) for c in value) + if isinstance(value, str) and value.startswith("#") and len(value) == 7: + return tuple(int(value[i:i + 2], 16) for i in (1, 3, 5)) + except (TypeError, ValueError): + pass + return default + + +def font_color(config: Optional[Dict[str, Any]], fonts: Optional[Dict[str, Any]], + font, default: Tuple[int, int, int] = (255, 255, 255)): + """Colour for whichever element owns this face. + + Matched on identity, and deliberately gives up when one object is + shared: the last-resort font path can hand the same face to several + keys, and there is no right answer for which element's colour that is. + White is what those draws used before, so ambiguity costs nothing. + """ + try: + fonts = fonts or {} + matches = [element for key, element in ELEMENT_FOR_FONT.items() + if fonts.get(key) is font] + if len(matches) == 1: + return element_color(config, matches[0], default) + except (AttributeError, TypeError): + pass + return default + + +def coerce_rgb(value, fallback): + """Turn a configured [R, G, B] list into a clamped (r, g, b) tuple.""" + # Checked before unpacking: a 3-character string ("123") would otherwise + # iterate into three digits and yield a colour rather than the fallback. + if not isinstance(value, (list, tuple)) or len(value) != 3: + return fallback + try: + r, g, b = (max(0, min(255, int(channel))) for channel in value) + except (TypeError, ValueError): + return fallback + return (r, g, b) + + +# --------------------------------------------------------------------------- +# Favourite teams +# --------------------------------------------------------------------------- + +def favorite_teams_for(config: Dict[str, Any], game: Dict[str, Any]) -> list: + """Favorite teams that apply to this game. + + Both sources are used. Games carry the league manager's *resolved* + favorites, which is the only place dynamic groups such as AP_TOP_25 + appear expanded; the config is read as well so an edit takes effect on + already-fetched games, and so hand-built game dicts (tests, other + callers) still work. + """ + favorites = list(game.get("favorite_teams") or []) + league_config = config.get(str(game.get("league", "") or "")) + if isinstance(league_config, dict): + favorites += list(league_config.get("favorite_teams") or []) + else: + favorites += list(config.get("favorite_teams") or []) + return favorites + + +def side_is_favorite(game: Dict[str, Any], side: str, favorites: set) -> bool: + """Is the home/away side of this game a favorite team? + + Reads both the flat (``home_abbr``) and nested (``home_team.abbrev``) + payload shapes, and matches on the ESPN id too, because a couple of + leagues (NRL) key favorites by id where abbreviations collide. + """ + candidates = [game.get(f"{side}_abbr"), game.get(f"{side}_id")] + team = game.get(f"{side}_team") + if isinstance(team, dict): + candidates += [team.get("abbrev"), team.get("abbreviation"), team.get("id")] + for value in candidates: + if value is not None and str(value).strip().upper() in favorites: + return True + return False + + +def side_score(game: Dict[str, Any], side: str) -> Optional[int]: + """Numeric score for one side, from either payload shape.""" + raw = None + team = game.get(f"{side}_team") + if isinstance(team, dict) and team.get("score") is not None: + raw = team.get("score") + if raw is None: + raw = game.get(f"{side}_score") + try: + return int(float(str(raw).strip())) + except (TypeError, ValueError): + return None + + +def favorite_result(config: Dict[str, Any], game: Dict[str, Any]) -> Optional[str]: + """Say how the favorite team did in a finished game. + + Returns 'win', 'loss' or 'tie', or None when there is no single team + to root for: no favorites configured, neither side is a favorite, or + *both* are -- a favorite-vs-favorite game has no losing side worth + flagging in red. Also None when the scores are not usable numbers. + """ + favorites = { + str(team).strip().upper() + for team in favorite_teams_for(config, game) + if str(team).strip() + } + if not favorites: + return None + + home_fav = side_is_favorite(game, "home", favorites) + away_fav = side_is_favorite(game, "away", favorites) + if home_fav == away_fav: + return None + + home_score = side_score(game, "home") + away_score = side_score(game, "away") + if home_score is None or away_score is None: + return None + + if home_score == away_score: + return "tie" + favorite_score, other_score = ( + (home_score, away_score) if home_fav else (away_score, home_score) + ) + return "win" if favorite_score > other_score else "loss" + + +def recent_score_color(config: Dict[str, Any], logger, game: Dict[str, Any], default): + """Fill color for a finished game's score, per favorite_result_colors.""" + try: + settings = (config.get("customization") or {}).get( + "favorite_result_colors" + ) or {} + if not settings.get("enabled", False): + return default + result = favorite_result(config, game) + if result is None: + return default + return coerce_rgb( + settings.get(f"{result}_color"), + FAVORITE_RESULT_COLOR_DEFAULTS[result], + ) + except Exception: + logger.debug("Could not resolve favorite result color", exc_info=True) + return default + + +def score_color_for(config: Dict[str, Any], logger, game: Dict[str, Any], + game_type: str, default=None): + """Fill color for a game card's score. Only finished games are tinted. + + The default is the configured score colour rather than a flat white, + so customization.score_text.text_color shows on games the favourite + tint does not apply to. The tint still wins where it applies. + """ + if default is None: + default = element_color(config, 'score_text') + if game_type != "recent": + return default + return recent_score_color(config, logger, game, default) + + +# --------------------------------------------------------------------------- +# Date and time +# --------------------------------------------------------------------------- + +def card_tzinfo(config: Optional[Dict[str, Any]], logger): + """Timezone for weekday/24h conversions; falls back to UTC.""" + configured = (config or {}).get("timezone") + if configured: + try: + return ZoneInfo(configured) + except (KeyError, ValueError, TypeError, OSError) as exc: + # KeyError covers ZoneInfoNotFoundError. A bad zone name in + # config should fall back to UTC, not blank the card. + logger.debug("Unusable timezone %r: %s", configured, exc) + return timezone.utc + + +def weekday_for(config: Optional[Dict[str, Any]], logger, + game: Optional[Dict]) -> str: + """Weekday abbreviation from the game's start time, or ''.""" + if not game: + return "" + raw = game.get("start_time_utc") or game.get("start_time") + if not raw: + return "" + try: + start = raw if isinstance(raw, datetime) else datetime.fromisoformat( + str(raw).replace("Z", "+00:00")) + return WEEKDAY_ABBR[start.astimezone(card_tzinfo(config, logger)).weekday()] + except (ValueError, TypeError): + return "" + + +def format_game_date(config: Optional[Dict[str, Any]], logger, date_text: str, + game: Optional[Dict] = None) -> str: + """Format an upcoming card's date per scroll_card.date_format.""" + raw = str(date_text or "").strip() + if not raw: + return "" + fmt = str(scroll_card_option(config, "date_format", "abbrev") or "abbrev") + if fmt == "numeric": + return raw + parts = raw.replace("-", "/").split("/") + if not (len(parts) >= 2 and parts[0].strip().isdigit() and parts[1].strip().isdigit()): + return raw + month, day = int(parts[0]), int(parts[1]) + if not 1 <= month <= 12: + return raw + name = MONTH_ABBR[month - 1] + if fmt == "numeric_day_first": + return f"{day}/{month}" + if fmt == "day_first": + return f"{day} {name}" + if fmt == "weekday": + weekday = weekday_for(config, logger, game) + return f"{weekday} {name} {day}" if weekday else f"{name} {day}" + return f"{name} {day}" + + +def format_game_time(config: Optional[Dict[str, Any]], time_text: str) -> str: + """Return the time as-is (12h) or converted to 24h.""" + raw = str(time_text or "").strip() + if not raw or str(scroll_card_option(config, "time_format", "12h")) != "24h": + return raw + cleaned = raw.upper().replace(" ", "") + meridiem = "AM" if cleaned.endswith("AM") else "PM" if cleaned.endswith("PM") else "" + if not meridiem: + return raw + try: + hh, _, mm = cleaned[:-2].partition(":") + hour, minute = int(hh), int(mm or 0) + except ValueError: + return raw + if not (0 <= hour <= 12 and 0 <= minute <= 59): + return raw + hour = hour % 12 + (12 if meridiem == "PM" else 0) + return f"{hour:02d}:{minute:02d}" + + +# --------------------------------------------------------------------------- +# Font sizing +# --------------------------------------------------------------------------- + +#: Per-schema caches, keyed by the schema's absolute path. Keyed rather than +#: global because each plugin declares its own defaults; keyed rather than +#: per-class because the helper has no class to hang it on. +_SCHEMA_FONT_SIZE_CACHE: Dict[str, Dict[str, int]] = {} + + +def crisp_size(font_file, desired, aliases=None, grid_table=None): + """Snap *desired* to the nearest size *font_file* renders crisply at. + + A face with no known grid is returned unchanged, so a user-supplied + font is never second-guessed. + + ``aliases`` and ``grid_table`` default to the shared tables; a plugin + that ships an extra face can pass its own without forking this. + """ + aliases = FONT_NAME_ALIASES if aliases is None else aliases + grid_table = FONT_PIXEL_GRID if grid_table is None else grid_table + font_file = aliases.get(font_file, font_file) + grid = grid_table.get(font_file) + if not grid or not desired or desired <= 0: + return desired + return max(grid, int(round(float(desired) / grid)) * grid) + + +def schema_font_size(schema_path: str, element_key) -> Optional[int]: + """The font_size this plugin's config_schema.json declares, or None. + + Cached per schema path. The plugins cached this on their own class; the + path is the same distinction expressed without one, so two plugins never + share an entry. + """ + if not element_key: + return None + cache = _SCHEMA_FONT_SIZE_CACHE.get(schema_path) + if cache is None: + cache = {} + try: + import json + with open(schema_path) as fh: + schema = json.load(fh) + props = (schema.get('properties', {}) + .get('customization', {}) + .get('properties', {})) + for key, spec in props.items(): + size = spec.get('properties', {}).get('font_size', {}).get('default') + if size is not None: + cache[key] = int(size) + except Exception as exc: + # See sports_shared._schema_font_size: an unreadable schema + # silently disables the pixel-grid snap for every element. + # Built once per schema path, so this cannot repeat per frame. + logger.warning( + "could not read %s (%s: %s); font sizes will skip their " + "pixel grid snap and may render a pixel narrow", + schema_path, type(exc).__name__, exc) + cache = {} + _SCHEMA_FONT_SIZE_CACHE[schema_path] = cache + return cache.get(element_key) + + +def resolve_font_size(schema_path: str, element_config, element_key, + default_size, font_name, aliases=None, grid_table=None): + """Size to render at: the user's choice, or a grid-snapped default. + + A configured size counts as a real choice only when it differs from + the schema default. The web UI writes the whole schema default block + on every save, so "font_size == schema default" carries no intent and + would otherwise pin every install to an anti-aliased size forever. + """ + configured = (element_config or {}).get('font_size') + if configured is not None: + try: + configured = int(configured) + if configured != schema_font_size(schema_path, element_key): + return configured + except (TypeError, ValueError): + pass + return crisp_size(font_name, default_size, aliases, grid_table) + + +def unshare_element_fonts(logger, fonts): + """Give each colourable element its own face object. + + The colour a draw gets is resolved from the face it was handed, and + several of these loaders legitimately hand one object to more than one + element -- a size resolver that lands two elements on the same face, a + fallback that fills every key from one default, football's narrowing + step that deliberately shrinks the clock along with the score. Sharing + the object makes the element ambiguous and the colour unresolvable. + + Re-instantiating from the same path and size gives a distinct object + with identical metrics, so nothing about the rendering changes; only + the ability to tell two elements apart does. Faces that cannot be + rebuilt (a BDF loaded through freetype.Face, anything without a usable + path) are left shared, and their draws stay white as before. + """ + try: + from PIL import ImageFont as _IF + except ImportError: # pragma: no cover + return fonts + seen = {} + for key in ELEMENT_FOR_FONT: + font = fonts.get(key) + if font is None: + continue + if id(font) not in seen: + seen[id(font)] = key + continue + path, size = getattr(font, "path", None), getattr(font, "size", None) + if not path or not size: + continue + try: + fonts[key] = _IF.truetype(path, size) + except (OSError, ValueError, TypeError): + logger.debug( + "Could not un-share the %s face; it keeps the default colour", key) + return fonts diff --git a/src/common/sports_game_renderer.py b/src/common/sports_game_renderer.py new file mode 100644 index 00000000..d42faf3f --- /dev/null +++ b/src/common/sports_game_renderer.py @@ -0,0 +1,261 @@ +"""The scroll/Vegas card geometry the sports scoreboards all share. + +Eight scoreboards -- afl, baseball, basketball, football, hockey, lacrosse, +nrl and soccer -- each carried their own ``game_renderer.py``. After the card +helpers moved to ``src/common/sports_card.py`` the settings lookups were +shared, but the *geometry* was not: nine methods that decide how wide the +centre gap is, how much room a logo gets, and where an upcoming card's date +and time land were still eight separate copies. Five were byte-identical +across all eight plugins; the other four were identical in seven, with a +different single outlier each time. + +That last detail is why this is a mixin rather than free functions. There is +no per-sport branching to write -- baseball needs its own +``_draw_upcoming_game_status`` and ``_logo_slot_width``, hockey its own +``_upcoming_date_and_time``, football its own ``_score_reserve_width``, and +every one of those is an ordinary override. The seven that agree inherit and +say nothing. + +It is deliberately *only* a mixin: no ``__init__``, no state of its own. The +plugins' constructors differ in six ways and none of that difference is worth +unifying, so adoption is one line on the class statement plus deleting the +methods that now come from here. + +What a host class must provide +------------------------------ +Attributes: ``display_width``, ``display_height``, ``config``, ``fonts``, +``logger``, ``_team_rankings_cache``. + +Methods: ``_draw_text_with_outline(draw, text, position, font, fill=None, +outline_color=(0, 0, 0))`` -- the one hook whose body genuinely varies -- plus +the ``sports_card`` delegations ``_scroll_card_option``, +``_upcoming_center_mode``, ``_vs_text``, ``_element_color``, +``_format_game_date`` and ``_format_game_time``. +""" + +import math +from typing import ClassVar, Dict, Tuple + +from PIL import Image, ImageDraw + + +class SportsGameRendererMixin: + """Shared card geometry for the sports scoreboards. See module docstring.""" + + #: Centre strip as a fraction of card width, before clamping. + CENTER_GAP_RATIO: ClassVar[float] = 0.28 + #: Clamp floor for the derived centre gap. + CENTER_GAP_MIN_PX: ClassVar[int] = 22 + #: Clamp ceiling for the derived centre gap. + CENTER_GAP_MAX_PX: ClassVar[int] = 40 + #: Breathing room kept between the score and each logo. + _SCORE_LOGO_GUTTER_PX: ClassVar[int] = 4 + #: Widest score the centre strip must fit. Leagues that can reach three + #: digits a side override this with "000-000". + _SCORE_PROBE: ClassVar[str] = "00-00" + + # ---- geometry ------------------------------------------------------ + + # Non-finite settings are rejected before any int()/round(): "inf" reaches + # these from config as a float or a string, passes an `isinstance` plus + # `>= 0` check unharmed, and then raises OverflowError out of int() -- + # which the old `except (TypeError, ValueError)` did not catch, so it + # aborted the whole card render. Present in all eight plugins before this + # moved to the core; fixing it here fixes it in all eight. + + def _score_reserve_width(self) -> int: + """Centre strip the score actually needs, measured rather than assumed. + + The gap was derived from the card width alone (width x + CENTER_GAP_RATIO, clamped to CENTER_GAP_MAX_PX) while the score's size + comes from config and the element-style resolver. Nothing compared the + two, so any score wider than the clamp was drawn over the logos. + Measuring it keeps the strip wide enough for whatever font is in play. + """ + try: + probe = ImageDraw.Draw(Image.new("RGB", (4, 4))) + width = probe.textlength(self._SCORE_PROBE, font=self.fonts['score']) + return int(width) + 2 * self._SCORE_LOGO_GUTTER_PX + except Exception: + self.logger.debug("Score reserve measurement failed", exc_info=True) + return 0 + + def _center_gap_width(self) -> int: + """Width of the middle strip kept clear of logos. + + ``scroll_card.center_gap`` pins it outright; otherwise it scales with + the card width between the configurable min and max. 0 restores + edge-to-edge logos. + """ + configured = self._scroll_card_option("center_gap") + if (isinstance(configured, (int, float)) + and math.isfinite(configured) and configured >= 0): + return int(configured) + ratio = self._scroll_card_option("center_gap_ratio", self.CENTER_GAP_RATIO) + low = self._scroll_card_option("center_gap_min", self.CENTER_GAP_MIN_PX) + high = self._scroll_card_option("center_gap_max", self.CENTER_GAP_MAX_PX) + try: + ratio, low, high = float(ratio), float(low), float(high) + if not (math.isfinite(ratio) and math.isfinite(low) + and math.isfinite(high)): + return self.CENTER_GAP_MIN_PX + scaled = round(self.display_width * ratio) + derived = int(max(int(low), min(int(high), scaled))) + # A strip narrower than the score is the bug, not a style choice. + # An explicit ``center_gap`` is still honoured above, including 0. + return max(derived, self._score_reserve_width()) + except (TypeError, ValueError, OverflowError): + return self.CENTER_GAP_MIN_PX + + def _logo_slot_width(self) -> int: + """Per-side logo slot, leaving the center gap clear. + + No longer capped at display_height: the card is sized as two + full-height logos plus the measured gap, so what is left after the gap + is exactly the logo's share. The cap was what froze the logos at 46px + on the old flat 128px card. + """ + available = (self.display_width - self._center_gap_width()) // 2 + return max(8, available) + + def _logo_cache_key(self, name: str) -> str: + """Cache key scoped to the logo slot. + + One cache dict is shared by renderers built for different card widths, + so a logo sized for a wide slot must not be handed to a narrow one. + """ + return f"{name}@{self._logo_slot_width()}x{self.display_height}" + + def _layout_offset(self, element: str, axis: str, default: int = 0) -> int: + """X/Y nudge for one element, from customization.layout. + + Same block the full-screen scorebug reads (sports.py + _get_layout_offset), so a nudge configured in the web UI now moves + the element on the scroll/Vegas card too -- previously the schema + advertised these offsets but this renderer ignored them. + """ + try: + layout = (self.config or {}).get("customization", {}).get("layout", {}) + value = (layout.get(element) or {}).get(axis, default) + if isinstance(value, bool): + return default + if isinstance(value, (int, float)): + return int(value) if math.isfinite(value) else default + if isinstance(value, str): + parsed = float(value) + return int(parsed) if math.isfinite(parsed) else default + except (TypeError, ValueError, OverflowError): + pass + return default + + # ---- upcoming cards ------------------------------------------------ + + def _upcoming_date_and_time(self, game: Dict) -> Tuple[str, str]: + """(date, time) for an upcoming card, from the extractor's flat keys.""" + return ( + str(game.get("game_date", "") or ""), + str(game.get("game_time", "") or ""), + ) + + def _draw_upcoming_center(self, draw: "ImageDraw.ImageDraw", game: Dict) -> None: + """Draw the middle of an upcoming card. + + Never a score: an upcoming game has not started, so the extractor's + 0-0 is noise. Either the VS text (default), the date and time stacked, + or nothing at all. + """ + mode = self._upcoming_center_mode() + if mode == "none": + return + + if mode == "vs": + vs_text = self._vs_text() + if not vs_text: + return + vs_width = draw.textlength(vs_text, font=self.fonts['score']) + vs_x = (self.display_width - vs_width) // 2 + self._layout_offset('score', 'x_offset') + vs_y = (self.display_height // 2) - 3 + self._layout_offset('score', 'y_offset') + self._draw_text_with_outline( + draw, vs_text, (vs_x, vs_y), self.fonts['score'], + fill=self._element_color('score_text') + ) + return + + date_text, time_text = self._upcoming_date_and_time(game) + lines = [] + if self._scroll_card_option("show_date", True): + lines.append(self._format_game_date(date_text, game)) + if self._scroll_card_option("show_time", True): + lines.append(self._format_game_time(time_text)) + lines = [t for t in lines if t] + if not lines: + return + font = self.fonts.get('detail') or self.fonts['time'] + line_h = 7 + top = (self.display_height // 2) - (len(lines) * line_h) // 2 + top += self._layout_offset('score', 'y_offset') + for i, line in enumerate(lines): + width = draw.textlength(line, font=font) + x = (self.display_width - width) // 2 + self._layout_offset('score', 'x_offset') + self._draw_text_with_outline( + draw, line, (x, top + i * line_h), font, + fill=self._element_color('detail_text') + ) + + def _draw_upcoming_game_status(self, draw: "ImageDraw.ImageDraw", game: Dict) -> None: + """Draw the date and time around an upcoming card. + + Time top and date bottom by default; scroll_card.swap_date_time puts + the date on top instead. Skipped when the pair is stacked in the + middle, which would otherwise print them twice. + """ + if self._upcoming_center_mode() == "date_time": + return + + date_raw, time_raw = self._upcoming_date_and_time(game) + date_text = (self._format_game_date(date_raw, game) + if self._scroll_card_option("show_date", True) else "") + time_text = (self._format_game_time(time_raw) + if self._scroll_card_option("show_time", True) else "") + + if self._scroll_card_option("swap_date_time", False): + top_text, top_el, bottom_text, bottom_el = ( + date_text, 'date', time_text, 'time') + top_font = self.fonts.get('detail') or self.fonts['time'] + bottom_font = self.fonts['time'] + top_color, bottom_color = 'detail_text', 'period_text' + else: + top_text, top_el, bottom_text, bottom_el = ( + time_text, 'time', date_text, 'date') + top_font = self.fonts['time'] + bottom_font = self.fonts.get('detail') or self.fonts['time'] + top_color, bottom_color = 'period_text', 'detail_text' + + if top_text: + top_width = draw.textlength(top_text, font=top_font) + top_x = (self.display_width - top_width) // 2 + self._layout_offset(top_el, 'x_offset') + top_y = 1 + self._layout_offset(top_el, 'y_offset') + self._draw_text_with_outline( + draw, top_text, (top_x, top_y), top_font, + fill=self._element_color(top_color) + ) + + if bottom_text: + bottom_width = draw.textlength(bottom_text, font=bottom_font) + bottom_x = ((self.display_width - bottom_width) // 2 + + self._layout_offset(bottom_el, 'x_offset')) + # Measured, not a fixed -7: the detail font is 6px in most plugins + # but 10px in soccer and nrl, where "Sep 19" ran past the card. + ink_bottom = draw.textbbox((0, 0), bottom_text, font=bottom_font)[3] + bottom_y = (max(0, self.display_height - ink_bottom - 1) + + self._layout_offset(bottom_el, 'y_offset')) + self._draw_text_with_outline( + draw, bottom_text, (bottom_x, bottom_y), bottom_font, + fill=self._element_color(bottom_color) + ) + + # ---- rankings ------------------------------------------------------ + + def set_rankings_cache(self, rankings: Dict[str, int]) -> None: + """Set the team rankings cache for display.""" + self._team_rankings_cache = rankings diff --git a/src/common/sports_shared.py b/src/common/sports_shared.py new file mode 100644 index 00000000..b120e532 --- /dev/null +++ b/src/common/sports_shared.py @@ -0,0 +1,1304 @@ +"""The sports.py surface that is byte-identical in every scoreboard. + +Nine plugins ship their own ``sports.py`` -- 41,326 lines in total. Comparing +executable ASTs across the eight that share a lineage, 48 method bodies are +byte-identical in all eight: 1,007 lines carried in eight copies, so 8,056 +duplicated lines that must be edited eight times to fix once. + +They are the parts with no sport in them. The selection and rotation engine +(``_round_robin_favorites``, ``_favorites_first``, ``_compose_selection``, +``_check_ranking_coverage``, ``_game_divisions``, ``_normalise_quality``), the +font/colour/date subsystem (``_scale_headline_fonts``, ``_scorebug_font``, +``_resolve_font_size``, ``_format_game_date``, ``_font_color``), and the +switch-mode upcoming card (``_draw_upcoming_center_switch``). Nothing here knows +what an inning or a possession is. + +Mixins rather than free functions, because every one of these reads host state +-- ``self.config``, ``self.fonts``, ``self.logger``, ``self.display_width``. +Rewriting 48 bodies into free functions would be a rewrite, not a move; as +mixins the bodies move verbatim, which is what keeps the renders identical. + +THREE OF THE 48 ARE DELIBERATELY LEFT BEHIND +-------------------------------------------- +Byte-identical bodies are not automatically safe to move: a body can bind a +module-level name that differs per plugin, and then it only *looks* the same. + +- ``_get_timezone`` calls ``resolve_timezone``, imported from a per-plugin + module (``hockey_timezone``, ``soccer_timezone``, ...). All eight of those + differ -- each carries its own ``_WRITEBACK_FIXED_IN`` version -- so moving + the caller here would silently bind every scoreboard to one plugin's copy. +- ``_extract_game_details`` and ``_fetch_data`` are ``@abstractmethod`` stubs. + They are the sport-specific contract; satisfying them from a mixin would let a + plugin instantiate without implementing its own sport. + +``_resolve_font_path`` went the other way: it is a module-level function in +sports.py rather than a method, identical in all eight, and ``_scale_headline_fonts`` +needs it -- so it is inlined below rather than left behind. + +WHAT A HOST MUST PROVIDE +------------------------ +Enumerated by walking every ``self.`` the mixins read and subtracting what +they define, so this list is derived rather than remembered. Everything below is +supplied by all eight scoreboards today. + +State: ``config``, ``fonts``, ``logger``, ``display_width``, ``display_height``, +``display_manager``, ``league``, ``sport``, ``mode_config``, ``session``, +``headers``, ``favorite_teams``, ``games_list``, ``current_game_index``, +``last_game_switch``, ``last_update``, ``update_interval``, +``no_data_interval``, ``game_display_duration``, ``stale_game_timeout``, +``other_games_min_quality``, ``schedule_lookback_days``, +``schedule_lookahead_days``, ``game_update_timestamps``, +``_zero_clock_timestamps``, ``_logo_cache``, ``_selection_pools``, +``_ranking_coverage_logged_at``, ``_empty_live_streak``, ``_last_warning_time``, +``_score_grew``. + +Methods that stay per-plugin, because they are not identical across the eight +(or, for ``_get_timezone``, because they bind per-plugin modules): +``_get_layout_offset``, ``_by_importance``, ``_other_games_window``, +``_upcoming_date_and_time_text``, ``_extract_game_details_common``, +``_load_division_team_ids``, ``_get_timezone``, ``_is_favorite_game``, +``_is_game_really_over``, ``_is_ranked_game``, ``_passes_other_filters``. + +Of the fourteen shared class constants, thirteen are identical everywhere and +live here. Only ``_SCORE_PROBE_TEXT`` varies -- afl and basketball reach three digits +a side and override it, the same two that override ``_SCORE_PROBE`` on +``SportsGameRendererMixin``. + +DELIBERATELY NOT MERGED WITH sports_card +---------------------------------------- +Fourteen of these have same-named twins in ``src/common/sports_card.py``, which +the scoreboards' ``game_renderer.py`` already uses. They are NOT wired together +here. Only five are provably equivalent by source comparison; the other nine +differ in ways inspection cannot settle, and a wrong guess silently changes what +every scoreboard draws. Merging them needs differential testing against both +implementations, and is left for its own change. +""" + +from __future__ import annotations + +import logging +import os +import sys +import time +from datetime import datetime, timedelta, timezone +from typing import Any, ClassVar, Dict, List, Optional, Tuple + +import pytz +import requests +from PIL import Image, ImageDraw, ImageFont + +logger = logging.getLogger(__name__) + +# How long a live mode may stretch its poll interval when nothing is happening, +# and the streak lengths that earn each stretch. Identical in all eight plugins. +_IDLE_SHORT_STREAK = 6 +_IDLE_SHORT_FACTOR = 2 +_IDLE_LONG_STREAK = 24 +_IDLE_LONG_FACTOR = 6 +_DEFAULT_LIVE_IDLE_MAX_SECONDS = 900 + + +def _resolve_font_path(path: str) -> str: + """Resolve a bundled font path without depending on the process cwd. + + These fonts ship with the LEDMatrix core, and every call site here named + them relative to the working directory. That holds under the packaged + systemd unit, whose WorkingDirectory is the install root, and breaks + everywhere else -- the plugin safety harness, a manual run from $HOME, a + unit file written without WorkingDirectory. The failure is quiet: the + load raises, the caller falls back, and the scoreboard renders in PIL's + default face instead of the pixel font it was laid out for. + + Resolution order matches the core's own resolver: the path as given + first, so behaviour is unchanged wherever it already worked and a + configured absolute path is returned untouched, then the core install + root, then the original string so callers still raise and fall back + exactly as they do today. + """ + if os.path.exists(path): + return path + try: + import src.font_manager as _core_fonts + + # The core grew this resolver in ChuckBuilds/LEDMatrix#425. Use it + # when it is there so both repos stay on one definition of "install + # root"; older cores fall through to the equivalent derivation below. + manager = getattr(_core_fonts, "FontManager", None) + resolver = getattr(manager, "_resolve_asset_path", None) + if resolver is not None: + resolved = resolver(path) + if resolved and os.path.exists(resolved): + return resolved + root = os.path.dirname(os.path.dirname(os.path.abspath(_core_fonts.__file__))) + candidate = os.path.join(root, path) + if os.path.exists(candidate): + return candidate + except (ImportError, AttributeError, OSError): + # No core on the path (standalone tooling), a core laid out + # differently, or an unreadable install. Returning the original keeps + # the caller's existing fallback intact. + return path + return path + + +class SportsCoreSharedMixin: + """The ``SportsCore`` bodies identical in all eight scoreboards.""" + + #: Design height the font scale is expressed against. + _FONT_DESIGN_HEIGHT: ClassVar[int] = 32 + #: Fraction of the centre strip a score may grow into. + _SCORE_GROWTH_BUDGET: ClassVar[float] = 0.65 + #: Widest score the scorebug sizes itself to hold. Leagues that reach three + #: digits a side override this with "000-000". + _SCORE_PROBE_TEXT: ClassVar[str] = "00-00" + #: Whether this sport's scorebug draws a score at all. + _DRAWS_SCORE: ClassVar[bool] = False + #: Fallback (font, size) rungs for a score that will not fit. + _NARROW_SCORE_RUNGS: ClassVar[Tuple[Tuple[str, int], ...]] = ( + ("4x6-font.ttf", 14), ("4x6-font.ttf", 7)) + #: Hard ceiling on score growth, in multiples of the configured size. + _SCORE_MAX_GROWTH: ClassVar[int] = 2 + #: Which colour setting owns each font slot. + _ELEMENT_FOR_FONT: ClassVar[Dict[str, str]] = { + "score": "score_text", "time": "period_text", "team": "team_text", + "detail": "detail_text", "status": "status_text"} + #: Default tint for a favourite team's finished game. + FAVORITE_RESULT_COLOR_DEFAULTS: ClassVar[Dict[str, Tuple[int, int, int]]] = { + "win": (0, 255, 0), "loss": (255, 0, 0), "tie": (255, 200, 0)} + _MONTH_ABBR: ClassVar[Tuple[str, ...]] = ( + "Jan", "Feb", "Mar", "Apr", "May", "Jun", + "Jul", "Aug", "Sep", "Oct", "Nov", "Dec") + _WEEKDAY_ABBR: ClassVar[Tuple[str, ...]] = ( + "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun") + #: Bitmap fonts snap to their native pixel grid. + _FONT_PIXEL_GRID: ClassVar[Dict[str, int]] = { + "PressStart2P-Regular.ttf": 8, "4x6-font.ttf": 7} + _FONT_NAME_ALIASES: ClassVar[Dict[str, str]] = { + "press_start": "PressStart2P-Regular.ttf", "four_by_six": "4x6-font.ttf"} + #: Accepted values for the other-games quality filter. + _QUALITY_CHOICES: ClassVar[frozenset] = frozenset({"any", "ranked"}) + #: How long to stay quiet between ranking-coverage warnings. + _RANKING_COVERAGE_SECONDS: ClassVar[int] = 60 * 60 + + def _get_season_schedule_dates(self) -> tuple[str, str]: + return "", "" + + def _draw_scorebug_layout(self, game: Dict, force_clear: bool = False) -> None: + """Placeholder draw method - subclasses should override.""" + # This base method will be simple, subclasses provide specifics + try: + img = Image.new("RGB", (self.display_width, self.display_height), (0, 0, 0)) + draw = ImageDraw.Draw(img) + status = game.get("status_text", "N/A") + self._draw_text_with_outline(draw, status, (2, 2), self.fonts["status"]) + self.display_manager.image.paste(img, (0, 0)) + # Don't call update_display here, let subclasses handle it after drawing + except Exception as e: + self.logger.error( + f"Error in base _draw_scorebug_layout: {e}", exc_info=True + ) + + @classmethod + def _crisp_size(cls, font_file, desired): + """Snap *desired* to the nearest size *font_file* renders crisply at. + + A face with no known grid is returned unchanged, so a user-supplied + font is never second-guessed. + """ + font_file = cls._FONT_NAME_ALIASES.get(font_file, font_file) + grid = cls._FONT_PIXEL_GRID.get(font_file) + if not grid or not desired or desired <= 0: + return desired + return max(grid, int(round(float(desired) / grid)) * grid) + + #: Absolute path of this plugin's directory, declared by the plugin + #: itself. The mixin cannot work it out -- see _plugin_dir. + _PLUGIN_DIR: ClassVar[Optional[str]] = None + + def _plugin_dir(self) -> Optional[str]: + """Directory of the plugin that owns this instance. + + In sports.py these methods could just use ``__file__``. Here that is + src/common/, so the directory has to come from the plugin. + + It is TOLD, not deduced. The first version walked the MRO for a class + whose module sits beside a config_schema.json. That works when a test + imports the plugin itself, and returns None under the real loader, for + a specific reason worth recording: + + PluginLoader._namespace_plugin_modules renames every bare module a + plugin brought in (sports, game_renderer, ...) to + "_plg__" and REMOVES the bare entry, so two + plugins owning a module of the same name cannot collide. + + The class still reports ``__module__ == "sports"``, but + ``sys.modules["sports"]`` no longer exists, so the walk finds no + __file__ and falls off the end. The failure was silent and expensive: + + _plugin_dir() -> None + _schema_font_size() -> None for every element + -> a configured size equal to the schema default stops looking + like a default and is treated as a deliberate user choice + -> the snap to the font's pixel grid is skipped + -> 4x6-font.ttf renders at 6 instead of 7: 3px-wide glyphs + instead of 4px + + On a 256x64 panel that made the odds, the team records and the date row + hard to read. It was found by a user counting pixels on the panel. No + gate here caught it: the tests imported plugins directly and the + safety harness loads them its own way, so neither reproduced the + loader's renaming. + + The MRO walk stays as a fallback for hosts that declare no + _PLUGIN_DIR -- the plugins' own probe harnesses build classes with + ``type()`` -- but it is no longer the primary answer. + """ + declared = getattr(self, "_PLUGIN_DIR", None) + if declared and os.path.isfile(os.path.join(declared, "config_schema.json")): + return declared + + for cls in type(self).__mro__: + module = sys.modules.get(getattr(cls, "__module__", ""), None) + path = getattr(module, "__file__", None) + if not path: + continue + directory = os.path.dirname(os.path.abspath(path)) + if os.path.isfile(os.path.join(directory, "config_schema.json")): + return directory + return None + + def _schema_font_size(self, element_key): + """The font_size this plugin's config_schema.json declares, or None.""" + if not element_key: + return None + cache = getattr(self.__class__, '_SCHEMA_FONT_SIZES', None) + if cache is None: + cache = {} + try: + import json + directory = self._plugin_dir() + if directory is None: + raise FileNotFoundError("no config_schema.json on the MRO") + with open(os.path.join(directory, 'config_schema.json')) as fh: + schema = json.load(fh) + props = (schema.get('properties', {}) + .get('customization', {}) + .get('properties', {})) + for key, spec in props.items(): + size = spec.get('properties', {}).get('font_size', {}).get('default') + if size is not None: + cache[key] = int(size) + except Exception as exc: + # Say so. An unreadable schema is not cosmetic: every element's + # configured size then stops matching "the schema default", is + # treated as a deliberate user choice, and skips the snap to the + # font's pixel grid -- which renders 4x6-font.ttf at 6 instead + # of 7, a 3px-wide glyph instead of 4px. That shipped once, + # silently, and was found by a user counting pixels on a photo + # of the panel. + # + # Logged, not raised: a missing schema must not stop a plugin + # rendering. The cache is built once per class, so this cannot + # repeat per frame. + logger.warning( + "%s: could not read config_schema.json (%s: %s); every font " + "size will be treated as user-chosen and will skip its pixel " + "grid snap. Font sizes may render a pixel narrow.", + type(self).__name__, type(exc).__name__, exc) + cache = {} + self.__class__._SCHEMA_FONT_SIZES = cache + return cache.get(element_key) + + def _resolve_font_size(self, element_config, element_key, default_size, font_name): + """Size to render at: the user's choice, or a grid-snapped default. + + A configured size counts as a real choice only when it differs from + the schema default. The web UI writes the whole schema default block + on every save, so "font_size == schema default" carries no intent and + would otherwise pin every install to an anti-aliased size forever. + """ + configured = (element_config or {}).get('font_size') + if configured is not None: + try: + configured = int(configured) + if configured != self._schema_font_size(element_key): + return configured + except (TypeError, ValueError): + pass + return self._crisp_size(font_name, default_size) + + def _card_option(self, key: str, default: Any = None) -> Any: + """Read one key from the scroll_card config block.""" + block = (self.config or {}).get("scroll_card") + if isinstance(block, dict) and block.get(key) is not None: + return block.get(key) + return default + + def _switch_upcoming_center(self) -> str: + """Middle of the full-screen upcoming scorebug: 'vs', 'date_time' or 'none'.""" + mode = str(self._card_option("switch_upcoming_center", "date_time") + or "date_time").lower() + if mode == "inherit": + mode = str(self._card_option("upcoming_center", "vs") or "vs").lower() + return mode if mode in ("vs", "date_time", "none") else "date_time" + + def _vs_text(self) -> str: + """Separator drawn between the teams -- "VS", "@", "at", anything.""" + return str(self._card_option("vs_text", "VS")) + + def _switch_date_format(self) -> str: + """Date style for the full-screen scorebug. + + Its own key rather than the shared ``date_format`` because the two + displays disagree about the default: the scroll card renders "Sep 19" + while _extract_game_details_common emits "9/19", the "numeric" style, + and this scorebug has always drawn it. Reading the shared key here + would restyle every existing panel on update -- and "leave it alone + when unset" is not available, because the core merges schema defaults + into the config on every load, so the key is never actually unset. + "inherit" opts into the scroll and Vegas setting. + """ + fmt = str(self._card_option("switch_date_format", "numeric") or "numeric").lower() + if fmt == "inherit": + fmt = str(self._card_option("date_format", "abbrev") or "abbrev").lower() + return fmt + + def _format_game_date(self, date_text: str, game: Optional[Dict] = None) -> str: + """Format an upcoming date per scroll_card.switch_date_format.""" + raw = str(date_text or "").strip() + if not raw: + return raw + fmt = self._switch_date_format() + if fmt == "numeric": + return raw + parts = raw.replace("-", "/").split("/") + if not (len(parts) >= 2 and parts[0].strip().isdigit() and parts[1].strip().isdigit()): + return raw + month, day = int(parts[0]), int(parts[1]) + if not 1 <= month <= 12: + return raw + name = self._MONTH_ABBR[month - 1] + if fmt == "numeric_day_first": + return f"{day}/{month}" + if fmt == "day_first": + return f"{day} {name}" + if fmt == "weekday": + weekday = self._weekday_for(game) + return f"{weekday} {name} {day}" if weekday else f"{name} {day}" + return f"{name} {day}" + + def _weekday_for(self, game: Optional[Dict]) -> str: + """Weekday abbreviation from the game's start time, or ''.""" + if not game: + return "" + raw = game.get("start_time_utc") or game.get("start_time") + if not raw: + return "" + try: + start = raw if isinstance(raw, datetime) else datetime.fromisoformat( + str(raw).replace("Z", "+00:00")) + return self._WEEKDAY_ABBR[start.astimezone(self._get_timezone()).weekday()] + except (ValueError, TypeError, OverflowError): + return "" + + def _format_game_time(self, time_text: str) -> str: + """Return the time as-is (12h) or converted to 24h.""" + raw = str(time_text or "").strip() + if not raw or str(self._card_option("time_format", "12h")) != "24h": + return raw + cleaned = raw.upper().replace(" ", "") + meridiem = "AM" if cleaned.endswith("AM") else "PM" if cleaned.endswith("PM") else "" + if not meridiem: + return raw + try: + hh, _, mm = cleaned[:-2].partition(":") + hour, minute = int(hh), int(mm or 0) + except ValueError: + return raw + if not (0 <= hour <= 12 and 0 <= minute <= 59): + return raw + hour = hour % 12 + (12 if meridiem == "PM" else 0) + return f"{hour:02d}:{minute:02d}" + + def _scorebug_font(self, draw, text: str, width: int): + """The face this scorebug draws its date and time in. + + Always the "time" face, which is what this display has used for both + rows for as long as it has existed: changing switch_upcoming_center + moves the two lines around, it is not meant to restyle them, so the + type stays put while the placement changes. + + The single exception is text that cannot fit the panel at all. Only + the "weekday" date can do that -- "Fri Sep 19" measures 80px in an + 8px face, on a board 64px wide -- and the smaller "detail" face is a + better answer there than running off both edges. Every other date and + time this display can produce fits, so in practice the face never + changes; it is a floor, not a style rule. + """ + font = self.fonts["time"] + if not text: + return font + try: + if draw.textlength(text, font=font) + 2 <= width: + return font + except (TypeError, ValueError): + return font + return self.fonts.get("detail") or font + + def _draw_upcoming_center_switch(self, draw, game: Dict, center_y: int, + game_date: str, game_time: str, + display_width: Optional[int] = None, + display_height: Optional[int] = None, + date_element: str = 'date', + time_element: str = 'time', + second_row_y_offset: bool = True) -> bool: + """Draw the middle of the full-screen upcoming scorebug. + + Returns True when the header above it ("Next Game", or the league + name) should still be drawn. In "vs" and "none" the date and time move + out of the middle and into the top and bottom slots, mirroring the + scroll card -- and the top slot is where the header used to be, so the + caller drops it. + + ``date_element``/``time_element``/``second_row_y_offset`` exist only so + the layout-offset keys stay exactly what each plugin's schema + advertises; this sport's defaults are the common case. + """ + width = self.display_width if display_width is None else display_width + height = self.display_height if display_height is None else display_height + mode = self._switch_upcoming_center() + date_text, time_text = self._upcoming_date_and_time_text( + game_date, game_time, game) + swapped = bool(self._card_option("swap_date_time", False)) + + if mode == "date_time": + # Historically the date sat at center_y - 7 with the time 9px + # under it, and the time's row was derived from the date's, so a + # date y_offset moved the pair. Both still hold; the slots only + # trade places when swap_date_time is set, and hiding one line + # leaves the other where it was rather than re-centering the stack. + slots = [(time_element, time_text), (date_element, date_text)] if swapped \ + else [(date_element, date_text), (time_element, time_text)] + row_y = center_y - 7 + for index, (element, text) in enumerate(slots): + if index: + row_y += 9 + if second_row_y_offset: + row_y += self._get_layout_offset(element, 'y_offset') + else: + row_y += self._get_layout_offset(element, 'y_offset') + if not text: + continue + font = self._scorebug_font(draw, text, width) + text_width = draw.textlength(text, font=font) + text_x = ((width - text_width) // 2 + + self._get_layout_offset(element, 'x_offset')) + self._draw_text_with_outline( + draw, text, (text_x, row_y), font + ) + return True + + if mode == "vs": + vs_text = self._vs_text() + if vs_text: + vs_width = draw.textlength(vs_text, font=self.fonts["score"]) + vs_x = ((width - vs_width) // 2 + + self._get_layout_offset('score', 'x_offset')) + vs_y = (center_y - 3 + + self._get_layout_offset('score', 'y_offset')) + self._draw_text_with_outline( + draw, vs_text, (vs_x, vs_y), self.fonts["score"] + ) + + # "vs" and "none" both push the date and time out to the edges, time + # on top unless swap_date_time says otherwise -- the same order the + # scroll card uses. + if swapped: + top_element, top_text = date_element, date_text + bottom_element, bottom_text = time_element, time_text + else: + top_element, top_text = time_element, time_text + bottom_element, bottom_text = date_element, date_text + + if top_text: + top_font = self._scorebug_font(draw, top_text, width) + top_width = draw.textlength(top_text, font=top_font) + top_x = ((width - top_width) // 2 + + self._get_layout_offset(top_element, 'x_offset')) + top_y = 1 + self._get_layout_offset(top_element, 'y_offset') + self._draw_text_with_outline( + draw, top_text, (top_x, top_y), top_font + ) + if bottom_text: + bottom_font = self._scorebug_font(draw, bottom_text, width) + bottom_width = draw.textlength(bottom_text, font=bottom_font) + bottom_x = ((width - bottom_width) // 2 + + self._get_layout_offset(bottom_element, 'x_offset')) + # Measured, not a fixed offset: the detail font is 6px in most + # plugins and 10px in soccer and nrl, where a fixed -7 ran the + # date off the panel. + ink_bottom = draw.textbbox((0, 0), bottom_text, font=bottom_font)[3] + bottom_y = (max(0, height - ink_bottom - 1) + + self._get_layout_offset(bottom_element, 'y_offset')) + self._draw_text_with_outline( + draw, bottom_text, (bottom_x, bottom_y), bottom_font + ) + return False + + @staticmethod + def _coerce_rgb(value, fallback): + """Turn a configured [R, G, B] list into a clamped (r, g, b) tuple.""" + # Checked before unpacking: a 3-character string ("123") would otherwise + # iterate into three digits and yield a colour rather than the fallback. + if not isinstance(value, (list, tuple)) or len(value) != 3: + return fallback + try: + r, g, b = (max(0, min(255, int(channel))) for channel in value) + except (TypeError, ValueError): + return fallback + return (r, g, b) + + @staticmethod + def _side_is_favorite(game: Dict, side: str, favorites: set) -> bool: + """Is the home/away side of this game a favorite team? + + Both the abbreviation and the ESPN id are checked, because a couple of + leagues (NRL) match favorites by id where abbreviations collide. + """ + for key in (f"{side}_abbr", f"{side}_id"): + value = game.get(key) + if value is not None and str(value).strip().upper() in favorites: + return True + return False + + def _favorite_result(self, game: Dict) -> Optional[str]: + """Say how the favorite team did in a finished game. + + Returns 'win', 'loss' or 'tie', or None when there is no single team + to root for: no favorites configured, neither side is a favorite, or + *both* are -- a favorite-vs-favorite game has no losing side worth + flagging in red. Also None when the scores are not usable numbers. + """ + favorites = getattr(self, "favorite_teams", None) or [] + favorites = {str(team).strip().upper() for team in favorites if str(team).strip()} + if not favorites: + return None + + home_fav = self._side_is_favorite(game, "home", favorites) + away_fav = self._side_is_favorite(game, "away", favorites) + if home_fav == away_fav: + return None + + try: + # int(float(...)) to match GameRenderer._side_score exactly -- the + # two paths must agree on what counts as a usable score. + home_score = int(float(str(game.get("home_score", "")).strip())) + away_score = int(float(str(game.get("away_score", "")).strip())) + except (TypeError, ValueError): + return None + + if home_score == away_score: + return "tie" + favorite_score, other_score = ( + (home_score, away_score) if home_fav else (away_score, home_score) + ) + return "win" if favorite_score > other_score else "loss" + + def _recent_score_color(self, game: Dict, default): + """Fill color for a finished game's score, per favorite_result_colors.""" + try: + settings = (self.config.get("customization") or {}).get( + "favorite_result_colors" + ) or {} + if not settings.get("enabled", False): + return default + result = self._favorite_result(game) + if result is None: + return default + return self._coerce_rgb( + settings.get(f"{result}_color"), + self.FAVORITE_RESULT_COLOR_DEFAULTS[result], + ) + except Exception: + self.logger.debug( + "Could not resolve favorite result color", exc_info=True + ) + return default + + def _score_font_size(self) -> int: + """Pixel size the score is currently drawn at.""" + return getattr(self.fonts.get("score"), "size", 8) or 8 + + def _time_font_size(self) -> int: + """Pixel size the clock/date face is currently drawn at.""" + return getattr(self.fonts.get("time"), "size", 8) or 8 + + def _user_chose_size(self, element_key: str) -> bool: + """True when customization..font_size is a real choice. + + The web UI's save flow writes the whole schema default block into + config.json on every save, whether or not the user touched that + section, so a size merely being PRESENT carries no intent. Only one + that differs from the schema default does. + """ + element = (self.config.get('customization', {}) or {}).get(element_key) or {} + configured = element.get('font_size') + if configured is None: + return False + try: + return int(configured) != self._schema_font_size(element_key) + except (TypeError, ValueError): + return False + + def _grid_scaled_size(self, font): + """(path, grid, size) for *font* regrown to this panel's height. + + None when the panel is at or below the design height (nothing to do), + or when the face has no known pixel grid -- a user-supplied font is + never second-guessed, because we do not know what it renders crisply + at. + """ + path = getattr(font, 'path', None) + base = getattr(font, 'size', None) + if not base or not isinstance(path, str): + return None + face = os.path.basename(path) + grid = self._FONT_PIXEL_GRID.get(self._FONT_NAME_ALIASES.get(face, face)) + if not grid: + return None + scale = float(self.display_height) / (self._FONT_DESIGN_HEIGHT or 32) + if scale <= 1.0: + return None + return path, grid, max(int(base), int(self._crisp_size(face, base * scale))) + + def _scale_headline_fonts(self, fonts): + """Grow the score with the panel, and hold the clock/date below it. + + The score is the one number the card exists to show, and it was the + only element not sized from the panel. Worse, it was not even bigger + than its neighbours: PressStart2P renders crisply on an 8px grid, so + the 10px default snapped to 8 -- the same 8 the period/clock above it + and the game date below it are drawn at. Three lines of identical + type, none of them the headline, which is what makes the score read as + lower priority than the time and the date rather than the point of the + card. + + So the score is sized from display_height and snapped to its face's + pixel grid (off the grid FreeType anti-aliases the strokes, and on an + LED matrix a part-lit pixel is a dim lamp rather than a soft edge), + then stepped back down that grid until it fits its share of the width. + The clock/date face is regrown the same way but held at least one grid + step below the score, so the ranking between them is visible rather + than implied. + + A 32-tall panel scales by exactly 1.0 and is left byte-identical; a + size the user set explicitly is never overridden. + """ + self._score_grew = False + if not self._DRAWS_SCORE: + # No score on this screen, so none of the sizing below is for it. + return fonts + try: + scaled = None if self._user_chose_size('score_text') else \ + self._grid_scaled_size(fonts.get('score')) + if scaled is not None: + path, grid, size = scaled + base = getattr(fonts['score'], 'size', size) or size + size = min(size, base * self._SCORE_MAX_GROWTH) + probe = ImageDraw.Draw(Image.new('RGB', (4, 4))) + budget = self.display_width * self._SCORE_GROWTH_BUDGET + # Measured from a fixed five-character score rather than the + # live one, so the card does not resize when a side passes 9. + while size > grid: + if probe.textlength( + self._SCORE_PROBE_TEXT, + font=ImageFont.truetype(path, size)) <= budget: + break + size -= grid + if size != getattr(fonts['score'], 'size', size): + fonts['score'] = ImageFont.truetype(path, size) + self._score_grew = True + + if not self._score_grew and not self._user_chose_size('score_text') \ + and self.display_height > self._FONT_DESIGN_HEIGHT: + # PressStart2P could not grow inside the budget -- its next crisp + # size is simply too wide for this panel. A narrower face still + # can: 4x6-font at 14px is nearly as tall as PressStart2P at 16 + # and about half as wide. This matters beyond the score itself, + # because a card whose score never grows never reserves the + # centre either, so its logos stay at the uncapped 1.5x and are + # drawn straight over the score -- which is what a three-digit + # basketball score does on a 128x64 board. + probe = ImageDraw.Draw(Image.new('RGB', (4, 4))) + budget = self.display_width * self._SCORE_GROWTH_BUDGET + current = getattr(fonts.get('score'), 'size', 0) or 0 + for _name, _size in self._NARROW_SCORE_RUNGS: + if _size <= current: + continue + _path = _resolve_font_path(f"assets/fonts/{_name}") + _candidate = ImageFont.truetype(_path, _size) + if probe.textlength(self._SCORE_PROBE_TEXT, + font=_candidate) <= budget: + fonts['score'] = _candidate + self._score_grew = True + break + + scaled = None if self._user_chose_size('period_text') else \ + self._grid_scaled_size(fonts.get('time')) + if scaled is not None: + path, grid, size = scaled + ceiling = getattr(fonts.get('score'), 'size', 0) or 0 + if ceiling and size >= ceiling: + size = max(grid, ceiling - grid) + if size != getattr(fonts['time'], 'size', size): + fonts['time'] = ImageFont.truetype(path, size) + except Exception: + self.logger.debug("Headline font scaling skipped", exc_info=True) + return fonts + + def _element_color(self, element: str, default: Tuple[int, int, int] = (255, 255, 255)): + """Per-element text colour from customization..text_color.""" + try: + cfg = (self.config or {}).get("customization", {}).get(element, {}) + value = cfg.get("text_color") + if isinstance(value, (list, tuple)) and len(value) == 3: + return tuple(max(0, min(255, int(c))) for c in value) + if isinstance(value, str) and value.startswith("#") and len(value) == 7: + return tuple(int(value[i:i + 2], 16) for i in (1, 3, 5)) + except (TypeError, ValueError): + pass + return default + + def _unshare_element_fonts(self, fonts): + """Give each colourable element its own face object. + + The colour a draw gets is resolved from the face it was handed, and + several of these loaders legitimately hand one object to more than one + element -- a size resolver that lands two elements on the same face, a + fallback that fills every key from one default, football's narrowing + step that deliberately shrinks the clock along with the score. Sharing + the object makes the element ambiguous and the colour unresolvable. + + Re-instantiating from the same path and size gives a distinct object + with identical metrics, so nothing about the rendering changes; only + the ability to tell two elements apart does. Faces that cannot be + rebuilt (a BDF loaded through freetype.Face, anything without a usable + path) are left shared, and their draws stay white as before. + """ + try: + from PIL import ImageFont as _IF + except ImportError: # pragma: no cover + return fonts + seen = {} + for key in self._ELEMENT_FOR_FONT: + font = fonts.get(key) + if font is None: + continue + if id(font) not in seen: + seen[id(font)] = key + continue + path, size = getattr(font, "path", None), getattr(font, "size", None) + if not path or not size: + continue + try: + fonts[key] = _IF.truetype(path, size) + except (OSError, ValueError, TypeError): + self.logger.debug( + "Could not un-share the %s face; it keeps the default colour", key) + return fonts + + def _font_color(self, font, default: Tuple[int, int, int] = (255, 255, 255)): + """Colour for whichever element owns this face. + + Matched on identity, and deliberately gives up when one object is + shared: the last-resort font path can hand the same face to several + keys, and there is no right answer for which element's colour that is. + White is what those draws used before, so ambiguity costs nothing. + """ + try: + fonts = getattr(self, "fonts", None) or {} + matches = [element for key, element in self._ELEMENT_FOR_FONT.items() + if fonts.get(key) is font] + if len(matches) == 1: + return self._element_color(matches[0], default) + except (AttributeError, TypeError): + pass + return default + + def _draw_text_with_outline( + self, draw, text, position, font, fill=None, outline_color=(0, 0, 0) + ): + """Draw text with a black outline for better readability.""" + # Disable anti-aliasing: pixel/bitmap fonts (e.g. PressStart2P) get + # anti-aliased into dim partial-lit pixels on a 1:1 LED matrix, muddying + # glyphs. 1-bit mode keeps strokes crisp. + # Defaults to the configured colour for whichever element owns + # this face rather than to white, so customization..text_color + # reaches every draw. The schema has offered those pickers all along + # and they only ever changed the font. An explicit fill still wins: + # the odds colours and the favourite-result score tint mean something + # the palette does not. + if fill is None: + fill = self._font_color(font) + draw.fontmode = "1" + x, y = position + for dx, dy in [ + (-1, -1), + (-1, 0), + (-1, 1), + (0, -1), + (0, 1), + (1, -1), + (1, 0), + (1, 1), + ]: + draw.text((x + dx, y + dy), text, font=font, fill=outline_color) + draw.text((x, y), text, font=font, fill=fill) + + def _should_log(self, warning_type: str, cooldown: int = 60) -> bool: + """Check if we should log a warning based on cooldown period.""" + current_time = time.time() + if current_time - self._last_warning_time > cooldown: + self._last_warning_time = current_time + return True + return False + + def _get_weeks_data(self) -> Optional[Dict]: + """ + Get partial data for immediate display while background fetch is in progress. + This fetches current/recent games only for quick response. + """ + try: + # Fetch current week and next few days for immediate display + now = datetime.now(pytz.utc) + immediate_events = [] + + start_date = now - timedelta(days=self.schedule_lookback_days) + end_date = now + timedelta(days=self.schedule_lookahead_days) + date_str = f"{start_date.strftime('%Y%m%d')}-{end_date.strftime('%Y%m%d')}" + url = f"https://site.api.espn.com/apis/site/v2/sports/{self.sport}/{self.league}/scoreboard" + response = self.session.get( + url, + params={"dates": date_str, "limit": 1000}, + headers=self.headers, + timeout=10, + ) + response.raise_for_status() + data = response.json() + immediate_events = data.get("events", []) + + if immediate_events: + self.logger.info(f"Fetched {len(immediate_events)} events {date_str}") + return {"events": immediate_events} + + except requests.exceptions.RequestException as e: + self.logger.warning( + f"Error fetching this weeks games for {self.sport} - {self.league} - {date_str}: {e}" + ) + return None + + def _custom_scorebug_layout(self, game: dict, draw_overlay: ImageDraw.ImageDraw): + pass + + def cleanup(self): + """Clean up resources when plugin is unloaded.""" + # Close HTTP session + if hasattr(self, 'session') and self.session: + try: + self.session.close() + except Exception as e: + self.logger.warning(f"Error closing session: {e}") + + # Clear caches + if hasattr(self, '_logo_cache'): + self._logo_cache.clear() + + self.logger.info(f"{self.__class__.__name__} cleanup completed") + + def _game_divisions(self, game: Dict) -> Optional[set]: + """Divisions of BOTH sides, or None when they cannot be told. + + Both sides are collected, but the caller only needs ONE of them to sit + in a checked division. Requiring every participant read as "FBS games + only" and removed a ranked side hosting an FCS school -- which is still + a game involving a team the viewer checked the box for, and on a real + Week 2 slate it silently dropped five of the twenty ranked matchups. + What the checkbox is for is keeping FCS-versus-FCS out of a board + configured for FBS, and that still holds: a game with no checked + division on either side is dropped. + """ + divisions = self._load_division_team_ids() + if not any(divisions.values()): + return None + try: + ids = [int(game.get("home_id")), int(game.get("away_id"))] + except (TypeError, ValueError): + return None + present = set() + for team_id in ids: + for name in ("fbs", "fcs"): + if team_id in divisions.get(name, set()): + present.add(name) + break + else: + present.add("other") + return present + + def _league_has_rankings(self) -> bool: + """Only college leagues publish a poll; everyone else 404s. + + This gate matters more than it looks. _fetch_team_rankings only + short-circuits when the cache is non-empty, so a failed fetch leaves it + empty and the next update tries again -- at a 30s interval that is + ~2,900 pointless requests a day, per league, all of them 404s. + """ + league = (self.league or "").lower() + return "college" in league or "ncaa" in league + + @staticmethod + def _normalise_divisions(raw) -> List[str]: + """Division names from config, in the shape the filter expects. + + A hand-edited config can hold "fbs" where the schema says ["fbs"], and + list("fbs") is ['f', 'b', 's'] -- three names that match no division, so + every non-favourite game is rejected by a setting the user believes says + the opposite. An empty list is left empty: that means "no division + filter" and is a legitimate choice, not a mistake to correct. + """ + if isinstance(raw, str): + raw = [raw] + try: + items = list(raw or []) + except TypeError: + return [] + return [str(d).strip().lower() for d in items if str(d).strip()] + + def _round_robin_favorites(self, games: List[Dict], limit: int) -> List[Dict]: + """Each favourite team's next game before any team's second one. + + Taking the soonest N favourite games spends the slots on whoever plays + most often. Walked across a real season with two favourites and a limit + of 2, nine days of it showed Auburn twice and Georgia not at all -- + Auburn played either side of a Georgia bye, so both slots went to + Auburn. The other-games pool already refuses to do this; favourites + were still doing it. + + Depth is kept where there is room: one favourite with three slots still + gets its next three games, because the round-robin only comes back for + a team's second game once every team has had a first. + + A game between two favourites is picked once and counts for both. + """ + if limit <= 0 or not games: + return [] + wanted = [t for t in (self.favorite_teams or []) if t] + if len(wanted) < 2: + return games[:limit] # nothing to share the slots between + + # Which side of a game belongs to which favourite is a per-lineage + # question: NRL matches on ESPN team IDs because its abbreviations are + # not unique ("NEW" is both Newcastle and New Zealand), while the rest + # match on abbreviation. Ask for the lineage's own matcher rather than + # assuming, or this silently groups nothing and every slot goes empty. + matcher = getattr(self, "_team_in", None) + if not callable(matcher): + matcher = None # an is-None test narrows for static analysis + if matcher is None: + def belongs(game, team): + return team in (game.get("home_abbr"), game.get("away_abbr")) + else: + def belongs(game, team): + return bool(matcher(game.get("home_id"), [team]) + or matcher(game.get("away_id"), [team])) + + queues = {team: [] for team in wanted} + for game in games: # already in kickoff order + for team in wanted: + if belongs(game, team): + queues[team].append(game) + + picked, taken = [], set() + while len(picked) < limit: + progressed = False + for team in wanted: + queue = queues[team] + while queue and queue[0].get("id") in taken: + queue.pop(0) + if queue and len(picked) < limit: + game = queue.pop(0) + taken.add(game.get("id")) + picked.append(game) + progressed = True + if not progressed: + break # every queue is empty + return picked + + def _normalise_quality(self, raw) -> str: + """other_games_min_quality, as one of the values the code implements. + + An unusable value used to fall through every branch of + _passes_other_filters and silently mean "any" -- a quality bar the + board believes it has and does not. + """ + value = str(raw or "").strip().lower() + if value in self._QUALITY_CHOICES: + return value + if value == "broadcast": + # Retired in football-scoreboard 3.0.0 and now here. Measured + # against a real Week 1 and Week 2 college slate it passed 174 of + # 175 games: ESPN publishes a broadcaster for nearly everything + # now, ESPN+ included, so the tier read as a quality bar and + # behaved as "any". Boards holding it get the bar they thought + # they were getting. + self.logger.warning( + "%s: other_games_min_quality 'broadcast' has been retired -- " + "it let through nearly every game -- using 'ranked'. Change " + "the setting to clear this.", getattr(self, "sport_key", "?"), + ) + return "ranked" + self.logger.warning( + "%s: ignoring unusable other_games_min_quality=%r, using 'ranked'", + getattr(self, "sport_key", "?"), raw, + ) + return "ranked" + + def _check_ranking_coverage(self, games: List[Dict]) -> None: + """Say so when a loaded poll matches nothing on the schedule. + + The table is keyed by the abbreviation the RANKINGS endpoint returns and + matched against the one the SCOREBOARD endpoint returns. Nothing + guarantees the two agree, and if they ever stop agreeing the filter + quietly removes every non-favourite game -- no exception, no log line, + just a shorter board. That is the same shape as the bug where rankings + were never loading at all, which survived until someone went looking. + + Throttled to once an hour: selection runs on every update. + """ + if self.other_games_min_quality != "ranked": + return + rankings = getattr(self, "_team_rankings_cache", None) or {} + if not rankings or not games: + return + if any(self._is_ranked_game(g) for g in games): + return + now = time.monotonic() + # Zero means never logged, not "logged at the epoch". monotonic() counts + # from an arbitrary origin -- on a freshly booted board it is a few + # hundred seconds -- so comparing against 0 swallowed the first warning + # for the first hour of uptime, which is exactly when a misconfigured + # board is being watched. CI caught this; a machine with days of uptime + # cannot. + if (self._ranking_coverage_logged_at + and now - self._ranking_coverage_logged_at < self._RANKING_COVERAGE_SECONDS): + return + self._ranking_coverage_logged_at = now + self.logger.warning( + "%s: %d ranked teams loaded, but none of the %d other games match " + "one -- the quality filter is removing every non-favourite game. " + "Ranked abbreviations look like: %s", + self.league, len(rankings), len(games), + ", ".join(sorted(rankings)[:8]), + ) + + def _favorites_first( + self, + processed_games: List[Dict], + favorite_limit: int, + other_limit: int, + newest_first: bool = False, + ) -> List[Dict]: + """Favourite games first, then a bounded number of everything else. + + This is the middle setting the plugin was missing. `show_favorite_teams_only` + used to be the whole story: on, and you saw nothing but your teams; off, + and your teams were ignored entirely -- the selection just took the next + N games league-wide, so a UGA fan with 946 upcoming college games in the + window saw UGA about as often as chance allowed. + + Both counts are TOTALS here, not per-team. In favourites-only mode + `upcoming_games_to_show` is a per-team budget, which is reasonable when + the list is your own teams; applied to a dynamic group it is not. With + AP_TOP_10 resolving to a dozen teams, three games each is 28 distinct + cards before a single non-favourite is added. A total keeps the rotation + the length the user asked for. + """ + if newest_first: + def key(g): + return g.get("start_time_utc") or datetime.min.replace(tzinfo=timezone.utc) + ordered = sorted(processed_games, key=key, reverse=True) + else: + def key(g): + return g.get("start_time_utc") or datetime.max.replace(tzinfo=timezone.utc) + ordered = sorted(processed_games, key=key) + + favorites, others, unfiltered = [], [], [] + for game in ordered: + if self._is_favorite_game(game): + favorites.append(game) # never filtered: your team is your team + continue + unfiltered.append(game) + if self._passes_other_filters(game): + others.append(game) + self._check_ranking_coverage(unfiltered) + + self._selection_pools = { + "favorites": favorites, + "others": self._by_importance(others, newest_first), + "unfiltered": self._by_importance(unfiltered, newest_first), + "favorite_limit": favorite_limit, + "other_limit": other_limit, + "newest_first": newest_first, + } + return self._compose_selection() + + def _compose_selection(self) -> List[Dict]: + """Favourites plus the current slice of others, in schedule order. + + Split out of _favorites_first so the slice can be re-cut between + fetches. The pools are settled -- which games exist, and which of them + are worth a slot -- while WHICH of the others is on screen is a display + decision, and gating it on the fetch made the rotation interval a lie: + update() returns early until upcoming_update_interval has passed, so a + four-minute rotation actually stepped fifteen windows once an hour. + Same lesson as _advance_live_game_if_due further down this file. + """ + pools = self._selection_pools + favorites, others = pools["favorites"], pools["others"] + favorite_limit, other_limit = pools["favorite_limit"], pools["other_limit"] + newest_first = pools["newest_first"] + if newest_first: + def key(g): + return g.get("start_time_utc") or datetime.min.replace(tzinfo=timezone.utc) + else: + def key(g): + return g.get("start_time_utc") or datetime.max.replace(tzinfo=timezone.utc) + + selected = self._round_robin_favorites(favorites, max(0, favorite_limit)) + selected.extend(self._other_games_window(others, max(0, other_limit))) + if not selected and other_limit > 0: + # Nothing survived at all: your teams are not playing inside the + # schedule window AND the filters removed every other game. Each + # check fails open on missing data, but a filter working exactly as + # asked can still match nothing on a given day, and with no + # favourite game left there is nothing to carry the mode -- an empty + # list is a blank panel, not a short one. Same whole-list fallback + # `_filtered_or_all` makes for a board with no favourites at all. + # `other_limit` of 0 is an explicit "favourites only", so that one + # is left to go quiet as asked. + selected = self._other_games_window(pools["unfiltered"], max(0, other_limit)) + # Re-sort so the card order still reads as a schedule. Selection decides + # WHICH games; it should not reorder them into favourites-then-others, + # which would show next week's UGA game before tonight's. + selected.sort(key=key, reverse=newest_first) + return selected + + +class SportsLiveSharedMixin: + """The ``SportsLive`` bodies identical in all eight scoreboards.""" + + def _detect_stale_games(self, games: List[Dict]) -> None: + """Remove games that appear stale or haven't updated.""" + current_time = time.time() + + for game in games[:]: # Copy list to iterate safely + game_id = game.get("id") + if not game_id: + continue + + # Check if game data is stale + timestamps = self.game_update_timestamps.get(game_id, {}) + last_seen = timestamps.get("last_seen", 0) + + if last_seen > 0 and current_time - last_seen > self.stale_game_timeout: + self.logger.warning( + f"Removing stale game {game.get('away_abbr')}@{game.get('home_abbr')} " + f"(last seen {int(current_time - last_seen)}s ago)" + ) + games.remove(game) + if game_id in self.game_update_timestamps: + del self.game_update_timestamps[game_id] + continue + + # Also check if game appears to be over + if self._is_game_really_over(game): + self.logger.debug( + f"Removing game that appears over: {game.get('away_abbr')}@{game.get('home_abbr')} " + f"(clock={game.get('clock')}, period={game.get('period')}, period_text={game.get('period_text')})" + ) + games.remove(game) + if game_id in self.game_update_timestamps: + del self.game_update_timestamps[game_id] + + def _idle_live_interval(self) -> int: + """How long to wait before looking for live games again, when there are none. + + Escalates the longer nothing turns up, and any live game resets it, so + an in-season gap between games costs at most one escalated wait while + an out-of-season league stops polling on a live cadence entirely. + + Capped rather than unbounded: the cost of backing off is how late the + first game after a quiet spell is noticed, and past the cap the saving + stops being worth that. + """ + streak = getattr(self, "_empty_live_streak", 0) + base = self.no_data_interval + ceiling = getattr(self, "live_idle_max_interval", + _DEFAULT_LIVE_IDLE_MAX_SECONDS) + # The ceiling bounds the un-escalated interval too. The two settings are + # independent integers with no cross-validation, so base > ceiling is a + # reachable config -- and returning base unclamped there made the wait + # *shrink* as the streak grew (3600s at streak 0, 900s at streak 24), + # the opposite of what the setting named "maximum" promises. + if streak >= _IDLE_LONG_STREAK: + return min(int(base * _IDLE_LONG_FACTOR), ceiling) + if streak >= _IDLE_SHORT_STREAK: + return min(int(base * _IDLE_SHORT_FACTOR), ceiling) + return min(base, ceiling) + + def _note_live_fetch(self, found_live: bool) -> None: + """Record whether a look for live games found any.""" + if found_live: + if getattr(self, "_empty_live_streak", 0): + self.logger.info( + "Live games found after %d empty check(s); back to the " + "live update interval", self._empty_live_streak) + self._empty_live_streak = 0 + else: + self._empty_live_streak = getattr(self, "_empty_live_streak", 0) + 1 + + +class SportsRecentSharedMixin: + """The ``SportsRecent`` bodies identical in all eight scoreboards.""" + + def __init__( + self, + config: Dict[str, Any], + display_manager, + cache_manager, + logger: logging.Logger, + sport_key: str, + ): + super().__init__(config, display_manager, cache_manager, logger, sport_key) + self.games_list = [] # Filtered list for display (favorite teams) + self.current_game_index = 0 + self.last_update = 0 + self.update_interval = self.mode_config.get( + "recent_update_interval", 3600 + ) # Check for recent games every hour + self.last_game_switch = 0 + self.game_display_duration = self.mode_config.get("recent_game_duration", 15) + self._zero_clock_timestamps: Dict[str, float] = {} # Track games at 0:00 + + def _get_zero_clock_duration(self, game_id: str) -> float: + """Track how long a game has been at 0:00 clock.""" + current_time = time.time() + if game_id not in self._zero_clock_timestamps: + self._zero_clock_timestamps[game_id] = current_time + return 0.0 + return current_time - self._zero_clock_timestamps[game_id] + + def _clear_zero_clock_tracking(self, game_id: str) -> None: + """Clear tracking when game clock moves away from 0:00 or game ends.""" + if game_id in self._zero_clock_timestamps: + del self._zero_clock_timestamps[game_id] + diff --git a/src/display_controller.py b/src/display_controller.py index e739e08e..bb650698 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -181,6 +181,16 @@ class DisplayController: self.plugin_modes = {} # mode -> plugin_instance mapping for plugin-first dispatch self.mode_to_plugin_id: Dict[str, str] = {} self.plugin_display_modes: Dict[str, List[str]] = {} + # plugin_display_modes is mutated only by _register_loaded_plugin / + # _unregister_plugin on the render thread, but the config-watcher + # thread reads it in _enabled_plugin_not_running. Both mutation sites + # run during reconcile (rare), so this lock never touches the per-frame + # path -- the hot-path reads are same-thread as the writes. + self._plugin_modes_lock = threading.Lock() + # Guards the consume-and-clear of _pending_plugin_reconcile. Only taken + # when a reconcile is actually pending or a config change arrives, both + # rare -- the per-frame path just reads the bool. + self._reconcile_flag_lock = threading.Lock() # Per-plugin config-change callbacks, kept so we can unsubscribe a # plugin when it is disabled live. self._plugin_config_callbacks: Dict[str, Callable] = {} @@ -463,8 +473,10 @@ class DisplayController: self._refresh_config_cache(new_config) # If a plugin was enabled/disabled, flag a reconcile for the main # loop to apply (loading/unloading off the watcher thread is unsafe). - if self._enabled_set_changed(old_config, new_config): - self._pending_plugin_reconcile = True + if (self._enabled_set_changed(old_config, new_config) + or self._enabled_plugin_not_running(new_config)): + with self._reconcile_flag_lock: + self._pending_plugin_reconcile = True self.config_service.subscribe(_controller_config_change) @@ -1749,11 +1761,12 @@ class DisplayController: # rebuilding available_modes happens here on the render thread so # it can't race with rendering. Deferred while on-demand is active # (the flag stays set) so we don't fight its temporary-enable. + # The lock-free read is a fast path only; it can be a false + # negative (the watcher setting the flag just after it is read + # is seen next iteration), never a false positive that loses a + # request. if self._pending_plugin_reconcile and not self.on_demand_active: - # Only clear the flag on success -- a retryable failure - # (e.g. discovery) leaves it set so the request isn't lost. - if self._reconcile_enabled_plugins(): - self._pending_plugin_reconcile = False + self._service_pending_reconcile() if not self.available_modes: # Nothing to render yet. Re-check _pending_plugin_reconcile @@ -2813,7 +2826,8 @@ class DisplayController: logger.debug("Using manifest display_modes for %s: %s", plugin_id, display_modes) if not (isinstance(display_modes, list) and display_modes): display_modes = [plugin_id] - self.plugin_display_modes[plugin_id] = list(display_modes) + with self._plugin_modes_lock: + self.plugin_display_modes[plugin_id] = list(display_modes) # Subscribe to config changes for per-plugin hot-reload. Bind plugin_id # and instance as defaults so each plugin's callback targets its own @@ -2847,7 +2861,8 @@ class DisplayController: def _unregister_plugin(self, plugin_id: str) -> None: """Remove a plugin's modes, config subscription and instance, then unload it. Used by live disable hot-reload.""" - modes = self.plugin_display_modes.pop(plugin_id, []) + with self._plugin_modes_lock: + modes = self.plugin_display_modes.pop(plugin_id, []) for mode in modes: if mode in self.available_modes: self.available_modes.remove(mode) @@ -2892,6 +2907,67 @@ class DisplayController: } return enabled_map(old_config) != enabled_map(new_config) + def _service_pending_reconcile(self) -> None: + """Consume a pending reconcile request and run it. + + The request is consumed BEFORE reconciling, not cleared after. Clearing + after would drop any config change that lands while reconcile is + running: reconcile has already read its config by then, so the clear + erases a request it never served and the newest config never + reconciles -- the same "your save did nothing" failure this whole path + exists to prevent. Consuming first means such a request stays set and + is picked up on the next pass. + + A retryable failure (e.g. discovery) re-arms the flag. + """ + with self._reconcile_flag_lock: + pending = self._pending_plugin_reconcile + self._pending_plugin_reconcile = False + if pending and not self._reconcile_enabled_plugins(): + with self._reconcile_flag_lock: + self._pending_plugin_reconcile = True + + def _enabled_plugin_not_running(self, new_config: Dict[str, Any]) -> bool: + """True when a discovered plugin is enabled in config but not running. + + ``_enabled_set_changed`` compares only top-level ``enabled`` flags, which + misses the case that strands a plugin: one whose ``validate_config()`` + returned False is absent from the running set, and the edit that fixes it + (enabling a league, filling in an API key) lives *nested* inside that + plugin's own section. No top-level flag changes, so no reconcile is + queued, and the save that should have fixed it appears to do nothing -- + only toggling some unrelated plugin recovers it. hockey-scoreboard sat + enabled-but-absent on a live rig for four days this way. + + Deliberately narrow: it fires only for ids the plugin manager has + actually discovered, so non-plugin sections that carry their own + ``enabled`` flag (``schedule``, ``display``, ...) don't queue a reconcile + on every save. In the steady state -- everything enabled is loaded -- + this is False and costs nothing. That matters because reconcile runs + ``discover_plugins()`` on the render thread, where a needless + filesystem scan per config save would show up as a frame hitch. + + Runs on the config-watcher thread, so both mappings it reads are + snapshotted under the lock that guards their writes. + """ + if self.plugin_manager is None: + return False + # Two snapshots, each taken under its own lock and never nested, so a + # half-written mapping is never observed and this can't deadlock + # against discovery (which holds the discovery lock while rebuilding). + try: + known = self.plugin_manager.discovered_plugin_ids() + except AttributeError: + # Older manager without the accessor: fall back to a plain read. + known = set(getattr(self.plugin_manager, 'plugin_manifests', ()) or ()) + with self._plugin_modes_lock: + running = set(self.plugin_display_modes) + for key, value in new_config.items(): + if (key in known and isinstance(value, dict) + and value.get('enabled', False) and key not in running): + return True + return False + def _reconcile_enabled_plugins(self) -> bool: """Load/unload plugins so the running set matches the enabled set in config. Runs on the main display thread (never the config-watcher diff --git a/src/display_manager.py b/src/display_manager.py index 9cc7f622..e578ad7c 100644 --- a/src/display_manager.py +++ b/src/display_manager.py @@ -364,6 +364,7 @@ class DisplayManager: # Create image with the (logical) display dimensions self.image = Image.new('RGB', (self.matrix.width, self.matrix.height)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs. logger.info(f"Image canvas created with dimensions: {self.matrix.width}x{self.matrix.height}") # Initialize font with Press Start 2P @@ -403,6 +404,7 @@ class DisplayManager: self.image = Image.new('RGB', (fallback_width, fallback_height)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs. # Simple fallback visualization so web UI shows a realistic canvas try: self.draw.rectangle([0, 0, fallback_width - 1, fallback_height - 1], outline=(255, 0, 0)) @@ -705,6 +707,7 @@ class DisplayManager: # self.image, so swapping the buffer below is enough on its own. self.image = Image.new('RGB', (target_w, target_h)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs. yield finally: self.matrix = real_matrix @@ -814,6 +817,7 @@ class DisplayManager: self.image = Image.new('RGB', (width, height)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs. logger.debug("Cleared display in fallback mode") return @@ -825,6 +829,7 @@ class DisplayManager: # Create a new black image self.image = Image.new('RGB', (self.matrix.width, self.matrix.height)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs. if not self._capture_mode_active: # Clear both canvases and the underlying matrix to ensure no artifacts. @@ -1258,6 +1263,7 @@ class DisplayManager: try: self.image = Image.new('RGB', (self.width, self.height)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs. except (OSError, RuntimeError, ValueError, MemoryError): logger.debug("Canvas reset during cleanup failed", exc_info=True) # Reset the singleton state when cleaning up diff --git a/src/logo_downloader.py b/src/logo_downloader.py index b799b7c1..d6fbd7bc 100644 --- a/src/logo_downloader.py +++ b/src/logo_downloader.py @@ -14,6 +14,7 @@ import json from typing import Dict, List, Optional, Tuple from pathlib import Path from PIL import Image, ImageDraw, ImageFont +from PIL.PngImagePlugin import PngInfo from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry from src.common.permission_utils import ( @@ -25,6 +26,96 @@ from src.common.permission_utils import ( logger = logging.getLogger(__name__) +#: PNG text key stamped into a generated placeholder so a later run can tell it +#: apart from a real logo that happens to be small. +PLACEHOLDER_MARKER = "ledmatrix_placeholder" + +#: Geometry of a generated placeholder, used to recognise ones written before +#: the marker existed. Those are already on users' disks and would otherwise +#: never be retried. +PLACEHOLDER_SIZE = (64, 64) +PLACEHOLDER_BG = (100, 100, 100, 255) + +#: How long a placeholder is trusted before the real logo is attempted again. +#: A placeholder means the download failed, and download failures are usually +#: transient (no network at boot, ESPN blipping). Retrying every frame would +#: hammer the API from a Pi that is also driving a panel; never retrying leaves +#: the team a grey box forever, which is the bug this exists to avoid. +PLACEHOLDER_RETRY_SECONDS = 6 * 60 * 60 + + +def is_placeholder_logo(filepath: Path) -> bool: + """True if the file at ``filepath`` is a generated placeholder, not a logo. + + Checks the marker first, then falls back to matching the placeholder's + exact geometry and background colour so files written before the marker was + introduced are still recognised. + """ + try: + with Image.open(filepath) as img: + if img.info.get(PLACEHOLDER_MARKER): + return True + if img.size != PLACEHOLDER_SIZE: + return False + return img.convert("RGBA").getpixel((0, 0)) == PLACEHOLDER_BG + except Exception: + # Unreadable file: not provably a placeholder, and the caller's own + # error handling is better placed to deal with it. + return False + + +def should_attempt_download(filepath: Path, force_download: bool = False) -> bool: + """Whether a real logo is worth (re)fetching for ``filepath``. + + True when nothing is there, when the caller forced it, or when what is + there is a placeholder old enough to retry. A *fresh* placeholder says a + download just failed, so retrying it immediately would hammer the API for + a result that is very unlikely to have changed. + """ + if force_download or not filepath.exists(): + return True + if not is_placeholder_logo(filepath): + return False + age = placeholder_age_seconds(filepath) + return age is None or age >= PLACEHOLDER_RETRY_SECONDS + + +def refresh_placeholder_timestamp(filepath: Path) -> bool: + """Restamp a placeholder so a failed retry restarts the back-off clock. + + Without this a stale placeholder stays stale: every later call sees an + expired timestamp, retries, fails, and leaves the timestamp untouched -- + which is a download attempt per call, the opposite of what the back-off is + for. + """ + try: + if not is_placeholder_logo(filepath): + return False + metadata = PngInfo() + metadata.add_text(PLACEHOLDER_MARKER, str(time.time())) + with Image.open(filepath) as img: + img.copy().save(filepath, "PNG", pnginfo=metadata) + return True + except Exception: + logger.debug("Could not refresh placeholder timestamp for %s", filepath, + exc_info=True) + return False + + +def placeholder_age_seconds(filepath: Path) -> Optional[float]: + """Seconds since a placeholder was written, or None if unknown.""" + try: + with Image.open(filepath) as img: + stamped = img.info.get(PLACEHOLDER_MARKER) + if stamped and stamped != "1": + return max(0.0, time.time() - float(stamped)) + except Exception: + pass + try: + return max(0.0, time.time() - filepath.stat().st_mtime) + except OSError: + return None + class LogoDownloader: """Centralized logo downloader for team logos from ESPN API.""" @@ -499,8 +590,10 @@ class LogoDownloader: filename = f"{self.normalize_abbreviation(abbreviation)}.png" filepath = Path(logo_dir) / filename - # Skip if already exists and not forcing download - if filepath.exists() and not force_download: + # A placeholder does not count as existing -- it is a previous + # failure, and a bulk pass is exactly where it should get another + # chance, subject to the same back-off as everywhere else. + if not should_attempt_download(filepath, force_download): logger.debug(f"Skipping {display_name}: {filename} already exists") continue @@ -559,8 +652,9 @@ class LogoDownloader: filename = f"{self.normalize_abbreviation(abbreviation)}.png" filepath = Path(logo_dir) / filename - # Skip if already exists and not forcing download - if filepath.exists() and not force_download: + # Same eligibility rule as every other download site: a stale + # placeholder is a failed download, not a logo. + if not should_attempt_download(filepath, force_download): logger.debug(f"Skipping {display_name} ({category}, {conference}): {filename} already exists") continue @@ -674,11 +768,16 @@ class LogoDownloader: # Fallback without font draw.text((16, 24), text, fill=(255, 255, 255, 255)) - logo.save(filepath) - + # Stamp it so a later run can tell this apart from a real logo and + # retry the download, instead of treating the file's existence as + # proof the logo was fetched. + metadata = PngInfo() + metadata.add_text(PLACEHOLDER_MARKER, str(time.time())) + logo.save(filepath, "PNG", pnginfo=metadata) + # Set proper file permissions after saving ensure_file_permissions(filepath, get_assets_file_mode()) - + logger.info(f"Created placeholder logo for {team_abbreviation} at {filepath}") return True @@ -770,9 +869,18 @@ def download_missing_logo(league: str, team_id: str, team_abbreviation: str, log # Use the exact filepath that was passed in (respects config settings) filepath = logo_path - if filepath.exists(): + if filepath.exists() and not should_attempt_download(filepath): + # Either a real logo, or a placeholder too fresh to be worth retrying. logger.debug(f"Logo already exists for {team_abbreviation} ({league})") return True + if filepath.exists(): + # A placeholder is a *failed* download wearing the real logo's + # filename. Treating it as "already exists" is what pinned a team to a + # grey box permanently after one transient failure. + logger.info( + "Logo for %s (%s) is a placeholder from a failed download; " + "retrying the real logo", team_abbreviation, league, + ) # Try to download the real logo first logger.info(f"Attempting to download logo for {team_abbreviation} from {league}") diff --git a/src/plugin_system/compatibility.py b/src/plugin_system/compatibility.py index 9b676cd2..4537be85 100644 --- a/src/plugin_system/compatibility.py +++ b/src/plugin_system/compatibility.py @@ -28,6 +28,7 @@ fixes their version string. See `docs/SPORTS_UNIFICATION.md`, phase B4. from __future__ import annotations import re +from pathlib import Path from typing import Any, Dict, Optional, Tuple # Below this, the core's self-reported version is not evidence of anything. @@ -35,6 +36,45 @@ from typing import Any, Dict, Optional, Tuple TRUSTWORTHY_FLOOR: Tuple[int, int, int] = (2, 0, 0) +# ``src/__init__.py`` relative to this file: src/plugin_system/ -> src/ +_VERSION_FILE = Path(__file__).resolve().parent.parent / "__init__.py" +_VERSION_RE = re.compile(r'^__version__\s*=\s*["\']([^"\']+)["\']', re.M) + + +def current_core_version() -> str: + """The core version as it is on disk right now, not as it was at import. + + ``from src import __version__`` binds whatever the process loaded at start. + The web UI runs as its own long-lived service (``ledmatrix-web.service``), + and updating the core replaces files on disk without restarting it -- the + update route says so explicitly and asks the user to restart. Its prompt + names the *display* service, so a user who follows it leaves the web + process holding the old number. + + The plugin store's gate lives in that web process. Stale by one release is + exactly the case that matters: every plugin flooring on the release you + just installed gets refused, with a message blaming a core version that is + already correct on disk. 3.3.0 is the first release where that hits a whole + plugin family at once -- all eight sports scoreboards floor there. + + Reading the file costs one stat and a small read per call, and only on the + install/update path. Any failure falls back to the imported value, so this + can only ever be as wrong as before, never worse. + """ + try: + text = _VERSION_FILE.read_text(encoding="utf-8") + match = _VERSION_RE.search(text) + if match: + return match.group(1) + except (OSError, UnicodeDecodeError): + pass + try: + from src import __version__ as imported + return imported + except Exception: # noqa: BLE001 - never let this raise + return "0.0.0" + + def parse_semver(value: Any) -> Optional[Tuple[int, int, int]]: """Parse ``X.Y.Z`` (extra parts and suffixes ignored) into a comparable 3-tuple, or ``None`` when unparseable. A leading ``v`` is tolerated.""" diff --git a/src/plugin_system/plugin_loader.py b/src/plugin_system/plugin_loader.py index d8ae0e3d..7a4969be 100644 --- a/src/plugin_system/plugin_loader.py +++ b/src/plugin_system/plugin_loader.py @@ -772,8 +772,8 @@ class PluginLoader: newer than the running core. Advisory only — never raises — so a plugin that guards optional features with try/except keeps working. """ - from src import __version__ as core_version from src.plugin_system import compatibility + core_version = compatibility.current_core_version() compatible, _reason = compatibility.check(manifest, core_version) if compatible: diff --git a/src/plugin_system/plugin_manager.py b/src/plugin_system/plugin_manager.py index 543017c9..0dc7a428 100644 --- a/src/plugin_system/plugin_manager.py +++ b/src/plugin_system/plugin_manager.py @@ -631,6 +631,17 @@ class PluginManager: return self.load_plugin(plugin_id) + def discovered_plugin_ids(self) -> set: + """Snapshot of the discovered plugin ids, taken under the discovery lock. + + Callers on other threads (the config watcher) must not iterate + ``plugin_manifests`` directly: discovery rebuilds it entry by entry, so + an unsynchronised reader can see a half-populated mapping or raise + "dictionary changed size during iteration". + """ + with self._discovery_lock: + return set(self.plugin_manifests) + def get_plugin(self, plugin_id: str) -> Optional[Any]: """ Get a loaded plugin instance by ID. diff --git a/src/plugin_system/plugin_state.py b/src/plugin_system/plugin_state.py index 269bb423..f9b9d7d0 100644 --- a/src/plugin_system/plugin_state.py +++ b/src/plugin_system/plugin_state.py @@ -6,14 +6,40 @@ with state transitions and queries. """ import threading +import time +from collections import deque from enum import Enum -from typing import Optional, Dict, Any +from typing import Optional, Dict, Any, Deque, List, Tuple from datetime import datetime import logging from src.logging_config import get_logger +# The history is diagnostic only -- nothing reads the entries themselves, just +# their count -- but it is appended to on the hot scheduling path: every update +# cycle records RUNNING on reserve and ENABLED on finish. Unbounded, that is +# 2,880 entries per plugin per day at the default 60s interval, which on a 1 GB +# Pi exhausts memory in weeks. +# +# Two limits, because a single entry count answers the wrong question. What a +# reader wants is "the last couple of hours", and how many transitions that is +# depends entirely on the plugin's update interval -- which on a real board +# spans 2s to 3600s. A flat 200 entries is 4.2 days for the slowest plugin and +# 3.3 minutes for the fastest, so the plugin churning hardest, the one worth +# looking at, keeps the least history. +# +# So: trim by AGE first, which makes the retained window comparable across +# plugins whatever their cadence... +STATE_HISTORY_MAX_AGE_SECONDS = 2 * 60 * 60 + +# ...and cap by COUNT second, purely as a memory ceiling for the fast pollers +# whose age window would otherwise run to thousands of entries. At ~230 bytes +# an entry this is ~0.5 MB per plugin worst case, and only plugins updating +# faster than roughly every 4s can reach it. +MAX_STATE_HISTORY_PER_PLUGIN = 2000 + + class PluginState(Enum): """Plugin state enumeration.""" UNLOADED = "unloaded" # Plugin not loaded @@ -37,11 +63,43 @@ class PluginStateManager: self.logger = logger or get_logger(__name__) self._lock = threading.RLock() self._states: Dict[str, PluginState] = {} - self._state_history: Dict[str, list] = {} + # (monotonic timestamp, transition). The clock is monotonic so a DST + # shift or an NTP step cannot make entries look old and flush the + # history; the human-readable timestamp lives inside the transition. + self._state_history: Dict[str, Deque[Tuple[float, Dict[str, Any]]]] = {} + # Lifetime transition totals, kept separately so the count reported by + # get_state_info() stays truthful once the history above starts rolling. + self._state_transition_counts: Dict[str, int] = {} self._error_info: Dict[str, Dict[str, Any]] = {} self._last_update: Dict[str, datetime] = {} self._last_display: Dict[str, datetime] = {} + def _record_transition( + self, + plugin_id: str, + transition: Dict[str, Any] + ) -> None: + """Append a transition to the plugin's bounded history. + + Callers must already hold ``_lock``. The deque discards its oldest + entry once it is full, so the history cannot grow without bound; the + lifetime total is tracked separately for get_state_info(). + """ + history = self._state_history.get(plugin_id) + if history is None: + history = deque(maxlen=MAX_STATE_HISTORY_PER_PLUGIN) + self._state_history[plugin_id] = history + now = time.monotonic() + history.append((now, transition)) + # Age out first; the deque's maxlen is the backstop for plugins that + # produce more than the ceiling within the window. + cutoff = now - STATE_HISTORY_MAX_AGE_SECONDS + while history and history[0][0] < cutoff: + history.popleft() + self._state_transition_counts[plugin_id] = ( + self._state_transition_counts.get(plugin_id, 0) + 1 + ) + def set_state( self, plugin_id: str, @@ -60,16 +118,13 @@ class PluginStateManager: old_state = self._states.get(plugin_id, PluginState.UNLOADED) self._states[plugin_id] = state - if plugin_id not in self._state_history: - self._state_history[plugin_id] = [] - transition = { 'timestamp': datetime.now(), 'from': old_state.value, 'to': state.value, 'error': str(error) if error else None } - self._state_history[plugin_id].append(transition) + self._record_transition(plugin_id, transition) # Store error info if transitioning to ERROR state if state == PluginState.ERROR and error: @@ -126,17 +181,29 @@ class PluginStateManager: state = self.get_state(plugin_id) return state == PluginState.ENABLED - def get_state_history(self, plugin_id: str) -> list: + def get_state_history(self, plugin_id: str) -> List[Dict[str, Any]]: """ Get state transition history for a plugin. - + + Retention is by age first -- transitions older than + STATE_HISTORY_MAX_AGE_SECONDS are dropped -- and by count second, at + MAX_STATE_HISTORY_PER_PLUGIN, which only binds for plugins updating + fast enough to exceed it inside that window. + Args: plugin_id: Plugin identifier - + Returns: - List of state transitions + List of recent state transitions, oldest first. Both the list and + the transition dicts are copies, so callers cannot mutate the + manager's own history. The values inside a transition are all + immutable, so a shallow copy per entry is enough. """ - return self._state_history.get(plugin_id, []) + with self._lock: + return [ + dict(transition) + for _stamp, transition in self._state_history.get(plugin_id, ()) + ] def set_error_info(self, plugin_id: str, error_info: Dict[str, Any]) -> None: """ @@ -179,9 +246,7 @@ class PluginStateManager: old_state = self._states.get(plugin_id, PluginState.UNLOADED) self._states[plugin_id] = state - if plugin_id not in self._state_history: - self._state_history[plugin_id] = [] - self._state_history[plugin_id].append({ + self._record_transition(plugin_id, { 'timestamp': datetime.now(), 'from': old_state.value, 'to': state.value, @@ -241,26 +306,40 @@ class PluginStateManager: Returns: Dictionary with state information """ - state = self.get_state(plugin_id) - info = { - 'state': state.value, - 'is_loaded': self.is_loaded(plugin_id), - 'is_enabled': self.is_enabled(plugin_id), - 'is_running': self.is_running(plugin_id), - 'is_error': self.is_error(plugin_id), - 'can_execute': self.can_execute(plugin_id), - 'last_update': self.get_last_update(plugin_id), - 'last_display': self.get_last_display(plugin_id), - 'error_info': self.get_error_info(plugin_id), - 'state_history_count': len(self.get_state_history(plugin_id)) - } + # One snapshot, one critical section. Each field was read under its own + # lock, so an unload running concurrently could be observed half-done: + # 'state' read before clear_state() removed it and + # 'state_history_count' read after, giving a caller a plugin that is + # ENABLED with zero transitions. _lock is an RLock, so the helpers + # below can still take it. + with self._lock: + state = self.get_state(plugin_id) + info = { + 'state': state.value, + 'is_loaded': self.is_loaded(plugin_id), + 'is_enabled': self.is_enabled(plugin_id), + 'is_running': self.is_running(plugin_id), + 'is_error': self.is_error(plugin_id), + 'can_execute': self.can_execute(plugin_id), + 'last_update': self.get_last_update(plugin_id), + 'last_display': self.get_last_display(plugin_id), + 'error_info': self.get_error_info(plugin_id), + 'state_history_count': self._state_transition_counts.get(plugin_id, 0) + } return info def clear_state(self, plugin_id: str) -> None: - """Clear all state information for a plugin.""" - self._states.pop(plugin_id, None) - self._state_history.pop(plugin_id, None) - self._error_info.pop(plugin_id, None) - self._last_update.pop(plugin_id, None) - self._last_display.pop(plugin_id, None) + """Clear all state information for a plugin. + + Held under ``_lock`` so the five dicts are dropped as one unit: every + other mutator takes the lock, and without it a concurrent set_state() + could interleave and leave a plugin with history but no state. + """ + with self._lock: + self._states.pop(plugin_id, None) + self._state_history.pop(plugin_id, None) + self._state_transition_counts.pop(plugin_id, None) + self._error_info.pop(plugin_id, None) + self._last_update.pop(plugin_id, None) + self._last_display.pop(plugin_id, None) diff --git a/src/plugin_system/store_manager.py b/src/plugin_system/store_manager.py index 5cf31cf8..122cdf9c 100644 --- a/src/plugin_system/store_manager.py +++ b/src/plugin_system/store_manager.py @@ -1467,8 +1467,11 @@ class PluginStoreManager: # already had. Allowing it costs them a plugin that raises # ModuleNotFoundError at load and is reported only as one line # in the journal. See docs/SPORTS_UNIFICATION.md (phase B4/B6). - from src import __version__ as core_version from src.plugin_system import compatibility + # On disk, not as imported: this process may predate the core + # update that made the plugin compatible. See + # compatibility.current_core_version. + core_version = compatibility.current_core_version() compatible, reason = compatibility.check(manifest, core_version) if not compatible: @@ -1602,6 +1605,26 @@ class PluginStoreManager: 'error': f'Manifest missing required fields: {", ".join(missing_fields)}' } + # Refuse a plugin that needs a newer core than this one, exactly as + # _install_plugin_impl does after its download. Sideloading is an + # explicit act rather than an automatic store update, but the floor + # is not advice about intent -- it is a statement that the plugin + # cannot run here, and letting it through produces the same silent + # PluginState.ERROR at load. This was the last of the three routes + # in that skipped the check. + # + # Before the move, so the `finally` below removes the temp tree and + # nothing half-installed is left behind. + from src.plugin_system import compatibility + core_version = compatibility.current_core_version() + + compatible, reason = compatibility.check(manifest, core_version) + if not compatible: + self.logger.error( + "Refusing to install %s from %s: %s", + plugin_id, repo_url, reason) + return {'success': False, 'error': reason} + # Validate version fields consistency (warnings only, not required) validation_errors = self._validate_manifest_version_fields(manifest) if validation_errors: @@ -2583,6 +2606,85 @@ class PluginStoreManager: self.logger.error(f"Error uninstalling plugin {plugin_id}: {e}") return False + def _gate_pulled_commit(self, plugin_id: str, plugin_path: Path, + previous_sha: Optional[str]) -> bool: + """Apply the compatibility gate to a commit that arrived via git pull. + + Every other route into an installed plugin goes through + ``install_plugin``, which gates in ``_install_plugin_impl``. This one + did not: a ``git pull`` could deliver a manifest flooring above this + core and nothing would notice until the plugin failed to load, which + surfaces as one line in the journal and a scoreboard that silently + stopped appearing. + + Checked after the pull rather than before it, for the same reason + ``_install_plugin_impl`` checks after the download: the registry + carries no compatibility field, so the incoming floor is only knowable + once the new commit is on disk. + + Undone with ``git reset --hard`` rather than by removing the directory. + This is a live checkout, the previous commit is still in the object + store, and the reset leaves the user on the exact version they were + already running -- the same promise ``_reinstall_with_rollback`` makes, + reached by the means this path actually has. It is also the gentler + option: no window in which the plugin directory does not exist, and no + ``.standalone-backup-`` debris if the process dies mid-way. + + A manifest that cannot be read is not evidence of incompatibility, so + it allows. ``compatibility.check`` refuses only on evidence for the + same reason: a wrong refusal breaks a working install, while a wrong + allowance degrades to exactly the behaviour this path had before the + gate existed. + """ + manifest_path = plugin_path / "manifest.json" + try: + with open(manifest_path, 'r', encoding='utf-8') as mf: + manifest = json.load(mf) + except (OSError, ValueError) as e: + self.logger.warning( + "Could not read %s after updating %s (%s); allowing the " + "update, as an unreadable manifest declares no floor", + manifest_path, plugin_id, e) + return True + + from src.plugin_system import compatibility + core_version = compatibility.current_core_version() + + compatible, reason = compatibility.check(manifest, core_version) + if compatible: + return True + + self.logger.error("Refusing the update to %s: %s", plugin_id, reason) + + if not previous_sha: + self.logger.error( + "Cannot roll %s back: the commit it was on before the pull is " + "unknown. It is now on a version this core cannot run — " + "reinstall it from the plugin store.", plugin_id) + return False + + # Safe by construction: update_plugin returns before pulling unless the + # tree was clean or successfully stashed, so there are no uncommitted + # tracked edits for --hard to discard. The stash is not popped on the + # success path either, so the reset leaves the working tree exactly + # where a successful pull would have. Say "commit", not "changes". + reset = subprocess.run( + ['git', '-C', str(plugin_path), 'reset', '--hard', previous_sha], + capture_output=True, text=True, timeout=60, check=False) + if reset.returncode != 0: + self.logger.error( + "CRITICAL: could not roll %s back to commit %s: %s. It is left " + "on a version this core cannot run; " + "`git -C %s reset --hard %s` restores it.", + plugin_id, previous_sha[:7], + (reset.stderr or reset.stdout or '').strip(), + plugin_path, previous_sha) + else: + self.logger.info( + "Rolled %s back to commit %s; it stays on the version it was " + "already running.", plugin_id, previous_sha[:7]) + return False + def _reinstall_with_rollback(self, plugin_id: str, plugin_path: Path) -> bool: """Replace an installed plugin with a fresh install, atomically. @@ -2860,6 +2962,8 @@ class PluginStoreManager: status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})() stash_info = "" + # Whether the pull can be undone without destroying work. + tree_is_recoverable = not has_changes if has_changes: self.logger.info(f"Stashing local changes in {plugin_id} before update") try: @@ -2873,12 +2977,37 @@ class PluginStoreManager: ) if stash_result.returncode == 0: stash_info = " (local changes were stashed)" + tree_is_recoverable = True self.logger.info(f"Stashed local changes (including untracked files) for {plugin_id}") else: self.logger.warning(f"Failed to stash local changes for {plugin_id}: {stash_result.stderr}") except subprocess.TimeoutExpired: self.logger.warning(f"Stash operation timed out for {plugin_id}, proceeding with pull") + # Do not pull what cannot be un-pulled. + # + # The compatibility gate below can refuse the commit this + # pull brings down, and its only way back is `git reset + # --hard`, which discards uncommitted tracked edits. Those + # edits are exactly what the stash above exists to protect, + # so a stash that failed or timed out leaves the rollback + # unable to run without destroying them. + # + # A pull does not necessarily refuse on a dirty tree -- git + # merges happily as long as the incoming commit touches + # different files -- so without this the update would + # succeed, the gate would refuse, and the reset would take + # the user's work with it. Refusing here costs an update in + # a case that already went wrong; the alternative costs + # data. + if not tree_is_recoverable: + self.logger.error( + "Refusing to update %s: it has local changes that could " + "not be stashed, and an incompatible update could then " + "only be rolled back by discarding them. Commit or stash " + "them by hand, then update.", plugin_id) + return False + # Pull from the determined remote branch self.logger.info(f"Pulling from origin/{remote_pull_branch} for {plugin_id}...") pull_result = subprocess.run( @@ -2901,6 +3030,14 @@ class PluginStoreManager: elif updated_sha: self.logger.info(f"Plugin {plugin_id} updated to commit {updated_sha[:7]}{stash_info}") + # The install gate, at the only point on this path where + # it can be answered. Every other route in goes through + # install_plugin, which gates in _install_plugin_impl; this + # one did not, so a pull could deliver a manifest flooring + # above this core and nothing would notice. + if not self._gate_pulled_commit(plugin_id, plugin_path, local_sha): + return False + self._install_dependencies(plugin_path) return True diff --git a/src/plugin_system/testing/visual_display_manager.py b/src/plugin_system/testing/visual_display_manager.py index 5211a226..e33d2309 100644 --- a/src/plugin_system/testing/visual_display_manager.py +++ b/src/plugin_system/testing/visual_display_manager.py @@ -70,6 +70,7 @@ class VisualTestDisplayManager: # Canvas self.image = Image.new('RGB', (width, height), (0, 0, 0)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows. # Matrix proxy (plugins access display_manager.matrix.width/height) self.matrix = _MatrixProxy(width, height) @@ -184,6 +185,7 @@ class VisualTestDisplayManager: self.clear_called = True self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows. def update_display(self): """No-op for hardware; marks that display was updated.""" @@ -211,6 +213,7 @@ class VisualTestDisplayManager: self.matrix = _MatrixProxy(target_w, target_h) self.image = Image.new('RGB', (target_w, target_h), (0, 0, 0)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows. yield finally: self._width, self._height = prev_w, prev_h @@ -569,6 +572,7 @@ class VisualTestDisplayManager: self.draw_calls = [] self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows. self._scrolling_state = { 'is_scrolling': False, 'last_scroll_activity': 0, @@ -582,3 +586,4 @@ class VisualTestDisplayManager: """Clean up resources.""" self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0)) self.draw = ImageDraw.Draw(self.image) + self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows. diff --git a/src/vegas_mode/coordinator.py b/src/vegas_mode/coordinator.py index 430cbef6..b9af5ea6 100644 --- a/src/vegas_mode/coordinator.py +++ b/src/vegas_mode/coordinator.py @@ -31,6 +31,18 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) +#: Degradation threshold, as a fraction of target_fps. A marquee jitters a +#: little all the time, so "anything under target" would report constantly and +#: mean nothing; 90% of target is the point where a shortfall is real. At a +#: 60fps target that is 54fps -- 55fps is a normal wobble and stays at DEBUG, +#: which is deliberate, not an off-by-one. +_FPS_HEALTHY_FRACTION = 0.9 + +#: A healthy marquee still reports this often, so silence means stopped +#: rather than fine. +_FPS_HEARTBEAT_INTERVAL = 300.0 + + def _percentile(ordered: List[float], fraction: float) -> float: """Nearest-rank percentile of an already-sorted list. @@ -96,6 +108,11 @@ class VegasModeCoordinator: self._is_active = False self._is_paused = False self._should_stop = False + # Frame-rate health, tracked across run_iteration() calls so the + # heartbeat is one-per-interval rather than one-per-cycle, and so a + # recovery spanning two cycles is still reported. Reset on start(). + self._fps_last_health_log = 0.0 + self._fps_was_degraded = False self._state_lock = threading.Lock() # Live priority tracking @@ -248,6 +265,11 @@ class VegasModeCoordinator: self._is_active = True self._should_stop = False self._start_time = time.time() + # A fresh run starts with a clean health slate: no stale + # "was degraded" from the previous run, and a heartbeat that is + # due immediately so the first sample confirms the marquee is up. + self._fps_last_health_log = 0.0 + self._fps_was_degraded = False # Line up the next group immediately, so the first extension is already # warm rather than stalling the scroll to fetch it. @@ -395,8 +417,18 @@ class VegasModeCoordinator: duration = self.render_pipeline.get_dynamic_duration() start_time = time.time() frame_count = 0 - fps_log_interval = 5.0 # Log FPS every 5 seconds - last_fps_log_time = start_time + fps_log_interval = 5.0 # Sample FPS every 5 seconds + # Health state lives on the coordinator, not here: run_iteration() is + # called once per cycle, so locals reset every few seconds. That made + # `last_fps_health_log = 0.0` fire the "heartbeat" on the first sample + # of every iteration rather than once per interval, and a recovery + # that crossed an iteration boundary was never reported at all -- + # was_degraded had already gone back to False. + # Monotonic, and deliberately not start_time: start_time is wall + # clock and is used below to report the iteration's duration. Mixing + # the two here would make every delta hugely negative and silence the + # frame-rate reporting altogether. + last_fps_log_time = time.monotonic() fps_frame_count = 0 # A mean hides stutter completely. At 120fps a five-second window is # ~600 frames, so a 200ms freeze -- plainly visible on a marquee -- @@ -408,7 +440,13 @@ class VegasModeCoordinator: logger.info("Starting Vegas iteration for %.1fs", duration) while True: - frame_started = time.time() + # Monotonic, like the FPS window below. These devices have no RTC, + # so the wall clock jumps by however wrong boot time was the moment + # NTP first syncs. A backward jump makes frame_elapsed negative, + # and `frame_interval - frame_elapsed` then sleeps for longer than + # the whole budget -- the render loop stalls for the size of the + # correction. A forward jump inflates p99 and worst-frame instead. + frame_started = time.monotonic() # Check for STATIC mode plugin that should pause scroll static_plugin = self._check_static_plugin_trigger() @@ -436,7 +474,7 @@ class VegasModeCoordinator: # quarter of the budget spent not rendering. Subtracting the work # already done keeps the pacing target while reclaiming that time, # and yields the GIL either way so other threads still run. - frame_elapsed = time.time() - frame_started + frame_elapsed = time.monotonic() - frame_started time.sleep(max(0.0, frame_interval - frame_elapsed)) # Measured before the sleep: time spent working, not pacing. @@ -448,16 +486,42 @@ class VegasModeCoordinator: frame_count += 1 fps_frame_count += 1 - # Periodic FPS logging - current_time = time.time() + # Periodic FPS logging. Reported at INFO only when the frame rate + # is actually worth an operator's attention -- a shortfall against + # target, or the recovery from one -- with a slow heartbeat so a + # healthy marquee still shows a pulse. + # + # Measured over two hours on a running rig: 1410 samples, 98.5% + # of them within 10% of target. The 1.5% that were not included a + # reading of 8.6fps against a target of 60 -- a real stall, and + # completely invisible inside 1389 lines reading "59.6". + # Monotonic: every use of this value in the block below is a + # duration, and these devices have no RTC, so the wall clock jumps + # by however wrong boot time was the moment NTP first syncs. That + # would not only mis-fire the heartbeat, it would corrupt the + # frame rate itself, since fps is frames divided by this delta. + current_time = time.monotonic() if current_time - last_fps_log_time >= fps_log_interval: fps = fps_frame_count / (current_time - last_fps_log_time) p99 = _percentile(sorted(frame_times), 0.99) - logger.info( - "Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms", - fps, self.vegas_config.target_fps, fps_frame_count, - p99 * 1000.0, frame_worst * 1000.0 - ) + target = self.vegas_config.target_fps + degraded = target > 0 and fps < target * _FPS_HEALTHY_FRACTION + due = (current_time - self._fps_last_health_log + >= _FPS_HEARTBEAT_INTERVAL) + if degraded or self._fps_was_degraded or due: + logger.info( + "Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms", + fps, target, fps_frame_count, + p99 * 1000.0, frame_worst * 1000.0 + ) + self._fps_last_health_log = current_time + else: + logger.debug( + "Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms", + fps, target, fps_frame_count, + p99 * 1000.0, frame_worst * 1000.0 + ) + self._fps_was_degraded = degraded last_fps_log_time = current_time fps_frame_count = 0 frame_worst = 0.0 diff --git a/src/web_interface/secret_helpers.py b/src/web_interface/secret_helpers.py index 6a00cb94..9aa2eccc 100644 --- a/src/web_interface/secret_helpers.py +++ b/src/web_interface/secret_helpers.py @@ -5,7 +5,7 @@ Provides functions for identifying, masking, separating, and filtering secret fields in plugin configurations based on JSON Schema x-secret markers. """ -from typing import Any, Dict, Set, Tuple +from typing import Any, Dict, Optional, Set, Tuple def find_secret_fields(properties: Dict[str, Any], prefix: str = '') -> Set[str]: @@ -202,11 +202,89 @@ def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]: nested = remove_empty_secrets(v) if nested: result[k] = nested + elif isinstance(v, list): + # Lists used to fall through to the scalar branch below and be + # kept verbatim, blanks and all. Because lists merge by + # *replacement*, saving any unrelated setting then wrote + # [{"token": ""}, ...] straight over the stored list and + # destroyed every credential in it. + pruned = _prune_secret_list(v) + if pruned is not None: + result[k] = pruned elif v is not None and not (isinstance(v, str) and v.strip() == ''): result[k] = v return result +def _prune_secret_list(items: list) -> Optional[list]: + """Strip blanks from inside a list of secrets, preserving every index. + + The rest of the system treats a secrets list as *parallel* to the regular + one -- ``sec[i]`` holds the secret fields of item ``i``, and ``{}`` means + "item i has none" (see ConfigManager._strip_secrets_recursive). So an + emptied dict item stays ``{}``: putting ``None`` there makes that list stop + looking parallel, and the stripper then drops the whole key from the main + config, taking the non-secret fields with it. + + A blank *scalar* becomes ``None``, meaning "no update at this index" -- + :func:`merge_secrets` substitutes whatever is stored there. Returns + ``None`` when nothing in the list carries a real value, so the caller drops + the key and leaves the stored list untouched. + """ + pruned: list = [] + has_real_value = False + for item in items: + if isinstance(item, dict): + kept = remove_empty_secrets(item) + pruned.append(kept) + has_real_value = has_real_value or bool(kept) + elif isinstance(item, list): + sub = _prune_secret_list(item) + pruned.append(sub if sub is not None else []) + has_real_value = has_real_value or sub is not None + elif item is not None and not (isinstance(item, str) and item.strip() == ''): + pruned.append(item) + has_real_value = True + else: + pruned.append(None) + return pruned if has_real_value else None + + +def merge_secrets(stored: Any, incoming: Any) -> Any: + """Merge submitted secrets over stored ones, element-wise inside lists. + + ``deep_merge`` replaces a list wholesale. For secrets that is destructive: + an incoming list that carries a real value for one entry and ``None`` for + the rest would drop the stored credentials of every other entry. Here a + list merges by index, and ``None`` means "keep what is stored". + + Entries are matched by *position*, which is what the config form gives us + -- there is no schema-declared identity to key on, and it is the same + contract ConfigManager._strip_secrets_recursive already relies on. The + incoming list's length wins, so deleting an item deletes its secrets; + an item the client left blank keeps whatever is stored at that index. + """ + if isinstance(stored, dict) and isinstance(incoming, dict): + merged = dict(stored) + for key, value in incoming.items(): + merged[key] = (merge_secrets(stored[key], value) + if key in stored else value) + return merged + if isinstance(stored, list) and isinstance(incoming, list): + # The incoming list sets the length -- the regular config's list is + # authoritative about how many items exist, and this one runs parallel + # to it. Removing an entry must therefore remove its secrets too. + merged_list = [] + for index, item in enumerate(incoming): + stored_item = stored[index] if index < len(stored) else None + merged_list.append(stored_item if item is None + else merge_secrets(stored_item, item)) + return merged_list + if incoming is None: + return stored + return incoming + + def strip_masked_values(secrets: Dict[str, Any]) -> Dict[str, Any]: """Remove values a client echoed back rather than changed. diff --git a/test/test_api_v3_non_finite_numbers.py b/test/test_api_v3_non_finite_numbers.py new file mode 100644 index 00000000..4e6c217b --- /dev/null +++ b/test/test_api_v3_non_finite_numbers.py @@ -0,0 +1,85 @@ +"""Non-finite JSON numbers must be rejected, not raise. + +json.loads accepts Infinity/-Infinity/NaN by default (they are not valid JSON, +but Python's parser emits them) and Flask's get_json passes them straight +through. int(float('inf')) raises OverflowError, which is neither ValueError +nor TypeError -- so validation blocks that carefully caught those let it +through and Flask turned it into a 500. + +The damage was not the status code. /config/dim-schedule answered with +CONFIG_SAVE_FAILED and suggested "Check file permissions on config directory" +and "Check available disk space" for what was actually an invalid number. + +NaN already returned 400 (int(nan) raises ValueError), which is why this only +showed up for the infinities. +""" +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + + +#: (route, field) that returned 500 before OverflowError was caught. Both +#: infinity signs are exercised: int() raises OverflowError for either, but +#: only one of them was in the original report, and a guard that special-cased +#: the sign would pass a one-sided test. +NON_FINITE_ROUTES = [ + ('/api/v3/config/dim-schedule', 'dim_brightness'), + ('/api/v3/errors/clear', 'max_age_hours'), + ('/api/v3/config/main', 'multiplexing'), + ('/api/v3/config/main', 'row_address_type'), +] +NON_FINITE_CASES = [ + (route, '{"%s": %s}' % (field, literal)) + for route, field in NON_FINITE_ROUTES + for literal in ('Infinity', '-Infinity') +] + + +@pytest.mark.parametrize("route,body", NON_FINITE_CASES) +def test_infinity_is_a_client_error_not_a_server_error(api_v3_client, route, body): + """Exactly 400, not merely "some 4xx". + + Accepting any 4xx would let a 404 pass, so renaming one of these routes + would leave the test green while testing nothing -- the failure mode this + whole file exists to catch. + """ + response = api_v3_client.post(route, data=body, content_type='application/json') + assert response.status_code == 400, ( + f"{route} with {body} answered {response.status_code}; expected 400" + ) + + +@pytest.mark.parametrize("route,body", [ + ('/api/v3/config/dim-schedule', '{"dim_brightness": NaN}'), + ('/api/v3/errors/clear', '{"max_age_hours": NaN}'), +]) +def test_nan_is_also_a_client_error(api_v3_client, route, body): + """int(nan) raises ValueError so this path already worked -- pinned so a + refactor that narrows the except tuple cannot quietly break it.""" + response = api_v3_client.post(route, data=body, content_type='application/json') + assert response.status_code == 400 + + +def test_a_valid_number_is_accepted(api_v3_client, api_v3_module, monkeypatch): + """Prove the widened except did not start swallowing ordinary input. + + Asserting "not a 400" would not show that: the mocked save path fails for + any input, so the assertion would hold even if validation had rejected the + value. Give load_config a real dict and stub the atomic save, and the + endpoint reaches its success response -- which only happens if 30 passed + validation. + """ + api_v3_module.api_v3.config_manager.load_config.return_value = {} + monkeypatch.setattr(api_v3_module, '_save_config_atomic', + lambda *a, **k: (True, '')) + response = api_v3_client.post( + '/api/v3/config/dim-schedule', + data='{"dim_brightness": 30}', + content_type='application/json', + ) + assert response.status_code == 200, response.get_data(as_text=True)[:200] diff --git a/test/test_background_fetch_dedupe.py b/test/test_background_fetch_dedupe.py new file mode 100644 index 00000000..4398a474 --- /dev/null +++ b/test/test_background_fetch_dedupe.py @@ -0,0 +1,450 @@ +"""A second request for a key already being fetched must join, not duplicate. + +request_id embeds a millisecond timestamp and active_requests is keyed by it, +so every submit looked new and nothing compared what was actually being +fetched. On a real board the season-schedule cache_key is requested by both +the Recent and the Upcoming manager: they miss the cache in the same +millisecond and each start a full download and parse of the same payload. +Measured on a running board, 138 background fetches in 24 hours arriving in +pairs at identical timestamps -- half of them redundant. + +The cost of a duplicate is a second download, a second JSON parse (the +expensive part on a Pi), and a second parsed copy resident at the same time. +Schedules on that board run from 256KB to 20MB. It also consumes a second of +the three executor slots with identical work, which is what makes two large +parses peak simultaneously. +""" + +import threading +import time +from unittest.mock import MagicMock, Mock, patch + +import pytest +import requests + +from src.background_data_service import BackgroundDataService, FetchStatus + + +PAYLOAD = {"events": [{"id": f"g{i}"} for i in range(20)]} + + +@pytest.fixture +def cache(): + m = MagicMock() + m.get.return_value = None # always a miss: force the fetch path + m.set.return_value = None + return m + + +@pytest.fixture +def service(cache): + svc = BackgroundDataService(cache, max_workers=3, request_timeout=5) + yield svc + svc.shutdown(wait=False) + + +def _resp(): + r = Mock() + r.json.return_value = PAYLOAD + r.raise_for_status.return_value = None + return r + + +def _wait(service, req_id, timeout=5): + deadline = time.time() + timeout + while not service.is_request_complete(req_id) and time.time() < deadline: + time.sleep(0.02) + + +class _BlockingSession: + """Holds the first fetch open so a second can be submitted mid-flight.""" + + def __init__(self): + self.calls = 0 + self.release = threading.Event() + self.started = threading.Event() + + def get(self, *a, **k): + self.calls += 1 + self.started.set() + self.release.wait(timeout=5) + return _resp() + + +def test_a_second_submit_for_the_same_key_does_not_fetch_twice(service): + session = _BlockingSession() + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="nba_2026", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + + second = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="nba_2026", + callback=lambda r: None, max_retries=0) + + assert second == first, "the joiner should share the in-flight request id" + session.release.set() + _wait(service, first) + + assert session.calls == 1, f"the payload was fetched {session.calls} times" + + +def test_the_joiner_still_gets_its_callback(service): + session = _BlockingSession() + seen = [] + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: seen.append("first"), max_retries=0) + assert session.started.wait(timeout=5) + joined = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: seen.append("second"), max_retries=0) + # Assert the coalescing happened, otherwise this passes trivially: + # two independent requests would each fire their own callback and the + # test would say nothing about the joined path. + assert joined == first + session.release.set() + _wait(service, first) + + deadline = time.time() + 5 + while len(seen) < 2 and time.time() < deadline: + time.sleep(0.02) + assert sorted(seen) == ["first", "second"], ( + f"both submitters must be called back, got {seen}") + + +def test_one_callback_raising_does_not_silence_the_other(service): + session = _BlockingSession() + seen = [] + + def boom(result): + raise RuntimeError("consumer blew up") + + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=boom, max_retries=0) + assert session.started.wait(timeout=5) + joined = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: seen.append("survivor"), max_retries=0) + # Same reason: without coalescing these are separate requests and + # neither callback can affect the other. + assert joined == first + session.release.set() + _wait(service, first) + + deadline = time.time() + 5 + while not seen and time.time() < deadline: + time.sleep(0.02) + assert seen == ["survivor"] + + +def test_different_keys_are_not_coalesced(service): + session = _BlockingSession() + with patch.object(service, "session", session): + a = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/a", cache_key="key_a", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + b = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/b", cache_key="key_b", + callback=lambda r: None, max_retries=0) + assert a != b, "different cache keys must not share a request" + session.release.set() + _wait(service, a) + _wait(service, b) + assert session.calls == 2 + + +def test_a_later_submit_after_completion_fetches_again(service): + """Dedupe is for concurrent requests only, not a second cache layer.""" + with patch.object(service.session, "get", side_effect=[_resp(), _resp()]) as get: + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + _wait(service, first) + second = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + _wait(service, second) + assert first != second + assert get.call_count == 2 + + +def test_cancelling_releases_the_key(service): + """A cancelled request must not wedge its key against future fetches.""" + session = _BlockingSession() + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + service.cancel_request(first) + assert "k" not in service._inflight_by_cache_key + session.release.set() + + +def test_a_stranded_index_entry_cannot_wedge_a_key(service): + """Defensive: the request is looked up, not trusted from the id alone.""" + service._inflight_by_cache_key["ghost"] = "no_such_request" + with patch.object(service.session, "get", return_value=_resp()): + req = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="ghost", + callback=lambda r: None, max_retries=0) + _wait(service, req) + assert service.get_result(req).success is True + + +def test_the_deduplicated_count_is_reported(service): + session = _BlockingSession() + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + max_retries=0) + session.release.set() + _wait(service, first) + assert service.get_statistics().get("deduplicated_requests") == 1 + + +def test_a_cancelled_worker_cannot_overwrite_its_replacement(service, cache): + """Cancelling frees the key, so a replacement may already own it. + + The worker cannot abort an HTTP call in flight, so when the cancelled one + finally returns it must discard its response rather than write it. Without + that, the sequence is: cancel A, submit B for the same key, B fetches and + caches fresh data, A returns and overwrites it with the response nobody + wanted -- and calls A's callbacks too. + """ + slow = _BlockingSession() + stale = {"events": [{"id": "STALE"}]} + slow_resp = Mock() + slow_resp.json.return_value = stale + slow_resp.raise_for_status.return_value = None + + def blocked_get(*a, **k): + slow.calls += 1 + slow.started.set() + slow.release.wait(timeout=5) + return slow_resp + + called = [] + with patch.object(service.session, "get", side_effect=blocked_get): + first = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: called.append("cancelled_one"), max_retries=0) + assert slow.started.wait(timeout=5) + + service.cancel_request(first) + assert "k" not in service._inflight_by_cache_key + + # The replacement writes the fresh value while the cancelled fetch is held. + fresh = {"events": [{"id": "FRESH"}]} + fresh_resp = Mock() + fresh_resp.json.return_value = fresh + fresh_resp.raise_for_status.return_value = None + with patch.object(service.session, "get", return_value=fresh_resp): + second = service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", cache_key="k", + callback=lambda r: called.append("replacement"), max_retries=0) + _wait(service, second) + + assert cache.set.call_args[0][1] == fresh, "replacement must own the cache" + + # Now let the cancelled fetch finish. It must write nothing and call nobody. + # Wait for the worker to actually finish rather than sleeping: a fixed + # sleep is a race under load, and a slow worker would make this pass for + # the wrong reason. A cancelled request is still filed in + # completed_requests, so that is the signal it has run to completion. + writes_before = cache.set.call_count + slow.release.set() + deadline = time.time() + 5 + while first not in service.completed_requests and time.time() < deadline: + time.sleep(0.02) + assert first in service.completed_requests, "cancelled worker never finished" + + assert cache.set.call_count == writes_before, ( + "the cancelled worker wrote to the cache after its replacement") + assert cache.set.call_args[0][1] == fresh, "stale data overwrote fresh" + assert "cancelled_one" not in called, ( + "a cancelled request must not deliver callbacks") + + +def test_request_ids_are_unique_within_a_millisecond(service): + """request_id was sport_year_milliseconds, which collides. + + Two submits inside the same millisecond produced the SAME id, so one + silently replaced the other in active_requests and completed_requests. + Dedupe hands this id back to every joiner as their handle for + get_result(), so uniqueness is now load-bearing rather than incidental. + """ + # Stub the executor rather than the session: this is about what submit + # hands back, and letting 50 workers loose would outlive the patch and + # make real network calls. + with patch.object(service.executor, "submit"): + ids = [ + service.submit_fetch_request( + sport="nba", year=2026, url="https://x/s", + cache_key=f"key_{i}", # distinct keys: no dedupe + callback=lambda r: None, max_retries=0) + for i in range(50) + ] + assert len(set(ids)) == len(ids), "request ids collided" + + +# --- cancellation must be terminal ------------------------------------------ +# +# Cancelling used to be advisory: three separate paths wrote request.status +# without checking whether the request had already been cancelled, so a cancel +# could be silently undone and the work it was meant to stop went ahead. + +URL = "http://example.invalid/scores" +KEY = "sched_nfl_2025" + + +class _CountingSession: + """Records whether an HTTP fetch was ever attempted.""" + + def __init__(self): + self.calls = 0 + + def get(self, *a, **k): + self.calls += 1 + return _resp() + + +class _BlockingFailingSession(_CountingSession): + """Holds the fetch open, then fails it. + + Cancelling while the worker is parked inside the HTTP call is the only way + to reach the exception handler as a cancelled request. Cancel it before + the call starts and the worker returns at the pre-start branch instead, + which would leave the except path untested. + """ + + def __init__(self): + super().__init__() + self.started = threading.Event() + self.release = threading.Event() + + def get(self, *a, **k): + self.calls += 1 + self.started.set() + self.release.wait(timeout=5) + raise requests.RequestException("connection reset") + + +def _fill_every_worker_slot(service, slots=3): + """Occupy the pool so the next submit is queued rather than started. + + This is what makes "cancel before the worker runs" deterministic instead + of a race the test would win only sometimes. Returns the gate that + releases the pool. + """ + gate = threading.Event() + for _ in range(slots): + service.executor.submit(gate.wait, 5) + return gate + + +def test_cancelling_before_the_worker_starts_stops_the_fetch(service, cache): + """The queued worker must honour a cancel, not overwrite it with IN_PROGRESS. + + Between submit and the worker picking the job up, the request sits in the + executor queue. Cancelling there is the cheapest possible cancel -- nothing + has been downloaded yet -- and it was the one that did not work. + """ + gate = _fill_every_worker_slot(service) + session = _CountingSession() + delivered = [] + + with patch.object(service, 'session', session): + rid = service.submit_fetch_request( + "nfl", 2025, URL, KEY, max_retries=0, callback=delivered.append + ) + assert service.cancel_request(rid) is True + gate.set() # let the queued worker run + _wait(service, rid) + + assert session.calls == 0, ( + "cancelled before it started, yet the worker still downloaded the payload" + ) + assert cache.set.call_count == 0, "a cancelled request wrote to the cache" + assert delivered == [], "a cancelled request invoked its callbacks" + + +def test_a_cancel_during_the_commit_is_refused(service, cache): + """Once the worker has claimed the commit, cancelling is too late. + + The claim and the cancelled-check happen in one critical section, so a + cancel arriving after it cannot retroactively abandon data already on its + way to the cache. Letting it through stranded every joiner: the payload + landed in the cache but the callbacks were suppressed, so a manager that + joined this fetch waited for a call that never came. + """ + gate = _fill_every_worker_slot(service) + late = {} + delivered = [] + + def cancel_mid_write(key, data, *a, **k): + late['returned'] = service.cancel_request(late['rid']) + + cache.set.side_effect = cancel_mid_write + + # Read the payload inside the callback. The service releases result.data + # once every callback has been delivered, so inspecting the FetchResult + # afterwards sees the released object, not what the caller was handed. + def record(result): + delivered.append((result.success, result.data)) + + with patch.object(service, 'session', _CountingSession()): + late['rid'] = service.submit_fetch_request( + "nfl", 2025, URL, KEY, max_retries=0, callback=record + ) + gate.set() # only now can the worker reach the commit + _wait(service, late['rid']) + + assert late.get('returned') is False, ( + "cancelled a request that had already committed" + ) + assert cache.set.call_count == 1, "the commit itself was lost" + assert delivered == [(True, PAYLOAD)], ( + "data reached the cache but the callbacks were suppressed -- " + "every joined submitter is left waiting forever" + ) + + +def test_a_failure_after_cancelling_stays_cancelled(service, cache): + """A cancelled request that then errors must not resurface as FAILED. + + The except path overwrote CANCELLED with FAILED, and the callback gate in + the finally block only suppresses callbacks for CANCELLED -- so cancelling + a request that was about to time out delivered a spurious error callback. + """ + session = _BlockingFailingSession() + delivered = [] + + with patch.object(service, 'session', session): + rid = service.submit_fetch_request( + "nfl", 2025, URL, KEY, max_retries=0, callback=delivered.append + ) + # Assert the worker is inside the HTTP call before cancelling, + # otherwise this silently degrades into the pre-start case and the + # exception handler is never exercised. + assert session.started.wait(timeout=5) + assert service.cancel_request(rid) is True + session.release.set() + _wait(service, rid) + + assert session.calls == 1, "the fetch never started, so nothing could fail" + + assert delivered == [], "a cancelled request delivered a failure callback" + assert service.get_request_status(rid) is FetchStatus.CANCELLED, ( + "a cancelled request that then errored was reported as FAILED" + ) diff --git a/test/test_background_payload_release.py b/test/test_background_payload_release.py new file mode 100644 index 00000000..2c90a8da --- /dev/null +++ b/test/test_background_payload_release.py @@ -0,0 +1,280 @@ +"""A delivered fetch payload must not stay resident on the stored result. + +BackgroundDataService kept the fetched body on the FetchResult it filed in +`completed_requests`, which is swept only hourly and capped at 500 entries by +count. For status records that is free; for a season schedule it is not. NCAA +football's 2026 schedule is 946 games, and on a 1GB Pi 3B+ the parsed payload +measured ~90MB -- a tenth of the board's memory, pinned for an hour after the +consumer had already been handed it. + +The cache-hit path was the worse of the two. It runs once per update interval +per sport, mints a fresh request_id each time, and hands back whatever the +cache returns -- so a memory-tier miss (the tier is capped at 150 entries) +re-parses the payload from disk into a genuinely new object. Those accumulate +as separate copies rather than shared references, which is the staircase seen +in the field: RSS stepping up ~90MB per sport as seasons loaded and never +coming back down. + +Releasing is safe because the payload is written to the cache under the +request's cache_key before the result is built, and that is where consumers +read it from -- the callback is handed the object directly and the plugins use +it only in passing before reading the cache back. + +Requests submitted *without* a callback keep their payload: polling +get_result() is then the only way to collect it, so releasing would break that +contract. + +And the release must happen after EVERY callback, not after each one. Callers +that joined an in-flight fetch share a single FetchResult, so releasing per +delivery strips the payload out from under everyone still queued -- see +TestJoinersAllGetTheData. +""" + +import threading +import time +import pytest +from unittest.mock import MagicMock, Mock, patch + +from src.background_data_service import BackgroundDataService + + +PAYLOAD = {"events": [{"id": f"g{i}"} for i in range(50)]} + + +@pytest.fixture +def cache(): + m = MagicMock() + m.get.return_value = None + m.set.return_value = None + return m + + +@pytest.fixture +def service(cache): + svc = BackgroundDataService(cache, max_workers=2, request_timeout=5) + yield svc + svc.shutdown(wait=False) + + +def _wait(service, req_id, timeout=5): + """Wait for the result to be FILED. + + Enough for anything that is true by the time the worker stores the result: + its success flag, its error, the cache write that happened during the + fetch. + """ + deadline = time.time() + timeout + while not service.is_request_complete(req_id) and time.time() < deadline: + time.sleep(0.02) + + +def _wait_for_release(service, req_id, timeout=5): + """Wait for the payload to be RELEASED, which is strictly later. + + The worker files the result, then runs the callback, then releases. So + is_request_complete() goes true while the callback still has not run -- + waiting on it alone leaves a window in which `seen` is empty and the + payload is still resident, and the assertions race the worker. It passes + in practice only because a one-line callback usually beats the 20ms poll. + + Release happens after the callback returns, so a released payload also + means the callback has finished: one wait covers both. + """ + deadline = time.time() + timeout + while time.time() < deadline: + result = service.get_result(req_id) + if result is not None and result.data is None: + return + time.sleep(0.02) + raise AssertionError( + f"payload for {req_id} was never released (callback may not have run)") + + +def _resp(): + r = Mock() + r.json.return_value = PAYLOAD + r.raise_for_status.return_value = None + return r + + +class TestFetchPath: + def test_callback_receives_the_payload_then_it_is_released(self, service, cache): + seen = {} + + def callback(result): + # The consumer's one look at the data happens here. + seen['events'] = len(result.data['events']) + + with patch.object(service.session, "get", return_value=_resp()): + req_id = service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", callback=callback, max_retries=0, + ) + _wait_for_release(service, req_id) + + assert seen['events'] == 50, "callback must still be handed the payload" + + stored = service.get_result(req_id) + assert stored is not None + assert stored.success is True + assert stored.data is None, "payload must not stay on the stored result" + + def test_nothing_is_lost_the_cache_holds_it(self, service, cache): + with patch.object(service.session, "get", return_value=_resp()): + req_id = service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", callback=lambda r: None, max_retries=0, + ) + _wait(service, req_id) + + cache.set.assert_called_once() + key, written = cache.set.call_args[0][:2] + assert key == "ncaa_fb_2026" + assert written == PAYLOAD, "the payload must be persisted before release" + + def test_without_a_callback_the_payload_is_kept(self, service, cache): + # Polling get_result() is then the only delivery mechanism. + with patch.object(service.session, "get", return_value=_resp()): + req_id = service.submit_fetch_request( + sport="nfl", year=2026, url="https://example.com/s", + cache_key="nfl_2026", max_retries=0, + ) + _wait(service, req_id) + + assert service.get_result(req_id).data == PAYLOAD + + def test_a_failed_fetch_still_records_its_error(self, service, cache): + with patch.object(service.session, "get", side_effect=Exception("boom")): + req_id = service.submit_fetch_request( + sport="nfl", year=2026, url="https://example.com/s", + cache_key="nfl_2026", callback=lambda r: None, max_retries=0, + ) + _wait(service, req_id) + + stored = service.get_result(req_id) + assert stored.success is False + assert stored.error is not None + + +class TestCacheHitPath: + def test_cache_hit_releases_after_the_callback(self, service, cache): + cache.get.return_value = PAYLOAD + seen = {} + + req_id = service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", + callback=lambda r: seen.update(events=len(r.data['events'])), + ) + + assert seen['events'] == 50 + assert service.get_result(req_id).data is None + + def test_repeated_cache_hits_do_not_accumulate_payloads(self, service, cache): + # The staircase: one entry per update interval per sport, each one + # potentially a freshly parsed copy after a memory-tier miss. + cache.get.return_value = PAYLOAD + + for _ in range(25): + service.submit_fetch_request( + sport="ncaa_fb", year=2026, url="https://example.com/s", + cache_key="ncaa_fb_2026", callback=lambda r: None, + ) + + retained = [r for r in service.completed_requests.values() if r.data is not None] + assert retained == [], f"{len(retained)} payloads still resident" + + def test_cache_hit_without_a_callback_is_unchanged(self, service, cache): + cache.get.return_value = PAYLOAD + req_id = service.submit_fetch_request( + sport="nfl", year=2026, url="https://example.com/s", + cache_key="nfl_2026", + ) + assert service.get_result(req_id).data == PAYLOAD + + +class TestJoinersAllGetTheData: + """Deduplicated callers share one FetchResult; releasing between them + empties it for the rest. + + Not a corner case. A sport's recent, upcoming and live managers all ask for + the same season schedule, so the second and third are joiners on almost + every cycle. Releasing inside the delivery loop handed the payload to + whichever ran first and gave the others `result.data is None`. + + The consequence was worse than a quiet degradation, because consumers do + `result.data.get('events')`: they raised AttributeError, the delivery loop + caught it, and the whole failure surfaced as a single + "Error in callback for request ..." line while that manager silently never + received its schedule. + """ + + class _BlockingSession: + """Holds the fetch open so a second submit lands while in flight.""" + + def __init__(self): + self.release = threading.Event() + self.started = threading.Event() + + def get(self, *a, **k): + self.started.set() + self.release.wait(timeout=5) + return _resp() + + def test_every_joiner_is_handed_the_payload(self, service): + session = self._BlockingSession() + seen = {} + + def record(name): + # Read it the way the sport managers do. `result.data['events']` + # would raise TypeError on None; `.get` raises AttributeError, + # which is the error actually seen in the field. + def cb(result): + seen[name] = result.data.get('events') if result.data else None + return cb + + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=record("first"), max_retries=0) + assert session.started.wait(timeout=5) + + joined = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=record("second"), max_retries=0) + # Without coalescing these are two independent fetches that each + # own their result, and the test proves nothing about sharing. + assert joined == first, "the joiner should share the in-flight id" + + session.release.set() + _wait(service, first) + + deadline = time.time() + 5 + while len(seen) < 2 and time.time() < deadline: + time.sleep(0.02) + + assert set(seen) == {"first", "second"}, f"both must be called, got {seen}" + for name, events in seen.items(): + assert events is not None, ( + f"{name!r} was handed a released payload: the result was " + f"emptied before every callback had been delivered") + assert len(events) == 50, f"{name!r} got {events!r}" + + def test_the_payload_is_still_released_once_they_have_all_had_it(self, service): + """The memory fix must survive the ordering fix.""" + session = self._BlockingSession() + + with patch.object(service, "session", session): + first = service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=lambda r: None, max_retries=0) + assert session.started.wait(timeout=5) + service.submit_fetch_request( + sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026", + callback=lambda r: None, max_retries=0) + session.release.set() + _wait_for_release(service, first) + + stored = service.get_result(first) + assert stored is not None and stored.data is None, ( + "the payload must still be dropped once every callback has run") diff --git a/test/test_core_version_freshness.py b/test/test_core_version_freshness.py new file mode 100644 index 00000000..55cdaef9 --- /dev/null +++ b/test/test_core_version_freshness.py @@ -0,0 +1,122 @@ +"""The gate must read the core version from disk, not from its own import. + +`from src import __version__` binds whatever the process loaded at start. The +web UI is a long-lived service of its own (ledmatrix-web.service), and updating +the core replaces files on disk without restarting it -- the update route says +so and asks the user to restart, but its prompt named only the *display* +service, so a user who followed it left the web process holding the old number. + +The plugin store's gate lives in that web process, so being stale by exactly +one release is the case that bites: every plugin flooring on the release you +just installed is refused, with a message blaming a core version that is +already correct on disk. + +Observed on hardware: after updating a rig to 3.3.0 and restarting only the +display service, all eight sports scoreboards were refused with +"supports LEDMatrix >=3.3.0, but this system is running 3.2.0" while +src/__init__.py on that machine read 3.3.0. +""" + +import importlib +import sys + +import pytest + +from src.plugin_system import compatibility + + +class TestCurrentCoreVersion: + def test_it_reads_the_file_rather_than_the_imported_value(self, monkeypatch, tmp_path): + # Simulate a process whose import predates the update: the module + # object says 3.2.0 while the file on disk says 3.3.0. + import src + monkeypatch.setattr(src, "__version__", "3.2.0") + fake = tmp_path / "__init__.py" + fake.write_text('__version__ = "3.3.0"\n', encoding="utf-8") + monkeypatch.setattr(compatibility, "_VERSION_FILE", fake) + assert compatibility.current_core_version() == "3.3.0" + + def test_it_matches_the_real_file_by_default(self): + import src + assert compatibility.current_core_version() == src.__version__ + + @pytest.mark.parametrize("body", [ + "__version__ = '3.4.1'\n", + '__version__="3.4.1"\n', + '"""doc"""\n\n__version__ = "3.4.1" # trailing comment\n', + ]) + def test_it_tolerates_the_ways_that_line_gets_written(self, monkeypatch, tmp_path, body): + fake = tmp_path / "__init__.py" + fake.write_text(body, encoding="utf-8") + monkeypatch.setattr(compatibility, "_VERSION_FILE", fake) + assert compatibility.current_core_version() == "3.4.1" + + def test_a_missing_file_falls_back_to_the_import(self, monkeypatch, tmp_path): + # Never worse than before: an unreadable file returns what the old + # code would have returned. + import src + monkeypatch.setattr(src, "__version__", "3.2.0") + monkeypatch.setattr(compatibility, "_VERSION_FILE", tmp_path / "gone.py") + assert compatibility.current_core_version() == "3.2.0" + + def test_a_file_without_the_line_falls_back(self, monkeypatch, tmp_path): + import src + monkeypatch.setattr(src, "__version__", "3.2.0") + fake = tmp_path / "__init__.py" + fake.write_text("# no version here\n", encoding="utf-8") + monkeypatch.setattr(compatibility, "_VERSION_FILE", fake) + assert compatibility.current_core_version() == "3.2.0" + + def test_it_never_raises(self, monkeypatch, tmp_path): + # This runs on the install path; an exception here would surface as a + # failed update rather than a version mismatch. + bad = tmp_path / "__init__.py" + bad.write_bytes(b"\xff\xfe\x00 not utf-8 \xff") + monkeypatch.setattr(compatibility, "_VERSION_FILE", bad) + assert isinstance(compatibility.current_core_version(), str) + + +class TestTheBugItFixes: + def test_a_stale_import_no_longer_refuses_a_compatible_plugin(self, monkeypatch, tmp_path): + """The exact hardware failure, as a test.""" + import src + monkeypatch.setattr(src, "__version__", "3.2.0") # what the process holds + fake = tmp_path / "__init__.py" + fake.write_text('__version__ = "3.3.0"\n', encoding="utf-8") # what is on disk + monkeypatch.setattr(compatibility, "_VERSION_FILE", fake) + + manifest = {"name": "Hockey Scoreboard", "min_ledmatrix_version": "3.3.0"} + + stale_ok, _ = compatibility.check(manifest, src.__version__) + assert stale_ok is False, "precondition: the stale value is what refused it" + + fresh_ok, reason = compatibility.check( + manifest, compatibility.current_core_version()) + assert fresh_ok is True, f"the disk version must allow it, got: {reason}" + + def test_it_still_refuses_when_the_core_really_is_too_old(self, monkeypatch, tmp_path): + # The gate must not become permissive: a genuinely old core still says no. + fake = tmp_path / "__init__.py" + fake.write_text('__version__ = "3.2.0"\n', encoding="utf-8") + monkeypatch.setattr(compatibility, "_VERSION_FILE", fake) + ok, reason = compatibility.check( + {"name": "Hockey", "min_ledmatrix_version": "3.3.0"}, + compatibility.current_core_version()) + assert ok is False + assert "3.2.0" in (reason or "") + + +class TestCallSites: + @pytest.mark.parametrize("module", [ + "src.plugin_system.store_manager", + "src.plugin_system.plugin_loader", + ]) + def test_no_gate_binds_the_version_at_import(self, module): + """Catch a future call site reintroducing the stale read.""" + import inspect + mod = importlib.import_module(module) + source = inspect.getsource(mod) + assert "from src import __version__ as core_version" not in source, ( + f"{module} binds __version__ at import; use " + f"compatibility.current_core_version() so a long-lived process " + f"sees a core update.") diff --git a/test/test_display_controller_plugin_toggle.py b/test/test_display_controller_plugin_toggle.py index 0980e729..e09933d0 100644 --- a/test/test_display_controller_plugin_toggle.py +++ b/test/test_display_controller_plugin_toggle.py @@ -6,6 +6,7 @@ These tests cover the reconcile path that loads/unloads plugins and rebuilds the dispatch maps on the main thread when the enabled set changes. """ +import copy from unittest.mock import MagicMock @@ -253,3 +254,182 @@ class TestEnabledSetChanged: {"a": {"enabled": True, "duration": 30}}, {"a": {"enabled": True, "duration": 45}}, ) is False + + +class TestEnabledPluginNotRunning: + """A plugin that fails validate_config() is enabled but absent, and the + config edit that fixes it is nested inside the plugin's own section -- so + the top-level ``enabled`` comparison never sees it. These cover the second + gate that queues a reconcile in that case. + """ + + def test_nested_edit_is_invisible_to_the_enabled_set_check(self, test_display_controller): + """The original gate: proves why a second one is needed.""" + controller = test_display_controller + old = {"hockey-scoreboard": {"enabled": True, "nhl": {"enabled": False}}} + new = {"hockey-scoreboard": {"enabled": True, "nhl": {"enabled": True}}} + # Enabling a league changes no top-level flag. + assert controller._enabled_set_changed(old, new) is False + + def test_queues_reconcile_when_enabled_plugin_is_absent(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} # failed to load + cfg = {"hockey-scoreboard": {"enabled": True, "nhl": {"enabled": True}}} + assert controller._enabled_plugin_not_running(cfg) is True + + def test_quiet_when_every_enabled_plugin_is_running(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {"hockey-scoreboard": ["nhl"]} + cfg = {"hockey-scoreboard": {"enabled": True}} + assert controller._enabled_plugin_not_running(cfg) is False + + def test_disabled_plugin_does_not_queue(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} + cfg = {"hockey-scoreboard": {"enabled": False}} + assert controller._enabled_plugin_not_running(cfg) is False + + def test_non_plugin_sections_do_not_queue(self, test_display_controller): + """``schedule``/``display`` carry their own ``enabled`` and are never + in plugin_display_modes -- without the manifest check they would queue + a reconcile, and therefore a filesystem scan, on every config save.""" + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {"hockey-scoreboard": ["nhl"]} + cfg = { + "hockey-scoreboard": {"enabled": True}, + "schedule": {"enabled": True}, + "display": {"enabled": True}, + } + assert controller._enabled_plugin_not_running(cfg) is False + + def test_non_dict_section_is_ignored(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} + assert controller._enabled_plugin_not_running({"hockey-scoreboard": "nonsense"}) is False + + def test_no_plugin_manager_is_quiet(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager = None + assert controller._enabled_plugin_not_running({"x": {"enabled": True}}) is False + + +class TestReconcileQueuedThroughSubscriber: + """End-to-end through the real config-change subscriber, not the helper. + + Without the second gate this is the four-day-outage path: the plugin is + enabled, absent, and the save that enables its league sets no flag. + """ + + @staticmethod + def _subscriber(controller): + subs = controller.config_service._subscribers['*'] + for cb in subs: + if getattr(cb, '__name__', '') == '_controller_config_change': + return cb + raise AssertionError(f"controller subscriber not found among {subs}") + + @staticmethod + def _configs(controller, plugin_section_old, plugin_section_new): + """Build two full configs differing only inside the plugin section -- + the subscriber refreshes its cache from these, so they must be real.""" + base = copy.deepcopy(controller.config) + old = copy.deepcopy(base) + new = copy.deepcopy(base) + old["hockey-scoreboard"] = plugin_section_old + new["hockey-scoreboard"] = plugin_section_new + return old, new + + def test_nested_edit_queues_reconcile_for_absent_plugin(self, test_display_controller): + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {} # validate_config() said False + controller._pending_plugin_reconcile = False + + old, new = self._configs( + controller, + {"enabled": True, "nhl": {"enabled": False}}, + {"enabled": True, "nhl": {"enabled": True}}, + ) + # The original gate is blind to this edit ... + assert controller._enabled_set_changed(old, new) is False + self._subscriber(controller)(old, new) + # ... but the reconcile is queued anyway. + assert controller._pending_plugin_reconcile is True + + def test_steady_state_does_not_queue_reconcile(self, test_display_controller): + """Everything enabled is running: an unrelated edit must not queue a + reconcile, or every config save drags a filesystem scan onto the + render thread.""" + controller = test_display_controller + controller.plugin_manager.plugin_manifests = {"hockey-scoreboard": {}} + controller.plugin_manager.discovered_plugin_ids.return_value = {"hockey-scoreboard"} + controller.plugin_display_modes = {"hockey-scoreboard": ["nhl"]} + controller._pending_plugin_reconcile = False + + old, new = self._configs( + controller, + {"enabled": True, "scroll_speed": 1}, + {"enabled": True, "scroll_speed": 2}, + ) + self._subscriber(controller)(old, new) + + assert controller._pending_plugin_reconcile is False + + +class TestPendingReconcileNotLost: + """A config change arriving *during* reconcile must not be discarded. + + The flag used to be cleared after a successful reconcile. Reconcile has + already read its config by then, so that clear erased a request it never + served and the newest config never reconciled -- the same "my save did + nothing" symptom this path exists to prevent. + """ + + def test_request_arriving_during_reconcile_survives(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = True + + def reconcile_and_race(): + # The watcher thread queues another change while we are mid-flight. + with controller._reconcile_flag_lock: + controller._pending_plugin_reconcile = True + return True + + controller._reconcile_enabled_plugins = reconcile_and_race + controller._service_pending_reconcile() + + assert controller._pending_plugin_reconcile is True, \ + "a config change landing during reconcile was discarded" + + def test_flag_cleared_on_a_quiet_success(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = True + controller._reconcile_enabled_plugins = lambda: True + controller._service_pending_reconcile() + assert controller._pending_plugin_reconcile is False + + def test_retryable_failure_rearms(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = True + controller._reconcile_enabled_plugins = lambda: False + controller._service_pending_reconcile() + assert controller._pending_plugin_reconcile is True + + def test_no_reconcile_when_nothing_pending(self, test_display_controller): + controller = test_display_controller + controller._pending_plugin_reconcile = False + calls = [] + controller._reconcile_enabled_plugins = lambda: calls.append(1) or True + controller._service_pending_reconcile() + assert calls == [] diff --git a/test/test_install_lowmem.py b/test/test_install_lowmem.py index 7143176a..7c9fbbc8 100644 --- a/test/test_install_lowmem.py +++ b/test/test_install_lowmem.py @@ -13,6 +13,7 @@ need root and mutate the system, so they are exercised manually instead. """ import subprocess +import tempfile from pathlib import Path import pytest @@ -31,6 +32,16 @@ def run_lib(snippet: str, env: dict | None = None) -> subprocess.CompletedProces ) +def _fstype_of(path: object) -> str: + """Filesystem type backing ``path``, via the same tool the helper uses.""" + result = subprocess.run( + ["findmnt", "-no", "FSTYPE", "--target", str(path)], + capture_output=True, text=True, + env={"PATH": "/usr/bin:/bin:/usr/sbin:/sbin"}, + ) + return result.stdout.strip() + + def call(fn: str, *args: object, env: dict | None = None) -> str: joined = " ".join(str(a) for a in args) result = run_lib(f"{fn} {joined}", env=env) @@ -195,8 +206,29 @@ class TestOomDetection: class TestDiskBackedTmpdir: def test_returns_nothing_when_tmpdir_is_already_disk_backed(self, tmp_path): - # tmp_path is on the regular filesystem, so the default must be kept. - assert call("lm_disk_backed_tmpdir", env={"TMPDIR": str(tmp_path)}) == "" + # Do not assume tmp_path is disk-backed. Debian 13 -- the platform this + # helper exists for -- mounts /tmp as tmpfs, and pytest puts tmp_path + # under /tmp, so this asserted against a *memory*-backed directory and + # failed on the target platform while the helper behaved exactly as + # designed. Search for a directory whose backing store is really disk. + scratch = None + disk_backed = None + for candidate in (tmp_path, Path("/var/tmp"), LIB.parent): + if _fstype_of(candidate) not in ("tmpfs", "ramfs", ""): + if candidate is tmp_path: + disk_backed = candidate + else: + scratch = Path(tempfile.mkdtemp(dir=str(candidate))) + disk_backed = scratch + break + if disk_backed is None: + pytest.skip("no disk-backed directory available to test against") + try: + assert call("lm_disk_backed_tmpdir", + env={"TMPDIR": str(disk_backed)}) == "" + finally: + if scratch is not None: + scratch.rmdir() def test_redirects_away_from_a_memory_backed_tmpdir(self): # Debian 13 mounts /tmp as tmpfs, which would otherwise hold the whole diff --git a/test/test_logging_config.py b/test/test_logging_config.py index d0cf0fe8..044a5d92 100644 --- a/test/test_logging_config.py +++ b/test/test_logging_config.py @@ -89,11 +89,17 @@ class TestContextualFormatter: assert "hello" in out def test_location_toggle(self): + # Assert on the whole "module.func:lineno" token, not a bare ":42". + # The formatted line starts with an HH:MM:SS timestamp, so a bare + # ":{lineno}" also matches the clock whenever the minute or second + # happens to equal the line number -- about 3% of runs, which is a + # flaky failure with nothing wrong. record = make_record() + location = f"{record.module}.{record.funcName}:{record.lineno}" with_loc = ContextualFormatter(include_location=True).format(record) without = ContextualFormatter(include_location=False).format(record) - assert f":{record.lineno}" in with_loc - assert f":{record.lineno}" not in without + assert location in with_loc + assert location not in without def test_record_not_mutated_no_double_prefix(self): # Regression: a record is formatted once PER HANDLER. The formatter diff --git a/test/test_logo_downloader.py b/test/test_logo_downloader.py index 9b79d45b..c14b3391 100644 --- a/test/test_logo_downloader.py +++ b/test/test_logo_downloader.py @@ -8,11 +8,27 @@ ensure_logo_directory, and the download_missing_logo function path """ import os +import time + import pytest from pathlib import Path from unittest.mock import patch, Mock, MagicMock -from src.logo_downloader import LogoDownloader +from PIL import Image +from PIL.PngImagePlugin import PngInfo + +from src.logo_downloader import ( + PLACEHOLDER_BG, + PLACEHOLDER_MARKER, + PLACEHOLDER_RETRY_SECONDS, + PLACEHOLDER_SIZE, + LogoDownloader, + download_missing_logo, + is_placeholder_logo, + placeholder_age_seconds, + refresh_placeholder_timestamp, + should_attempt_download, +) # --------------------------------------------------------------------------- @@ -127,3 +143,211 @@ class TestEnsureLogoDirectory: with patch("builtins.open", side_effect=mock_open): result = downloader.ensure_logo_directory(test_dir) assert result is False + + +# --------------------------------------------------------------------------- +# Placeholder detection and retry +# +# A failed download used to be cached as a placeholder wearing the real logo's +# filename, and download_missing_logo returned early on "the file exists". One +# transient failure therefore pinned a team to a grey box permanently. +# --------------------------------------------------------------------------- + +class TestPlaceholderLogos: + def _placeholder(self, tmp_path, abbrev="COLL"): + downloader = LogoDownloader() + assert downloader.create_placeholder_logo(abbrev, str(tmp_path)) is True + return tmp_path / f"{abbrev}.png" + + def test_generated_placeholder_is_recognised(self, tmp_path): + assert is_placeholder_logo(self._placeholder(tmp_path)) is True + + def test_real_logo_is_not_a_placeholder(self, tmp_path): + real = tmp_path / "REAL.png" + Image.new("RGBA", (500, 500), (12, 34, 56, 255)).save(real) + assert is_placeholder_logo(real) is False + + def test_legacy_unmarked_placeholder_is_recognised(self, tmp_path): + """Placeholders written before the marker existed must still be caught. + + They are already sitting on users' disks; if they were not recognised + those teams would stay grey boxes forever even after this fix. + """ + legacy = tmp_path / "LEGACY.png" + Image.new("RGBA", PLACEHOLDER_SIZE, PLACEHOLDER_BG).save(legacy) + assert is_placeholder_logo(legacy) is True + + def test_same_size_but_different_colour_is_not_a_placeholder(self, tmp_path): + real = tmp_path / "SMALL.png" + Image.new("RGBA", PLACEHOLDER_SIZE, (10, 200, 10, 255)).save(real) + assert is_placeholder_logo(real) is False + + def test_missing_file_is_not_a_placeholder(self, tmp_path): + assert is_placeholder_logo(tmp_path / "nope.png") is False + + def test_existing_real_logo_short_circuits_without_downloading(self, tmp_path): + real = tmp_path / "REAL.png" + Image.new("RGBA", (500, 500), (1, 2, 3, 255)).save(real) + with patch.object(LogoDownloader, "download_logo") as download: + assert download_missing_logo( + "afl", "1", "REAL", real, logo_url="http://example/x.png") is True + download.assert_not_called() + + def _age_placeholder(self, path, seconds): + """Rewrite a placeholder's marker so it reads as `seconds` old.""" + metadata = PngInfo() + metadata.add_text(PLACEHOLDER_MARKER, str(time.time() - seconds)) + with Image.open(path) as img: + img.copy().save(path, "PNG", pnginfo=metadata) + + def test_stale_placeholder_triggers_a_retry(self, tmp_path): + path = self._placeholder(tmp_path) + self._age_placeholder(path, PLACEHOLDER_RETRY_SECONDS + 60) + assert placeholder_age_seconds(path) > PLACEHOLDER_RETRY_SECONDS + + with patch.object(LogoDownloader, "download_logo", return_value=True) as download: + assert download_missing_logo( + "afl", "1", "COLL", path, + logo_url="http://example/coll.png") is True + download.assert_called_once() + + def test_placeholder_age_survives_an_mtime_touch(self, tmp_path): + """The age comes from the stamp, not the filesystem. + + Anything that rewrites file times -- a backup restore, an rsync, a + permissions fix script -- would otherwise reset the retry clock. + """ + path = self._placeholder(tmp_path) + self._age_placeholder(path, PLACEHOLDER_RETRY_SECONDS + 60) + now = time.time() + os.utime(path, (now, now)) + assert placeholder_age_seconds(path) > PLACEHOLDER_RETRY_SECONDS + + def test_fresh_placeholder_does_not_retry(self, tmp_path): + """Rate limiting: a placeholder written seconds ago must not re-download. + + Without this the fix would trade a permanent grey box for an ESPN + request on every frame. + """ + path = self._placeholder(tmp_path) + with patch.object(LogoDownloader, "download_logo") as download: + assert download_missing_logo( + "afl", "1", "COLL", path, + logo_url="http://example/coll.png") is True + download.assert_not_called() + + +class TestDownloadEligibility: + """One rule, shared by every download site. + + The two bulk loops and the single-logo path each had their own idea of what + counted as "already have it", which is how one of them ended up retrying + fresh placeholders and the other skipping stale ones forever. + """ + + def _placeholder(self, tmp_path, abbrev="COLL"): + assert LogoDownloader().create_placeholder_logo(abbrev, str(tmp_path)) + return tmp_path / f"{abbrev}.png" + + def _age(self, path, seconds): + metadata = PngInfo() + metadata.add_text(PLACEHOLDER_MARKER, str(time.time() - seconds)) + with Image.open(path) as img: + img.copy().save(path, "PNG", pnginfo=metadata) + + def test_missing_file_is_eligible(self, tmp_path): + assert should_attempt_download(tmp_path / "nope.png") is True + + def test_real_logo_is_not_eligible(self, tmp_path): + real = tmp_path / "REAL.png" + Image.new("RGBA", (500, 500), (1, 2, 3, 255)).save(real) + assert should_attempt_download(real) is False + + def test_force_download_beats_a_real_logo(self, tmp_path): + real = tmp_path / "REAL.png" + Image.new("RGBA", (500, 500), (1, 2, 3, 255)).save(real) + assert should_attempt_download(real, force_download=True) is True + + def test_fresh_placeholder_is_not_eligible(self, tmp_path): + assert should_attempt_download(self._placeholder(tmp_path)) is False + + def test_stale_placeholder_is_eligible(self, tmp_path): + path = self._placeholder(tmp_path) + self._age(path, PLACEHOLDER_RETRY_SECONDS + 60) + assert should_attempt_download(path) is True + + def test_league_bulk_loop_skips_a_fresh_placeholder(self, tmp_path): + """A bulk pass honours the same back-off as everything else.""" + self._placeholder(tmp_path, "AAA") + downloader = LogoDownloader() + teams = [{"abbreviation": "AAA", "display_name": "A", "logo_url": "http://x/a.png"}] + with patch.object(LogoDownloader, "get_logo_directory", return_value=str(tmp_path)): + with patch.object(LogoDownloader, "fetch_teams_data", return_value={"sports": [{}]}): + with patch.object(LogoDownloader, "extract_teams_from_data", return_value=teams): + with patch.object(LogoDownloader, "download_logo") as download: + downloader.download_missing_logos_for_league("nfl") + download.assert_not_called() + + def test_league_bulk_loop_retries_a_stale_placeholder(self, tmp_path): + path = self._placeholder(tmp_path, "AAA") + self._age(path, PLACEHOLDER_RETRY_SECONDS + 60) + downloader = LogoDownloader() + teams = [{"abbreviation": "AAA", "display_name": "A", "logo_url": "http://x/a.png"}] + with patch.object(LogoDownloader, "get_logo_directory", return_value=str(tmp_path)): + with patch.object(LogoDownloader, "fetch_teams_data", return_value={"sports": [{}]}): + with patch.object(LogoDownloader, "extract_teams_from_data", return_value=teams): + with patch.object(LogoDownloader, "download_logo", return_value=True) as download: + downloader.download_missing_logos_for_league("nfl") + download.assert_called_once() + + def test_ncaa_bulk_loop_retries_a_stale_placeholder(self, tmp_path): + """This loop skipped placeholders forever; it now shares the rule.""" + path = self._placeholder(tmp_path, "AAA") + self._age(path, PLACEHOLDER_RETRY_SECONDS + 60) + downloader = LogoDownloader() + teams = [{"abbreviation": "AAA", "display_name": "A", + "logo_url": "http://x/a.png", "category": "FBS", + "conference": "SEC"}] + with patch.object(LogoDownloader, "get_logo_directory", return_value=str(tmp_path)): + with patch.object(LogoDownloader, "fetch_teams_data", return_value={"sports": [{}]}): + with patch.object(LogoDownloader, "extract_teams_from_data", return_value=teams): + with patch.object(LogoDownloader, "download_logo", return_value=True) as download: + downloader.download_all_ncaa_football_logos() + download.assert_called_once() + + def test_ncaa_bulk_loop_skips_a_fresh_placeholder(self, tmp_path): + self._placeholder(tmp_path, "AAA") + downloader = LogoDownloader() + teams = [{"abbreviation": "AAA", "display_name": "A", + "logo_url": "http://x/a.png", "category": "FBS", + "conference": "SEC"}] + with patch.object(LogoDownloader, "get_logo_directory", return_value=str(tmp_path)): + with patch.object(LogoDownloader, "fetch_teams_data", return_value={"sports": [{}]}): + with patch.object(LogoDownloader, "extract_teams_from_data", return_value=teams): + with patch.object(LogoDownloader, "download_logo") as download: + downloader.download_all_ncaa_football_logos() + download.assert_not_called() + + +class TestRefreshPlaceholderTimestamp: + def test_restarts_the_back_off(self, tmp_path): + assert LogoDownloader().create_placeholder_logo("COLL", str(tmp_path)) + path = tmp_path / "COLL.png" + metadata = PngInfo() + metadata.add_text(PLACEHOLDER_MARKER, str(time.time() - (PLACEHOLDER_RETRY_SECONDS + 60))) + with Image.open(path) as img: + img.copy().save(path, "PNG", pnginfo=metadata) + assert should_attempt_download(path) is True + + assert refresh_placeholder_timestamp(path) is True + assert should_attempt_download(path) is False + + def test_refuses_to_touch_a_real_logo(self, tmp_path): + real = tmp_path / "REAL.png" + Image.new("RGBA", (500, 500), (1, 2, 3, 255)).save(real) + before = real.read_bytes() + assert refresh_placeholder_timestamp(real) is False + assert real.read_bytes() == before + + def test_missing_file_is_not_an_error(self, tmp_path): + assert refresh_placeholder_timestamp(tmp_path / "nope.png") is False diff --git a/test/test_logo_helper.py b/test/test_logo_helper.py index 0b02af7d..cb5824b2 100644 --- a/test/test_logo_helper.py +++ b/test/test_logo_helper.py @@ -421,3 +421,76 @@ class TestSessionConfiguration: def test_user_agent_and_accept_headers(self, helper): assert helper.session.headers["User-Agent"] == "LEDMatrix-Common/1.0" assert helper.session.headers["Accept"] == "image/*" + + +class TestStalePlaceholderHandling: + """load_logo_with_download must not be fooled by a cached placeholder. + + A placeholder wears the real logo's filename, so both the file cache and + the in-memory cache can hold one and look like a hit. + """ + + def _placeholder(self, tmp_path, abbrev="COLL"): + from src.logo_downloader import LogoDownloader + assert LogoDownloader().create_placeholder_logo(abbrev, str(tmp_path)) + return tmp_path / f"{abbrev}.png" + + def _make_stale(self, path): + import time + from PIL.PngImagePlugin import PngInfo + from src.logo_downloader import PLACEHOLDER_MARKER, PLACEHOLDER_RETRY_SECONDS + metadata = PngInfo() + metadata.add_text(PLACEHOLDER_MARKER, str(time.time() - (PLACEHOLDER_RETRY_SECONDS + 60))) + with Image.open(path) as img: + img.copy().save(path, "PNG", pnginfo=metadata) + + def test_fresh_placeholder_is_served_without_a_download(self, helper, tmp_path): + path = self._placeholder(tmp_path) + with patch.object(LogoHelper, "_download_logo") as download: + assert helper.load_logo_with_download("COLL", path, "http://x/c.png") is not None + download.assert_not_called() + + def test_stale_placeholder_triggers_a_download(self, helper, tmp_path): + path = self._placeholder(tmp_path) + self._make_stale(path) + with patch.object(LogoHelper, "_download_logo") as download: + helper.load_logo_with_download("COLL", path, "http://x/c.png") + download.assert_called_once() + + def test_replacement_logo_is_not_masked_by_the_cached_placeholder(self, helper, tmp_path): + """The bug this guards: load_logo answers from cache before the disk. + + Without invalidation the freshly downloaded logo would not appear until + the process restarted. + """ + path = self._placeholder(tmp_path) + first = helper.load_logo_with_download("COLL", path, "http://x/c.png") + assert first is not None + + self._make_stale(path) + + def fake_download(_self, _url, file_path): + Image.new("RGB", (500, 500), (7, 8, 9)).save(file_path, format="PNG") + + with patch.object(LogoHelper, "_download_logo", fake_download): + second = helper.load_logo_with_download("COLL", path, "http://x/c.png") + + assert second is not None + from src.logo_downloader import is_placeholder_logo + assert is_placeholder_logo(path) is False + assert second.getpixel((0, 0))[:3] == (7, 8, 9) + + def test_failed_retry_restarts_the_back_off(self, helper, tmp_path): + """Otherwise a stale placeholder means a download attempt per call.""" + from src.logo_downloader import should_attempt_download + path = self._placeholder(tmp_path) + self._make_stale(path) + assert should_attempt_download(path) is True + + def boom(_self, _url, _file_path): + raise OSError("network down") + + with patch.object(LogoHelper, "_download_logo", boom): + helper.load_logo_with_download("COLL", path, "http://x/c.png") + + assert should_attempt_download(path) is False diff --git a/test/test_plugin_compatibility_gate.py b/test/test_plugin_compatibility_gate.py index 26834ffa..9ab858a9 100644 --- a/test/test_plugin_compatibility_gate.py +++ b/test/test_plugin_compatibility_gate.py @@ -19,6 +19,7 @@ The rules being pinned here, in priority order: """ import json +import subprocess from pathlib import Path from unittest.mock import MagicMock @@ -146,9 +147,18 @@ def store(tmp_path, monkeypatch): class TestInstallGate: - """`install_plugin` is the chokepoint: `_reinstall_with_rollback` calls it, - so gating there covers updates too, and a refused update restores the - version the user already had.""" + """`install_plugin` is the chokepoint for every route that re-downloads: + `_reinstall_with_rollback` calls it, so a refused update restores the + version the user already had. + + It is not the *only* route in, and saying so here once is cheaper than + rediscovering it. `update_plugin` also has a git branch that pulls in + place and never re-downloads; that one is gated separately by + `_gate_pulled_commit` and pinned in `TestGitPullGate` below. A third + route, `install_from_url` (sideloading from a URL), is gated in that + function and pinned in `TestSideloadGate`. All three refuse on the same + rule. + """ def _install_with_manifest(self, store, manifest, core_version, monkeypatch): mgr, plugins_dir = store @@ -208,6 +218,291 @@ class TestInstallGate: assert (path / "manifest.json").exists() +class TestSideloadGate: + """`install_from_url` never looked at the core version. + + Sideloading is an explicit act rather than an automatic store update, so + the argument for gating it is different: not "the user did not choose + this", but that the floor states the plugin *cannot run here*. Letting it + through produces the same silent PluginState.ERROR at load that the store + gate exists to prevent, and the user who typed the URL is no better placed + to diagnose it than one who pressed Update. + """ + + def _sideload(self, store, manifest, core_version, monkeypatch): + mgr, plugins_dir = store + plugin_id = manifest["id"] + + def fake_clone(repo_url, target_path, branches=None): + target_path.mkdir(parents=True, exist_ok=True) + (target_path / "manifest.json").write_text( + json.dumps(manifest), encoding="utf-8") + (target_path / "manager.py").write_text( + "class P: pass\n", encoding="utf-8") + return "main" + + monkeypatch.setattr(mgr, "_install_via_git", fake_clone) + monkeypatch.setattr(mgr, "_install_dependencies", lambda *a, **k: True) + + import src + monkeypatch.setattr(src, "__version__", core_version) + return mgr.install_from_url("https://example.invalid/plugin"), \ + plugins_dir / plugin_id + + def test_refuses_a_plugin_that_needs_a_newer_core(self, store, monkeypatch): + manifest = { + "id": "sideload-newer", "name": "Sideload", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "9.9.9", + } + result, path = self._sideload(store, manifest, "3.2.0", monkeypatch) + + assert result["success"] is False + assert "9.9.9" in result["error"], result["error"] + assert not path.exists(), ( + "a refused sideload must not leave the plugin installed") + + def test_allows_a_compatible_plugin(self, store, monkeypatch): + """The guard against over-refusing: a gate that blocks everything + passes the test above and breaks sideloading entirely.""" + manifest = { + "id": "sideload-fine", "name": "Sideload", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "3.0.0", + } + result, path = self._sideload(store, manifest, "3.2.0", monkeypatch) + + assert result["success"] is True, result.get("error") + assert (path / "manifest.json").exists() + + def test_untrustworthy_core_does_not_block_a_2_0_0_floor( + self, store, monkeypatch): + """Same rule as the other two routes: a v3.1.0 release reports 1.0.0, + and nearly every published manifest floors at 2.0.0.""" + manifest = { + "id": "sideload-floored", "name": "Sideload", "class_name": "P", + "display_modes": ["a"], "versions": [{"ledmatrix_min": "2.0.0"}], + } + result, _ = self._sideload(store, manifest, "1.0.0", monkeypatch) + + assert result["success"] is True, result.get("error") + +# -------------------------------------------------------------------------- +# The git-pull update path — the one route that does not re-download +# -------------------------------------------------------------------------- + +def _git_available() -> bool: + # OSError, not just a non-zero exit: with no git on PATH subprocess raises + # FileNotFoundError, and this runs at import time -- before skipif can act, + # so the whole module would error out instead of skipping. + try: + return subprocess.run( + ['git', '--version'], capture_output=True).returncode == 0 + except OSError: + return False + + +_HAS_GIT = _git_available() + + +def _git(*args, cwd): + return subprocess.run(['git', *args], cwd=str(cwd), + capture_output=True, text=True, check=True) + + +@pytest.mark.skipif(not _HAS_GIT, reason='git not available') +class TestGitPullGate: + """A plugin installed as a git checkout updates by pulling in place, so it + never passes through `install_plugin` and was never gated. + + Real git repositories rather than mocks, because the claim under test is + that `git reset --hard` puts the checkout back — a mock of git would only + prove the call was made, which is the easy half. + + Only the handful of registry entries with no `plugin_path` reach this path + in practice; monorepo plugins install as archives and update through + `_reinstall_with_rollback`. It is gated anyway because the sunset rule in + `docs/plugin-development/08-shared-sports-code.md` states the core enforces + the floor "at install/update time", and a precondition that is documented + but not true is worse than one that is merely missing. + """ + + @pytest.fixture + def checkout(self, tmp_path, monkeypatch): + """An origin repo holding a plugin, and a clone of it installed as + `plugin-repos/gitplug`, with the store pointed at it.""" + from src.plugin_system.store_manager import PluginStoreManager + + origin = tmp_path / 'origin' + origin.mkdir() + _git('init', '--initial-branch=main', '--bare', cwd=origin) + + seed = tmp_path / 'seed' + _git('clone', str(origin), str(seed), cwd=tmp_path) + _git('config', 'user.email', 'test@example.com', cwd=seed) + _git('config', 'user.name', 'Test', cwd=seed) + (seed / 'manifest.json').write_text(json.dumps({ + "id": "gitplug", "name": "Git Plug", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "2.0.0", + }), encoding='utf-8') + (seed / 'manager.py').write_text("class P: pass\n", encoding='utf-8') + _git('add', '-A', cwd=seed) + _git('commit', '-m', 'initial', cwd=seed) + _git('push', '-u', 'origin', 'main', cwd=seed) + + plugins_dir = tmp_path / 'plugin-repos' + plugins_dir.mkdir() + work = plugins_dir / 'gitplug' + _git('clone', str(origin), str(work), cwd=tmp_path) + _git('config', 'user.email', 'test@example.com', cwd=work) + _git('config', 'user.name', 'Test', cwd=work) + + mgr = PluginStoreManager(plugins_dir=str(plugins_dir)) + mgr.logger = MagicMock() + # No registry entry, so update_plugin takes the plain-pull branch + # rather than the remote-mismatch or already-current shortcuts. + monkeypatch.setattr(mgr, 'fetch_registry', lambda *a, **k: None) + monkeypatch.setattr(mgr, 'get_plugin_info', lambda *a, **k: None) + monkeypatch.setattr(mgr, '_install_dependencies', lambda *a, **k: True) + return mgr, seed, work + + def _push(self, seed, manifest_text): + (seed / 'manifest.json').write_text(manifest_text, encoding='utf-8') + _git('add', '-A', cwd=seed) + _git('commit', '-m', 'update', cwd=seed) + _git('push', 'origin', 'main', cwd=seed) + + def _head(self, work): + return _git('rev-parse', 'HEAD', cwd=work).stdout.strip() + + def test_refuses_a_pulled_commit_that_needs_a_newer_core( + self, checkout, monkeypatch): + mgr, seed, work = checkout + before = self._head(work) + self._push(seed, json.dumps({ + "id": "gitplug", "name": "Git Plug", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "9.9.9", + })) + + import src + monkeypatch.setattr(src, '__version__', '3.2.0') + assert mgr.update_plugin('gitplug') is False + + assert self._head(work) == before, ( + "a refused update must leave the checkout on the commit it was " + "already running, not on the one it cannot load") + floor = json.loads((work / 'manifest.json').read_text()) + assert floor['min_ledmatrix_version'] == '2.0.0', ( + "the reset must restore the working tree, not just the ref") + + def test_allows_a_pulled_commit_the_core_can_run( + self, checkout, monkeypatch): + """The guard against over-refusing. A gate that refuses everything + passes the test above and breaks every update.""" + mgr, seed, work = checkout + before = self._head(work) + self._push(seed, json.dumps({ + "id": "gitplug", "name": "Git Plug", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "3.0.0", + })) + + import src + monkeypatch.setattr(src, '__version__', '3.2.0') + assert mgr.update_plugin('gitplug') is True + assert self._head(work) != before + + def test_an_unreadable_manifest_after_pull_is_not_a_refusal( + self, checkout, monkeypatch): + """Rule 1 of this file — refuse only on evidence. A manifest that + will not parse declares no floor, so it is not evidence of anything.""" + mgr, seed, work = checkout + self._push(seed, '{ this is not json') + + import src + monkeypatch.setattr(src, '__version__', '3.2.0') + assert mgr.update_plugin('gitplug') is True + + def test_an_untrustworthy_core_does_not_block_a_2_0_0_floor_on_pull( + self, checkout, monkeypatch): + """The same regression guard as + `test_untrustworthy_core_does_not_block_installs`, because this path + now shares that rule: a v3.1.0 release reports 1.0.0, and nearly every + published manifest floors at 2.0.0.""" + mgr, seed, work = checkout + before = self._head(work) + self._push(seed, json.dumps({ + "id": "gitplug", "name": "Git Plug", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "2.0.0", + "description": "changed", + })) + + import src + monkeypatch.setattr(src, '__version__', '1.0.0') + assert mgr.update_plugin('gitplug') is True + assert self._head(work) != before + + def test_a_failed_stash_stops_the_update_before_pulling( + self, checkout, monkeypatch): + """Uncommitted work is not collateral for the gate. + + The gate's only rollback is `git reset --hard`, which discards + uncommitted tracked edits. update_plugin stashes them first -- but when + that stash fails it used to pull anyway, and a pull touching different + files succeeds on a dirty tree. Refuse, refuse the rollback's rollback, + and the user's edits go with it. + """ + mgr, seed, work = checkout + before = self._head(work) + (work / 'manager.py').write_text( + "class P:\n MINE = 'do not lose this'\n", encoding='utf-8') + self._push(seed, json.dumps({ + "id": "gitplug", "name": "Git Plug", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "9.9.9", + })) + + real_run = subprocess.run + + def fail_stash(cmd, *a, **k): + if isinstance(cmd, list) and 'stash' in cmd: + return subprocess.CompletedProcess(cmd, 1, '', 'stash boom') + return real_run(cmd, *a, **k) + + import src + monkeypatch.setattr(src, '__version__', '3.2.0') + monkeypatch.setattr( + 'src.plugin_system.store_manager.subprocess.run', fail_stash) + assert mgr.update_plugin('gitplug') is False + + assert self._head(work) == before, "must not pull what it cannot undo" + assert 'do not lose this' in (work / 'manager.py').read_text(), ( + "the local edit the stash failed to save must still be there") + + def test_rollback_reports_the_recovery_command_when_git_fails( + self, checkout, monkeypatch): + """If the reset itself fails the user is left on a version that cannot + load, so the log line has to carry the command that fixes it — + it is the only thing standing between them and a manual reinstall.""" + mgr, seed, work = checkout + self._push(seed, json.dumps({ + "id": "gitplug", "name": "Git Plug", "class_name": "P", + "display_modes": ["a"], "min_ledmatrix_version": "9.9.9", + })) + + real_run = subprocess.run + + def fail_reset(cmd, *a, **k): + if isinstance(cmd, list) and 'reset' in cmd: + return subprocess.CompletedProcess(cmd, 1, '', 'reset boom') + return real_run(cmd, *a, **k) + + import src + monkeypatch.setattr(src, '__version__', '3.2.0') + monkeypatch.setattr( + 'src.plugin_system.store_manager.subprocess.run', fail_reset) + assert mgr.update_plugin('gitplug') is False + + logged = ' '.join(str(c) for c in mgr.logger.error.call_args_list) + assert 'reset --hard' in logged, logged + + class TestLoaderAndStoreAgree: """Both read the same manifests; a disagreement means one of them is lying to the user.""" diff --git a/test/test_plugin_manager_discovered_ids.py b/test/test_plugin_manager_discovered_ids.py new file mode 100644 index 00000000..5bedba8d --- /dev/null +++ b/test/test_plugin_manager_discovered_ids.py @@ -0,0 +1,63 @@ +"""Tests for PluginManager.discovered_plugin_ids(). + +The config-watcher thread needs the set of discovered plugin ids while the +render thread may be rebuilding plugin_manifests. Iterating that dict directly +can observe a half-populated mapping or raise "dictionary changed size during +iteration", so the accessor snapshots it under the discovery lock. +""" + +import tempfile +import threading +from pathlib import Path + +import pytest + +from src.plugin_system.plugin_manager import PluginManager + + +@pytest.fixture +def pm(): + with tempfile.TemporaryDirectory() as tmp: + yield PluginManager(plugins_dir=str(Path(tmp) / "plugins")) + + +def test_returns_the_discovered_ids(pm): + pm.plugin_manifests = {"clock-simple": {}, "hockey-scoreboard": {}} + assert pm.discovered_plugin_ids() == {"clock-simple", "hockey-scoreboard"} + + +def test_empty_when_nothing_discovered(pm): + pm.plugin_manifests = {} + assert pm.discovered_plugin_ids() == set() + + +def test_is_a_snapshot_not_a_live_view(pm): + """The caller iterates the result on another thread; it must not alias + the mapping discovery is still writing to.""" + pm.plugin_manifests = {"clock-simple": {}} + snapshot = pm.discovered_plugin_ids() + pm.plugin_manifests["hockey-scoreboard"] = {} + assert snapshot == {"clock-simple"} + + +def test_takes_the_discovery_lock(pm): + """Guards against the lock being dropped in a later refactor: with the + lock held by another thread the call must block rather than read.""" + pm.plugin_manifests = {"clock-simple": {}} + finished = threading.Event() + + def call(): + pm.discovered_plugin_ids() + finished.set() + + pm._discovery_lock.acquire() + try: + # RLock is reentrant per-thread, so use a *different* thread to prove + # the accessor actually waits on it. + t = threading.Thread(target=call, daemon=True) + t.start() + assert not finished.wait(timeout=0.3), "accessor did not take the discovery lock" + finally: + pm._discovery_lock.release() + t.join(timeout=2) + assert finished.is_set() diff --git a/test/test_plugin_state_history_cap.py b/test/test_plugin_state_history_cap.py new file mode 100644 index 00000000..1d5e20d5 --- /dev/null +++ b/test/test_plugin_state_history_cap.py @@ -0,0 +1,166 @@ +"""Plugin state history must not grow without bound. + +`PluginStateManager` recorded every state transition in a per-plugin list and +never trimmed it. The only code that removed entries was `clear_state()`, called +solely from `PluginManager.unload_plugin()`, so a plugin that stays loaded -- +i.e. normal operation -- never released a single entry. + +The list is written on the hot scheduling path. Every update cycle appends +twice: `_reserve_for_update()` sets RUNNING and `_finish()` sets ENABLED back +again. At the default 60-second update interval that is 2,880 entries per +plugin per day, and nothing ever reads the entries -- `get_state_info()` only +takes their `len()`. It is pure dead weight. + +Measured against the unpatched class, ten plugins on a 60s interval retain +864,010 transitions after thirty simulated days, for 231 MB of heap. On a 1 GB +Pi that is fatal on its own, and the failure is not a clean OOM: once +MemAvailable falls far enough, fork() starts returning ENOMEM, so sshd accepts +connections and closes them before its banner while the kernel still answers +pings. The board looks like a hardware fault and needs a power cycle. + +These tests pin the cap, the retention order, and the one piece of behaviour the +cap must not change: `state_history_count` is surfaced through the web API, so +it has to keep reporting the lifetime total rather than plateauing at the cap. +""" + +import os +import sys + +import pytest + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) + +from src.plugin_system.plugin_state import ( # noqa: E402 + MAX_STATE_HISTORY_PER_PLUGIN, + PluginState, + PluginStateManager, +) + + +def _cycle_updates(manager, plugin_id, cycles): + """Drive the real scheduling path: RUNNING on reserve, ENABLED on finish.""" + for _ in range(cycles): + manager.set_state(plugin_id, PluginState.RUNNING) + manager.set_state(plugin_id, PluginState.ENABLED) + + +def test_state_history_is_capped(): + """A day of updates must not retain a day of transitions.""" + manager = PluginStateManager() + manager.set_state("clock", PluginState.ENABLED) + + # One simulated day at the default 60s update interval. + _cycle_updates(manager, "clock", 1440) + + history = manager.get_state_history("clock") + assert len(history) <= MAX_STATE_HISTORY_PER_PLUGIN, ( + f"history grew to {len(history)} entries; it is never trimmed" + ) + + +def test_state_history_keeps_the_most_recent_transitions(): + """Trimming drops the oldest entries, not the newest.""" + manager = PluginStateManager() + manager.set_state("clock", PluginState.ENABLED) + _cycle_updates(manager, "clock", MAX_STATE_HISTORY_PER_PLUGIN) + + history = manager.get_state_history("clock") + + # The scheduling cycle ends on ENABLED, so the newest entry is the + # RUNNING -> ENABLED half of the last cycle. + assert history[-1]["from"] == PluginState.RUNNING.value + assert history[-1]["to"] == PluginState.ENABLED.value + + # And the very first ENABLED transition has aged out. + assert history[0]["from"] != PluginState.UNLOADED.value + + +def test_state_history_count_reports_lifetime_total(): + """The count exposed through the API must not plateau at the cap. + + `get_state_info()['state_history_count']` is surfaced by the web UI. Capping + the retained list must not turn it into "entries we happen to still hold". + """ + manager = PluginStateManager() + manager.set_state("clock", PluginState.ENABLED) + total = 1 + + cycles = MAX_STATE_HISTORY_PER_PLUGIN * 2 + _cycle_updates(manager, "clock", cycles) + total += cycles * 2 + + info = manager.get_state_info("clock") + assert info["state_history_count"] == total + assert len(manager.get_state_history("clock")) <= MAX_STATE_HISTORY_PER_PLUGIN + + +def test_error_transitions_are_capped_too(): + """set_state_with_error() appends to the same list and needs the same cap.""" + manager = PluginStateManager() + manager.set_state("clock", PluginState.ENABLED) + + for _ in range(MAX_STATE_HISTORY_PER_PLUGIN * 2): + manager.set_state_with_error( + "clock", + PluginState.ENABLED, + {"reason": "update timeout"}, + error=RuntimeError("boom"), + ) + + assert len(manager.get_state_history("clock")) <= MAX_STATE_HISTORY_PER_PLUGIN + + +def test_history_is_isolated_per_plugin(): + """The cap is per plugin, not shared across the manager.""" + manager = PluginStateManager() + for plugin_id in ("clock", "weather"): + manager.set_state(plugin_id, PluginState.ENABLED) + _cycle_updates(manager, plugin_id, 50) + + assert len(manager.get_state_history("clock")) == 101 + assert len(manager.get_state_history("weather")) == 101 + + +def test_get_state_history_returns_a_copy(): + """Callers must not be able to mutate the manager's internal history.""" + manager = PluginStateManager() + manager.set_state("clock", PluginState.ENABLED) + + history = manager.get_state_history("clock") + history.clear() + + assert len(manager.get_state_history("clock")) == 1 + + +def test_get_state_history_entries_are_copies(): + """Copying the outer list is not enough -- the entries are handed out too. + + A caller holding a returned transition must not be able to rewrite the + manager's record of what happened. + """ + manager = PluginStateManager() + manager.set_state("clock", PluginState.ENABLED) + + entry = manager.get_state_history("clock")[0] + entry["to"] = "tampered" + entry["error"] = "injected" + + stored = manager.get_state_history("clock")[0] + assert stored["to"] == PluginState.ENABLED.value + assert stored["error"] is None + + +def test_clear_state_drops_history(): + """Unloading a plugin still releases everything it accumulated.""" + manager = PluginStateManager() + manager.set_state("clock", PluginState.ENABLED) + _cycle_updates(manager, "clock", 10) + + manager.clear_state("clock") + + assert manager.get_state_history("clock") == [] + assert manager.get_state_info("clock")["state_history_count"] == 0 + + +if __name__ == "__main__": + sys.exit(pytest.main([__file__, "-v"])) diff --git a/test/test_plugin_state_history_retention.py b/test/test_plugin_state_history_retention.py new file mode 100644 index 00000000..a7cf1edf --- /dev/null +++ b/test/test_plugin_state_history_retention.py @@ -0,0 +1,209 @@ +"""Retention is bounded by age first and by count second. + +The cap added in the parent change is a flat entry count, and an entry count +answers the wrong question. What a reader wants from this history is "the last +couple of hours"; how many transitions that is depends entirely on the +plugin's update interval, which on a real board spans 2s to 3600s. A flat 200 +entries is 4.2 days of history for the slowest plugin and 3.3 minutes for the +fastest -- so the plugin churning hardest, the one actually worth looking at, +keeps the least. + +Trimming by age makes the retained window comparable whatever the cadence, and +the count then serves only as a memory ceiling for pollers fast enough to +produce thousands of transitions inside that window. +""" + +import time +import pytest + +from src.plugin_system.plugin_state import ( + PluginState, + PluginStateManager, + MAX_STATE_HISTORY_PER_PLUGIN, + STATE_HISTORY_MAX_AGE_SECONDS, +) + + +class FakeClock: + """A monotonic clock the test drives, so no test has to sleep.""" + + def __init__(self): + self.t = 1000.0 + + def __call__(self): + return self.t + + def advance(self, seconds): + self.t += seconds + + +@pytest.fixture +def clock(monkeypatch): + c = FakeClock() + monkeypatch.setattr("src.plugin_system.plugin_state.time.monotonic", c) + return c + + +def _cycle(manager, plugin_id, clock, interval, cycles): + """One update cycle: RUNNING on reserve, ENABLED on finish.""" + for _ in range(cycles): + manager.set_state(plugin_id, PluginState.RUNNING) + manager.set_state(plugin_id, PluginState.ENABLED) + clock.advance(interval) + + +def test_transitions_older_than_the_window_are_dropped(clock): + m = PluginStateManager() + _cycle(m, "clock", clock, interval=60, cycles=10) + assert len(m.get_state_history("clock")) == 20 + + # Nothing happens for longer than the window, then one more cycle. + clock.advance(STATE_HISTORY_MAX_AGE_SECONDS + 1) + _cycle(m, "clock", clock, interval=60, cycles=1) + + assert len(m.get_state_history("clock")) == 2, ( + "only the transitions inside the window should survive") + + +def test_every_plugin_keeps_the_same_WINDOW_not_the_same_COUNT(clock): + """The point of the age policy, stated as the property that distinguishes it. + + Run both plugins for three times the retention window. Under a flat count + cap the slow one would still be holding transitions from hours before the + window, because it never produces enough entries to evict them. Under the + age policy each plugin retains its own last two hours and no more -- + different entry counts, same span of time. + """ + window = STATE_HISTORY_MAX_AGE_SECONDS + m = PluginStateManager() + + _cycle(m, "slow", clock, interval=60, cycles=(3 * window) // 60) + slow = len(m.get_state_history("slow")) + + # Assert the property directly rather than a derived count. The guarantee + # is about the SPAN of retained history, not its age against the current + # clock: trimming happens on append, so a plugin that has gone quiet keeps + # its last window until it writes again. That is intentional -- it is + # bounded either way, and a lazy trim costs nothing on the hot path. + stamps = [stamp for stamp, _ in m._state_history["slow"]] + assert stamps[-1] - stamps[0] <= window, ( + f"retained history spans {stamps[-1] - stamps[0]:.0f}s, " + f"window is {window}s") + assert slow < 2 * ((3 * window) // 60), ( + f"slow plugin kept {slow} entries -- three windows' worth was retained") + + clock.t = 1000.0 + _cycle(m, "fast", clock, interval=2, cycles=(3 * window) // 2) + fast = len(m.get_state_history("fast")) + + # Different counts, and the fast poller keeps more of them -- under a flat + # count cap these would be equal and the fast one would cover minutes. + assert fast > slow, f"fast={fast} slow={slow}" + + +def test_the_count_ceiling_still_bounds_a_fast_poller(clock): + """Age alone would let a 2s plugin hold 7,200 entries.""" + m = PluginStateManager() + _cycle(m, "flights", clock, interval=2, cycles=STATE_HISTORY_MAX_AGE_SECONDS) + assert len(m.get_state_history("flights")) <= MAX_STATE_HISTORY_PER_PLUGIN + + +def test_a_burst_inside_the_window_is_capped_not_kept(clock): + """Transitions with no time between them still cannot grow without bound.""" + m = PluginStateManager() + for _ in range(MAX_STATE_HISTORY_PER_PLUGIN * 3): + m.set_state("flapping", PluginState.RUNNING) # clock never advances + assert len(m.get_state_history("flapping")) <= MAX_STATE_HISTORY_PER_PLUGIN + + +def test_ageing_out_does_not_disturb_the_lifetime_count(clock): + m = PluginStateManager() + _cycle(m, "clock", clock, interval=60, cycles=10) + clock.advance(STATE_HISTORY_MAX_AGE_SECONDS + 1) + _cycle(m, "clock", clock, interval=60, cycles=1) + + assert len(m.get_state_history("clock")) == 2 + assert m.get_state_info("clock")["state_history_count"] == 22, ( + "the lifetime total must survive trimming, it is the flap signal") + + +def test_the_surviving_entries_are_the_recent_ones(clock): + m = PluginStateManager() + _cycle(m, "clock", clock, interval=60, cycles=5) + clock.advance(STATE_HISTORY_MAX_AGE_SECONDS + 1) + m.set_state("clock", PluginState.ERROR) + + history = m.get_state_history("clock") + assert [h["to"] for h in history] == ["error"] + + +def test_a_monotonic_clock_is_used_not_the_wall_clock(clock): + """A DST shift or NTP step must not flush the history. + + The trim reads time.monotonic(); the human-readable datetime inside each + transition is for display only. + """ + m = PluginStateManager() + _cycle(m, "clock", clock, interval=60, cycles=3) + before = len(m.get_state_history("clock")) + + import datetime as real_datetime + + class ShiftedDatetime(real_datetime.datetime): + @classmethod + def now(cls, tz=None): + return real_datetime.datetime(1999, 1, 1) # clock jumps backwards + + import src.plugin_system.plugin_state as ps + original = ps.datetime + ps.datetime = ShiftedDatetime + try: + m.set_state("clock", PluginState.ENABLED) + finally: + ps.datetime = original + + assert len(m.get_state_history("clock")) == before + 1, ( + "a wall-clock jump must not trim anything") + + +def test_get_state_info_is_a_consistent_snapshot(): + """An unload running concurrently must not be observed half-done. + + Each field used to be read under its own lock, so clear_state() could + interleave: 'state' read before the removal, 'state_history_count' after, + handing a caller a plugin that is ENABLED with zero transitions. The whole + payload is now built in one critical section. + """ + import threading + + m = PluginStateManager() + for _ in range(50): + m.set_state("clock", PluginState.RUNNING) + m.set_state("clock", PluginState.ENABLED) + + inconsistent = [] + stop = threading.Event() + + def reader(): + while not stop.is_set(): + info = m.get_state_info("clock") + # Either fully present or fully cleared -- never a live state with + # a wiped count. + if info["state"] != PluginState.UNLOADED.value and \ + info["state_history_count"] == 0: + inconsistent.append(info) + return + + def clearer(): + for _ in range(200): + for _ in range(20): + m.set_state("clock", PluginState.ENABLED) + m.clear_state("clock") + + t = threading.Thread(target=reader, daemon=True) + t.start() + clearer() + stop.set() + t.join(timeout=5) + + assert not inconsistent, f"observed a torn snapshot: {inconsistent[:1]}" diff --git a/test/test_sports_card.py b/test/test_sports_card.py new file mode 100644 index 00000000..8de04e47 --- /dev/null +++ b/test/test_sports_card.py @@ -0,0 +1,195 @@ +"""The card helpers the eight scoreboards now share. + +These bodies lived in eight byte-identical copies. Moving them here means one +fix reaches every scoreboard — and that a mistake does too, which is what this +file guards. Each case below is one the plugins' own code already handled; the +point is that it keeps handling it. + +The functions take ``config``/``logger``/``fonts`` as arguments rather than +reading them off an instance, so a plugin keeps its method and delegates the +body. That is what let all eight adopt this with byte-identical renders. +""" + +import logging +import json +import os + +import pytest + +from src.common import sports_card as C + + +@pytest.fixture +def log(): + return logging.getLogger("test_sports_card") + + +class TestSettingsLookup: + def test_reads_the_scroll_card_block(self): + cfg = {"scroll_card": {"vs_text": "@"}} + assert C.scroll_card_option(cfg, "vs_text", "VS") == "@" + + @pytest.mark.parametrize("cfg", [None, {}, {"scroll_card": None}, + {"scroll_card": {"vs_text": None}}]) + def test_missing_or_null_falls_back(self, cfg): + """A null in config means "unset", not "empty string".""" + assert C.scroll_card_option(cfg, "vs_text", "VS") == "VS" + + def test_upcoming_center_rejects_unknown_modes(self): + for bad in ("sideways", "", None, 7): + assert C.upcoming_center_mode({"scroll_card": {"upcoming_center": bad}}) == "vs" + assert C.upcoming_center_mode({"scroll_card": {"upcoming_center": "DATE_TIME"}}) == "date_time" + + +class TestColour: + def test_rgb_list_and_hex_both_work(self): + assert C.element_color({"customization": {"score_text": {"text_color": [1, 2, 3]}}}, + "score_text") == (1, 2, 3) + assert C.element_color({"customization": {"score_text": {"text_color": "#ff8000"}}}, + "score_text") == (255, 128, 0) + + @pytest.mark.parametrize("value", ["nope", "#fff", [1, 2], None, ["a", "b", "c"]]) + def test_unusable_colour_falls_back(self, value): + cfg = {"customization": {"score_text": {"text_color": value}}} + assert C.element_color(cfg, "score_text", (9, 9, 9)) == (9, 9, 9) + + def test_coerce_rgb_clamps_rather_than_rejecting(self): + assert C.coerce_rgb([300, -5, 20], (0, 0, 0)) == (255, 0, 20) + + def test_coerce_rgb_refuses_a_three_character_string(self): + """"123" would otherwise iterate into three digits and yield a colour.""" + assert C.coerce_rgb("123", (7, 7, 7)) == (7, 7, 7) + + def test_font_colour_is_resolved_by_identity(self): + a, b = object(), object() + cfg = {"customization": {"score_text": {"text_color": [4, 5, 6]}}} + assert C.font_color(cfg, {"score": a, "team": b}, a) == (4, 5, 6) + + def test_a_shared_face_gives_up_rather_than_guessing(self): + """One object used for two elements has no single right colour.""" + shared = object() + cfg = {"customization": {"score_text": {"text_color": [4, 5, 6]}}} + assert C.font_color(cfg, {"score": shared, "team": shared}, shared) == (255, 255, 255) + + +class TestFavourites: + GAME = {"home_abbr": "TB", "away_abbr": "NO", "home_score": "21", "away_score": "17"} + + def test_win_loss_and_tie(self): + cfg = {"favorite_teams": ["TB"]} + assert C.favorite_result(cfg, self.GAME) == "win" + assert C.favorite_result({"favorite_teams": ["NO"]}, self.GAME) == "loss" + tied = dict(self.GAME, home_score="3", away_score="3") + assert C.favorite_result(cfg, tied) == "tie" + + def test_no_verdict_without_exactly_one_favourite_side(self): + assert C.favorite_result({}, self.GAME) is None + assert C.favorite_result({"favorite_teams": ["TB", "NO"]}, self.GAME) is None + assert C.favorite_result({"favorite_teams": ["SEA"]}, self.GAME) is None + + def test_unusable_scores_give_no_verdict(self): + bad = dict(self.GAME, home_score="x") + assert C.favorite_result({"favorite_teams": ["TB"]}, bad) is None + + def test_nested_payload_shape_is_read_too(self): + game = {"home_team": {"abbrev": "TB", "score": 9}, + "away_team": {"abbrev": "NO", "score": 2}} + assert C.side_score(game, "home") == 9 + assert C.side_is_favorite(game, "home", {"TB"}) is True + + def test_matches_on_id_where_abbreviations_collide(self): + """NRL keys favourites by ESPN id; abbreviations are not unique there.""" + game = {"home_abbr": "SYD", "home_id": "4321"} + assert C.side_is_favorite(game, "home", {"4321"}) is True + + def test_game_and_config_favourites_are_both_used(self): + """Games carry resolved dynamic groups; config catches later edits.""" + game = dict(self.GAME, favorite_teams=["NO"], league="nfl") + assert set(C.favorite_teams_for({"nfl": {"favorite_teams": ["TB"]}}, game)) == {"NO", "TB"} + + +class TestDateAndTime: + def test_date_formats(self, log): + for fmt, want in [("abbrev", "Sep 5"), ("numeric", "9/5"), + ("day_first", "5 Sep"), ("numeric_day_first", "5/9")]: + cfg = {"scroll_card": {"date_format": fmt}} + assert C.format_game_date(cfg, log, "9/5") == want + + @pytest.mark.parametrize("raw", ["", "garbage", "13/40", "no/slash/here"]) + def test_unparseable_dates_pass_through(self, log, raw): + assert C.format_game_date({}, log, raw) == raw.strip() + + def test_24h_conversion(self): + cfg = {"scroll_card": {"time_format": "24h"}} + assert C.format_game_time(cfg, "7:30 PM") == "19:30" + assert C.format_game_time(cfg, "12:00 AM") == "00:00" + assert C.format_game_time(cfg, "12:15 PM") == "12:15" + + def test_12h_is_left_alone_and_junk_survives(self): + assert C.format_game_time({}, "7:30 PM") == "7:30 PM" + assert C.format_game_time({"scroll_card": {"time_format": "24h"}}, "soon") == "soon" + + def test_a_bad_timezone_falls_back_to_utc(self, log): + """A typo in config should not blank the card.""" + from datetime import timezone + assert C.card_tzinfo({"timezone": "Not/AZone"}, log) is timezone.utc + + +class TestFontSizing: + def test_snaps_to_the_faces_pixel_grid(self): + assert C.crisp_size("4x6-font.ttf", 6) == 7 # 7px grid + assert C.crisp_size("PressStart2P-Regular.ttf", 10) == 8 + assert C.crisp_size("PressStart2P-Regular.ttf", 13) == 16 + + def test_an_unknown_face_is_never_second_guessed(self): + assert C.crisp_size("SomeUserFont.ttf", 11) == 11 + + def test_aliases_resolve_before_the_grid_lookup(self): + assert C.crisp_size("four_by_six", 6) == C.crisp_size("4x6-font.ttf", 6) + + @pytest.mark.parametrize("desired", [0, -3, None]) + def test_unusable_sizes_pass_through_without_raising(self, desired): + """None reached this in the field; football's variant raised TypeError.""" + assert C.crisp_size("4x6-font.ttf", desired) == desired + + def test_schema_cache_is_keyed_per_schema_not_globally(self, tmp_path): + """Two plugins declaring different defaults must not share an answer.""" + a, b = tmp_path / "a.json", tmp_path / "b.json" + for path, size in ((a, 11), (b, 22)): + path.write_text(json.dumps({"properties": {"customization": {"properties": { + "score_text": {"properties": {"font_size": {"default": size}}}}}}})) + assert C.schema_font_size(str(a), "score_text") == 11 + assert C.schema_font_size(str(b), "score_text") == 22 + + def test_a_missing_schema_is_not_an_error(self, tmp_path): + assert C.schema_font_size(str(tmp_path / "nope.json"), "score_text") is None + + def test_a_configured_size_matching_the_schema_default_is_not_a_choice(self, tmp_path): + """The web UI writes the whole default block on every save, so + font_size == schema default carries no intent and must not pin the + install to an off-grid size forever.""" + schema = tmp_path / "s.json" + schema.write_text(json.dumps({"properties": {"customization": {"properties": { + "score_text": {"properties": {"font_size": {"default": 10}}}}}}})) + got = C.resolve_font_size(str(schema), {"font_size": 10}, "score_text", 10, + "PressStart2P-Regular.ttf") + assert got == 8, "a default-valued size should snap to the grid" + + def test_a_real_choice_wins(self, tmp_path): + schema = tmp_path / "s.json" + schema.write_text(json.dumps({"properties": {"customization": {"properties": { + "score_text": {"properties": {"font_size": {"default": 10}}}}}}})) + got = C.resolve_font_size(str(schema), {"font_size": 13}, "score_text", 10, + "PressStart2P-Regular.ttf") + assert got == 13, "an explicit size the user chose is not second-guessed" + + +class TestTables: + def test_every_font_key_maps_to_an_element(self): + assert set(C.ELEMENT_FOR_FONT) == {"score", "time", "team", "status", "detail", "rank"} + + def test_result_colours_cover_every_verdict(self): + assert set(C.FAVORITE_RESULT_COLOR_DEFAULTS) == {"win", "loss", "tie"} + + def test_month_and_weekday_tables_are_complete(self): + assert len(C.MONTH_ABBR) == 12 and len(C.WEEKDAY_ABBR) == 7 diff --git a/test/test_sports_game_renderer.py b/test/test_sports_game_renderer.py new file mode 100644 index 00000000..50590005 --- /dev/null +++ b/test/test_sports_game_renderer.py @@ -0,0 +1,258 @@ +"""The shared card geometry, exercised against a host that supplies only what +the mixin's contract names. + +The point of these is the contract, not the arithmetic. The mixin reaches for +``display_width``, ``fonts``, ``config`` and six ``sports_card`` delegations +through ``self``, and the eight plugins are what actually provide them. A +stub host that provides exactly the documented surface and nothing else is +what catches the mixin quietly growing a dependency the plugins do not have. +""" + +import pytest +from PIL import Image, ImageDraw, ImageFont + +from src.common.sports_game_renderer import SportsGameRendererMixin + + +class Host(SportsGameRendererMixin): + """The documented contract, and not one attribute more.""" + + def __init__(self, width=128, height=32, config=None, scroll=None): + self.display_width = width + self.display_height = height + self.config = config or {} + self.logger = _Logger() + self._team_rankings_cache = {} + self._scroll = scroll or {} + font = ImageFont.load_default() + self.fonts = {'score': font, 'time': font, 'detail': font} + self.drawn = [] + + # -- the six sports_card delegations the mixin calls -- + def _scroll_card_option(self, key, default=None): + return self._scroll.get(key, default) + + def _upcoming_center_mode(self): + return self._scroll.get('upcoming_center', 'vs') + + def _vs_text(self): + return self._scroll.get('vs_text', 'VS') + + def _element_color(self, element): + return (255, 255, 255) + + def _format_game_date(self, raw, game): + return raw + + def _format_game_time(self, raw): + return raw + + # -- the one hook whose body genuinely varies per plugin -- + def _draw_text_with_outline(self, draw, text, position, font, + fill=None, outline_color=(0, 0, 0)): + self.drawn.append((text, position)) + + +class _Logger: + def debug(self, *a, **k): + pass + + +def _draw(): + return ImageDraw.Draw(Image.new("RGB", (256, 64))) + + +class TestCenterGap: + def test_explicit_center_gap_wins_outright(self): + assert Host(scroll={'center_gap': 31})._center_gap_width() == 31 + + def test_explicit_zero_restores_edge_to_edge_logos(self): + # 0 is a real setting, not a falsy miss -- the guard is `>= 0`. + assert Host(scroll={'center_gap': 0})._center_gap_width() == 0 + + def test_otherwise_it_scales_with_card_width_within_the_clamp(self): + h = Host(width=512) + # 512 * 0.28 = 143, clamped to the 40px ceiling. + assert h._center_gap_width() >= h.CENTER_GAP_MIN_PX + + def test_the_gap_never_ends_up_narrower_than_the_score(self): + # This is the bug the measurement exists to prevent: a derived gap + # smaller than the rendered score drew the score over the logos. + h = Host(width=64) + assert h._center_gap_width() >= h._score_reserve_width() + + def test_a_junk_ratio_falls_back_to_the_floor(self): + h = Host(scroll={'center_gap_ratio': 'wide'}) + assert h._center_gap_width() == h.CENTER_GAP_MIN_PX + + +class TestNonFiniteSettings: + """inf reaches int() and raises OverflowError, which the old + `except (TypeError, ValueError)` did not catch -- so one bad config value + aborted the entire card render rather than falling back.""" + + @pytest.mark.parametrize("bad", [float("inf"), float("-inf")]) + def test_a_non_finite_center_gap_falls_back(self, bad): + h = Host(scroll={'center_gap': bad}) + assert h._center_gap_width() >= h.CENTER_GAP_MIN_PX + + @pytest.mark.parametrize("bad", [float("inf"), float("-inf"), float("nan")]) + def test_a_non_finite_ratio_falls_back_to_the_floor(self, bad): + h = Host(scroll={'center_gap_ratio': bad}) + assert h._center_gap_width() == h.CENTER_GAP_MIN_PX + + @pytest.mark.parametrize("bad", [float("inf"), float("-inf")]) + def test_non_finite_clamp_bounds_fall_back(self, bad): + h = Host(scroll={'center_gap_min': bad, 'center_gap_max': bad}) + assert h._center_gap_width() == h.CENTER_GAP_MIN_PX + + @pytest.mark.parametrize("bad", [float("inf"), float("-inf"), "inf", "-inf", "nan"]) + def test_a_non_finite_layout_offset_gives_the_default(self, bad): + cfg = {'customization': {'layout': {'score': {'x_offset': bad}}}} + assert Host(config=cfg)._layout_offset('score', 'x_offset', 7) == 7 + + def test_a_finite_value_is_still_honoured(self): + # The guard must not swallow ordinary settings. + assert Host(scroll={'center_gap': 31})._center_gap_width() == 31 + cfg = {'customization': {'layout': {'score': {'x_offset': -3}}}} + assert Host(config=cfg)._layout_offset('score', 'x_offset', 7) == -3 + + +class TestScoreReserve: + def test_it_measures_the_probe_plus_both_gutters(self): + h = Host() + assert h._score_reserve_width() > 2 * h._SCORE_LOGO_GUTTER_PX + + def test_a_wider_probe_reserves_more(self): + class Wide(Host): + _SCORE_PROBE = "000-000" + assert Wide()._score_reserve_width() > Host()._score_reserve_width() + + def test_an_unmeasurable_font_reserves_nothing_rather_than_raising(self): + h = Host() + h.fonts = {'score': object()} + assert h._score_reserve_width() == 0 + + +class TestLogoSlot: + def test_the_slot_is_what_is_left_after_the_gap(self): + h = Host(width=128, scroll={'center_gap': 40}) + assert h._logo_slot_width() == 44 + + def test_it_is_not_capped_at_the_card_height(self): + # The height cap is what froze logos at 46px on a 128px card. + h = Host(width=512, height=32, scroll={'center_gap': 40}) + assert h._logo_slot_width() > h.display_height + + def test_a_gap_wider_than_the_card_still_leaves_a_usable_slot(self): + assert Host(width=64, scroll={'center_gap': 200})._logo_slot_width() == 8 + + def test_the_cache_key_is_scoped_to_the_slot_not_just_the_name(self): + # One cache dict is shared by renderers of different card widths. + narrow = Host(width=64, scroll={'center_gap': 20})._logo_cache_key("NYY") + wide = Host(width=256, scroll={'center_gap': 20})._logo_cache_key("NYY") + assert narrow != wide + + +class TestLayoutOffset: + def _host(self, value): + return Host(config={'customization': {'layout': {'score': {'x_offset': value}}}}) + + def test_it_reads_the_same_block_as_the_full_screen_scorebug(self): + assert self._host(5)._layout_offset('score', 'x_offset') == 5 + + def test_a_string_offset_from_the_web_ui_is_coerced(self): + assert self._host("-3")._layout_offset('score', 'x_offset') == -3 + + def test_a_bool_is_not_silently_an_offset_of_one(self): + assert self._host(True)._layout_offset('score', 'x_offset', 9) == 9 + + @pytest.mark.parametrize("cfg", [{}, {'customization': {}}, + {'customization': {'layout': {}}}]) + def test_a_missing_block_gives_the_default(self, cfg): + assert Host(config=cfg)._layout_offset('score', 'x_offset', 7) == 7 + + def test_an_unparseable_offset_gives_the_default(self): + assert self._host("left")._layout_offset('score', 'x_offset', 4) == 4 + + +class TestUpcomingCenter: + def test_none_draws_nothing(self): + h = Host(scroll={'upcoming_center': 'none'}) + h._draw_upcoming_center(_draw(), {}) + assert h.drawn == [] + + def test_vs_is_the_default_and_never_a_score(self): + # An upcoming game has not started; the extractor's 0-0 is noise. + h = Host() + h._draw_upcoming_center(_draw(), {'home_score': 0, 'away_score': 0}) + assert [t for t, _ in h.drawn] == ['VS'] + + def test_an_empty_vs_string_draws_nothing(self): + h = Host(scroll={'vs_text': ''}) + h._draw_upcoming_center(_draw(), {}) + assert h.drawn == [] + + def test_date_time_stacks_both_lines(self): + h = Host(scroll={'upcoming_center': 'date_time'}) + h._draw_upcoming_center(_draw(), {'game_date': 'Sep 19', 'game_time': '7:00 PM'}) + assert [t for t, _ in h.drawn] == ['Sep 19', '7:00 PM'] + + def test_hiding_both_lines_draws_nothing(self): + h = Host(scroll={'upcoming_center': 'date_time', + 'show_date': False, 'show_time': False}) + h._draw_upcoming_center(_draw(), {'game_date': 'Sep 19', 'game_time': '7:00 PM'}) + assert h.drawn == [] + + +class TestUpcomingStatus: + def test_time_on_top_and_date_below_by_default(self): + h = Host() + h._draw_upcoming_game_status(_draw(), {'game_date': 'Sep 19', 'game_time': '7:00 PM'}) + assert [t for t, _ in h.drawn] == ['7:00 PM', 'Sep 19'] + + def test_swap_date_time_reverses_them(self): + h = Host(scroll={'swap_date_time': True}) + h._draw_upcoming_game_status(_draw(), {'game_date': 'Sep 19', 'game_time': '7:00 PM'}) + assert [t for t, _ in h.drawn] == ['Sep 19', '7:00 PM'] + + def test_it_stays_out_of_the_way_when_the_centre_already_has_them(self): + # Otherwise the date and time print twice on the same card. + h = Host(scroll={'upcoming_center': 'date_time'}) + h._draw_upcoming_game_status(_draw(), {'game_date': 'Sep 19', 'game_time': '7:00 PM'}) + assert h.drawn == [] + + def test_the_bottom_line_is_measured_not_a_fixed_offset(self): + # A fixed -7 ran "Sep 19" past the card wherever the detail font is + # 10px rather than 6px. + h = Host(height=64) + h._draw_upcoming_game_status(_draw(), {'game_date': 'Sep 19', 'game_time': '7:00 PM'}) + bottom_y = h.drawn[1][1][1] + assert 0 <= bottom_y < h.display_height + + +class TestRankings: + def test_set_rankings_cache_replaces_the_cache(self): + h = Host() + h.set_rankings_cache({'UGA': 1}) + assert h._team_rankings_cache == {'UGA': 1} + + +class TestContract: + def test_the_mixin_carries_no_state_of_its_own(self): + # Adoption must be one line on the class statement; a mixin with an + # __init__ would force eight constructors to cooperate. + assert '__init__' not in SportsGameRendererMixin.__dict__ + + def test_a_host_providing_the_documented_surface_needs_nothing_more(self): + # Host defines exactly what the module docstring names. If the mixin + # grows a new self.* dependency, this is what fails. + h = Host() + h._center_gap_width() + h._logo_slot_width() + h._logo_cache_key("X") + h._layout_offset('score', 'x_offset') + h._upcoming_date_and_time({}) + h._draw_upcoming_center(_draw(), {}) + h._draw_upcoming_game_status(_draw(), {}) + h.set_rankings_cache({}) diff --git a/test/test_sports_shared.py b/test/test_sports_shared.py new file mode 100644 index 00000000..bd402ae3 --- /dev/null +++ b/test/test_sports_shared.py @@ -0,0 +1,376 @@ +"""The shared sports.py mixins: their host contract, and _plugin_dir. + +Two things are worth testing here and the rest is not. The 45 method bodies +moved verbatim from the plugins, so they are covered by the plugins' own tests +and by 176 byte-identical safety-harness renders. What is genuinely new is: + +1. The contract. Every ``self.`` a mixin reads must be defined on the + mixin, or a host that does not happen to declare it raises AttributeError at + runtime. Two were missed on the first pass (_QUALITY_CHOICES and + _RANKING_COVERAGE_SECONDS); the eight plugins all declare them, so nothing + failed -- it would only have bitten a ninth. The test derives the list rather + than restating it, so the next omission fails here instead of in the field. + +2. ``_plugin_dir``. This is the only line of genuinely new logic in the move. In + sports.py these methods found config_schema.json with ``__file__``; here that + is src/common/, so the plugin directory has to be recovered from the + instance -- and getting it wrong is silent, costing grid-snapped font sizes + (measured at 81% anti-aliased edges) rather than raising. +""" + +import ast +import os +import sys +import types +from abc import ABC + +import pytest + +from src.common import sports_shared +from src.common.sports_shared import ( + SportsCoreSharedMixin, SportsLiveSharedMixin, SportsRecentSharedMixin) + +MIXINS = (SportsCoreSharedMixin, SportsLiveSharedMixin, SportsRecentSharedMixin) + + +def _constants_read_by_mixins(): + """Every ALL-CAPS ``self.X`` the mixin bodies read, found by parsing them.""" + tree = ast.parse(open(sports_shared.__file__).read()) + names = set() + for node in ast.walk(tree): + if (isinstance(node, ast.Attribute) + and isinstance(node.value, ast.Name) + and node.value.id == "self" + and node.attr.upper() == node.attr): + names.add(node.attr) + return names + + +class TestHostContract: + def test_every_constant_read_is_also_defined(self): + # Otherwise a host that does not declare it raises AttributeError the + # first time the code path runs -- which for these is mid-render. + missing = sorted( + name for name in _constants_read_by_mixins() + if not any(hasattr(m, name) for m in MIXINS)) + assert missing == [], ( + f"read but never defined on a mixin: {missing}. Give each a default " + f"on SportsCoreSharedMixin and document it in the module docstring.") + + @pytest.mark.parametrize("name,expected", [ + ("_QUALITY_CHOICES", frozenset({"any", "ranked"})), + ("_RANKING_COVERAGE_SECONDS", 3600), + ("_SCORE_PROBE_TEXT", "00-00"), + ("_FONT_DESIGN_HEIGHT", 32), + ]) + def test_defaults_match_what_the_plugins_ship(self, name, expected): + # The eight plugins declare their own copies, which shadow these. The + # values must still agree, or a ninth plugin inheriting the default + # behaves differently from the eight. + assert getattr(SportsCoreSharedMixin, name) == expected + + def test_only_the_recent_mixin_carries_a_constructor(self): + # SportsCore and SportsLive keep their own __init__ -- those differ per + # plugin. SportsRecent.__init__ was one of the 48 byte-identical bodies, + # so it moved with the rest; that is deliberate, not an oversight. + assert "__init__" not in SportsCoreSharedMixin.__dict__ + assert "__init__" not in SportsLiveSharedMixin.__dict__ + assert "__init__" in SportsRecentSharedMixin.__dict__ + + def test_the_recent_constructor_still_chains_to_the_host(self): + """Its zero-arg super() binds to where it is DEFINED, not where it is used. + + Moving a body containing bare ``super()`` is the one move that can + change meaning: the compiler closes over __class__ = the defining class, + so after the move that is SportsRecentSharedMixin rather than the + plugin's SportsRecent. It still works only because the mixin is listed + first, leaving the host class next in the MRO -- adopt it in the other + order and the chain silently skips the host's __init__. + """ + calls = [] + + class Host: + def __init__(self, config, display_manager, cache_manager, logger, sport_key): + calls.append(sport_key) + self.mode_config = {} + + class Recent(SportsRecentSharedMixin, Host): + pass + + inst = Recent({}, None, None, None, "nhl") + assert calls == ["nhl"], "the host constructor must still run" + assert inst.current_game_index == 0 + assert inst.update_interval == 3600 + assert inst._zero_clock_timestamps == {} + + def test_adopting_the_recent_mixin_second_would_skip_the_host(self): + # The failure mode the ordering above prevents, pinned so nobody + # "tidies" the base list. + calls = [] + + class Host: + def __init__(self, *a): + calls.append(a) + self.mode_config = {} + + class Wrong(Host, SportsRecentSharedMixin): + pass + + Wrong({}, None, None, None, "nhl") + # Host.__init__ wins and the mixin's setup never runs at all. + assert not hasattr(Wrong({}, None, None, None, "nhl"), "current_game_index") + + +class _Host(SportsCoreSharedMixin): + pass + + +def _write_plugin(tmp_path, name="fakeplug", schema=True): + """A throwaway package on sys.path, with or without a config_schema.json.""" + d = tmp_path / name + d.mkdir() + (d / "__init__.py").write_text("") + (d / "mod.py").write_text("class Leaf:\n pass\n") + if schema: + (d / "config_schema.json").write_text( + '{"properties": {"customization": {"properties": ' + '{"score": {"properties": {"font_size": {"default": 16}}}}}}}') + return d + + +class TestPluginDir: + def test_it_finds_the_directory_holding_config_schema_json(self, tmp_path, monkeypatch): + d = _write_plugin(tmp_path) + monkeypatch.syspath_prepend(str(tmp_path)) + mod = __import__("fakeplug.mod", fromlist=["Leaf"]) + host = type("H", (mod.Leaf, SportsCoreSharedMixin), {})() + assert host._plugin_dir() == str(d) + + def test_a_class_built_by_type_still_resolves(self, tmp_path, monkeypatch): + # SportsCore is an ABC, so type(name, bases, ns) reports __module__ as + # "abc" rather than the plugin -- which is exactly what the plugins' + # own tests build. Walking the MRO is what steps past it. + d = _write_plugin(tmp_path, "abcplug") + monkeypatch.syspath_prepend(str(tmp_path)) + mod = __import__("abcplug.mod", fromlist=["Leaf"]) + + class Base(SportsCoreSharedMixin, mod.Leaf, ABC): + pass + + synthetic = type("Probe", (Base,), {}) + assert synthetic.__module__ == "abc", "precondition: the trap this guards" + assert synthetic.__new__(synthetic)._plugin_dir() == str(d) + + def test_it_returns_none_when_no_schema_is_anywhere_on_the_mro(self, tmp_path, monkeypatch): + d = _write_plugin(tmp_path, "noschema", schema=False) + monkeypatch.syspath_prepend(str(tmp_path)) + mod = __import__("noschema.mod", fromlist=["Leaf"]) + host = type("H", (mod.Leaf, SportsCoreSharedMixin), {})() + # None rather than a wrong guess: _schema_font_size then caches empty + # and every element keeps its own default. + assert host._plugin_dir() is None + + def test_it_never_returns_the_core_module_directory(self): + # The bug this replaced: __file__ pointed at src/common/, so the schema + # was never found and font sizes silently stopped snapping to the grid. + host = _Host() + core_common = os.path.dirname(os.path.abspath(sports_shared.__file__)) + assert host._plugin_dir() != core_common + + def test_a_module_with_no_file_is_skipped_not_crashed_on(self, monkeypatch): + # Namespace packages and some frozen/dynamic modules have no __file__. + ghost = types.ModuleType("ghost_no_file") + if hasattr(ghost, "__file__"): + del ghost.__file__ + monkeypatch.setitem(sys.modules, "ghost_no_file", ghost) + cls = type("H", (SportsCoreSharedMixin,), {"__module__": "ghost_no_file"}) + assert cls.__new__(cls)._plugin_dir() is None + + +class TestSchemaFontSize: + def test_it_reads_the_plugin_schema_not_the_cores(self, tmp_path, monkeypatch): + d = _write_plugin(tmp_path, "sizeplug") + monkeypatch.syspath_prepend(str(tmp_path)) + mod = __import__("sizeplug.mod", fromlist=["Leaf"]) + host = type("H", (mod.Leaf, SportsCoreSharedMixin), {})() + assert host._schema_font_size("score") == 16 + + def test_an_unknown_element_is_none(self, tmp_path, monkeypatch): + _write_plugin(tmp_path, "unkplug") + monkeypatch.syspath_prepend(str(tmp_path)) + mod = __import__("unkplug.mod", fromlist=["Leaf"]) + host = type("H", (mod.Leaf, SportsCoreSharedMixin), {})() + assert host._schema_font_size("nonesuch") is None + + def test_an_empty_key_is_none_without_touching_the_disk(self): + assert _Host()._schema_font_size("") is None + + def test_a_missing_schema_degrades_to_none_rather_than_raising(self, tmp_path, monkeypatch): + _write_plugin(tmp_path, "bareplug", schema=False) + monkeypatch.syspath_prepend(str(tmp_path)) + mod = __import__("bareplug.mod", fromlist=["Leaf"]) + host = type("H", (mod.Leaf, SportsCoreSharedMixin), {})() + assert host._schema_font_size("score") is None + + +class _LiveHost(SportsLiveSharedMixin): + """The documented contract for the live mixin, and nothing else.""" + + def __init__(self, no_data_interval=300, stale_game_timeout=600, over=()): + self.no_data_interval = no_data_interval + self.stale_game_timeout = stale_game_timeout + self.game_update_timestamps = {} + self._over = set(over) + + class _L: + def __getattr__(self, _n): + return lambda *a, **k: None + self.logger = _L() + + def _is_game_really_over(self, game): + return game.get("id") in self._over + + +class TestLiveMixin: + """These three moved to the core, so they are tested here. + + They were already covered by hockey's and lacrosse's own tests, but those + two plugins disable live mode in their safety-harness fixtures, so the 176 + renders never exercise this path. Testing the mixin directly means the + coverage no longer depends on which plugin happens to have a unit test. + """ + + def test_a_stale_game_is_dropped_and_forgotten(self): + h = _LiveHost(stale_game_timeout=600) + import time as _t + h.game_update_timestamps["g1"] = {"last_seen": _t.time() - 5000} + games = [{"id": "g1", "home_abbr": "H", "away_abbr": "A"}] + h._detect_stale_games(games) + assert games == [] + assert "g1" not in h.game_update_timestamps, "its timestamp must go too" + + def test_a_fresh_game_survives(self): + h = _LiveHost(stale_game_timeout=600) + import time as _t + h.game_update_timestamps["g1"] = {"last_seen": _t.time() - 5} + games = [{"id": "g1"}] + h._detect_stale_games(games) + assert len(games) == 1 + + def test_a_game_never_seen_is_not_treated_as_stale(self): + # last_seen 0 means "no reading", not "seen at the epoch". + h = _LiveHost() + games = [{"id": "g1"}] + h._detect_stale_games(games) + assert len(games) == 1 + + def test_a_game_with_no_id_is_left_alone(self): + h = _LiveHost() + games = [{"home_abbr": "H"}] + h._detect_stale_games(games) + assert len(games) == 1 + + def test_a_finished_game_is_dropped_even_when_fresh(self): + h = _LiveHost(over=("g2",)) + games = [{"id": "g1"}, {"id": "g2"}] + h._detect_stale_games(games) + assert [g["id"] for g in games] == ["g1"] + + def test_removing_several_does_not_skip_any(self): + # It iterates a copy for exactly this reason; mutating the live list + # while looping would step over the element after each removal. + h = _LiveHost(over=("g1", "g2", "g3")) + games = [{"id": "g1"}, {"id": "g2"}, {"id": "g3"}] + h._detect_stale_games(games) + assert games == [] + + def test_the_idle_interval_escalates_with_the_empty_streak(self): + h = _LiveHost(no_data_interval=60) + h.live_idle_max_interval = 100000 + base = h._idle_live_interval() + h._empty_live_streak = 6 + short = h._idle_live_interval() + h._empty_live_streak = 24 + long = h._idle_live_interval() + assert base < short < long + + def test_the_ceiling_bounds_even_the_unescalated_interval(self): + # base > ceiling is a reachable config: the two settings are + # independent integers with no cross-validation. Returning base + # unclamped made the wait SHRINK as the streak grew. + h = _LiveHost(no_data_interval=3600) + h.live_idle_max_interval = 900 + h._empty_live_streak = 0 + assert h._idle_live_interval() == 900 + h._empty_live_streak = 24 + assert h._idle_live_interval() == 900 + + def test_finding_a_live_game_resets_the_streak(self): + h = _LiveHost() + h._note_live_fetch(False) + h._note_live_fetch(False) + assert h._empty_live_streak == 2 + h._note_live_fetch(True) + assert h._empty_live_streak == 0 + + def test_the_streak_starts_from_absent_state(self): + # The host is not required to pre-declare _empty_live_streak. + h = _LiveHost() + assert not hasattr(h, "_empty_live_streak") + h._note_live_fetch(False) + assert h._empty_live_streak == 1 + + +class TestPluginDirIsToldNotDeduced: + """The regression that shipped: _plugin_dir returned None in production. + + The first version walked the MRO for a class whose module sits beside a + config_schema.json. That passes when a test imports the plugin directly -- + which is how it was verified -- and returns None under the real plugin + loader, which imports modules by a path that leaves no such entry on the + MRO. + + Silent, and expensive: no schema means _schema_font_size returns None for + every element, so a configured size equal to the schema default stops + looking like a default, is treated as a deliberate choice, and skips the + grid snap. 4x6-font.ttf then renders at 6 rather than 7 -- 3px-wide glyphs + instead of 4px. On a 256x64 panel that made the odds, the records and the + date row illegible. A user counted the pixels; no gate here caught it. + """ + + def test_a_declared_directory_is_used(self, tmp_path): + d = _write_plugin(tmp_path, "declared") + host = type("H", (SportsCoreSharedMixin,), {"_PLUGIN_DIR": str(d)})() + assert host._plugin_dir() == str(d) + + def test_it_works_when_no_module_on_the_mro_helps(self, tmp_path, monkeypatch): + """The production case: nothing on the MRO sits beside a schema.""" + d = _write_plugin(tmp_path, "loaderstyle") + # A class whose module is not importable by name, as the loader produces. + cls = type("Loaded", (SportsCoreSharedMixin,), {"_PLUGIN_DIR": str(d)}) + cls.__module__ = "a.module.name.that.is.not.in.sys.modules" + assert cls.__new__(cls)._plugin_dir() == str(d), ( + "the declared directory must win when the MRO walk cannot help") + + def test_without_it_the_mro_walk_would_have_failed(self): + # Pin the precondition, so this test still means something if the + # fallback is ever changed. + cls = type("Orphan", (SportsCoreSharedMixin,), {}) + cls.__module__ = "not.a.real.module" + assert cls.__new__(cls)._plugin_dir() is None + + def test_a_declared_directory_without_a_schema_is_not_trusted(self, tmp_path): + # A stale or wrong path must not shadow the fallback. + empty = tmp_path / "noschema" + empty.mkdir() + d = _write_plugin(tmp_path, "realone") + monkey = type("H", (SportsCoreSharedMixin,), {"_PLUGIN_DIR": str(empty)}) + assert monkey.__new__(monkey)._plugin_dir() != str(empty) + + def test_the_font_size_consequence(self, tmp_path): + """End to end: a declared directory restores the schema lookup.""" + d = _write_plugin(tmp_path, "sizeconseq") + host = type("H", (SportsCoreSharedMixin,), {"_PLUGIN_DIR": str(d)})() + assert host._schema_font_size("score") == 16, ( + "without the schema this is None, which is what made a configured " + "size look user-chosen and skipped the grid snap") diff --git a/test/test_sports_sunset_matrix.py b/test/test_sports_sunset_matrix.py new file mode 100644 index 00000000..db6da2fb --- /dev/null +++ b/test/test_sports_sunset_matrix.py @@ -0,0 +1,310 @@ +"""What happens to an adopted sports plugin on each core it can meet. + +B5 moved the eight scoreboards onto `src.common.sports_scroll` behind a guarded +import, keeping a bundled copy as the fallback. B6 deletes those copies. The +two phases make *different* promises, and only the second one is obvious: + +| | bundled copy present | bundled copy removed | +|--------------------|---------------------------------|-----------------------------| +| **pinned old core**| loads -- B5's whole guarantee | ERROR, naming the module | +| **current core** | loads, using core code | loads, using core code | + +The top-left cell is the one worth having: nothing else in this suite proves an +adopted plugin still runs on a core that predates the module, and that claim is +the entire basis for having shipped B5 ahead of B6's gate. + +The bottom-left cell is the B6 failure mode, and it is asserted through +`PluginManager.load_plugin` rather than a bare import on purpose. The manager +catches the `ModuleNotFoundError`, so nothing propagates to a caller: a test +that expected `pytest.raises` would pass against a core where the module is +merely *broken* rather than absent, and would say nothing about what the user +actually experiences. What they get is a plugin parked in ERROR and one log +line -- which is precisely why B6 needs the install gate rather than trusting +the failure to be noticed. + +This covers the load path. The install/update gate -- which is what should stop +a sunset plugin reaching an old core in the first place -- is the other half of +the guarantee and is tested in test_plugin_compatibility_gate.py. + +See docs/SPORTS_UNIFICATION.md, "B6 -- why the sunset needs more than a version +floor". +""" + +import itertools +import json +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +project_root = Path(__file__).parent.parent +if str(project_root) not in sys.path: + sys.path.insert(0, str(project_root)) + +from src.plugin_system.plugin_manager import PluginManager +from src.plugin_system.plugin_state import PluginState + + +_PLUGIN_IDS = itertools.count() + +CORE_MODULE = "src.common.sports_scroll" + +# Only the leaf. A pre-3.2.0 core still ships `src/common/` -- scroll_helper +# and friends live there -- and it is `sports_scroll.py` alone that is absent. +# Hiding the whole package would be a different, harsher core than any that +# shipped, and it would make the load failure name the package rather than the +# module, which is the thing a reader needs to see. +HIDDEN = (CORE_MODULE,) + + +class _PinnedOldCore: + """Make `src.common.sports_scroll` un-importable for the duration. + + A meta_path finder rather than a monkeypatched `__import__`: the plugin is + executed by the real loader through `exec_module`, so the block has to live + in the import system itself to be reached. + """ + + def __init__(self, *names): + self.names = set(names) + self._saved = {} + + def find_spec(self, fullname, path=None, target=None): + if fullname in self.names: + raise ModuleNotFoundError(f"No module named {fullname!r}", name=fullname) + return None + + def __enter__(self): + for name in list(sys.modules): + if name in self.names: + self._saved[name] = sys.modules.pop(name) + sys.meta_path.insert(0, self) + return self + + def __exit__(self, *exc): + sys.meta_path.remove(self) + sys.modules.update(self._saved) + return False + + +# The two source shapes, kept as literals rather than copied from a plugin at +# runtime: this file lives in the core repo and must not depend on a plugin +# checkout being present, and pinning the shapes here means a plugin that +# drifts away from one is a visible edit, not a silently weakened test. + +# B5, as the eight scoreboards ship today: prefer the core module, fall back to +# the bundled copy. The except clause is narrow on purpose -- a bare +# `except ImportError` would also swallow a failure raised *inside* a core +# module that is present, quietly loading the legacy copy and hiding a broken +# core install. +ADOPTED_WITH_FALLBACK = ''' +_USING_CORE_SCROLL = False +try: + from src.common.sports_scroll import SportsScrollDisplay as _Base + _USING_CORE_SCROLL = True +except ModuleNotFoundError as exc: + if exc.name not in {"src", "src.common", "src.common.sports_scroll"}: + raise + _Base = None + +if not _USING_CORE_SCROLL: + from scroll_display_legacy import LegacyScrollDisplay as _Base +''' + +# B6, once the copies are deleted: there is nothing to fall back to, so the +# guard goes with them. Keeping the try/except while removing the file it +# falls back to would only mislabel the failure -- the plugin would report a +# missing `scroll_display_legacy` and say nothing about the core module that +# is actually absent. +SUNSET_NO_FALLBACK = ''' +from src.common.sports_scroll import SportsScrollDisplay as _Base + +_USING_CORE_SCROLL = True +''' + +PLUGIN_BODY = ''' +from src.plugin_system.base_plugin import BasePlugin + + +class SunsetProbe(BasePlugin): + """Records which scroll implementation the guarded import selected.""" + + using_core_scroll = _USING_CORE_SCROLL + scroll_base = _Base + + def update(self): + pass + + def display(self, force_clear=False): + pass +''' + +LEGACY_COPY = ''' +class LegacyScrollDisplay: + """Stands in for the bundled pre-3.2.0 implementation.""" +''' + + +def _write_plugin(plugins_dir: Path, plugin_id: str, *, bundled_copy: bool) -> Path: + path = plugins_dir / plugin_id + path.mkdir(parents=True) + (path / "manifest.json").write_text( + json.dumps({ + "id": plugin_id, + "name": "Sunset Probe", + "version": "1.0.0", + "entry_point": "manager.py", + "class_name": "SunsetProbe", + "display_modes": ["sunset_probe"], + }), + encoding="utf-8", + ) + shape = ADOPTED_WITH_FALLBACK if bundled_copy else SUNSET_NO_FALLBACK + (path / "manager.py").write_text(shape + PLUGIN_BODY, encoding="utf-8") + if bundled_copy: + (path / "scroll_display_legacy.py").write_text(LEGACY_COPY, encoding="utf-8") + return path + + +@pytest.fixture +def load(tmp_path): + """Load a synthetic adopted plugin through the real PluginManager. + + Returns a callable taking the two axes of the matrix and handing back the + manager, so the caller can ask it for state and recorded error. + """ + plugins_dir = tmp_path / "plugin-repos" + plugins_dir.mkdir() + + def _load(*, core_has_module: bool, bundled_copy: bool, hide=HIDDEN): + # A distinct id per cell, from a counter that spans the whole session. + # The loader names plugin modules after the plugin id and sys.modules + # is process-global, so a per-test counter would hand the second test + # the first test's already-imported module -- which passes or fails on + # the wrong plugin's import. + plugin_id = f"sunset-probe-{next(_PLUGIN_IDS)}" + _write_plugin(plugins_dir, plugin_id, bundled_copy=bundled_copy) + + with patch('src.common.permission_utils.ensure_directory_permissions'): + manager = PluginManager( + plugins_dir=str(plugins_dir), + config_manager=MagicMock(), + display_manager=MagicMock(), + cache_manager=MagicMock(), + font_manager=MagicMock(), + ) + manager.discover_plugins() + if core_has_module: + ok = manager.load_plugin(plugin_id) + else: + with _PinnedOldCore(*hide): + ok = manager.load_plugin(plugin_id) + return manager, plugin_id, ok + + return _load + + +def _assert_loaded(manager, plugin_id, ok): + assert ok is True, ( + f"load_plugin returned False; state is " + f"{manager.state_manager.get_state(plugin_id)}, error " + f"{manager.state_manager.get_error_info(plugin_id)}" + ) + assert manager.state_manager.get_state(plugin_id) is not PluginState.ERROR + + +class TestBundledCopyPresent: + """B5's shape: the guarded import with the fallback still shipped.""" + + def test_old_core_falls_back_and_still_loads(self, load): + """The claim that made it safe to ship B5 before B6's gate.""" + manager, plugin_id, ok = load(core_has_module=False, bundled_copy=True) + + _assert_loaded(manager, plugin_id, ok) + plugin = manager.plugins[plugin_id] + assert plugin.using_core_scroll is False, ( + "the core module was hidden, so the plugin must be on its bundled copy" + ) + assert plugin.scroll_base.__name__ == "LegacyScrollDisplay" + + def test_current_core_prefers_the_core_module(self, load): + manager, plugin_id, ok = load(core_has_module=True, bundled_copy=True) + + _assert_loaded(manager, plugin_id, ok) + plugin = manager.plugins[plugin_id] + assert plugin.using_core_scroll is True, ( + "the bundled copy must not win while the core module is importable" + ) + assert plugin.scroll_base.__name__ == "SportsScrollDisplay" + + + def test_a_core_without_the_package_at_all_still_falls_back(self, load): + """The guard's other accepted names. + + Its except clause accepts `src` and `src.common` as well as the module + itself, so those branches exist in all eight shipped plugins. No core + that old is likely still running, but the code claiming to handle it is + real and nothing else exercises it -- an untested branch in a fallback + is exactly the kind that rots unnoticed until the fallback is needed. + """ + manager, plugin_id, ok = load( + core_has_module=False, bundled_copy=True, hide=(CORE_MODULE, "src.common") + ) + + _assert_loaded(manager, plugin_id, ok) + assert manager.plugins[plugin_id].using_core_scroll is False + + +class TestBundledCopyRemoved: + """B6's shape: the copies are gone and only the core module remains.""" + + def test_current_core_still_loads(self, load): + manager, plugin_id, ok = load(core_has_module=True, bundled_copy=False) + + _assert_loaded(manager, plugin_id, ok) + assert manager.plugins[plugin_id].using_core_scroll is True + + def test_old_core_errors_and_records_the_missing_module(self, load): + """The B6 failure mode, as the user meets it. + + Not `pytest.raises`: load_plugin catches it, so nothing reaches a + caller. The observable consequences are the ERROR state and the + recorded error -- and the error has to name the module, or whoever + reads the log cannot tell a missing core module from any other + import failure. + """ + manager, plugin_id, ok = load(core_has_module=False, bundled_copy=False) + + assert ok is False, "a plugin with no scroll implementation must not load" + assert manager.state_manager.get_state(plugin_id) is PluginState.ERROR + assert plugin_id not in manager.plugins, ( + "a plugin that failed to load must not be left registered" + ) + + info = manager.state_manager.get_error_info(plugin_id) + assert info is not None, "ERROR state recorded no error to explain it" + assert info['error_type'] == 'ModuleNotFoundError', info + assert CORE_MODULE in info['error'], ( + f"the recorded error must name the module that was missing, got {info['error']!r}" + ) + + +def test_the_matrix_has_one_failing_cell(load): + """Guards the shape of the table itself. + + Each cell above is asserted on its own, so a change that broke two of them + in compensating ways could leave every individual test passing. This says + the outcome depends on both axes and fails in exactly one combination. + """ + outcomes = { + (core, bundled): load(core_has_module=core, bundled_copy=bundled)[2] + for core in (True, False) + for bundled in (True, False) + } + assert outcomes == { + (True, True): True, + (True, False): True, + (False, True): True, + (False, False): False, + }, outcomes diff --git a/test/test_system_status_available_memory.py b/test/test_system_status_available_memory.py new file mode 100644 index 00000000..c08bde22 --- /dev/null +++ b/test/test_system_status_available_memory.py @@ -0,0 +1,92 @@ +"""/api/v3/system/status must report MemAvailable, not just used/total. + +"Memory used %" cannot tell a healthy board from one about to fail. Page cache +counts as used and is reclaimable on demand, so a Pi can read 70% used and be +perfectly fine, or read the same and be minutes from trouble. MemAvailable is +the kernel's own estimate of what a new allocation can actually obtain, and it +is the number that tracked the failure on a 1GB Pi 3B+: healthy running sat at +500MB+, the crash happened at 73MB, and by then fork() was failing -- sshd +could not spawn a session and systemd could not respawn the display, while the +kernel carried on answering pings. + +psutil.virtual_memory().available is MemAvailable on Linux. total - used is not +a substitute: they diverge exactly when unreclaimable memory (shmem, tmpfs) is +in play, which is when the distinction matters. +""" + +import json +import sys +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest +from flask import Flask + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +MB = 1024 * 1024 + + +@pytest.fixture +def client(): + pytest.importorskip("psutil") + app = Flask(__name__) + app.config["TESTING"] = True + from web_interface.blueprints.api_v3 import api_v3 + for attr in ("config_manager", "plugin_manager", "cache_manager"): + setattr(api_v3, attr, MagicMock()) + if "api_v3" not in app.blueprints: + app.register_blueprint(api_v3, url_prefix="/api/v3") + return app.test_client() + + +def _memory(total_mb, used_mb, available_mb): + m = MagicMock() + m.total = total_mb * MB + m.used = used_mb * MB + m.available = available_mb * MB + m.percent = round(used_mb / total_mb * 100, 1) + return m + + +def _get_status(client, memory): + # The endpoint caches for 10s; bypass so each case is measured fresh. + with patch("web_interface.cache.get_cached", return_value=None), \ + patch("psutil.virtual_memory", return_value=memory), \ + patch("psutil.cpu_percent", return_value=5.0), \ + patch("psutil.boot_time", return_value=0.0): + resp = client.get("/api/v3/system/status") + assert resp.status_code == 200, resp.data + return json.loads(resp.data)["data"] + + +def test_available_memory_is_reported(client): + data = _get_status(client, _memory(total_mb=905, used_mb=620, available_mb=284)) + assert "memory_available_mb" in data + assert data["memory_available_mb"] == pytest.approx(284, abs=0.5) + + +def test_available_is_not_total_minus_used(client): + # The case the readout exists for: 600MB is "not used", but only 300MB can + # actually be allocated. Reporting used% alone would call this healthy. + data = _get_status(client, _memory(total_mb=1000, used_mb=400, available_mb=300)) + + derived = data["memory_total_mb"] - data["memory_used_mb"] + assert derived == pytest.approx(600, abs=1) + assert data["memory_available_mb"] == pytest.approx(300, abs=0.5) + assert data["memory_available_mb"] != pytest.approx(derived, abs=1), \ + "available must come from MemAvailable, not be derived from used" + + +def test_existing_memory_fields_are_unchanged(client): + data = _get_status(client, _memory(total_mb=905, used_mb=620, available_mb=284)) + assert data["memory_total_mb"] == pytest.approx(905, abs=0.5) + assert data["memory_used_mb"] == pytest.approx(620, abs=0.5) + assert "memory_used_percent" in data + + +def test_a_nearly_exhausted_board_reports_a_small_number(client): + # 73MB available is what the board actually read when it stopped being able + # to fork. The readout has to surface that rather than round it away. + data = _get_status(client, _memory(total_mb=905, used_mb=800, available_mb=73)) + assert data["memory_available_mb"] == pytest.approx(73, abs=0.5) diff --git a/test/test_vegas_fps_health.py b/test/test_vegas_fps_health.py new file mode 100644 index 00000000..7db126e6 --- /dev/null +++ b/test/test_vegas_fps_health.py @@ -0,0 +1,99 @@ +"""Frame pacing and FPS health reporting must not depend on the wall clock. + +These devices have no RTC, so the system clock jumps by however wrong boot +time was the moment NTP first syncs. The render loop sleeps the *remainder* +of each frame budget: + + frame_elapsed = - frame_started + time.sleep(max(0.0, frame_interval - frame_elapsed)) + +With a wall-clock `now`, a backward jump makes frame_elapsed negative, so +`frame_interval - frame_elapsed` exceeds the whole budget and the render loop +stalls for the size of the correction. A forward jump instead inflates the +p99 and worst-frame numbers the telemetry reports. +""" +import ast +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +COORD = (Path(__file__).resolve().parent.parent + / "src" / "vegas_mode" / "coordinator.py") +TREE = ast.parse(COORD.read_text(encoding="utf-8")) + + +def _assignments_of(name): + """Every `name = ` in the module, as unparsed source.""" + out = [] + for node in ast.walk(TREE): + if isinstance(node, ast.Assign): + for target in node.targets: + if isinstance(target, ast.Name) and target.id == name: + out.append((node.lineno, ast.unparse(node.value))) + return out + + +def test_per_frame_timestamps_are_monotonic(): + for name in ("frame_started", "frame_elapsed"): + assigns = _assignments_of(name) + assert assigns, f"{name} is no longer assigned -- has the loop changed?" + for lineno, expr in assigns: + assert "time.time()" not in expr, ( + f"{name} at line {lineno} uses the wall clock ({expr!r}). A " + "backward NTP step makes the per-frame delta negative and the " + "loop then sleeps longer than the whole frame budget.") + assert "time.monotonic()" in expr, ( + f"{name} at line {lineno} is {expr!r}, expected monotonic") + + +def test_the_fps_window_is_monotonic(): + for lineno, expr in _assignments_of("current_time"): + assert "time.monotonic()" in expr, ( + f"current_time at line {lineno} is {expr!r}; fps is frames divided " + "by this delta, so a clock step would corrupt the rate itself") + + +def test_health_state_is_not_reset_every_iteration(): + """run_iteration() runs once per cycle -- locals here reset every few seconds. + + As locals, `last_fps_health_log = 0.0` made the 300s heartbeat fire on the + first sample of every iteration, and a recovery spanning two iterations was + never reported because was_degraded had already gone back to False. + """ + run_iteration = next( + (n for n in ast.walk(TREE) + if isinstance(n, ast.FunctionDef) and n.name == "run_iteration"), None) + assert run_iteration is not None, "run_iteration() not found" + + local_names = {t.id for n in ast.walk(run_iteration) + if isinstance(n, ast.Assign) + for t in n.targets if isinstance(t, ast.Name)} + for leaked in ("last_fps_health_log", "was_degraded"): + assert leaked not in local_names, ( + f"{leaked} is a local of run_iteration() again, so it resets every " + "cycle -- the heartbeat degenerates to once per iteration") + + body = ast.unparse(run_iteration) + assert "self._fps_last_health_log" in body and "self._fps_was_degraded" in body, ( + "the health state should live on the coordinator, across iterations") + + +def test_start_clears_stale_health_state(): + """A new run must not inherit "was degraded" from the previous one.""" + start = next((n for n in ast.walk(TREE) + if isinstance(n, ast.FunctionDef) and n.name == "start"), None) + assert start is not None, "start() not found" + body = ast.unparse(start) + assert "self._fps_last_health_log" in body and "self._fps_was_degraded" in body, ( + "start() does not reset the FPS health state") + + +def test_the_degraded_threshold_is_documented(): + """The 90% band is deliberate; say so where the constant is defined.""" + source = COORD.read_text(encoding="utf-8") + idx = source.index("_FPS_HEALTHY_FRACTION = ") + preamble = source[max(0, idx - 700):idx] + assert "90%" in preamble or "0.9" in preamble, ( + "the degradation threshold is not explained at its definition, so " + "'below target' reads as a bug rather than a deliberate band") diff --git a/test/web_interface/test_api_v3_secret_roundtrip.py b/test/web_interface/test_api_v3_secret_roundtrip.py index 1d70f009..9d07a011 100644 --- a/test/web_interface/test_api_v3_secret_roundtrip.py +++ b/test/web_interface/test_api_v3_secret_roundtrip.py @@ -229,6 +229,44 @@ class TestSavePluginConfig: "REAL-KEY-0123456789", "an unrelated edit destroyed the API key" assert env.fresh_load()[PLUGIN_ID]["city"] == "Dallas" + def test_an_unrelated_edit_does_not_erase_array_item_secrets(self, env): + """The scalar api_key case above, but for a list of credentials. + + remove_empty_secrets recursed into dicts only, so a list went into + deep_merge untouched -- and lists merge by *replacement*. Saving any + unrelated field posted [{"token": ""}, ...] straight over the stored + array and destroyed every token in it at once. + """ + assert self._save(env, {"accounts": [ + {"name": "a", "token": "REAL-A"}, + {"name": "b", "token": "REAL-B"}, + ], "city": "Austin"}).status_code == 200 + + # the user changes the city; both masked tokens ride along blank + assert self._save(env, {"accounts": [ + {"name": "a", "token": ""}, + {"name": "b", "token": ""}, + ], "city": "Dallas"}).status_code == 200 + + merged = env.fresh_load()[PLUGIN_ID] + assert [a.get("token") for a in merged["accounts"]] == \ + ["REAL-A", "REAL-B"], "an unrelated edit destroyed the array secrets" + assert [a["name"] for a in merged["accounts"]] == ["a", "b"] + assert merged["city"] == "Dallas" + + def test_one_array_secret_can_be_changed_without_losing_the_rest(self, env): + assert self._save(env, {"accounts": [ + {"name": "a", "token": "REAL-A"}, + {"name": "b", "token": "REAL-B"}, + ]}).status_code == 200 + assert self._save(env, {"accounts": [ + {"name": "a", "token": ""}, + {"name": "b", "token": "NEW-B"}, + ]}).status_code == 200 + + merged = env.fresh_load()[PLUGIN_ID] + assert [a.get("token") for a in merged["accounts"]] == ["REAL-A", "NEW-B"] + def test_a_secret_can_still_be_changed(self, env): """Dropping blanks must not stop a real new value from being saved.""" self._save(env, {"api_key": "first-key"}) diff --git a/test/web_interface/test_config_logging_omits_secrets.py b/test/web_interface/test_config_logging_omits_secrets.py new file mode 100644 index 00000000..031eb4f1 --- /dev/null +++ b/test/web_interface/test_config_logging_omits_secrets.py @@ -0,0 +1,45 @@ +"""The validation logging ran before separate_secrets, so it logged credentials. + +api_v3's plugin-config save logged `Full config: {plugin_config}` at INFO and +`Config that failed: {plugin_config}` at ERROR. Both run *before* +separate_secrets(), so plugin_config still held the values the user just typed +into the form -- API keys and tokens went to the journal in clear text. +""" +import re +from pathlib import Path + +import pytest + +SOURCE = (Path(__file__).resolve().parents[2] + / "web_interface" / "blueprints" / "api_v3.py") + +#: Objects that still hold submitted secret values at the point these log +#: calls run. Interpolating one whole into a log message leaks credentials. +UNREDACTED = ("plugin_config", "secrets_config", "current_secrets") + + +def _logging_lines(): + for number, line in enumerate(SOURCE.read_text(encoding="utf-8").splitlines(), 1): + stripped = line.strip() + if stripped.startswith("#"): + continue + if re.match(r"logger\.(debug|info|warning|error|critical|exception)\(", stripped): + yield number, stripped + + +@pytest.mark.parametrize("name", UNREDACTED) +def test_no_log_call_interpolates_a_whole_secret_bearing_object(name): + # {name} or {name['k']} leaks; {list(name.keys())} and {len(name)} do not. + bare = re.compile(r"\{" + re.escape(name) + r"(\[[^\]]*\])*\}") + offenders = [f"{n}: {text}" for n, text in _logging_lines() if bare.search(text)] + assert not offenders, ( + f"{name} still holds submitted secrets where these log calls run:\n " + + "\n ".join(offenders)) + + +def test_the_guard_would_notice_a_reintroduced_leak(): + """Pin the detector itself, so a rewrite cannot silently stop matching.""" + bare = re.compile(r"\{" + re.escape("plugin_config") + r"(\[[^\]]*\])*\}") + assert bare.search('logger.info(f"Full config: {plugin_config}")') + assert bare.search("logger.error(f\"{plugin_config['api_key']}\")") + assert not bare.search('logger.info(f"{list(plugin_config.keys())}")') diff --git a/test/web_interface/test_secret_helpers.py b/test/web_interface/test_secret_helpers.py index 5f33e899..a9eb4a7a 100644 --- a/test/web_interface/test_secret_helpers.py +++ b/test/web_interface/test_secret_helpers.py @@ -17,6 +17,7 @@ from src.web_interface.secret_helpers import ( separate_secrets, mask_secret_fields, mask_all_secret_values, + merge_secrets, remove_empty_secrets, ) @@ -239,3 +240,67 @@ class TestRemoveEmptySecrets: def test_keeps_falsey_non_string_values(self): # 0 and False are neither None nor blank strings — they are kept. assert remove_empty_secrets({"a": 0, "b": False}) == {"a": 0, "b": False} + + +class TestArrayItemSecrets: + """Lists merge by replacement, so a blanked array wipes stored credentials. + + remove_empty_secrets recursed into dicts but let a list through untouched, + so [{"token": ""}] went straight into deep_merge and overwrote the stored + list. Saving any unrelated setting destroyed every token in the array. + """ + + STORED = {"accounts": [{"name": "a", "token": "REAL-A"}, + {"name": "b", "token": "REAL-B"}]} + + def test_an_unrelated_save_keeps_every_stored_token(self): + posted = {"accounts": [{"name": "a", "token": ""}, + {"name": "b", "token": ""}]} + merged = merge_secrets(self.STORED, remove_empty_secrets(posted)) + assert [a["token"] for a in merged["accounts"]] == ["REAL-A", "REAL-B"] + + def test_editing_one_entry_leaves_the_others_alone(self): + posted = {"accounts": [{"name": "a", "token": ""}, + {"name": "b", "token": "NEW-B"}]} + merged = merge_secrets(self.STORED, remove_empty_secrets(posted)) + assert [a["token"] for a in merged["accounts"]] == ["REAL-A", "NEW-B"] + + def test_a_new_entry_is_appended(self): + posted = {"accounts": [{"name": "a", "token": ""}, + {"name": "b", "token": ""}, + {"name": "c", "token": "NEW-C"}]} + merged = merge_secrets(self.STORED, remove_empty_secrets(posted)) + assert [a["token"] for a in merged["accounts"]] == \ + ["REAL-A", "REAL-B", "NEW-C"] + + def test_a_list_of_bare_strings_merges_by_index(self): + merged = merge_secrets({"keys": ["K1", "K2", "K3"]}, + remove_empty_secrets({"keys": ["", "K2-NEW", ""]})) + assert merged["keys"] == ["K1", "K2-NEW", "K3"] + + def test_an_all_blank_list_is_dropped_entirely(self): + posted = {"accounts": [{"token": ""}, {"token": ""}]} + assert "accounts" not in remove_empty_secrets(posted) + + def test_plain_dict_secrets_are_unaffected(self): + merged = merge_secrets({"api_key": "OLD", "other": "keep"}, + remove_empty_secrets({"api_key": "", "other": "changed"})) + assert merged == {"api_key": "OLD", "other": "changed"} + + def test_a_removed_entry_takes_its_secret_with_it(self): + """The regular config's list is authoritative about how many items + exist, and the secrets list runs parallel to it -- see + ConfigManager._strip_secrets_recursive. So a shorter incoming list + must shorten the stored secrets too, or the two fall out of step.""" + posted = {"accounts": [{"name": "a", "token": "NEW-A"}]} + merged = merge_secrets(self.STORED, remove_empty_secrets(posted)) + assert [a["token"] for a in merged["accounts"]] == ["NEW-A"] + + def test_an_emptied_item_stays_a_dict_not_none(self): + """None there stops the list looking parallel, and + _strip_secrets_recursive then drops the whole key from the main + config -- deleting the item's non-secret fields as well.""" + pruned = remove_empty_secrets( + {"accounts": [{"token": "real"}, {"token": ""}]}) + assert pruned["accounts"] == [{"token": "real"}, {}] + assert None not in pruned["accounts"] diff --git a/web_interface/blueprints/api_v3.py b/web_interface/blueprints/api_v3.py index 37df1141..35dfd885 100644 --- a/web_interface/blueprints/api_v3.py +++ b/web_interface/blueprints/api_v3.py @@ -22,7 +22,8 @@ logger = logging.getLogger(__name__) from src.web_interface.api_helpers import success_response, error_response, validate_request_json from src.web_interface.errors import ErrorCode from src.web_interface.secret_helpers import (find_secret_fields, mask_all_secret_values, - remove_empty_secrets, separate_secrets, + merge_secrets, remove_empty_secrets, + separate_secrets, strip_masked_values) from src.web_interface.error_handler import describe_exception, redact_text from src.plugin_system.operation_types import OperationType @@ -597,7 +598,7 @@ def save_dim_schedule_config(): dim_brightness = 30 else: dim_brightness = int(dim_brightness_raw) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return error_response( ErrorCode.VALIDATION_ERROR, "dim_brightness must be an integer between 0 and 100", @@ -797,7 +798,7 @@ def save_main_config(): }), 400 try: target_fps = int(raw_target_fps) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({ 'status': 'error', 'message': "Invalid value for target_fps: must be an integer" @@ -867,7 +868,7 @@ def save_main_config(): mux_val = int(data['multiplexing']) if mux_val < 0 or mux_val > 22: return jsonify({'status': 'error', 'message': f"Invalid multiplexing value '{data['multiplexing']}'. Must be an integer from 0 to 22."}), 400 - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({'status': 'error', 'message': f"Invalid multiplexing value '{data['multiplexing']}'. Must be an integer from 0 to 22."}), 400 # Validate pixel_mapper_config (free-form mapper string, e.g. "U-mapper;Rotate:90") @@ -885,7 +886,7 @@ def save_main_config(): rat_val = int(data['row_address_type']) if rat_val < 0 or rat_val > 4: return jsonify({'status': 'error', 'message': f"Invalid row_address_type '{data['row_address_type']}'. Must be an integer from 0 to 4."}), 400 - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({'status': 'error', 'message': f"Invalid row_address_type '{data['row_address_type']}'. Must be an integer from 0 to 4."}), 400 # Handle hardware settings @@ -910,7 +911,7 @@ def save_main_config(): if rp1_val not in (0, 1): return jsonify({'status': 'error', 'message': "rp1_rio must be 0 (PIO) or 1 (RIO)"}), 400 current_config['display']['runtime']['rp1_rio'] = rp1_val - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({'status': 'error', 'message': "rp1_rio must be 0 or 1"}), 400 # Handle checkboxes - coerce to bool to ensure proper JSON types @@ -963,7 +964,7 @@ def save_main_config(): copies = None try: copies = int(data['double_sided_copies']) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): if enabled: return jsonify({'status': 'error', 'message': "Double-sided copies must be an integer"}), 400 if copies is not None and not (2 <= copies <= 8): @@ -1036,7 +1037,7 @@ def save_main_config(): if data.get('vegas_extend_threshold_screens') not in ('', None): try: screens = float(data['vegas_extend_threshold_screens']) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({ 'status': 'error', 'message': "Invalid value for vegas_extend_threshold_screens: " @@ -1053,7 +1054,7 @@ def save_main_config(): if data.get('vegas_max_plugin_width_ratio') not in ('', None): try: ratio = float(data['vegas_max_plugin_width_ratio']) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({ 'status': 'error', 'message': "Invalid value for vegas_max_plugin_width_ratio: " @@ -1101,7 +1102,7 @@ def save_main_config(): continue try: int_value = int(raw_value) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({ 'status': 'error', 'message': f"Invalid value for {field_name}: must be an integer" @@ -1153,7 +1154,7 @@ def save_main_config(): if not (1024 <= port_val <= 65535): return jsonify({'status': 'error', 'message': "sync_port must be between 1024 and 65535"}), 400 current_config['sync']['port'] = port_val - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({'status': 'error', 'message': "sync_port must be an integer"}), 400 if "sync_follower_position" in data: @@ -1197,7 +1198,7 @@ def save_main_config(): raw_value = data.pop(field) try: int_value = int(raw_value) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({'status': 'error', 'message': f"Invalid duration for {field}: must be an integer"}), 400 current_config['display']['display_durations'][field] = int_value @@ -1220,7 +1221,7 @@ def save_main_config(): continue try: int_value = int(raw_value) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({'status': 'error', 'message': f"Invalid duration for mode '{mode_key}': must be an integer"}), 400 current_config['display']['display_durations'][mode_key] = int_value @@ -1296,7 +1297,10 @@ def save_main_config(): if secrets_config: if plugin_id not in current_secrets: current_secrets[plugin_id] = {} - current_secrets[plugin_id] = deep_merge(current_secrets[plugin_id], secrets_config) + # Lists merge by replacement, so deep_merge here wrote a + # blanked array straight over the stored credentials. + current_secrets[plugin_id] = merge_secrets( + current_secrets[plugin_id], secrets_config) # Save secrets file api_v3.config_manager.save_raw_file_content('secrets', current_secrets) @@ -1559,6 +1563,11 @@ def get_system_status(): 'memory_used_percent': round(memory_percent, 1), 'memory_total_mb': round(memory.total / (1024 * 1024), 1), 'memory_used_mb': round(memory.used / (1024 * 1024), 1), + # MemAvailable, not total-minus-used: it accounts for reclaimable + # page cache, so it is what actually predicts memory trouble. A + # board can read 70% "used" and be fine, or read the same and be + # about to fail fork(), and only this number tells them apart. + 'memory_available_mb': round(memory.available / (1024 * 1024), 1), 'cpu_temp': round(cpu_temp, 1) if cpu_temp is not None else None, 'disk_used_percent': round(disk_percent, 1), 'disk_total_gb': round(disk.total / (1024 * 1024 * 1024), 1), @@ -5118,7 +5127,7 @@ def save_plugin_config(): converted_array.append(int(v)) else: converted_array.append(float(v)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): converted_array.append(v) else: converted_array.append(v) @@ -5143,7 +5152,7 @@ def save_plugin_config(): converted_array.append(int(v)) else: converted_array.append(float(v)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): converted_array.append(v) else: converted_array.append(v) @@ -5180,7 +5189,7 @@ def save_plugin_config(): converted_array.append(int(v)) else: converted_array.append(float(v)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): converted_array.append(v) else: converted_array.append(v) @@ -5204,7 +5213,7 @@ def save_plugin_config(): converted_array.append(int(v)) else: converted_array.append(float(v)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): converted_array.append(v) else: converted_array.append(v) @@ -5371,7 +5380,7 @@ def save_plugin_config(): if isinstance(v, str): try: converted.append(int(v) if item_type == 'integer' else float(v)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): converted.append(v) else: converted.append(v) @@ -5496,7 +5505,7 @@ def save_plugin_config(): try: normalized[key] = int(value_stripped) continue - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): pass elif isinstance(value, (int, float)): normalized[key] = int(value) @@ -5514,7 +5523,7 @@ def save_plugin_config(): try: normalized[key] = float(value_stripped) continue - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): pass elif isinstance(value, (int, float)): normalized[key] = float(value) @@ -5569,7 +5578,7 @@ def save_plugin_config(): try: normalized_array.append(int(v)) continue - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): pass elif isinstance(v, (int, float)): normalized_array.append(int(v)) @@ -5579,7 +5588,7 @@ def save_plugin_config(): try: normalized_array.append(float(v)) continue - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): pass elif isinstance(v, (int, float)): normalized_array.append(float(v)) @@ -5595,7 +5604,7 @@ def save_plugin_config(): if isinstance(v, str): try: normalized_array.append(int(v)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): normalized_array.append(v) elif isinstance(v, (int, float)): normalized_array.append(int(v)) @@ -5609,7 +5618,7 @@ def save_plugin_config(): if isinstance(v, str): try: normalized_array.append(float(v)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): normalized_array.append(v) else: normalized_array.append(v) @@ -5632,7 +5641,7 @@ def save_plugin_config(): if isinstance(value, str): try: normalized[key] = int(value) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): normalized[key] = value else: normalized[key] = value @@ -5641,7 +5650,7 @@ def save_plugin_config(): if isinstance(value, str): try: normalized[key] = float(value) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): normalized[key] = value else: normalized[key] = value @@ -5675,8 +5684,10 @@ def save_plugin_config(): if schema: # Log what we're validating for debugging logger.info(f"Validating config for {plugin_id}") + # Only the shape. plugin_config still holds the submitted secret + # values at this point -- separate_secrets does not run until + # below -- so logging it wrote live credentials to the journal. logger.info(f"Config keys being validated: {list(plugin_config.keys())}") - logger.info(f"Full config: {plugin_config}") # Get enhanced schema keys (including injected core properties) # We need to create an enhanced schema to get the actual allowed keys @@ -5699,7 +5710,8 @@ def save_plugin_config(): # Log validation errors for debugging logger.error(f"Config validation failed for {plugin_id}") logger.error(f"Validation errors: {validation_errors}") - logger.error(f"Config that failed: {plugin_config}") + # Keys only, for the same reason as above. + logger.error(f"Config keys that failed: {list(plugin_config.keys())}") logger.error(f"Schema properties: {list(enhanced_schema.get('properties', {}).keys())}") # Also print to console for immediate visibility @@ -5750,7 +5762,9 @@ def save_plugin_config(): if secrets_config: if plugin_id not in current_secrets: current_secrets[plugin_id] = {} - current_secrets[plugin_id] = deep_merge(current_secrets[plugin_id], secrets_config) + # See above -- secrets lists must merge element-wise. + current_secrets[plugin_id] = merge_secrets( + current_secrets[plugin_id], secrets_config) # Save secrets file try: api_v3.config_manager.save_raw_file_content('secrets', current_secrets) @@ -6779,7 +6793,7 @@ def get_font_preview() -> tuple[Response, int] | Response: # Safe integer parsing for size try: size = int(request.args.get('size', 12)) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return jsonify({'status': 'error', 'message': 'Invalid font size'}), 400 if not font_filename: @@ -8360,7 +8374,7 @@ def clear_old_errors(): context={'provided_value': raw_max_age}, status_code=400 ) - except (ValueError, TypeError): + except (ValueError, TypeError, OverflowError): return error_response( error_code=ErrorCode.INVALID_INPUT, message="max_age_hours must be a valid integer", diff --git a/web_interface/static/v3/app.js b/web_interface/static/v3/app.js index 581c66f2..638204b3 100644 --- a/web_interface/static/v3/app.js +++ b/web_interface/static/v3/app.js @@ -126,7 +126,17 @@ window.showRestartPending = function(message) { } catch { /* private browsing */ } const banner = document.getElementById('restart-pending-banner'); const text = document.getElementById('restart-pending-text'); - if (text && message) text.textContent = message; + if (text) { + // Without the else-branch a config save inherited whatever wording the + // previous update left in the DOM: showRestartPending() clears the + // stored text but used to leave the element itself alone. The default + // is read back from the server-rendered copy rather than duplicated + // here, so the template stays the one place that owns the string. + if (text.dataset.defaultText === undefined) { + text.dataset.defaultText = text.textContent.trim(); + } + text.textContent = message || text.dataset.defaultText; + } if (banner) banner.style.display = 'block'; }; diff --git a/web_interface/templates/v3/base.html b/web_interface/templates/v3/base.html index e739dd8c..053724b4 100644 --- a/web_interface/templates/v3/base.html +++ b/web_interface/templates/v3/base.html @@ -1164,9 +1164,16 @@ // The pull replaced files on disk; the running services still // hold the code they loaded at boot. Ask for the restart that // makes the update actually take effect. + // + // Both services, not just the display. The plugin store's + // compatibility gate runs in the web process, so a web service + // still holding the previous version refuses every plugin that + // floors on the release just installed -- blaming a core + // version that is already correct on disk. if (data.restart_required && typeof window.showRestartPending === 'function') { window.showRestartPending( - 'Update installed \u2014 restart the display to run the new code'); + 'Update installed \u2014 restart the display and web ' + + 'services to run the new code'); } } if (typeof showNotification === 'function') { diff --git a/web_interface/templates/v3/partials/tools.html b/web_interface/templates/v3/partials/tools.html index f4f7cfdd..1d0fb104 100644 --- a/web_interface/templates/v3/partials/tools.html +++ b/web_interface/templates/v3/partials/tools.html @@ -687,12 +687,35 @@ const mUsedGb = d.memory_used_mb != null ? (d.memory_used_mb / 1024).toFixed(1) : null; const mTotGb = d.memory_total_mb != null ? (d.memory_total_mb / 1024).toFixed(1) : null; const temp = d.cpu_temp != null ? d.cpu_temp + '°C' : 'N/A'; + // Available memory is the number that predicts trouble. When it + // runs out the board does not fail cleanly: fork() starts + // returning ENOMEM, so sshd cannot spawn a session and systemd + // cannot respawn the display, while the kernel keeps answering + // pings. Thresholds are drawn from that failure -- it was + // measured at 73MB free, and healthy running sits well above. + // Round ONCE, then colour and label off the same number. + // The API sends one decimal place, so classifying the raw + // value and displaying the rounded one disagreed at the + // boundaries: 149.6 rendered as "150 MB" in red, and 299.6 as + // "300 MB" in amber, both contradicting the threshold the + // colour claims to apply. Which side of the line a spare + // 0.4MB falls on does not matter; the tile agreeing with + // itself does. + const availMb = d.memory_available_mb == null + ? null : Math.round(d.memory_available_mb); + const availColor = availMb == null ? 'text-gray-400' + : availMb < 150 ? 'text-red-600' + : availMb < 300 ? 'text-amber-500' + : 'text-green-600'; panel.innerHTML = diagTile('fa-microchip', 'text-blue-600', 'CPU Usage', (d.cpu_percent != null ? d.cpu_percent : '--') + '%', null) + diagTile('fa-memory', 'text-green-600', 'Memory', (d.memory_used_percent != null ? d.memory_used_percent : '--') + '%', (mUsedGb && mTotGb) ? `${mUsedGb} / ${mTotGb} GB` : null) + + diagTile('fa-memory', availColor, 'Available Memory', + availMb != null ? `${availMb} MB` : '--', + mTotGb ? `of ${mTotGb} GB total` : null) + diagTile('fa-thermometer-half', 'text-red-600', 'CPU Temp', temp, null) + diagTile('fa-hdd', 'text-indigo-600', 'Disk', (d.disk_used_percent != null ? d.disk_used_percent : '--') + '%',