mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-20 13:39:13 +00:00
Compare commits
80
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7603728e5a | ||
|
|
6c23994b2f | ||
|
|
f12d11334a | ||
|
|
be7a7b7baf | ||
|
|
406e68fba9 | ||
|
|
ea54e56bed | ||
|
|
696acdbc7b | ||
|
|
91d15a8943 | ||
|
|
6bea1a7c21 | ||
|
|
0730d95200 | ||
|
|
32d637a446 | ||
|
|
bc2dbf3824 | ||
|
|
cb0545ecb3 | ||
|
|
300cdaa250 | ||
|
|
10da2f97fd | ||
|
|
92f9d06af9 | ||
|
|
6e361e05cc | ||
|
|
a686932c7e | ||
|
|
154525beb8 | ||
|
|
9b522d412c | ||
|
|
cbc540a679 | ||
|
|
4aeb0033e0 | ||
|
|
eae063700f | ||
|
|
af96bd5cb0 | ||
|
|
5e5979973e | ||
|
|
bdced206dc | ||
|
|
39e7f8cbe0 | ||
|
|
f90638a9ec | ||
|
|
333fd17d28 | ||
|
|
a4a55a23fc | ||
|
|
085fb93a87 | ||
|
|
c321b94085 | ||
|
|
5a1f121e6b | ||
|
|
6138a3cbef | ||
|
|
568cb6d77f | ||
|
|
2f42d179f6 | ||
|
|
37fc1b56b5 | ||
|
|
acc55ef119 | ||
|
|
1a0864e5d4 | ||
|
|
732c7d1a30 | ||
|
|
d42593e7ce | ||
|
|
f0bef7784c | ||
|
|
986f74e38b | ||
|
|
e450a6dfb6 | ||
|
|
5133643600 | ||
|
|
5929190e36 | ||
|
|
79ba93f5a6 | ||
|
|
e499efb1f0 | ||
|
|
6b74506695 | ||
|
|
5f29243e87 | ||
|
|
1fbe244e49 | ||
|
|
863e4a1ecd | ||
|
|
9c0c0dc851 | ||
|
|
71739d85d1 | ||
|
|
cc258aaffd | ||
|
|
fe5a3aa99d | ||
|
|
10e75b977f | ||
|
|
cf0a551f7b | ||
|
|
9018fa23cd | ||
|
|
0c5b9c57d3 | ||
|
|
9083df9f5c | ||
|
|
0901d044d3 | ||
|
|
08265c1135 | ||
|
|
5713fd20a7 | ||
|
|
9cf30bbbef | ||
|
|
a51fb7ce11 | ||
|
|
fce1fdac57 | ||
|
|
7171e6c022 | ||
|
|
9fbdd71941 | ||
|
|
2add759f40 | ||
|
|
bb1a1671ec | ||
|
|
8159afca43 | ||
|
|
44f59ede07 | ||
|
|
ca26c1b83b | ||
|
|
f887063434 | ||
|
|
6287acd591 | ||
|
|
ee59caa577 | ||
|
|
cd7e16e58e | ||
|
|
47e3021fc3 | ||
|
|
e319540c6e |
@@ -72,4 +72,4 @@ jobs:
|
|||||||
--ignore=test/plugins \
|
--ignore=test/plugins \
|
||||||
--cov=src --cov=web_interface \
|
--cov=src --cov=web_interface \
|
||||||
--cov-report=term \
|
--cov-report=term \
|
||||||
--cov-fail-under=45
|
--cov-fail-under=52
|
||||||
|
|||||||
@@ -17,6 +17,81 @@ release that ships it.
|
|||||||
accepts both, but the store flags the old spelling as deprecated
|
accepts both, but the store flags the old spelling as deprecated
|
||||||
(`store_manager.py`) and only the new one is in `schema/manifest_schema.json`.
|
(`store_manager.py`) and only the new one is in `schema/manifest_schema.json`.
|
||||||
|
|
||||||
|
## 3.3.0
|
||||||
|
|
||||||
|
**The release the sports scoreboards floor on to delete their bundled copies.**
|
||||||
|
3.2.0 shipped the unified sports library and made `ledmatrix_min_version`
|
||||||
|
enforceable; this ships the last three shared modules and completes the store
|
||||||
|
gate, so a scoreboard can now floor here and carry no fallback at all.
|
||||||
|
|
||||||
|
New modules a plugin may import via `src.*` and floor on 3.3.0 for:
|
||||||
|
|
||||||
|
- `src/common/sports_card.py` — settings, colour, font and date helpers for a
|
||||||
|
scoreboard's `game_renderer.py`. Free functions taking `config`/`fonts`
|
||||||
|
explicitly, so nothing about the caller's class is assumed.
|
||||||
|
- `src/common/sports_game_renderer.py` — `SportsGameRendererMixin`: scroll/Vegas
|
||||||
|
card geometry (centre gap, logo slot and cache key, layout offsets, the
|
||||||
|
upcoming-card date and time layout). No `__init__` and no state, so adoption
|
||||||
|
is one line on the class statement.
|
||||||
|
- `src/common/sports_shared.py` — `SportsCoreSharedMixin`,
|
||||||
|
`SportsLiveSharedMixin`, `SportsRecentSharedMixin`: the `sports.py` bodies
|
||||||
|
byte-identical in all eight lineage-sharing scoreboards.
|
||||||
|
|
||||||
|
All three sit under `src/common/` rather than `src/base_classes/sports/`,
|
||||||
|
deliberately: importing that package pulls `core.py` → `DisplayManager` →
|
||||||
|
`rgbmatrix`, and these are pure logic. Plugins importing them must not acquire a
|
||||||
|
hardware dependency.
|
||||||
|
|
||||||
|
Three notes for anyone adopting `sports_shared`:
|
||||||
|
|
||||||
|
- `SportsRecentSharedMixin` defines `__init__`. Its bare `super()` binds to the
|
||||||
|
mixin, so it reaches the host only when the mixin is listed **first** in the
|
||||||
|
bases. Reversing that order silently skips the host constructor.
|
||||||
|
- Methods that resolve the plugin's `config_schema.json` use `_plugin_dir()`,
|
||||||
|
which walks the MRO rather than reading `__file__` — `__file__` is now
|
||||||
|
`src/common/`. It walks because `SportsCore` is an ABC: a subclass built with
|
||||||
|
`type(name, bases, ns)` reports `__module__` as `"abc"`.
|
||||||
|
- `_get_timezone`, `_extract_game_details` and `_fetch_data` are byte-identical
|
||||||
|
across the eight but stay in the plugins. The first binds a per-plugin
|
||||||
|
timezone module whose contents differ; the other two are the abstract stubs
|
||||||
|
that define the sport.
|
||||||
|
|
||||||
|
**The store's compatibility gate is now on every registry-managed route.** 3.2.0
|
||||||
|
gated `install_plugin`. This release gates the git-pull update path and
|
||||||
|
`install_from_url`, so a plugin whose floor the core cannot meet is refused
|
||||||
|
after download with no partial directory left behind.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **ESPN 403s.** `site.api` began rejecting the User-Agent strings this repo
|
||||||
|
sent on 2026-08-04; every shared-data-source scoreboard returned
|
||||||
|
`403 Forbidden`. Requests now send an identifying token with a project URL —
|
||||||
|
browser-style strings and bare custom tokens are both refused.
|
||||||
|
- **Low-memory boards becoming unreachable under load** while still answering
|
||||||
|
pings and serving the web UI. Fetched payloads are released after delivery
|
||||||
|
rather than pinned on the completed request for up to an hour, malloc arenas
|
||||||
|
are capped, and log volume and SD writes are reduced. Available memory is now
|
||||||
|
reported in Tools diagnostics.
|
||||||
|
- **A failed logo download pinning a team to a grey box**: the placeholder was
|
||||||
|
written under the real logo's filename, so later attempts found a file and
|
||||||
|
reported success without retrying.
|
||||||
|
- **A plugin enabled but never loaded is retried** rather than staying absent
|
||||||
|
with `error = null`.
|
||||||
|
- `ttl` now controls cache expiry; abandoned cache writes no longer leave temp
|
||||||
|
files; one cache-cleanup thread per directory rather than per manager.
|
||||||
|
- Odds are fetched for the games displayed, not the whole schedule window, and a
|
||||||
|
stalled ESPN no longer stalls the whole plugin update.
|
||||||
|
- Array-item secrets are no longer wiped or logged.
|
||||||
|
- A restored backup matches the device it was taken from.
|
||||||
|
- The web UI reports the real error instead of "unknown", rejects non-finite
|
||||||
|
JSON numbers, and stops checkbox groups posting back hidden options.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- `display.hardware.orientation` for panels mounted upside down.
|
||||||
|
- Vegas keeps live content in the ticker rather than being preempted by it.
|
||||||
|
- Schemas can label enum dropdown options.
|
||||||
|
|
||||||
## 3.2.0
|
## 3.2.0
|
||||||
|
|
||||||
**The first release shipping the unified sports library.** This is the version
|
**The first release shipping the unified sports library.** This is the version
|
||||||
|
|||||||
@@ -161,9 +161,11 @@ The system supports live, recent, and upcoming game information for multiple spo
|
|||||||
### LED Matrix Panels
|
### LED Matrix Panels
|
||||||
(2x in a horizontal chain is recommended)
|
(2x in a horizontal chain is recommended)
|
||||||
- [Adafruit 64×32](https://www.adafruit.com/product/2278) – designed for 128×32 but works with dynamic scaling on many displays (pixel pitch is user preference)
|
- [Adafruit 64×32](https://www.adafruit.com/product/2278) – designed for 128×32 but works with dynamic scaling on many displays (pixel pitch is user preference)
|
||||||
|
**Warning: Lately the Waveshare Panels have had different variations - only some are compatible with this project. I hope to identify what is different to fix it but so far there is a decent chance you get a mis-matched set of panels if you don't buy them all at once! **
|
||||||
- [Waveshare 64×32](https://amzn.to/3Kw55jK) - Does not require E addressable pad
|
- [Waveshare 64×32](https://amzn.to/3Kw55jK) - Does not require E addressable pad
|
||||||
- [Waveshare 96×48](https://amzn.to/4bydNcv) – higher resolution, requires soldering the **E addressable pad** on the [Adafruit RGB Bonnet](https://www.adafruit.com/product/3211) to “8” **OR** toggling the DIP switch on the Adafruit Triple LED Matrix Bonnet *(no soldering required!)*
|
- [Waveshare 96×48](https://amzn.to/4bydNcv) – higher resolution, requires soldering the **E addressable pad** on the [Adafruit RGB Bonnet](https://www.adafruit.com/product/3211) to “8” **OR** toggling the DIP switch on the Adafruit Triple LED Matrix Bonnet *(no soldering required!)*
|
||||||
> Amazon Affiliate Link – ChuckBuilds receives a small commission on purchases
|
- There are some Panels on Aliexpress that have worked fine for me, shop around! I think Adafruit is probably the "safest" but they do have some limitation on resolution and layout.
|
||||||
|
> Amazon Affiliate Links – ChuckBuilds receives a small commission on purchases
|
||||||
|
|
||||||
### Power Supply
|
### Power Supply
|
||||||
- [5V 4A DC Power Supply](https://www.adafruit.com/product/658) (good for 2 -3 displays, depending on brightness and pixel density, you'll need higher amperage for more)
|
- [5V 4A DC Power Supply](https://www.adafruit.com/product/658) (good for 2 -3 displays, depending on brightness and pixel density, you'll need higher amperage for more)
|
||||||
@@ -600,6 +602,14 @@ These settings are typically only needed for non-standard panels or custom confi
|
|||||||
- Leave empty unless you need custom mapping
|
- Leave empty unless you need custom mapping
|
||||||
- See rpi-rgb-led-matrix documentation for full options
|
- See rpi-rgb-led-matrix documentation for full options
|
||||||
|
|
||||||
|
- **`orientation`** (string, default: "normal")
|
||||||
|
- Rotates the rendered image to match how the panel is physically mounted
|
||||||
|
- Set to `"180"` (or use the "Upside Down" option in the web UI's Display
|
||||||
|
settings) if the panel is mounted upside down — useful for optimizing
|
||||||
|
where the Raspberry Pi and wiring sit relative to the mounting location
|
||||||
|
- Applied independently of `pixel_mapper_config` (appended as a trailing
|
||||||
|
`Rotate:180` mapper), so custom mapper configs keep working alongside it
|
||||||
|
|
||||||
- **`row_address_type`** (integer, default: 0)
|
- **`row_address_type`** (integer, default: 0)
|
||||||
- How rows are addressed on the panel
|
- How rows are addressed on the panel
|
||||||
- Most panels use 0 (direct addressing)
|
- Most panels use 0 (direct addressing)
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 467 B |
Binary file not shown.
|
Before Width: | Height: | Size: 39 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 105 KiB |
@@ -112,6 +112,7 @@
|
|||||||
"led_rgb_sequence": "RGB",
|
"led_rgb_sequence": "RGB",
|
||||||
"limit_refresh_rate_hz": 100,
|
"limit_refresh_rate_hz": 100,
|
||||||
"pixel_mapper_config": "",
|
"pixel_mapper_config": "",
|
||||||
|
"orientation": "normal",
|
||||||
"row_address_type": 0,
|
"row_address_type": 0,
|
||||||
"multiplexing": 0,
|
"multiplexing": 0,
|
||||||
"panel_type": ""
|
"panel_type": ""
|
||||||
@@ -129,6 +130,9 @@
|
|||||||
"plugin_rotation_order": [],
|
"plugin_rotation_order": [],
|
||||||
"use_short_date_format": true,
|
"use_short_date_format": true,
|
||||||
"vegas_scroll": {
|
"vegas_scroll": {
|
||||||
|
"live_in_ticker": false,
|
||||||
|
"live_weight": 3,
|
||||||
|
"favorite_live_weight": 5,
|
||||||
"enabled": false,
|
"enabled": false,
|
||||||
"scroll_speed": 50,
|
"scroll_speed": 50,
|
||||||
"separator_width": 32,
|
"separator_width": 32,
|
||||||
@@ -149,7 +153,7 @@
|
|||||||
"min_plugin_width": 8,
|
"min_plugin_width": 8,
|
||||||
"lead_in_width": 0,
|
"lead_in_width": 0,
|
||||||
"plugins_per_cycle": 6,
|
"plugins_per_cycle": 6,
|
||||||
"max_plugin_width_ratio": 3.0,
|
"max_plugin_width_ratio": 0.0,
|
||||||
"overflow_mode": "rotate",
|
"overflow_mode": "rotate",
|
||||||
"dynamic_duration_enabled": true,
|
"dynamic_duration_enabled": true,
|
||||||
"min_cycle_duration": 60,
|
"min_cycle_duration": 60,
|
||||||
|
|||||||
@@ -64,10 +64,98 @@ JSON is optional.
|
|||||||
| `target_fps` | `125` | Target frame rate |
|
| `target_fps` | `125` | Target frame rate |
|
||||||
| `buffer_ahead` | `2` | Number of plugins buffered ahead |
|
| `buffer_ahead` | `2` | Number of plugins buffered ahead |
|
||||||
|
|
||||||
This table is a subset — `display.vegas_scroll` supports 26 keys in
|
This table is a subset — `display.vegas_scroll` supports 30 keys in
|
||||||
total. See the full list in
|
total. See the full list in
|
||||||
[CONFIG_REFERENCE.md](CONFIG_REFERENCE.md#displayvegas_scroll--continuous-scroll-mode).
|
[CONFIG_REFERENCE.md](CONFIG_REFERENCE.md#displayvegas_scroll--continuous-scroll-mode).
|
||||||
|
|
||||||
|
### Live Content in the Ticker
|
||||||
|
|
||||||
|
By default, live content **preempts** Vegas mode: while any plugin reports
|
||||||
|
live priority, the display controller refuses to run the ticker and shows
|
||||||
|
that plugin's full-screen display instead. You get a big readable scoreboard,
|
||||||
|
but the marquee stops entirely for the duration of the game.
|
||||||
|
|
||||||
|
Set `live_in_ticker` to keep the ticker running and let live content take
|
||||||
|
**extra turns inside it** instead:
|
||||||
|
|
||||||
|
```json
|
||||||
|
"vegas_scroll": {
|
||||||
|
"live_in_ticker": true,
|
||||||
|
"live_weight": 3,
|
||||||
|
"favorite_live_weight": 5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Why weights exist
|
||||||
|
|
||||||
|
The rotation is otherwise a strict round robin — every plugin appears exactly
|
||||||
|
once per cycle. With a dozen plugins enabled, a live score comes round once a
|
||||||
|
lap and can be minutes old by the time you see it. A weight of *N* gives a
|
||||||
|
plugin *N* slots per cycle.
|
||||||
|
|
||||||
|
The slots are placed by **Smooth Weighted Round-Robin**, the same scheduler
|
||||||
|
the sports plugins use internally to rotate their own games. The important
|
||||||
|
property is that repeats are *spread through the cycle* rather than clumped:
|
||||||
|
three appearances in a row followed by a long silence would be worse than not
|
||||||
|
boosting at all.
|
||||||
|
|
||||||
|
Twelve plugins, with a favorite's baseball game and an ordinary live hockey
|
||||||
|
game (`live_weight: 3`, `favorite_live_weight: 5`):
|
||||||
|
|
||||||
|
```
|
||||||
|
baseball > hockey > weather > clock > baseball
|
||||||
|
stocks > news > flights > baseball > hockey
|
||||||
|
calendar > f1 > music > baseball > tides
|
||||||
|
birds > hockey > baseball
|
||||||
|
```
|
||||||
|
|
||||||
|
18 slots for 12 plugins. Baseball appears 5 times, hockey 3, everything else
|
||||||
|
once, and no plugin ever appears twice in a row — **including across the seam**
|
||||||
|
where the cycle loops back on itself. Smooth Weighted Round-Robin schedules the
|
||||||
|
heaviest item first and usually last as well, so the strip would otherwise show
|
||||||
|
it twice running at exactly the one join a within-cycle check cannot see. The
|
||||||
|
trailing repeat is moved into the widest remaining gap. Where a double is
|
||||||
|
unavoidable — a plugin holding most of the slots has to neighbour itself — the
|
||||||
|
schedule is left as it is.
|
||||||
|
|
||||||
|
#### Where the weight comes from
|
||||||
|
|
||||||
|
For each plugin in the rotation, in order:
|
||||||
|
|
||||||
|
1. **The plugin's own answer.** If it implements
|
||||||
|
`get_vegas_priority_weight()` and returns a number, that wins. This is the
|
||||||
|
only route for favorite-team awareness — the core can see *that* a game is
|
||||||
|
live, but not *whose*, so a scoreboard has to say so itself.
|
||||||
|
2. **The core's default.** When the plugin returns `None` (the base-class
|
||||||
|
default), a plugin where both `has_live_priority()` and `has_live_content()`
|
||||||
|
are true gets `live_weight`.
|
||||||
|
3. **Everything else** gets 1.
|
||||||
|
|
||||||
|
Because of step 2, **existing plugins need no changes** — any scoreboard with
|
||||||
|
`live_priority` enabled already gets extra turns. Step 1 is opt-in, for
|
||||||
|
plugins that want to distinguish a favorite's game from any other live game.
|
||||||
|
|
||||||
|
Weights are clamped to 1–10. A weight of 1 is no boost; a weight below 1 would
|
||||||
|
drop the plugin from the rotation entirely, which is never what is meant.
|
||||||
|
|
||||||
|
#### Things worth knowing
|
||||||
|
|
||||||
|
- **Weights are per plugin, not per game.** A scoreboard showing four live
|
||||||
|
games still occupies one slot at a time, rotating its own games within that
|
||||||
|
slot using its own `favorite_live_boost`. This controls how often the
|
||||||
|
*plugin* comes round.
|
||||||
|
- **The ticker is zero-sum.** Giving baseball 5 slots does not make the cycle
|
||||||
|
faster; it makes the cycle *longer* and everything else proportionally
|
||||||
|
rarer. If you want live scores sooner in wall-clock terms, pair this with a
|
||||||
|
smaller `plugins_per_cycle`.
|
||||||
|
- **Frequency is not freshness.** Each appearance redraws from the plugin's
|
||||||
|
current data (`refresh_updated_plugins()` drops cached content when a
|
||||||
|
plugin's data changes), but how current that data is depends on the
|
||||||
|
plugin's own `live_update_interval`. Showing a stale score five times a lap
|
||||||
|
is no better than showing it once.
|
||||||
|
- **Everything still appears.** A boost never starves another plugin out of
|
||||||
|
the cycle; low-weight plugins keep their single slot.
|
||||||
|
|
||||||
### Per-Plugin Configuration
|
### Per-Plugin Configuration
|
||||||
|
|
||||||
Override Vegas behavior for specific plugins:
|
Override Vegas behavior for specific plugins:
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ tooling against it.
|
|||||||
| `web_display_autostart` | bool, `true` | Whether the web interface service starts with the system | `scripts/utils/start_web_conditionally.py` |
|
| `web_display_autostart` | bool, `true` | Whether the web interface service starts with the system | `scripts/utils/start_web_conditionally.py` |
|
||||||
| `timezone` | string, `"America/New_York"` | IANA timezone for schedules and displays | `ConfigManager.get_timezone()` |
|
| `timezone` | string, `"America/New_York"` | IANA timezone for schedules and displays | `ConfigManager.get_timezone()` |
|
||||||
| `target_fps` | int, `100` | Frame-rate ceiling for plugin rendering | `src/plugin_system/base_plugin.py`, `src/common/sports_scroll.py` |
|
| `target_fps` | int, `100` | Frame-rate ceiling for plugin rendering | `src/plugin_system/base_plugin.py`, `src/common/sports_scroll.py` |
|
||||||
| `location` | object | `city` / `state` / `country`, offered to plugins that need a location (weather, etc.) | plugins via merged config |
|
| `location` | object | `city` / `state` / `country`. Supplies the **default** for a plugin's own `location_city` / `location_state` / `location_country` setting, so weather, radar and friends follow this device without being configured twice. A value saved on the plugin itself still overrides it. | `SchemaManager.apply_device_location()`, then plugins via merged config |
|
||||||
|
|
||||||
## `schedule` — display on/off hours
|
## `schedule` — display on/off hours
|
||||||
|
|
||||||
@@ -66,6 +66,7 @@ in `DisplayManager` (`src/display_manager.py`, ~lines 270–295).
|
|||||||
| `led_rgb_sequence` | string, `"RGB"` |
|
| `led_rgb_sequence` | string, `"RGB"` |
|
||||||
| `limit_refresh_rate_hz` | int, `100` (code default 90) |
|
| `limit_refresh_rate_hz` | int, `100` (code default 90) |
|
||||||
| `pixel_mapper_config` | string, `""` — e.g. `"U-mapper"` / `"Rotate:90"` |
|
| `pixel_mapper_config` | string, `""` — e.g. `"U-mapper"` / `"Rotate:90"` |
|
||||||
|
| `orientation` | string, `"normal"` — `"180"` rotates the rendered image 180° for panels physically mounted upside down (e.g. to move the Pi/wiring to a more convenient side); composed onto `pixel_mapper_config` as a trailing `Rotate:180` mapper, so it stays independent of any custom `pixel_mapper_config` value |
|
||||||
| `row_address_type` | int, `0` — non-standard panel row addressing |
|
| `row_address_type` | int, `0` — non-standard panel row addressing |
|
||||||
| `multiplexing` | int, `0` — panel multiplexing scheme |
|
| `multiplexing` | int, `0` — panel multiplexing scheme |
|
||||||
| `panel_type` | string, `""` — set to `"FM6126A"` or `"FM6127"` for panels needing init |
|
| `panel_type` | string, `""` — set to `"FM6126A"` or `"FM6127"` for panels needing init |
|
||||||
@@ -103,7 +104,8 @@ logical image to multiple chained physical panels.
|
|||||||
## `display.vegas_scroll` — continuous scroll mode
|
## `display.vegas_scroll` — continuous scroll mode
|
||||||
|
|
||||||
Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See
|
Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See
|
||||||
[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for behavior details.
|
[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for behavior details, including
|
||||||
|
[live content in the ticker](ADVANCED_FEATURES.md#live-content-in-the-ticker).
|
||||||
|
|
||||||
| Key | Type / default |
|
| Key | Type / default |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -127,13 +129,16 @@ Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See
|
|||||||
| `min_plugin_width` | int, `8` |
|
| `min_plugin_width` | int, `8` |
|
||||||
| `lead_in_width` | int, `0` |
|
| `lead_in_width` | int, `0` |
|
||||||
| `plugins_per_cycle` | int, `6` |
|
| `plugins_per_cycle` | int, `6` |
|
||||||
| `max_plugin_width_ratio` | float, `3.0` |
|
| `max_plugin_width_ratio` | float, `0.0` |
|
||||||
| `overflow_mode` | string, `"rotate"` |
|
| `overflow_mode` | string, `"rotate"` |
|
||||||
| `dynamic_duration_enabled` | bool, `true` |
|
| `dynamic_duration_enabled` | bool, `true` |
|
||||||
| `min_cycle_duration` | int, `60` |
|
| `min_cycle_duration` | int, `60` |
|
||||||
| `max_cycle_duration` | int, `240` |
|
| `max_cycle_duration` | int, `240` |
|
||||||
| `frame_based_scrolling` | bool, `true` — frame-count-based scroll stepping |
|
| `frame_based_scrolling` | bool, `true` — frame-count-based scroll stepping |
|
||||||
| `scroll_delay` | float, `0.02` — seconds between scroll updates (~50 FPS) |
|
| `scroll_delay` | float, `0.02` — seconds between scroll updates (~50 FPS) |
|
||||||
|
| `live_in_ticker` | bool, `false` — keep scrolling during live games instead of handing the display to a full-screen scoreboard |
|
||||||
|
| `live_weight` | int, `3` (1–10) — slots per cycle for a plugin with live content |
|
||||||
|
| `favorite_live_weight` | int, `5` (1–10) — slots per cycle when a plugin reports a favorite team is live |
|
||||||
|
|
||||||
## `sync` — multi-display synchronization
|
## `sync` — multi-display synchronization
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,115 @@
|
|||||||
|
# Running on Low-Memory Boards
|
||||||
|
|
||||||
|
Applies to the Pi Zero 2 W (512 MB), Pi 3 / 3B+ (1 GB), and the 1 GB Pi 4.
|
||||||
|
If your board has 2 GB or more you can skip this document.
|
||||||
|
|
||||||
|
## The failure this prevents
|
||||||
|
|
||||||
|
The display process is the largest thing on the board. On a 1 GB Pi 3B+ with
|
||||||
|
around 20 plugins enabled it settles near **600 MB of 905 MB usable**, leaving
|
||||||
|
under 200 MB of headroom for everything else.
|
||||||
|
|
||||||
|
When that headroom runs out, the board does not crash cleanly. `fork()` starts
|
||||||
|
failing, and because a new process is needed to do almost anything, the
|
||||||
|
symptoms look nothing like "out of memory":
|
||||||
|
|
||||||
|
| What you see | Why |
|
||||||
|
|---|---|
|
||||||
|
| SSH accepts the connection then closes it instantly, before any banner | `sshd` forks a session per connection; the fork fails |
|
||||||
|
| The web UI still responds quickly | Already running, serves from existing threads, forks nothing |
|
||||||
|
| Ping is perfect, 0% loss | Handled entirely in the kernel |
|
||||||
|
| The panel is dark | The display process was killed and cannot be respawned |
|
||||||
|
| The clock is wrong after the next boot | `fake-hwclock`'s periodic save is a scheduled job, and it cannot fork either |
|
||||||
|
|
||||||
|
The board looks healthy from the outside and cannot be logged into. Only a
|
||||||
|
power cycle clears it. If you are here because SSH stopped working, also see
|
||||||
|
[SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md), which
|
||||||
|
covers the more common cause (AP mode).
|
||||||
|
|
||||||
|
## Check your headroom
|
||||||
|
|
||||||
|
```bash
|
||||||
|
free -m
|
||||||
|
ps -eo rss,comm --sort=-rss | head -5
|
||||||
|
```
|
||||||
|
|
||||||
|
If `MemAvailable` is under ~150 MB while the display is running, you are close
|
||||||
|
to the edge. To watch it over time:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
watch -n 30 'free -m | head -2'
|
||||||
|
```
|
||||||
|
|
||||||
|
Available memory that falls steadily rather than holding flat means you will
|
||||||
|
reach the wall; it is a question of when.
|
||||||
|
|
||||||
|
## What to do
|
||||||
|
|
||||||
|
**1. Enable the memory cgroup controller.** Without it, the `MemoryMax=85%` in
|
||||||
|
`systemd/ledmatrix.service` is accepted by systemd and silently ignored, so the
|
||||||
|
service has no ceiling and a runaway takes the whole board down instead of just
|
||||||
|
restarting. Raspberry Pi firmware disables this controller by default.
|
||||||
|
|
||||||
|
`first_time_install.sh` does this for you. To check it took effect:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep memory /sys/fs/cgroup/cgroup.controllers
|
||||||
|
```
|
||||||
|
|
||||||
|
If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the
|
||||||
|
kernel command line and reboot. Edit whichever file your image uses —
|
||||||
|
`/boot/firmware/cmdline.txt` on current Raspberry Pi OS, `/boot/cmdline.txt` on
|
||||||
|
older layouts (the installer checks the first and falls back to the second).
|
||||||
|
Everything must stay on a single line.
|
||||||
|
|
||||||
|
This changes the failure mode from "the board becomes unreachable" to "the
|
||||||
|
display service restarts". It is a safety net, not a fix.
|
||||||
|
|
||||||
|
**2. Run fewer plugins.** This is the actual remedy. Every enabled plugin costs
|
||||||
|
memory permanently — its module, its parsed config, and its cached API
|
||||||
|
responses. On a 512 MB or 1 GB board, keep the enabled set small and prefer
|
||||||
|
plugins that poll infrequently.
|
||||||
|
|
||||||
|
**3. Lower the cache ceiling.** The in-memory cache is sized from total RAM
|
||||||
|
(150 entries at 1 GB and below, up to 1500 at 8 GB). To go lower still:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
# /etc/systemd/system/ledmatrix.service.d/override.conf
|
||||||
|
[Service]
|
||||||
|
Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75
|
||||||
|
```
|
||||||
|
|
||||||
|
Writing the file does not change the running service. Reload systemd and
|
||||||
|
restart it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl restart ledmatrix
|
||||||
|
```
|
||||||
|
|
||||||
|
Fewer entries means more API calls, so lower this only while you are actually
|
||||||
|
short of memory.
|
||||||
|
|
||||||
|
**4. Consider `MemoryHigh`.** `MemoryMax` kills and restarts. `MemoryHigh`
|
||||||
|
throttles and reclaims instead, which is gentler — but on a board where the
|
||||||
|
process genuinely wants more than the limit, sustained reclaim can stall the
|
||||||
|
render loop and show as visible stutter on the panel. Add it only if you prefer
|
||||||
|
degraded output to a restart:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Service]
|
||||||
|
MemoryHigh=70%
|
||||||
|
```
|
||||||
|
|
||||||
|
## Keep your logs
|
||||||
|
|
||||||
|
These images default to volatile journald storage, so every reboot destroys the
|
||||||
|
logs — including the ones explaining why the board rebooted. `first_time_install.sh`
|
||||||
|
enables persistent storage capped at 64 MB. To confirm:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
journalctl --list-boots
|
||||||
|
```
|
||||||
|
|
||||||
|
More than one boot listed means logs are surviving reboots. If only one is
|
||||||
|
listed, journald is still writing to `/run` (tmpfs).
|
||||||
@@ -170,6 +170,47 @@ Default returns `False`.
|
|||||||
List of display modes to show during a live takeover. Default returns the
|
List of display modes to show during a live takeover. Default returns the
|
||||||
plugin's `display_modes` from its manifest.
|
plugin's `display_modes` from its manifest.
|
||||||
|
|
||||||
|
#### `get_vegas_priority_weight() -> Optional[int]`
|
||||||
|
|
||||||
|
How many slots per Vegas cycle this plugin should get. Default returns
|
||||||
|
`None`, which defers to the core.
|
||||||
|
|
||||||
|
The Vegas ticker is otherwise a strict round robin — every plugin appears
|
||||||
|
exactly once per cycle — so with a dozen plugins enabled a live score can be
|
||||||
|
minutes stale by the time it comes round. A weight of *N* gives the plugin
|
||||||
|
*N* slots per cycle, spread evenly through it rather than clumped.
|
||||||
|
|
||||||
|
**You usually do not need this.** When the hook returns `None`, the core
|
||||||
|
already gives a plugin `vegas_scroll.live_weight` whenever
|
||||||
|
`has_live_priority()` and `has_live_content()` are both true. Live sports get
|
||||||
|
extra turns with no code at all.
|
||||||
|
|
||||||
|
Implement it only when the plugin knows something the core cannot. The
|
||||||
|
motivating case is favorite teams — the core can see *that* a game is live,
|
||||||
|
but not *whose*:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_vegas_priority_weight(self):
|
||||||
|
if not (self.has_live_priority() and self.has_live_content()):
|
||||||
|
return None # let the core decide
|
||||||
|
vegas = self.global_config.get('display', {}).get('vegas_scroll', {})
|
||||||
|
if self._favorite_is_live():
|
||||||
|
return vegas.get('favorite_live_weight', 5)
|
||||||
|
return vegas.get('live_weight', 3)
|
||||||
|
```
|
||||||
|
|
||||||
|
The weight is per *plugin*, not per game: a scoreboard showing four live games
|
||||||
|
still occupies one slot at a time and rotates its own games within it. Values
|
||||||
|
are clamped to 1–10 by the caller. An exception here is caught and logged, and
|
||||||
|
the core then falls back to its own live-content check — so a plugin whose
|
||||||
|
weight calculation is broken still gets `live_weight` for a game that really
|
||||||
|
is live, rather than being demoted to 1.
|
||||||
|
|
||||||
|
Only consulted when the user has set `vegas_scroll.live_in_ticker`. With the
|
||||||
|
default (`false`) live content preempts Vegas entirely and there is no ticker
|
||||||
|
to be weighted within. See
|
||||||
|
[ADVANCED_FEATURES.md](ADVANCED_FEATURES.md#live-content-in-the-ticker).
|
||||||
|
|
||||||
### Vegas scroll hooks
|
### Vegas scroll hooks
|
||||||
|
|
||||||
Vegas mode shows multiple plugins as a single continuous scroll instead of
|
Vegas mode shows multiple plugins as a single continuous scroll instead of
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ the one-shot installer. The pages here go deeper.
|
|||||||
5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes
|
5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes
|
||||||
6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install
|
6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install
|
||||||
7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems
|
7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems
|
||||||
|
8. [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) — Pi Zero 2 W / 3B+ / 1GB Pi 4 memory limits
|
||||||
|
|
||||||
## I want to write a plugin
|
## I want to write a plugin
|
||||||
|
|
||||||
|
|||||||
+169
-37
@@ -27,7 +27,7 @@ These are independent concerns. Conflating them is what produces god classes.
|
|||||||
| Plugin loads on a core that predates a module | Guarded import with a bundled fallback (`try: from src.X import Y / except ModuleNotFoundError: from y import Y`) |
|
| Plugin loads on a core that predates a module | Guarded import with a bundled fallback (`try: from src.X import Y / except ModuleNotFoundError: from y import Y`) |
|
||||||
| Plugin loads on a core that predates a *method* | Capability probing — `hasattr(SportsCore, "_detect_stale_games")` — never a version comparison. The loader's compat check is advisory-only (it logs and continues), so probing is the real protection. |
|
| Plugin loads on a core that predates a *method* | Capability probing — `hasattr(SportsCore, "_detect_stale_games")` — never a version comparison. The loader's compat check is advisory-only (it logs and continues), so probing is the real protection. |
|
||||||
| Core changes never break a plugin's rendering | The **view-model contract**: `_extract_game_details_common` returns a dict whose `GUARANTEED_KEYS` are frozen by `test/test_skin_system.py::TestViewModelContract`. Keys may be added, never renamed or removed. |
|
| Core changes never break a plugin's rendering | The **view-model contract**: `_extract_game_details_common` returns a dict whose `GUARANTEED_KEYS` are frozen by `test/test_skin_system.py::TestViewModelContract`. Keys may be added, never renamed or removed. |
|
||||||
| A plugin can drop its bundled copy safely | The **sunset rule**: its manifest must floor `ledmatrix_min_version` at the first core release shipping the module (recorded in `CHANGELOG.md`) — *necessary but not sufficient*. Nothing enforces that floor today, so the copy also waits for the B6 gate below. |
|
| A plugin can drop its bundled copy safely | The **sunset rule**: its manifest must floor `ledmatrix_min_version` at the first core release shipping the module (recorded in `CHANGELOG.md`) — *necessary but not sufficient*. The store enforces that floor on every registry-managed install and on both supported update paths (sideloading via `install_from_url` is not gated), but a floor cannot reach a user who never updates, so the copy also waits for the B6 gate below. |
|
||||||
|
|
||||||
The core API is **additive-only**. A method the plugins call is never removed or
|
The core API is **additive-only**. A method the plugins call is never removed or
|
||||||
given a new required parameter; new behavior arrives as new methods with
|
given a new required parameter; new behavior arrives as new methods with
|
||||||
@@ -204,7 +204,7 @@ legacy compatibility rather than the mechanism.
|
|||||||
B0–B3 are merged and shipping in core 3.2.0. Everything that remains is
|
B0–B3 are merged and shipping in core 3.2.0. Everything that remains is
|
||||||
**rollout**, and it splits into three phases with very different risk profiles.
|
**rollout**, and it splits into three phases with very different risk profiles.
|
||||||
The original plan folded the last two together; they are separated here because
|
The original plan folded the last two together; they are separated here because
|
||||||
one of them is safe by construction and the other is not.
|
one of them cannot break a user on an old core and the other can.
|
||||||
|
|
||||||
| Phase | Scope | Status | Gate |
|
| Phase | Scope | Status | Gate |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
@@ -212,9 +212,9 @@ one of them is safe by construction and the other is not.
|
|||||||
| **B1** | Promote the nine universal methods; convert `sports.py` → package | ✅ | Characterization suite green; no behavior change intended |
|
| **B1** | Promote the nine universal methods; convert `sports.py` → package | ✅ | Characterization suite green; no behavior change intended |
|
||||||
| **B2** | `CelebrationMixin` + rotation strategies as opt-in capabilities | ✅ | Non-adopters have zero new code in their MRO; strategies checked against verbatim plugin transcriptions |
|
| **B2** | `CelebrationMixin` + rotation strategies as opt-in capabilities | ✅ | Non-adopters have zero new code in their MRO; strategies checked against verbatim plugin transcriptions |
|
||||||
| **B3** | Upstream the scroll **orchestration** layer as `src/common/sports_scroll.py`, reading `global_config['target_fps']` natively | ✅ | Content building stays per-sport |
|
| **B3** | Upstream the scroll **orchestration** layer as `src/common/sports_scroll.py`, reading `global_config['target_fps']` natively | ✅ | Content building stays per-sport |
|
||||||
| **B4** | Ship 3.2.0 *and* make version reporting trustworthy | ⏳ **next** | Tag, release, and `src.__version__` agree; compatibility gate merged |
|
| **B4** | Ship 3.2.0 *and* make version reporting trustworthy | ✅ | Released 2026-08-03; tag, release and `src.__version__` agree; compatibility gate merged (#428, #431, #433) |
|
||||||
| **B5** | Adoption — guarded core imports: three pilots, then the remaining six. **Bundled copies stay.** | after B4 | Per plugin: harness + goldens byte-identical, then a device soak |
|
| **B5** | Adoption — guarded core imports, all eight. **Bundled copies stay.** | ✅ | All eight adopted; harness byte-identical; see the B5 retrospective below — four shipped broken and were repaired in plugins #251 |
|
||||||
| **B6** | Sunset — delete the bundled copies | **blocked** | B4's gate shipped *and* in users' hands (see below) |
|
| **B6** | Sunset — delete the bundled copies | ✅ | Ran 2026-09-01, all eight. Floors at 3.2.0; the store refuses on all three routes in (#431/#433, #508, #510). See "B6 — what actually happened" |
|
||||||
|
|
||||||
### B4 — what "ship 3.2.0" actually requires
|
### B4 — what "ship 3.2.0" actually requires
|
||||||
|
|
||||||
@@ -234,6 +234,20 @@ a floor can be trusted against, and today it is not:
|
|||||||
compares the plugin's manifest version against the registry's
|
compares the plugin's manifest version against the registry's
|
||||||
`latest_version` and nothing else.
|
`latest_version` and nothing else.
|
||||||
|
|
||||||
|
*Fixed, in two parts.* `install_plugin` gained the gate in #431/#433, which
|
||||||
|
covers every registry-managed install and, through `_reinstall_with_rollback`,
|
||||||
|
the update path that re-downloads.
|
||||||
|
`update_plugin`'s git branch pulls in place and re-downloads nothing, so it
|
||||||
|
stayed ungated until `_gate_pulled_commit` closed it — checked after the pull
|
||||||
|
(the registry carries no floor field, so the incoming floor is unknowable
|
||||||
|
before it) and undone with `git reset --hard` to the pre-pull commit. That
|
||||||
|
route is rare in practice, since monorepo plugins install as archives; it was
|
||||||
|
closed because the sunset rule in the plugins repo's
|
||||||
|
`08-shared-sports-code.md` states as **condition 3** that the core enforces
|
||||||
|
the floor "at install/update time", and B6 rests on that being true rather
|
||||||
|
than merely written down. `install_from_url` — sideloading a plugin from a
|
||||||
|
URL — is still ungated.
|
||||||
|
|
||||||
So B4 is: tag and release 3.2.0; make the tag, the release, and `__version__`
|
So B4 is: tag and release 3.2.0; make the tag, the release, and `__version__`
|
||||||
agree, and keep them agreeing; reconsider the `< 2.0.0` skip; migrate manifests
|
agree, and keep them agreeing; reconsider the `< 2.0.0` skip; migrate manifests
|
||||||
from `ledmatrix_min` to `ledmatrix_min_version`; and add the install/update
|
from `ledmatrix_min` to `ledmatrix_min_version`; and add the install/update
|
||||||
@@ -262,13 +276,21 @@ default is the more restrictive.
|
|||||||
`compatible_versions`. No manifest still carries it, so there is nothing to
|
`compatible_versions`. No manifest still carries it, so there is nothing to
|
||||||
migrate there.)
|
migrate there.)
|
||||||
|
|
||||||
### B5 — adoption is safe by construction
|
### B5 — the *fallback* is safe by construction; the modern path is not
|
||||||
|
|
||||||
|
The heading matters, because the unqualified version of this claim is false and
|
||||||
|
this document proves it two sections down: four of the eight adopted plugins
|
||||||
|
shipped with scroll mode broken on a 3.2.0 core. What is safe by construction is
|
||||||
|
narrower than "adoption".
|
||||||
|
|
||||||
A plugin adopting core imports keeps its bundled copy and reaches it through the
|
A plugin adopting core imports keeps its bundled copy and reaches it through the
|
||||||
guarded import (see the Upgradability table above). On a core that ships the
|
guarded import (see the Upgradability table above). On a core that doesn't ship
|
||||||
module the plugin uses core code; on one that doesn't it falls back and behaves
|
the module the plugin falls back and behaves exactly as it does today. That
|
||||||
exactly as it does today. There is no version of this step that breaks a user,
|
fallback compatibility — and only that — is safe by construction. On a core that
|
||||||
which is why it does not wait for B6's gate.
|
*does* ship the module, correctness is not automatic — object-level and scroll-mode validation
|
||||||
|
(building both classes and comparing, per the retrospective below) is required
|
||||||
|
to prove full behavior. There is no version of this step that breaks a user *on
|
||||||
|
an old core*, which is why it does not wait for B6's gate.
|
||||||
|
|
||||||
The hockey scroll-display pilot is **already validated**: adopted against a core
|
The hockey scroll-display pilot is **already validated**: adopted against a core
|
||||||
carrying 3.2.0, `scroll_display.py` went from 691 to 289 lines and all 16 harness
|
carrying 3.2.0, `scroll_display.py` went from 691 to 289 lines and all 16 harness
|
||||||
@@ -297,7 +319,8 @@ been in users' hands long enough that the population running a core without it
|
|||||||
is small.** The bundled copies cost disk space; deleting them early costs
|
is small.** The bundled copies cost disk space; deleting them early costs
|
||||||
scoreboards, silently. That trade is not close.
|
scoreboards, silently. That trade is not close.
|
||||||
|
|
||||||
Before the first sunset, add a **compatibility regression test**. It has to
|
Before the first sunset, add a **compatibility regression test**. **Built:**
|
||||||
|
core `test/test_sports_sunset_matrix.py` (#505). It has to
|
||||||
cover four cases, not one — B5's safety claim and B6's failure mode are
|
cover four cases, not one — B5's safety claim and B6's failure mode are
|
||||||
different propositions and only the second is obvious:
|
different propositions and only the second is obvious:
|
||||||
|
|
||||||
@@ -321,35 +344,144 @@ gate rather than trusting the failure to be noticed.
|
|||||||
The same suite should exercise the install/update gate, since it is the other
|
The same suite should exercise the install/update gate, since it is the other
|
||||||
half of the guarantee.
|
half of the guarantee.
|
||||||
|
|
||||||
|
### B6 — what actually happened
|
||||||
|
|
||||||
|
**Ran 2026-09-01, across all eight scoreboards.** Held from 2026-08-05 to
|
||||||
|
2026-09-01 on the argument below, which is kept because the reasoning applies to
|
||||||
|
the next module, not because it is still in force.
|
||||||
|
|
||||||
|
**The hold, and why it lifted.** The stated gate was evidence of 3.2.0 uptake —
|
||||||
|
"a few months of it being the default download, or store-side install data".
|
||||||
|
That evidence never arrived and could not: the core updates by
|
||||||
|
`git pull --rebase`, so release-asset counts cannot measure uptake, and no
|
||||||
|
store-side telemetry exists. What changed instead is that the *risk* the gate
|
||||||
|
protected against was closed directly. The store now refuses a plugin whose
|
||||||
|
floor exceeds the running core on **all three** routes in:
|
||||||
|
|
||||||
|
| route | gated by |
|
||||||
|
|---|---|
|
||||||
|
| `install_plugin` — every path that re-downloads, `_reinstall_with_rollback` included | #431, #433 |
|
||||||
|
| `update_plugin`'s git branch — pulls in place, re-downloads nothing | #508 |
|
||||||
|
| `install_from_url` — sideloading | #510 |
|
||||||
|
|
||||||
|
With all three closed a pre-3.2.0 user cannot receive a sunset plugin at all;
|
||||||
|
they keep the version they already run. The population the hold existed to
|
||||||
|
protect is protected by refusal rather than by a bundled copy — which is what
|
||||||
|
the copy was standing in for.
|
||||||
|
|
||||||
|
**What shipped.** Eight plugins, ~5,800 lines of frozen fallback deleted. Each:
|
||||||
|
copy removed, guarded import collapsed to a plain one, floor raised to 3.2.0,
|
||||||
|
`test_core_fallback.py` rewritten as `test_core_scroll.py` asserting the sunset
|
||||||
|
rather than the fallback. `scripts/check_scroll_adoption.py` gained
|
||||||
|
`sunset_violations` and a `SUNSET_PLUGINS` set naming all eight, so a
|
||||||
|
resurrected copy or a returned guard fails CI.
|
||||||
|
|
||||||
|
Delivered as plugins #346 (hockey, later folded into #351), #349 (football),
|
||||||
|
#350 (baseball), #351 (the remaining six).
|
||||||
|
|
||||||
|
**Two things found by doing it, both worth carrying forward:**
|
||||||
|
|
||||||
|
- **Only one fallback held orchestration logic the core lacked.** baseball's
|
||||||
|
`_configure_scroll_helper` reinterpreted `scroll_speed` as pixels-per-*frame*
|
||||||
|
when `speed × delay` fell outside the 0.1–5.0 window — measured, 10–20×
|
||||||
|
faster than configured for a speed between 1.0 and 5.0. Standardised onto the
|
||||||
|
core's behaviour (honour the documented px/sec, clamp) rather than preserved.
|
||||||
|
Every other difference across the eight was a docstring, an unreachable
|
||||||
|
`scroll_helper is None` guard, or an equivalent diagnostic.
|
||||||
|
- **Two tests had been leaning on the guard without anyone knowing.**
|
||||||
|
`soccer/test_live_screens.py` stubbed `src` in a way that shadowed the core,
|
||||||
|
so its guarded import fell back and the test had been exercising the *frozen
|
||||||
|
copy* rather than the shipping class since B5. Before sunsetting anything else
|
||||||
|
that carries a guarded core import, grep for tests that stub `src`.
|
||||||
|
|
||||||
|
**The floor-raising traps still apply** to any future sunset: four plugins
|
||||||
|
declare their floor top-level, where editing `versions[0]` is a silent no-op,
|
||||||
|
and the floor has three live spellings (`min_ledmatrix_version`,
|
||||||
|
`requires.min_ledmatrix_version`, `versions[].ledmatrix_min_version`, plus
|
||||||
|
deprecated `ledmatrix_min`). See
|
||||||
|
`src/plugin_system/compatibility.py:declared_min_version` for the resolution
|
||||||
|
order any floor-raising tool must reproduce — and note the name is **inverted**
|
||||||
|
between the top level and `versions[]`.
|
||||||
|
|
||||||
|
**Still not adopted, deliberately:** `data_sources.py`, `game_renderer.py` and
|
||||||
|
`base_odds_manager.py`. The standing decision held them until B6 closed; it now
|
||||||
|
has, so they can be reconsidered — with B5's lesson applied, which is to build
|
||||||
|
the object and diff rendered output rather than trust a static check.
|
||||||
|
|
||||||
|
### B5 retrospective — what the adoption actually cost
|
||||||
|
|
||||||
|
Recorded because it is the evidence behind the two decisions above, and because
|
||||||
|
"the adoption went fine" is not what happened.
|
||||||
|
|
||||||
|
**Four of the eight shipped with scroll mode broken** on a 3.2.0 core, and were
|
||||||
|
repaired in plugins-repo #251. The restructure lifted the content methods
|
||||||
|
verbatim but left the state they read off `self` behind: separator-icon
|
||||||
|
constants (hockey, basketball, lacrosse) and the game-renderer cache (afl).
|
||||||
|
hockey/basketball/lacrosse could not construct the scroll display at all; afl
|
||||||
|
raised inside `prepare_scroll_content`, which the core base *catches*, so its
|
||||||
|
only symptom was scroll mode silently drawing nothing.
|
||||||
|
|
||||||
|
Three things are worth carrying forward:
|
||||||
|
|
||||||
|
- **The bundled fallback did not protect anyone from this.** The break was on
|
||||||
|
the modern path, which the fallback never touches. Carrying the second copy
|
||||||
|
bought nothing against the actual defect while creating the divergence that
|
||||||
|
produced it. That is an argument *for* B6, not against it.
|
||||||
|
- **Every gate was green.** The safety harness renders the scoreboard screens,
|
||||||
|
not scroll mode; `test_core_fallback.py` checked that methods existed and that
|
||||||
|
their *globals* resolved, and `self.NHL_SEPARATOR_ICON` is an attribute read,
|
||||||
|
invisible to an AST scan for `Name` loads. The fix was to stop reasoning about
|
||||||
|
source and **build the object**: construct both classes on both paths, compare
|
||||||
|
the separator icons they end up with, and assert the adopted class ends up
|
||||||
|
with every instance attribute the bundled one sets.
|
||||||
|
- **Test what the change touches, not what is convenient to render.** Scroll
|
||||||
|
mode had no coverage because the harness could not reach it. A comparison
|
||||||
|
harness that renders the same games through both paths and diffs the pixels
|
||||||
|
needs no per-sport knowledge of the right answer, only that adopting core code
|
||||||
|
did not change it.
|
||||||
|
|
||||||
|
**The ledger.** Before adoption, eight duplicated copies totalled 5,685 lines.
|
||||||
|
After adoption plus the frozen legacy copies it was 10,610; removing the dead
|
||||||
|
inline duplication (plugins #252) brought it to roughly 8,620. B6 would take it
|
||||||
|
to about 3,300 including the shared core module — some 2,400 fewer than before
|
||||||
|
this project started. **Until B6 runs, the adoption is net negative on disk**,
|
||||||
|
and its one delivered user-visible gain is that adopted plugins honour the
|
||||||
|
global `target_fps` instead of hardcoding ~100 FPS.
|
||||||
|
|
||||||
|
### Decision: stop adopting further modules until B6 closes
|
||||||
|
|
||||||
|
`data_sources.py` (9 copies), `game_renderer.py` (8) and `base_odds_manager.py`
|
||||||
|
are the obvious next candidates. **Do not adopt them yet.** Each adoption adds
|
||||||
|
carrying cost — a second copy to keep in step — against a payoff that is
|
||||||
|
contingent on B6, and B6 is gated on an installed base we cannot currently
|
||||||
|
measure. Consolidate what is already committed; revisit when B6 does.
|
||||||
|
|
||||||
## What's next
|
## What's next
|
||||||
|
|
||||||
In order. Each step is independently useful and independently revertible.
|
Steps 1–5 of the original plan are **done**: 3.2.0 is tagged and published with
|
||||||
|
a version number CI now asserts (#428), the compatibility gate is in
|
||||||
|
`install_plugin` and reads `compatible_versions` as well as the floor
|
||||||
|
(#431, #433), the newest manifest entry is required to use `ledmatrix_min_version`
|
||||||
|
(plugins #244), and all eight plugins have adopted the scroll orchestration
|
||||||
|
(plugins #245–#249, repaired in #251, tidied in #252).
|
||||||
|
|
||||||
1. **Tag and publish v3.2.0.** The code is already on `main` (`21825cbf`).
|
What actually remains, smallest first:
|
||||||
Nothing else blocks this, and it is what makes `ledmatrix_min_version:
|
|
||||||
"3.2.0"` refer to something real.
|
1. **Soak the adoptions on hardware.** football and hockey have been run on a
|
||||||
2. **Make the version number honest.** Have the release process assert that the
|
live rig through real games; baseball was watched through one earlier. The
|
||||||
tag, the GitHub release, and `src.__version__` agree — a check in CI is
|
rest are proven by harness, unit tests and pixel comparison. Out-of-season
|
||||||
cheaper than the confusion of the last two releases. Then revisit the
|
sports cannot be soaked until their season starts. When you do, **check the
|
||||||
`< 2.0.0` skip in `_warn_if_incompatible`, which currently silences the
|
rig's `*_display_mode` first** — a board in `switch` mode will happily load a
|
||||||
warning for the users who most need it.
|
sunset plugin and tell you nothing about the scroll code the sunset changed.
|
||||||
3. **Add the compatibility gate** to `StoreManager.install_plugin` and
|
2. **Cut 3.3.0.** Not required by B6 — its floors are 3.2.0, which is released —
|
||||||
`.update_plugin`: refuse a plugin whose declared floor exceeds
|
but `calendar` 1.2.3 floors at 3.3.0 for the device-authorization endpoints
|
||||||
`src.__version__`, and surface the reason in the store UI rather than only
|
that landed after 3.2.0, so it is un-installable until the release exists.
|
||||||
the log. This is the single change that turns the floor from documentation
|
3. **Reconsider the held modules** (`data_sources.py`, `game_renderer.py`,
|
||||||
into a guarantee, and B6 depends on it.
|
`base_odds_manager.py`) now that the sunset has closed. `game_renderer.py` is
|
||||||
4. **Migrate the manifests** to `ledmatrix_min_version`, and reconcile them with
|
the largest single duplication left: ~11,500 lines across eight plugins, with
|
||||||
`compatible_versions` (see above — that field is the required, canonical one,
|
~36,500 more in the eight `sports.py`. Note that core already ships
|
||||||
and the gate does not read it yet). Currently 28 plugins spell the floor both
|
`src/base_classes/sports/` (~143KB, promoted in B1/B2) that **no plugin
|
||||||
ways across their `versions[]` entries, 12 use only the old spelling, and 2
|
imports** — check whether it has drifted before treating it as the target.
|
||||||
only the new. Scope the sweep to the nine sports plugins if a 42-plugin
|
|
||||||
version-bump wave isn't worth it — but the `compatible_versions` half has to
|
|
||||||
cover every manifest the gate can refuse, or define explicit legacy handling,
|
|
||||||
before the gate is allowed to block anything.
|
|
||||||
5. **Run B5 adoption** — hockey, soccer, football, then the remaining six.
|
|
||||||
Bundled copies stay. Byte-identical harness output per plugin, then a soak.
|
|
||||||
6. **Only then plan B6**, with the compatibility regression test described above
|
|
||||||
in CI first.
|
|
||||||
|
|
||||||
## How to keep this project healthy
|
## How to keep this project healthy
|
||||||
|
|
||||||
|
|||||||
@@ -20,7 +20,22 @@ The installation script:
|
|||||||
- Installs and configures `dnsmasq` (DHCP server for AP mode)
|
- Installs and configures `dnsmasq` (DHCP server for AP mode)
|
||||||
- These services can interfere with normal WiFi client mode
|
- These services can interfere with normal WiFi client mode
|
||||||
|
|
||||||
### 3. Reboot After Installation
|
### 3. The Board Ran Out of Memory
|
||||||
|
|
||||||
|
On a 512MB or 1GB board, memory exhaustion stops `sshd` being able to fork a
|
||||||
|
session process. The connection is accepted and then closed immediately, before
|
||||||
|
any banner:
|
||||||
|
|
||||||
|
```text
|
||||||
|
kex_exchange_identification: Connection closed by remote host
|
||||||
|
```
|
||||||
|
|
||||||
|
The giveaway is that the board is otherwise healthy — ping is clean and the web
|
||||||
|
UI still responds — but nothing that needs to start a new process works, and
|
||||||
|
the panel is usually dark. Only a power cycle clears it. See
|
||||||
|
[LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md).
|
||||||
|
|
||||||
|
### 4. Reboot After Installation
|
||||||
|
|
||||||
If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode.
|
If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode.
|
||||||
|
|
||||||
@@ -190,11 +205,23 @@ The web interface allows you to:
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
**SSH becomes unavailable because**:
|
**SSH becomes unavailable because** — two unrelated causes, and they need
|
||||||
|
different responses:
|
||||||
|
|
||||||
|
*AP mode (most common):*
|
||||||
- WiFi monitor service enables AP mode when WiFi disconnects
|
- WiFi monitor service enables AP mode when WiFi disconnects
|
||||||
- AP mode switches WiFi from client to access point mode
|
- AP mode switches WiFi from client to access point mode
|
||||||
- Pi loses connection to your original network
|
- Pi loses connection to your original network
|
||||||
|
|
||||||
|
*Memory exhaustion (low-memory boards):*
|
||||||
|
- The board runs out of memory, so `sshd` cannot fork a session process
|
||||||
|
- The connection is accepted and closed before any banner
|
||||||
|
- Ping still answers and the web UI still responds, so it looks healthy
|
||||||
|
- The panel is usually dark and the service cannot restart
|
||||||
|
- **Only a power cycle clears this** — there is no remote recovery, because
|
||||||
|
every remote route needs a new process
|
||||||
|
- Prevention and tuning: [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md)
|
||||||
|
|
||||||
**To regain SSH**:
|
**To regain SSH**:
|
||||||
1. Connect to **LEDMatrix-Setup** AP network (password: `ledmatrix123`)
|
1. Connect to **LEDMatrix-Setup** AP network (password: `ledmatrix123`)
|
||||||
2. SSH to `192.168.4.1`
|
2. SSH to `192.168.4.1`
|
||||||
|
|||||||
Regular → Executable
+101
-3
@@ -1419,9 +1419,16 @@ $ACTUAL_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_ROOT_DIR/scripts/fix_perms/
|
|||||||
EOF
|
EOF
|
||||||
if [ -n "$JOURNALCTL_PATH" ]; then
|
if [ -n "$JOURNALCTL_PATH" ]; then
|
||||||
cat >> /tmp/ledmatrix_web_sudoers << EOF
|
cat >> /tmp/ledmatrix_web_sudoers << EOF
|
||||||
$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *
|
# NOEXEC, because these rules end in a wildcard and journalctl starts a pager
|
||||||
$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *
|
# when its output is a terminal. From that pager (less) a "!sh" is a root
|
||||||
$ACTUAL_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *
|
# shell -- the standard journalctl escalation. The web interface always passes
|
||||||
|
# --no-pager, so nothing here needs it, but the rule cannot require a flag that
|
||||||
|
# sits in the middle of the command line. NOEXEC stops the command executing
|
||||||
|
# another program at all, which closes the hole without depending on wildcard
|
||||||
|
# matching subtleties.
|
||||||
|
$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix.service *
|
||||||
|
$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix *
|
||||||
|
$ACTUAL_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -t ledmatrix *
|
||||||
EOF
|
EOF
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -1688,6 +1695,97 @@ else
|
|||||||
echo "✗ $CMDLINE_FILE not found; skipping isolcpus optimization"
|
echo "✗ $CMDLINE_FILE not found; skipping isolcpus optimization"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Enable the memory cgroup controller (idempotent).
|
||||||
|
# The Pi firmware boots with cgroup_disable=memory, so systemd's MemoryMax= is
|
||||||
|
# accepted and silently ignored — the display service then has no ceiling, and
|
||||||
|
# a runaway takes the whole board down (sshd can no longer fork, the panel goes
|
||||||
|
# dark) rather than just restarting the one service.
|
||||||
|
if [ "$SKIP_PERF" != "1" ] && [ -f "$CMDLINE_FILE" ]; then
|
||||||
|
# Both parameters are required for the memory controller, and they can get
|
||||||
|
# separated -- an image, another tool or a half-applied earlier run can
|
||||||
|
# leave one without the other. Checking only cgroup_enable=memory would
|
||||||
|
# report success while MemoryMax= silently does nothing, so each is checked
|
||||||
|
# and appended independently.
|
||||||
|
cgroup_missing=""
|
||||||
|
for cgroup_param in cgroup_enable=memory cgroup_memory=1; do
|
||||||
|
if ! grep -qw "$cgroup_param" "$CMDLINE_FILE"; then
|
||||||
|
cgroup_missing="$cgroup_missing $cgroup_param"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
if [ -z "$cgroup_missing" ]; then
|
||||||
|
echo "cgroup memory parameters already present in $CMDLINE_FILE"
|
||||||
|
else
|
||||||
|
echo "Adding${cgroup_missing} to $CMDLINE_FILE..."
|
||||||
|
cp "$CMDLINE_FILE" "$CMDLINE_FILE.bak" 2>/dev/null || true
|
||||||
|
# The kernel command line must stay on one line.
|
||||||
|
sed -i "1 s|\$|${cgroup_missing}|" "$CMDLINE_FILE"
|
||||||
|
echo " Takes effect after reboot. Verify with:"
|
||||||
|
echo " grep memory /sys/fs/cgroup/cgroup.controllers"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Persist the journal (idempotent).
|
||||||
|
# These images default to volatile storage: journald keeps everything in /run
|
||||||
|
# (tmpfs), so every reboot destroys the logs — including the ones that would
|
||||||
|
# explain why the board rebooted. Capped so an SD card is not worn out by logs.
|
||||||
|
# A non-empty /var/log/journal does not prove journald is configured the way
|
||||||
|
# this needs: the directory survives a switch back to volatile storage, and it
|
||||||
|
# says nothing about whether a size cap is set. Read the effective
|
||||||
|
# configuration instead, and only write the keys the user has not set
|
||||||
|
# themselves so an explicit local limit is preserved.
|
||||||
|
journald_effective() {
|
||||||
|
# systemd-analyze merges journald.conf with every drop-in; grep is the
|
||||||
|
# fallback for images that ship without it.
|
||||||
|
if command -v systemd-analyze >/dev/null 2>&1 &&
|
||||||
|
systemd-analyze cat-config systemd/journald.conf >/dev/null 2>&1; then
|
||||||
|
systemd-analyze cat-config systemd/journald.conf 2>/dev/null
|
||||||
|
else
|
||||||
|
cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
journald_conf="$(journald_effective)"
|
||||||
|
# Both settings are optional, and stock images ship them commented out. grep
|
||||||
|
# exits 1 on no match, which pipefail turns fatal under set -e — an absent
|
||||||
|
# setting must read as empty, not abort the install.
|
||||||
|
journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)"
|
||||||
|
journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)"
|
||||||
|
|
||||||
|
if [ "$journald_storage" = "persistent" ] && [ -n "$journald_cap" ]; then
|
||||||
|
echo "Persistent journald storage already configured (SystemMaxUse=$journald_cap)"
|
||||||
|
else
|
||||||
|
echo "Enabling persistent journald storage..."
|
||||||
|
mkdir -p /etc/systemd/journald.conf.d
|
||||||
|
{
|
||||||
|
echo "# Installed by LEDMatrix first_time_install.sh"
|
||||||
|
echo "[Journal]"
|
||||||
|
echo "Storage=persistent"
|
||||||
|
if [ -n "$journald_cap" ]; then
|
||||||
|
echo "# SystemMaxUse left to your existing setting ($journald_cap)"
|
||||||
|
else
|
||||||
|
# Capped so logs cannot wear out or fill an SD card.
|
||||||
|
echo "SystemMaxUse=64M"
|
||||||
|
fi
|
||||||
|
} > /etc/systemd/journald.conf.d/ledmatrix-persistent.conf
|
||||||
|
mkdir -p /var/log/journal
|
||||||
|
systemd-tmpfiles --create --prefix /var/log/journal >/dev/null 2>&1 || true
|
||||||
|
systemctl restart systemd-journald >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
# Drop-ins are applied in lexical order, so a locally added file that sorts
|
||||||
|
# after ledmatrix-persistent.conf (zz-local.conf and friends) still wins.
|
||||||
|
# Writing the file is not evidence it took effect -- re-read and say so
|
||||||
|
# plainly rather than reporting success we cannot confirm.
|
||||||
|
journald_now="$(journald_effective | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]' || true)"
|
||||||
|
if [ "$journald_now" = "persistent" ]; then
|
||||||
|
echo " Persistent journald storage active"
|
||||||
|
else
|
||||||
|
echo " WARNING: journald storage is still '${journald_now:-unset}' after"
|
||||||
|
echo " writing /etc/systemd/journald.conf.d/ledmatrix-persistent.conf."
|
||||||
|
echo " Another drop-in that sorts later is overriding it. Check:"
|
||||||
|
echo " systemd-analyze cat-config systemd/journald.conf | grep -n Storage="
|
||||||
|
echo " Logs will not survive a reboot until that is resolved."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
# Ensure dtparam=audio=off in config.txt (idempotent)
|
# Ensure dtparam=audio=off in config.txt (idempotent)
|
||||||
if [ "$SKIP_PERF" = "1" ]; then
|
if [ "$SKIP_PERF" = "1" ]; then
|
||||||
: # skipped
|
: # skipped
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ from src.plugin_system.testing.loading import ( # noqa: E402
|
|||||||
)
|
)
|
||||||
from src.plugin_system.testing.harness import ( # noqa: E402
|
from src.plugin_system.testing.harness import ( # noqa: E402
|
||||||
RenderResult, render_plugin_matrix, compare_to_goldens, write_goldens,
|
RenderResult, render_plugin_matrix, compare_to_goldens, write_goldens,
|
||||||
|
check_empty_claimed,
|
||||||
check_scale_up,
|
check_scale_up,
|
||||||
)
|
)
|
||||||
from src.plugin_system.testing.sizes import ( # noqa: E402
|
from src.plugin_system.testing.sizes import ( # noqa: E402
|
||||||
@@ -115,6 +116,11 @@ def check_one(plugin_id: str, search_dirs: List[str], sizes, mock_data: Dict,
|
|||||||
declared = load_manifest(plugin_dir).get("display", {}).get("design_size", {})
|
declared = load_manifest(plugin_dir).get("display", {}).get("design_size", {})
|
||||||
design_size = (int(declared.get("width", 128)), int(declared.get("height", 32)))
|
design_size = (int(declared.get("width", 128)), int(declared.get("height", 32)))
|
||||||
fill_strict = spec.get("fill_check") == "strict"
|
fill_strict = spec.get("fill_check") == "strict"
|
||||||
|
# A mode that renders nothing without returning False is never skipped by
|
||||||
|
# the display controller, so it holds a blank panel for its whole duration.
|
||||||
|
# Warn-only by default: a scroll mode's first frame is legitimately its
|
||||||
|
# blank scroll-in buffer.
|
||||||
|
empty_strict = spec.get("empty_check") == "strict"
|
||||||
|
|
||||||
# Every run: the base config, plus one per harness.json "variant" —
|
# Every run: the base config, plus one per harness.json "variant" —
|
||||||
# a config overlay with its own golden dir (e.g. adaptive layout mode
|
# a config overlay with its own golden dir (e.g. adaptive layout mode
|
||||||
@@ -142,6 +148,7 @@ def check_one(plugin_id: str, search_dirs: List[str], sizes, mock_data: Dict,
|
|||||||
compare_to_goldens(results, golden_dir)
|
compare_to_goldens(results, golden_dir)
|
||||||
|
|
||||||
check_scale_up(results, design_size=design_size, strict=fill_strict)
|
check_scale_up(results, design_size=design_size, strict=fill_strict)
|
||||||
|
check_empty_claimed(results, strict=empty_strict)
|
||||||
|
|
||||||
# Tag variant runs so the report and PNG dumps stay distinguishable.
|
# Tag variant runs so the report and PNG dumps stay distinguishable.
|
||||||
if variant_name:
|
if variant_name:
|
||||||
@@ -178,6 +185,9 @@ def print_report(all_results: Dict[str, List[RenderResult]]) -> bool:
|
|||||||
# warn-only underfill: big panel left mostly empty
|
# warn-only underfill: big panel left mostly empty
|
||||||
ex, ey = r.fill_extent
|
ex, ey = r.fill_extent
|
||||||
detail += f" (fill warn: extent {ex:.0%}x{ey:.0%})"
|
detail += f" (fill warn: extent {ex:.0%}x{ey:.0%})"
|
||||||
|
if r.empty_claimed and r.empty_ok is None:
|
||||||
|
detail += (f" (empty warn: drew nothing but display() returned"
|
||||||
|
f" {r.display_returned!r}, so the mode is not skipped)")
|
||||||
else:
|
else:
|
||||||
everything_ok = False
|
everything_ok = False
|
||||||
if r.error is not None:
|
if r.error is not None:
|
||||||
@@ -191,6 +201,11 @@ def print_report(all_results: Dict[str, List[RenderResult]]) -> bool:
|
|||||||
ex, ey = r.fill_extent or (0.0, 0.0)
|
ex, ey = r.fill_extent or (0.0, 0.0)
|
||||||
status = "FAIL"
|
status = "FAIL"
|
||||||
detail = f" fill: extent {ex:.0%}x{ey:.0%} below required coverage"
|
detail = f" fill: extent {ex:.0%}x{ey:.0%} below required coverage"
|
||||||
|
elif r.empty_ok is False:
|
||||||
|
status = "FAIL"
|
||||||
|
detail = (f" drew nothing but display() returned"
|
||||||
|
f" {r.display_returned!r}; return False so the"
|
||||||
|
f" controller skips the mode")
|
||||||
else:
|
else:
|
||||||
status, detail = "FAIL", ""
|
status, detail = "FAIL", ""
|
||||||
print(f" [{status}] {r.size_label:>7} {r.mode}{detail}")
|
print(f" [{status}] {r.size_label:>7} {r.mode}{detail}")
|
||||||
|
|||||||
@@ -24,9 +24,29 @@ echo "========================================"
|
|||||||
# Auto-detect latest version if needed
|
# Auto-detect latest version if needed
|
||||||
if [ "$PIXLET_VERSION" = "latest" ]; then
|
if [ "$PIXLET_VERSION" = "latest" ]; then
|
||||||
echo "Detecting latest version..."
|
echo "Detecting latest version..."
|
||||||
PIXLET_VERSION=$(curl -s "https://api.github.com/repos/${REPO}/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
|
# When this response arrives on a single line -- as it did on the device
|
||||||
if [ -z "$PIXLET_VERSION" ]; then
|
# where Starlark apps were failing -- `grep '"tag_name"'` matches the whole
|
||||||
echo "Failed to detect latest version, using fallback"
|
# document and a greedy `sed 's/.*"([^"]+)".*/\1/'` captures the LAST
|
||||||
|
# quoted token in it rather than the tag. That resolved to
|
||||||
|
# "mentions_count", which built a download URL for a release that does not
|
||||||
|
# exist. (The API is pretty-printed by default, which is why the old
|
||||||
|
# command looks correct when you try it by hand -- but the formatting is
|
||||||
|
# not something to depend on.) Match the field itself and take the value
|
||||||
|
# after it, which is right for either shape.
|
||||||
|
PIXLET_VERSION=$(curl -fsSL "https://api.github.com/repos/${REPO}/releases/latest" \
|
||||||
|
| grep -o '"tag_name"[[:space:]]*:[[:space:]]*"[^"]*"' \
|
||||||
|
| head -n1 \
|
||||||
|
| sed -E 's/.*:[[:space:]]*"([^"]*)".*/\1/')
|
||||||
|
|
||||||
|
# A wrong-but-non-empty value is what made the old bug silent, so check the
|
||||||
|
# shape rather than just that something came back. Anchored at both ends: a
|
||||||
|
# partial match would accept "v0.53garbage" or "0.53" and build a URL for a
|
||||||
|
# release that cannot exist, which is the failure this check is here to
|
||||||
|
# stop. Every tronbyt/pixlet release to date is vX.Y.Z; the optional suffix
|
||||||
|
# leaves room for a future -rc.1 or +build tag.
|
||||||
|
if ! printf '%s' "$PIXLET_VERSION" \
|
||||||
|
| grep -qE '^v[0-9]+\.[0-9]+\.[0-9]+([-+][0-9A-Za-z.-]+)?$'; then
|
||||||
|
echo "Could not detect the latest version (got: '${PIXLET_VERSION:-<empty>}'), using fallback"
|
||||||
PIXLET_VERSION="v0.50.2"
|
PIXLET_VERSION="v0.50.2"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
@@ -67,8 +87,26 @@ download_binary() {
|
|||||||
temp_dir=$(mktemp -d -p "$PROJECT_ROOT" -t pixlet_download.XXXXXXXXXX)
|
temp_dir=$(mktemp -d -p "$PROJECT_ROOT" -t pixlet_download.XXXXXXXXXX)
|
||||||
local temp_file="$temp_dir/$archive_name"
|
local temp_file="$temp_dir/$archive_name"
|
||||||
|
|
||||||
if ! curl -L -o "$temp_file" "$url" 2>/dev/null; then
|
# -f so an HTTP error is a failure. Without it curl writes the 404 body
|
||||||
echo "✗ Failed to download $arch"
|
# to the file and exits 0, and the first sign of trouble is tar saying
|
||||||
|
# "not in gzip format" about what is actually a page of HTML.
|
||||||
|
if ! curl -fL -o "$temp_file" "$url" 2>/dev/null; then
|
||||||
|
echo "✗ Failed to download $arch from $url"
|
||||||
|
rm -rf "$temp_dir"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Belt and braces: a mirror or proxy can return 200 with an error page.
|
||||||
|
if ! gzip -t "$temp_file" 2>/dev/null; then
|
||||||
|
echo "✗ Downloaded file is not a gzip archive: $url"
|
||||||
|
# These bytes come from whatever answered the request, so strip
|
||||||
|
# everything non-printable before echoing them: an error page carrying
|
||||||
|
# terminal escapes would otherwise be able to rewrite this output or
|
||||||
|
# bury it in a CI log. Printable characters are kept rather than
|
||||||
|
# hex-encoding the lot, because "<!DOCTYPE html>" is the diagnostic.
|
||||||
|
local first_bytes
|
||||||
|
first_bytes=$(head -c 60 "$temp_file" | tr -cd '[:print:]')
|
||||||
|
printf ' (first bytes: %s)\n' "$first_bytes"
|
||||||
rm -rf "$temp_dir"
|
rm -rf "$temp_dir"
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
@@ -100,10 +100,15 @@ TEMP_SUDOERS="/tmp/ledmatrix_web_sudoers_$$"
|
|||||||
echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web.service"
|
echo "$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart ledmatrix-web.service"
|
||||||
|
|
||||||
# Optional: journalctl (non-critical — skip if not found)
|
# Optional: journalctl (non-critical — skip if not found)
|
||||||
|
#
|
||||||
|
# NOEXEC, matching first_time_install.sh. These rules end in a wildcard and
|
||||||
|
# journalctl starts a pager, so without it the caller can reach a shell:
|
||||||
|
# less runs "!command" as the user the pager belongs to, which here is
|
||||||
|
# root. NOEXEC stops the granted command executing anything of its own.
|
||||||
if [ -n "$JOURNALCTL_PATH" ]; then
|
if [ -n "$JOURNALCTL_PATH" ]; then
|
||||||
echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix.service *"
|
echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix.service *"
|
||||||
echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -u ledmatrix *"
|
echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -u ledmatrix *"
|
||||||
echo "$WEB_USER ALL=(ALL) NOPASSWD: $JOURNALCTL_PATH -t ledmatrix *"
|
echo "$WEB_USER ALL=(ALL) NOPASSWD:NOEXEC: $JOURNALCTL_PATH -t ledmatrix *"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Required: python3, bash
|
# Required: python3, bash
|
||||||
|
|||||||
@@ -25,9 +25,44 @@ if [ "$EUID" -eq 0 ]; then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Resolve command paths against a fixed PATH, and check what we resolved.
|
||||||
|
#
|
||||||
|
# Every path found here is written into a sudoers file as a NOPASSWD grant, so
|
||||||
|
# whoever controls the binary at that path controls root. first_time_install.sh
|
||||||
|
# re-execs itself with `sudo -E`, which preserves the invoking user's
|
||||||
|
# environment -- PATH included -- so without pinning it, `which nmcli` can
|
||||||
|
# resolve to anything on that PATH: a writable directory early in it turns a
|
||||||
|
# compromise of the low-privilege web user into permanent root.
|
||||||
|
PATH=/usr/sbin:/usr/bin:/sbin:/bin
|
||||||
|
export PATH
|
||||||
|
|
||||||
|
# A binary named in a sudoers rule must be root-owned and writable by nobody
|
||||||
|
# else, or the grant hands root to whoever can rewrite it.
|
||||||
|
require_trusted_binary() {
|
||||||
|
local label="$1" path="$2"
|
||||||
|
if [ ! -x "$path" ]; then
|
||||||
|
echo "✗ $label: $path is not an executable file"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
local owner perms
|
||||||
|
owner=$(stat -c '%u' "$path") || exit 1
|
||||||
|
perms=$(stat -c '%a' "$path") || exit 1
|
||||||
|
if [ "$owner" != "0" ]; then
|
||||||
|
echo "✗ $label: $path is not owned by root (uid $owner); refusing to"
|
||||||
|
echo " grant it NOPASSWD sudo."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
# Group- or world-writable means someone other than root can replace it.
|
||||||
|
case "$perms" in
|
||||||
|
*[2367]) echo "✗ $label: $path is writable by group or other ($perms);"
|
||||||
|
echo " refusing to grant it NOPASSWD sudo."
|
||||||
|
exit 1 ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
# Get the full paths to commands
|
# Get the full paths to commands
|
||||||
NMCLI_PATH=$(which nmcli || echo "/usr/bin/nmcli")
|
NMCLI_PATH=$(command -v nmcli || echo "/usr/bin/nmcli")
|
||||||
SYSTEMCTL_PATH=$(which systemctl)
|
SYSTEMCTL_PATH=$(command -v systemctl)
|
||||||
|
|
||||||
echo "Command paths:"
|
echo "Command paths:"
|
||||||
echo " nmcli: $NMCLI_PATH"
|
echo " nmcli: $NMCLI_PATH"
|
||||||
@@ -37,6 +72,18 @@ echo " systemctl: $SYSTEMCTL_PATH"
|
|||||||
echo ""
|
echo ""
|
||||||
echo "Step 1: Configuring sudo permissions for nmcli..."
|
echo "Step 1: Configuring sudo permissions for nmcli..."
|
||||||
SUDOERS_FILE="/etc/sudoers.d/ledmatrix_wifi"
|
SUDOERS_FILE="/etc/sudoers.d/ledmatrix_wifi"
|
||||||
|
SYSCTL_PATH=$(command -v sysctl || echo /usr/sbin/sysctl)
|
||||||
|
NFT_PATH=$(command -v nft || echo /usr/sbin/nft)
|
||||||
|
RFKILL_PATH=$(command -v rfkill || echo /usr/sbin/rfkill)
|
||||||
|
MKDIR_PATH=$(command -v mkdir || echo /usr/bin/mkdir)
|
||||||
|
|
||||||
|
# Checked before any of them reaches the sudoers file.
|
||||||
|
require_trusted_binary "nmcli" "$NMCLI_PATH"
|
||||||
|
require_trusted_binary "systemctl" "$SYSTEMCTL_PATH"
|
||||||
|
require_trusted_binary "sysctl" "$SYSCTL_PATH"
|
||||||
|
require_trusted_binary "nft" "$NFT_PATH"
|
||||||
|
require_trusted_binary "rfkill" "$RFKILL_PATH"
|
||||||
|
require_trusted_binary "mkdir" "$MKDIR_PATH"
|
||||||
|
|
||||||
# Create a temporary sudoers file using mktemp (handles permissions better)
|
# Create a temporary sudoers file using mktemp (handles permissions better)
|
||||||
TEMP_SUDOERS=$(mktemp) || {
|
TEMP_SUDOERS=$(mktemp) || {
|
||||||
@@ -62,6 +109,36 @@ $WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH start dnsmasq
|
|||||||
$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop dnsmasq
|
$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH stop dnsmasq
|
||||||
$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart dnsmasq
|
$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart dnsmasq
|
||||||
$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart NetworkManager
|
$WEB_USER ALL=(ALL) NOPASSWD: $SYSTEMCTL_PATH restart NetworkManager
|
||||||
|
# The captive portal turns IP forwarding on while the access point is up and
|
||||||
|
# restores the previous value when it comes down (wifi_manager._setup_iptables_
|
||||||
|
# redirect / _teardown_iptables_redirect). Without this rule that sudo call
|
||||||
|
# needs a password, so forwarding stays off and clients associate to the AP but
|
||||||
|
# cannot route. It goes unnoticed on a stock Raspberry Pi image, where
|
||||||
|
# /etc/sudoers.d/010_pi-nopasswd grants the default user blanket NOPASSWD and
|
||||||
|
# masks every gap in this file -- it only bites once that blanket rule is
|
||||||
|
# removed.
|
||||||
|
$WEB_USER ALL=(ALL) NOPASSWD: $SYSCTL_PATH -w net.ipv4.ip_forward=0
|
||||||
|
$WEB_USER ALL=(ALL) NOPASSWD: $SYSCTL_PATH -w net.ipv4.ip_forward=1
|
||||||
|
# The portal's redirect lives in its own nftables table, created when the AP
|
||||||
|
# comes up and deleted when it goes down, and the radio has to be unblocked
|
||||||
|
# before the AP can start at all. Same story as the sysctl rules above: called
|
||||||
|
# with sudo, never granted here, and invisible on a stock Pi image.
|
||||||
|
$WEB_USER ALL=(ALL) NOPASSWD: $NFT_PATH add table ip ledmatrix
|
||||||
|
$WEB_USER ALL=(ALL) NOPASSWD: $NFT_PATH delete table ip ledmatrix
|
||||||
|
$WEB_USER ALL=(ALL) NOPASSWD: $RFKILL_PATH unblock wifi
|
||||||
|
# NetworkManager's dnsmasq drop-in directory, exact path.
|
||||||
|
$WEB_USER ALL=(ALL) NOPASSWD: $MKDIR_PATH -p /etc/NetworkManager/dnsmasq-shared.d
|
||||||
|
#
|
||||||
|
# iptables is deliberately NOT granted here. Its rules are built from the live
|
||||||
|
# interface name and port, so a rule covering them needs a trailing wildcard --
|
||||||
|
# and `iptables --modprobe=/path/to/anything` runs that path as root, so
|
||||||
|
# `NOPASSWD: iptables *` is a root shell for the web user by another name. That
|
||||||
|
# is a worse outcome than the gap it would close, which today is masked anyway
|
||||||
|
# by the blanket NOPASSWD rule on stock Pi images.
|
||||||
|
#
|
||||||
|
# Closing it safely means a wrapper script that builds the rules itself and
|
||||||
|
# takes only an interface and a port, granted the way safe_plugin_rm.sh already
|
||||||
|
# is. That belongs in its own change rather than being smuggled into this one.
|
||||||
|
|
||||||
# Allow copying hostapd and dnsmasq config files into place
|
# Allow copying hostapd and dnsmasq config files into place
|
||||||
$WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/cp /tmp/hostapd.conf /etc/hostapd/hostapd.conf
|
$WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/cp /tmp/hostapd.conf /etc/hostapd/hostapd.conf
|
||||||
|
|||||||
Regular → Executable
Regular → Executable
@@ -52,6 +52,10 @@ def main() -> int:
|
|||||||
parser.add_argument('--height', type=int, default=32, help='Display height (default: 32)')
|
parser.add_argument('--height', type=int, default=32, help='Display height (default: 32)')
|
||||||
parser.add_argument('--skip-update', action='store_true',
|
parser.add_argument('--skip-update', action='store_true',
|
||||||
help='Skip calling update() (render display only)')
|
help='Skip calling update() (render display only)')
|
||||||
|
parser.add_argument('--display-mode', default=None,
|
||||||
|
help='Display mode to render, for plugins that declare '
|
||||||
|
'more than one in their manifest (e.g. nrl_live). '
|
||||||
|
'Omitted, the plugin picks its own default.')
|
||||||
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
@@ -141,7 +145,25 @@ def main() -> int:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning("update() raised: %s — continuing to display()", e)
|
logger.warning("update() raised: %s — continuing to display()", e)
|
||||||
|
|
||||||
|
# A plugin that declares several display modes usually renders nothing
|
||||||
|
# useful without being told which one to draw: the scoreboards keep their
|
||||||
|
# state on per-mode sub-managers and their no-argument path returns False.
|
||||||
|
# Only pass the argument when asked for, so the many plugins whose display()
|
||||||
|
# takes no display_mode keep working untouched.
|
||||||
try:
|
try:
|
||||||
|
if args.display_mode:
|
||||||
|
try:
|
||||||
|
plugin_instance.display(display_mode=args.display_mode,
|
||||||
|
force_clear=True)
|
||||||
|
except TypeError as error:
|
||||||
|
if ("unexpected keyword argument" not in str(error)
|
||||||
|
or "display_mode" not in str(error)):
|
||||||
|
raise
|
||||||
|
logger.warning(
|
||||||
|
"%s.display() does not accept display_mode; rendering its "
|
||||||
|
"default screen instead", args.plugin)
|
||||||
|
plugin_instance.display(force_clear=True)
|
||||||
|
else:
|
||||||
plugin_instance.display(force_clear=True)
|
plugin_instance.display(force_clear=True)
|
||||||
logger.debug("display() completed")
|
logger.debug("display() completed")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
|||||||
+1
-1
@@ -4,5 +4,5 @@ LEDMatrix Display System
|
|||||||
Core source package for the LED Matrix Display project.
|
Core source package for the LED Matrix Display project.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
__version__ = "3.2.0"
|
__version__ = "3.3.0"
|
||||||
|
|
||||||
|
|||||||
@@ -14,11 +14,12 @@ Key Features:
|
|||||||
- Memory-efficient data storage
|
- Memory-efficient data storage
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import itertools
|
||||||
import time
|
import time
|
||||||
import logging
|
import logging
|
||||||
import threading
|
import threading
|
||||||
import requests
|
import requests
|
||||||
from typing import Dict, Any, Optional, Callable
|
from typing import Dict, Any, Optional, Callable, List
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
import queue
|
import queue
|
||||||
@@ -50,14 +51,33 @@ class FetchRequest:
|
|||||||
max_retries: int = 3
|
max_retries: int = 3
|
||||||
priority: int = 1 # Higher number = higher priority
|
priority: int = 1 # Higher number = higher priority
|
||||||
callback: Optional[Callable] = None
|
callback: Optional[Callable] = None
|
||||||
|
# Callbacks from submitters that JOINED this fetch instead of starting a
|
||||||
|
# duplicate one. The primary `callback` above belongs to whoever created
|
||||||
|
# the request; these belong to everyone who asked for the same cache_key
|
||||||
|
# while it was still in flight.
|
||||||
|
extra_callbacks: List[Callable] = field(default_factory=list)
|
||||||
created_at: float = field(default_factory=time.time)
|
created_at: float = field(default_factory=time.time)
|
||||||
status: FetchStatus = FetchStatus.PENDING
|
status: FetchStatus = FetchStatus.PENDING
|
||||||
|
# Set once the worker has decided this response will be cached, while it
|
||||||
|
# still holds the lock. From that point cancelling is refused: the write
|
||||||
|
# is already authorised, and abandoning it here would put the payload in
|
||||||
|
# the cache with the callbacks suppressed -- joiners waiting forever for a
|
||||||
|
# fetch that did, in fact, succeed.
|
||||||
|
commit_claimed: bool = False
|
||||||
result: Optional[Any] = None
|
result: Optional[Any] = None
|
||||||
error: Optional[str] = None
|
error: Optional[str] = None
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class FetchResult:
|
class FetchResult:
|
||||||
"""Result of a background fetch operation."""
|
"""Result of a background fetch operation.
|
||||||
|
|
||||||
|
``data`` survives on the stored result only for requests submitted without
|
||||||
|
a ``callback``, where polling ``get_result()`` is the sole way to collect
|
||||||
|
it. When a callback was given, the payload has already been delivered and
|
||||||
|
the service releases it -- see :meth:`BackgroundDataService._release_payload`.
|
||||||
|
Either way the data remains in the cache under the request's ``cache_key``,
|
||||||
|
which is where consumers read it from.
|
||||||
|
"""
|
||||||
request_id: str
|
request_id: str
|
||||||
success: bool
|
success: bool
|
||||||
data: Optional[Any] = None
|
data: Optional[Any] = None
|
||||||
@@ -66,6 +86,10 @@ class FetchResult:
|
|||||||
fetch_time: float = 0.0
|
fetch_time: float = 0.0
|
||||||
retry_count: int = 0
|
retry_count: int = 0
|
||||||
completed_at: float = field(default_factory=time.time) # Timestamp when request completed
|
completed_at: float = field(default_factory=time.time) # Timestamp when request completed
|
||||||
|
# The request's final status, recorded so a finished request can still be
|
||||||
|
# reported accurately. Without it a caller can only be told COMPLETED or
|
||||||
|
# FAILED, which turns "you cancelled this" into "this errored".
|
||||||
|
final_status: Optional[FetchStatus] = None
|
||||||
|
|
||||||
class BackgroundDataService:
|
class BackgroundDataService:
|
||||||
"""
|
"""
|
||||||
@@ -90,6 +114,20 @@ class BackgroundDataService:
|
|||||||
|
|
||||||
# Thread management
|
# Thread management
|
||||||
self.executor = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix="BackgroundData")
|
self.executor = ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix="BackgroundData")
|
||||||
|
# cache_key -> request_id for fetches currently in flight. Submitting
|
||||||
|
# the same key twice used to start two identical fetches: request_id
|
||||||
|
# carries a millisecond timestamp, so every submit looked new, and
|
||||||
|
# active_requests is keyed by it rather than by what is being fetched.
|
||||||
|
# On a real board the season-schedule key is requested by both the
|
||||||
|
# Recent and the Upcoming manager, which miss the cache in the same
|
||||||
|
# millisecond and each download and parse the same payload.
|
||||||
|
self._inflight_by_cache_key: Dict[str, str] = {}
|
||||||
|
# request_id was sport_year_milliseconds, which is not unique: two
|
||||||
|
# submits inside the same millisecond produced the SAME id, so one
|
||||||
|
# silently replaced the other in active_requests and completed_requests.
|
||||||
|
# Rare before, but dedupe hands this id back to every joiner as their
|
||||||
|
# handle for get_result(), so it has to be unique. A counter is enough.
|
||||||
|
self._request_seq = itertools.count()
|
||||||
self.active_requests: Dict[str, FetchRequest] = {}
|
self.active_requests: Dict[str, FetchRequest] = {}
|
||||||
self.completed_requests: Dict[str, FetchResult] = {}
|
self.completed_requests: Dict[str, FetchResult] = {}
|
||||||
self.request_queue = queue.PriorityQueue()
|
self.request_queue = queue.PriorityQueue()
|
||||||
@@ -177,7 +215,9 @@ class BackgroundDataService:
|
|||||||
if cache_key is None:
|
if cache_key is None:
|
||||||
cache_key = self.get_sport_cache_key(sport)
|
cache_key = self.get_sport_cache_key(sport)
|
||||||
|
|
||||||
request_id = f"{sport}_{year}_{int(time.time() * 1000)}"
|
with self._lock:
|
||||||
|
request_id = (f"{sport}_{year}_{int(time.time() * 1000)}"
|
||||||
|
f"_{next(self._request_seq)}")
|
||||||
|
|
||||||
# Check cache first
|
# Check cache first
|
||||||
cached_data = self.cache_manager.get(cache_key)
|
cached_data = self.cache_manager.get(cache_key)
|
||||||
@@ -191,6 +231,10 @@ class BackgroundDataService:
|
|||||||
cached=True,
|
cached=True,
|
||||||
fetch_time=0.0
|
fetch_time=0.0
|
||||||
)
|
)
|
||||||
|
# Filed before the callback runs, as it always was: a callback
|
||||||
|
# that queries get_result()/is_request_complete() for its own
|
||||||
|
# request must still find it. Releasing afterwards mutates the
|
||||||
|
# same object the dict holds.
|
||||||
self.completed_requests[request_id] = result
|
self.completed_requests[request_id] = result
|
||||||
|
|
||||||
if callback:
|
if callback:
|
||||||
@@ -198,6 +242,7 @@ class BackgroundDataService:
|
|||||||
callback(result)
|
callback(result)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error in callback for request {request_id}: {e}")
|
logger.error(f"Error in callback for request {request_id}: {e}")
|
||||||
|
self._release_payload(result)
|
||||||
|
|
||||||
logger.debug(f"Cache hit for {sport} {year} data")
|
logger.debug(f"Cache hit for {sport} {year} data")
|
||||||
return request_id
|
return request_id
|
||||||
@@ -218,7 +263,29 @@ class BackgroundDataService:
|
|||||||
)
|
)
|
||||||
|
|
||||||
with self._lock:
|
with self._lock:
|
||||||
|
existing_id = self._inflight_by_cache_key.get(cache_key)
|
||||||
|
existing = self.active_requests.get(existing_id) if existing_id else None
|
||||||
|
if existing_id and existing is None:
|
||||||
|
# Stranded index entry: the request it names is gone. Drop it and
|
||||||
|
# fetch normally. Looking the request up rather than trusting the
|
||||||
|
# id is what stops a stale entry wedging a key forever.
|
||||||
|
del self._inflight_by_cache_key[cache_key]
|
||||||
|
if existing is not None:
|
||||||
|
# Someone is already fetching this key. Ride along rather than
|
||||||
|
# duplicating the download, the parse and the resident copy.
|
||||||
|
if callback:
|
||||||
|
existing.extra_callbacks.append(callback)
|
||||||
|
self.stats['deduplicated_requests'] = (
|
||||||
|
self.stats.get('deduplicated_requests', 0) + 1
|
||||||
|
)
|
||||||
|
logger.info(
|
||||||
|
"Joined in-flight fetch %s for %s (cache_key=%s) instead of "
|
||||||
|
"starting a duplicate", existing_id, sport, cache_key
|
||||||
|
)
|
||||||
|
return existing_id
|
||||||
|
|
||||||
self.active_requests[request_id] = request
|
self.active_requests[request_id] = request
|
||||||
|
self._inflight_by_cache_key[cache_key] = request_id
|
||||||
self.stats['total_requests'] += 1
|
self.stats['total_requests'] += 1
|
||||||
self.stats['cache_misses'] += 1
|
self.stats['cache_misses'] += 1
|
||||||
|
|
||||||
@@ -243,7 +310,31 @@ class BackgroundDataService:
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
with self._lock:
|
with self._lock:
|
||||||
|
# A request cancelled while it sat in the executor queue must
|
||||||
|
# stay cancelled. Overwriting the status here undid the cancel
|
||||||
|
# outright: the worker went on to download, cache and call back
|
||||||
|
# for work the caller had already withdrawn.
|
||||||
|
if request.status == FetchStatus.CANCELLED:
|
||||||
|
cancelled_before_start = True
|
||||||
|
else:
|
||||||
|
cancelled_before_start = False
|
||||||
request.status = FetchStatus.IN_PROGRESS
|
request.status = FetchStatus.IN_PROGRESS
|
||||||
|
if cancelled_before_start:
|
||||||
|
logger.info(
|
||||||
|
"Request %s was cancelled before its worker started; "
|
||||||
|
"skipping the fetch entirely", request.id
|
||||||
|
)
|
||||||
|
# Assign before returning: the finally block stores `result`
|
||||||
|
# in completed_requests, so building a fresh one here would
|
||||||
|
# file the untouched placeholder instead of this outcome.
|
||||||
|
result = FetchResult(
|
||||||
|
request_id=request.id,
|
||||||
|
success=False,
|
||||||
|
error="cancelled",
|
||||||
|
fetch_time=time.time() - start_time,
|
||||||
|
retry_count=request.retry_count
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
logger.info(f"Starting background fetch for {request.sport} {request.year}")
|
logger.info(f"Starting background fetch for {request.sport} {request.year}")
|
||||||
|
|
||||||
@@ -269,6 +360,37 @@ class BackgroundDataService:
|
|||||||
# Log data validation
|
# Log data validation
|
||||||
logger.debug(f"Validated {len(events)} events for {request.sport} {request.year}")
|
logger.debug(f"Validated {len(events)} events for {request.sport} {request.year}")
|
||||||
|
|
||||||
|
# A cancelled request must not commit anything. Cancelling
|
||||||
|
# releases the cache_key, so a replacement fetch for the same key
|
||||||
|
# may already be in flight or finished -- writing this response to
|
||||||
|
# the cache now would overwrite fresher data with the response
|
||||||
|
# nobody wanted. The worker has no way to abort the HTTP call, so
|
||||||
|
# this is where the work gets discarded.
|
||||||
|
with self._lock:
|
||||||
|
cancelled = request.status == FetchStatus.CANCELLED
|
||||||
|
if not cancelled:
|
||||||
|
# Claim the commit in the same critical section that read
|
||||||
|
# the status, so a cancel cannot slip in between the check
|
||||||
|
# and the cache write below. The write itself stays outside
|
||||||
|
# the lock: it serialises a multi-megabyte payload to the
|
||||||
|
# SD card, and holding the service lock across that would
|
||||||
|
# stall every submit, status query and cancel behind it.
|
||||||
|
request.commit_claimed = True
|
||||||
|
if cancelled:
|
||||||
|
logger.info(
|
||||||
|
"Discarding response for cancelled request %s; %s may "
|
||||||
|
"already belong to a replacement fetch",
|
||||||
|
request.id, request.cache_key
|
||||||
|
)
|
||||||
|
result = FetchResult(
|
||||||
|
request_id=request.id,
|
||||||
|
success=False,
|
||||||
|
error="cancelled",
|
||||||
|
fetch_time=time.time() - start_time,
|
||||||
|
retry_count=request.retry_count
|
||||||
|
)
|
||||||
|
return result
|
||||||
|
|
||||||
# Cache the data
|
# Cache the data
|
||||||
self.cache_manager.set(request.cache_key, data)
|
self.cache_manager.set(request.cache_key, data)
|
||||||
|
|
||||||
@@ -294,6 +416,11 @@ class BackgroundDataService:
|
|||||||
logger.error(f"Failed to fetch {request.sport} {request.year} data: {error_msg}")
|
logger.error(f"Failed to fetch {request.sport} {request.year} data: {error_msg}")
|
||||||
|
|
||||||
with self._lock:
|
with self._lock:
|
||||||
|
# Don't relabel a cancelled request. The callback gate in the
|
||||||
|
# finally block only suppresses CANCELLED, so promoting it to
|
||||||
|
# FAILED here delivered an error callback for a fetch nobody
|
||||||
|
# was waiting on any more.
|
||||||
|
if request.status != FetchStatus.CANCELLED:
|
||||||
request.status = FetchStatus.FAILED
|
request.status = FetchStatus.FAILED
|
||||||
request.error = error_msg
|
request.error = error_msg
|
||||||
|
|
||||||
@@ -308,9 +435,26 @@ class BackgroundDataService:
|
|||||||
finally:
|
finally:
|
||||||
# Store result and clean up
|
# Store result and clean up
|
||||||
with self._lock:
|
with self._lock:
|
||||||
|
result.final_status = request.status
|
||||||
self.completed_requests[request.id] = result
|
self.completed_requests[request.id] = result
|
||||||
if request.id in self.active_requests:
|
if request.id in self.active_requests:
|
||||||
del self.active_requests[request.id]
|
del self.active_requests[request.id]
|
||||||
|
# Stop accepting joiners and take the callback list in the same
|
||||||
|
# critical section. A submitter that arrives after this point
|
||||||
|
# finds no in-flight entry and either hits the cache (written
|
||||||
|
# above, before the result was built) or starts a fresh fetch --
|
||||||
|
# what it must never do is join a fetch whose callbacks have
|
||||||
|
# already run and then never be called.
|
||||||
|
if self._inflight_by_cache_key.get(request.cache_key) == request.id:
|
||||||
|
del self._inflight_by_cache_key[request.cache_key]
|
||||||
|
# A cancelled request delivers nothing: its joiners were told
|
||||||
|
# about a fetch that has been abandoned, and a replacement will
|
||||||
|
# call them via its own request.
|
||||||
|
if request.status == FetchStatus.CANCELLED:
|
||||||
|
callbacks = []
|
||||||
|
else:
|
||||||
|
callbacks = ([request.callback] if request.callback else [])
|
||||||
|
callbacks.extend(request.extra_callbacks)
|
||||||
|
|
||||||
# Update statistics
|
# Update statistics
|
||||||
if result.success:
|
if result.success:
|
||||||
@@ -327,15 +471,54 @@ class BackgroundDataService:
|
|||||||
# Periodic cleanup after storing result
|
# Periodic cleanup after storing result
|
||||||
self._cleanup_completed_requests()
|
self._cleanup_completed_requests()
|
||||||
|
|
||||||
# Call callback if provided
|
# Call every callback: the original submitter's and any that joined
|
||||||
if request.callback:
|
# this fetch. One raising must not stop the others being delivered.
|
||||||
|
for cb in callbacks:
|
||||||
try:
|
try:
|
||||||
request.callback(result)
|
cb(result)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error in callback for request {request.id}: {e}")
|
logger.error(f"Error in callback for request {request.id}: {e}")
|
||||||
|
|
||||||
|
# Released AFTER the loop, not inside it. Every callback here holds
|
||||||
|
# the same FetchResult, so releasing per-delivery handed the first
|
||||||
|
# one the data and every joiner `result.data is None` -- which is
|
||||||
|
# not a quiet degradation: they read `result.data.get('events')` and
|
||||||
|
# raise AttributeError, which this very loop catches and logs, so
|
||||||
|
# the symptom was one ERROR line and a manager that silently never
|
||||||
|
# got its schedule. Deduplication is the normal case, not a corner:
|
||||||
|
# a sport's recent, upcoming and live managers all ride one season
|
||||||
|
# fetch.
|
||||||
|
#
|
||||||
|
# Guarded on `callbacks`, because a request submitted without one
|
||||||
|
# has no other way to collect its payload than polling get_result().
|
||||||
|
# The old per-delivery release got that right by accident: an empty
|
||||||
|
# list never entered the loop body.
|
||||||
|
if callbacks:
|
||||||
|
self._release_payload(result)
|
||||||
|
request.result = None
|
||||||
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _release_payload(result: FetchResult) -> None:
|
||||||
|
"""Drop a delivered payload, keeping the result's status and timings.
|
||||||
|
|
||||||
|
Only called once EVERY callback has been handed the data -- callers
|
||||||
|
that joined an in-flight fetch share this object, so releasing between
|
||||||
|
deliveries strips the payload out from under the ones still queued.
|
||||||
|
Consumers read fetched data back from the cache under ``cache_key``;
|
||||||
|
the copy carried
|
||||||
|
here was pinning a parsed season schedule -- 946 games for NCAA
|
||||||
|
football, roughly a tenth of total RAM on a 1GB Pi -- in memory until
|
||||||
|
the hourly sweep.
|
||||||
|
|
||||||
|
The cache-hit path matters most: it runs once per update interval per
|
||||||
|
sport, mints a fresh request_id each time, and a memory-tier miss
|
||||||
|
re-parses the payload from disk. Those were genuinely separate copies
|
||||||
|
accumulating toward the 500-entry cap, not shared references.
|
||||||
|
"""
|
||||||
|
result.data = None
|
||||||
|
|
||||||
def _make_request_with_retry(self, request: FetchRequest) -> requests.Response:
|
def _make_request_with_retry(self, request: FetchRequest) -> requests.Response:
|
||||||
"""
|
"""
|
||||||
Make HTTP request with retry logic and exponential backoff.
|
Make HTTP request with retry logic and exponential backoff.
|
||||||
@@ -422,6 +605,8 @@ class BackgroundDataService:
|
|||||||
return self.active_requests[request_id].status
|
return self.active_requests[request_id].status
|
||||||
elif request_id in self.completed_requests:
|
elif request_id in self.completed_requests:
|
||||||
result = self.completed_requests[request_id]
|
result = self.completed_requests[request_id]
|
||||||
|
if result.final_status is not None:
|
||||||
|
return result.final_status
|
||||||
return FetchStatus.COMPLETED if result.success else FetchStatus.FAILED
|
return FetchStatus.COMPLETED if result.success else FetchStatus.FAILED
|
||||||
return None
|
return None
|
||||||
|
|
||||||
@@ -438,8 +623,22 @@ class BackgroundDataService:
|
|||||||
with self._lock:
|
with self._lock:
|
||||||
if request_id in self.active_requests:
|
if request_id in self.active_requests:
|
||||||
request = self.active_requests[request_id]
|
request = self.active_requests[request_id]
|
||||||
|
if request.commit_claimed:
|
||||||
|
# Too late: the worker holds an authorised commit. Report
|
||||||
|
# the failure rather than half-cancelling a request whose
|
||||||
|
# data is about to land in the cache.
|
||||||
|
logger.debug(
|
||||||
|
"Not cancelling %s: its response is already being "
|
||||||
|
"committed", request_id
|
||||||
|
)
|
||||||
|
return False
|
||||||
request.status = FetchStatus.CANCELLED
|
request.status = FetchStatus.CANCELLED
|
||||||
del self.active_requests[request_id]
|
del self.active_requests[request_id]
|
||||||
|
# Cancelling is the other way a request leaves active_requests,
|
||||||
|
# so the in-flight index has to be released here too or the key
|
||||||
|
# stays pointed at a request that no longer exists.
|
||||||
|
if self._inflight_by_cache_key.get(request.cache_key) == request_id:
|
||||||
|
del self._inflight_by_cache_key[request.cache_key]
|
||||||
logger.info(f"Cancelled request {request_id}")
|
logger.info(f"Cancelled request {request_id}")
|
||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|||||||
@@ -145,8 +145,7 @@ class SportsUpcoming(SportsCore):
|
|||||||
if (game['home_abbr'] in self.favorite_teams or
|
if (game['home_abbr'] in self.favorite_teams or
|
||||||
game['away_abbr'] in self.favorite_teams):
|
game['away_abbr'] in self.favorite_teams):
|
||||||
favorite_games_found += 1
|
favorite_games_found += 1
|
||||||
if self.show_odds:
|
# Odds are NOT fetched here -- see after selection below.
|
||||||
self._fetch_odds(game)
|
|
||||||
|
|
||||||
# Enhanced logging for debugging
|
# Enhanced logging for debugging
|
||||||
self.logger.info(f"Found {all_upcoming_games} total upcoming games in data")
|
self.logger.info(f"Found {all_upcoming_games} total upcoming games in data")
|
||||||
@@ -190,6 +189,20 @@ class SportsUpcoming(SportsCore):
|
|||||||
# Limit to the specified number of upcoming games
|
# Limit to the specified number of upcoming games
|
||||||
team_games = team_games[:self.upcoming_games_to_show]
|
team_games = team_games[:self.upcoming_games_to_show]
|
||||||
|
|
||||||
|
# Odds are fetched here, for the games that survived selection,
|
||||||
|
# rather than in the loop that collects them. That loop walks every
|
||||||
|
# upcoming game in the schedule window, and for a college league
|
||||||
|
# the window is enormous -- a live rig logged 946 upcoming games in
|
||||||
|
# one cycle and displayed 1 of them. The comment up there claimed
|
||||||
|
# odds were fetched "only for games that will be displayed", but
|
||||||
|
# the only narrowing it applied was show_favorite_teams_only, which
|
||||||
|
# is not the default; in the usual case nothing narrowed it at all
|
||||||
|
# and every game cost a separate ESPN request on a Pi that is also
|
||||||
|
# driving the panel.
|
||||||
|
if self.show_odds:
|
||||||
|
for game in team_games:
|
||||||
|
self._fetch_odds(game)
|
||||||
|
|
||||||
# Log changes or periodically
|
# Log changes or periodically
|
||||||
should_log = (
|
should_log = (
|
||||||
current_time - self.last_log_time >= self.log_interval or
|
current_time - self.last_log_time >= self.log_interval or
|
||||||
|
|||||||
+71
-10
@@ -12,6 +12,8 @@ Follows LEDMatrix configuration management patterns:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
import time
|
||||||
|
|
||||||
import requests
|
import requests
|
||||||
import json
|
import json
|
||||||
from typing import Dict, Any, Optional, List
|
from typing import Dict, Any, Optional, List
|
||||||
@@ -43,9 +45,34 @@ class BaseOddsManager:
|
|||||||
self.logger = logging.getLogger(__name__)
|
self.logger = logging.getLogger(__name__)
|
||||||
self.base_url = "https://sports.core.api.espn.com/v2/sports"
|
self.base_url = "https://sports.core.api.espn.com/v2/sports"
|
||||||
|
|
||||||
|
# This path used a bare requests.get, so it identified itself as
|
||||||
|
# python-requests/x.y -- the one thing ESPN is known to reject. Around
|
||||||
|
# 2026-08-04 it began 403ing browser strings and bare custom tokens
|
||||||
|
# alike; what it accepts is a token with a URL that says who is
|
||||||
|
# calling. Every other ESPN caller in the tree already sends this
|
||||||
|
# (src/common/api_helper.py, src/base_classes/data_sources.py); the
|
||||||
|
# odds path was simply missed, and it is the one whose failures cost
|
||||||
|
# the caller its whole update budget.
|
||||||
|
#
|
||||||
|
# Deliberately no retry adapter, unlike api_helper: retries multiply
|
||||||
|
# request_timeout, which is set to 5s precisely to stay inside that
|
||||||
|
# budget. One try, then the cooldown below.
|
||||||
|
self.session = requests.Session()
|
||||||
|
self.session.headers.update({
|
||||||
|
'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)',
|
||||||
|
'Accept': 'application/json',
|
||||||
|
})
|
||||||
|
|
||||||
# Configuration with defaults
|
# Configuration with defaults
|
||||||
self.update_interval = 3600 # 1 hour default
|
self.update_interval = 3600 # 1 hour default
|
||||||
self.request_timeout = 30 # 30 seconds default
|
# Well under the plugin executor's 30s operation budget. At 30s a
|
||||||
|
# single stalled ESPN request consumed the entire budget and the whole
|
||||||
|
# update() was killed -- and odds are fetched per live game, inside the
|
||||||
|
# live update loop, with show_odds defaulting on. Losing one game's
|
||||||
|
# odds beats losing the update that carries every game's score.
|
||||||
|
self.request_timeout = 5
|
||||||
|
# Set when a request fails; until then, skip the network entirely.
|
||||||
|
self._skip_network_until = 0.0
|
||||||
self.cache_ttl = 1800 # 30 minutes default
|
self.cache_ttl = 1800 # 30 minutes default
|
||||||
|
|
||||||
# Load configuration if available
|
# Load configuration if available
|
||||||
@@ -73,6 +100,14 @@ class BaseOddsManager:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.warning(f"Failed to load BaseOddsManager configuration: {e}")
|
self.logger.warning(f"Failed to load BaseOddsManager configuration: {e}")
|
||||||
|
|
||||||
|
# After a network failure, stop trying for this long and serve cache only.
|
||||||
|
# A short per-request timeout bounds one stall, but a full Sunday slate is
|
||||||
|
# ~16 games fetched in a loop, so 16 consecutive timeouts still blow the
|
||||||
|
# budget. When ESPN is unreachable it is unreachable for all of them, so
|
||||||
|
# the first failure is enough to know: skip the rest of this pass and try
|
||||||
|
# again shortly.
|
||||||
|
_FAILURE_COOLDOWN = 60.0
|
||||||
|
|
||||||
def get_odds(self, sport: str | None, league: str | None, event_id: str,
|
def get_odds(self, sport: str | None, league: str | None, event_id: str,
|
||||||
update_interval_seconds: int = None) -> Optional[Dict[str, Any]]:
|
update_interval_seconds: int = None) -> Optional[Dict[str, Any]]:
|
||||||
"""
|
"""
|
||||||
@@ -101,6 +136,16 @@ class BaseOddsManager:
|
|||||||
self.logger.info(f"Using cached odds from ESPN for {cache_key}")
|
self.logger.info(f"Using cached odds from ESPN for {cache_key}")
|
||||||
return cached_data
|
return cached_data
|
||||||
|
|
||||||
|
if time.monotonic() < self._skip_network_until:
|
||||||
|
# A recent request failed, so ESPN is very likely still unreachable.
|
||||||
|
# Returning now keeps the caller's update inside its time budget
|
||||||
|
# instead of paying the timeout again for every remaining game.
|
||||||
|
self.logger.debug(
|
||||||
|
"Skipping odds fetch for %s: a recent request failed, holding off "
|
||||||
|
"for another %.0fs", cache_key,
|
||||||
|
self._skip_network_until - time.monotonic())
|
||||||
|
return None
|
||||||
|
|
||||||
self.logger.info(f"Cache miss - fetching fresh odds from ESPN for {cache_key}")
|
self.logger.info(f"Cache miss - fetching fresh odds from ESPN for {cache_key}")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -117,10 +162,12 @@ class BaseOddsManager:
|
|||||||
url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds"
|
url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds"
|
||||||
self.logger.info(f"Requesting odds from URL: {url}")
|
self.logger.info(f"Requesting odds from URL: {url}")
|
||||||
|
|
||||||
response = requests.get(url, timeout=self.request_timeout)
|
response = self.session.get(url, timeout=self.request_timeout)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
raw_data = response.json()
|
raw_data = response.json()
|
||||||
|
|
||||||
|
self._skip_network_until = 0.0 # reachable again
|
||||||
|
|
||||||
self.logger.debug(f"Received raw odds data from ESPN: {json.dumps(raw_data, indent=2)}")
|
self.logger.debug(f"Received raw odds data from ESPN: {json.dumps(raw_data, indent=2)}")
|
||||||
|
|
||||||
odds_data = self._extract_espn_data(raw_data)
|
odds_data = self._extract_espn_data(raw_data)
|
||||||
@@ -140,7 +187,11 @@ class BaseOddsManager:
|
|||||||
return odds_data
|
return odds_data
|
||||||
|
|
||||||
except requests.exceptions.RequestException as e:
|
except requests.exceptions.RequestException as e:
|
||||||
self.logger.error(f"Error fetching odds from ESPN API for {cache_key}: {e}")
|
self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN
|
||||||
|
self.logger.error(
|
||||||
|
"Error fetching odds from ESPN API for %s: %s. Holding off on odds "
|
||||||
|
"for %.0fs so a slate of games does not pay this timeout each.",
|
||||||
|
cache_key, e, self._FAILURE_COOLDOWN)
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.")
|
self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.")
|
||||||
|
|
||||||
@@ -163,19 +214,25 @@ class BaseOddsManager:
|
|||||||
item = data["items"][0]
|
item = data["items"][0]
|
||||||
self.logger.debug(f"First item keys: {list(item.keys())}")
|
self.logger.debug(f"First item keys: {list(item.keys())}")
|
||||||
|
|
||||||
# The ESPN API returns odds data directly in the item, not in a providers array
|
# The ESPN API returns odds data directly in the item, not in a
|
||||||
# Extract the odds data directly from the item
|
# providers array. ESPN sends explicit JSON nulls for absent
|
||||||
|
# sides ("homeTeamOdds": null), so every level uses `or {}` —
|
||||||
|
# .get's default only applies when the key is missing entirely.
|
||||||
|
home = item.get("homeTeamOdds") or {}
|
||||||
|
away = item.get("awayTeamOdds") or {}
|
||||||
extracted_data = {
|
extracted_data = {
|
||||||
"details": item.get("details"),
|
"details": item.get("details"),
|
||||||
"over_under": item.get("overUnder"),
|
"over_under": item.get("overUnder"),
|
||||||
"spread": item.get("spread"),
|
"spread": item.get("spread"),
|
||||||
"home_team_odds": {
|
"home_team_odds": {
|
||||||
"money_line": item.get("homeTeamOdds", {}).get("moneyLine"),
|
"money_line": home.get("moneyLine"),
|
||||||
"spread_odds": item.get("homeTeamOdds", {}).get("current", {}).get("pointSpread", {}).get("value")
|
"spread_odds": ((home.get("current") or {})
|
||||||
|
.get("pointSpread") or {}).get("value")
|
||||||
},
|
},
|
||||||
"away_team_odds": {
|
"away_team_odds": {
|
||||||
"money_line": item.get("awayTeamOdds", {}).get("moneyLine"),
|
"money_line": away.get("moneyLine"),
|
||||||
"spread_odds": item.get("awayTeamOdds", {}).get("current", {}).get("pointSpread", {}).get("value")
|
"spread_odds": ((away.get("current") or {})
|
||||||
|
.get("pointSpread") or {}).get("value")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
self.logger.debug(f"Returning extracted odds data: {json.dumps(extracted_data, indent=2)}")
|
self.logger.debug(f"Returning extracted odds data: {json.dumps(extracted_data, indent=2)}")
|
||||||
@@ -260,7 +317,11 @@ class BaseOddsManager:
|
|||||||
Returns:
|
Returns:
|
||||||
Formatted odds summary string
|
Formatted odds summary string
|
||||||
"""
|
"""
|
||||||
if not self.is_odds_available(odds_data):
|
# Gate only on truly-empty / negative-cached data. is_odds_available
|
||||||
|
# deliberately ignores money lines (its callers decide whether to
|
||||||
|
# RENDER an odds widget), but a summary of money-line-only odds is
|
||||||
|
# still meaningful — the parts loop below handles them.
|
||||||
|
if not odds_data or odds_data.get('no_odds'):
|
||||||
return "No odds available"
|
return "No odds available"
|
||||||
|
|
||||||
parts = []
|
parts = []
|
||||||
|
|||||||
Vendored
+80
-1
@@ -14,6 +14,13 @@ import zlib
|
|||||||
from typing import Dict, Any, Optional, Protocol
|
from typing import Dict, Any, Optional, Protocol
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
|
# How old an abandoned write's temp file must be before the sweep removes it.
|
||||||
|
# A real write holds its temp file for milliseconds, so an hour is far beyond
|
||||||
|
# any in-flight write while still clearing the same day's debris. Deliberately
|
||||||
|
# not tied to the retention policies: those describe how long data stays
|
||||||
|
# useful, and a half-written file was never useful.
|
||||||
|
_ORPHAN_TEMP_MAX_AGE_SECONDS = 3600
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
class CacheStrategyProtocol(Protocol):
|
class CacheStrategyProtocol(Protocol):
|
||||||
@@ -112,6 +119,22 @@ class DiskCache:
|
|||||||
record_ts = None
|
record_ts = None
|
||||||
|
|
||||||
now = time.time()
|
now = time.time()
|
||||||
|
|
||||||
|
# An explicit per-entry ttl wins over the caller's max_age. The
|
||||||
|
# caller that wrote the record knows what its data is; max_age is
|
||||||
|
# inferred from substrings in the key ("live", "odds", "stock") and
|
||||||
|
# is only a fallback for records that never said. Until now the ttl
|
||||||
|
# was stored and ignored, so `set(key, data, ttl=...)` did nothing
|
||||||
|
# at all -- 48 plugin call sites and 4 in the core were writing a
|
||||||
|
# number no read path consulted.
|
||||||
|
effective_max_age = max_age
|
||||||
|
if isinstance(record, dict):
|
||||||
|
stored_ttl = record.get('ttl')
|
||||||
|
if isinstance(stored_ttl, (int, float)) and not isinstance(stored_ttl, bool) \
|
||||||
|
and stored_ttl >= 0:
|
||||||
|
effective_max_age = stored_ttl
|
||||||
|
max_age = effective_max_age
|
||||||
|
|
||||||
# max_age=None means "never expires" (mirrors MemoryCache and the
|
# max_age=None means "never expires" (mirrors MemoryCache and the
|
||||||
# cache_manager docstring). Guard it explicitly — otherwise the
|
# cache_manager docstring). Guard it explicitly — otherwise the
|
||||||
# comparison below raises TypeError and the record is treated as a
|
# comparison below raises TypeError and the record is treated as a
|
||||||
@@ -331,6 +354,23 @@ class DiskCache:
|
|||||||
"""Get the cache directory path."""
|
"""Get the cache directory path."""
|
||||||
return self.cache_dir
|
return self.cache_dir
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _is_orphaned_temp(filename: str) -> bool:
|
||||||
|
"""Whether a name is one of set()'s temp files rather than real data.
|
||||||
|
|
||||||
|
Matches only what this class creates: mkstemp with a prefix of
|
||||||
|
".<cache filename>." , so ".weather.json.a1b2c3d4". The shape is
|
||||||
|
checked rather than just the leading dot, because this predicate
|
||||||
|
deletes things -- a stray dotfile someone left in the cache directory
|
||||||
|
is not ours to remove, and a completed ".json" never is either.
|
||||||
|
"""
|
||||||
|
if not filename.startswith('.') or filename.endswith('.json'):
|
||||||
|
return False
|
||||||
|
head, sep, suffix = filename.rpartition('.json.')
|
||||||
|
# head is the key (non-empty after the leading dot), suffix is
|
||||||
|
# mkstemp's random component.
|
||||||
|
return bool(sep) and len(head) > 1 and bool(suffix)
|
||||||
|
|
||||||
def cleanup_expired_files(self, cache_strategy: CacheStrategyProtocol, retention_policies: Dict[str, int]) -> Dict[str, Any]:
|
def cleanup_expired_files(self, cache_strategy: CacheStrategyProtocol, retention_policies: Dict[str, int]) -> Dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
Clean up expired cache files based on retention policies.
|
Clean up expired cache files based on retention policies.
|
||||||
@@ -365,12 +405,51 @@ class DiskCache:
|
|||||||
try:
|
try:
|
||||||
with self._lock:
|
with self._lock:
|
||||||
# Get snapshot of files while holding lock briefly
|
# Get snapshot of files while holding lock briefly
|
||||||
filenames = [f for f in os.listdir(self.cache_dir) if f.endswith('.json')]
|
entries = os.listdir(self.cache_dir)
|
||||||
except OSError as list_error:
|
except OSError as list_error:
|
||||||
self.logger.error("Error listing cache directory %s: %s", self.cache_dir, list_error, exc_info=True)
|
self.logger.error("Error listing cache directory %s: %s", self.cache_dir, list_error, exc_info=True)
|
||||||
stats['errors'] += 1
|
stats['errors'] += 1
|
||||||
return stats
|
return stats
|
||||||
|
|
||||||
|
filenames = [f for f in entries if f.endswith('.json')]
|
||||||
|
|
||||||
|
# Sweep temp files abandoned by a write that never finished. set()
|
||||||
|
# removes its own in a finally, so these are the ones where the
|
||||||
|
# process died between mkstemp and os.replace -- a SIGKILL, a lost
|
||||||
|
# restart race, a power cut. Nothing ever collected them: they are
|
||||||
|
# named ".<key>.json.<random>", and the scan above only matches
|
||||||
|
# names ending in .json, so they accumulated indefinitely. Measured
|
||||||
|
# on a live rig: 76 files, 1,050 MB, 81% of the whole cache
|
||||||
|
# directory, the oldest six months old.
|
||||||
|
stats['orphan_temp_files_deleted'] = 0
|
||||||
|
for filename in (f for f in entries if self._is_orphaned_temp(f)):
|
||||||
|
# Counted as scanned like any other candidate, so files_deleted
|
||||||
|
# can never exceed files_scanned and the summary line reads
|
||||||
|
# honestly ("77/8864", not "77/0").
|
||||||
|
stats['files_scanned'] += 1
|
||||||
|
path = os.path.join(self.cache_dir, filename)
|
||||||
|
try:
|
||||||
|
# An in-flight write lives for milliseconds, so anything
|
||||||
|
# this old is certainly abandoned rather than in progress.
|
||||||
|
if (current_time - os.path.getmtime(path)) <= _ORPHAN_TEMP_MAX_AGE_SECONDS:
|
||||||
|
continue
|
||||||
|
with self._lock:
|
||||||
|
size = os.path.getsize(path)
|
||||||
|
os.remove(path)
|
||||||
|
stats['files_deleted'] += 1
|
||||||
|
stats['orphan_temp_files_deleted'] += 1
|
||||||
|
stats['space_freed_bytes'] += size
|
||||||
|
except FileNotFoundError:
|
||||||
|
continue # another sweep got there first
|
||||||
|
except OSError as e:
|
||||||
|
stats['errors'] += 1
|
||||||
|
self.logger.warning("Error deleting orphaned temp file %s: %s", filename, e)
|
||||||
|
|
||||||
|
if stats['orphan_temp_files_deleted']:
|
||||||
|
self.logger.info(
|
||||||
|
"Removed %d abandoned cache temp file(s)",
|
||||||
|
stats['orphan_temp_files_deleted'])
|
||||||
|
|
||||||
# Process files outside the lock to avoid blocking get/set operations
|
# Process files outside the lock to avoid blocking get/set operations
|
||||||
for filename in filenames:
|
for filename in filenames:
|
||||||
stats['files_scanned'] += 1
|
stats['files_scanned'] += 1
|
||||||
|
|||||||
Vendored
+85
-16
@@ -4,11 +4,58 @@ Memory Cache
|
|||||||
Handles in-memory caching with TTL support, size limits, and automatic cleanup.
|
Handles in-memory caching with TTL support, size limits, and automatic cleanup.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
import time
|
import time
|
||||||
import threading
|
import threading
|
||||||
import logging
|
import logging
|
||||||
from typing import Dict, Any, Optional
|
from typing import Dict, Any, Optional
|
||||||
|
|
||||||
|
# Historical fixed ceiling, kept as the fallback when RAM cannot be read.
|
||||||
|
DEFAULT_MAX_SIZE = 1000
|
||||||
|
|
||||||
|
|
||||||
|
def _total_memory_mb() -> Optional[float]:
|
||||||
|
"""Physical RAM in MB, or None where /proc/meminfo is unavailable."""
|
||||||
|
try:
|
||||||
|
with open('/proc/meminfo', 'r', encoding='utf-8') as fh:
|
||||||
|
for line in fh:
|
||||||
|
if line.startswith('MemTotal:'):
|
||||||
|
return int(line.split()[1]) / 1024
|
||||||
|
except (OSError, ValueError, IndexError):
|
||||||
|
return None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def default_max_size() -> int:
|
||||||
|
"""Entry ceiling scaled to this machine's RAM.
|
||||||
|
|
||||||
|
One fixed ceiling cannot serve both a 512 MB Pi Zero 2 W and an 8 GB Pi 5.
|
||||||
|
Entries here are parsed API payloads that routinely run tens of kilobytes
|
||||||
|
each, so a thousand of them is a comfortable cache on a large board and a
|
||||||
|
substantial fraction of total RAM on a small one — where the process
|
||||||
|
competing for that RAM is also driving the panel. Set
|
||||||
|
LEDMATRIX_CACHE_MAX_ENTRIES to override.
|
||||||
|
"""
|
||||||
|
override = os.environ.get('LEDMATRIX_CACHE_MAX_ENTRIES')
|
||||||
|
if override:
|
||||||
|
try:
|
||||||
|
value = int(override)
|
||||||
|
if value > 0:
|
||||||
|
return value
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
total_mb = _total_memory_mb()
|
||||||
|
if total_mb is None:
|
||||||
|
return DEFAULT_MAX_SIZE
|
||||||
|
if total_mb < 1536: # 512 MB and 1 GB boards
|
||||||
|
return 150
|
||||||
|
if total_mb < 3072: # 2 GB
|
||||||
|
return 400
|
||||||
|
if total_mb < 6144: # 4 GB
|
||||||
|
return 800
|
||||||
|
return 1500 # 8 GB and up
|
||||||
|
|
||||||
|
|
||||||
class MemoryCache:
|
class MemoryCache:
|
||||||
"""Manages in-memory cache with TTL and size limits."""
|
"""Manages in-memory cache with TTL and size limits."""
|
||||||
@@ -57,6 +104,16 @@ class MemoryCache:
|
|||||||
if timestamp is None:
|
if timestamp is None:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
# An explicit per-entry ttl wins over the caller's max_age, matching
|
||||||
|
# DiskCache. max_age is inferred from substrings in the key and is
|
||||||
|
# only a fallback for records that did not say what they wanted.
|
||||||
|
record = self._cache[key]
|
||||||
|
if isinstance(record, dict):
|
||||||
|
stored_ttl = record.get('ttl')
|
||||||
|
if isinstance(stored_ttl, (int, float)) and not isinstance(stored_ttl, bool) \
|
||||||
|
and stored_ttl >= 0:
|
||||||
|
max_age = stored_ttl
|
||||||
|
|
||||||
# Check expiration
|
# Check expiration
|
||||||
if max_age is not None and (now - timestamp) > max_age:
|
if max_age is not None and (now - timestamp) > max_age:
|
||||||
# Expired - remove it
|
# Expired - remove it
|
||||||
@@ -77,6 +134,32 @@ class MemoryCache:
|
|||||||
with self._lock:
|
with self._lock:
|
||||||
self._cache[key] = value
|
self._cache[key] = value
|
||||||
self._timestamps[key] = time.time()
|
self._timestamps[key] = time.time()
|
||||||
|
# Enforce the ceiling here rather than leaving it to the periodic
|
||||||
|
# cleanup, which only runs every cleanup_interval seconds (300 by
|
||||||
|
# default). A burst of inserts between two sweeps could otherwise
|
||||||
|
# take the cache far past _max_size, which is the memory growth this
|
||||||
|
# limit exists to prevent -- and on a 1GB board that is the
|
||||||
|
# difference between a bounded cache and an unreachable Pi.
|
||||||
|
self._evict_over_limit_locked()
|
||||||
|
|
||||||
|
def _evict_over_limit_locked(self) -> int:
|
||||||
|
"""Drop oldest entries until the cache is within _max_size.
|
||||||
|
|
||||||
|
Caller must hold self._lock. Returns the number of entries removed.
|
||||||
|
"""
|
||||||
|
excess = len(self._cache) - self._max_size
|
||||||
|
if excess <= 0:
|
||||||
|
return 0
|
||||||
|
oldest = sorted(
|
||||||
|
self._timestamps.items(),
|
||||||
|
key=lambda item: float(item[1]) if isinstance(item[1], (int, float)) else 0.0
|
||||||
|
)
|
||||||
|
removed = 0
|
||||||
|
for key, _ in oldest[:excess]:
|
||||||
|
self._cache.pop(key, None)
|
||||||
|
self._timestamps.pop(key, None)
|
||||||
|
removed += 1
|
||||||
|
return removed
|
||||||
|
|
||||||
def clear(self, key: Optional[str] = None) -> None:
|
def clear(self, key: Optional[str] = None) -> None:
|
||||||
"""
|
"""
|
||||||
@@ -133,22 +216,8 @@ class MemoryCache:
|
|||||||
self._timestamps.pop(key, None)
|
self._timestamps.pop(key, None)
|
||||||
removed_count += 1
|
removed_count += 1
|
||||||
|
|
||||||
# Enforce size limit by removing oldest entries if cache is too large
|
# Same ceiling enforcement set() uses, so the two cannot drift.
|
||||||
if len(self._cache) > self._max_size:
|
removed_count += self._evict_over_limit_locked()
|
||||||
# Sort by timestamp (oldest first)
|
|
||||||
sorted_entries = sorted(
|
|
||||||
self._timestamps.items(),
|
|
||||||
key=lambda x: float(x[1]) if isinstance(x[1], (int, float)) else 0
|
|
||||||
)
|
|
||||||
|
|
||||||
# Remove oldest entries until we're under the limit
|
|
||||||
excess_count = len(self._cache) - self._max_size
|
|
||||||
for i in range(excess_count):
|
|
||||||
if i < len(sorted_entries):
|
|
||||||
key = sorted_entries[i][0]
|
|
||||||
self._cache.pop(key, None)
|
|
||||||
self._timestamps.pop(key, None)
|
|
||||||
removed_count += 1
|
|
||||||
|
|
||||||
self._last_cleanup = current_time
|
self._last_cleanup = current_time
|
||||||
|
|
||||||
|
|||||||
+48
-5
@@ -33,7 +33,7 @@ import logging
|
|||||||
import threading
|
import threading
|
||||||
import tempfile
|
import tempfile
|
||||||
from src.exceptions import CacheError
|
from src.exceptions import CacheError
|
||||||
from src.cache.memory_cache import MemoryCache
|
from src.cache.memory_cache import MemoryCache, default_max_size
|
||||||
from src.cache.disk_cache import DiskCache
|
from src.cache.disk_cache import DiskCache
|
||||||
from src.cache.cache_strategy import CacheStrategy
|
from src.cache.cache_strategy import CacheStrategy
|
||||||
from src.cache.cache_metrics import CacheMetrics
|
from src.cache.cache_metrics import CacheMetrics
|
||||||
@@ -47,6 +47,20 @@ from src.cache.disk_cache import DateTimeEncoder # noqa: F401 - deliberate re-e
|
|||||||
class CacheManager:
|
class CacheManager:
|
||||||
"""Manages caching of API responses to reduce API calls."""
|
"""Manages caching of API responses to reduce API calls."""
|
||||||
|
|
||||||
|
# Which cache directories already have a cleanup thread in this process.
|
||||||
|
#
|
||||||
|
# The sweep is directory-scoped work -- it lists a directory and deletes
|
||||||
|
# from it -- so one per directory is the right number no matter how many
|
||||||
|
# managers exist. Nothing enforced that before: every instance started its
|
||||||
|
# own, and because the loop closes over `self`, a discarded manager could
|
||||||
|
# never be collected and its thread woke to re-scan the same directory
|
||||||
|
# every 24 hours for the life of the process. Startup validation runs
|
||||||
|
# twice and built a throwaway manager each time, so a display process
|
||||||
|
# carried three threads for one cache.
|
||||||
|
_cleanup_owners: Dict[str, 'CacheManager'] = {}
|
||||||
|
_cleanup_owners_lock = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
# Initialize logger first
|
# Initialize logger first
|
||||||
self.logger: logging.Logger = get_logger(__name__)
|
self.logger: logging.Logger = get_logger(__name__)
|
||||||
@@ -70,7 +84,9 @@ class CacheManager:
|
|||||||
self.logger.warning("ConfigManager not available, using default cache intervals")
|
self.logger.warning("ConfigManager not available, using default cache intervals")
|
||||||
|
|
||||||
# Initialize cache components using composition
|
# Initialize cache components using composition
|
||||||
self._memory_cache_component = MemoryCache(max_size=1000, cleanup_interval=300.0)
|
self._memory_cache_component = MemoryCache(
|
||||||
|
max_size=default_max_size(), cleanup_interval=300.0
|
||||||
|
)
|
||||||
self._disk_cache_component = DiskCache(cache_dir=self.cache_dir, logger=self.logger)
|
self._disk_cache_component = DiskCache(cache_dir=self.cache_dir, logger=self.logger)
|
||||||
self._strategy_component = CacheStrategy(config_manager=self.config_manager, logger=self.logger)
|
self._strategy_component = CacheStrategy(config_manager=self.config_manager, logger=self.logger)
|
||||||
self._metrics_component = CacheMetrics(logger=self.logger)
|
self._metrics_component = CacheMetrics(logger=self.logger)
|
||||||
@@ -594,8 +610,10 @@ class CacheManager:
|
|||||||
Args:
|
Args:
|
||||||
key: Cache key
|
key: Cache key
|
||||||
data: Data to cache
|
data: Data to cache
|
||||||
ttl: Optional time-to-live in seconds (stored for compatibility but
|
ttl: Time-to-live in seconds for this entry. Takes precedence over
|
||||||
expiration is still controlled via max_age when reading)
|
the max_age a reader would otherwise apply, which is inferred
|
||||||
|
from the key and is only a fallback for entries that did not
|
||||||
|
say. Omit it to keep that inferred behaviour.
|
||||||
"""
|
"""
|
||||||
cache_data = {
|
cache_data = {
|
||||||
'data': data,
|
'data': data,
|
||||||
@@ -716,11 +734,29 @@ class CacheManager:
|
|||||||
}
|
}
|
||||||
|
|
||||||
def start_cleanup_thread(self) -> None:
|
def start_cleanup_thread(self) -> None:
|
||||||
"""Start background thread for periodic disk cache cleanup."""
|
"""Start background thread for periodic disk cache cleanup.
|
||||||
|
|
||||||
|
At most one thread per cache directory per process: the sweep is
|
||||||
|
directory-scoped, so a second one only duplicates the scan.
|
||||||
|
"""
|
||||||
if self._cleanup_thread and self._cleanup_thread.is_alive():
|
if self._cleanup_thread and self._cleanup_thread.is_alive():
|
||||||
self.logger.debug("Cleanup thread already running")
|
self.logger.debug("Cleanup thread already running")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
with CacheManager._cleanup_owners_lock:
|
||||||
|
owner = CacheManager._cleanup_owners.get(self.cache_dir)
|
||||||
|
if owner is not None and owner is not self:
|
||||||
|
thread = owner._cleanup_thread
|
||||||
|
if thread is not None and thread.is_alive():
|
||||||
|
self.logger.debug(
|
||||||
|
"Cleanup thread for %s already owned by another cache "
|
||||||
|
"manager in this process; not starting a second",
|
||||||
|
self.cache_dir)
|
||||||
|
return
|
||||||
|
# The owner's thread died or was stopped -- take over.
|
||||||
|
CacheManager._cleanup_owners[self.cache_dir] = self
|
||||||
|
|
||||||
|
|
||||||
def cleanup_loop():
|
def cleanup_loop():
|
||||||
"""Background loop that runs cleanup periodically."""
|
"""Background loop that runs cleanup periodically."""
|
||||||
self.logger.info("Disk cache cleanup thread started (interval: %d hours)",
|
self.logger.info("Disk cache cleanup thread started (interval: %d hours)",
|
||||||
@@ -768,6 +804,13 @@ class CacheManager:
|
|||||||
Signals the thread to stop and waits for it to finish (with timeout).
|
Signals the thread to stop and waits for it to finish (with timeout).
|
||||||
This allows for clean shutdown during testing or application termination.
|
This allows for clean shutdown during testing or application termination.
|
||||||
"""
|
"""
|
||||||
|
# Release ownership first and unconditionally, so a manager that never
|
||||||
|
# started a thread (or whose thread already exited) cannot keep the
|
||||||
|
# directory claimed and block a live manager from sweeping it.
|
||||||
|
with CacheManager._cleanup_owners_lock:
|
||||||
|
if CacheManager._cleanup_owners.get(self.cache_dir) is self:
|
||||||
|
del CacheManager._cleanup_owners[self.cache_dir]
|
||||||
|
|
||||||
if not self._cleanup_thread or not self._cleanup_thread.is_alive():
|
if not self._cleanup_thread or not self._cleanup_thread.is_alive():
|
||||||
self.logger.debug("Cleanup thread not running")
|
self.logger.debug("Cleanup thread not running")
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -273,19 +273,34 @@ class APIHelper:
|
|||||||
"""
|
"""
|
||||||
Clear cache data.
|
Clear cache data.
|
||||||
|
|
||||||
|
Uses CacheManager's real surface (clear_cache / delete /
|
||||||
|
list_cache_files); safely no-ops on managers without it. The old
|
||||||
|
implementation guarded on a nonexistent ``clear`` method, so it
|
||||||
|
silently never cleared anything.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
pattern: Optional pattern to match cache keys
|
pattern: Optional substring to match cache keys; only matching
|
||||||
|
entries are deleted.
|
||||||
"""
|
"""
|
||||||
if self.cache_manager:
|
if not self.cache_manager:
|
||||||
if hasattr(self.cache_manager, 'clear'):
|
return
|
||||||
if pattern:
|
if pattern:
|
||||||
# Clear only keys matching pattern
|
if (hasattr(self.cache_manager, 'list_cache_files')
|
||||||
keys = self.cache_manager.keys()
|
and hasattr(self.cache_manager, 'delete')):
|
||||||
for key in keys:
|
for entry in self.cache_manager.list_cache_files():
|
||||||
if pattern in key:
|
key = entry.get('key') if isinstance(entry, dict) else None
|
||||||
|
if key and pattern in key:
|
||||||
self.cache_manager.delete(key)
|
self.cache_manager.delete(key)
|
||||||
else:
|
else:
|
||||||
|
self.logger.debug(
|
||||||
|
"Cache manager lacks list_cache_files/delete; "
|
||||||
|
"cannot clear by pattern")
|
||||||
|
elif hasattr(self.cache_manager, 'clear_cache'):
|
||||||
|
self.cache_manager.clear_cache()
|
||||||
|
elif hasattr(self.cache_manager, 'clear'):
|
||||||
self.cache_manager.clear()
|
self.cache_manager.clear()
|
||||||
|
else:
|
||||||
|
self.logger.debug("Cache manager exposes no clear method; no-op")
|
||||||
|
|
||||||
def _get_from_cache(self, key: str) -> Optional[Any]:
|
def _get_from_cache(self, key: str) -> Optional[Any]:
|
||||||
"""Get data from cache."""
|
"""Get data from cache."""
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ Handles configuration management and validation for LED matrix plugins.
|
|||||||
Extracted from LEDMatrix core to provide reusable functionality for plugins.
|
Extracted from LEDMatrix core to provide reusable functionality for plugins.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import copy
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -160,17 +161,20 @@ class ConfigHelper:
|
|||||||
override_config: Configuration to merge in (takes precedence)
|
override_config: Configuration to merge in (takes precedence)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Merged configuration dictionary
|
Merged configuration dictionary (fully independent of both
|
||||||
|
inputs — a shallow copy would alias un-overridden nested dicts,
|
||||||
|
so mutating the result would mutate the caller's base config).
|
||||||
"""
|
"""
|
||||||
merged = base_config.copy()
|
merged = copy.deepcopy(base_config)
|
||||||
|
|
||||||
for key, value in override_config.items():
|
for key, value in override_config.items():
|
||||||
if key in merged and isinstance(merged[key], dict) and isinstance(value, dict):
|
if key in merged and isinstance(merged[key], dict) and isinstance(value, dict):
|
||||||
# Recursively merge nested dictionaries
|
# Recursively merge nested dictionaries
|
||||||
merged[key] = self.merge_configs(merged[key], value)
|
merged[key] = self.merge_configs(merged[key], value)
|
||||||
else:
|
else:
|
||||||
# Override with new value
|
# Override with new value — deep-copied so mutating the
|
||||||
merged[key] = value
|
# merged result can't reach back into override_config.
|
||||||
|
merged[key] = copy.deepcopy(value)
|
||||||
|
|
||||||
return merged
|
return merged
|
||||||
|
|
||||||
|
|||||||
@@ -115,18 +115,14 @@ class DisplayHelper:
|
|||||||
if home_logo and away_logo:
|
if home_logo and away_logo:
|
||||||
self._draw_logos(main_img, home_logo, away_logo)
|
self._draw_logos(main_img, home_logo, away_logo)
|
||||||
|
|
||||||
# Draw status/period text (top center)
|
# Draw one combined top line (period/status/clock all share y=1 —
|
||||||
if status_text or period_text:
|
# drawing them separately overprinted each other).
|
||||||
status_display = f"{period_text} {status_text}".strip()
|
top_line = " ".join(p for p in [period_text, status_text, clock] if p)
|
||||||
if status_display:
|
if top_line:
|
||||||
self._draw_centered_text(draw, status_display,
|
self._draw_centered_text(draw, top_line,
|
||||||
fonts.get('time', fonts.get('status')),
|
fonts.get('time', fonts.get('status')),
|
||||||
y_position=1)
|
y_position=1)
|
||||||
|
|
||||||
# Draw clock if available
|
|
||||||
if clock:
|
|
||||||
self._draw_centered_text(draw, clock, fonts.get('time'), y_position=1)
|
|
||||||
|
|
||||||
# Draw scores (center)
|
# Draw scores (center)
|
||||||
score_text = f"{away_score}-{home_score}"
|
score_text = f"{away_score}-{home_score}"
|
||||||
self._draw_centered_text(draw, score_text, fonts.get('score'),
|
self._draw_centered_text(draw, score_text, fonts.get('score'),
|
||||||
@@ -153,12 +149,18 @@ class DisplayHelper:
|
|||||||
"""
|
"""
|
||||||
Draw a ticker/scrolling text layout.
|
Draw a ticker/scrolling text layout.
|
||||||
|
|
||||||
|
Renders a single static frame with the text at the left edge; the
|
||||||
|
caller advances the scroll by re-rendering or shifting. The
|
||||||
|
scroll_speed parameter is accepted for API compatibility but does
|
||||||
|
not affect this frame. (Previously the text was drawn starting at
|
||||||
|
x=display_width — entirely off-canvas — so every frame was blank.)
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
text: Text to display
|
text: Text to display
|
||||||
font: Font to use
|
font: Font to use
|
||||||
background_color: Background color
|
background_color: Background color
|
||||||
text_color: Text color
|
text_color: Text color
|
||||||
scroll_speed: Pixels to scroll per frame
|
scroll_speed: Accepted for compatibility; unused per-frame
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
PIL Image with ticker layout
|
PIL Image with ticker layout
|
||||||
@@ -166,11 +168,7 @@ class DisplayHelper:
|
|||||||
img = self.create_base_image(background_color)
|
img = self.create_base_image(background_color)
|
||||||
draw = ImageDraw.Draw(img)
|
draw = ImageDraw.Draw(img)
|
||||||
|
|
||||||
# Start text off-screen to the right
|
self._draw_text_with_outline(draw, text, (0, self.display_height // 2 - 6),
|
||||||
x_position = self.display_width
|
|
||||||
|
|
||||||
# Draw text
|
|
||||||
self._draw_text_with_outline(draw, text, (x_position, self.display_height // 2 - 6),
|
|
||||||
font, fill=text_color)
|
font, fill=text_color)
|
||||||
|
|
||||||
return img
|
return img
|
||||||
@@ -214,15 +212,9 @@ class DisplayHelper:
|
|||||||
Returns:
|
Returns:
|
||||||
PIL Image with error message
|
PIL Image with error message
|
||||||
"""
|
"""
|
||||||
img = self.create_base_image((50, 0, 0)) # Dark red background
|
# Dark red background, white text
|
||||||
|
|
||||||
# Use default font
|
|
||||||
font = ImageFont.load_default()
|
font = ImageFont.load_default()
|
||||||
|
return self.draw_centered_text(message, font, (50, 0, 0), (255, 255, 255))
|
||||||
# Draw centered error message
|
|
||||||
self._draw_centered_text(message, font, (50, 0, 0), (255, 255, 255))
|
|
||||||
|
|
||||||
return img
|
|
||||||
|
|
||||||
def draw_no_data_message(self, message: str = "No Data") -> Image.Image:
|
def draw_no_data_message(self, message: str = "No Data") -> Image.Image:
|
||||||
"""
|
"""
|
||||||
@@ -234,11 +226,8 @@ class DisplayHelper:
|
|||||||
Returns:
|
Returns:
|
||||||
PIL Image with no data message
|
PIL Image with no data message
|
||||||
"""
|
"""
|
||||||
img = self.create_base_image((0, 0, 0))
|
|
||||||
font = ImageFont.load_default()
|
font = ImageFont.load_default()
|
||||||
self._draw_centered_text(message, font, (0, 0, 0), (150, 150, 150))
|
return self.draw_centered_text(message, font, (0, 0, 0), (150, 150, 150))
|
||||||
|
|
||||||
return img
|
|
||||||
|
|
||||||
def get_display_dimensions(self) -> Tuple[int, int]:
|
def get_display_dimensions(self) -> Tuple[int, int]:
|
||||||
"""
|
"""
|
||||||
|
|||||||
+111
-9
@@ -6,6 +6,8 @@ Extracted from LEDMatrix core to provide reusable functionality for plugins.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Dict, List, Optional, Union
|
from typing import Dict, List, Optional, Union
|
||||||
|
|
||||||
@@ -19,6 +21,10 @@ from src.common.permission_utils import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Well above any real team logo; bounds what a remote URL can write to disk.
|
||||||
|
MAX_LOGO_BYTES = 10 * 1024 * 1024
|
||||||
|
|
||||||
|
|
||||||
class LogoHelper:
|
class LogoHelper:
|
||||||
"""
|
"""
|
||||||
Helper class for logo loading, caching, and resizing.
|
Helper class for logo loading, caching, and resizing.
|
||||||
@@ -137,8 +143,10 @@ class LogoHelper:
|
|||||||
"""
|
"""
|
||||||
logo_path = Path(logo_path)
|
logo_path = Path(logo_path)
|
||||||
|
|
||||||
# Try to load existing logo first
|
# Try to load existing logo first. A placeholder written by a previous
|
||||||
if logo_path.exists():
|
# failed download does not count: it wears the real logo's filename, so
|
||||||
|
# trusting the file's existence is what left teams as grey boxes.
|
||||||
|
if logo_path.exists() and not self._is_stale_placeholder(logo_path):
|
||||||
return self.load_logo(team_abbr, logo_path, max_width, max_height)
|
return self.load_logo(team_abbr, logo_path, max_width, max_height)
|
||||||
|
|
||||||
# Download if URL provided and file doesn't exist
|
# Download if URL provided and file doesn't exist
|
||||||
@@ -146,13 +154,61 @@ class LogoHelper:
|
|||||||
try:
|
try:
|
||||||
self.logger.info(f"Downloading logo for {team_abbr} from {logo_url}")
|
self.logger.info(f"Downloading logo for {team_abbr} from {logo_url}")
|
||||||
self._download_logo(logo_url, logo_path)
|
self._download_logo(logo_url, logo_path)
|
||||||
|
# The file on disk just changed. Any cached image for it is the
|
||||||
|
# placeholder we came here to replace, and load_logo() answers
|
||||||
|
# from the cache before touching the disk -- so without this the
|
||||||
|
# real logo would not appear until the process restarted.
|
||||||
|
self._invalidate_cached_logo(team_abbr, logo_path)
|
||||||
return self.load_logo(team_abbr, logo_path, max_width, max_height)
|
return self.load_logo(team_abbr, logo_path, max_width, max_height)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(f"Failed to download logo for {team_abbr}: {e}")
|
self.logger.error(f"Failed to download logo for {team_abbr}: {e}")
|
||||||
|
# The retry failed, so restart the back-off. The stale
|
||||||
|
# placeholder is still on disk with its old timestamp, and
|
||||||
|
# leaving it there means the next call retries immediately --
|
||||||
|
# a download attempt per call, which is what the back-off
|
||||||
|
# exists to prevent.
|
||||||
|
self._refresh_stale_placeholder(logo_path)
|
||||||
|
|
||||||
# Create placeholder if all else fails
|
# Create placeholder if all else fails
|
||||||
return self._create_placeholder_logo(team_abbr, max_width, max_height)
|
return self._create_placeholder_logo(team_abbr, max_width, max_height)
|
||||||
|
|
||||||
|
def _invalidate_cached_logo(self, team_abbr: str, logo_path: Path) -> None:
|
||||||
|
"""Drop every cached size of one logo after its file changed on disk."""
|
||||||
|
prefix = f"{team_abbr}_{logo_path}_"
|
||||||
|
for key in [k for k in self._logo_cache if k.startswith(prefix)]:
|
||||||
|
self._logo_cache.pop(key, None)
|
||||||
|
if key in self._cache_order:
|
||||||
|
self._cache_order.remove(key)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _refresh_stale_placeholder(logo_path: Path) -> None:
|
||||||
|
"""Restart the retry back-off after a failed download attempt."""
|
||||||
|
try:
|
||||||
|
from src.logo_downloader import refresh_placeholder_timestamp
|
||||||
|
except ImportError:
|
||||||
|
return
|
||||||
|
refresh_placeholder_timestamp(logo_path)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _is_stale_placeholder(logo_path: Path) -> bool:
|
||||||
|
"""True if the file is a placeholder old enough to be worth retrying.
|
||||||
|
|
||||||
|
Imported lazily so this module keeps working against a core build whose
|
||||||
|
logo_downloader predates placeholder marking.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from src.logo_downloader import (
|
||||||
|
PLACEHOLDER_RETRY_SECONDS,
|
||||||
|
is_placeholder_logo,
|
||||||
|
placeholder_age_seconds,
|
||||||
|
)
|
||||||
|
except ImportError:
|
||||||
|
return False
|
||||||
|
if not is_placeholder_logo(logo_path):
|
||||||
|
return False
|
||||||
|
age = placeholder_age_seconds(logo_path)
|
||||||
|
return age is None or age >= PLACEHOLDER_RETRY_SECONDS
|
||||||
|
|
||||||
def get_logo_variations(self, team_abbr: str) -> List[str]:
|
def get_logo_variations(self, team_abbr: str) -> List[str]:
|
||||||
"""
|
"""
|
||||||
Get possible filename variations for a team abbreviation.
|
Get possible filename variations for a team abbreviation.
|
||||||
@@ -226,7 +282,10 @@ class LogoHelper:
|
|||||||
return {
|
return {
|
||||||
'cached_logos': len(self._logo_cache),
|
'cached_logos': len(self._logo_cache),
|
||||||
'cache_size_limit': self.cache_size,
|
'cache_size_limit': self.cache_size,
|
||||||
'cache_usage_percent': (len(self._logo_cache) / self.cache_size) * 100
|
'cache_usage_percent': (
|
||||||
|
(len(self._logo_cache) / self.cache_size) * 100
|
||||||
|
if self.cache_size else 0
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
def _resize_logo(self, logo: Image.Image, max_width: Optional[int] = None,
|
def _resize_logo(self, logo: Image.Image, max_width: Optional[int] = None,
|
||||||
@@ -258,17 +317,60 @@ class LogoHelper:
|
|||||||
self._cache_order.append(cache_key)
|
self._cache_order.append(cache_key)
|
||||||
|
|
||||||
def _download_logo(self, url: str, file_path: Path) -> None:
|
def _download_logo(self, url: str, file_path: Path) -> None:
|
||||||
"""Download logo from URL."""
|
"""Download logo from URL.
|
||||||
|
|
||||||
|
The response size is capped and the saved file is verified as a
|
||||||
|
decodable image before it is left on disk: a logo URL is remote
|
||||||
|
input, and without this an oversized or malformed response would
|
||||||
|
be cached for every later load_logo() call to trip over.
|
||||||
|
|
||||||
|
The body is streamed and counted as it arrives rather than read
|
||||||
|
through response.content, which buffers the whole thing first —
|
||||||
|
a server that omits Content-Length and never stops sending would
|
||||||
|
exhaust memory before any size check could run. Nothing lands at
|
||||||
|
file_path until the download completes and decodes, so a failed
|
||||||
|
download cannot leave a truncated logo behind either.
|
||||||
|
"""
|
||||||
# Ensure directory exists with proper permissions
|
# Ensure directory exists with proper permissions
|
||||||
ensure_directory_permissions(file_path.parent, get_assets_dir_mode())
|
ensure_directory_permissions(file_path.parent, get_assets_dir_mode())
|
||||||
|
|
||||||
# Download with timeout
|
# A unique temp name, not a fixed "<name>.part": two plugins can
|
||||||
response = self.session.get(url, timeout=30)
|
# ask for the same logo at once, and a shared name would let them
|
||||||
|
# interleave writes into one file, publish the mixture, or delete
|
||||||
|
# each other's partial. Same directory, so os.replace stays atomic.
|
||||||
|
fd, tmp_name = tempfile.mkstemp(
|
||||||
|
dir=str(file_path.parent), prefix=file_path.name + '.', suffix='.part')
|
||||||
|
tmp_path = Path(tmp_name)
|
||||||
|
try:
|
||||||
|
# fdopen outermost so the descriptor mkstemp handed back is
|
||||||
|
# always adopted and closed, including when the request itself
|
||||||
|
# raises — load_logo_with_download swallows that, so a leak
|
||||||
|
# here would accumulate quietly on a URL that keeps failing.
|
||||||
|
with os.fdopen(fd, 'wb') as f:
|
||||||
|
with self.session.get(url, timeout=30, stream=True) as response:
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
|
downloaded = 0
|
||||||
|
for chunk in response.iter_content(chunk_size=64 * 1024):
|
||||||
|
if not chunk:
|
||||||
|
continue
|
||||||
|
downloaded += len(chunk)
|
||||||
|
if downloaded > MAX_LOGO_BYTES:
|
||||||
|
raise ValueError(
|
||||||
|
f"Logo at {url} exceeds the "
|
||||||
|
f"{MAX_LOGO_BYTES}-byte limit; not saved")
|
||||||
|
f.write(chunk)
|
||||||
|
|
||||||
# Save to file
|
# Verify it decodes before it becomes the cached logo. PIL
|
||||||
with open(file_path, 'wb') as f:
|
# raises DecompressionBombError past its own pixel limit; a
|
||||||
f.write(response.content)
|
# partial or non-image response raises UnidentifiedImageError
|
||||||
|
# (an OSError subclass).
|
||||||
|
with Image.open(tmp_path) as probe:
|
||||||
|
probe.load()
|
||||||
|
|
||||||
|
os.replace(tmp_path, file_path)
|
||||||
|
except BaseException:
|
||||||
|
tmp_path.unlink(missing_ok=True)
|
||||||
|
raise
|
||||||
|
|
||||||
# Set proper file permissions after saving
|
# Set proper file permissions after saving
|
||||||
ensure_file_permissions(file_path, get_assets_file_mode())
|
ensure_file_permissions(file_path, get_assets_file_mode())
|
||||||
|
|||||||
@@ -328,7 +328,11 @@ class ScrollHelper:
|
|||||||
elapsed_time = current_time - (self.scroll_start_time or current_time)
|
elapsed_time = current_time - (self.scroll_start_time or current_time)
|
||||||
# The image already includes display_width padding, so we only need total_scroll_width
|
# The image already includes display_width padding, so we only need total_scroll_width
|
||||||
required_total_distance = self.total_scroll_width
|
required_total_distance = self.total_scroll_width
|
||||||
self.logger.info(
|
# Progress telemetry, emitted every few seconds for the whole of
|
||||||
|
# every scroll. It says how far along a marquee is, which is what
|
||||||
|
# you turn debug on to watch and not something an operator needs
|
||||||
|
# in the journal on a device that scrolls all day.
|
||||||
|
self.logger.debug(
|
||||||
"Scroll progress: elapsed=%.2fs, target=%.2fs, total_scrolled=%.0f/%d px (%.1f%%)",
|
"Scroll progress: elapsed=%.2fs, target=%.2fs, total_scrolled=%.0f/%d px (%.1f%%)",
|
||||||
elapsed_time,
|
elapsed_time,
|
||||||
self.calculated_duration,
|
self.calculated_duration,
|
||||||
|
|||||||
@@ -0,0 +1,470 @@
|
|||||||
|
"""Card-drawing helpers shared by every sports scoreboard plugin.
|
||||||
|
|
||||||
|
The eight scoreboards each carried byte-identical copies of the functions
|
||||||
|
below: the colour pickers, the settings lookup, the date and time formatting,
|
||||||
|
the favourite-team rules and the font-size grid snapping. One fix had to be
|
||||||
|
made eight times, and a new scoreboard started by copying them a ninth.
|
||||||
|
|
||||||
|
Everything here is a **free function taking explicit arguments**, not a base
|
||||||
|
class. Adoption is therefore per-function and reversible: a plugin keeps its
|
||||||
|
method and delegates the body, so the call sites and the override points are
|
||||||
|
untouched. That is also why `config`, `logger` and `fonts` are parameters
|
||||||
|
rather than attributes -- the helper never reaches back into the caller.
|
||||||
|
|
||||||
|
The bodies are the plugins' own code, moved rather than rewritten. The one
|
||||||
|
deliberate difference is `crisp_size`, which takes the seven-plugin guard
|
||||||
|
(`not desired`) instead of football's: they agree on every real input, and
|
||||||
|
the extra guard only stops a None size raising TypeError.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Dict, Optional, Tuple
|
||||||
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ELEMENT_FOR_FONT", "FAVORITE_RESULT_COLOR_DEFAULTS", "FONT_NAME_ALIASES",
|
||||||
|
"FONT_PIXEL_GRID", "MONTH_ABBR", "WEEKDAY_ABBR",
|
||||||
|
"scroll_card_option", "element_color", "font_color", "coerce_rgb",
|
||||||
|
"score_color_for", "recent_score_color", "favorite_teams_for",
|
||||||
|
"side_is_favorite", "side_score", "favorite_result",
|
||||||
|
"card_tzinfo", "weekday_for", "format_game_date", "format_game_time",
|
||||||
|
"vs_text", "upcoming_center_mode", "crisp_size", "schema_font_size",
|
||||||
|
"resolve_font_size", "unshare_element_fonts",
|
||||||
|
]
|
||||||
|
|
||||||
|
#: Which customization element owns each font key, for colour resolution.
|
||||||
|
ELEMENT_FOR_FONT: Dict[str, str] = {
|
||||||
|
"score": "score_text",
|
||||||
|
"time": "period_text",
|
||||||
|
"team": "team_name",
|
||||||
|
"status": "status_text",
|
||||||
|
"detail": "detail_text",
|
||||||
|
"rank": "rank_text",
|
||||||
|
}
|
||||||
|
|
||||||
|
#: Fallback colours when favourite_result_colors is on but a slot is unset.
|
||||||
|
FAVORITE_RESULT_COLOR_DEFAULTS: Dict[str, Tuple[int, int, int]] = {
|
||||||
|
"win": (0, 255, 0),
|
||||||
|
"loss": (255, 0, 0),
|
||||||
|
"tie": (255, 200, 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
#: Family aliases the web UI may write, mapped to the shipped filename.
|
||||||
|
FONT_NAME_ALIASES: Dict[str, str] = {
|
||||||
|
"press_start": "PressStart2P-Regular.ttf",
|
||||||
|
"four_by_six": "4x6-font.ttf",
|
||||||
|
}
|
||||||
|
|
||||||
|
#: Pixel grid each face renders crisply on. Off-grid sizes anti-alias, which
|
||||||
|
#: on an LED matrix is a dim lamp rather than a soft edge.
|
||||||
|
FONT_PIXEL_GRID: Dict[str, int] = {
|
||||||
|
"PressStart2P-Regular.ttf": 8,
|
||||||
|
"4x6-font.ttf": 7,
|
||||||
|
}
|
||||||
|
|
||||||
|
MONTH_ABBR = ("Jan", "Feb", "Mar", "Apr", "May", "Jun",
|
||||||
|
"Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
|
||||||
|
WEEKDAY_ABBR = ("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Settings lookup
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def scroll_card_option(config: Optional[Dict[str, Any]], key: str,
|
||||||
|
default: Any = None) -> Any:
|
||||||
|
"""Read one key from the scroll_card config block."""
|
||||||
|
block = (config or {}).get("scroll_card")
|
||||||
|
if isinstance(block, dict) and block.get(key) is not None:
|
||||||
|
return block.get(key)
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def vs_text(config: Optional[Dict[str, Any]]) -> str:
|
||||||
|
"""Separator drawn between the teams -- "VS", "@", "at", anything."""
|
||||||
|
return str(scroll_card_option(config, "vs_text", "VS"))
|
||||||
|
|
||||||
|
|
||||||
|
def upcoming_center_mode(config: Optional[Dict[str, Any]]) -> str:
|
||||||
|
"""Middle of an upcoming card: 'vs', 'date_time' or 'none'."""
|
||||||
|
mode = str(scroll_card_option(config, "upcoming_center", "vs") or "vs").lower()
|
||||||
|
return mode if mode in ("vs", "date_time", "none") else "vs"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Colour
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def element_color(config: Optional[Dict[str, Any]], element: str,
|
||||||
|
default: Tuple[int, int, int] = (255, 255, 255)):
|
||||||
|
"""Per-element text colour from customization.<element>.text_color."""
|
||||||
|
try:
|
||||||
|
cfg = (config or {}).get("customization", {}).get(element, {})
|
||||||
|
value = cfg.get("text_color")
|
||||||
|
if isinstance(value, (list, tuple)) and len(value) == 3:
|
||||||
|
return tuple(max(0, min(255, int(c))) for c in value)
|
||||||
|
if isinstance(value, str) and value.startswith("#") and len(value) == 7:
|
||||||
|
return tuple(int(value[i:i + 2], 16) for i in (1, 3, 5))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def font_color(config: Optional[Dict[str, Any]], fonts: Optional[Dict[str, Any]],
|
||||||
|
font, default: Tuple[int, int, int] = (255, 255, 255)):
|
||||||
|
"""Colour for whichever element owns this face.
|
||||||
|
|
||||||
|
Matched on identity, and deliberately gives up when one object is
|
||||||
|
shared: the last-resort font path can hand the same face to several
|
||||||
|
keys, and there is no right answer for which element's colour that is.
|
||||||
|
White is what those draws used before, so ambiguity costs nothing.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
fonts = fonts or {}
|
||||||
|
matches = [element for key, element in ELEMENT_FOR_FONT.items()
|
||||||
|
if fonts.get(key) is font]
|
||||||
|
if len(matches) == 1:
|
||||||
|
return element_color(config, matches[0], default)
|
||||||
|
except (AttributeError, TypeError):
|
||||||
|
pass
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def coerce_rgb(value, fallback):
|
||||||
|
"""Turn a configured [R, G, B] list into a clamped (r, g, b) tuple."""
|
||||||
|
# Checked before unpacking: a 3-character string ("123") would otherwise
|
||||||
|
# iterate into three digits and yield a colour rather than the fallback.
|
||||||
|
if not isinstance(value, (list, tuple)) or len(value) != 3:
|
||||||
|
return fallback
|
||||||
|
try:
|
||||||
|
r, g, b = (max(0, min(255, int(channel))) for channel in value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return fallback
|
||||||
|
return (r, g, b)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Favourite teams
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def favorite_teams_for(config: Dict[str, Any], game: Dict[str, Any]) -> list:
|
||||||
|
"""Favorite teams that apply to this game.
|
||||||
|
|
||||||
|
Both sources are used. Games carry the league manager's *resolved*
|
||||||
|
favorites, which is the only place dynamic groups such as AP_TOP_25
|
||||||
|
appear expanded; the config is read as well so an edit takes effect on
|
||||||
|
already-fetched games, and so hand-built game dicts (tests, other
|
||||||
|
callers) still work.
|
||||||
|
"""
|
||||||
|
favorites = list(game.get("favorite_teams") or [])
|
||||||
|
league_config = config.get(str(game.get("league", "") or ""))
|
||||||
|
if isinstance(league_config, dict):
|
||||||
|
favorites += list(league_config.get("favorite_teams") or [])
|
||||||
|
else:
|
||||||
|
favorites += list(config.get("favorite_teams") or [])
|
||||||
|
return favorites
|
||||||
|
|
||||||
|
|
||||||
|
def side_is_favorite(game: Dict[str, Any], side: str, favorites: set) -> bool:
|
||||||
|
"""Is the home/away side of this game a favorite team?
|
||||||
|
|
||||||
|
Reads both the flat (``home_abbr``) and nested (``home_team.abbrev``)
|
||||||
|
payload shapes, and matches on the ESPN id too, because a couple of
|
||||||
|
leagues (NRL) key favorites by id where abbreviations collide.
|
||||||
|
"""
|
||||||
|
candidates = [game.get(f"{side}_abbr"), game.get(f"{side}_id")]
|
||||||
|
team = game.get(f"{side}_team")
|
||||||
|
if isinstance(team, dict):
|
||||||
|
candidates += [team.get("abbrev"), team.get("abbreviation"), team.get("id")]
|
||||||
|
for value in candidates:
|
||||||
|
if value is not None and str(value).strip().upper() in favorites:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def side_score(game: Dict[str, Any], side: str) -> Optional[int]:
|
||||||
|
"""Numeric score for one side, from either payload shape."""
|
||||||
|
raw = None
|
||||||
|
team = game.get(f"{side}_team")
|
||||||
|
if isinstance(team, dict) and team.get("score") is not None:
|
||||||
|
raw = team.get("score")
|
||||||
|
if raw is None:
|
||||||
|
raw = game.get(f"{side}_score")
|
||||||
|
try:
|
||||||
|
return int(float(str(raw).strip()))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def favorite_result(config: Dict[str, Any], game: Dict[str, Any]) -> Optional[str]:
|
||||||
|
"""Say how the favorite team did in a finished game.
|
||||||
|
|
||||||
|
Returns 'win', 'loss' or 'tie', or None when there is no single team
|
||||||
|
to root for: no favorites configured, neither side is a favorite, or
|
||||||
|
*both* are -- a favorite-vs-favorite game has no losing side worth
|
||||||
|
flagging in red. Also None when the scores are not usable numbers.
|
||||||
|
"""
|
||||||
|
favorites = {
|
||||||
|
str(team).strip().upper()
|
||||||
|
for team in favorite_teams_for(config, game)
|
||||||
|
if str(team).strip()
|
||||||
|
}
|
||||||
|
if not favorites:
|
||||||
|
return None
|
||||||
|
|
||||||
|
home_fav = side_is_favorite(game, "home", favorites)
|
||||||
|
away_fav = side_is_favorite(game, "away", favorites)
|
||||||
|
if home_fav == away_fav:
|
||||||
|
return None
|
||||||
|
|
||||||
|
home_score = side_score(game, "home")
|
||||||
|
away_score = side_score(game, "away")
|
||||||
|
if home_score is None or away_score is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if home_score == away_score:
|
||||||
|
return "tie"
|
||||||
|
favorite_score, other_score = (
|
||||||
|
(home_score, away_score) if home_fav else (away_score, home_score)
|
||||||
|
)
|
||||||
|
return "win" if favorite_score > other_score else "loss"
|
||||||
|
|
||||||
|
|
||||||
|
def recent_score_color(config: Dict[str, Any], logger, game: Dict[str, Any], default):
|
||||||
|
"""Fill color for a finished game's score, per favorite_result_colors."""
|
||||||
|
try:
|
||||||
|
settings = (config.get("customization") or {}).get(
|
||||||
|
"favorite_result_colors"
|
||||||
|
) or {}
|
||||||
|
if not settings.get("enabled", False):
|
||||||
|
return default
|
||||||
|
result = favorite_result(config, game)
|
||||||
|
if result is None:
|
||||||
|
return default
|
||||||
|
return coerce_rgb(
|
||||||
|
settings.get(f"{result}_color"),
|
||||||
|
FAVORITE_RESULT_COLOR_DEFAULTS[result],
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
logger.debug("Could not resolve favorite result color", exc_info=True)
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def score_color_for(config: Dict[str, Any], logger, game: Dict[str, Any],
|
||||||
|
game_type: str, default=None):
|
||||||
|
"""Fill color for a game card's score. Only finished games are tinted.
|
||||||
|
|
||||||
|
The default is the configured score colour rather than a flat white,
|
||||||
|
so customization.score_text.text_color shows on games the favourite
|
||||||
|
tint does not apply to. The tint still wins where it applies.
|
||||||
|
"""
|
||||||
|
if default is None:
|
||||||
|
default = element_color(config, 'score_text')
|
||||||
|
if game_type != "recent":
|
||||||
|
return default
|
||||||
|
return recent_score_color(config, logger, game, default)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Date and time
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def card_tzinfo(config: Optional[Dict[str, Any]], logger):
|
||||||
|
"""Timezone for weekday/24h conversions; falls back to UTC."""
|
||||||
|
configured = (config or {}).get("timezone")
|
||||||
|
if configured:
|
||||||
|
try:
|
||||||
|
return ZoneInfo(configured)
|
||||||
|
except (KeyError, ValueError, TypeError, OSError) as exc:
|
||||||
|
# KeyError covers ZoneInfoNotFoundError. A bad zone name in
|
||||||
|
# config should fall back to UTC, not blank the card.
|
||||||
|
logger.debug("Unusable timezone %r: %s", configured, exc)
|
||||||
|
return timezone.utc
|
||||||
|
|
||||||
|
|
||||||
|
def weekday_for(config: Optional[Dict[str, Any]], logger,
|
||||||
|
game: Optional[Dict]) -> str:
|
||||||
|
"""Weekday abbreviation from the game's start time, or ''."""
|
||||||
|
if not game:
|
||||||
|
return ""
|
||||||
|
raw = game.get("start_time_utc") or game.get("start_time")
|
||||||
|
if not raw:
|
||||||
|
return ""
|
||||||
|
try:
|
||||||
|
start = raw if isinstance(raw, datetime) else datetime.fromisoformat(
|
||||||
|
str(raw).replace("Z", "+00:00"))
|
||||||
|
return WEEKDAY_ABBR[start.astimezone(card_tzinfo(config, logger)).weekday()]
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def format_game_date(config: Optional[Dict[str, Any]], logger, date_text: str,
|
||||||
|
game: Optional[Dict] = None) -> str:
|
||||||
|
"""Format an upcoming card's date per scroll_card.date_format."""
|
||||||
|
raw = str(date_text or "").strip()
|
||||||
|
if not raw:
|
||||||
|
return ""
|
||||||
|
fmt = str(scroll_card_option(config, "date_format", "abbrev") or "abbrev")
|
||||||
|
if fmt == "numeric":
|
||||||
|
return raw
|
||||||
|
parts = raw.replace("-", "/").split("/")
|
||||||
|
if not (len(parts) >= 2 and parts[0].strip().isdigit() and parts[1].strip().isdigit()):
|
||||||
|
return raw
|
||||||
|
month, day = int(parts[0]), int(parts[1])
|
||||||
|
if not 1 <= month <= 12:
|
||||||
|
return raw
|
||||||
|
name = MONTH_ABBR[month - 1]
|
||||||
|
if fmt == "numeric_day_first":
|
||||||
|
return f"{day}/{month}"
|
||||||
|
if fmt == "day_first":
|
||||||
|
return f"{day} {name}"
|
||||||
|
if fmt == "weekday":
|
||||||
|
weekday = weekday_for(config, logger, game)
|
||||||
|
return f"{weekday} {name} {day}" if weekday else f"{name} {day}"
|
||||||
|
return f"{name} {day}"
|
||||||
|
|
||||||
|
|
||||||
|
def format_game_time(config: Optional[Dict[str, Any]], time_text: str) -> str:
|
||||||
|
"""Return the time as-is (12h) or converted to 24h."""
|
||||||
|
raw = str(time_text or "").strip()
|
||||||
|
if not raw or str(scroll_card_option(config, "time_format", "12h")) != "24h":
|
||||||
|
return raw
|
||||||
|
cleaned = raw.upper().replace(" ", "")
|
||||||
|
meridiem = "AM" if cleaned.endswith("AM") else "PM" if cleaned.endswith("PM") else ""
|
||||||
|
if not meridiem:
|
||||||
|
return raw
|
||||||
|
try:
|
||||||
|
hh, _, mm = cleaned[:-2].partition(":")
|
||||||
|
hour, minute = int(hh), int(mm or 0)
|
||||||
|
except ValueError:
|
||||||
|
return raw
|
||||||
|
if not (0 <= hour <= 12 and 0 <= minute <= 59):
|
||||||
|
return raw
|
||||||
|
hour = hour % 12 + (12 if meridiem == "PM" else 0)
|
||||||
|
return f"{hour:02d}:{minute:02d}"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Font sizing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
#: Per-schema caches, keyed by the schema's absolute path. Keyed rather than
|
||||||
|
#: global because each plugin declares its own defaults; keyed rather than
|
||||||
|
#: per-class because the helper has no class to hang it on.
|
||||||
|
_SCHEMA_FONT_SIZE_CACHE: Dict[str, Dict[str, int]] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def crisp_size(font_file, desired, aliases=None, grid_table=None):
|
||||||
|
"""Snap *desired* to the nearest size *font_file* renders crisply at.
|
||||||
|
|
||||||
|
A face with no known grid is returned unchanged, so a user-supplied
|
||||||
|
font is never second-guessed.
|
||||||
|
|
||||||
|
``aliases`` and ``grid_table`` default to the shared tables; a plugin
|
||||||
|
that ships an extra face can pass its own without forking this.
|
||||||
|
"""
|
||||||
|
aliases = FONT_NAME_ALIASES if aliases is None else aliases
|
||||||
|
grid_table = FONT_PIXEL_GRID if grid_table is None else grid_table
|
||||||
|
font_file = aliases.get(font_file, font_file)
|
||||||
|
grid = grid_table.get(font_file)
|
||||||
|
if not grid or not desired or desired <= 0:
|
||||||
|
return desired
|
||||||
|
return max(grid, int(round(float(desired) / grid)) * grid)
|
||||||
|
|
||||||
|
|
||||||
|
def schema_font_size(schema_path: str, element_key) -> Optional[int]:
|
||||||
|
"""The font_size this plugin's config_schema.json declares, or None.
|
||||||
|
|
||||||
|
Cached per schema path. The plugins cached this on their own class; the
|
||||||
|
path is the same distinction expressed without one, so two plugins never
|
||||||
|
share an entry.
|
||||||
|
"""
|
||||||
|
if not element_key:
|
||||||
|
return None
|
||||||
|
cache = _SCHEMA_FONT_SIZE_CACHE.get(schema_path)
|
||||||
|
if cache is None:
|
||||||
|
cache = {}
|
||||||
|
try:
|
||||||
|
import json
|
||||||
|
with open(schema_path) as fh:
|
||||||
|
schema = json.load(fh)
|
||||||
|
props = (schema.get('properties', {})
|
||||||
|
.get('customization', {})
|
||||||
|
.get('properties', {}))
|
||||||
|
for key, spec in props.items():
|
||||||
|
size = spec.get('properties', {}).get('font_size', {}).get('default')
|
||||||
|
if size is not None:
|
||||||
|
cache[key] = int(size)
|
||||||
|
except Exception as exc:
|
||||||
|
# See sports_shared._schema_font_size: an unreadable schema
|
||||||
|
# silently disables the pixel-grid snap for every element.
|
||||||
|
# Built once per schema path, so this cannot repeat per frame.
|
||||||
|
logger.warning(
|
||||||
|
"could not read %s (%s: %s); font sizes will skip their "
|
||||||
|
"pixel grid snap and may render a pixel narrow",
|
||||||
|
schema_path, type(exc).__name__, exc)
|
||||||
|
cache = {}
|
||||||
|
_SCHEMA_FONT_SIZE_CACHE[schema_path] = cache
|
||||||
|
return cache.get(element_key)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_font_size(schema_path: str, element_config, element_key,
|
||||||
|
default_size, font_name, aliases=None, grid_table=None):
|
||||||
|
"""Size to render at: the user's choice, or a grid-snapped default.
|
||||||
|
|
||||||
|
A configured size counts as a real choice only when it differs from
|
||||||
|
the schema default. The web UI writes the whole schema default block
|
||||||
|
on every save, so "font_size == schema default" carries no intent and
|
||||||
|
would otherwise pin every install to an anti-aliased size forever.
|
||||||
|
"""
|
||||||
|
configured = (element_config or {}).get('font_size')
|
||||||
|
if configured is not None:
|
||||||
|
try:
|
||||||
|
configured = int(configured)
|
||||||
|
if configured != schema_font_size(schema_path, element_key):
|
||||||
|
return configured
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return crisp_size(font_name, default_size, aliases, grid_table)
|
||||||
|
|
||||||
|
|
||||||
|
def unshare_element_fonts(logger, fonts):
|
||||||
|
"""Give each colourable element its own face object.
|
||||||
|
|
||||||
|
The colour a draw gets is resolved from the face it was handed, and
|
||||||
|
several of these loaders legitimately hand one object to more than one
|
||||||
|
element -- a size resolver that lands two elements on the same face, a
|
||||||
|
fallback that fills every key from one default, football's narrowing
|
||||||
|
step that deliberately shrinks the clock along with the score. Sharing
|
||||||
|
the object makes the element ambiguous and the colour unresolvable.
|
||||||
|
|
||||||
|
Re-instantiating from the same path and size gives a distinct object
|
||||||
|
with identical metrics, so nothing about the rendering changes; only
|
||||||
|
the ability to tell two elements apart does. Faces that cannot be
|
||||||
|
rebuilt (a BDF loaded through freetype.Face, anything without a usable
|
||||||
|
path) are left shared, and their draws stay white as before.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from PIL import ImageFont as _IF
|
||||||
|
except ImportError: # pragma: no cover
|
||||||
|
return fonts
|
||||||
|
seen = {}
|
||||||
|
for key in ELEMENT_FOR_FONT:
|
||||||
|
font = fonts.get(key)
|
||||||
|
if font is None:
|
||||||
|
continue
|
||||||
|
if id(font) not in seen:
|
||||||
|
seen[id(font)] = key
|
||||||
|
continue
|
||||||
|
path, size = getattr(font, "path", None), getattr(font, "size", None)
|
||||||
|
if not path or not size:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
fonts[key] = _IF.truetype(path, size)
|
||||||
|
except (OSError, ValueError, TypeError):
|
||||||
|
logger.debug(
|
||||||
|
"Could not un-share the %s face; it keeps the default colour", key)
|
||||||
|
return fonts
|
||||||
@@ -0,0 +1,261 @@
|
|||||||
|
"""The scroll/Vegas card geometry the sports scoreboards all share.
|
||||||
|
|
||||||
|
Eight scoreboards -- afl, baseball, basketball, football, hockey, lacrosse,
|
||||||
|
nrl and soccer -- each carried their own ``game_renderer.py``. After the card
|
||||||
|
helpers moved to ``src/common/sports_card.py`` the settings lookups were
|
||||||
|
shared, but the *geometry* was not: nine methods that decide how wide the
|
||||||
|
centre gap is, how much room a logo gets, and where an upcoming card's date
|
||||||
|
and time land were still eight separate copies. Five were byte-identical
|
||||||
|
across all eight plugins; the other four were identical in seven, with a
|
||||||
|
different single outlier each time.
|
||||||
|
|
||||||
|
That last detail is why this is a mixin rather than free functions. There is
|
||||||
|
no per-sport branching to write -- baseball needs its own
|
||||||
|
``_draw_upcoming_game_status`` and ``_logo_slot_width``, hockey its own
|
||||||
|
``_upcoming_date_and_time``, football its own ``_score_reserve_width``, and
|
||||||
|
every one of those is an ordinary override. The seven that agree inherit and
|
||||||
|
say nothing.
|
||||||
|
|
||||||
|
It is deliberately *only* a mixin: no ``__init__``, no state of its own. The
|
||||||
|
plugins' constructors differ in six ways and none of that difference is worth
|
||||||
|
unifying, so adoption is one line on the class statement plus deleting the
|
||||||
|
methods that now come from here.
|
||||||
|
|
||||||
|
What a host class must provide
|
||||||
|
------------------------------
|
||||||
|
Attributes: ``display_width``, ``display_height``, ``config``, ``fonts``,
|
||||||
|
``logger``, ``_team_rankings_cache``.
|
||||||
|
|
||||||
|
Methods: ``_draw_text_with_outline(draw, text, position, font, fill=None,
|
||||||
|
outline_color=(0, 0, 0))`` -- the one hook whose body genuinely varies -- plus
|
||||||
|
the ``sports_card`` delegations ``_scroll_card_option``,
|
||||||
|
``_upcoming_center_mode``, ``_vs_text``, ``_element_color``,
|
||||||
|
``_format_game_date`` and ``_format_game_time``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import math
|
||||||
|
from typing import ClassVar, Dict, Tuple
|
||||||
|
|
||||||
|
from PIL import Image, ImageDraw
|
||||||
|
|
||||||
|
|
||||||
|
class SportsGameRendererMixin:
|
||||||
|
"""Shared card geometry for the sports scoreboards. See module docstring."""
|
||||||
|
|
||||||
|
#: Centre strip as a fraction of card width, before clamping.
|
||||||
|
CENTER_GAP_RATIO: ClassVar[float] = 0.28
|
||||||
|
#: Clamp floor for the derived centre gap.
|
||||||
|
CENTER_GAP_MIN_PX: ClassVar[int] = 22
|
||||||
|
#: Clamp ceiling for the derived centre gap.
|
||||||
|
CENTER_GAP_MAX_PX: ClassVar[int] = 40
|
||||||
|
#: Breathing room kept between the score and each logo.
|
||||||
|
_SCORE_LOGO_GUTTER_PX: ClassVar[int] = 4
|
||||||
|
#: Widest score the centre strip must fit. Leagues that can reach three
|
||||||
|
#: digits a side override this with "000-000".
|
||||||
|
_SCORE_PROBE: ClassVar[str] = "00-00"
|
||||||
|
|
||||||
|
# ---- geometry ------------------------------------------------------
|
||||||
|
|
||||||
|
# Non-finite settings are rejected before any int()/round(): "inf" reaches
|
||||||
|
# these from config as a float or a string, passes an `isinstance` plus
|
||||||
|
# `>= 0` check unharmed, and then raises OverflowError out of int() --
|
||||||
|
# which the old `except (TypeError, ValueError)` did not catch, so it
|
||||||
|
# aborted the whole card render. Present in all eight plugins before this
|
||||||
|
# moved to the core; fixing it here fixes it in all eight.
|
||||||
|
|
||||||
|
def _score_reserve_width(self) -> int:
|
||||||
|
"""Centre strip the score actually needs, measured rather than assumed.
|
||||||
|
|
||||||
|
The gap was derived from the card width alone (width x
|
||||||
|
CENTER_GAP_RATIO, clamped to CENTER_GAP_MAX_PX) while the score's size
|
||||||
|
comes from config and the element-style resolver. Nothing compared the
|
||||||
|
two, so any score wider than the clamp was drawn over the logos.
|
||||||
|
Measuring it keeps the strip wide enough for whatever font is in play.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
probe = ImageDraw.Draw(Image.new("RGB", (4, 4)))
|
||||||
|
width = probe.textlength(self._SCORE_PROBE, font=self.fonts['score'])
|
||||||
|
return int(width) + 2 * self._SCORE_LOGO_GUTTER_PX
|
||||||
|
except Exception:
|
||||||
|
self.logger.debug("Score reserve measurement failed", exc_info=True)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def _center_gap_width(self) -> int:
|
||||||
|
"""Width of the middle strip kept clear of logos.
|
||||||
|
|
||||||
|
``scroll_card.center_gap`` pins it outright; otherwise it scales with
|
||||||
|
the card width between the configurable min and max. 0 restores
|
||||||
|
edge-to-edge logos.
|
||||||
|
"""
|
||||||
|
configured = self._scroll_card_option("center_gap")
|
||||||
|
if (isinstance(configured, (int, float))
|
||||||
|
and math.isfinite(configured) and configured >= 0):
|
||||||
|
return int(configured)
|
||||||
|
ratio = self._scroll_card_option("center_gap_ratio", self.CENTER_GAP_RATIO)
|
||||||
|
low = self._scroll_card_option("center_gap_min", self.CENTER_GAP_MIN_PX)
|
||||||
|
high = self._scroll_card_option("center_gap_max", self.CENTER_GAP_MAX_PX)
|
||||||
|
try:
|
||||||
|
ratio, low, high = float(ratio), float(low), float(high)
|
||||||
|
if not (math.isfinite(ratio) and math.isfinite(low)
|
||||||
|
and math.isfinite(high)):
|
||||||
|
return self.CENTER_GAP_MIN_PX
|
||||||
|
scaled = round(self.display_width * ratio)
|
||||||
|
derived = int(max(int(low), min(int(high), scaled)))
|
||||||
|
# A strip narrower than the score is the bug, not a style choice.
|
||||||
|
# An explicit ``center_gap`` is still honoured above, including 0.
|
||||||
|
return max(derived, self._score_reserve_width())
|
||||||
|
except (TypeError, ValueError, OverflowError):
|
||||||
|
return self.CENTER_GAP_MIN_PX
|
||||||
|
|
||||||
|
def _logo_slot_width(self) -> int:
|
||||||
|
"""Per-side logo slot, leaving the center gap clear.
|
||||||
|
|
||||||
|
No longer capped at display_height: the card is sized as two
|
||||||
|
full-height logos plus the measured gap, so what is left after the gap
|
||||||
|
is exactly the logo's share. The cap was what froze the logos at 46px
|
||||||
|
on the old flat 128px card.
|
||||||
|
"""
|
||||||
|
available = (self.display_width - self._center_gap_width()) // 2
|
||||||
|
return max(8, available)
|
||||||
|
|
||||||
|
def _logo_cache_key(self, name: str) -> str:
|
||||||
|
"""Cache key scoped to the logo slot.
|
||||||
|
|
||||||
|
One cache dict is shared by renderers built for different card widths,
|
||||||
|
so a logo sized for a wide slot must not be handed to a narrow one.
|
||||||
|
"""
|
||||||
|
return f"{name}@{self._logo_slot_width()}x{self.display_height}"
|
||||||
|
|
||||||
|
def _layout_offset(self, element: str, axis: str, default: int = 0) -> int:
|
||||||
|
"""X/Y nudge for one element, from customization.layout.
|
||||||
|
|
||||||
|
Same block the full-screen scorebug reads (sports.py
|
||||||
|
_get_layout_offset), so a nudge configured in the web UI now moves
|
||||||
|
the element on the scroll/Vegas card too -- previously the schema
|
||||||
|
advertised these offsets but this renderer ignored them.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
layout = (self.config or {}).get("customization", {}).get("layout", {})
|
||||||
|
value = (layout.get(element) or {}).get(axis, default)
|
||||||
|
if isinstance(value, bool):
|
||||||
|
return default
|
||||||
|
if isinstance(value, (int, float)):
|
||||||
|
return int(value) if math.isfinite(value) else default
|
||||||
|
if isinstance(value, str):
|
||||||
|
parsed = float(value)
|
||||||
|
return int(parsed) if math.isfinite(parsed) else default
|
||||||
|
except (TypeError, ValueError, OverflowError):
|
||||||
|
pass
|
||||||
|
return default
|
||||||
|
|
||||||
|
# ---- upcoming cards ------------------------------------------------
|
||||||
|
|
||||||
|
def _upcoming_date_and_time(self, game: Dict) -> Tuple[str, str]:
|
||||||
|
"""(date, time) for an upcoming card, from the extractor's flat keys."""
|
||||||
|
return (
|
||||||
|
str(game.get("game_date", "") or ""),
|
||||||
|
str(game.get("game_time", "") or ""),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _draw_upcoming_center(self, draw: "ImageDraw.ImageDraw", game: Dict) -> None:
|
||||||
|
"""Draw the middle of an upcoming card.
|
||||||
|
|
||||||
|
Never a score: an upcoming game has not started, so the extractor's
|
||||||
|
0-0 is noise. Either the VS text (default), the date and time stacked,
|
||||||
|
or nothing at all.
|
||||||
|
"""
|
||||||
|
mode = self._upcoming_center_mode()
|
||||||
|
if mode == "none":
|
||||||
|
return
|
||||||
|
|
||||||
|
if mode == "vs":
|
||||||
|
vs_text = self._vs_text()
|
||||||
|
if not vs_text:
|
||||||
|
return
|
||||||
|
vs_width = draw.textlength(vs_text, font=self.fonts['score'])
|
||||||
|
vs_x = (self.display_width - vs_width) // 2 + self._layout_offset('score', 'x_offset')
|
||||||
|
vs_y = (self.display_height // 2) - 3 + self._layout_offset('score', 'y_offset')
|
||||||
|
self._draw_text_with_outline(
|
||||||
|
draw, vs_text, (vs_x, vs_y), self.fonts['score'],
|
||||||
|
fill=self._element_color('score_text')
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
date_text, time_text = self._upcoming_date_and_time(game)
|
||||||
|
lines = []
|
||||||
|
if self._scroll_card_option("show_date", True):
|
||||||
|
lines.append(self._format_game_date(date_text, game))
|
||||||
|
if self._scroll_card_option("show_time", True):
|
||||||
|
lines.append(self._format_game_time(time_text))
|
||||||
|
lines = [t for t in lines if t]
|
||||||
|
if not lines:
|
||||||
|
return
|
||||||
|
font = self.fonts.get('detail') or self.fonts['time']
|
||||||
|
line_h = 7
|
||||||
|
top = (self.display_height // 2) - (len(lines) * line_h) // 2
|
||||||
|
top += self._layout_offset('score', 'y_offset')
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
width = draw.textlength(line, font=font)
|
||||||
|
x = (self.display_width - width) // 2 + self._layout_offset('score', 'x_offset')
|
||||||
|
self._draw_text_with_outline(
|
||||||
|
draw, line, (x, top + i * line_h), font,
|
||||||
|
fill=self._element_color('detail_text')
|
||||||
|
)
|
||||||
|
|
||||||
|
def _draw_upcoming_game_status(self, draw: "ImageDraw.ImageDraw", game: Dict) -> None:
|
||||||
|
"""Draw the date and time around an upcoming card.
|
||||||
|
|
||||||
|
Time top and date bottom by default; scroll_card.swap_date_time puts
|
||||||
|
the date on top instead. Skipped when the pair is stacked in the
|
||||||
|
middle, which would otherwise print them twice.
|
||||||
|
"""
|
||||||
|
if self._upcoming_center_mode() == "date_time":
|
||||||
|
return
|
||||||
|
|
||||||
|
date_raw, time_raw = self._upcoming_date_and_time(game)
|
||||||
|
date_text = (self._format_game_date(date_raw, game)
|
||||||
|
if self._scroll_card_option("show_date", True) else "")
|
||||||
|
time_text = (self._format_game_time(time_raw)
|
||||||
|
if self._scroll_card_option("show_time", True) else "")
|
||||||
|
|
||||||
|
if self._scroll_card_option("swap_date_time", False):
|
||||||
|
top_text, top_el, bottom_text, bottom_el = (
|
||||||
|
date_text, 'date', time_text, 'time')
|
||||||
|
top_font = self.fonts.get('detail') or self.fonts['time']
|
||||||
|
bottom_font = self.fonts['time']
|
||||||
|
top_color, bottom_color = 'detail_text', 'period_text'
|
||||||
|
else:
|
||||||
|
top_text, top_el, bottom_text, bottom_el = (
|
||||||
|
time_text, 'time', date_text, 'date')
|
||||||
|
top_font = self.fonts['time']
|
||||||
|
bottom_font = self.fonts.get('detail') or self.fonts['time']
|
||||||
|
top_color, bottom_color = 'period_text', 'detail_text'
|
||||||
|
|
||||||
|
if top_text:
|
||||||
|
top_width = draw.textlength(top_text, font=top_font)
|
||||||
|
top_x = (self.display_width - top_width) // 2 + self._layout_offset(top_el, 'x_offset')
|
||||||
|
top_y = 1 + self._layout_offset(top_el, 'y_offset')
|
||||||
|
self._draw_text_with_outline(
|
||||||
|
draw, top_text, (top_x, top_y), top_font,
|
||||||
|
fill=self._element_color(top_color)
|
||||||
|
)
|
||||||
|
|
||||||
|
if bottom_text:
|
||||||
|
bottom_width = draw.textlength(bottom_text, font=bottom_font)
|
||||||
|
bottom_x = ((self.display_width - bottom_width) // 2
|
||||||
|
+ self._layout_offset(bottom_el, 'x_offset'))
|
||||||
|
# Measured, not a fixed -7: the detail font is 6px in most plugins
|
||||||
|
# but 10px in soccer and nrl, where "Sep 19" ran past the card.
|
||||||
|
ink_bottom = draw.textbbox((0, 0), bottom_text, font=bottom_font)[3]
|
||||||
|
bottom_y = (max(0, self.display_height - ink_bottom - 1)
|
||||||
|
+ self._layout_offset(bottom_el, 'y_offset'))
|
||||||
|
self._draw_text_with_outline(
|
||||||
|
draw, bottom_text, (bottom_x, bottom_y), bottom_font,
|
||||||
|
fill=self._element_color(bottom_color)
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- rankings ------------------------------------------------------
|
||||||
|
|
||||||
|
def set_rankings_cache(self, rankings: Dict[str, int]) -> None:
|
||||||
|
"""Set the team rankings cache for display."""
|
||||||
|
self._team_rankings_cache = rankings
|
||||||
File diff suppressed because it is too large
Load Diff
+88
-30
@@ -19,6 +19,7 @@ Port default: 5765 (UDP). Open this port on both Pis if ufw is active:
|
|||||||
|
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
|
import math
|
||||||
import os
|
import os
|
||||||
import socket
|
import socket
|
||||||
import struct
|
import struct
|
||||||
@@ -37,6 +38,13 @@ _RAW_MAGIC = b'SYNC_RAW'
|
|||||||
_RAW_HEADER = struct.Struct('<HH') # width, height (uint16 LE)
|
_RAW_HEADER = struct.Struct('<HH') # width, height (uint16 LE)
|
||||||
|
|
||||||
|
|
||||||
|
# Upper bound on a decoded frame/scroll image. Generous for any real scroll
|
||||||
|
# image (a leader's full cycle is long but only panel-height tall), and low
|
||||||
|
# enough that a crafted image from any host on the LAN cannot force a large
|
||||||
|
# allocation on the render thread. Applied on both receive paths — the TCP
|
||||||
|
# image server and the follower's legacy-PNG UDP fallback.
|
||||||
|
_MAX_FRAME_W, _MAX_FRAME_H = 100_000, 256
|
||||||
|
|
||||||
SYNC_PORT = 5765
|
SYNC_PORT = 5765
|
||||||
HELLO_INTERVAL = 5.0 # follower broadcasts hello every 5 s
|
HELLO_INTERVAL = 5.0 # follower broadcasts hello every 5 s
|
||||||
HEARTBEAT_INTERVAL = 2.0 # follower sends heartbeat every 2 s
|
HEARTBEAT_INTERVAL = 2.0 # follower sends heartbeat every 2 s
|
||||||
@@ -101,6 +109,7 @@ class DisplaySyncManager:
|
|||||||
self._peer_chain: int = 0
|
self._peer_chain: int = 0
|
||||||
self._last_heartbeat_time: float = 0.0
|
self._last_heartbeat_time: float = 0.0
|
||||||
self._leader_width: int = 0 # set by display_controller after init
|
self._leader_width: int = 0 # set by display_controller after init
|
||||||
|
self._oversized_frame_warned: bool = False
|
||||||
|
|
||||||
# Follower state
|
# Follower state
|
||||||
self._follower_state = FollowerState.STANDALONE
|
self._follower_state = FollowerState.STANDALONE
|
||||||
@@ -174,6 +183,10 @@ class DisplaySyncManager:
|
|||||||
continue
|
continue
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
self.logger.debug("Sync leader recv error: %s", exc)
|
self.logger.debug("Sync leader recv error: %s", exc)
|
||||||
|
# Brief backoff: a socket left in a bad state raises
|
||||||
|
# immediately, which would otherwise spin this thread at
|
||||||
|
# 100% CPU logging the same error.
|
||||||
|
time.sleep(0.1)
|
||||||
|
|
||||||
def _handle_hello(self, msg: dict, sender_ip: str) -> None:
|
def _handle_hello(self, msg: dict, sender_ip: str) -> None:
|
||||||
hw = self._hw_config
|
hw = self._hw_config
|
||||||
@@ -273,11 +286,10 @@ class DisplaySyncManager:
|
|||||||
break
|
break
|
||||||
data.extend(chunk)
|
data.extend(chunk)
|
||||||
img = Image.open(io.BytesIO(data))
|
img = Image.open(io.BytesIO(data))
|
||||||
_MAX_W, _MAX_H = 100_000, 256 # generous for any real scroll image
|
if img.width > _MAX_FRAME_W or img.height > _MAX_FRAME_H:
|
||||||
if img.width > _MAX_W or img.height > _MAX_H:
|
|
||||||
self.logger.warning(
|
self.logger.warning(
|
||||||
"Sync: rejected oversized scroll image %dx%d (max %dx%d) from %s",
|
"Sync: rejected oversized scroll image %dx%d (max %dx%d) from %s",
|
||||||
img.width, img.height, _MAX_W, _MAX_H, addr,
|
img.width, img.height, _MAX_FRAME_W, _MAX_FRAME_H, addr,
|
||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
try:
|
try:
|
||||||
@@ -396,7 +408,7 @@ class DisplaySyncManager:
|
|||||||
data = header + arr.tobytes()
|
data = header + arr.tobytes()
|
||||||
if len(data) <= 65000:
|
if len(data) <= 65000:
|
||||||
self._send_sock.sendto(data, (self._peer_ip, self.port))
|
self._send_sock.sendto(data, (self._peer_ip, self.port))
|
||||||
elif not getattr(self, '_oversized_frame_warned', False):
|
elif not self._oversized_frame_warned:
|
||||||
self._oversized_frame_warned = True
|
self._oversized_frame_warned = True
|
||||||
self.logger.warning(
|
self.logger.warning(
|
||||||
"Sync: frame too large for UDP (%d bytes, max 65000) — "
|
"Sync: frame too large for UDP (%d bytes, max 65000) — "
|
||||||
@@ -451,25 +463,8 @@ class DisplaySyncManager:
|
|||||||
)
|
)
|
||||||
self.write_status_file()
|
self.write_status_file()
|
||||||
|
|
||||||
def _follower_recv_loop(self) -> None:
|
def _handle_received_frame(self, img: Image.Image, sender_ip: str) -> None:
|
||||||
while self._running:
|
"""Record a decoded leader frame and enter follower mode if needed."""
|
||||||
try:
|
|
||||||
data, addr = self._recv_sock.recvfrom(65535)
|
|
||||||
sender_ip = addr[0]
|
|
||||||
|
|
||||||
if data[:8] == _RAW_MAGIC or len(data) > 512:
|
|
||||||
# Frame data: prefer magic-tagged raw RGB; fall back to legacy PNG
|
|
||||||
try:
|
|
||||||
if data[:8] == _RAW_MAGIC:
|
|
||||||
w, h = _RAW_HEADER.unpack(data[8:12])
|
|
||||||
raw = data[12:]
|
|
||||||
img = Image.frombuffer(
|
|
||||||
"RGB", (w, h), raw, "raw", "RGB", 0, 1
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
# Fallback: try legacy PNG
|
|
||||||
img = Image.open(io.BytesIO(data))
|
|
||||||
img.load()
|
|
||||||
with self._frame_lock:
|
with self._frame_lock:
|
||||||
self._latest_frame = img
|
self._latest_frame = img
|
||||||
self._last_leader_frame_time = time.time()
|
self._last_leader_frame_time = time.time()
|
||||||
@@ -482,12 +477,62 @@ class DisplaySyncManager:
|
|||||||
sender_ip,
|
sender_ip,
|
||||||
)
|
)
|
||||||
self.write_status_file()
|
self.write_status_file()
|
||||||
|
|
||||||
|
def _follower_recv_loop(self) -> None:
|
||||||
|
while self._running:
|
||||||
|
try:
|
||||||
|
data, addr = self._recv_sock.recvfrom(65535)
|
||||||
|
sender_ip = addr[0]
|
||||||
|
|
||||||
|
if data[:8] == _RAW_MAGIC:
|
||||||
|
# Magic-tagged raw RGB frame — self-describing, no guessing.
|
||||||
|
try:
|
||||||
|
w, h = _RAW_HEADER.unpack(data[8:12])
|
||||||
|
raw = data[12:]
|
||||||
|
img = Image.frombuffer(
|
||||||
|
"RGB", (w, h), raw, "raw", "RGB", 0, 1
|
||||||
|
)
|
||||||
|
self._handle_received_frame(img, sender_ip)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
self.logger.debug("Sync: frame decode error: %s", exc)
|
self.logger.debug("Sync: frame decode error: %s", exc)
|
||||||
else:
|
else:
|
||||||
# Control message
|
# No magic prefix. Whether the payload parses as JSON
|
||||||
|
# decides between a control message and a legacy
|
||||||
|
# (pre-magic) PNG frame — both wire formats are
|
||||||
|
# self-describing, so no size heuristic is needed. A
|
||||||
|
# >512-byte control message used to be misrouted into
|
||||||
|
# image decode and silently dropped.
|
||||||
try:
|
try:
|
||||||
msg = json.loads(data.decode("utf-8"))
|
msg = json.loads(data.decode("utf-8"))
|
||||||
|
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||||
|
# Not JSON — try a legacy PNG frame.
|
||||||
|
try:
|
||||||
|
img = Image.open(io.BytesIO(data))
|
||||||
|
if img.width > _MAX_FRAME_W or img.height > _MAX_FRAME_H:
|
||||||
|
# Same cap the TCP image path applies: decode
|
||||||
|
# is deferred until load(), so check first.
|
||||||
|
self.logger.debug(
|
||||||
|
"Sync: rejected oversized legacy frame %dx%d from %s",
|
||||||
|
img.width, img.height, sender_ip,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
img.load()
|
||||||
|
self._handle_received_frame(img, sender_ip)
|
||||||
|
except Exception as exc:
|
||||||
|
self.logger.debug("Sync: frame decode error: %s", exc)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# It parsed, so it is a control message and never a
|
||||||
|
# frame. Read and validate its fields under a guard —
|
||||||
|
# a UDP payload is attacker-shaped, so a non-object
|
||||||
|
# body makes .get() raise AttributeError and an "sx"
|
||||||
|
# carrying a non-numeric x raises ValueError/TypeError
|
||||||
|
# — but dispatch the callback *outside* it. Running
|
||||||
|
# the callback in here would let a fault in someone
|
||||||
|
# else's code read as a malformed packet and be
|
||||||
|
# logged as one.
|
||||||
|
fire_new_cycle = False
|
||||||
|
try:
|
||||||
t = msg.get("t")
|
t = msg.get("t")
|
||||||
if t == "hello_ack":
|
if t == "hello_ack":
|
||||||
self._leader_ip = sender_ip
|
self._leader_ip = sender_ip
|
||||||
@@ -501,7 +546,17 @@ class DisplaySyncManager:
|
|||||||
self.write_status_file()
|
self.write_status_file()
|
||||||
elif t == "sx":
|
elif t == "sx":
|
||||||
# Vegas scroll-position sync — tiny message, renders locally
|
# Vegas scroll-position sync — tiny message, renders locally
|
||||||
self._latest_scroll_x = float(msg["x"])
|
scroll_x = float(msg["x"])
|
||||||
|
if not math.isfinite(scroll_x):
|
||||||
|
# json.loads accepts the NaN/Infinity literals,
|
||||||
|
# and float("nan") accepts the strings, so a
|
||||||
|
# non-finite x reaches here intact. Left alone
|
||||||
|
# it poisons every offset computed from it —
|
||||||
|
# NaN comparisons are all false, so the
|
||||||
|
# follower renders a frame it can never scroll
|
||||||
|
# back from. Treat it as malformed.
|
||||||
|
raise ValueError(f"non-finite scroll x: {msg['x']!r}")
|
||||||
|
self._latest_scroll_x = scroll_x
|
||||||
self._last_leader_frame_time = time.time()
|
self._last_leader_frame_time = time.time()
|
||||||
self._leader_ip = sender_ip
|
self._leader_ip = sender_ip
|
||||||
if self._follower_state == FollowerState.STANDALONE:
|
if self._follower_state == FollowerState.STANDALONE:
|
||||||
@@ -511,19 +566,22 @@ class DisplaySyncManager:
|
|||||||
sender_ip,
|
sender_ip,
|
||||||
)
|
)
|
||||||
self.write_status_file()
|
self.write_status_file()
|
||||||
if self._on_new_cycle:
|
fire_new_cycle = True # build initial scroll image
|
||||||
self._on_new_cycle() # build initial scroll image
|
|
||||||
elif t == "nc":
|
elif t == "nc":
|
||||||
# Leader started a new scroll cycle — rebuild local image
|
# Leader started a new scroll cycle — rebuild local image
|
||||||
if self._on_new_cycle:
|
fire_new_cycle = True
|
||||||
|
except (KeyError, AttributeError, TypeError, ValueError) as exc:
|
||||||
|
self.logger.debug("Sync: malformed control message: %s", exc)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if fire_new_cycle and self._on_new_cycle:
|
||||||
self._on_new_cycle()
|
self._on_new_cycle()
|
||||||
except (json.JSONDecodeError, UnicodeDecodeError, KeyError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
except socket.timeout:
|
except socket.timeout:
|
||||||
continue
|
continue
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
self.logger.debug("Sync follower recv error: %s", exc)
|
self.logger.debug("Sync follower recv error: %s", exc)
|
||||||
|
time.sleep(0.1)
|
||||||
|
|
||||||
def _follower_announce_loop(self) -> None:
|
def _follower_announce_loop(self) -> None:
|
||||||
hw = self._hw_config
|
hw = self._hw_config
|
||||||
|
|||||||
+64
-9
@@ -269,20 +269,47 @@ class ConfigManager:
|
|||||||
self.logger.error(error_msg, exc_info=True)
|
self.logger.error(error_msg, exc_info=True)
|
||||||
raise ConfigError(error_msg, config_path=self.config_path) from e
|
raise ConfigError(error_msg, config_path=self.config_path) from e
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _is_parallel_secrets_list(value: Any) -> bool:
|
||||||
|
"""True for the parallel-placeholder list shape emitted by
|
||||||
|
``secret_helpers.separate_secrets`` for array-item secrets: a
|
||||||
|
non-empty list whose elements are ALL dicts (``{}`` marks an item
|
||||||
|
with no secrets). Any other list-shaped secrets value is a
|
||||||
|
whole-key secret (e.g. a list of secret scalars)."""
|
||||||
|
return (isinstance(value, list) and bool(value)
|
||||||
|
and all(isinstance(item, dict) for item in value))
|
||||||
|
|
||||||
def _strip_secrets_recursive(self, data_to_filter: Dict[str, Any], secrets: Dict[str, Any]) -> Dict[str, Any]:
|
def _strip_secrets_recursive(self, data_to_filter: Dict[str, Any], secrets: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
"""Recursively remove secret keys from a dictionary."""
|
"""Recursively remove secret keys from a dictionary."""
|
||||||
result = {}
|
result = {}
|
||||||
for key, value in data_to_filter.items():
|
for key, value in data_to_filter.items():
|
||||||
if key in secrets:
|
if key not in secrets:
|
||||||
if isinstance(value, dict) and isinstance(secrets[key], dict):
|
|
||||||
# This key is a shared group, recurse
|
|
||||||
stripped_sub_dict = self._strip_secrets_recursive(value, secrets[key])
|
|
||||||
if stripped_sub_dict: # Only add if there's non-secret data left
|
|
||||||
result[key] = stripped_sub_dict
|
|
||||||
# Else, it's a secret key at this level, so we skip it
|
|
||||||
else:
|
|
||||||
# This key is not in secrets, so we keep it
|
# This key is not in secrets, so we keep it
|
||||||
result[key] = value
|
result[key] = value
|
||||||
|
continue
|
||||||
|
sec = secrets[key]
|
||||||
|
if isinstance(value, dict) and isinstance(sec, dict):
|
||||||
|
# This key is a shared group, recurse
|
||||||
|
stripped_sub_dict = self._strip_secrets_recursive(value, sec)
|
||||||
|
if stripped_sub_dict: # Only add if there's non-secret data left
|
||||||
|
result[key] = stripped_sub_dict
|
||||||
|
elif isinstance(value, list) and self._is_parallel_secrets_list(sec):
|
||||||
|
# Parallel-list shape from separate_secrets: sec[i] holds the
|
||||||
|
# secret fields of value[i] ({} = item i has none). Strip each
|
||||||
|
# item and ALWAYS keep the list — indices must survive so the
|
||||||
|
# merge-on-load can realign secrets with their items. The
|
||||||
|
# regular list's length is authoritative: extra secrets
|
||||||
|
# entries are ignored.
|
||||||
|
stripped_items = []
|
||||||
|
for i, item in enumerate(value):
|
||||||
|
s_item = sec[i] if i < len(sec) else {}
|
||||||
|
if isinstance(item, dict) and s_item:
|
||||||
|
stripped_items.append(self._strip_secrets_recursive(item, s_item))
|
||||||
|
else:
|
||||||
|
stripped_items.append(item)
|
||||||
|
result[key] = stripped_items
|
||||||
|
# Else: whole-key secret (scalar, list of secret scalars, or a
|
||||||
|
# shape mismatch) -> drop the key entirely. Never leak.
|
||||||
return result
|
return result
|
||||||
|
|
||||||
def _load_secrets_for_save(self) -> Dict[str, Any]:
|
def _load_secrets_for_save(self) -> Dict[str, Any]:
|
||||||
@@ -358,11 +385,39 @@ class ConfigManager:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
def _deep_merge(self, target: Dict[str, Any], source: Dict[str, Any]) -> None:
|
def _deep_merge(self, target: Dict[str, Any], source: Dict[str, Any]) -> None:
|
||||||
"""Deep merge source dict into target dict."""
|
"""Deep merge source dict into target dict.
|
||||||
|
|
||||||
|
Sole call site: merging config_secrets.json into the loaded config.
|
||||||
|
Understands the parallel-list shape separate_secrets emits for
|
||||||
|
array-item secrets (see _is_parallel_secrets_list): each secrets
|
||||||
|
list item is merged into the config list item at the same index
|
||||||
|
({} placeholders skipped). The config list's length is
|
||||||
|
authoritative — a user deleting an array item from config.json
|
||||||
|
must not have it resurrected from a stale secrets entry."""
|
||||||
for key, value in source.items():
|
for key, value in source.items():
|
||||||
if key in target and isinstance(target[key], dict) and isinstance(value, dict):
|
if key in target and isinstance(target[key], dict) and isinstance(value, dict):
|
||||||
self._deep_merge(target[key], value)
|
self._deep_merge(target[key], value)
|
||||||
|
elif (key in target and isinstance(target[key], list)
|
||||||
|
and self._is_parallel_secrets_list(value)):
|
||||||
|
tlist = target[key]
|
||||||
|
for i, s_item in enumerate(value):
|
||||||
|
if i >= len(tlist):
|
||||||
|
# Interpolate only config-side data here — nothing
|
||||||
|
# iterated out of the secrets dict (not even the key
|
||||||
|
# name) may reach the log.
|
||||||
|
self.logger.warning(
|
||||||
|
"A secrets list is longer than the config list it "
|
||||||
|
"parallels (config has %d item(s)); ignoring the "
|
||||||
|
"extra entries", len(tlist))
|
||||||
|
break
|
||||||
|
if not s_item:
|
||||||
|
continue # {} placeholder: item i has no secrets
|
||||||
|
if isinstance(tlist[i], dict):
|
||||||
|
self._deep_merge(tlist[i], s_item)
|
||||||
else:
|
else:
|
||||||
|
tlist[i] = s_item # shape drift; the secret wins
|
||||||
|
else:
|
||||||
|
# Scalars AND whole-secret scalar arrays: replace (legacy).
|
||||||
target[key] = value
|
target[key] = value
|
||||||
|
|
||||||
def _create_config_from_template(self) -> None:
|
def _create_config_from_template(self) -> None:
|
||||||
|
|||||||
+161
-13
@@ -44,6 +44,20 @@ from src.common.sync_manager import DisplaySyncManager, SyncRole
|
|||||||
# Get logger with consistent configuration
|
# Get logger with consistent configuration
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
# How long startup will wait for plugins to fetch their first data before
|
||||||
|
# showing anything. Each plugin's update blocks for up to the executor's 30s
|
||||||
|
# timeout and they run one after another, so the uncapped total is the sum of
|
||||||
|
# every slow plugin: 82 seconds on the worst boot measured, with a blank panel
|
||||||
|
# throughout. Whatever does not finish in time is picked up by the scheduled
|
||||||
|
# update tick moments later, with the display already running.
|
||||||
|
_INITIAL_UPDATE_BUDGET_SECONDS = 20.0
|
||||||
|
|
||||||
|
# The least budget worth starting a plugin with. Below this the plugin is
|
||||||
|
# deferred instead: granting it a floor would let the pass run past its
|
||||||
|
# deadline, and granting it the true remainder would record a timeout for a
|
||||||
|
# slot it never had a chance to use.
|
||||||
|
_MIN_INITIAL_UPDATE_TIMEOUT_SECONDS = 2.0
|
||||||
|
|
||||||
# Vegas mode import (lazy loaded to avoid circular imports)
|
# Vegas mode import (lazy loaded to avoid circular imports)
|
||||||
_vegas_mode_imported = False
|
_vegas_mode_imported = False
|
||||||
VegasModeCoordinator = None
|
VegasModeCoordinator = None
|
||||||
@@ -90,7 +104,8 @@ class DisplayController:
|
|||||||
# Validate startup configuration
|
# Validate startup configuration
|
||||||
try:
|
try:
|
||||||
from src.startup_validator import StartupValidator
|
from src.startup_validator import StartupValidator
|
||||||
validator = StartupValidator(self.config_manager)
|
validator = StartupValidator(self.config_manager,
|
||||||
|
cache_manager=self.cache_manager)
|
||||||
is_valid, errors, warnings = validator.validate_all()
|
is_valid, errors, warnings = validator.validate_all()
|
||||||
|
|
||||||
if warnings:
|
if warnings:
|
||||||
@@ -166,6 +181,16 @@ class DisplayController:
|
|||||||
self.plugin_modes = {} # mode -> plugin_instance mapping for plugin-first dispatch
|
self.plugin_modes = {} # mode -> plugin_instance mapping for plugin-first dispatch
|
||||||
self.mode_to_plugin_id: Dict[str, str] = {}
|
self.mode_to_plugin_id: Dict[str, str] = {}
|
||||||
self.plugin_display_modes: Dict[str, List[str]] = {}
|
self.plugin_display_modes: Dict[str, List[str]] = {}
|
||||||
|
# plugin_display_modes is mutated only by _register_loaded_plugin /
|
||||||
|
# _unregister_plugin on the render thread, but the config-watcher
|
||||||
|
# thread reads it in _enabled_plugin_not_running. Both mutation sites
|
||||||
|
# run during reconcile (rare), so this lock never touches the per-frame
|
||||||
|
# path -- the hot-path reads are same-thread as the writes.
|
||||||
|
self._plugin_modes_lock = threading.Lock()
|
||||||
|
# Guards the consume-and-clear of _pending_plugin_reconcile. Only taken
|
||||||
|
# when a reconcile is actually pending or a config change arrives, both
|
||||||
|
# rare -- the per-frame path just reads the bool.
|
||||||
|
self._reconcile_flag_lock = threading.Lock()
|
||||||
# Per-plugin config-change callbacks, kept so we can unsubscribe a
|
# Per-plugin config-change callbacks, kept so we can unsubscribe a
|
||||||
# plugin when it is disabled live.
|
# plugin when it is disabled live.
|
||||||
self._plugin_config_callbacks: Dict[str, Callable] = {}
|
self._plugin_config_callbacks: Dict[str, Callable] = {}
|
||||||
@@ -258,7 +283,8 @@ class DisplayController:
|
|||||||
# Validate plugins after plugin manager is created
|
# Validate plugins after plugin manager is created
|
||||||
try:
|
try:
|
||||||
from src.startup_validator import StartupValidator
|
from src.startup_validator import StartupValidator
|
||||||
validator = StartupValidator(self.config_manager, self.plugin_manager)
|
validator = StartupValidator(self.config_manager, self.plugin_manager,
|
||||||
|
cache_manager=self.cache_manager)
|
||||||
is_valid, errors, warnings = validator.validate_all()
|
is_valid, errors, warnings = validator.validate_all()
|
||||||
|
|
||||||
if warnings:
|
if warnings:
|
||||||
@@ -447,7 +473,9 @@ class DisplayController:
|
|||||||
self._refresh_config_cache(new_config)
|
self._refresh_config_cache(new_config)
|
||||||
# If a plugin was enabled/disabled, flag a reconcile for the main
|
# If a plugin was enabled/disabled, flag a reconcile for the main
|
||||||
# loop to apply (loading/unloading off the watcher thread is unsafe).
|
# loop to apply (loading/unloading off the watcher thread is unsafe).
|
||||||
if self._enabled_set_changed(old_config, new_config):
|
if (self._enabled_set_changed(old_config, new_config)
|
||||||
|
or self._enabled_plugin_not_running(new_config)):
|
||||||
|
with self._reconcile_flag_lock:
|
||||||
self._pending_plugin_reconcile = True
|
self._pending_plugin_reconcile = True
|
||||||
|
|
||||||
self.config_service.subscribe(_controller_config_change)
|
self.config_service.subscribe(_controller_config_change)
|
||||||
@@ -461,7 +489,7 @@ class DisplayController:
|
|||||||
# Initial data update for plugins (ensures data available on first display)
|
# Initial data update for plugins (ensures data available on first display)
|
||||||
logger.info("Performing initial plugin data update...")
|
logger.info("Performing initial plugin data update...")
|
||||||
update_start = time.time()
|
update_start = time.time()
|
||||||
self._update_modules()
|
self._update_modules(deadline=update_start + _INITIAL_UPDATE_BUDGET_SECONDS)
|
||||||
logger.info("Initial plugin update completed in %.3f seconds", time.time() - update_start)
|
logger.info("Initial plugin update completed in %.3f seconds", time.time() - update_start)
|
||||||
|
|
||||||
# Initialize Vegas mode coordinator
|
# Initialize Vegas mode coordinator
|
||||||
@@ -817,14 +845,42 @@ class DisplayController:
|
|||||||
self._cached_target_brightness = normal_brightness # persist for minute-gate
|
self._cached_target_brightness = normal_brightness # persist for minute-gate
|
||||||
return normal_brightness
|
return normal_brightness
|
||||||
|
|
||||||
def _update_modules(self):
|
def _update_modules(self, deadline: Optional[float] = None):
|
||||||
"""Update all plugin modules."""
|
"""Update all plugin modules.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
deadline: Wall-clock time after which remaining plugins are left
|
||||||
|
for the scheduled update tick instead of being waited on. Each
|
||||||
|
update blocks this thread for up to the executor's timeout, and
|
||||||
|
they run one after another, so without a bound the total is the
|
||||||
|
sum of every slow plugin on the system. Measured at startup on
|
||||||
|
a live rig: 82 seconds, 55 and 26 on the two boots before -- all
|
||||||
|
of it with nothing on the panel.
|
||||||
|
"""
|
||||||
if not self.plugin_manager:
|
if not self.plugin_manager:
|
||||||
return
|
return
|
||||||
|
|
||||||
# Update all loaded plugins
|
# Update all loaded plugins
|
||||||
plugins_dict = getattr(self.plugin_manager, 'loaded_plugins', None) or getattr(self.plugin_manager, 'plugins', {})
|
plugins_dict = getattr(self.plugin_manager, 'loaded_plugins', None) or getattr(self.plugin_manager, 'plugins', {})
|
||||||
|
deferred = []
|
||||||
for plugin_id, plugin_instance in plugins_dict.items():
|
for plugin_id, plugin_instance in plugins_dict.items():
|
||||||
|
update_timeout = None
|
||||||
|
if deadline is not None:
|
||||||
|
update_timeout = deadline - time.time()
|
||||||
|
if update_timeout < _MIN_INITIAL_UPDATE_TIMEOUT_SECONDS:
|
||||||
|
# Too little left to be worth starting. Deferring rather
|
||||||
|
# than granting a floor keeps the budget a real ceiling --
|
||||||
|
# clamping up to a minimum let a plugin that began with a
|
||||||
|
# sliver left run on past the deadline -- and a plugin
|
||||||
|
# handed a slot it cannot use would just be recorded as
|
||||||
|
# having timed out.
|
||||||
|
#
|
||||||
|
# Nothing is lost either way: a plugin that has never
|
||||||
|
# updated is immediately due, so run_scheduled_updates()
|
||||||
|
# picks it up within seconds, with the display already
|
||||||
|
# running.
|
||||||
|
deferred.append(plugin_id)
|
||||||
|
continue
|
||||||
# Check circuit breaker before attempting update
|
# Check circuit breaker before attempting update
|
||||||
if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker:
|
if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker:
|
||||||
if self.plugin_manager.health_tracker.should_skip_plugin(plugin_id):
|
if self.plugin_manager.health_tracker.should_skip_plugin(plugin_id):
|
||||||
@@ -833,7 +889,13 @@ class DisplayController:
|
|||||||
|
|
||||||
# Use PluginExecutor if available for safe execution
|
# Use PluginExecutor if available for safe execution
|
||||||
if hasattr(self.plugin_manager, 'plugin_executor'):
|
if hasattr(self.plugin_manager, 'plugin_executor'):
|
||||||
success = self.plugin_manager.plugin_executor.execute_update(plugin_instance, plugin_id)
|
# The remaining budget is the timeout, so the pass cannot
|
||||||
|
# run past its deadline. Bounding the loop alone did not do
|
||||||
|
# it: the last plugin to start could still block for the
|
||||||
|
# executor's full 30s, which turned a 20s budget into a 31.8s
|
||||||
|
# pass on the rig.
|
||||||
|
success = self.plugin_manager.plugin_executor.execute_update(
|
||||||
|
plugin_instance, plugin_id, timeout=update_timeout)
|
||||||
if success and hasattr(self.plugin_manager, 'plugin_last_update'):
|
if success and hasattr(self.plugin_manager, 'plugin_last_update'):
|
||||||
self.plugin_manager.plugin_last_update[plugin_id] = time.time()
|
self.plugin_manager.plugin_last_update[plugin_id] = time.time()
|
||||||
else:
|
else:
|
||||||
@@ -852,6 +914,12 @@ class DisplayController:
|
|||||||
if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker:
|
if hasattr(self.plugin_manager, 'health_tracker') and self.plugin_manager.health_tracker:
|
||||||
self.plugin_manager.health_tracker.record_failure(plugin_id, exc)
|
self.plugin_manager.health_tracker.record_failure(plugin_id, exc)
|
||||||
|
|
||||||
|
if deferred:
|
||||||
|
logger.info(
|
||||||
|
"Initial update budget spent; %d plugin(s) left to the update "
|
||||||
|
"tick so the display can start: %s",
|
||||||
|
len(deferred), ", ".join(deferred))
|
||||||
|
|
||||||
def _tick_plugin_updates_for_vegas(self) -> None:
|
def _tick_plugin_updates_for_vegas(self) -> None:
|
||||||
"""Run scheduled plugin updates and tell Vegas mode which plugins
|
"""Run scheduled plugin updates and tell Vegas mode which plugins
|
||||||
actually got fresh data, so it can hot-swap them into the scroll
|
actually got fresh data, so it can hot-swap them into the scroll
|
||||||
@@ -1638,6 +1706,12 @@ class DisplayController:
|
|||||||
logger.warning("Error checking live priority for %s: %s", mode_name, e)
|
logger.warning("Error checking live priority for %s: %s", mode_name, e)
|
||||||
return live
|
return live
|
||||||
|
|
||||||
|
def _vegas_keeps_live_in_ticker(self) -> bool:
|
||||||
|
"""Whether live content should stay in the ticker instead of preempting it."""
|
||||||
|
coordinator = getattr(self, 'vegas_coordinator', None)
|
||||||
|
config = getattr(coordinator, 'vegas_config', None)
|
||||||
|
return bool(getattr(config, 'live_in_ticker', False))
|
||||||
|
|
||||||
def _check_live_priority(self, advance=False):
|
def _check_live_priority(self, advance=False):
|
||||||
"""Return the live-priority mode to display, or None if nothing is live.
|
"""Return the live-priority mode to display, or None if nothing is live.
|
||||||
|
|
||||||
@@ -1687,11 +1761,12 @@ class DisplayController:
|
|||||||
# rebuilding available_modes happens here on the render thread so
|
# rebuilding available_modes happens here on the render thread so
|
||||||
# it can't race with rendering. Deferred while on-demand is active
|
# it can't race with rendering. Deferred while on-demand is active
|
||||||
# (the flag stays set) so we don't fight its temporary-enable.
|
# (the flag stays set) so we don't fight its temporary-enable.
|
||||||
|
# The lock-free read is a fast path only; it can be a false
|
||||||
|
# negative (the watcher setting the flag just after it is read
|
||||||
|
# is seen next iteration), never a false positive that loses a
|
||||||
|
# request.
|
||||||
if self._pending_plugin_reconcile and not self.on_demand_active:
|
if self._pending_plugin_reconcile and not self.on_demand_active:
|
||||||
# Only clear the flag on success -- a retryable failure
|
self._service_pending_reconcile()
|
||||||
# (e.g. discovery) leaves it set so the request isn't lost.
|
|
||||||
if self._reconcile_enabled_plugins():
|
|
||||||
self._pending_plugin_reconcile = False
|
|
||||||
|
|
||||||
if not self.available_modes:
|
if not self.available_modes:
|
||||||
# Nothing to render yet. Re-check _pending_plugin_reconcile
|
# Nothing to render yet. Re-check _pending_plugin_reconcile
|
||||||
@@ -1851,14 +1926,24 @@ class DisplayController:
|
|||||||
# Check for live priority content and switch to it immediately.
|
# Check for live priority content and switch to it immediately.
|
||||||
# advance=True so multiple simultaneously-live games take turns
|
# advance=True so multiple simultaneously-live games take turns
|
||||||
# (round-robin) instead of pinning to the first plugin.
|
# (round-robin) instead of pinning to the first plugin.
|
||||||
if not self.on_demand_active and not wifi_status_data:
|
# Skipped when the ticker is keeping live content: switching
|
||||||
|
# the rotation underneath Vegas would move current_mode_index
|
||||||
|
# and stash a resume point for a takeover that never happens.
|
||||||
|
if (not self.on_demand_active and not wifi_status_data
|
||||||
|
and not (self._is_vegas_mode_active()
|
||||||
|
and self._vegas_keeps_live_in_ticker())):
|
||||||
live_priority_mode = self._check_live_priority(advance=True)
|
live_priority_mode = self._check_live_priority(advance=True)
|
||||||
self._apply_live_priority(live_priority_mode)
|
self._apply_live_priority(live_priority_mode)
|
||||||
|
|
||||||
# Vegas scroll mode - continuous ticker across all plugins
|
# Vegas scroll mode - continuous ticker across all plugins
|
||||||
# Priority: on-demand > wifi-status > live-priority > vegas > normal rotation
|
# Priority: on-demand > wifi-status > live-priority > vegas > normal rotation
|
||||||
if self._is_vegas_mode_active() and not wifi_status_data:
|
if self._is_vegas_mode_active() and not wifi_status_data:
|
||||||
live_mode = self._check_live_priority()
|
# Live content normally preempts the ticker entirely. With
|
||||||
|
# vegas_scroll.live_in_ticker the marquee keeps running and
|
||||||
|
# the live plugin takes extra turns inside it instead --
|
||||||
|
# see StreamManager._apply_priority_weights.
|
||||||
|
live_mode = (None if self._vegas_keeps_live_in_ticker()
|
||||||
|
else self._check_live_priority())
|
||||||
if not live_mode:
|
if not live_mode:
|
||||||
try:
|
try:
|
||||||
# Run Vegas mode iteration
|
# Run Vegas mode iteration
|
||||||
@@ -2741,6 +2826,7 @@ class DisplayController:
|
|||||||
logger.debug("Using manifest display_modes for %s: %s", plugin_id, display_modes)
|
logger.debug("Using manifest display_modes for %s: %s", plugin_id, display_modes)
|
||||||
if not (isinstance(display_modes, list) and display_modes):
|
if not (isinstance(display_modes, list) and display_modes):
|
||||||
display_modes = [plugin_id]
|
display_modes = [plugin_id]
|
||||||
|
with self._plugin_modes_lock:
|
||||||
self.plugin_display_modes[plugin_id] = list(display_modes)
|
self.plugin_display_modes[plugin_id] = list(display_modes)
|
||||||
|
|
||||||
# Subscribe to config changes for per-plugin hot-reload. Bind plugin_id
|
# Subscribe to config changes for per-plugin hot-reload. Bind plugin_id
|
||||||
@@ -2775,6 +2861,7 @@ class DisplayController:
|
|||||||
def _unregister_plugin(self, plugin_id: str) -> None:
|
def _unregister_plugin(self, plugin_id: str) -> None:
|
||||||
"""Remove a plugin's modes, config subscription and instance, then
|
"""Remove a plugin's modes, config subscription and instance, then
|
||||||
unload it. Used by live disable hot-reload."""
|
unload it. Used by live disable hot-reload."""
|
||||||
|
with self._plugin_modes_lock:
|
||||||
modes = self.plugin_display_modes.pop(plugin_id, [])
|
modes = self.plugin_display_modes.pop(plugin_id, [])
|
||||||
for mode in modes:
|
for mode in modes:
|
||||||
if mode in self.available_modes:
|
if mode in self.available_modes:
|
||||||
@@ -2820,6 +2907,67 @@ class DisplayController:
|
|||||||
}
|
}
|
||||||
return enabled_map(old_config) != enabled_map(new_config)
|
return enabled_map(old_config) != enabled_map(new_config)
|
||||||
|
|
||||||
|
def _service_pending_reconcile(self) -> None:
|
||||||
|
"""Consume a pending reconcile request and run it.
|
||||||
|
|
||||||
|
The request is consumed BEFORE reconciling, not cleared after. Clearing
|
||||||
|
after would drop any config change that lands while reconcile is
|
||||||
|
running: reconcile has already read its config by then, so the clear
|
||||||
|
erases a request it never served and the newest config never
|
||||||
|
reconciles -- the same "your save did nothing" failure this whole path
|
||||||
|
exists to prevent. Consuming first means such a request stays set and
|
||||||
|
is picked up on the next pass.
|
||||||
|
|
||||||
|
A retryable failure (e.g. discovery) re-arms the flag.
|
||||||
|
"""
|
||||||
|
with self._reconcile_flag_lock:
|
||||||
|
pending = self._pending_plugin_reconcile
|
||||||
|
self._pending_plugin_reconcile = False
|
||||||
|
if pending and not self._reconcile_enabled_plugins():
|
||||||
|
with self._reconcile_flag_lock:
|
||||||
|
self._pending_plugin_reconcile = True
|
||||||
|
|
||||||
|
def _enabled_plugin_not_running(self, new_config: Dict[str, Any]) -> bool:
|
||||||
|
"""True when a discovered plugin is enabled in config but not running.
|
||||||
|
|
||||||
|
``_enabled_set_changed`` compares only top-level ``enabled`` flags, which
|
||||||
|
misses the case that strands a plugin: one whose ``validate_config()``
|
||||||
|
returned False is absent from the running set, and the edit that fixes it
|
||||||
|
(enabling a league, filling in an API key) lives *nested* inside that
|
||||||
|
plugin's own section. No top-level flag changes, so no reconcile is
|
||||||
|
queued, and the save that should have fixed it appears to do nothing --
|
||||||
|
only toggling some unrelated plugin recovers it. hockey-scoreboard sat
|
||||||
|
enabled-but-absent on a live rig for four days this way.
|
||||||
|
|
||||||
|
Deliberately narrow: it fires only for ids the plugin manager has
|
||||||
|
actually discovered, so non-plugin sections that carry their own
|
||||||
|
``enabled`` flag (``schedule``, ``display``, ...) don't queue a reconcile
|
||||||
|
on every save. In the steady state -- everything enabled is loaded --
|
||||||
|
this is False and costs nothing. That matters because reconcile runs
|
||||||
|
``discover_plugins()`` on the render thread, where a needless
|
||||||
|
filesystem scan per config save would show up as a frame hitch.
|
||||||
|
|
||||||
|
Runs on the config-watcher thread, so both mappings it reads are
|
||||||
|
snapshotted under the lock that guards their writes.
|
||||||
|
"""
|
||||||
|
if self.plugin_manager is None:
|
||||||
|
return False
|
||||||
|
# Two snapshots, each taken under its own lock and never nested, so a
|
||||||
|
# half-written mapping is never observed and this can't deadlock
|
||||||
|
# against discovery (which holds the discovery lock while rebuilding).
|
||||||
|
try:
|
||||||
|
known = self.plugin_manager.discovered_plugin_ids()
|
||||||
|
except AttributeError:
|
||||||
|
# Older manager without the accessor: fall back to a plain read.
|
||||||
|
known = set(getattr(self.plugin_manager, 'plugin_manifests', ()) or ())
|
||||||
|
with self._plugin_modes_lock:
|
||||||
|
running = set(self.plugin_display_modes)
|
||||||
|
for key, value in new_config.items():
|
||||||
|
if (key in known and isinstance(value, dict)
|
||||||
|
and value.get('enabled', False) and key not in running):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
def _reconcile_enabled_plugins(self) -> bool:
|
def _reconcile_enabled_plugins(self) -> bool:
|
||||||
"""Load/unload plugins so the running set matches the enabled set in
|
"""Load/unload plugins so the running set matches the enabled set in
|
||||||
config. Runs on the main display thread (never the config-watcher
|
config. Runs on the main display thread (never the config-watcher
|
||||||
|
|||||||
+118
-3
@@ -25,6 +25,7 @@ the same object.
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import socket
|
||||||
import tempfile
|
import tempfile
|
||||||
if os.getenv("EMULATOR", "false") == "true":
|
if os.getenv("EMULATOR", "false") == "true":
|
||||||
from RGBMatrixEmulator import RGBMatrix, RGBMatrixOptions
|
from RGBMatrixEmulator import RGBMatrix, RGBMatrixOptions
|
||||||
@@ -258,6 +259,26 @@ class DisplayManager:
|
|||||||
# Initialize managers
|
# Initialize managers
|
||||||
# Calendar manager is now initialized by DisplayController
|
# Calendar manager is now initialized by DisplayController
|
||||||
|
|
||||||
|
# Orientation setting -> rpi-rgb-led-matrix "Rotate:<deg>" pixel-mapper suffix.
|
||||||
|
# "normal" needs no suffix since 0 degrees is the identity transform.
|
||||||
|
_ORIENTATION_ROTATE_DEGREES = {'normal': None, '90': 90, '180': 180, '270': 270}
|
||||||
|
|
||||||
|
def _build_pixel_mapper_config(self, hardware_config: dict) -> str:
|
||||||
|
"""Compose the raw pixel_mapper_config string with the orientation setting.
|
||||||
|
|
||||||
|
`pixel_mapper_config` stays available as a free-form advanced field (e.g.
|
||||||
|
for "U-mapper" chain layouts); `orientation` is the user-facing dropdown
|
||||||
|
for physical mounting (e.g. panels mounted upside down) and is appended as
|
||||||
|
a "Rotate:<deg>" mapper rather than overwriting any existing config.
|
||||||
|
"""
|
||||||
|
base_mapper = (hardware_config.get('pixel_mapper_config') or '').strip()
|
||||||
|
orientation = hardware_config.get('orientation', 'normal')
|
||||||
|
degrees = self._ORIENTATION_ROTATE_DEGREES.get(orientation)
|
||||||
|
if degrees is None:
|
||||||
|
return base_mapper
|
||||||
|
rotate_mapper = f'Rotate:{degrees}'
|
||||||
|
return f'{base_mapper};{rotate_mapper}' if base_mapper else rotate_mapper
|
||||||
|
|
||||||
def _setup_matrix(self):
|
def _setup_matrix(self):
|
||||||
"""Initialize the RGB matrix with configuration settings."""
|
"""Initialize the RGB matrix with configuration settings."""
|
||||||
_init_error_str = None
|
_init_error_str = None
|
||||||
@@ -283,7 +304,7 @@ class DisplayManager:
|
|||||||
options.pwm_bits = hardware_config.get('pwm_bits', 10)
|
options.pwm_bits = hardware_config.get('pwm_bits', 10)
|
||||||
options.pwm_lsb_nanoseconds = hardware_config.get('pwm_lsb_nanoseconds', 150)
|
options.pwm_lsb_nanoseconds = hardware_config.get('pwm_lsb_nanoseconds', 150)
|
||||||
options.led_rgb_sequence = hardware_config.get('led_rgb_sequence', 'RGB')
|
options.led_rgb_sequence = hardware_config.get('led_rgb_sequence', 'RGB')
|
||||||
options.pixel_mapper_config = hardware_config.get('pixel_mapper_config', '')
|
options.pixel_mapper_config = self._build_pixel_mapper_config(hardware_config)
|
||||||
options.row_address_type = hardware_config.get('row_address_type', 0)
|
options.row_address_type = hardware_config.get('row_address_type', 0)
|
||||||
options.multiplexing = hardware_config.get('multiplexing', 0)
|
options.multiplexing = hardware_config.get('multiplexing', 0)
|
||||||
options.panel_type = hardware_config.get('panel_type', '')
|
options.panel_type = hardware_config.get('panel_type', '')
|
||||||
@@ -343,6 +364,7 @@ class DisplayManager:
|
|||||||
# Create image with the (logical) display dimensions
|
# Create image with the (logical) display dimensions
|
||||||
self.image = Image.new('RGB', (self.matrix.width, self.matrix.height))
|
self.image = Image.new('RGB', (self.matrix.width, self.matrix.height))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs.
|
||||||
logger.info(f"Image canvas created with dimensions: {self.matrix.width}x{self.matrix.height}")
|
logger.info(f"Image canvas created with dimensions: {self.matrix.width}x{self.matrix.height}")
|
||||||
|
|
||||||
# Initialize font with Press Start 2P
|
# Initialize font with Press Start 2P
|
||||||
@@ -382,6 +404,7 @@ class DisplayManager:
|
|||||||
|
|
||||||
self.image = Image.new('RGB', (fallback_width, fallback_height))
|
self.image = Image.new('RGB', (fallback_width, fallback_height))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs.
|
||||||
# Simple fallback visualization so web UI shows a realistic canvas
|
# Simple fallback visualization so web UI shows a realistic canvas
|
||||||
try:
|
try:
|
||||||
self.draw.rectangle([0, 0, fallback_width - 1, fallback_height - 1], outline=(255, 0, 0))
|
self.draw.rectangle([0, 0, fallback_width - 1, fallback_height - 1], outline=(255, 0, 0))
|
||||||
@@ -497,6 +520,91 @@ class DisplayManager:
|
|||||||
logger.warning(f"[BRIGHTNESS] Matrix does not support brightness property: {e}", exc_info=True)
|
logger.warning(f"[BRIGHTNESS] Matrix does not support brightness property: {e}", exc_info=True)
|
||||||
return -1
|
return -1
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _local_ip() -> Optional[str]:
|
||||||
|
"""This device's address on the network it routes through, or None.
|
||||||
|
|
||||||
|
Deliberately not `hostname -I` or a systemctl probe for AP mode, which
|
||||||
|
is how the web launcher does it: both spawn processes with multi-second
|
||||||
|
timeouts, and this runs on the startup path the rest of this change
|
||||||
|
exists to shorten. Connecting a UDP socket sends no packets -- it only
|
||||||
|
asks the kernel which source address it would use -- so it costs
|
||||||
|
microseconds and works with the network down, as long as a route
|
||||||
|
exists.
|
||||||
|
"""
|
||||||
|
sock = None
|
||||||
|
try:
|
||||||
|
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||||
|
sock.settimeout(0.2)
|
||||||
|
sock.connect(("8.8.8.8", 80)) # nosec B104 - no traffic; selects a route
|
||||||
|
ip = sock.getsockname()[0]
|
||||||
|
return ip if ip and not ip.startswith("127.") else None
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
if sock is not None:
|
||||||
|
try:
|
||||||
|
sock.close()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _fitting_font(self, lines, width):
|
||||||
|
"""The largest font from the usual ladder that fits every line."""
|
||||||
|
candidates = [self.font,
|
||||||
|
("assets/fonts/4x6-font.ttf", 6)]
|
||||||
|
for candidate in candidates:
|
||||||
|
try:
|
||||||
|
font = candidate
|
||||||
|
if isinstance(candidate, tuple):
|
||||||
|
font = ImageFont.truetype(candidate[0], candidate[1])
|
||||||
|
if all(self.draw.textlength(t, font=font) <= width for t in lines):
|
||||||
|
return font
|
||||||
|
except (OSError, ValueError, AttributeError):
|
||||||
|
continue
|
||||||
|
return self.font
|
||||||
|
|
||||||
|
def _draw_startup_banner(self, lines, width: int, height: int) -> None:
|
||||||
|
"""Centre `lines` over whatever the test pattern already drew.
|
||||||
|
|
||||||
|
This screen stays on the panel for the whole initial plugin update, and
|
||||||
|
on a headless Pi it is the only place the device's address appears
|
||||||
|
without going looking for it -- so it has to be readable off a wall,
|
||||||
|
not merely present.
|
||||||
|
|
||||||
|
The font is chosen to fit rather than fixed at 8px: "Initializing" is
|
||||||
|
96px in PressStart2P, which ran off the side of a 64px panel even
|
||||||
|
before an address was added. And the pattern is punched out behind the
|
||||||
|
text, because the diagonal runs through the middle of the panel, which
|
||||||
|
is exactly where this sits.
|
||||||
|
|
||||||
|
The text stays blue. It is not decoration: the pattern draws one pure
|
||||||
|
channel per element -- red border, green diagonal, blue text -- so that
|
||||||
|
a glance at the panel says whether led_rgb_sequence is right. Swap the
|
||||||
|
wiring to BGR and the border comes up blue and this text red. Drawing
|
||||||
|
it white would light all three channels and destroy the only blue
|
||||||
|
reference on the screen, which is why it is worth a comment rather
|
||||||
|
than a quiet preference.
|
||||||
|
"""
|
||||||
|
if not lines:
|
||||||
|
return
|
||||||
|
font = self._fitting_font(lines, width - 2)
|
||||||
|
line_height = self.draw.textbbox((0, 0), "Ag", font=font)[3] + 1
|
||||||
|
block_height = line_height * len(lines)
|
||||||
|
block_top = max(1, (height - block_height) // 2)
|
||||||
|
block_width = max(self.draw.textlength(t, font=font) for t in lines)
|
||||||
|
block_left = max(0, (width - block_width) // 2)
|
||||||
|
|
||||||
|
self.draw.rectangle(
|
||||||
|
[block_left - 2, block_top - 1,
|
||||||
|
block_left + block_width + 1, block_top + block_height],
|
||||||
|
fill=(0, 0, 0))
|
||||||
|
|
||||||
|
for row, line in enumerate(lines):
|
||||||
|
line_width = self.draw.textlength(line, font=font)
|
||||||
|
self.draw.text(
|
||||||
|
(max(0, (width - line_width) // 2), block_top + row * line_height),
|
||||||
|
line, font=font, fill=(0, 0, 255))
|
||||||
|
|
||||||
def _draw_test_pattern(self):
|
def _draw_test_pattern(self):
|
||||||
"""Draw a test pattern to verify the display is working."""
|
"""Draw a test pattern to verify the display is working."""
|
||||||
try:
|
try:
|
||||||
@@ -516,8 +624,11 @@ class DisplayManager:
|
|||||||
# Draw a diagonal line
|
# Draw a diagonal line
|
||||||
self.draw.line([0, 0, self.matrix.width-1, self.matrix.height-1], fill=(0, 255, 0))
|
self.draw.line([0, 0, self.matrix.width-1, self.matrix.height-1], fill=(0, 255, 0))
|
||||||
|
|
||||||
# Draw some text - changed from "TEST" to "Initializing" with smaller font
|
lines = ["Initializing"]
|
||||||
self.draw.text((10, 10), "Initializing", font=self.font, fill=(0, 0, 255))
|
ip = self._local_ip()
|
||||||
|
if ip:
|
||||||
|
lines.append(ip)
|
||||||
|
self._draw_startup_banner(lines, self.matrix.width, self.matrix.height)
|
||||||
|
|
||||||
# Update the display once after everything is drawn
|
# Update the display once after everything is drawn
|
||||||
self.update_display()
|
self.update_display()
|
||||||
@@ -596,6 +707,7 @@ class DisplayManager:
|
|||||||
# self.image, so swapping the buffer below is enough on its own.
|
# self.image, so swapping the buffer below is enough on its own.
|
||||||
self.image = Image.new('RGB', (target_w, target_h))
|
self.image = Image.new('RGB', (target_w, target_h))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs.
|
||||||
yield
|
yield
|
||||||
finally:
|
finally:
|
||||||
self.matrix = real_matrix
|
self.matrix = real_matrix
|
||||||
@@ -705,6 +817,7 @@ class DisplayManager:
|
|||||||
|
|
||||||
self.image = Image.new('RGB', (width, height))
|
self.image = Image.new('RGB', (width, height))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs.
|
||||||
logger.debug("Cleared display in fallback mode")
|
logger.debug("Cleared display in fallback mode")
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -716,6 +829,7 @@ class DisplayManager:
|
|||||||
# Create a new black image
|
# Create a new black image
|
||||||
self.image = Image.new('RGB', (self.matrix.width, self.matrix.height))
|
self.image = Image.new('RGB', (self.matrix.width, self.matrix.height))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs.
|
||||||
|
|
||||||
if not self._capture_mode_active:
|
if not self._capture_mode_active:
|
||||||
# Clear both canvases and the underlying matrix to ensure no artifacts.
|
# Clear both canvases and the underlying matrix to ensure no artifacts.
|
||||||
@@ -1149,6 +1263,7 @@ class DisplayManager:
|
|||||||
try:
|
try:
|
||||||
self.image = Image.new('RGB', (self.width, self.height))
|
self.image = Image.new('RGB', (self.width, self.height))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # 1-bit text: the panel has no partial brightness, so AA only smears glyphs.
|
||||||
except (OSError, RuntimeError, ValueError, MemoryError):
|
except (OSError, RuntimeError, ValueError, MemoryError):
|
||||||
logger.debug("Canvas reset during cleanup failed", exc_info=True)
|
logger.debug("Canvas reset during cleanup failed", exc_info=True)
|
||||||
# Reset the singleton state when cleaning up
|
# Reset the singleton state when cleaning up
|
||||||
|
|||||||
@@ -168,9 +168,13 @@ class DynamicTeamResolver:
|
|||||||
# Sort by ranking (1, 2, 3, etc.)
|
# Sort by ranking (1, 2, 3, etc.)
|
||||||
sorted_rankings = dict(sorted(rankings.items(), key=lambda x: x[1]))
|
sorted_rankings = dict(sorted(rankings.items(), key=lambda x: x[1]))
|
||||||
|
|
||||||
# Cache the results
|
# Cache the results ON THE CLASS. Assigning through self
|
||||||
self._rankings_cache = sorted_rankings
|
# would create instance attributes that shadow the shared
|
||||||
self._cache_timestamp = current_time
|
# class-level cache, making it per-instance — and every
|
||||||
|
# scoreboard constructs its own resolver, so the cache
|
||||||
|
# would never actually be shared.
|
||||||
|
DynamicTeamResolver._rankings_cache = sorted_rankings
|
||||||
|
DynamicTeamResolver._cache_timestamp = current_time
|
||||||
|
|
||||||
self.logger.info(f"Fetched rankings for {len(sorted_rankings)} teams")
|
self.logger.info(f"Fetched rankings for {len(sorted_rankings)} teams")
|
||||||
return sorted_rankings
|
return sorted_rankings
|
||||||
@@ -216,9 +220,11 @@ class DynamicTeamResolver:
|
|||||||
return any(pattern in team_name.upper() for pattern in dynamic_patterns)
|
return any(pattern in team_name.upper() for pattern in dynamic_patterns)
|
||||||
|
|
||||||
def clear_cache(self):
|
def clear_cache(self):
|
||||||
"""Clear the rankings cache to force fresh data on next request."""
|
"""Clear the SHARED rankings cache to force fresh data on next
|
||||||
self._rankings_cache = {}
|
request. Writes through the class — assigning via self would only
|
||||||
self._cache_timestamp = 0
|
shadow the shared cache for this instance."""
|
||||||
|
DynamicTeamResolver._rankings_cache = {}
|
||||||
|
DynamicTeamResolver._cache_timestamp = 0
|
||||||
self.logger.info("Cleared dynamic team rankings cache")
|
self.logger.info("Cleared dynamic team rankings cache")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+97
-5
@@ -5,6 +5,7 @@ Provides consistent logging configuration across the LEDMatrix application.
|
|||||||
Supports structured logging with context information and appropriate log levels.
|
Supports structured logging with context information and appropriate log levels.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import copy
|
||||||
import logging
|
import logging
|
||||||
import sys
|
import sys
|
||||||
import os
|
import os
|
||||||
@@ -65,8 +66,12 @@ class ContextualFormatter(logging.Formatter):
|
|||||||
self.include_context = include_context
|
self.include_context = include_context
|
||||||
|
|
||||||
def format(self, record: logging.LogRecord) -> str:
|
def format(self, record: logging.LogRecord) -> str:
|
||||||
"""Format log record with context."""
|
"""Format log record with context.
|
||||||
# Add context to message if present
|
|
||||||
|
Works on a shallow copy of the record: a record is formatted once
|
||||||
|
PER HANDLER, so mutating record.msg in place (the old behavior)
|
||||||
|
prepended the context prefix again for every additional handler.
|
||||||
|
"""
|
||||||
if self.include_context:
|
if self.include_context:
|
||||||
context_parts = []
|
context_parts = []
|
||||||
|
|
||||||
@@ -81,6 +86,7 @@ class ContextualFormatter(logging.Formatter):
|
|||||||
context_parts.append(f"[{key}: {value}]")
|
context_parts.append(f"[{key}: {value}]")
|
||||||
|
|
||||||
if context_parts:
|
if context_parts:
|
||||||
|
record = copy.copy(record)
|
||||||
record.msg = ' '.join(context_parts) + ' ' + str(record.msg)
|
record.msg = ' '.join(context_parts) + ' ' + str(record.msg)
|
||||||
|
|
||||||
return super().format(record)
|
return super().format(record)
|
||||||
@@ -124,7 +130,12 @@ def setup_logging(
|
|||||||
# Console handler (always add)
|
# Console handler (always add)
|
||||||
console_handler = logging.StreamHandler(sys.stdout)
|
console_handler = logging.StreamHandler(sys.stdout)
|
||||||
console_handler.setLevel(level)
|
console_handler.setLevel(level)
|
||||||
console_handler.setFormatter(formatter)
|
# Under systemd, tag each line so the journal records the real severity
|
||||||
|
# rather than filing everything as informational. The file handler below
|
||||||
|
# keeps the plain formatter: the prefix is meaningful to journald and noise
|
||||||
|
# anywhere else.
|
||||||
|
console_handler.setFormatter(
|
||||||
|
JournalPriorityFormatter(formatter) if _under_systemd() else formatter)
|
||||||
root_logger.addHandler(console_handler)
|
root_logger.addHandler(console_handler)
|
||||||
|
|
||||||
# File handler (if specified)
|
# File handler (if specified)
|
||||||
@@ -139,6 +150,84 @@ def setup_logging(
|
|||||||
sys.stderr.write(f"Warning: Could not set up file logging to {log_file}: {e}\n")
|
sys.stderr.write(f"Warning: Could not set up file logging to {log_file}: {e}\n")
|
||||||
|
|
||||||
|
|
||||||
|
#: syslog priorities, which is what systemd parses from a "<N>" prefix on
|
||||||
|
#: stdout. Mapped from Python's levels.
|
||||||
|
_SYSLOG_PRIORITY = {
|
||||||
|
logging.CRITICAL: 2, # LOG_CRIT
|
||||||
|
logging.ERROR: 3, # LOG_ERR
|
||||||
|
logging.WARNING: 4, # LOG_WARNING
|
||||||
|
logging.INFO: 6, # LOG_INFO
|
||||||
|
logging.DEBUG: 7, # LOG_DEBUG
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class JournalPriorityFormatter(logging.Formatter):
|
||||||
|
"""Wraps a formatter, prefixing each line with its syslog priority.
|
||||||
|
|
||||||
|
Under systemd everything this process writes to stdout lands in the journal
|
||||||
|
as PRIORITY=6, whatever the Python level was. Measured on a live rig: 55
|
||||||
|
ERROR lines and 13 WARNING lines in a day, every one of them recorded as
|
||||||
|
informational, so `journalctl -p err -u ledmatrix` returned nothing at all
|
||||||
|
while errors were being logged. Anyone triaging has to grep the message
|
||||||
|
text instead, which is both slower and wrong -- a search for "oom" matches
|
||||||
|
the radar logging "zoom=9".
|
||||||
|
|
||||||
|
systemd reads a leading "<N>" on each line and uses it as the priority
|
||||||
|
(sd-daemon(3)), so this needs no extra dependency. Multi-line records get
|
||||||
|
the prefix on every line, since the journal splits them and an unprefixed
|
||||||
|
continuation would fall back to the default.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, inner: logging.Formatter):
|
||||||
|
super().__init__()
|
||||||
|
self._inner = inner
|
||||||
|
|
||||||
|
@property
|
||||||
|
def inner(self) -> logging.Formatter:
|
||||||
|
"""The formatter doing the actual work.
|
||||||
|
|
||||||
|
Whether journald tagging is applied depends on JOURNAL_STREAM, so it is
|
||||||
|
on under systemd and off in a terminal -- and anything asserting which
|
||||||
|
formatter setup_logging() selected would otherwise get a different
|
||||||
|
answer in CI than on a developer's machine. Exposing the inner one lets
|
||||||
|
those checks stay about format_type, which is what they mean.
|
||||||
|
"""
|
||||||
|
return self._inner
|
||||||
|
|
||||||
|
def format(self, record: logging.LogRecord) -> str:
|
||||||
|
text = self._inner.format(record)
|
||||||
|
prefix = f"<{_SYSLOG_PRIORITY.get(record.levelno, 6)}>"
|
||||||
|
return "\n".join(prefix + line for line in text.split("\n"))
|
||||||
|
|
||||||
|
|
||||||
|
def _under_systemd() -> bool:
|
||||||
|
"""True when stdout really is the journal.
|
||||||
|
|
||||||
|
systemd sets JOURNAL_STREAM to "dev:ino" for services whose output it
|
||||||
|
captures. Presence alone is not enough to act on: the variable is
|
||||||
|
inherited by child processes and survives redirection, so a subprocess
|
||||||
|
whose stdout is a pipe or a file still sees it and would emit the "<N>"
|
||||||
|
priority prefixes as literal noise into that output. systemd's own
|
||||||
|
guidance is to fstat the descriptor and compare st_dev/st_ino, which is
|
||||||
|
what distinguishes "the journal is somewhere in my ancestry" from "my
|
||||||
|
stdout is the journal".
|
||||||
|
"""
|
||||||
|
declared = os.environ.get("JOURNAL_STREAM")
|
||||||
|
if not declared:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
dev_text, ino_text = declared.split(":", 1)
|
||||||
|
declared_ids = (int(dev_text), int(ino_text))
|
||||||
|
except (ValueError, AttributeError):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
stat_result = os.fstat(sys.stdout.fileno())
|
||||||
|
except (OSError, ValueError, AttributeError):
|
||||||
|
# No usable stdout: captured by pytest, detached, or already closed.
|
||||||
|
return False
|
||||||
|
return (stat_result.st_dev, stat_result.st_ino) == declared_ids
|
||||||
|
|
||||||
|
|
||||||
class PluginLoggerAdapter(logging.LoggerAdapter):
|
class PluginLoggerAdapter(logging.LoggerAdapter):
|
||||||
"""LoggerAdapter that stamps every record with its plugin_id.
|
"""LoggerAdapter that stamps every record with its plugin_id.
|
||||||
|
|
||||||
@@ -224,8 +313,11 @@ def log_warning(logger: logging.Logger, message: str, **kwargs) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def log_error(logger: logging.Logger, message: str, **kwargs) -> None:
|
def log_error(logger: logging.Logger, message: str, **kwargs) -> None:
|
||||||
"""Log error message with context."""
|
"""Log error message with context. Defaults exc_info=True; a caller
|
||||||
log_with_context(logger, logging.ERROR, message, **kwargs, exc_info=True)
|
passing exc_info explicitly wins (the old hardcoded keyword raised
|
||||||
|
TypeError on that duplicate)."""
|
||||||
|
kwargs.setdefault('exc_info', True)
|
||||||
|
log_with_context(logger, logging.ERROR, message, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
def log_debug(logger: logging.Logger, message: str, **kwargs) -> None:
|
def log_debug(logger: logging.Logger, message: str, **kwargs) -> None:
|
||||||
|
|||||||
+114
-6
@@ -14,6 +14,7 @@ import json
|
|||||||
from typing import Dict, List, Optional, Tuple
|
from typing import Dict, List, Optional, Tuple
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from PIL import Image, ImageDraw, ImageFont
|
from PIL import Image, ImageDraw, ImageFont
|
||||||
|
from PIL.PngImagePlugin import PngInfo
|
||||||
from requests.adapters import HTTPAdapter
|
from requests.adapters import HTTPAdapter
|
||||||
from urllib3.util.retry import Retry
|
from urllib3.util.retry import Retry
|
||||||
from src.common.permission_utils import (
|
from src.common.permission_utils import (
|
||||||
@@ -25,6 +26,96 @@ from src.common.permission_utils import (
|
|||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
#: PNG text key stamped into a generated placeholder so a later run can tell it
|
||||||
|
#: apart from a real logo that happens to be small.
|
||||||
|
PLACEHOLDER_MARKER = "ledmatrix_placeholder"
|
||||||
|
|
||||||
|
#: Geometry of a generated placeholder, used to recognise ones written before
|
||||||
|
#: the marker existed. Those are already on users' disks and would otherwise
|
||||||
|
#: never be retried.
|
||||||
|
PLACEHOLDER_SIZE = (64, 64)
|
||||||
|
PLACEHOLDER_BG = (100, 100, 100, 255)
|
||||||
|
|
||||||
|
#: How long a placeholder is trusted before the real logo is attempted again.
|
||||||
|
#: A placeholder means the download failed, and download failures are usually
|
||||||
|
#: transient (no network at boot, ESPN blipping). Retrying every frame would
|
||||||
|
#: hammer the API from a Pi that is also driving a panel; never retrying leaves
|
||||||
|
#: the team a grey box forever, which is the bug this exists to avoid.
|
||||||
|
PLACEHOLDER_RETRY_SECONDS = 6 * 60 * 60
|
||||||
|
|
||||||
|
|
||||||
|
def is_placeholder_logo(filepath: Path) -> bool:
|
||||||
|
"""True if the file at ``filepath`` is a generated placeholder, not a logo.
|
||||||
|
|
||||||
|
Checks the marker first, then falls back to matching the placeholder's
|
||||||
|
exact geometry and background colour so files written before the marker was
|
||||||
|
introduced are still recognised.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
with Image.open(filepath) as img:
|
||||||
|
if img.info.get(PLACEHOLDER_MARKER):
|
||||||
|
return True
|
||||||
|
if img.size != PLACEHOLDER_SIZE:
|
||||||
|
return False
|
||||||
|
return img.convert("RGBA").getpixel((0, 0)) == PLACEHOLDER_BG
|
||||||
|
except Exception:
|
||||||
|
# Unreadable file: not provably a placeholder, and the caller's own
|
||||||
|
# error handling is better placed to deal with it.
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def should_attempt_download(filepath: Path, force_download: bool = False) -> bool:
|
||||||
|
"""Whether a real logo is worth (re)fetching for ``filepath``.
|
||||||
|
|
||||||
|
True when nothing is there, when the caller forced it, or when what is
|
||||||
|
there is a placeholder old enough to retry. A *fresh* placeholder says a
|
||||||
|
download just failed, so retrying it immediately would hammer the API for
|
||||||
|
a result that is very unlikely to have changed.
|
||||||
|
"""
|
||||||
|
if force_download or not filepath.exists():
|
||||||
|
return True
|
||||||
|
if not is_placeholder_logo(filepath):
|
||||||
|
return False
|
||||||
|
age = placeholder_age_seconds(filepath)
|
||||||
|
return age is None or age >= PLACEHOLDER_RETRY_SECONDS
|
||||||
|
|
||||||
|
|
||||||
|
def refresh_placeholder_timestamp(filepath: Path) -> bool:
|
||||||
|
"""Restamp a placeholder so a failed retry restarts the back-off clock.
|
||||||
|
|
||||||
|
Without this a stale placeholder stays stale: every later call sees an
|
||||||
|
expired timestamp, retries, fails, and leaves the timestamp untouched --
|
||||||
|
which is a download attempt per call, the opposite of what the back-off is
|
||||||
|
for.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if not is_placeholder_logo(filepath):
|
||||||
|
return False
|
||||||
|
metadata = PngInfo()
|
||||||
|
metadata.add_text(PLACEHOLDER_MARKER, str(time.time()))
|
||||||
|
with Image.open(filepath) as img:
|
||||||
|
img.copy().save(filepath, "PNG", pnginfo=metadata)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
logger.debug("Could not refresh placeholder timestamp for %s", filepath,
|
||||||
|
exc_info=True)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def placeholder_age_seconds(filepath: Path) -> Optional[float]:
|
||||||
|
"""Seconds since a placeholder was written, or None if unknown."""
|
||||||
|
try:
|
||||||
|
with Image.open(filepath) as img:
|
||||||
|
stamped = img.info.get(PLACEHOLDER_MARKER)
|
||||||
|
if stamped and stamped != "1":
|
||||||
|
return max(0.0, time.time() - float(stamped))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
return max(0.0, time.time() - filepath.stat().st_mtime)
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
|
||||||
class LogoDownloader:
|
class LogoDownloader:
|
||||||
"""Centralized logo downloader for team logos from ESPN API."""
|
"""Centralized logo downloader for team logos from ESPN API."""
|
||||||
|
|
||||||
@@ -499,8 +590,10 @@ class LogoDownloader:
|
|||||||
filename = f"{self.normalize_abbreviation(abbreviation)}.png"
|
filename = f"{self.normalize_abbreviation(abbreviation)}.png"
|
||||||
filepath = Path(logo_dir) / filename
|
filepath = Path(logo_dir) / filename
|
||||||
|
|
||||||
# Skip if already exists and not forcing download
|
# A placeholder does not count as existing -- it is a previous
|
||||||
if filepath.exists() and not force_download:
|
# failure, and a bulk pass is exactly where it should get another
|
||||||
|
# chance, subject to the same back-off as everywhere else.
|
||||||
|
if not should_attempt_download(filepath, force_download):
|
||||||
logger.debug(f"Skipping {display_name}: {filename} already exists")
|
logger.debug(f"Skipping {display_name}: {filename} already exists")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
@@ -559,8 +652,9 @@ class LogoDownloader:
|
|||||||
filename = f"{self.normalize_abbreviation(abbreviation)}.png"
|
filename = f"{self.normalize_abbreviation(abbreviation)}.png"
|
||||||
filepath = Path(logo_dir) / filename
|
filepath = Path(logo_dir) / filename
|
||||||
|
|
||||||
# Skip if already exists and not forcing download
|
# Same eligibility rule as every other download site: a stale
|
||||||
if filepath.exists() and not force_download:
|
# placeholder is a failed download, not a logo.
|
||||||
|
if not should_attempt_download(filepath, force_download):
|
||||||
logger.debug(f"Skipping {display_name} ({category}, {conference}): {filename} already exists")
|
logger.debug(f"Skipping {display_name} ({category}, {conference}): {filename} already exists")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
@@ -674,7 +768,12 @@ class LogoDownloader:
|
|||||||
# Fallback without font
|
# Fallback without font
|
||||||
draw.text((16, 24), text, fill=(255, 255, 255, 255))
|
draw.text((16, 24), text, fill=(255, 255, 255, 255))
|
||||||
|
|
||||||
logo.save(filepath)
|
# Stamp it so a later run can tell this apart from a real logo and
|
||||||
|
# retry the download, instead of treating the file's existence as
|
||||||
|
# proof the logo was fetched.
|
||||||
|
metadata = PngInfo()
|
||||||
|
metadata.add_text(PLACEHOLDER_MARKER, str(time.time()))
|
||||||
|
logo.save(filepath, "PNG", pnginfo=metadata)
|
||||||
|
|
||||||
# Set proper file permissions after saving
|
# Set proper file permissions after saving
|
||||||
ensure_file_permissions(filepath, get_assets_file_mode())
|
ensure_file_permissions(filepath, get_assets_file_mode())
|
||||||
@@ -770,9 +869,18 @@ def download_missing_logo(league: str, team_id: str, team_abbreviation: str, log
|
|||||||
# Use the exact filepath that was passed in (respects config settings)
|
# Use the exact filepath that was passed in (respects config settings)
|
||||||
filepath = logo_path
|
filepath = logo_path
|
||||||
|
|
||||||
if filepath.exists():
|
if filepath.exists() and not should_attempt_download(filepath):
|
||||||
|
# Either a real logo, or a placeholder too fresh to be worth retrying.
|
||||||
logger.debug(f"Logo already exists for {team_abbreviation} ({league})")
|
logger.debug(f"Logo already exists for {team_abbreviation} ({league})")
|
||||||
return True
|
return True
|
||||||
|
if filepath.exists():
|
||||||
|
# A placeholder is a *failed* download wearing the real logo's
|
||||||
|
# filename. Treating it as "already exists" is what pinned a team to a
|
||||||
|
# grey box permanently after one transient failure.
|
||||||
|
logger.info(
|
||||||
|
"Logo for %s (%s) is a placeholder from a failed download; "
|
||||||
|
"retrying the real logo", team_abbreviation, league,
|
||||||
|
)
|
||||||
|
|
||||||
# Try to download the real logo first
|
# Try to download the real logo first
|
||||||
logger.info(f"Attempting to download logo for {team_abbreviation} from {league}")
|
logger.info(f"Attempting to download logo for {team_abbreviation} from {league}")
|
||||||
|
|||||||
@@ -364,8 +364,10 @@ class BasePlugin(ABC):
|
|||||||
# Handle None case
|
# Handle None case
|
||||||
if duration is None:
|
if duration is None:
|
||||||
pass # Fall through to config
|
pass # Fall through to config
|
||||||
# Try to convert to float if it's a number or numeric string
|
# Try to convert to float if it's a number or numeric string.
|
||||||
elif isinstance(duration, (int, float)):
|
# bool is excluded: it's an int subclass, and True would
|
||||||
|
# otherwise read as a 1-second duration.
|
||||||
|
elif isinstance(duration, (int, float)) and not isinstance(duration, bool):
|
||||||
if duration > 0:
|
if duration > 0:
|
||||||
return float(duration)
|
return float(duration)
|
||||||
else:
|
else:
|
||||||
@@ -403,8 +405,9 @@ class BasePlugin(ABC):
|
|||||||
# Fall back to config
|
# Fall back to config
|
||||||
config_duration = self.config.get("display_duration", 15.0)
|
config_duration = self.config.get("display_duration", 15.0)
|
||||||
try:
|
try:
|
||||||
# Ensure config value is also a valid float
|
# Ensure config value is also a valid float (bool excluded — an
|
||||||
if isinstance(config_duration, (int, float)):
|
# int subclass that would otherwise read True as 1 second)
|
||||||
|
if isinstance(config_duration, (int, float)) and not isinstance(config_duration, bool):
|
||||||
if config_duration > 0:
|
if config_duration > 0:
|
||||||
return float(config_duration)
|
return float(config_duration)
|
||||||
else:
|
else:
|
||||||
@@ -552,6 +555,48 @@ class BasePlugin(ABC):
|
|||||||
"""
|
"""
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def get_vegas_priority_weight(self) -> Optional[int]:
|
||||||
|
"""How many slots per Vegas cycle this plugin should get, or None.
|
||||||
|
|
||||||
|
The Vegas ticker is otherwise a strict round robin: every plugin
|
||||||
|
appears exactly once per cycle. With a dozen plugins enabled that puts
|
||||||
|
minutes between a live score and its next appearance. A weight of N
|
||||||
|
gives the plugin N slots per cycle, spread evenly through it rather
|
||||||
|
than clumped together.
|
||||||
|
|
||||||
|
Return ``None`` (the default) to let the core decide. It gives a
|
||||||
|
plugin ``vegas_scroll.live_weight`` when ``has_live_priority()`` and
|
||||||
|
``has_live_content()`` are both true, and 1 otherwise -- so live sports
|
||||||
|
already get extra turns without implementing this at all.
|
||||||
|
|
||||||
|
Implement it only when the plugin knows something the core cannot. The
|
||||||
|
motivating case is favorite teams: the core can see *that* a game is
|
||||||
|
live but not *whose*, so a scoreboard that wants its favorite's game
|
||||||
|
shown more often than other live games has to say so::
|
||||||
|
|
||||||
|
def get_vegas_priority_weight(self):
|
||||||
|
if not (self.has_live_priority() and self.has_live_content()):
|
||||||
|
return None # let the core decide
|
||||||
|
cfg = self.global_config.get('display', {}).get('vegas_scroll', {})
|
||||||
|
if self._favorite_is_live():
|
||||||
|
return cfg.get('favorite_live_weight', 5)
|
||||||
|
return cfg.get('live_weight', 3)
|
||||||
|
|
||||||
|
The weight is per *plugin*, not per game. A scoreboard showing four
|
||||||
|
live games still occupies one slot at a time and rotates its own games
|
||||||
|
within that slot; this controls how often the plugin itself comes
|
||||||
|
round.
|
||||||
|
|
||||||
|
Raising is safe: the core logs it and falls back to its own
|
||||||
|
live-content check, so a broken weight calculation costs the plugin
|
||||||
|
the favorite distinction but not the live boost.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Slots per cycle (clamped to 1..10 by the caller), or None to
|
||||||
|
defer to the core's own live-content weighting.
|
||||||
|
"""
|
||||||
|
return None
|
||||||
|
|
||||||
def get_live_modes(self) -> List[str]:
|
def get_live_modes(self) -> List[str]:
|
||||||
"""
|
"""
|
||||||
Get list of display modes that should be used during live priority takeover.
|
Get list of display modes that should be used during live priority takeover.
|
||||||
@@ -794,10 +839,12 @@ class BasePlugin(ABC):
|
|||||||
self.logger.error("'enabled' must be a boolean")
|
self.logger.error("'enabled' must be a boolean")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
# Check display_duration if present
|
# Check display_duration if present. bool is excluded explicitly:
|
||||||
|
# it's an int subclass, and get_display_duration rejects it too.
|
||||||
if "display_duration" in self.config:
|
if "display_duration" in self.config:
|
||||||
duration = self.config["display_duration"]
|
duration = self.config["display_duration"]
|
||||||
if not isinstance(duration, (int, float)) or duration <= 0:
|
if (not isinstance(duration, (int, float))
|
||||||
|
or isinstance(duration, bool) or duration <= 0):
|
||||||
self.logger.error("'display_duration' must be a positive number")
|
self.logger.error("'display_duration' must be a positive number")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ fixes their version string. See `docs/SPORTS_UNIFICATION.md`, phase B4.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import re
|
import re
|
||||||
|
from pathlib import Path
|
||||||
from typing import Any, Dict, Optional, Tuple
|
from typing import Any, Dict, Optional, Tuple
|
||||||
|
|
||||||
# Below this, the core's self-reported version is not evidence of anything.
|
# Below this, the core's self-reported version is not evidence of anything.
|
||||||
@@ -35,6 +36,45 @@ from typing import Any, Dict, Optional, Tuple
|
|||||||
TRUSTWORTHY_FLOOR: Tuple[int, int, int] = (2, 0, 0)
|
TRUSTWORTHY_FLOOR: Tuple[int, int, int] = (2, 0, 0)
|
||||||
|
|
||||||
|
|
||||||
|
# ``src/__init__.py`` relative to this file: src/plugin_system/ -> src/
|
||||||
|
_VERSION_FILE = Path(__file__).resolve().parent.parent / "__init__.py"
|
||||||
|
_VERSION_RE = re.compile(r'^__version__\s*=\s*["\']([^"\']+)["\']', re.M)
|
||||||
|
|
||||||
|
|
||||||
|
def current_core_version() -> str:
|
||||||
|
"""The core version as it is on disk right now, not as it was at import.
|
||||||
|
|
||||||
|
``from src import __version__`` binds whatever the process loaded at start.
|
||||||
|
The web UI runs as its own long-lived service (``ledmatrix-web.service``),
|
||||||
|
and updating the core replaces files on disk without restarting it -- the
|
||||||
|
update route says so explicitly and asks the user to restart. Its prompt
|
||||||
|
names the *display* service, so a user who follows it leaves the web
|
||||||
|
process holding the old number.
|
||||||
|
|
||||||
|
The plugin store's gate lives in that web process. Stale by one release is
|
||||||
|
exactly the case that matters: every plugin flooring on the release you
|
||||||
|
just installed gets refused, with a message blaming a core version that is
|
||||||
|
already correct on disk. 3.3.0 is the first release where that hits a whole
|
||||||
|
plugin family at once -- all eight sports scoreboards floor there.
|
||||||
|
|
||||||
|
Reading the file costs one stat and a small read per call, and only on the
|
||||||
|
install/update path. Any failure falls back to the imported value, so this
|
||||||
|
can only ever be as wrong as before, never worse.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
text = _VERSION_FILE.read_text(encoding="utf-8")
|
||||||
|
match = _VERSION_RE.search(text)
|
||||||
|
if match:
|
||||||
|
return match.group(1)
|
||||||
|
except (OSError, UnicodeDecodeError):
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
from src import __version__ as imported
|
||||||
|
return imported
|
||||||
|
except Exception: # noqa: BLE001 - never let this raise
|
||||||
|
return "0.0.0"
|
||||||
|
|
||||||
|
|
||||||
def parse_semver(value: Any) -> Optional[Tuple[int, int, int]]:
|
def parse_semver(value: Any) -> Optional[Tuple[int, int, int]]:
|
||||||
"""Parse ``X.Y.Z`` (extra parts and suffixes ignored) into a comparable
|
"""Parse ``X.Y.Z`` (extra parts and suffixes ignored) into a comparable
|
||||||
3-tuple, or ``None`` when unparseable. A leading ``v`` is tolerated."""
|
3-tuple, or ``None`` when unparseable. A leading ``v`` is tolerated."""
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ and circuit breaker state. Provides automatic recovery mechanisms.
|
|||||||
|
|
||||||
import time
|
import time
|
||||||
import logging
|
import logging
|
||||||
from typing import Dict, Optional, Any
|
from typing import Dict, Optional, Any, Tuple
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
|
|
||||||
|
|
||||||
@@ -64,10 +64,48 @@ class PluginHealthTracker:
|
|||||||
cache_key, max_age=None, memory_ttl=0 if force_reload else None
|
cache_key, max_age=None, memory_ttl=0 if force_reload else None
|
||||||
)
|
)
|
||||||
|
|
||||||
if cached:
|
if isinstance(cached, dict) and cached:
|
||||||
return cached
|
# Complete it rather than trusting it: a persisted record can be
|
||||||
|
# missing fields the callers index directly (a partial write, a
|
||||||
|
# restored backup, an older schema), and returning it verbatim makes
|
||||||
|
# record_success / record_failure raise KeyError, which takes the
|
||||||
|
# display down in a restart loop that survives reboots because the
|
||||||
|
# bad entry is on disk.
|
||||||
|
state, repaired = self._repair_health_state(cached)
|
||||||
|
if repaired:
|
||||||
|
self.logger.warning(
|
||||||
|
f"Repaired health state for {plugin_id}: "
|
||||||
|
f"{sorted(repaired)} missing or invalid, using defaults for those."
|
||||||
|
)
|
||||||
|
return state
|
||||||
|
|
||||||
# Default state
|
# Not a dict at all: written by something other than
|
||||||
|
# _save_health_state (a key collision, a corrupted entry). Nothing to
|
||||||
|
# salvage.
|
||||||
|
if cached is not None and not isinstance(cached, dict):
|
||||||
|
self.logger.warning(
|
||||||
|
f"Discarding malformed health state for {plugin_id}: expected "
|
||||||
|
f"dict, got {type(cached).__name__}. Falling back to defaults."
|
||||||
|
)
|
||||||
|
|
||||||
|
return self._default_health_state()
|
||||||
|
|
||||||
|
def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None:
|
||||||
|
"""Save health state to cache."""
|
||||||
|
cache_key = self._get_health_key(plugin_id)
|
||||||
|
self.cache_manager.set(cache_key, state) # Persist indefinitely
|
||||||
|
self._health_state[plugin_id] = state
|
||||||
|
|
||||||
|
# The fields callers index directly (state['circuit_state'] and friends).
|
||||||
|
# A cached dict missing any of them raises KeyError deep in record_success /
|
||||||
|
# record_failure, so the value is completed before it is handed out.
|
||||||
|
_COUNTER_FIELDS = ('consecutive_failures', 'total_failures', 'total_successes')
|
||||||
|
_TIMESTAMP_FIELDS = ('last_success_time', 'last_failure_time',
|
||||||
|
'circuit_opened_time', 'half_open_start_time')
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _default_health_state() -> Dict[str, Any]:
|
||||||
|
"""A fresh state with every field the callers expect."""
|
||||||
return {
|
return {
|
||||||
'consecutive_failures': 0,
|
'consecutive_failures': 0,
|
||||||
'total_failures': 0,
|
'total_failures': 0,
|
||||||
@@ -77,14 +115,55 @@ class PluginHealthTracker:
|
|||||||
'circuit_state': CircuitState.CLOSED.value,
|
'circuit_state': CircuitState.CLOSED.value,
|
||||||
'circuit_opened_time': None,
|
'circuit_opened_time': None,
|
||||||
'half_open_start_time': None,
|
'half_open_start_time': None,
|
||||||
'last_error': None
|
'last_error': None,
|
||||||
}
|
}
|
||||||
|
|
||||||
def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None:
|
@classmethod
|
||||||
"""Save health state to cache."""
|
def _repair_health_state(cls, cached: Dict[str, Any]) -> Tuple[Dict[str, Any], list]:
|
||||||
cache_key = self._get_health_key(plugin_id)
|
"""Return `cached` completed against the defaults, plus what was repaired.
|
||||||
self.cache_manager.set(cache_key, state) # Persist indefinitely
|
|
||||||
self._health_state[plugin_id] = state
|
Per-field rather than all-or-nothing: a record that has real failure
|
||||||
|
counts but is missing `last_error` should keep the counts, not be reset
|
||||||
|
to healthy. Only values that are absent or the wrong type fall back to
|
||||||
|
the default, so a partial or older-schema record survives with whatever
|
||||||
|
it does carry, while every field the callers index is guaranteed present
|
||||||
|
and of a usable type.
|
||||||
|
"""
|
||||||
|
state = cls._default_health_state()
|
||||||
|
repaired = []
|
||||||
|
for field, default in state.items():
|
||||||
|
if field not in cached:
|
||||||
|
repaired.append(field)
|
||||||
|
continue
|
||||||
|
value = cached[field]
|
||||||
|
if field in cls._COUNTER_FIELDS:
|
||||||
|
ok = isinstance(value, int) and not isinstance(value, bool) and value >= 0
|
||||||
|
elif field in cls._TIMESTAMP_FIELDS:
|
||||||
|
# bool is a subclass of int, so True would pass as a timestamp
|
||||||
|
# and then compare as 1.0 -- expiring a cooldown the instant it
|
||||||
|
# opens, or (False) making the elapsed check never fire.
|
||||||
|
ok = value is None or (
|
||||||
|
isinstance(value, (int, float)) and not isinstance(value, bool)
|
||||||
|
)
|
||||||
|
elif field == 'circuit_state':
|
||||||
|
# Membership first requires the value to be hashable: a list or
|
||||||
|
# dict here would raise TypeError out of the repair itself,
|
||||||
|
# which is the crash this whole path exists to prevent.
|
||||||
|
ok = isinstance(value, str) and value in {
|
||||||
|
member.value for member in CircuitState
|
||||||
|
}
|
||||||
|
else: # last_error
|
||||||
|
ok = value is None or isinstance(value, str)
|
||||||
|
if ok:
|
||||||
|
state[field] = value
|
||||||
|
else:
|
||||||
|
repaired.append(field)
|
||||||
|
# Anything the schema has since grown (degraded, degraded_reason) is
|
||||||
|
# read with .get() by its callers, so carry it through untouched.
|
||||||
|
for field, value in cached.items():
|
||||||
|
if field not in state:
|
||||||
|
state[field] = value
|
||||||
|
return state, repaired
|
||||||
|
|
||||||
def get_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]:
|
def get_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]:
|
||||||
"""Get current health state for a plugin.
|
"""Get current health state for a plugin.
|
||||||
@@ -99,10 +178,20 @@ class PluginHealthTracker:
|
|||||||
)
|
)
|
||||||
return self._health_state[plugin_id]
|
return self._health_state[plugin_id]
|
||||||
|
|
||||||
|
# Fields the circuit breaker is rebuilt from after a restart. Everything
|
||||||
|
# else in a health record is reporting, read only for display.
|
||||||
|
_DURABLE_FIELDS = ('consecutive_failures', 'circuit_state',
|
||||||
|
'circuit_opened_time', 'half_open_start_time')
|
||||||
|
|
||||||
|
def _durable(self, state: Dict[str, Any]) -> tuple:
|
||||||
|
"""The part of a health record whose loss would change behaviour."""
|
||||||
|
return tuple(state.get(field) for field in self._DURABLE_FIELDS)
|
||||||
|
|
||||||
def record_success(self, plugin_id: str) -> None:
|
def record_success(self, plugin_id: str) -> None:
|
||||||
"""Record a successful plugin execution."""
|
"""Record a successful plugin execution."""
|
||||||
state = self.get_health_state(plugin_id)
|
state = self.get_health_state(plugin_id)
|
||||||
current_time = time.time()
|
current_time = time.time()
|
||||||
|
durable_before = self._durable(state)
|
||||||
|
|
||||||
# Reset consecutive failures
|
# Reset consecutive failures
|
||||||
state['consecutive_failures'] = 0
|
state['consecutive_failures'] = 0
|
||||||
@@ -120,6 +209,17 @@ class PluginHealthTracker:
|
|||||||
state['circuit_state'] = CircuitState.CLOSED.value
|
state['circuit_state'] = CircuitState.CLOSED.value
|
||||||
state['circuit_opened_time'] = None
|
state['circuit_opened_time'] = None
|
||||||
|
|
||||||
|
# A healthy plugin reports success every cycle, and in that steady state
|
||||||
|
# the only fields changed above are a counter and a timestamp that
|
||||||
|
# nothing reads back after a restart. Persisting them anyway rewrites a
|
||||||
|
# small file per plugin per cycle: on a rig running 24 plugins, a
|
||||||
|
# five-minute sample measured 22 rewrites, about 4.4 a minute or 6,300 a
|
||||||
|
# day. Those land on an SD card, where the cost is an erase-block cycle
|
||||||
|
# rather than the 400 bytes involved, and where wear is what eventually
|
||||||
|
# kills the card.
|
||||||
|
# In-memory state is still updated every time, so the health API and web
|
||||||
|
# UI show exactly what they did before; only the write is skipped.
|
||||||
|
if self._durable(state) != durable_before:
|
||||||
self._save_health_state(plugin_id, state)
|
self._save_health_state(plugin_id, state)
|
||||||
|
|
||||||
def record_failure(self, plugin_id: str, error: Optional[Exception] = None) -> None:
|
def record_failure(self, plugin_id: str, error: Optional[Exception] = None) -> None:
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import sys
|
|||||||
import subprocess
|
import subprocess
|
||||||
import threading
|
import threading
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Dict, Any, Optional, Tuple, Type
|
from typing import Dict, Any, List, Optional, Tuple, Type
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
from packaging.requirements import InvalidRequirement, Requirement
|
from packaging.requirements import InvalidRequirement, Requirement
|
||||||
@@ -45,6 +45,76 @@ def requirements_has_real_deps(requirements_file: str) -> bool:
|
|||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _extra_dependencies(dist_name: str, extras) -> Optional[List[Requirement]]:
|
||||||
|
"""Dependencies a distribution declares *only* behind the given extras.
|
||||||
|
|
||||||
|
Returns None when the installed metadata cannot be read or parsed, so the
|
||||||
|
caller can fall back to running pip rather than assuming anything.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
meta = importlib.metadata.metadata(dist_name)
|
||||||
|
except importlib.metadata.PackageNotFoundError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
gated: List[Requirement] = []
|
||||||
|
for raw in meta.get_all('Requires-Dist') or []:
|
||||||
|
try:
|
||||||
|
dep = Requirement(raw)
|
||||||
|
except InvalidRequirement:
|
||||||
|
return None
|
||||||
|
if dep.marker is None:
|
||||||
|
continue
|
||||||
|
# Keep only what the distribution gates behind an extra we asked for:
|
||||||
|
# satisfied when `extra` is that name, but not when no extra is
|
||||||
|
# requested. A marker that holds either way (python_version, sys_platform)
|
||||||
|
# belongs to the base install and is already covered by the version check.
|
||||||
|
if dep.marker.evaluate({'extra': ''}):
|
||||||
|
continue
|
||||||
|
if any(dep.marker.evaluate({'extra': extra}) for extra in extras):
|
||||||
|
gated.append(dep)
|
||||||
|
return gated
|
||||||
|
|
||||||
|
|
||||||
|
def _extras_are_satisfied(req: Requirement, _visited: Optional[set] = None) -> bool:
|
||||||
|
"""Check the dependencies pulled in by req's extras are installed.
|
||||||
|
|
||||||
|
Follows extras through nested extras. A gated dependency can itself request
|
||||||
|
one (`requests[socks]`), and checking only that `requests` is installed at
|
||||||
|
an acceptable version says nothing about whether the socks extra's own
|
||||||
|
dependency is there -- so the caller would skip pip and the plugin would
|
||||||
|
fail at import instead. Plain dependencies are still checked one level
|
||||||
|
deep, which is all that is needed to tell "the extra was installed" from
|
||||||
|
"the extra was never installed".
|
||||||
|
|
||||||
|
`_visited` carries the (distribution, extras) pairs already seen, so a
|
||||||
|
dependency cycle between extras terminates instead of recursing forever.
|
||||||
|
Anything unreadable returns False, so the caller still falls through to pip.
|
||||||
|
"""
|
||||||
|
if _visited is None:
|
||||||
|
_visited = set()
|
||||||
|
marker = (req.name.lower(), frozenset(e.lower() for e in req.extras))
|
||||||
|
if marker in _visited:
|
||||||
|
# Already accounted for higher up the chain; treating a cycle as
|
||||||
|
# satisfied here is safe because the outer frame still has to pass.
|
||||||
|
return True
|
||||||
|
_visited.add(marker)
|
||||||
|
|
||||||
|
gated = _extra_dependencies(req.name, req.extras)
|
||||||
|
if gated is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
for dep in gated:
|
||||||
|
try:
|
||||||
|
dep_version = importlib.metadata.version(dep.name)
|
||||||
|
except importlib.metadata.PackageNotFoundError:
|
||||||
|
return False
|
||||||
|
if dep.specifier and not dep.specifier.contains(dep_version, prereleases=True):
|
||||||
|
return False
|
||||||
|
if dep.extras and not _extras_are_satisfied(dep, _visited):
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
def requirements_are_satisfied(requirements_file: str) -> bool:
|
def requirements_are_satisfied(requirements_file: str) -> bool:
|
||||||
"""
|
"""
|
||||||
Check whether every real requirement line in requirements.txt is already
|
Check whether every real requirement line in requirements.txt is already
|
||||||
@@ -76,9 +146,6 @@ def requirements_are_satisfied(requirements_file: str) -> bool:
|
|||||||
except InvalidRequirement:
|
except InvalidRequirement:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
if req.extras:
|
|
||||||
return False # verifying extras' sub-dependencies isn't worth it here
|
|
||||||
|
|
||||||
if req.marker is not None and not req.marker.evaluate():
|
if req.marker is not None and not req.marker.evaluate():
|
||||||
continue # not applicable on this platform/interpreter
|
continue # not applicable on this platform/interpreter
|
||||||
|
|
||||||
@@ -90,6 +157,9 @@ def requirements_are_satisfied(requirements_file: str) -> bool:
|
|||||||
if req.specifier and not req.specifier.contains(installed_version, prereleases=True):
|
if req.specifier and not req.specifier.contains(installed_version, prereleases=True):
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
if req.extras and not _extras_are_satisfied(req):
|
||||||
|
return False
|
||||||
|
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
@@ -702,8 +772,8 @@ class PluginLoader:
|
|||||||
newer than the running core. Advisory only — never raises — so a
|
newer than the running core. Advisory only — never raises — so a
|
||||||
plugin that guards optional features with try/except keeps working.
|
plugin that guards optional features with try/except keeps working.
|
||||||
"""
|
"""
|
||||||
from src import __version__ as core_version
|
|
||||||
from src.plugin_system import compatibility
|
from src.plugin_system import compatibility
|
||||||
|
core_version = compatibility.current_core_version()
|
||||||
|
|
||||||
compatible, _reason = compatibility.check(manifest, core_version)
|
compatible, _reason = compatibility.check(manifest, core_version)
|
||||||
if compatible:
|
if compatible:
|
||||||
|
|||||||
@@ -71,11 +71,15 @@ class PluginManager:
|
|||||||
self.plugin_loader = PluginLoader(logger=self.logger)
|
self.plugin_loader = PluginLoader(logger=self.logger)
|
||||||
self.plugin_executor = PluginExecutor(default_timeout=30.0, logger=self.logger)
|
self.plugin_executor = PluginExecutor(default_timeout=30.0, logger=self.logger)
|
||||||
self.state_manager = PluginStateManager(logger=self.logger)
|
self.state_manager = PluginStateManager(logger=self.logger)
|
||||||
self.schema_manager = SchemaManager(plugins_dir=self.plugins_dir, logger=self.logger)
|
self.schema_manager = SchemaManager(plugins_dir=self.plugins_dir, logger=self.logger,
|
||||||
|
config_manager=self.config_manager)
|
||||||
|
|
||||||
# Lock protecting plugin_manifests and plugin_directories from
|
# Lock protecting plugin_manifests and plugin_directories from
|
||||||
# concurrent mutation (background reconciliation) and reads (requests).
|
# concurrent mutation (background reconciliation) and reads (requests).
|
||||||
self._discovery_lock = threading.RLock()
|
self._discovery_lock = threading.RLock()
|
||||||
|
#: Directories already reported as unloadable, so the warning is
|
||||||
|
#: emitted once rather than on every discovery scan.
|
||||||
|
self._skip_reported: set = set()
|
||||||
|
|
||||||
# Lock protecting plugin_last_update from concurrent mutation/iteration.
|
# Lock protecting plugin_last_update from concurrent mutation/iteration.
|
||||||
# It's written from run_scheduled_updates()/update_all_plugins() (main
|
# It's written from run_scheduled_updates()/update_all_plugins() (main
|
||||||
@@ -125,6 +129,14 @@ class PluginManager:
|
|||||||
self._plugin_locks: Dict[str, threading.Lock] = {}
|
self._plugin_locks: Dict[str, threading.Lock] = {}
|
||||||
self._plugin_locks_guard = threading.Lock()
|
self._plugin_locks_guard = threading.Lock()
|
||||||
self._update_worker: Optional[threading.Thread] = None
|
self._update_worker: Optional[threading.Thread] = None
|
||||||
|
# Plugin ids whose update() has finished since the last time anyone
|
||||||
|
# asked. Updates are dispatched to a worker thread, so a caller that
|
||||||
|
# wants to know "whose data just changed" cannot learn it by diffing
|
||||||
|
# plugin_last_update around run_scheduled_updates() -- that call only
|
||||||
|
# enqueues, and the timestamp is stamped later, on the worker. See
|
||||||
|
# run_scheduled_updates_with_changes().
|
||||||
|
self._completed_updates: set = set()
|
||||||
|
self._completed_updates_lock = threading.Lock()
|
||||||
self._synchronous_updates = False
|
self._synchronous_updates = False
|
||||||
if self.config_manager is not None:
|
if self.config_manager is not None:
|
||||||
try:
|
try:
|
||||||
@@ -187,18 +199,59 @@ class PluginManager:
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
manifest_path = item / "manifest.json"
|
manifest_path = item / "manifest.json"
|
||||||
if manifest_path.exists():
|
if not manifest_path.exists():
|
||||||
|
# Once per directory per process. Discovery runs on every
|
||||||
|
# web UI page load and every config reconcile, so warning
|
||||||
|
# unconditionally would put a line in the journal each
|
||||||
|
# time someone opened a page -- the same log-volume
|
||||||
|
# problem this is meant to help diagnose.
|
||||||
|
# A directory here that carries no manifest is not a
|
||||||
|
# plugin. Said once, because the alternative is a plugin
|
||||||
|
# that is enabled in config, enabled in plugin state,
|
||||||
|
# present on disk, and simply absent from the running
|
||||||
|
# process with nothing anywhere to say why. Working that
|
||||||
|
# out afterwards means reading cache-file mtimes.
|
||||||
|
if item.name not in self._skip_reported:
|
||||||
|
self._skip_reported.add(item.name)
|
||||||
|
self.logger.warning(
|
||||||
|
"Skipping %s: no manifest.json, so it cannot be "
|
||||||
|
"loaded as a plugin", item.name)
|
||||||
|
continue
|
||||||
try:
|
try:
|
||||||
with open(manifest_path, 'r', encoding='utf-8') as f:
|
with open(manifest_path, 'r', encoding='utf-8') as f:
|
||||||
manifest = json.load(f)
|
manifest = json.load(f)
|
||||||
plugin_id = manifest.get('id')
|
|
||||||
if plugin_id:
|
|
||||||
plugin_ids.append(plugin_id)
|
|
||||||
new_manifests[plugin_id] = manifest
|
|
||||||
new_directories[plugin_id] = item
|
|
||||||
except (json.JSONDecodeError, PermissionError, OSError) as e:
|
except (json.JSONDecodeError, PermissionError, OSError) as e:
|
||||||
self.logger.warning("Error reading manifest from %s: %s", manifest_path, e, exc_info=True)
|
self.logger.warning("Error reading manifest from %s: %s", manifest_path, e, exc_info=True)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
# json.load accepts any JSON value, so a manifest holding
|
||||||
|
# null, [] or "text" parses and then raises AttributeError on
|
||||||
|
# .get(). Nothing here catches that -- the outer handler takes
|
||||||
|
# OSError/PermissionError only -- so a single malformed
|
||||||
|
# manifest aborted the whole scan and every other plugin on
|
||||||
|
# disk, however healthy, silently failed to register.
|
||||||
|
if not isinstance(manifest, dict):
|
||||||
|
if item.name not in self._skip_reported:
|
||||||
|
self._skip_reported.add(item.name)
|
||||||
|
self.logger.warning(
|
||||||
|
"Skipping %s: its manifest.json is %s, not a JSON "
|
||||||
|
"object", item.name, type(manifest).__name__)
|
||||||
|
continue
|
||||||
|
|
||||||
|
plugin_id = manifest.get('id')
|
||||||
|
if not plugin_id:
|
||||||
|
# Parsed but unusable. This was the quietest path of all:
|
||||||
|
# the manifest is read successfully and then dropped.
|
||||||
|
if item.name not in self._skip_reported:
|
||||||
|
self._skip_reported.add(item.name)
|
||||||
|
self.logger.warning(
|
||||||
|
"Skipping %s: its manifest.json has no \"id\", so "
|
||||||
|
"there is nothing to register it under", item.name)
|
||||||
|
continue
|
||||||
|
|
||||||
|
plugin_ids.append(plugin_id)
|
||||||
|
new_manifests[plugin_id] = manifest
|
||||||
|
new_directories[plugin_id] = item
|
||||||
except (OSError, PermissionError) as e:
|
except (OSError, PermissionError) as e:
|
||||||
self.logger.error("Error scanning directory %s: %s", directory, e, exc_info=True)
|
self.logger.error("Error scanning directory %s: %s", directory, e, exc_info=True)
|
||||||
|
|
||||||
@@ -578,6 +631,17 @@ class PluginManager:
|
|||||||
|
|
||||||
return self.load_plugin(plugin_id)
|
return self.load_plugin(plugin_id)
|
||||||
|
|
||||||
|
def discovered_plugin_ids(self) -> set:
|
||||||
|
"""Snapshot of the discovered plugin ids, taken under the discovery lock.
|
||||||
|
|
||||||
|
Callers on other threads (the config watcher) must not iterate
|
||||||
|
``plugin_manifests`` directly: discovery rebuilds it entry by entry, so
|
||||||
|
an unsynchronised reader can see a half-populated mapping or raise
|
||||||
|
"dictionary changed size during iteration".
|
||||||
|
"""
|
||||||
|
with self._discovery_lock:
|
||||||
|
return set(self.plugin_manifests)
|
||||||
|
|
||||||
def get_plugin(self, plugin_id: str) -> Optional[Any]:
|
def get_plugin(self, plugin_id: str) -> Optional[Any]:
|
||||||
"""
|
"""
|
||||||
Get a loaded plugin instance by ID.
|
Get a loaded plugin instance by ID.
|
||||||
@@ -1025,6 +1089,7 @@ class PluginManager:
|
|||||||
if success:
|
if success:
|
||||||
with self._plugin_last_update_lock:
|
with self._plugin_last_update_lock:
|
||||||
self.plugin_last_update[plugin_id] = scheduled_time
|
self.plugin_last_update[plugin_id] = scheduled_time
|
||||||
|
self._note_update_completed(plugin_id)
|
||||||
self.state_manager.record_update(plugin_id)
|
self.state_manager.record_update(plugin_id)
|
||||||
self.state_manager.set_state(plugin_id, PluginState.ENABLED)
|
self.state_manager.set_state(plugin_id, PluginState.ENABLED)
|
||||||
if self.health_tracker:
|
if self.health_tracker:
|
||||||
@@ -1089,28 +1154,41 @@ class PluginManager:
|
|||||||
|
|
||||||
def run_scheduled_updates_with_changes(self, current_time: Optional[float] = None) -> List[str]:
|
def run_scheduled_updates_with_changes(self, current_time: Optional[float] = None) -> List[str]:
|
||||||
"""
|
"""
|
||||||
Like run_scheduled_updates(), but also returns the plugin_ids whose
|
Like run_scheduled_updates(), but also reports which plugins have
|
||||||
plugin_last_update timestamp actually advanced during this call.
|
fresh data -- the ids whose update() has finished since the last
|
||||||
|
call, not necessarily the ones enqueued by this one.
|
||||||
|
|
||||||
The before/after snapshots and the update pass itself are each
|
That distinction is the whole point. This used to snapshot
|
||||||
individually lock-protected against concurrent plugin_last_update
|
plugin_last_update, call run_scheduled_updates(), and diff. But
|
||||||
mutation (Vegas mode calls this from its own background
|
run_scheduled_updates() only *enqueues*: the work runs on the
|
||||||
update-tick thread, racing the main render loop's plugin updates),
|
update worker and the timestamp is stamped there, after this method
|
||||||
so callers get an atomic "who got fresh data" answer without
|
has already returned. The two snapshots were therefore always
|
||||||
reaching into plugin_last_update themselves. The lock is not held
|
identical and the result was always empty, so Vegas never learned
|
||||||
across the update pass so slow/blocking plugin update() calls don't
|
that any plugin's data had changed and kept scrolling whatever a
|
||||||
serialize against other plugin_last_update readers.
|
segment was first built from -- last night's live game still drawn
|
||||||
|
as live the next morning. The only path that ever worked was the
|
||||||
|
synchronous kill-switch, where update() runs inline.
|
||||||
|
|
||||||
|
Reporting completions instead of enqueues costs a poll's worth of
|
||||||
|
latency (the Vegas tick runs every ~4s) and is correct regardless of
|
||||||
|
which side of the queue the work lands on.
|
||||||
"""
|
"""
|
||||||
with self._plugin_last_update_lock:
|
|
||||||
old_times = dict(self.plugin_last_update)
|
|
||||||
|
|
||||||
self.run_scheduled_updates(current_time)
|
self.run_scheduled_updates(current_time)
|
||||||
|
return self.drain_completed_updates()
|
||||||
|
|
||||||
with self._plugin_last_update_lock:
|
def _note_update_completed(self, plugin_id: str) -> None:
|
||||||
return [
|
"""Record that a plugin's update() finished, for the next poll."""
|
||||||
plugin_id for plugin_id, new_time in self.plugin_last_update.items()
|
with self._completed_updates_lock:
|
||||||
if new_time > old_times.get(plugin_id, 0.0)
|
self._completed_updates.add(plugin_id)
|
||||||
]
|
|
||||||
|
def drain_completed_updates(self) -> List[str]:
|
||||||
|
"""Return and clear the plugin ids whose update() has since finished."""
|
||||||
|
with self._completed_updates_lock:
|
||||||
|
if not self._completed_updates:
|
||||||
|
return []
|
||||||
|
done = sorted(self._completed_updates)
|
||||||
|
self._completed_updates.clear()
|
||||||
|
return done
|
||||||
|
|
||||||
def update_all_plugins(self) -> None:
|
def update_all_plugins(self) -> None:
|
||||||
"""
|
"""
|
||||||
@@ -1135,6 +1213,7 @@ class PluginManager:
|
|||||||
if success:
|
if success:
|
||||||
with self._plugin_last_update_lock:
|
with self._plugin_last_update_lock:
|
||||||
self.plugin_last_update[plugin_id] = time.time()
|
self.plugin_last_update[plugin_id] = time.time()
|
||||||
|
self._note_update_completed(plugin_id)
|
||||||
self.state_manager.record_update(plugin_id)
|
self.state_manager.record_update(plugin_id)
|
||||||
self.state_manager.set_state(plugin_id, PluginState.ENABLED)
|
self.state_manager.set_state(plugin_id, PluginState.ENABLED)
|
||||||
else:
|
else:
|
||||||
|
|||||||
@@ -6,14 +6,40 @@ with state transitions and queries.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import threading
|
import threading
|
||||||
|
import time
|
||||||
|
from collections import deque
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
from typing import Optional, Dict, Any
|
from typing import Optional, Dict, Any, Deque, List, Tuple
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
from src.logging_config import get_logger
|
from src.logging_config import get_logger
|
||||||
|
|
||||||
|
|
||||||
|
# The history is diagnostic only -- nothing reads the entries themselves, just
|
||||||
|
# their count -- but it is appended to on the hot scheduling path: every update
|
||||||
|
# cycle records RUNNING on reserve and ENABLED on finish. Unbounded, that is
|
||||||
|
# 2,880 entries per plugin per day at the default 60s interval, which on a 1 GB
|
||||||
|
# Pi exhausts memory in weeks.
|
||||||
|
#
|
||||||
|
# Two limits, because a single entry count answers the wrong question. What a
|
||||||
|
# reader wants is "the last couple of hours", and how many transitions that is
|
||||||
|
# depends entirely on the plugin's update interval -- which on a real board
|
||||||
|
# spans 2s to 3600s. A flat 200 entries is 4.2 days for the slowest plugin and
|
||||||
|
# 3.3 minutes for the fastest, so the plugin churning hardest, the one worth
|
||||||
|
# looking at, keeps the least history.
|
||||||
|
#
|
||||||
|
# So: trim by AGE first, which makes the retained window comparable across
|
||||||
|
# plugins whatever their cadence...
|
||||||
|
STATE_HISTORY_MAX_AGE_SECONDS = 2 * 60 * 60
|
||||||
|
|
||||||
|
# ...and cap by COUNT second, purely as a memory ceiling for the fast pollers
|
||||||
|
# whose age window would otherwise run to thousands of entries. At ~230 bytes
|
||||||
|
# an entry this is ~0.5 MB per plugin worst case, and only plugins updating
|
||||||
|
# faster than roughly every 4s can reach it.
|
||||||
|
MAX_STATE_HISTORY_PER_PLUGIN = 2000
|
||||||
|
|
||||||
|
|
||||||
class PluginState(Enum):
|
class PluginState(Enum):
|
||||||
"""Plugin state enumeration."""
|
"""Plugin state enumeration."""
|
||||||
UNLOADED = "unloaded" # Plugin not loaded
|
UNLOADED = "unloaded" # Plugin not loaded
|
||||||
@@ -37,11 +63,43 @@ class PluginStateManager:
|
|||||||
self.logger = logger or get_logger(__name__)
|
self.logger = logger or get_logger(__name__)
|
||||||
self._lock = threading.RLock()
|
self._lock = threading.RLock()
|
||||||
self._states: Dict[str, PluginState] = {}
|
self._states: Dict[str, PluginState] = {}
|
||||||
self._state_history: Dict[str, list] = {}
|
# (monotonic timestamp, transition). The clock is monotonic so a DST
|
||||||
|
# shift or an NTP step cannot make entries look old and flush the
|
||||||
|
# history; the human-readable timestamp lives inside the transition.
|
||||||
|
self._state_history: Dict[str, Deque[Tuple[float, Dict[str, Any]]]] = {}
|
||||||
|
# Lifetime transition totals, kept separately so the count reported by
|
||||||
|
# get_state_info() stays truthful once the history above starts rolling.
|
||||||
|
self._state_transition_counts: Dict[str, int] = {}
|
||||||
self._error_info: Dict[str, Dict[str, Any]] = {}
|
self._error_info: Dict[str, Dict[str, Any]] = {}
|
||||||
self._last_update: Dict[str, datetime] = {}
|
self._last_update: Dict[str, datetime] = {}
|
||||||
self._last_display: Dict[str, datetime] = {}
|
self._last_display: Dict[str, datetime] = {}
|
||||||
|
|
||||||
|
def _record_transition(
|
||||||
|
self,
|
||||||
|
plugin_id: str,
|
||||||
|
transition: Dict[str, Any]
|
||||||
|
) -> None:
|
||||||
|
"""Append a transition to the plugin's bounded history.
|
||||||
|
|
||||||
|
Callers must already hold ``_lock``. The deque discards its oldest
|
||||||
|
entry once it is full, so the history cannot grow without bound; the
|
||||||
|
lifetime total is tracked separately for get_state_info().
|
||||||
|
"""
|
||||||
|
history = self._state_history.get(plugin_id)
|
||||||
|
if history is None:
|
||||||
|
history = deque(maxlen=MAX_STATE_HISTORY_PER_PLUGIN)
|
||||||
|
self._state_history[plugin_id] = history
|
||||||
|
now = time.monotonic()
|
||||||
|
history.append((now, transition))
|
||||||
|
# Age out first; the deque's maxlen is the backstop for plugins that
|
||||||
|
# produce more than the ceiling within the window.
|
||||||
|
cutoff = now - STATE_HISTORY_MAX_AGE_SECONDS
|
||||||
|
while history and history[0][0] < cutoff:
|
||||||
|
history.popleft()
|
||||||
|
self._state_transition_counts[plugin_id] = (
|
||||||
|
self._state_transition_counts.get(plugin_id, 0) + 1
|
||||||
|
)
|
||||||
|
|
||||||
def set_state(
|
def set_state(
|
||||||
self,
|
self,
|
||||||
plugin_id: str,
|
plugin_id: str,
|
||||||
@@ -60,16 +118,13 @@ class PluginStateManager:
|
|||||||
old_state = self._states.get(plugin_id, PluginState.UNLOADED)
|
old_state = self._states.get(plugin_id, PluginState.UNLOADED)
|
||||||
self._states[plugin_id] = state
|
self._states[plugin_id] = state
|
||||||
|
|
||||||
if plugin_id not in self._state_history:
|
|
||||||
self._state_history[plugin_id] = []
|
|
||||||
|
|
||||||
transition = {
|
transition = {
|
||||||
'timestamp': datetime.now(),
|
'timestamp': datetime.now(),
|
||||||
'from': old_state.value,
|
'from': old_state.value,
|
||||||
'to': state.value,
|
'to': state.value,
|
||||||
'error': str(error) if error else None
|
'error': str(error) if error else None
|
||||||
}
|
}
|
||||||
self._state_history[plugin_id].append(transition)
|
self._record_transition(plugin_id, transition)
|
||||||
|
|
||||||
# Store error info if transitioning to ERROR state
|
# Store error info if transitioning to ERROR state
|
||||||
if state == PluginState.ERROR and error:
|
if state == PluginState.ERROR and error:
|
||||||
@@ -126,17 +181,29 @@ class PluginStateManager:
|
|||||||
state = self.get_state(plugin_id)
|
state = self.get_state(plugin_id)
|
||||||
return state == PluginState.ENABLED
|
return state == PluginState.ENABLED
|
||||||
|
|
||||||
def get_state_history(self, plugin_id: str) -> list:
|
def get_state_history(self, plugin_id: str) -> List[Dict[str, Any]]:
|
||||||
"""
|
"""
|
||||||
Get state transition history for a plugin.
|
Get state transition history for a plugin.
|
||||||
|
|
||||||
|
Retention is by age first -- transitions older than
|
||||||
|
STATE_HISTORY_MAX_AGE_SECONDS are dropped -- and by count second, at
|
||||||
|
MAX_STATE_HISTORY_PER_PLUGIN, which only binds for plugins updating
|
||||||
|
fast enough to exceed it inside that window.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
plugin_id: Plugin identifier
|
plugin_id: Plugin identifier
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of state transitions
|
List of recent state transitions, oldest first. Both the list and
|
||||||
|
the transition dicts are copies, so callers cannot mutate the
|
||||||
|
manager's own history. The values inside a transition are all
|
||||||
|
immutable, so a shallow copy per entry is enough.
|
||||||
"""
|
"""
|
||||||
return self._state_history.get(plugin_id, [])
|
with self._lock:
|
||||||
|
return [
|
||||||
|
dict(transition)
|
||||||
|
for _stamp, transition in self._state_history.get(plugin_id, ())
|
||||||
|
]
|
||||||
|
|
||||||
def set_error_info(self, plugin_id: str, error_info: Dict[str, Any]) -> None:
|
def set_error_info(self, plugin_id: str, error_info: Dict[str, Any]) -> None:
|
||||||
"""
|
"""
|
||||||
@@ -179,9 +246,7 @@ class PluginStateManager:
|
|||||||
old_state = self._states.get(plugin_id, PluginState.UNLOADED)
|
old_state = self._states.get(plugin_id, PluginState.UNLOADED)
|
||||||
self._states[plugin_id] = state
|
self._states[plugin_id] = state
|
||||||
|
|
||||||
if plugin_id not in self._state_history:
|
self._record_transition(plugin_id, {
|
||||||
self._state_history[plugin_id] = []
|
|
||||||
self._state_history[plugin_id].append({
|
|
||||||
'timestamp': datetime.now(),
|
'timestamp': datetime.now(),
|
||||||
'from': old_state.value,
|
'from': old_state.value,
|
||||||
'to': state.value,
|
'to': state.value,
|
||||||
@@ -241,6 +306,13 @@ class PluginStateManager:
|
|||||||
Returns:
|
Returns:
|
||||||
Dictionary with state information
|
Dictionary with state information
|
||||||
"""
|
"""
|
||||||
|
# One snapshot, one critical section. Each field was read under its own
|
||||||
|
# lock, so an unload running concurrently could be observed half-done:
|
||||||
|
# 'state' read before clear_state() removed it and
|
||||||
|
# 'state_history_count' read after, giving a caller a plugin that is
|
||||||
|
# ENABLED with zero transitions. _lock is an RLock, so the helpers
|
||||||
|
# below can still take it.
|
||||||
|
with self._lock:
|
||||||
state = self.get_state(plugin_id)
|
state = self.get_state(plugin_id)
|
||||||
info = {
|
info = {
|
||||||
'state': state.value,
|
'state': state.value,
|
||||||
@@ -252,14 +324,21 @@ class PluginStateManager:
|
|||||||
'last_update': self.get_last_update(plugin_id),
|
'last_update': self.get_last_update(plugin_id),
|
||||||
'last_display': self.get_last_display(plugin_id),
|
'last_display': self.get_last_display(plugin_id),
|
||||||
'error_info': self.get_error_info(plugin_id),
|
'error_info': self.get_error_info(plugin_id),
|
||||||
'state_history_count': len(self.get_state_history(plugin_id))
|
'state_history_count': self._state_transition_counts.get(plugin_id, 0)
|
||||||
}
|
}
|
||||||
return info
|
return info
|
||||||
|
|
||||||
def clear_state(self, plugin_id: str) -> None:
|
def clear_state(self, plugin_id: str) -> None:
|
||||||
"""Clear all state information for a plugin."""
|
"""Clear all state information for a plugin.
|
||||||
|
|
||||||
|
Held under ``_lock`` so the five dicts are dropped as one unit: every
|
||||||
|
other mutator takes the lock, and without it a concurrent set_state()
|
||||||
|
could interleave and leave a plugin with history but no state.
|
||||||
|
"""
|
||||||
|
with self._lock:
|
||||||
self._states.pop(plugin_id, None)
|
self._states.pop(plugin_id, None)
|
||||||
self._state_history.pop(plugin_id, None)
|
self._state_history.pop(plugin_id, None)
|
||||||
|
self._state_transition_counts.pop(plugin_id, None)
|
||||||
self._error_info.pop(plugin_id, None)
|
self._error_info.pop(plugin_id, None)
|
||||||
self._last_update.pop(plugin_id, None)
|
self._last_update.pop(plugin_id, None)
|
||||||
self._last_display.pop(plugin_id, None)
|
self._last_display.pop(plugin_id, None)
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import time
|
|||||||
import logging
|
import logging
|
||||||
import threading
|
import threading
|
||||||
from typing import Dict, Optional, Any, Callable
|
from typing import Dict, Optional, Any, Callable
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field, fields
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import psutil
|
import psutil
|
||||||
@@ -49,6 +49,20 @@ class ResourceMetrics:
|
|||||||
self.total_execution_time = self.total_execution_time / self.call_count
|
self.total_execution_time = self.total_execution_time / self.call_count
|
||||||
|
|
||||||
|
|
||||||
|
#: How often a plugin's metrics are written to the cache, in seconds.
|
||||||
|
#:
|
||||||
|
#: Persisting on every call meant a small file rewritten roughly nine times a
|
||||||
|
#: minute per plugin. On a rig with fourteen active plugins that was ~126
|
||||||
|
#: writes a minute for metrics alone, and since each ~350-byte file costs a
|
||||||
|
#: 4KB block plus an ext4 journal entry, it dominated the device's write
|
||||||
|
#: volume -- on an SD card, which wears out.
|
||||||
|
#:
|
||||||
|
#: The in-memory copy stays authoritative and exact; only the cross-process
|
||||||
|
#: snapshot the web UI reads is delayed, and telemetry up to half a minute old
|
||||||
|
#: is still a fair description of a long-running plugin.
|
||||||
|
_METRICS_PERSIST_INTERVAL = 30.0
|
||||||
|
|
||||||
|
|
||||||
class PluginResourceMonitor:
|
class PluginResourceMonitor:
|
||||||
"""
|
"""
|
||||||
Monitors resource usage for plugins.
|
Monitors resource usage for plugins.
|
||||||
@@ -75,6 +89,10 @@ class PluginResourceMonitor:
|
|||||||
# Resource metrics per plugin
|
# Resource metrics per plugin
|
||||||
self._metrics: Dict[str, ResourceMetrics] = {}
|
self._metrics: Dict[str, ResourceMetrics] = {}
|
||||||
self._limits: Dict[str, ResourceLimits] = {}
|
self._limits: Dict[str, ResourceLimits] = {}
|
||||||
|
# When each plugin's metrics last reached the cache. Metrics change on
|
||||||
|
# every call, so they cannot be de-duplicated the way health state can;
|
||||||
|
# they are rate-limited instead. See _METRICS_PERSIST_INTERVAL.
|
||||||
|
self._metrics_persisted_at: Dict[str, float] = {}
|
||||||
|
|
||||||
# Thread-local storage for execution tracking
|
# Thread-local storage for execution tracking
|
||||||
self._local = threading.local()
|
self._local = threading.local()
|
||||||
@@ -102,6 +120,66 @@ class PluginResourceMonitor:
|
|||||||
"psutil not available - resource monitoring will be limited to execution time only"
|
"psutil not available - resource monitoring will be limited to execution time only"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def _metrics_from_cache(self, plugin_id: str, cached: Any) -> "ResourceMetrics":
|
||||||
|
"""Build metrics from a cached record, ignoring anything unrecognised.
|
||||||
|
|
||||||
|
ResourceMetrics(**cached) raises TypeError on a single unexpected key,
|
||||||
|
and that exception escapes into plugin_manager, which reports it as
|
||||||
|
"plugin <id> operation failed". Every plugin fails, and the plugin
|
||||||
|
system never finishes initialising.
|
||||||
|
|
||||||
|
Seen on a live rig: every plugin failing with
|
||||||
|
|
||||||
|
ResourceMetrics.__init__() got an unexpected keyword argument
|
||||||
|
'consecutive_failures'
|
||||||
|
|
||||||
|
which is a plugin_health field, not a metrics one. How a health-shaped
|
||||||
|
record came to sit under a plugin_metrics key on that machine is not
|
||||||
|
established -- a restored backup that mixed two machines' caches is the
|
||||||
|
likeliest explanation -- but the loader should not be brittle enough for
|
||||||
|
it to matter. plugin_health already repairs its records field by field
|
||||||
|
rather than trusting whatever is on disk; this does the same.
|
||||||
|
|
||||||
|
Unknown keys are dropped and named once, so a genuine schema change is
|
||||||
|
visible in the log instead of silently discarded.
|
||||||
|
"""
|
||||||
|
if not isinstance(cached, dict):
|
||||||
|
self.logger.warning(
|
||||||
|
"Ignoring cached metrics for %s: expected a mapping, got %s",
|
||||||
|
plugin_id, type(cached).__name__)
|
||||||
|
return ResourceMetrics()
|
||||||
|
|
||||||
|
known = {f.name for f in fields(ResourceMetrics)}
|
||||||
|
unknown = sorted(set(cached) - known)
|
||||||
|
if unknown:
|
||||||
|
self.logger.warning(
|
||||||
|
"Dropping unrecognised field(s) from cached metrics for %s: %s",
|
||||||
|
plugin_id, ", ".join(unknown))
|
||||||
|
# A dataclass does not enforce its annotations, so
|
||||||
|
# ResourceMetrics(call_count="not a number") builds happily and only
|
||||||
|
# blows up later, deep inside monitor_call ("can only concatenate str
|
||||||
|
# (not \"int\") to str"). Coerce here, where there is still a cache
|
||||||
|
# key to name in the warning.
|
||||||
|
declared = {f.name: f.type for f in fields(ResourceMetrics)}
|
||||||
|
usable = {}
|
||||||
|
for key, value in cached.items():
|
||||||
|
if key not in known:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
usable[key] = int(value) if declared[key] in ('int', int) else float(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
self.logger.warning(
|
||||||
|
"Cached metrics for %s have a bad %s (%r); starting fresh",
|
||||||
|
plugin_id, key, value)
|
||||||
|
return ResourceMetrics()
|
||||||
|
try:
|
||||||
|
return ResourceMetrics(**usable)
|
||||||
|
except (TypeError, ValueError) as e:
|
||||||
|
self.logger.warning(
|
||||||
|
"Cached metrics for %s unusable (%s); starting fresh",
|
||||||
|
plugin_id, e)
|
||||||
|
return ResourceMetrics()
|
||||||
|
|
||||||
def _get_metrics_key(self, plugin_id: str) -> str:
|
def _get_metrics_key(self, plugin_id: str) -> str:
|
||||||
"""Get cache key for plugin metrics."""
|
"""Get cache key for plugin metrics."""
|
||||||
return f"plugin_metrics:{plugin_id}"
|
return f"plugin_metrics:{plugin_id}"
|
||||||
@@ -126,7 +204,7 @@ class PluginResourceMonitor:
|
|||||||
cache_key, max_age=None, memory_ttl=0 if force_reload else None
|
cache_key, max_age=None, memory_ttl=0 if force_reload else None
|
||||||
)
|
)
|
||||||
if cached:
|
if cached:
|
||||||
metrics = ResourceMetrics(**cached)
|
metrics = self._metrics_from_cache(plugin_id, cached)
|
||||||
else:
|
else:
|
||||||
metrics = ResourceMetrics()
|
metrics = ResourceMetrics()
|
||||||
self._metrics[plugin_id] = metrics
|
self._metrics[plugin_id] = metrics
|
||||||
@@ -232,18 +310,8 @@ class PluginResourceMonitor:
|
|||||||
# CPU is harder to measure per-call, so we track it separately
|
# CPU is harder to measure per-call, so we track it separately
|
||||||
metrics.cpu_percent = self._get_process_cpu_percent()
|
metrics.cpu_percent = self._get_process_cpu_percent()
|
||||||
|
|
||||||
# Persist metrics
|
# Persist metrics, at most once per interval per plugin.
|
||||||
cache_key = self._get_metrics_key(plugin_id)
|
self._persist_metrics(plugin_id, metrics)
|
||||||
self.cache_manager.set(cache_key, {
|
|
||||||
'memory_mb': metrics.memory_mb,
|
|
||||||
'cpu_percent': metrics.cpu_percent,
|
|
||||||
'execution_time': metrics.execution_time,
|
|
||||||
'call_count': metrics.call_count,
|
|
||||||
'total_execution_time': metrics.total_execution_time,
|
|
||||||
'max_execution_time': metrics.max_execution_time,
|
|
||||||
'min_execution_time': metrics.min_execution_time if metrics.min_execution_time != float('inf') else 0.0,
|
|
||||||
'last_update_time': metrics.last_update_time
|
|
||||||
})
|
|
||||||
|
|
||||||
# Check limits
|
# Check limits
|
||||||
if limits:
|
if limits:
|
||||||
@@ -363,6 +431,44 @@ class PluginResourceMonitor:
|
|||||||
summaries[plugin_id] = self.get_metrics_summary(plugin_id)
|
summaries[plugin_id] = self.get_metrics_summary(plugin_id)
|
||||||
return summaries
|
return summaries
|
||||||
|
|
||||||
|
def _persist_metrics(self, plugin_id: str, metrics: ResourceMetrics,
|
||||||
|
force: bool = False) -> None:
|
||||||
|
"""Write a plugin's metrics to the cache, at most once per interval.
|
||||||
|
|
||||||
|
Caller must hold ``self._lock``.
|
||||||
|
"""
|
||||||
|
# Monotonic, not wall clock: these devices have no RTC, so the clock
|
||||||
|
# jumps by however far off boot-time was the moment NTP first syncs.
|
||||||
|
# A forward jump would allow an early write, a backward one would
|
||||||
|
# stall the snapshot well past the interval.
|
||||||
|
#
|
||||||
|
# The sentinel for "never written" is None, not 0.0. monotonic() is
|
||||||
|
# time since boot on Linux, and systemd starts this service *at* boot,
|
||||||
|
# so `now - 0.0 < 30` was true for the first half-minute of every
|
||||||
|
# single run -- the throttle swallowed the very first snapshot, which
|
||||||
|
# is the one that matters most after a restart.
|
||||||
|
now = time.monotonic()
|
||||||
|
last_written = self._metrics_persisted_at.get(plugin_id)
|
||||||
|
if (not force and last_written is not None
|
||||||
|
and now - last_written < _METRICS_PERSIST_INTERVAL):
|
||||||
|
return
|
||||||
|
cache_key = self._get_metrics_key(plugin_id)
|
||||||
|
self.cache_manager.set(cache_key, {
|
||||||
|
'memory_mb': metrics.memory_mb,
|
||||||
|
'cpu_percent': metrics.cpu_percent,
|
||||||
|
'execution_time': metrics.execution_time,
|
||||||
|
'call_count': metrics.call_count,
|
||||||
|
'total_execution_time': metrics.total_execution_time,
|
||||||
|
'max_execution_time': metrics.max_execution_time,
|
||||||
|
'min_execution_time': (metrics.min_execution_time
|
||||||
|
if metrics.min_execution_time != float('inf')
|
||||||
|
else 0.0),
|
||||||
|
'last_update_time': metrics.last_update_time,
|
||||||
|
})
|
||||||
|
# Only after the write lands. Marking it first would mean a failed
|
||||||
|
# set() bought the next interval's silence without leaving a snapshot.
|
||||||
|
self._metrics_persisted_at[plugin_id] = now
|
||||||
|
|
||||||
def reset_metrics(self, plugin_id: str) -> None:
|
def reset_metrics(self, plugin_id: str) -> None:
|
||||||
"""Reset metrics for a plugin."""
|
"""Reset metrics for a plugin."""
|
||||||
with self._lock:
|
with self._lock:
|
||||||
@@ -370,4 +476,7 @@ class PluginResourceMonitor:
|
|||||||
self._metrics[plugin_id] = ResourceMetrics()
|
self._metrics[plugin_id] = ResourceMetrics()
|
||||||
cache_key = self._get_metrics_key(plugin_id)
|
cache_key = self._get_metrics_key(plugin_id)
|
||||||
self.cache_manager.delete(cache_key)
|
self.cache_manager.delete(cache_key)
|
||||||
|
# Let the next call persist immediately rather than leaving the
|
||||||
|
# deleted key absent for the rest of the interval.
|
||||||
|
self._metrics_persisted_at.pop(plugin_id, None)
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ Manages saved GitHub repository URLs for easy plugin discovery and installation.
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import List, Dict, Optional
|
from typing import List, Dict, Optional
|
||||||
|
|
||||||
@@ -43,20 +44,45 @@ class SavedRepositoriesManager:
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
def _save_repositories(self) -> bool:
|
def _save_repositories(self) -> bool:
|
||||||
"""Save repositories to file."""
|
"""Save repositories to file atomically.
|
||||||
|
|
||||||
|
Writes to a temp file in the same directory and os.replace()s it
|
||||||
|
over the target, so a failed write can never truncate or
|
||||||
|
half-overwrite an existing saved_repositories.json.
|
||||||
|
"""
|
||||||
|
tmp_path = self.config_path.with_suffix(self.config_path.suffix + '.tmp')
|
||||||
try:
|
try:
|
||||||
# Ensure directory exists
|
# Ensure directory exists
|
||||||
self.config_path.parent.mkdir(parents=True, exist_ok=True)
|
self.config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
with open(self.config_path, 'w') as f:
|
with open(tmp_path, 'w') as f:
|
||||||
json.dump(self.repositories, f, indent=2)
|
json.dump(self.repositories, f, indent=2)
|
||||||
|
f.flush()
|
||||||
|
os.fsync(f.fileno())
|
||||||
|
os.replace(tmp_path, self.config_path)
|
||||||
|
|
||||||
self.logger.info(f"Saved {len(self.repositories)} repositories to {self.config_path}")
|
self.logger.info(f"Saved {len(self.repositories)} repositories to {self.config_path}")
|
||||||
return True
|
return True
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(f"Error saving repositories: {e}")
|
self.logger.error(f"Error saving repositories: {e}")
|
||||||
|
try:
|
||||||
|
tmp_path.unlink(missing_ok=True)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _clean_url(repo_url: str) -> str:
|
||||||
|
"""Normalize a repo URL: strip whitespace, trailing slashes, and a
|
||||||
|
trailing ``.git`` suffix ONLY. (The old ``.replace('.git', '')``
|
||||||
|
was an unanchored substring replace that mangled URLs merely
|
||||||
|
containing ``.git``, e.g. ``https://github.com/user/my.github.io``.)
|
||||||
|
"""
|
||||||
|
repo_url = repo_url.strip().rstrip('/')
|
||||||
|
if repo_url.endswith('.git'):
|
||||||
|
repo_url = repo_url[:-4]
|
||||||
|
return repo_url
|
||||||
|
|
||||||
def get_all(self) -> List[Dict[str, str]]:
|
def get_all(self) -> List[Dict[str, str]]:
|
||||||
"""Get all saved repositories."""
|
"""Get all saved repositories."""
|
||||||
return self.repositories.copy()
|
return self.repositories.copy()
|
||||||
@@ -72,8 +98,7 @@ class SavedRepositoriesManager:
|
|||||||
Returns:
|
Returns:
|
||||||
True if added successfully
|
True if added successfully
|
||||||
"""
|
"""
|
||||||
# Clean URL
|
repo_url = self._clean_url(repo_url)
|
||||||
repo_url = repo_url.strip().rstrip('/').replace('.git', '')
|
|
||||||
|
|
||||||
# Check if already exists
|
# Check if already exists
|
||||||
for repo in self.repositories:
|
for repo in self.repositories:
|
||||||
@@ -96,7 +121,12 @@ class SavedRepositoriesManager:
|
|||||||
'type': 'registry' if 'plugins.json' in repo_url or 'ledmatrix-plugins' in repo_url.lower() else 'single'
|
'type': 'registry' if 'plugins.json' in repo_url or 'ledmatrix-plugins' in repo_url.lower() else 'single'
|
||||||
})
|
})
|
||||||
|
|
||||||
return self._save_repositories()
|
if not self._save_repositories():
|
||||||
|
# Keep memory consistent with disk: a failed save must not leave
|
||||||
|
# a phantom entry that only this process can see.
|
||||||
|
self.repositories.pop()
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
def remove(self, repo_url: str) -> bool:
|
def remove(self, repo_url: str) -> bool:
|
||||||
"""
|
"""
|
||||||
@@ -108,21 +138,25 @@ class SavedRepositoriesManager:
|
|||||||
Returns:
|
Returns:
|
||||||
True if removed successfully
|
True if removed successfully
|
||||||
"""
|
"""
|
||||||
# Clean URL
|
repo_url = self._clean_url(repo_url)
|
||||||
repo_url = repo_url.strip().rstrip('/').replace('.git', '')
|
|
||||||
|
|
||||||
original_count = len(self.repositories)
|
previous = self.repositories
|
||||||
self.repositories = [r for r in self.repositories if r.get('url') != repo_url]
|
remaining = [r for r in previous if r.get('url') != repo_url]
|
||||||
|
|
||||||
if len(self.repositories) < original_count:
|
if len(remaining) < len(previous):
|
||||||
return self._save_repositories()
|
self.repositories = remaining
|
||||||
|
if not self._save_repositories():
|
||||||
|
# Failed save: restore so memory matches disk.
|
||||||
|
self.repositories = previous
|
||||||
|
return False
|
||||||
|
return True
|
||||||
else:
|
else:
|
||||||
self.logger.warning(f"Repository not found: {repo_url}")
|
self.logger.warning(f"Repository not found: {repo_url}")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
def has(self, repo_url: str) -> bool:
|
def has(self, repo_url: str) -> bool:
|
||||||
"""Check if a repository is already saved."""
|
"""Check if a repository is already saved."""
|
||||||
repo_url = repo_url.strip().rstrip('/').replace('.git', '')
|
repo_url = self._clean_url(repo_url)
|
||||||
return any(r.get('url') == repo_url for r in self.repositories)
|
return any(r.get('url') == repo_url for r in self.repositories)
|
||||||
|
|
||||||
def get_registry_repositories(self) -> List[Dict[str, str]]:
|
def get_registry_repositories(self) -> List[Dict[str, str]]:
|
||||||
|
|||||||
@@ -26,7 +26,25 @@ class SchemaManager:
|
|||||||
- Cache invalidation on plugin changes
|
- Cache invalidation on plugin changes
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, plugins_dir: Optional[Path] = None, project_root: Optional[Path] = None, logger: Optional[logging.Logger] = None):
|
# Plugin config keys that mean "where this device is". A plugin declaring
|
||||||
|
# any of these in its schema gets the device-wide ``location`` block from
|
||||||
|
# config.json as the *default* for that field, instead of whatever city the
|
||||||
|
# plugin author happened to ship. A value the user set on the plugin itself
|
||||||
|
# always wins -- this only ever replaces the schema default, so an explicit
|
||||||
|
# per-plugin location is still honoured.
|
||||||
|
#
|
||||||
|
# Only these fully-namespaced keys are substituted. A bare ``state`` or
|
||||||
|
# ``city`` key is deliberately left alone: plugins use those for unrelated
|
||||||
|
# things (ledmatrix-elections' ``state`` is a two-letter code, not a place
|
||||||
|
# name), and silently rewriting them would break those plugins.
|
||||||
|
DEVICE_LOCATION_KEYS: Dict[str, str] = {
|
||||||
|
'location_city': 'city',
|
||||||
|
'location_state': 'state',
|
||||||
|
'location_country': 'country',
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(self, plugins_dir: Optional[Path] = None, project_root: Optional[Path] = None,
|
||||||
|
logger: Optional[logging.Logger] = None, config_manager: Optional[Any] = None):
|
||||||
"""
|
"""
|
||||||
Initialize the Schema Manager.
|
Initialize the Schema Manager.
|
||||||
|
|
||||||
@@ -34,10 +52,14 @@ class SchemaManager:
|
|||||||
plugins_dir: Base plugins directory path
|
plugins_dir: Base plugins directory path
|
||||||
project_root: Project root directory path
|
project_root: Project root directory path
|
||||||
logger: Optional logger instance
|
logger: Optional logger instance
|
||||||
|
config_manager: Optional config manager, used to resolve the
|
||||||
|
device-wide ``location`` that seeds plugin location defaults.
|
||||||
|
Omitting it simply leaves schema defaults untouched.
|
||||||
"""
|
"""
|
||||||
self.logger = logger or logging.getLogger(__name__)
|
self.logger = logger or logging.getLogger(__name__)
|
||||||
self.plugins_dir = plugins_dir
|
self.plugins_dir = plugins_dir
|
||||||
self.project_root = project_root or Path.cwd()
|
self.project_root = project_root or Path.cwd()
|
||||||
|
self.config_manager = config_manager
|
||||||
|
|
||||||
# Schema cache: plugin_id -> schema dict
|
# Schema cache: plugin_id -> schema dict
|
||||||
self._schema_cache: Dict[str, Dict[str, Any]] = {}
|
self._schema_cache: Dict[str, Dict[str, Any]] = {}
|
||||||
@@ -212,10 +234,70 @@ class SchemaManager:
|
|||||||
|
|
||||||
return defaults
|
return defaults
|
||||||
|
|
||||||
|
def get_device_location(self) -> Optional[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Return the device-wide ``location`` block from config.json, or None.
|
||||||
|
|
||||||
|
This is the City/State/Country the user sets once under General
|
||||||
|
settings. Returns None when there is no config manager wired, the
|
||||||
|
config can't be read, or no location has been configured.
|
||||||
|
"""
|
||||||
|
if self.config_manager is None:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
config = self.config_manager.load_config()
|
||||||
|
except Exception as e:
|
||||||
|
# A config that can't be read must never stop defaults being
|
||||||
|
# generated -- the plugin's own schema defaults still apply.
|
||||||
|
self.logger.debug(f"Could not read device location from config: {e}")
|
||||||
|
return None
|
||||||
|
if not isinstance(config, dict):
|
||||||
|
return None
|
||||||
|
location = config.get('location')
|
||||||
|
return location if isinstance(location, dict) else None
|
||||||
|
|
||||||
|
def apply_device_location(self, defaults: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Replace location-shaped schema defaults with the device's own location.
|
||||||
|
|
||||||
|
Without this, a plugin that ships ``"location_city": "Dallas"`` as its
|
||||||
|
schema default silently reports Dallas weather (and centres its radar
|
||||||
|
there) for every user who never opened that plugin's config form --
|
||||||
|
even though they set their real city under General settings. The
|
||||||
|
substituted value is still only a *default*: ``merge_with_defaults``
|
||||||
|
lets any per-plugin value the user saved win over it.
|
||||||
|
|
||||||
|
Mutates and returns ``defaults`` for convenience.
|
||||||
|
"""
|
||||||
|
if not defaults:
|
||||||
|
return defaults
|
||||||
|
if not any(key in defaults for key in self.DEVICE_LOCATION_KEYS):
|
||||||
|
return defaults
|
||||||
|
|
||||||
|
location = self.get_device_location()
|
||||||
|
if not location:
|
||||||
|
return defaults
|
||||||
|
|
||||||
|
for key, field in self.DEVICE_LOCATION_KEYS.items():
|
||||||
|
if key not in defaults:
|
||||||
|
continue
|
||||||
|
value = location.get(field)
|
||||||
|
# Only a non-empty string is a real answer; a blank or missing
|
||||||
|
# field means "not configured", which leaves the schema default.
|
||||||
|
if isinstance(value, str) and value.strip():
|
||||||
|
defaults[key] = value.strip()
|
||||||
|
|
||||||
|
return defaults
|
||||||
|
|
||||||
def generate_default_config(self, plugin_id: str, use_cache: bool = True) -> Dict[str, Any]:
|
def generate_default_config(self, plugin_id: str, use_cache: bool = True) -> Dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
Generate default configuration for a plugin from its schema.
|
Generate default configuration for a plugin from its schema.
|
||||||
|
|
||||||
|
Location fields (see ``DEVICE_LOCATION_KEYS``) default to the device's
|
||||||
|
configured location rather than the plugin author's. That substitution
|
||||||
|
is applied on the way out rather than being cached, so changing the
|
||||||
|
device location takes effect without invalidating the defaults cache.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
plugin_id: Plugin identifier
|
plugin_id: Plugin identifier
|
||||||
use_cache: If True, return cached defaults if available
|
use_cache: If True, return cached defaults if available
|
||||||
@@ -225,7 +307,7 @@ class SchemaManager:
|
|||||||
"""
|
"""
|
||||||
# Check cache first
|
# Check cache first
|
||||||
if use_cache and plugin_id in self._defaults_cache:
|
if use_cache and plugin_id in self._defaults_cache:
|
||||||
return self._defaults_cache[plugin_id].copy()
|
return self.apply_device_location(self._defaults_cache[plugin_id].copy())
|
||||||
|
|
||||||
schema = self.load_schema(plugin_id, use_cache=use_cache)
|
schema = self.load_schema(plugin_id, use_cache=use_cache)
|
||||||
if not schema:
|
if not schema:
|
||||||
@@ -249,10 +331,11 @@ class SchemaManager:
|
|||||||
if 'live_priority' not in defaults:
|
if 'live_priority' not in defaults:
|
||||||
defaults['live_priority'] = schema.get('properties', {}).get('live_priority', {}).get('default', False)
|
defaults['live_priority'] = schema.get('properties', {}).get('live_priority', {}).get('default', False)
|
||||||
|
|
||||||
# Cache the defaults
|
# Cache the defaults *before* the device location is layered on, so a
|
||||||
|
# later change to the device location is picked up by the next call.
|
||||||
self._defaults_cache[plugin_id] = defaults.copy()
|
self._defaults_cache[plugin_id] = defaults.copy()
|
||||||
|
|
||||||
return defaults
|
return self.apply_device_location(defaults)
|
||||||
|
|
||||||
def validate_config_against_schema(self, config: Dict[str, Any], schema: Dict[str, Any],
|
def validate_config_against_schema(self, config: Dict[str, Any], schema: Dict[str, Any],
|
||||||
plugin_id: Optional[str] = None) -> Tuple[bool, List[str]]:
|
plugin_id: Optional[str] = None) -> Tuple[bool, List[str]]:
|
||||||
|
|||||||
@@ -1467,8 +1467,11 @@ class PluginStoreManager:
|
|||||||
# already had. Allowing it costs them a plugin that raises
|
# already had. Allowing it costs them a plugin that raises
|
||||||
# ModuleNotFoundError at load and is reported only as one line
|
# ModuleNotFoundError at load and is reported only as one line
|
||||||
# in the journal. See docs/SPORTS_UNIFICATION.md (phase B4/B6).
|
# in the journal. See docs/SPORTS_UNIFICATION.md (phase B4/B6).
|
||||||
from src import __version__ as core_version
|
|
||||||
from src.plugin_system import compatibility
|
from src.plugin_system import compatibility
|
||||||
|
# On disk, not as imported: this process may predate the core
|
||||||
|
# update that made the plugin compatible. See
|
||||||
|
# compatibility.current_core_version.
|
||||||
|
core_version = compatibility.current_core_version()
|
||||||
|
|
||||||
compatible, reason = compatibility.check(manifest, core_version)
|
compatible, reason = compatibility.check(manifest, core_version)
|
||||||
if not compatible:
|
if not compatible:
|
||||||
@@ -1602,6 +1605,26 @@ class PluginStoreManager:
|
|||||||
'error': f'Manifest missing required fields: {", ".join(missing_fields)}'
|
'error': f'Manifest missing required fields: {", ".join(missing_fields)}'
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Refuse a plugin that needs a newer core than this one, exactly as
|
||||||
|
# _install_plugin_impl does after its download. Sideloading is an
|
||||||
|
# explicit act rather than an automatic store update, but the floor
|
||||||
|
# is not advice about intent -- it is a statement that the plugin
|
||||||
|
# cannot run here, and letting it through produces the same silent
|
||||||
|
# PluginState.ERROR at load. This was the last of the three routes
|
||||||
|
# in that skipped the check.
|
||||||
|
#
|
||||||
|
# Before the move, so the `finally` below removes the temp tree and
|
||||||
|
# nothing half-installed is left behind.
|
||||||
|
from src.plugin_system import compatibility
|
||||||
|
core_version = compatibility.current_core_version()
|
||||||
|
|
||||||
|
compatible, reason = compatibility.check(manifest, core_version)
|
||||||
|
if not compatible:
|
||||||
|
self.logger.error(
|
||||||
|
"Refusing to install %s from %s: %s",
|
||||||
|
plugin_id, repo_url, reason)
|
||||||
|
return {'success': False, 'error': reason}
|
||||||
|
|
||||||
# Validate version fields consistency (warnings only, not required)
|
# Validate version fields consistency (warnings only, not required)
|
||||||
validation_errors = self._validate_manifest_version_fields(manifest)
|
validation_errors = self._validate_manifest_version_fields(manifest)
|
||||||
if validation_errors:
|
if validation_errors:
|
||||||
@@ -2583,6 +2606,85 @@ class PluginStoreManager:
|
|||||||
self.logger.error(f"Error uninstalling plugin {plugin_id}: {e}")
|
self.logger.error(f"Error uninstalling plugin {plugin_id}: {e}")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def _gate_pulled_commit(self, plugin_id: str, plugin_path: Path,
|
||||||
|
previous_sha: Optional[str]) -> bool:
|
||||||
|
"""Apply the compatibility gate to a commit that arrived via git pull.
|
||||||
|
|
||||||
|
Every other route into an installed plugin goes through
|
||||||
|
``install_plugin``, which gates in ``_install_plugin_impl``. This one
|
||||||
|
did not: a ``git pull`` could deliver a manifest flooring above this
|
||||||
|
core and nothing would notice until the plugin failed to load, which
|
||||||
|
surfaces as one line in the journal and a scoreboard that silently
|
||||||
|
stopped appearing.
|
||||||
|
|
||||||
|
Checked after the pull rather than before it, for the same reason
|
||||||
|
``_install_plugin_impl`` checks after the download: the registry
|
||||||
|
carries no compatibility field, so the incoming floor is only knowable
|
||||||
|
once the new commit is on disk.
|
||||||
|
|
||||||
|
Undone with ``git reset --hard`` rather than by removing the directory.
|
||||||
|
This is a live checkout, the previous commit is still in the object
|
||||||
|
store, and the reset leaves the user on the exact version they were
|
||||||
|
already running -- the same promise ``_reinstall_with_rollback`` makes,
|
||||||
|
reached by the means this path actually has. It is also the gentler
|
||||||
|
option: no window in which the plugin directory does not exist, and no
|
||||||
|
``.standalone-backup-`` debris if the process dies mid-way.
|
||||||
|
|
||||||
|
A manifest that cannot be read is not evidence of incompatibility, so
|
||||||
|
it allows. ``compatibility.check`` refuses only on evidence for the
|
||||||
|
same reason: a wrong refusal breaks a working install, while a wrong
|
||||||
|
allowance degrades to exactly the behaviour this path had before the
|
||||||
|
gate existed.
|
||||||
|
"""
|
||||||
|
manifest_path = plugin_path / "manifest.json"
|
||||||
|
try:
|
||||||
|
with open(manifest_path, 'r', encoding='utf-8') as mf:
|
||||||
|
manifest = json.load(mf)
|
||||||
|
except (OSError, ValueError) as e:
|
||||||
|
self.logger.warning(
|
||||||
|
"Could not read %s after updating %s (%s); allowing the "
|
||||||
|
"update, as an unreadable manifest declares no floor",
|
||||||
|
manifest_path, plugin_id, e)
|
||||||
|
return True
|
||||||
|
|
||||||
|
from src.plugin_system import compatibility
|
||||||
|
core_version = compatibility.current_core_version()
|
||||||
|
|
||||||
|
compatible, reason = compatibility.check(manifest, core_version)
|
||||||
|
if compatible:
|
||||||
|
return True
|
||||||
|
|
||||||
|
self.logger.error("Refusing the update to %s: %s", plugin_id, reason)
|
||||||
|
|
||||||
|
if not previous_sha:
|
||||||
|
self.logger.error(
|
||||||
|
"Cannot roll %s back: the commit it was on before the pull is "
|
||||||
|
"unknown. It is now on a version this core cannot run — "
|
||||||
|
"reinstall it from the plugin store.", plugin_id)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Safe by construction: update_plugin returns before pulling unless the
|
||||||
|
# tree was clean or successfully stashed, so there are no uncommitted
|
||||||
|
# tracked edits for --hard to discard. The stash is not popped on the
|
||||||
|
# success path either, so the reset leaves the working tree exactly
|
||||||
|
# where a successful pull would have. Say "commit", not "changes".
|
||||||
|
reset = subprocess.run(
|
||||||
|
['git', '-C', str(plugin_path), 'reset', '--hard', previous_sha],
|
||||||
|
capture_output=True, text=True, timeout=60, check=False)
|
||||||
|
if reset.returncode != 0:
|
||||||
|
self.logger.error(
|
||||||
|
"CRITICAL: could not roll %s back to commit %s: %s. It is left "
|
||||||
|
"on a version this core cannot run; "
|
||||||
|
"`git -C %s reset --hard %s` restores it.",
|
||||||
|
plugin_id, previous_sha[:7],
|
||||||
|
(reset.stderr or reset.stdout or '').strip(),
|
||||||
|
plugin_path, previous_sha)
|
||||||
|
else:
|
||||||
|
self.logger.info(
|
||||||
|
"Rolled %s back to commit %s; it stays on the version it was "
|
||||||
|
"already running.", plugin_id, previous_sha[:7])
|
||||||
|
return False
|
||||||
|
|
||||||
def _reinstall_with_rollback(self, plugin_id: str, plugin_path: Path) -> bool:
|
def _reinstall_with_rollback(self, plugin_id: str, plugin_path: Path) -> bool:
|
||||||
"""Replace an installed plugin with a fresh install, atomically.
|
"""Replace an installed plugin with a fresh install, atomically.
|
||||||
|
|
||||||
@@ -2860,6 +2962,8 @@ class PluginStoreManager:
|
|||||||
status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
|
status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
|
||||||
|
|
||||||
stash_info = ""
|
stash_info = ""
|
||||||
|
# Whether the pull can be undone without destroying work.
|
||||||
|
tree_is_recoverable = not has_changes
|
||||||
if has_changes:
|
if has_changes:
|
||||||
self.logger.info(f"Stashing local changes in {plugin_id} before update")
|
self.logger.info(f"Stashing local changes in {plugin_id} before update")
|
||||||
try:
|
try:
|
||||||
@@ -2873,12 +2977,37 @@ class PluginStoreManager:
|
|||||||
)
|
)
|
||||||
if stash_result.returncode == 0:
|
if stash_result.returncode == 0:
|
||||||
stash_info = " (local changes were stashed)"
|
stash_info = " (local changes were stashed)"
|
||||||
|
tree_is_recoverable = True
|
||||||
self.logger.info(f"Stashed local changes (including untracked files) for {plugin_id}")
|
self.logger.info(f"Stashed local changes (including untracked files) for {plugin_id}")
|
||||||
else:
|
else:
|
||||||
self.logger.warning(f"Failed to stash local changes for {plugin_id}: {stash_result.stderr}")
|
self.logger.warning(f"Failed to stash local changes for {plugin_id}: {stash_result.stderr}")
|
||||||
except subprocess.TimeoutExpired:
|
except subprocess.TimeoutExpired:
|
||||||
self.logger.warning(f"Stash operation timed out for {plugin_id}, proceeding with pull")
|
self.logger.warning(f"Stash operation timed out for {plugin_id}, proceeding with pull")
|
||||||
|
|
||||||
|
# Do not pull what cannot be un-pulled.
|
||||||
|
#
|
||||||
|
# The compatibility gate below can refuse the commit this
|
||||||
|
# pull brings down, and its only way back is `git reset
|
||||||
|
# --hard`, which discards uncommitted tracked edits. Those
|
||||||
|
# edits are exactly what the stash above exists to protect,
|
||||||
|
# so a stash that failed or timed out leaves the rollback
|
||||||
|
# unable to run without destroying them.
|
||||||
|
#
|
||||||
|
# A pull does not necessarily refuse on a dirty tree -- git
|
||||||
|
# merges happily as long as the incoming commit touches
|
||||||
|
# different files -- so without this the update would
|
||||||
|
# succeed, the gate would refuse, and the reset would take
|
||||||
|
# the user's work with it. Refusing here costs an update in
|
||||||
|
# a case that already went wrong; the alternative costs
|
||||||
|
# data.
|
||||||
|
if not tree_is_recoverable:
|
||||||
|
self.logger.error(
|
||||||
|
"Refusing to update %s: it has local changes that could "
|
||||||
|
"not be stashed, and an incompatible update could then "
|
||||||
|
"only be rolled back by discarding them. Commit or stash "
|
||||||
|
"them by hand, then update.", plugin_id)
|
||||||
|
return False
|
||||||
|
|
||||||
# Pull from the determined remote branch
|
# Pull from the determined remote branch
|
||||||
self.logger.info(f"Pulling from origin/{remote_pull_branch} for {plugin_id}...")
|
self.logger.info(f"Pulling from origin/{remote_pull_branch} for {plugin_id}...")
|
||||||
pull_result = subprocess.run(
|
pull_result = subprocess.run(
|
||||||
@@ -2901,6 +3030,14 @@ class PluginStoreManager:
|
|||||||
elif updated_sha:
|
elif updated_sha:
|
||||||
self.logger.info(f"Plugin {plugin_id} updated to commit {updated_sha[:7]}{stash_info}")
|
self.logger.info(f"Plugin {plugin_id} updated to commit {updated_sha[:7]}{stash_info}")
|
||||||
|
|
||||||
|
# The install gate, at the only point on this path where
|
||||||
|
# it can be answered. Every other route in goes through
|
||||||
|
# install_plugin, which gates in _install_plugin_impl; this
|
||||||
|
# one did not, so a pull could deliver a manifest flooring
|
||||||
|
# above this core and nothing would notice.
|
||||||
|
if not self._gate_pulled_commit(plugin_id, plugin_path, local_sha):
|
||||||
|
return False
|
||||||
|
|
||||||
self._install_dependencies(plugin_path)
|
self._install_dependencies(plugin_path)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|||||||
@@ -73,6 +73,11 @@ class RenderResult:
|
|||||||
golden_ok: Optional[bool] = None
|
golden_ok: Optional[bool] = None
|
||||||
golden_diff_pixels: int = 0
|
golden_diff_pixels: int = 0
|
||||||
golden_max_delta: int = 0
|
golden_max_delta: int = 0
|
||||||
|
# what display() handed back; the controller skips a mode only on False
|
||||||
|
display_returned: Any = None
|
||||||
|
# empty-frame check: rendered nothing while not reporting "no content"
|
||||||
|
empty_claimed: Optional[bool] = None # True when that happened
|
||||||
|
empty_ok: Optional[bool] = None # False only in strict mode
|
||||||
# fill / scale-up check (populated only for sizes >= 2x the design size)
|
# fill / scale-up check (populated only for sizes >= 2x the design size)
|
||||||
fill_checked: bool = False
|
fill_checked: bool = False
|
||||||
fill_ok: Optional[bool] = None # False only in strict mode
|
fill_ok: Optional[bool] = None # False only in strict mode
|
||||||
@@ -92,6 +97,8 @@ class RenderResult:
|
|||||||
return False
|
return False
|
||||||
if self.fill_ok is False:
|
if self.fill_ok is False:
|
||||||
return False
|
return False
|
||||||
|
if self.empty_ok is False:
|
||||||
|
return False
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
@@ -132,21 +139,25 @@ def _instantiate(plugin_id: str, manifest: Dict[str, Any], plugin_dir: Path,
|
|||||||
return plugin_instance
|
return plugin_instance
|
||||||
|
|
||||||
|
|
||||||
def _render_mode(plugin_instance: Any, mode: str) -> None:
|
def _render_mode(plugin_instance: Any, mode: str) -> Any:
|
||||||
"""Render a specific screen. Prefer an explicit display_mode kwarg; otherwise
|
"""Render a specific screen. Prefer an explicit display_mode kwarg; otherwise
|
||||||
drive the plugin's internal mode state machine (first display() call renders
|
drive the plugin's internal mode state machine (first display() call renders
|
||||||
modes[current_mode_index] when current_display_mode is None)."""
|
modes[current_mode_index] when current_display_mode is None).
|
||||||
|
|
||||||
|
Returns whatever display() returned. The display controller skips a mode
|
||||||
|
whose display() returns False, so that value decides whether an empty mode
|
||||||
|
is rotated past or sat on -- which makes it worth reporting rather than
|
||||||
|
discarding."""
|
||||||
sig = inspect.signature(plugin_instance.display)
|
sig = inspect.signature(plugin_instance.display)
|
||||||
if "display_mode" in sig.parameters:
|
if "display_mode" in sig.parameters:
|
||||||
plugin_instance.display(force_clear=True, display_mode=mode)
|
return plugin_instance.display(force_clear=True, display_mode=mode)
|
||||||
return
|
|
||||||
|
|
||||||
modes = getattr(plugin_instance, "modes", None)
|
modes = getattr(plugin_instance, "modes", None)
|
||||||
if modes and mode in modes:
|
if modes and mode in modes:
|
||||||
plugin_instance.current_mode_index = list(modes).index(mode)
|
plugin_instance.current_mode_index = list(modes).index(mode)
|
||||||
if hasattr(plugin_instance, "current_display_mode"):
|
if hasattr(plugin_instance, "current_display_mode"):
|
||||||
plugin_instance.current_display_mode = None
|
plugin_instance.current_display_mode = None
|
||||||
plugin_instance.display(force_clear=False)
|
return plugin_instance.display(force_clear=False)
|
||||||
|
|
||||||
|
|
||||||
def _freeze(freeze_time: Optional[str]):
|
def _freeze(freeze_time: Optional[str]):
|
||||||
@@ -234,7 +245,7 @@ def _render_size(plugin_id, manifest, plugin_dir, config, mock_data,
|
|||||||
logger.warning("update() raised a non-connectivity error for %s [%s]: %s",
|
logger.warning("update() raised a non-connectivity error for %s [%s]: %s",
|
||||||
plugin_id, mode, e)
|
plugin_id, mode, e)
|
||||||
if result.error is None:
|
if result.error is None:
|
||||||
_render_mode(inst, mode)
|
result.display_returned = _render_mode(inst, mode)
|
||||||
result.image = dm.get_image()
|
result.image = dm.get_image()
|
||||||
result.overflow = dm.check_overflow()
|
result.overflow = dm.check_overflow()
|
||||||
except Exception as e: # noqa: BLE001 — a display crash is a real failure
|
except Exception as e: # noqa: BLE001 — a display crash is a real failure
|
||||||
@@ -341,6 +352,44 @@ def fill_metrics(image: Image.Image) -> Tuple[float, float, float]:
|
|||||||
return (extent_x, extent_y, ink)
|
return (extent_x, extent_y, ink)
|
||||||
|
|
||||||
|
|
||||||
|
def check_empty_claimed(results: List[RenderResult],
|
||||||
|
strict: bool = False) -> List[RenderResult]:
|
||||||
|
"""Flag a mode that rendered nothing without reporting "no content".
|
||||||
|
|
||||||
|
The display controller skips a mode whose ``display()`` returns False, and
|
||||||
|
treats anything else -- including None -- as "content was shown". A mode
|
||||||
|
that draws nothing and does not return False therefore holds whatever is on
|
||||||
|
the panel for its whole display duration. Since a mode switch clears first,
|
||||||
|
that is a blank screen. Two sports plugins shipped exactly this: their
|
||||||
|
``display()`` returned None on every path, so an out-of-season league sat
|
||||||
|
blank for its full duration rather than being rotated past.
|
||||||
|
|
||||||
|
Warn-only by default, because a blank frame is not automatically wrong: a
|
||||||
|
scroll mode whose first frame is its blank scroll-in buffer renders empty
|
||||||
|
and is behaving correctly. ``strict=True`` sets ``empty_claimed`` such that
|
||||||
|
``RenderResult.ok`` fails -- opt in per plugin via harness.json
|
||||||
|
``{"empty_check": "strict"}`` once its modes are known to draw on the
|
||||||
|
fixture data.
|
||||||
|
|
||||||
|
Note this can only catch what the fixtures actually render. A plugin whose
|
||||||
|
harness fixture seeds content never exercises its empty path here; the
|
||||||
|
source-level gate in the plugins repo covers that case.
|
||||||
|
"""
|
||||||
|
for r in results:
|
||||||
|
if r.image is None or r.error is not None:
|
||||||
|
continue
|
||||||
|
# An explicit False is the plugin correctly saying "nothing to show".
|
||||||
|
if r.display_returned is False:
|
||||||
|
continue
|
||||||
|
if r.image.convert("L").point(
|
||||||
|
lambda p: 255 if p > _LIT_THRESHOLD else 0).getbbox() is not None:
|
||||||
|
continue
|
||||||
|
r.empty_claimed = True
|
||||||
|
if strict:
|
||||||
|
r.empty_ok = False
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
def check_scale_up(results: List[RenderResult],
|
def check_scale_up(results: List[RenderResult],
|
||||||
design_size: Tuple[int, int] = (128, 32),
|
design_size: Tuple[int, int] = (128, 32),
|
||||||
min_extent: float = _MIN_FILL_EXTENT,
|
min_extent: float = _MIN_FILL_EXTENT,
|
||||||
|
|||||||
@@ -70,6 +70,7 @@ class VisualTestDisplayManager:
|
|||||||
# Canvas
|
# Canvas
|
||||||
self.image = Image.new('RGB', (width, height), (0, 0, 0))
|
self.image = Image.new('RGB', (width, height), (0, 0, 0))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows.
|
||||||
|
|
||||||
# Matrix proxy (plugins access display_manager.matrix.width/height)
|
# Matrix proxy (plugins access display_manager.matrix.width/height)
|
||||||
self.matrix = _MatrixProxy(width, height)
|
self.matrix = _MatrixProxy(width, height)
|
||||||
@@ -184,6 +185,7 @@ class VisualTestDisplayManager:
|
|||||||
self.clear_called = True
|
self.clear_called = True
|
||||||
self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0))
|
self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows.
|
||||||
|
|
||||||
def update_display(self):
|
def update_display(self):
|
||||||
"""No-op for hardware; marks that display was updated."""
|
"""No-op for hardware; marks that display was updated."""
|
||||||
@@ -211,6 +213,7 @@ class VisualTestDisplayManager:
|
|||||||
self.matrix = _MatrixProxy(target_w, target_h)
|
self.matrix = _MatrixProxy(target_w, target_h)
|
||||||
self.image = Image.new('RGB', (target_w, target_h), (0, 0, 0))
|
self.image = Image.new('RGB', (target_w, target_h), (0, 0, 0))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows.
|
||||||
yield
|
yield
|
||||||
finally:
|
finally:
|
||||||
self._width, self._height = prev_w, prev_h
|
self._width, self._height = prev_w, prev_h
|
||||||
@@ -569,6 +572,7 @@ class VisualTestDisplayManager:
|
|||||||
self.draw_calls = []
|
self.draw_calls = []
|
||||||
self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0))
|
self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows.
|
||||||
self._scrolling_state = {
|
self._scrolling_state = {
|
||||||
'is_scrolling': False,
|
'is_scrolling': False,
|
||||||
'last_scroll_activity': 0,
|
'last_scroll_activity': 0,
|
||||||
@@ -582,3 +586,4 @@ class VisualTestDisplayManager:
|
|||||||
"""Clean up resources."""
|
"""Clean up resources."""
|
||||||
self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0))
|
self.image = Image.new('RGB', (self._width, self._height), (0, 0, 0))
|
||||||
self.draw = ImageDraw.Draw(self.image)
|
self.draw = ImageDraw.Draw(self.image)
|
||||||
|
self.draw.fontmode = "1" # Match production: 1-bit text, so goldens show what the panel shows.
|
||||||
|
|||||||
+102
-1
@@ -15,16 +15,23 @@ from src.logging_config import get_logger
|
|||||||
class StartupValidator:
|
class StartupValidator:
|
||||||
"""Validates system state on startup."""
|
"""Validates system state on startup."""
|
||||||
|
|
||||||
def __init__(self, config_manager: Any, plugin_manager: Optional[Any] = None) -> None:
|
def __init__(self, config_manager: Any, plugin_manager: Optional[Any] = None,
|
||||||
|
cache_manager: Optional[Any] = None) -> None:
|
||||||
"""
|
"""
|
||||||
Initialize the startup validator.
|
Initialize the startup validator.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
config_manager: ConfigManager instance
|
config_manager: ConfigManager instance
|
||||||
plugin_manager: Optional PluginManager instance
|
plugin_manager: Optional PluginManager instance
|
||||||
|
cache_manager: The CacheManager the application will actually use.
|
||||||
|
Pass it. Without one this validator builds its own just to read
|
||||||
|
a directory path, which reports on a cache the app does not
|
||||||
|
use and leaves behind a cleanup thread that nothing stops --
|
||||||
|
validation runs twice per startup, so that was two of them.
|
||||||
"""
|
"""
|
||||||
self.config_manager = config_manager
|
self.config_manager = config_manager
|
||||||
self.plugin_manager = plugin_manager
|
self.plugin_manager = plugin_manager
|
||||||
|
self.cache_manager = cache_manager
|
||||||
self.logger = get_logger(__name__)
|
self.logger = get_logger(__name__)
|
||||||
self.errors: List[str] = []
|
self.errors: List[str] = []
|
||||||
self.warnings: List[str] = []
|
self.warnings: List[str] = []
|
||||||
@@ -38,6 +45,11 @@ class StartupValidator:
|
|||||||
"""
|
"""
|
||||||
self.logger.info("Starting startup validation...")
|
self.logger.info("Starting startup validation...")
|
||||||
|
|
||||||
|
# Fresh lists each run — without this, calling validate_all() twice
|
||||||
|
# duplicated every message.
|
||||||
|
self.errors = []
|
||||||
|
self.warnings = []
|
||||||
|
|
||||||
# Validate configuration
|
# Validate configuration
|
||||||
self._validate_config()
|
self._validate_config()
|
||||||
|
|
||||||
@@ -51,6 +63,9 @@ class StartupValidator:
|
|||||||
if self.plugin_manager:
|
if self.plugin_manager:
|
||||||
self._validate_plugins()
|
self._validate_plugins()
|
||||||
|
|
||||||
|
# Warn when the running systemd unit no longer matches the repo's
|
||||||
|
self._validate_systemd_units()
|
||||||
|
|
||||||
is_valid = len(self.errors) == 0
|
is_valid = len(self.errors) == 0
|
||||||
|
|
||||||
if is_valid:
|
if is_valid:
|
||||||
@@ -62,6 +77,80 @@ class StartupValidator:
|
|||||||
|
|
||||||
return (is_valid, self.errors.copy(), self.warnings.copy())
|
return (is_valid, self.errors.copy(), self.warnings.copy())
|
||||||
|
|
||||||
|
#: Units this project installs, and where each is installed to.
|
||||||
|
_UNITS = (
|
||||||
|
("systemd/ledmatrix.service", "/etc/systemd/system/ledmatrix.service"),
|
||||||
|
("systemd/ledmatrix-web.service", "/etc/systemd/system/ledmatrix-web.service"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _validate_systemd_units(self) -> None:
|
||||||
|
"""Warn when an installed unit has drifted from the repo's template.
|
||||||
|
|
||||||
|
Nothing re-applies these after the first install. `git pull` -- which is
|
||||||
|
what the web UI's update button runs -- brings a new template into the
|
||||||
|
checkout, but nothing copies it to /etc/systemd/system and nothing runs
|
||||||
|
`systemctl daemon-reload`, so the unit that actually runs is whatever
|
||||||
|
first_time_install.sh wrote on day one.
|
||||||
|
|
||||||
|
That makes every hardening added to a unit inert on existing installs.
|
||||||
|
Measured on one rig: the installed unit was thirteen days older than the
|
||||||
|
repo's and differed in content, so a MemoryMax the repo had specified
|
||||||
|
was not being enforced at all -- `systemctl show` reported
|
||||||
|
MemoryMax=infinity.
|
||||||
|
|
||||||
|
A warning rather than an error, and certainly not a silent rewrite:
|
||||||
|
editing files under /etc and restarting services is the installer's job,
|
||||||
|
not something a display process should do to a machine while it boots.
|
||||||
|
The remedy is to re-run scripts/install/install_service.sh.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
project_root = Path(__file__).resolve().parent.parent
|
||||||
|
for template_rel, installed_path in self._UNITS:
|
||||||
|
template = project_root / template_rel
|
||||||
|
installed = Path(installed_path)
|
||||||
|
if not template.is_file() or not installed.is_file():
|
||||||
|
continue
|
||||||
|
|
||||||
|
# The template carries placeholders the installer substitutes,
|
||||||
|
# so compare the substituted form rather than the raw file.
|
||||||
|
expected = template.read_text(encoding="utf-8")
|
||||||
|
expected = expected.replace("__PROJECT_ROOT_DIR__", str(project_root))
|
||||||
|
expected = expected.replace("__USER__", "root")
|
||||||
|
|
||||||
|
try:
|
||||||
|
actual = installed.read_text(encoding="utf-8")
|
||||||
|
except PermissionError:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if self._unit_body(expected) != self._unit_body(actual):
|
||||||
|
self.warnings.append(
|
||||||
|
f"{installed.name} differs from {template_rel}; the "
|
||||||
|
"installed unit is not refreshed by an update, so "
|
||||||
|
"settings added to the template are not in effect. "
|
||||||
|
"Re-run scripts/install/install_service.sh to apply them."
|
||||||
|
)
|
||||||
|
except OSError as e:
|
||||||
|
self.logger.debug("Could not compare systemd units: %s", e)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _unit_body(text: str) -> str:
|
||||||
|
"""A unit's meaningful lines, in order: no comments, no blanks.
|
||||||
|
|
||||||
|
Order is preserved deliberately. This used to sort, which made the
|
||||||
|
comparison insensitive to two changes that matter in a systemd unit:
|
||||||
|
repeated directives such as ExecStartPre= and ExecStartPost= run in
|
||||||
|
the order they appear, and a directive that moves between [Unit],
|
||||||
|
[Service] and [Install] means something different -- or nothing --
|
||||||
|
where it lands. A drift check that normalises those away reports no
|
||||||
|
drift for a unit that has genuinely changed.
|
||||||
|
"""
|
||||||
|
lines = []
|
||||||
|
for line in text.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if line and not line.startswith("#"):
|
||||||
|
lines.append(line)
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
def _validate_config(self) -> None:
|
def _validate_config(self) -> None:
|
||||||
"""Validate configuration files."""
|
"""Validate configuration files."""
|
||||||
try:
|
try:
|
||||||
@@ -86,8 +175,20 @@ class StartupValidator:
|
|||||||
def _validate_cache_directory(self) -> None:
|
def _validate_cache_directory(self) -> None:
|
||||||
"""Validate cache directory permissions."""
|
"""Validate cache directory permissions."""
|
||||||
try:
|
try:
|
||||||
|
cache_manager = self.cache_manager
|
||||||
|
if cache_manager is None:
|
||||||
|
# No caller supplied one (older embedders, direct use in a
|
||||||
|
# script). Build one, but do not leave its cleanup thread
|
||||||
|
# running behind us -- this instance is discarded on the next
|
||||||
|
# line but the thread is a closure over it, so it would never
|
||||||
|
# be collected.
|
||||||
from src.cache_manager import CacheManager
|
from src.cache_manager import CacheManager
|
||||||
cache_manager = CacheManager()
|
cache_manager = CacheManager()
|
||||||
|
try:
|
||||||
|
cache_dir = cache_manager.get_cache_dir()
|
||||||
|
finally:
|
||||||
|
cache_manager.stop_cleanup_thread()
|
||||||
|
else:
|
||||||
cache_dir = cache_manager.get_cache_dir()
|
cache_dir = cache_manager.get_cache_dir()
|
||||||
|
|
||||||
if not cache_dir:
|
if not cache_dir:
|
||||||
|
|||||||
@@ -104,15 +104,53 @@ class VegasModeConfig:
|
|||||||
overflow_mode: str = "rotate"
|
overflow_mode: str = "rotate"
|
||||||
|
|
||||||
# Cap on one plugin's share of a cycle, as a multiple of display width.
|
# Cap on one plugin's share of a cycle, as a multiple of display width.
|
||||||
# A single ticker returning 7,000px would otherwise hold the panel for over
|
# 0 (the default) disables the cap, so every plugin contributes all of its
|
||||||
# two minutes. Overflow is deferred to later cycles rather than discarded.
|
# content and is always entered at its beginning.
|
||||||
# 0 disables the cap.
|
#
|
||||||
max_plugin_width_ratio: float = 3.0
|
# Capping was the default until it proved to cost more than it bought.
|
||||||
|
# Measured over a 17-plugin fleet on a 512px panel, only four plugins were
|
||||||
|
# ever wide enough to hit a 3.0 cap; for those four it produced two visible
|
||||||
|
# faults. Content resumed mid-item on each appearance (a news ticker entered
|
||||||
|
# at column 6027 of its own strip), and the final window of a rotation was
|
||||||
|
# whatever happened to be left — 348px of a 1840px stocks ticker, seven
|
||||||
|
# seconds of panel time. Both read as the display being broken rather than
|
||||||
|
# as deferral working.
|
||||||
|
#
|
||||||
|
# A wide plugin does hold the panel for a long time uncapped: set the cap
|
||||||
|
# per plugin with vegas_max_width_screens where that matters, rather than
|
||||||
|
# globally where it mostly hurts plugins that were never the problem.
|
||||||
|
max_plugin_width_ratio: float = 0.0
|
||||||
|
|
||||||
# Plugin management
|
# Plugin management
|
||||||
plugin_order: List[str] = field(default_factory=list)
|
plugin_order: List[str] = field(default_factory=list)
|
||||||
excluded_plugins: Set[str] = field(default_factory=set)
|
excluded_plugins: Set[str] = field(default_factory=set)
|
||||||
|
|
||||||
|
# --- Live content in the ticker -------------------------------------
|
||||||
|
#
|
||||||
|
# By default a live game preempts Vegas entirely: the display controller
|
||||||
|
# refuses to run the ticker while any plugin reports live priority, and you
|
||||||
|
# get the full-screen scoreboard instead. Set live_in_ticker to keep the
|
||||||
|
# marquee running and let live content take extra turns within it.
|
||||||
|
#
|
||||||
|
# The rotation is otherwise a strict round robin -- every plugin appears
|
||||||
|
# exactly once per cycle -- so with a dozen plugins enabled a live score
|
||||||
|
# comes round once a lap and can be minutes old on screen. Weighting lets a
|
||||||
|
# plugin claim several slots per cycle instead.
|
||||||
|
#
|
||||||
|
# Weights are per plugin, not per game: a scoreboard showing four live
|
||||||
|
# games still occupies one slot at a time, and rotates its own games within
|
||||||
|
# that slot using its own favorite_live_boost.
|
||||||
|
live_in_ticker: bool = False
|
||||||
|
|
||||||
|
# Slots per cycle for a plugin reporting live content. 1 disables the boost
|
||||||
|
# and restores the plain round robin.
|
||||||
|
live_weight: int = 3
|
||||||
|
|
||||||
|
# Slots per cycle for a plugin whose live content involves a favorite team.
|
||||||
|
# Only plugins implementing get_vegas_priority_weight() can claim this --
|
||||||
|
# the core cannot tell whose game is on, so the plugin reports it.
|
||||||
|
favorite_live_weight: int = 5
|
||||||
|
|
||||||
# Performance settings
|
# Performance settings
|
||||||
target_fps: int = 125 # Target frame rate
|
target_fps: int = 125 # Target frame rate
|
||||||
buffer_ahead: int = 2 # Number of plugins to buffer ahead
|
buffer_ahead: int = 2 # Number of plugins to buffer ahead
|
||||||
@@ -159,10 +197,16 @@ class VegasModeConfig:
|
|||||||
lead_in_width=int(vegas_config.get('lead_in_width', 0)),
|
lead_in_width=int(vegas_config.get('lead_in_width', 0)),
|
||||||
plugins_per_cycle=int(vegas_config.get('plugins_per_cycle', 6)),
|
plugins_per_cycle=int(vegas_config.get('plugins_per_cycle', 6)),
|
||||||
max_plugin_width_ratio=float(
|
max_plugin_width_ratio=float(
|
||||||
vegas_config.get('max_plugin_width_ratio', 3.0)),
|
vegas_config.get('max_plugin_width_ratio', 0.0)),
|
||||||
overflow_mode=str(vegas_config.get('overflow_mode', 'rotate')),
|
overflow_mode=str(vegas_config.get('overflow_mode', 'rotate')),
|
||||||
plugin_order=list(vegas_config.get('plugin_order', [])),
|
plugin_order=list(vegas_config.get('plugin_order', [])),
|
||||||
excluded_plugins=set(vegas_config.get('excluded_plugins', [])),
|
excluded_plugins=set(vegas_config.get('excluded_plugins', [])),
|
||||||
|
live_in_ticker=bool(vegas_config.get('live_in_ticker', False)),
|
||||||
|
# Clamped: a weight below 1 would drop the plugin from the rotation
|
||||||
|
# entirely, and a very large one starves everything else.
|
||||||
|
live_weight=max(1, min(10, int(vegas_config.get('live_weight', 3)))),
|
||||||
|
favorite_live_weight=max(
|
||||||
|
1, min(10, int(vegas_config.get('favorite_live_weight', 5)))),
|
||||||
target_fps=int(vegas_config.get('target_fps', 125)),
|
target_fps=int(vegas_config.get('target_fps', 125)),
|
||||||
buffer_ahead=int(vegas_config.get('buffer_ahead', 2)),
|
buffer_ahead=int(vegas_config.get('buffer_ahead', 2)),
|
||||||
frame_based_scrolling=vegas_config.get('frame_based_scrolling', True),
|
frame_based_scrolling=vegas_config.get('frame_based_scrolling', True),
|
||||||
@@ -192,6 +236,9 @@ class VegasModeConfig:
|
|||||||
'lead_in_width': self.lead_in_width,
|
'lead_in_width': self.lead_in_width,
|
||||||
'plugins_per_cycle': self.plugins_per_cycle,
|
'plugins_per_cycle': self.plugins_per_cycle,
|
||||||
'max_plugin_width_ratio': self.max_plugin_width_ratio,
|
'max_plugin_width_ratio': self.max_plugin_width_ratio,
|
||||||
|
'live_in_ticker': self.live_in_ticker,
|
||||||
|
'live_weight': self.live_weight,
|
||||||
|
'favorite_live_weight': self.favorite_live_weight,
|
||||||
'overflow_mode': self.overflow_mode,
|
'overflow_mode': self.overflow_mode,
|
||||||
'plugin_order': self.plugin_order,
|
'plugin_order': self.plugin_order,
|
||||||
'excluded_plugins': list(self.excluded_plugins),
|
'excluded_plugins': list(self.excluded_plugins),
|
||||||
@@ -359,6 +406,15 @@ class VegasModeConfig:
|
|||||||
|
|
||||||
if 'enabled' in vegas_config:
|
if 'enabled' in vegas_config:
|
||||||
self.enabled = vegas_config['enabled']
|
self.enabled = vegas_config['enabled']
|
||||||
|
if 'live_in_ticker' in vegas_config:
|
||||||
|
self.live_in_ticker = bool(vegas_config['live_in_ticker'])
|
||||||
|
# Clamped exactly as from_config does: a weight below 1 would drop the
|
||||||
|
# plugin from the rotation, and a huge one starves everything else.
|
||||||
|
if 'live_weight' in vegas_config:
|
||||||
|
self.live_weight = max(1, min(10, int(vegas_config['live_weight'])))
|
||||||
|
if 'favorite_live_weight' in vegas_config:
|
||||||
|
self.favorite_live_weight = max(
|
||||||
|
1, min(10, int(vegas_config['favorite_live_weight'])))
|
||||||
if 'scroll_speed' in vegas_config:
|
if 'scroll_speed' in vegas_config:
|
||||||
self.scroll_speed = float(vegas_config['scroll_speed'])
|
self.scroll_speed = float(vegas_config['scroll_speed'])
|
||||||
if 'separator_width' in vegas_config:
|
if 'separator_width' in vegas_config:
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ Supports three display modes per plugin:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
import math
|
||||||
import time
|
import time
|
||||||
import threading
|
import threading
|
||||||
from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING
|
from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING
|
||||||
@@ -30,6 +31,33 @@ if TYPE_CHECKING:
|
|||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
#: Degradation threshold, as a fraction of target_fps. A marquee jitters a
|
||||||
|
#: little all the time, so "anything under target" would report constantly and
|
||||||
|
#: mean nothing; 90% of target is the point where a shortfall is real. At a
|
||||||
|
#: 60fps target that is 54fps -- 55fps is a normal wobble and stays at DEBUG,
|
||||||
|
#: which is deliberate, not an off-by-one.
|
||||||
|
_FPS_HEALTHY_FRACTION = 0.9
|
||||||
|
|
||||||
|
#: A healthy marquee still reports this often, so silence means stopped
|
||||||
|
#: rather than fine.
|
||||||
|
_FPS_HEARTBEAT_INTERVAL = 300.0
|
||||||
|
|
||||||
|
|
||||||
|
def _percentile(ordered: List[float], fraction: float) -> float:
|
||||||
|
"""Nearest-rank percentile of an already-sorted list.
|
||||||
|
|
||||||
|
Index ceil(n * fraction) - 1, so 100 samples at 0.99 give the 99th-ranked
|
||||||
|
value. The obvious int(n * fraction) is off by one and, at exactly 100
|
||||||
|
samples, lands on the maximum -- which is the number already reported
|
||||||
|
alongside this one as the worst frame, so the two columns would agree
|
||||||
|
precisely when the sample was smallest.
|
||||||
|
"""
|
||||||
|
if not ordered:
|
||||||
|
return 0.0
|
||||||
|
index = math.ceil(len(ordered) * fraction) - 1
|
||||||
|
return ordered[min(len(ordered) - 1, max(0, index))]
|
||||||
|
|
||||||
|
|
||||||
class VegasModeCoordinator:
|
class VegasModeCoordinator:
|
||||||
"""
|
"""
|
||||||
Orchestrates Vegas scroll mode operation.
|
Orchestrates Vegas scroll mode operation.
|
||||||
@@ -80,6 +108,11 @@ class VegasModeCoordinator:
|
|||||||
self._is_active = False
|
self._is_active = False
|
||||||
self._is_paused = False
|
self._is_paused = False
|
||||||
self._should_stop = False
|
self._should_stop = False
|
||||||
|
# Frame-rate health, tracked across run_iteration() calls so the
|
||||||
|
# heartbeat is one-per-interval rather than one-per-cycle, and so a
|
||||||
|
# recovery spanning two cycles is still reported. Reset on start().
|
||||||
|
self._fps_last_health_log = 0.0
|
||||||
|
self._fps_was_degraded = False
|
||||||
self._state_lock = threading.Lock()
|
self._state_lock = threading.Lock()
|
||||||
|
|
||||||
# Live priority tracking
|
# Live priority tracking
|
||||||
@@ -232,6 +265,11 @@ class VegasModeCoordinator:
|
|||||||
self._is_active = True
|
self._is_active = True
|
||||||
self._should_stop = False
|
self._should_stop = False
|
||||||
self._start_time = time.time()
|
self._start_time = time.time()
|
||||||
|
# A fresh run starts with a clean health slate: no stale
|
||||||
|
# "was degraded" from the previous run, and a heartbeat that is
|
||||||
|
# due immediately so the first sample confirms the marquee is up.
|
||||||
|
self._fps_last_health_log = 0.0
|
||||||
|
self._fps_was_degraded = False
|
||||||
|
|
||||||
# Line up the next group immediately, so the first extension is already
|
# Line up the next group immediately, so the first extension is already
|
||||||
# warm rather than stalling the scroll to fetch it.
|
# warm rather than stalling the scroll to fetch it.
|
||||||
@@ -379,14 +417,36 @@ class VegasModeCoordinator:
|
|||||||
duration = self.render_pipeline.get_dynamic_duration()
|
duration = self.render_pipeline.get_dynamic_duration()
|
||||||
start_time = time.time()
|
start_time = time.time()
|
||||||
frame_count = 0
|
frame_count = 0
|
||||||
fps_log_interval = 5.0 # Log FPS every 5 seconds
|
fps_log_interval = 5.0 # Sample FPS every 5 seconds
|
||||||
last_fps_log_time = start_time
|
# Health state lives on the coordinator, not here: run_iteration() is
|
||||||
|
# called once per cycle, so locals reset every few seconds. That made
|
||||||
|
# `last_fps_health_log = 0.0` fire the "heartbeat" on the first sample
|
||||||
|
# of every iteration rather than once per interval, and a recovery
|
||||||
|
# that crossed an iteration boundary was never reported at all --
|
||||||
|
# was_degraded had already gone back to False.
|
||||||
|
# Monotonic, and deliberately not start_time: start_time is wall
|
||||||
|
# clock and is used below to report the iteration's duration. Mixing
|
||||||
|
# the two here would make every delta hugely negative and silence the
|
||||||
|
# frame-rate reporting altogether.
|
||||||
|
last_fps_log_time = time.monotonic()
|
||||||
fps_frame_count = 0
|
fps_frame_count = 0
|
||||||
|
# A mean hides stutter completely. At 120fps a five-second window is
|
||||||
|
# ~600 frames, so a 200ms freeze -- plainly visible on a marquee --
|
||||||
|
# moves the average from 120.0 to 115.4 and reads as healthy. What a
|
||||||
|
# viewer actually notices is the worst frame, so track that too.
|
||||||
|
frame_worst = 0.0
|
||||||
|
frame_times: List[float] = []
|
||||||
|
|
||||||
logger.info("Starting Vegas iteration for %.1fs", duration)
|
logger.info("Starting Vegas iteration for %.1fs", duration)
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
frame_started = time.time()
|
# Monotonic, like the FPS window below. These devices have no RTC,
|
||||||
|
# so the wall clock jumps by however wrong boot time was the moment
|
||||||
|
# NTP first syncs. A backward jump makes frame_elapsed negative,
|
||||||
|
# and `frame_interval - frame_elapsed` then sleeps for longer than
|
||||||
|
# the whole budget -- the render loop stalls for the size of the
|
||||||
|
# correction. A forward jump inflates p99 and worst-frame instead.
|
||||||
|
frame_started = time.monotonic()
|
||||||
|
|
||||||
# Check for STATIC mode plugin that should pause scroll
|
# Check for STATIC mode plugin that should pause scroll
|
||||||
static_plugin = self._check_static_plugin_trigger()
|
static_plugin = self._check_static_plugin_trigger()
|
||||||
@@ -414,23 +474,58 @@ class VegasModeCoordinator:
|
|||||||
# quarter of the budget spent not rendering. Subtracting the work
|
# quarter of the budget spent not rendering. Subtracting the work
|
||||||
# already done keeps the pacing target while reclaiming that time,
|
# already done keeps the pacing target while reclaiming that time,
|
||||||
# and yields the GIL either way so other threads still run.
|
# and yields the GIL either way so other threads still run.
|
||||||
frame_elapsed = time.time() - frame_started
|
frame_elapsed = time.monotonic() - frame_started
|
||||||
time.sleep(max(0.0, frame_interval - frame_elapsed))
|
time.sleep(max(0.0, frame_interval - frame_elapsed))
|
||||||
|
|
||||||
|
# Measured before the sleep: time spent working, not pacing.
|
||||||
|
if frame_elapsed > frame_worst:
|
||||||
|
frame_worst = frame_elapsed
|
||||||
|
frame_times.append(frame_elapsed)
|
||||||
|
|
||||||
# Increment frame count and check for interrupt periodically
|
# Increment frame count and check for interrupt periodically
|
||||||
frame_count += 1
|
frame_count += 1
|
||||||
fps_frame_count += 1
|
fps_frame_count += 1
|
||||||
|
|
||||||
# Periodic FPS logging
|
# Periodic FPS logging. Reported at INFO only when the frame rate
|
||||||
current_time = time.time()
|
# is actually worth an operator's attention -- a shortfall against
|
||||||
|
# target, or the recovery from one -- with a slow heartbeat so a
|
||||||
|
# healthy marquee still shows a pulse.
|
||||||
|
#
|
||||||
|
# Measured over two hours on a running rig: 1410 samples, 98.5%
|
||||||
|
# of them within 10% of target. The 1.5% that were not included a
|
||||||
|
# reading of 8.6fps against a target of 60 -- a real stall, and
|
||||||
|
# completely invisible inside 1389 lines reading "59.6".
|
||||||
|
# Monotonic: every use of this value in the block below is a
|
||||||
|
# duration, and these devices have no RTC, so the wall clock jumps
|
||||||
|
# by however wrong boot time was the moment NTP first syncs. That
|
||||||
|
# would not only mis-fire the heartbeat, it would corrupt the
|
||||||
|
# frame rate itself, since fps is frames divided by this delta.
|
||||||
|
current_time = time.monotonic()
|
||||||
if current_time - last_fps_log_time >= fps_log_interval:
|
if current_time - last_fps_log_time >= fps_log_interval:
|
||||||
fps = fps_frame_count / (current_time - last_fps_log_time)
|
fps = fps_frame_count / (current_time - last_fps_log_time)
|
||||||
|
p99 = _percentile(sorted(frame_times), 0.99)
|
||||||
|
target = self.vegas_config.target_fps
|
||||||
|
degraded = target > 0 and fps < target * _FPS_HEALTHY_FRACTION
|
||||||
|
due = (current_time - self._fps_last_health_log
|
||||||
|
>= _FPS_HEARTBEAT_INTERVAL)
|
||||||
|
if degraded or self._fps_was_degraded or due:
|
||||||
logger.info(
|
logger.info(
|
||||||
"Vegas FPS: %.1f (target: %d, frames: %d)",
|
"Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms",
|
||||||
fps, self.vegas_config.target_fps, fps_frame_count
|
fps, target, fps_frame_count,
|
||||||
|
p99 * 1000.0, frame_worst * 1000.0
|
||||||
)
|
)
|
||||||
|
self._fps_last_health_log = current_time
|
||||||
|
else:
|
||||||
|
logger.debug(
|
||||||
|
"Vegas FPS: %.1f (target: %d, frames: %d) p99 %.1fms worst %.1fms",
|
||||||
|
fps, target, fps_frame_count,
|
||||||
|
p99 * 1000.0, frame_worst * 1000.0
|
||||||
|
)
|
||||||
|
self._fps_was_degraded = degraded
|
||||||
last_fps_log_time = current_time
|
last_fps_log_time = current_time
|
||||||
fps_frame_count = 0
|
fps_frame_count = 0
|
||||||
|
frame_worst = 0.0
|
||||||
|
frame_times.clear()
|
||||||
|
|
||||||
if (self._interrupt_check and
|
if (self._interrupt_check and
|
||||||
frame_count % self._interrupt_check_interval == 0):
|
frame_count % self._interrupt_check_interval == 0):
|
||||||
@@ -497,6 +592,12 @@ class VegasModeCoordinator:
|
|||||||
if not self._live_priority_check:
|
if not self._live_priority_check:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
if self.vegas_config.live_in_ticker:
|
||||||
|
# The ticker keeps live content rather than yielding to it; the
|
||||||
|
# extra turns are arranged in the rotation itself, so there is
|
||||||
|
# nothing to pause for.
|
||||||
|
return False
|
||||||
|
|
||||||
try:
|
try:
|
||||||
live_mode = self._live_priority_check()
|
live_mode = self._live_priority_check()
|
||||||
if live_mode:
|
if live_mode:
|
||||||
|
|||||||
@@ -68,7 +68,22 @@ class PluginAdapter:
|
|||||||
# always the same opening items.
|
# always the same opening items.
|
||||||
self._item_offsets: dict = {}
|
self._item_offsets: dict = {}
|
||||||
|
|
||||||
logger.info(
|
# What the matching entry in _item_offsets is an offset *into*, as
|
||||||
|
# (kind, size). An offset only means anything against the content it
|
||||||
|
# was derived from, and there are three incompatible kinds:
|
||||||
|
#
|
||||||
|
# ('rows', n) index into a list of n images
|
||||||
|
# ('cuts', n) index into the n item boundaries of one image
|
||||||
|
# ('cols', w) pixel column in a w-wide image with no item boundaries
|
||||||
|
#
|
||||||
|
# Without this the offsets were reused across kinds — a plugin that
|
||||||
|
# returned one wide image on one fetch and several rows on the next had
|
||||||
|
# a pixel column of 1400 read back as a row index — and across content
|
||||||
|
# changes, where a column recorded against a 9,793px news strip pointed
|
||||||
|
# into unrelated headlines once the strip refreshed to 9,505px.
|
||||||
|
self._offset_shapes: dict = {}
|
||||||
|
|
||||||
|
logger.debug(
|
||||||
"PluginAdapter initialized: display=%dx%d",
|
"PluginAdapter initialized: display=%dx%d",
|
||||||
self.display_width, self.display_height
|
self.display_width, self.display_height
|
||||||
)
|
)
|
||||||
@@ -94,7 +109,7 @@ class PluginAdapter:
|
|||||||
Returns:
|
Returns:
|
||||||
List of PIL Images representing plugin content, or None if no content
|
List of PIL Images representing plugin content, or None if no content
|
||||||
"""
|
"""
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Getting content (class=%s)",
|
"[%s] Getting content (class=%s)",
|
||||||
plugin_id, plugin.__class__.__name__
|
plugin_id, plugin.__class__.__name__
|
||||||
)
|
)
|
||||||
@@ -103,7 +118,7 @@ class PluginAdapter:
|
|||||||
cached = self._get_cached(plugin_id)
|
cached = self._get_cached(plugin_id)
|
||||||
if cached is not None:
|
if cached is not None:
|
||||||
total_width = sum(img.width for img in cached)
|
total_width = sum(img.width for img in cached)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Using cached content: %d images, %dpx total",
|
"[%s] Using cached content: %d images, %dpx total",
|
||||||
plugin_id, len(cached), total_width
|
plugin_id, len(cached), total_width
|
||||||
)
|
)
|
||||||
@@ -111,46 +126,46 @@ class PluginAdapter:
|
|||||||
|
|
||||||
# Try native Vegas content method first
|
# Try native Vegas content method first
|
||||||
has_native = hasattr(plugin, 'get_vegas_content')
|
has_native = hasattr(plugin, 'get_vegas_content')
|
||||||
logger.info("[%s] Has get_vegas_content: %s", plugin_id, has_native)
|
logger.debug("[%s] Has get_vegas_content: %s", plugin_id, has_native)
|
||||||
if has_native:
|
if has_native:
|
||||||
content = self._get_native_content(plugin, plugin_id, offscreen_only)
|
content = self._get_native_content(plugin, plugin_id, offscreen_only)
|
||||||
if content:
|
if content:
|
||||||
total_width = sum(img.width for img in content)
|
total_width = sum(img.width for img in content)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Native content SUCCESS: %d images, %dpx total",
|
"[%s] Native content SUCCESS: %d images, %dpx total",
|
||||||
plugin_id, len(content), total_width
|
plugin_id, len(content), total_width
|
||||||
)
|
)
|
||||||
return self._finalize(content, plugin_id, 'native', plugin)
|
return self._finalize(content, plugin_id, 'native', plugin)
|
||||||
logger.info("[%s] Native content returned None", plugin_id)
|
logger.debug("[%s] Native content returned None", plugin_id)
|
||||||
|
|
||||||
# Try to get scroll_helper's cached image (for scrolling plugins like stocks/odds)
|
# Try to get scroll_helper's cached image (for scrolling plugins like stocks/odds)
|
||||||
has_scroll_helper = hasattr(plugin, 'scroll_helper')
|
has_scroll_helper = hasattr(plugin, 'scroll_helper')
|
||||||
logger.info("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper)
|
logger.debug("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper)
|
||||||
content = self._get_scroll_helper_content(plugin, plugin_id, offscreen_only)
|
content = self._get_scroll_helper_content(plugin, plugin_id, offscreen_only)
|
||||||
if content:
|
if content:
|
||||||
total_width = sum(img.width for img in content)
|
total_width = sum(img.width for img in content)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] ScrollHelper content SUCCESS: %d images, %dpx total",
|
"[%s] ScrollHelper content SUCCESS: %d images, %dpx total",
|
||||||
plugin_id, len(content), total_width
|
plugin_id, len(content), total_width
|
||||||
)
|
)
|
||||||
return self._finalize(content, plugin_id, 'scroll_helper', plugin)
|
return self._finalize(content, plugin_id, 'scroll_helper', plugin)
|
||||||
if has_scroll_helper:
|
if has_scroll_helper:
|
||||||
logger.info("[%s] ScrollHelper content returned None", plugin_id)
|
logger.debug("[%s] ScrollHelper content returned None", plugin_id)
|
||||||
|
|
||||||
if offscreen_only:
|
if offscreen_only:
|
||||||
# Display capture needs the shared canvas; leave it to the caller.
|
# Display capture needs the shared canvas; leave it to the caller.
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Needs display capture, deferring to the render thread",
|
"[%s] Needs display capture, deferring to the render thread",
|
||||||
plugin_id
|
plugin_id
|
||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
# Fall back to display capture
|
# Fall back to display capture
|
||||||
logger.info("[%s] Trying fallback display capture...", plugin_id)
|
logger.debug("[%s] Trying fallback display capture...", plugin_id)
|
||||||
content = self._capture_display_content(plugin, plugin_id)
|
content = self._capture_display_content(plugin, plugin_id)
|
||||||
if content:
|
if content:
|
||||||
total_width = sum(img.width for img in content)
|
total_width = sum(img.width for img in content)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Fallback capture SUCCESS: %d images, %dpx total",
|
"[%s] Fallback capture SUCCESS: %d images, %dpx total",
|
||||||
plugin_id, len(content), total_width
|
plugin_id, len(content), total_width
|
||||||
)
|
)
|
||||||
@@ -211,7 +226,7 @@ class PluginAdapter:
|
|||||||
kept.append(result.image)
|
kept.append(result.image)
|
||||||
|
|
||||||
if not kept:
|
if not kept:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] All %d image(s) from %s were blank — contributing nothing",
|
"[%s] All %d image(s) from %s were blank — contributing nothing",
|
||||||
plugin_id, len(images), source
|
plugin_id, len(images), source
|
||||||
)
|
)
|
||||||
@@ -220,14 +235,14 @@ class PluginAdapter:
|
|||||||
trimmed_width = sum(img.width for img in kept)
|
trimmed_width = sum(img.width for img in kept)
|
||||||
|
|
||||||
if trimmed_width < self.config.min_plugin_width:
|
if trimmed_width < self.config.min_plugin_width:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Trimmed content %dpx is below min_plugin_width %dpx — skipping",
|
"[%s] Trimmed content %dpx is below min_plugin_width %dpx — skipping",
|
||||||
plugin_id, trimmed_width, self.config.min_plugin_width
|
plugin_id, trimmed_width, self.config.min_plugin_width
|
||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
if trimmed_width != original_width or dropped_blank:
|
if trimmed_width != original_width or dropped_blank:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Trimmed %s content: %dpx -> %dpx (%.0f%% reclaimed), "
|
"[%s] Trimmed %s content: %dpx -> %dpx (%.0f%% reclaimed), "
|
||||||
"%d image(s) kept, %d blank dropped",
|
"%d image(s) kept, %d blank dropped",
|
||||||
plugin_id, source, original_width, trimmed_width,
|
plugin_id, source, original_width, trimmed_width,
|
||||||
@@ -398,6 +413,88 @@ class PluginAdapter:
|
|||||||
return 0
|
return 0
|
||||||
return int(self.display_width * ratio)
|
return int(self.display_width * ratio)
|
||||||
|
|
||||||
|
def _resume_offset(self, plugin_id: str, shape: Tuple[str, int]) -> int:
|
||||||
|
"""
|
||||||
|
The plugin's stored rotation offset, if it still applies.
|
||||||
|
|
||||||
|
An offset is only meaningful against content shaped the way it was
|
||||||
|
when the offset was recorded. When the shape has changed — a different
|
||||||
|
number of rows, a re-rendered strip with different item boundaries —
|
||||||
|
the stored value points somewhere arbitrary, so rotation restarts.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
plugin_id: Plugin identifier
|
||||||
|
shape: (kind, size) describing what an offset would index into now
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The stored offset, or 0 when it no longer applies
|
||||||
|
"""
|
||||||
|
if self._offset_shapes.get(plugin_id) != shape:
|
||||||
|
if plugin_id in self._item_offsets:
|
||||||
|
logger.debug(
|
||||||
|
"[%s] Content is %s now, was %s — restarting the rotation "
|
||||||
|
"rather than resuming at a position that no longer means "
|
||||||
|
"anything", plugin_id, shape,
|
||||||
|
self._offset_shapes.get(plugin_id))
|
||||||
|
self._item_offsets.pop(plugin_id, None)
|
||||||
|
self._offset_shapes[plugin_id] = shape
|
||||||
|
return 0
|
||||||
|
return self._item_offsets.get(plugin_id, 0)
|
||||||
|
|
||||||
|
def _record_offset(
|
||||||
|
self, plugin_id: str, offset: int, shape: Tuple[str, int]
|
||||||
|
) -> None:
|
||||||
|
"""Store where the next window should resume, with what it indexes."""
|
||||||
|
if offset:
|
||||||
|
self._item_offsets[plugin_id] = offset
|
||||||
|
self._offset_shapes[plugin_id] = shape
|
||||||
|
else:
|
||||||
|
# A wrapped-to-zero rotation is the same as no state at all, and
|
||||||
|
# keeping the key would report a window as active when the next
|
||||||
|
# pass starts from the top anyway.
|
||||||
|
self._item_offsets.pop(plugin_id, None)
|
||||||
|
self._offset_shapes.pop(plugin_id, None)
|
||||||
|
|
||||||
|
def _clear_offset(self, plugin_id: str) -> None:
|
||||||
|
"""Forget any rotation state for a plugin."""
|
||||||
|
self._item_offsets.pop(plugin_id, None)
|
||||||
|
self._offset_shapes.pop(plugin_id, None)
|
||||||
|
|
||||||
|
def _merge_trailing_runt(self, end: int, width: int, budget: int) -> int:
|
||||||
|
"""
|
||||||
|
Extend a window to the end of the content when what would be left over
|
||||||
|
is too small to be worth its own pass.
|
||||||
|
|
||||||
|
Windows were placed by walking forward from the last one, which makes
|
||||||
|
the final window whatever happens to remain. Measured on a live panel
|
||||||
|
that produced a 1,840px stocks ticker splitting 1,492 + 348 — the
|
||||||
|
second pass showing seven seconds of content before cutting, which
|
||||||
|
reads as the display failing rather than as a rotation.
|
||||||
|
|
||||||
|
Absorbing the remainder overruns the budget by less than one window
|
||||||
|
floor, which is a better trade than a fragment: the budget is a guard
|
||||||
|
against one plugin holding the panel for minutes, not a hard limit.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
end: Column the window would otherwise end at
|
||||||
|
width: Full content width
|
||||||
|
budget: Width budget being applied
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
``end``, or ``width`` when the remainder is below the floor
|
||||||
|
"""
|
||||||
|
remainder = width - end
|
||||||
|
# Measured against the budget rather than the panel: snapping to item
|
||||||
|
# boundaries means an ordinary window already lands short of the budget
|
||||||
|
# (a 512px budget over 182px-pitch items yields 348px windows), so an
|
||||||
|
# absolute floor would merge windows that were never fragments. Half a
|
||||||
|
# budget separates "a short last pass" from "a sliver", and caps the
|
||||||
|
# overrun this can cause at 1.5 budgets.
|
||||||
|
floor = budget // 2
|
||||||
|
if 0 < remainder < floor:
|
||||||
|
return width
|
||||||
|
return end
|
||||||
|
|
||||||
def _apply_width_budget(
|
def _apply_width_budget(
|
||||||
self, images: List[Image.Image], plugin_id: str,
|
self, images: List[Image.Image], plugin_id: str,
|
||||||
plugin: Optional['BasePlugin'] = None
|
plugin: Optional['BasePlugin'] = None
|
||||||
@@ -435,45 +532,62 @@ class PluginAdapter:
|
|||||||
|
|
||||||
if not budget or total <= budget:
|
if not budget or total <= budget:
|
||||||
# Fits, so reset rotation — the whole segment is being shown.
|
# Fits, so reset rotation — the whole segment is being shown.
|
||||||
self._item_offsets.pop(plugin_id, None)
|
self._clear_offset(plugin_id)
|
||||||
return images
|
return images
|
||||||
|
|
||||||
if len(images) == 1:
|
if len(images) == 1:
|
||||||
return [self._crop_to_budget(images[0], budget, plugin_id, mode)]
|
return [self._crop_to_budget(images[0], budget, plugin_id, mode)]
|
||||||
|
|
||||||
|
shape = ('rows', len(images))
|
||||||
if mode == 'truncate':
|
if mode == 'truncate':
|
||||||
# Ordered content: always show from the top. Deliberately does not
|
# Ordered content: always show from the top. Deliberately does not
|
||||||
# advance the offset, so the same opening items appear every time
|
# advance the offset, so the same opening items appear every time
|
||||||
# rather than the viewer being shown the middle of a ranked list.
|
# rather than the viewer being shown the middle of a ranked list.
|
||||||
start = 0
|
start = 0
|
||||||
else:
|
else:
|
||||||
start = self._item_offsets.get(plugin_id, 0) % len(images)
|
start = self._resume_offset(plugin_id, shape) % len(images)
|
||||||
selected: List[Image.Image] = []
|
selected: List[Image.Image] = []
|
||||||
used = 0
|
used = 0
|
||||||
consumed = 0
|
consumed = 0
|
||||||
|
|
||||||
# Walk forward from the rotation offset, taking whole items only, so a
|
# Walk forward from the rotation offset, taking whole items only, so a
|
||||||
# cut never lands in the middle of one.
|
# cut never lands in the middle of one.
|
||||||
|
#
|
||||||
|
# A window may overrun the budget while it is still shorter than the
|
||||||
|
# runt floor, for the same reason _merge_trailing_runt exists on the
|
||||||
|
# single-image path: a pass far shorter than its neighbours reads as
|
||||||
|
# the display failing rather than as a rotation. Rows of 450, 450 and
|
||||||
|
# 100 against a 512px budget used to give the 100 a pass of its own --
|
||||||
|
# two seconds against nine. Wrapping does not prevent that, because it
|
||||||
|
# only helps when the row wrapped to actually fits.
|
||||||
|
floor = budget // 2
|
||||||
for step in range(len(images)):
|
for step in range(len(images)):
|
||||||
img = images[(start + step) % len(images)]
|
img = images[(start + step) % len(images)]
|
||||||
cost = img.width
|
cost = img.width
|
||||||
if selected:
|
if selected:
|
||||||
cost += self._row_gap(selected[-1], img)
|
cost += self._row_gap(selected[-1], img)
|
||||||
if selected and used + cost > budget:
|
if selected and used + cost > budget:
|
||||||
|
# Keep the overrun bounded at the same 1.5 budgets the
|
||||||
|
# single-image path allows. A next row too wide to absorb
|
||||||
|
# leaves a short window standing -- better than a window of
|
||||||
|
# 1.9 budgets, and the same trade the always-take-the-first
|
||||||
|
# rule below already makes.
|
||||||
|
if used >= floor or used + cost > budget + floor:
|
||||||
break
|
break
|
||||||
selected.append(img)
|
selected.append(img)
|
||||||
used += cost
|
used += cost
|
||||||
consumed += 1
|
consumed += 1
|
||||||
|
|
||||||
if mode == 'truncate':
|
if mode == 'truncate':
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Width budget %dpx: showing the first %d of %d row(s) "
|
"[%s] Width budget %dpx: showing the first %d of %d row(s) "
|
||||||
"(%dpx incl. gaps); the rest are not shown (overflow=truncate)",
|
"(%dpx incl. gaps); the rest are not shown (overflow=truncate)",
|
||||||
plugin_id, budget, len(selected), len(images), used
|
plugin_id, budget, len(selected), len(images), used
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
self._item_offsets[plugin_id] = (start + consumed) % len(images)
|
self._record_offset(
|
||||||
logger.info(
|
plugin_id, (start + consumed) % len(images), shape)
|
||||||
|
logger.debug(
|
||||||
"[%s] Width budget %dpx: showing %d of %d row(s) (%dpx incl. gaps) "
|
"[%s] Width budget %dpx: showing %d of %d row(s) (%dpx incl. gaps) "
|
||||||
"from offset %d; remainder deferred to a later cycle",
|
"from offset %d; remainder deferred to a later cycle",
|
||||||
plugin_id, budget, len(selected), len(images), used, start
|
plugin_id, budget, len(selected), len(images), used, start
|
||||||
@@ -490,16 +604,13 @@ class PluginAdapter:
|
|||||||
|
|
||||||
The cut is snapped to the nearest blank column so it does not slice
|
The cut is snapped to the nearest blank column so it does not slice
|
||||||
through a glyph or logo and leave half a character at the panel edge.
|
through a glyph or logo and leave half a character at the panel edge.
|
||||||
"""
|
|
||||||
if mode == 'truncate':
|
|
||||||
# Always the start of the strip, so a ranked table is never entered
|
|
||||||
# from the middle.
|
|
||||||
offset = 0
|
|
||||||
else:
|
|
||||||
offset = self._item_offsets.get(plugin_id, 0)
|
|
||||||
if offset >= img.width:
|
|
||||||
offset = 0
|
|
||||||
|
|
||||||
|
Rotation is tracked as an index into the strip's item boundaries rather
|
||||||
|
than as a pixel column, because a ticker re-renders between fetches. A
|
||||||
|
column recorded against one render points at unrelated content in the
|
||||||
|
next as soon as anything ahead of it changes width — a digit in a
|
||||||
|
price, a shorter headline. The Nth boundary stays the Nth boundary.
|
||||||
|
"""
|
||||||
# Cut only where the plugin left a real gap between items. Snapping to
|
# Cut only where the plugin left a real gap between items. Snapping to
|
||||||
# any blank column used to pick the single-column gaps between
|
# any blank column used to pick the single-column gaps between
|
||||||
# characters, splitting a word and orphaning its tail into the next
|
# characters, splitting a word and orphaning its tail into the next
|
||||||
@@ -514,10 +625,18 @@ class PluginAdapter:
|
|||||||
# budget exactly. The gap rule exists to protect discrete items
|
# budget exactly. The gap rule exists to protect discrete items
|
||||||
# (words, ticker entries); it would be wrong to let a solid image
|
# (words, ticker entries); it would be wrong to let a solid image
|
||||||
# escape the cap in its name.
|
# escape the cap in its name.
|
||||||
end = min(offset + budget, img.width)
|
#
|
||||||
|
# With no items to index, the offset here has to stay a column, so
|
||||||
|
# it is only reusable while the image keeps its width.
|
||||||
|
shape = ('cols', img.width)
|
||||||
|
offset = 0 if mode == 'truncate' else self._resume_offset(
|
||||||
|
plugin_id, shape)
|
||||||
|
end = self._merge_trailing_runt(
|
||||||
|
min(offset + budget, img.width), img.width, budget)
|
||||||
if mode != 'truncate':
|
if mode != 'truncate':
|
||||||
self._item_offsets[plugin_id] = 0 if end >= img.width else end
|
self._record_offset(
|
||||||
logger.info(
|
plugin_id, 0 if end >= img.width else end, shape)
|
||||||
|
logger.debug(
|
||||||
"[%s] Width budget %dpx: cropped continuous %dpx image to "
|
"[%s] Width budget %dpx: cropped continuous %dpx image to "
|
||||||
"[%d:%d] (no item gaps of %dpx+ to align to)%s",
|
"[%d:%d] (no item gaps of %dpx+ to align to)%s",
|
||||||
plugin_id, budget, img.width, offset, end, min_run,
|
plugin_id, budget, img.width, offset, end, min_run,
|
||||||
@@ -528,8 +647,15 @@ class PluginAdapter:
|
|||||||
# Cut mid-gap so the content either side keeps some breathing room.
|
# Cut mid-gap so the content either side keeps some breathing room.
|
||||||
cuts = sorted({0, img.width} | {(a + b) // 2 for a, b in gaps})
|
cuts = sorted({0, img.width} | {(a + b) // 2 for a, b in gaps})
|
||||||
|
|
||||||
start = max((c for c in cuts if c <= offset), default=0)
|
shape = ('cuts', len(cuts))
|
||||||
later = [c for c in cuts if c > start]
|
index = 0 if mode == 'truncate' else self._resume_offset(
|
||||||
|
plugin_id, shape)
|
||||||
|
# Clamped rather than wrapped: a stale index past the end means the
|
||||||
|
# strip shrank, and restarting reads better than landing near the end.
|
||||||
|
start_index = index if 0 <= index < len(cuts) - 1 else 0
|
||||||
|
start = cuts[start_index]
|
||||||
|
|
||||||
|
later = cuts[start_index + 1:]
|
||||||
if not later:
|
if not later:
|
||||||
end = img.width
|
end = img.width
|
||||||
else:
|
else:
|
||||||
@@ -537,15 +663,22 @@ class PluginAdapter:
|
|||||||
# No boundary inside the budget: take the next one and overrun,
|
# No boundary inside the budget: take the next one and overrun,
|
||||||
# because the alternative is cutting through an item.
|
# because the alternative is cutting through an item.
|
||||||
end = max(within) if within else min(later)
|
end = max(within) if within else min(later)
|
||||||
|
end = self._merge_trailing_runt(end, img.width, budget)
|
||||||
|
# Every candidate for `end` came from `cuts` (which includes img.width),
|
||||||
|
# so this always resolves; the fallback is defensive only.
|
||||||
|
end_index = cuts.index(end) if end in cuts else len(cuts) - 1
|
||||||
|
|
||||||
if mode != 'truncate':
|
if mode != 'truncate':
|
||||||
# Next cycle resumes where this one stopped; wrap when the strip ends.
|
# Next cycle resumes at the boundary this one stopped on; wrap when
|
||||||
self._item_offsets[plugin_id] = 0 if end >= img.width else end
|
# the strip ends.
|
||||||
|
self._record_offset(
|
||||||
|
plugin_id, 0 if end >= img.width else end_index, shape)
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Width budget %dpx: cropped single %dpx image to [%d:%d] "
|
"[%s] Width budget %dpx: cropped single %dpx image to [%d:%d] "
|
||||||
"(%dpx) at item boundaries, %s",
|
"(%dpx) at item boundaries %d-%d of %d, %s",
|
||||||
plugin_id, budget, img.width, start, end, end - start,
|
plugin_id, budget, img.width, start, end, end - start,
|
||||||
|
start_index, end_index, len(cuts) - 1,
|
||||||
"showing the start only (overflow=truncate)"
|
"showing the start only (overflow=truncate)"
|
||||||
if mode == 'truncate' else "window advances next cycle"
|
if mode == 'truncate' else "window advances next cycle"
|
||||||
)
|
)
|
||||||
@@ -565,7 +698,7 @@ class PluginAdapter:
|
|||||||
List of images or None
|
List of images or None
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
logger.info("[%s] Native: calling get_vegas_content()", plugin_id)
|
logger.debug("[%s] Native: calling get_vegas_content()", plugin_id)
|
||||||
|
|
||||||
# Tell the plugin how much width the ticker wants it to use, and
|
# Tell the plugin how much width the ticker wants it to use, and
|
||||||
# narrow the canvas for the duration of the call. A plugin that
|
# narrow the canvas for the duration of the call. A plugin that
|
||||||
@@ -574,7 +707,7 @@ class PluginAdapter:
|
|||||||
# be explicit can read get_vegas_render_width().
|
# be explicit can read get_vegas_render_width().
|
||||||
render_width = self.resolve_render_width(plugin, plugin_id)
|
render_width = self.resolve_render_width(plugin, plugin_id)
|
||||||
if render_width != self.display_width:
|
if render_width != self.display_width:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Native: requesting %dpx instead of %dpx",
|
"[%s] Native: requesting %dpx instead of %dpx",
|
||||||
plugin_id, render_width, self.display_width
|
plugin_id, render_width, self.display_width
|
||||||
)
|
)
|
||||||
@@ -602,19 +735,19 @@ class PluginAdapter:
|
|||||||
plugin._vegas_render_width = None
|
plugin._vegas_render_width = None
|
||||||
|
|
||||||
if result is None:
|
if result is None:
|
||||||
logger.info("[%s] Native: get_vegas_content() returned None", plugin_id)
|
logger.debug("[%s] Native: get_vegas_content() returned None", plugin_id)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
# Normalize to list
|
# Normalize to list
|
||||||
if isinstance(result, Image.Image):
|
if isinstance(result, Image.Image):
|
||||||
images = [result]
|
images = [result]
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Native: got single Image %dx%d",
|
"[%s] Native: got single Image %dx%d",
|
||||||
plugin_id, result.width, result.height
|
plugin_id, result.width, result.height
|
||||||
)
|
)
|
||||||
elif isinstance(result, (list, tuple)):
|
elif isinstance(result, (list, tuple)):
|
||||||
images = list(result)
|
images = list(result)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Native: got %d items in list/tuple",
|
"[%s] Native: got %d items in list/tuple",
|
||||||
plugin_id, len(images)
|
plugin_id, len(images)
|
||||||
)
|
)
|
||||||
@@ -635,14 +768,14 @@ class PluginAdapter:
|
|||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Native: item[%d] is %dx%d, mode=%s",
|
"[%s] Native: item[%d] is %dx%d, mode=%s",
|
||||||
plugin_id, i, img.width, img.height, img.mode
|
plugin_id, i, img.width, img.height, img.mode
|
||||||
)
|
)
|
||||||
|
|
||||||
# Ensure correct height
|
# Ensure correct height
|
||||||
if img.height != self.display_height:
|
if img.height != self.display_height:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Native: resizing item[%d]: %dx%d -> %dx%d",
|
"[%s] Native: resizing item[%d]: %dx%d -> %dx%d",
|
||||||
plugin_id, i, img.width, img.height,
|
plugin_id, i, img.width, img.height,
|
||||||
img.width, self.display_height
|
img.width, self.display_height
|
||||||
@@ -660,13 +793,13 @@ class PluginAdapter:
|
|||||||
|
|
||||||
if valid_images:
|
if valid_images:
|
||||||
total_width = sum(img.width for img in valid_images)
|
total_width = sum(img.width for img in valid_images)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Native: SUCCESS - %d images, %dpx total width",
|
"[%s] Native: SUCCESS - %d images, %dpx total width",
|
||||||
plugin_id, len(valid_images), total_width
|
plugin_id, len(valid_images), total_width
|
||||||
)
|
)
|
||||||
return valid_images
|
return valid_images
|
||||||
|
|
||||||
logger.info("[%s] Native: no valid images after validation", plugin_id)
|
logger.debug("[%s] Native: no valid images after validation", plugin_id)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
except (AttributeError, TypeError, ValueError, OSError) as e:
|
except (AttributeError, TypeError, ValueError, OSError) as e:
|
||||||
@@ -700,20 +833,20 @@ class PluginAdapter:
|
|||||||
logger.debug("[%s] No scroll_helper attribute", plugin_id)
|
logger.debug("[%s] No scroll_helper attribute", plugin_id)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Found scroll_helper: %s",
|
"[%s] Found scroll_helper: %s",
|
||||||
plugin_id, type(scroll_helper).__name__
|
plugin_id, type(scroll_helper).__name__
|
||||||
)
|
)
|
||||||
|
|
||||||
cached_image = getattr(scroll_helper, 'cached_image', None)
|
cached_image = getattr(scroll_helper, 'cached_image', None)
|
||||||
if cached_image is None:
|
if cached_image is None:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] scroll_helper.cached_image is None, triggering content generation",
|
"[%s] scroll_helper.cached_image is None, triggering content generation",
|
||||||
plugin_id
|
plugin_id
|
||||||
)
|
)
|
||||||
if offscreen_only:
|
if offscreen_only:
|
||||||
# Generating it calls display(), which needs the canvas.
|
# Generating it calls display(), which needs the canvas.
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] scroll_helper cache empty; deferring generation "
|
"[%s] scroll_helper cache empty; deferring generation "
|
||||||
"to the render thread", plugin_id
|
"to the render thread", plugin_id
|
||||||
)
|
)
|
||||||
@@ -726,13 +859,13 @@ class PluginAdapter:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
if not isinstance(cached_image, Image.Image):
|
if not isinstance(cached_image, Image.Image):
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] scroll_helper.cached_image is not an Image: %s",
|
"[%s] scroll_helper.cached_image is not an Image: %s",
|
||||||
plugin_id, type(cached_image).__name__
|
plugin_id, type(cached_image).__name__
|
||||||
)
|
)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] scroll_helper.cached_image found: %dx%d, mode=%s",
|
"[%s] scroll_helper.cached_image found: %dx%d, mode=%s",
|
||||||
plugin_id, cached_image.width, cached_image.height, cached_image.mode
|
plugin_id, cached_image.width, cached_image.height, cached_image.mode
|
||||||
)
|
)
|
||||||
@@ -755,7 +888,7 @@ class PluginAdapter:
|
|||||||
|
|
||||||
# Ensure correct height
|
# Ensure correct height
|
||||||
if img.height != self.display_height:
|
if img.height != self.display_height:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Resizing scroll_helper content: %dx%d -> %dx%d",
|
"[%s] Resizing scroll_helper content: %dx%d -> %dx%d",
|
||||||
plugin_id, img.width, img.height,
|
plugin_id, img.width, img.height,
|
||||||
img.width, self.display_height
|
img.width, self.display_height
|
||||||
@@ -769,7 +902,7 @@ class PluginAdapter:
|
|||||||
if img.mode != 'RGB':
|
if img.mode != 'RGB':
|
||||||
img = img.convert('RGB')
|
img = img.convert('RGB')
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] ScrollHelper content ready: %dx%d",
|
"[%s] ScrollHelper content ready: %dx%d",
|
||||||
plugin_id, img.width, img.height
|
plugin_id, img.width, img.height
|
||||||
)
|
)
|
||||||
@@ -869,7 +1002,7 @@ class PluginAdapter:
|
|||||||
with self._capture():
|
with self._capture():
|
||||||
# Method 1: Try _create_scrolling_display (stocks pattern)
|
# Method 1: Try _create_scrolling_display (stocks pattern)
|
||||||
if hasattr(plugin, '_create_scrolling_display'):
|
if hasattr(plugin, '_create_scrolling_display'):
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Triggering via _create_scrolling_display()",
|
"[%s] Triggering via _create_scrolling_display()",
|
||||||
plugin_id
|
plugin_id
|
||||||
)
|
)
|
||||||
@@ -877,7 +1010,7 @@ class PluginAdapter:
|
|||||||
plugin._create_scrolling_display()
|
plugin._create_scrolling_display()
|
||||||
cached_image = getattr(scroll_helper, 'cached_image', None)
|
cached_image = getattr(scroll_helper, 'cached_image', None)
|
||||||
if cached_image is not None and isinstance(cached_image, Image.Image):
|
if cached_image is not None and isinstance(cached_image, Image.Image):
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] _create_scrolling_display() SUCCESS: %dx%d",
|
"[%s] _create_scrolling_display() SUCCESS: %dx%d",
|
||||||
plugin_id, cached_image.width, cached_image.height
|
plugin_id, cached_image.width, cached_image.height
|
||||||
)
|
)
|
||||||
@@ -889,7 +1022,7 @@ class PluginAdapter:
|
|||||||
|
|
||||||
# Method 2: Try display(force_clear=True) which typically builds scroll content
|
# Method 2: Try display(force_clear=True) which typically builds scroll content
|
||||||
if hasattr(plugin, 'display'):
|
if hasattr(plugin, 'display'):
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Triggering via display(force_clear=True)",
|
"[%s] Triggering via display(force_clear=True)",
|
||||||
plugin_id
|
plugin_id
|
||||||
)
|
)
|
||||||
@@ -898,12 +1031,12 @@ class PluginAdapter:
|
|||||||
plugin.display(force_clear=True)
|
plugin.display(force_clear=True)
|
||||||
cached_image = getattr(scroll_helper, 'cached_image', None)
|
cached_image = getattr(scroll_helper, 'cached_image', None)
|
||||||
if cached_image is not None and isinstance(cached_image, Image.Image):
|
if cached_image is not None and isinstance(cached_image, Image.Image):
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] display(force_clear=True) SUCCESS: %dx%d",
|
"[%s] display(force_clear=True) SUCCESS: %dx%d",
|
||||||
plugin_id, cached_image.width, cached_image.height
|
plugin_id, cached_image.width, cached_image.height
|
||||||
)
|
)
|
||||||
return cached_image
|
return cached_image
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] display(force_clear=True) did not populate cached_image",
|
"[%s] display(force_clear=True) did not populate cached_image",
|
||||||
plugin_id
|
plugin_id
|
||||||
)
|
)
|
||||||
@@ -912,7 +1045,7 @@ class PluginAdapter:
|
|||||||
"[%s] display(force_clear=True) failed", plugin_id
|
"[%s] display(force_clear=True) failed", plugin_id
|
||||||
)
|
)
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Could not trigger scroll content generation",
|
"[%s] Could not trigger scroll content generation",
|
||||||
plugin_id
|
plugin_id
|
||||||
)
|
)
|
||||||
@@ -944,15 +1077,15 @@ class PluginAdapter:
|
|||||||
try:
|
try:
|
||||||
# Save current display state
|
# Save current display state
|
||||||
original_image = self.display_manager.image.copy()
|
original_image = self.display_manager.image.copy()
|
||||||
logger.info("[%s] Fallback: saved original display state", plugin_id)
|
logger.debug("[%s] Fallback: saved original display state", plugin_id)
|
||||||
|
|
||||||
# Ensure plugin has fresh data before capturing
|
# Ensure plugin has fresh data before capturing
|
||||||
has_update_data = hasattr(plugin, 'update_data')
|
has_update_data = hasattr(plugin, 'update_data')
|
||||||
logger.info("[%s] Fallback: has update_data=%s", plugin_id, has_update_data)
|
logger.debug("[%s] Fallback: has update_data=%s", plugin_id, has_update_data)
|
||||||
if has_update_data:
|
if has_update_data:
|
||||||
try:
|
try:
|
||||||
plugin.update_data()
|
plugin.update_data()
|
||||||
logger.info("[%s] Fallback: update_data() called", plugin_id)
|
logger.debug("[%s] Fallback: update_data() called", plugin_id)
|
||||||
except (AttributeError, RuntimeError, OSError):
|
except (AttributeError, RuntimeError, OSError):
|
||||||
logger.exception("[%s] Fallback: update_data() failed", plugin_id)
|
logger.exception("[%s] Fallback: update_data() failed", plugin_id)
|
||||||
|
|
||||||
@@ -964,41 +1097,41 @@ class PluginAdapter:
|
|||||||
# arrangement rather than one that has to be cropped afterwards.
|
# arrangement rather than one that has to be cropped afterwards.
|
||||||
render_width = self.resolve_render_width(plugin, plugin_id)
|
render_width = self.resolve_render_width(plugin, plugin_id)
|
||||||
if render_width != self.display_width:
|
if render_width != self.display_width:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Fallback: rendering at %dpx instead of %dpx",
|
"[%s] Fallback: rendering at %dpx instead of %dpx",
|
||||||
plugin_id, render_width, self.display_width
|
plugin_id, render_width, self.display_width
|
||||||
)
|
)
|
||||||
|
|
||||||
with self._capture(), self._render_at(render_width):
|
with self._capture(), self._render_at(render_width):
|
||||||
self.display_manager.clear()
|
self.display_manager.clear()
|
||||||
logger.info("[%s] Fallback: display cleared, calling display()", plugin_id)
|
logger.debug("[%s] Fallback: display cleared, calling display()", plugin_id)
|
||||||
|
|
||||||
# First try without force_clear (some plugins behave better this way)
|
# First try without force_clear (some plugins behave better this way)
|
||||||
try:
|
try:
|
||||||
plugin.display()
|
plugin.display()
|
||||||
logger.info("[%s] Fallback: display() called successfully", plugin_id)
|
logger.debug("[%s] Fallback: display() called successfully", plugin_id)
|
||||||
except TypeError:
|
except TypeError:
|
||||||
# Plugin may require force_clear argument
|
# Plugin may require force_clear argument
|
||||||
logger.info("[%s] Fallback: display() failed, trying with force_clear=True", plugin_id)
|
logger.debug("[%s] Fallback: display() failed, trying with force_clear=True", plugin_id)
|
||||||
plugin.display(force_clear=True)
|
plugin.display(force_clear=True)
|
||||||
|
|
||||||
# Capture the result
|
# Capture the result
|
||||||
captured = self.display_manager.image.copy()
|
captured = self.display_manager.image.copy()
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Fallback: captured frame %dx%d, mode=%s",
|
"[%s] Fallback: captured frame %dx%d, mode=%s",
|
||||||
plugin_id, captured.width, captured.height, captured.mode
|
plugin_id, captured.width, captured.height, captured.mode
|
||||||
)
|
)
|
||||||
|
|
||||||
# Check if captured image has content (not all black)
|
# Check if captured image has content (not all black)
|
||||||
is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True)
|
is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Fallback: brightness check - %.3f%% bright pixels (threshold=0.5%%)",
|
"[%s] Fallback: brightness check - %.3f%% bright pixels (threshold=0.5%%)",
|
||||||
plugin_id, bright_ratio * 100
|
plugin_id, bright_ratio * 100
|
||||||
)
|
)
|
||||||
|
|
||||||
if is_blank:
|
if is_blank:
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Fallback: first capture blank, retrying with force_clear",
|
"[%s] Fallback: first capture blank, retrying with force_clear",
|
||||||
plugin_id
|
plugin_id
|
||||||
)
|
)
|
||||||
@@ -1009,7 +1142,7 @@ class PluginAdapter:
|
|||||||
captured = self.display_manager.image.copy()
|
captured = self.display_manager.image.copy()
|
||||||
|
|
||||||
is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True)
|
is_blank, bright_ratio = self._is_blank_image(captured, return_ratio=True)
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Fallback: retry brightness - %.3f%% bright pixels",
|
"[%s] Fallback: retry brightness - %.3f%% bright pixels",
|
||||||
plugin_id, bright_ratio * 100
|
plugin_id, bright_ratio * 100
|
||||||
)
|
)
|
||||||
@@ -1026,7 +1159,7 @@ class PluginAdapter:
|
|||||||
if captured.mode != 'RGB':
|
if captured.mode != 'RGB':
|
||||||
captured = captured.convert('RGB')
|
captured = captured.convert('RGB')
|
||||||
|
|
||||||
logger.info(
|
logger.debug(
|
||||||
"[%s] Fallback: SUCCESS - captured %dx%d",
|
"[%s] Fallback: SUCCESS - captured %dx%d",
|
||||||
plugin_id, captured.width, captured.height
|
plugin_id, captured.width, captured.height
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -406,6 +406,8 @@ class StreamManager:
|
|||||||
)
|
)
|
||||||
logger.info("Ordered plugins: %s", ordered_plugins)
|
logger.info("Ordered plugins: %s", ordered_plugins)
|
||||||
|
|
||||||
|
ordered_plugins = self._apply_priority_weights(ordered_plugins)
|
||||||
|
|
||||||
# Atomically update shared state under lock to avoid races with prefetchers
|
# Atomically update shared state under lock to avoid races with prefetchers
|
||||||
with self._buffer_lock:
|
with self._buffer_lock:
|
||||||
self._ordered_plugins = ordered_plugins
|
self._ordered_plugins = ordered_plugins
|
||||||
@@ -417,6 +419,143 @@ class StreamManager:
|
|||||||
|
|
||||||
logger.info("=" * 60)
|
logger.info("=" * 60)
|
||||||
|
|
||||||
|
def _plugin_weight(self, plugin_id: str) -> int:
|
||||||
|
"""Slots per cycle for one plugin.
|
||||||
|
|
||||||
|
A plugin may answer for itself via get_vegas_priority_weight() -- the
|
||||||
|
only way favorite-team awareness can reach here, since the core can see
|
||||||
|
that a game is live but not whose. When it declines (returns None, the
|
||||||
|
default), live content earns ``live_weight`` and everything else 1.
|
||||||
|
"""
|
||||||
|
plugin = None
|
||||||
|
try:
|
||||||
|
plugin = self.plugin_manager.plugins.get(plugin_id)
|
||||||
|
except (AttributeError, TypeError):
|
||||||
|
return 1
|
||||||
|
if plugin is None:
|
||||||
|
return 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
if hasattr(plugin, 'get_vegas_priority_weight'):
|
||||||
|
declared = plugin.get_vegas_priority_weight()
|
||||||
|
if declared is not None:
|
||||||
|
return max(1, min(10, int(declared)))
|
||||||
|
except Exception:
|
||||||
|
# Deliberately falls through to the core's own live check rather
|
||||||
|
# than demoting to 1. The plugin's weight calculation is broken,
|
||||||
|
# but has_live_priority() and has_live_content() are separate
|
||||||
|
# methods guarded separately below -- a plugin that genuinely has
|
||||||
|
# a live game should still get live_weight for it.
|
||||||
|
logger.exception("[%s] get_vegas_priority_weight() failed", plugin_id)
|
||||||
|
|
||||||
|
try:
|
||||||
|
if (hasattr(plugin, 'has_live_priority')
|
||||||
|
and hasattr(plugin, 'has_live_content')
|
||||||
|
and plugin.has_live_priority()
|
||||||
|
and plugin.has_live_content()):
|
||||||
|
return self.config.live_weight
|
||||||
|
except Exception:
|
||||||
|
logger.exception("[%s] live-content check failed", plugin_id)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
def _apply_priority_weights(self, ordered: List[str]) -> List[str]:
|
||||||
|
"""Expand the rotation so weighted plugins take several turns per cycle.
|
||||||
|
|
||||||
|
Smooth Weighted Round-Robin, the same scheduler the sports plugins use
|
||||||
|
to rotate their own games: a plugin of weight N appears N times per
|
||||||
|
cycle, and the repeats are spaced through the cycle rather than
|
||||||
|
clumped, so a live score is never three-in-a-row followed by a long
|
||||||
|
silence.
|
||||||
|
|
||||||
|
Returns the input unchanged when nothing is weighted, which is both the
|
||||||
|
common case and the pre-existing behaviour.
|
||||||
|
"""
|
||||||
|
if not ordered or not self.config.live_in_ticker:
|
||||||
|
return ordered
|
||||||
|
|
||||||
|
weights = {pid: self._plugin_weight(pid) for pid in ordered}
|
||||||
|
total = sum(weights.values())
|
||||||
|
if total <= len(ordered):
|
||||||
|
return ordered # nothing boosted; plain round robin
|
||||||
|
|
||||||
|
current = {pid: 0 for pid in ordered}
|
||||||
|
schedule: List[str] = []
|
||||||
|
for _ in range(total):
|
||||||
|
for pid in ordered:
|
||||||
|
current[pid] += weights[pid]
|
||||||
|
picked = max(current, key=lambda p: current[p])
|
||||||
|
current[picked] -= total
|
||||||
|
schedule.append(picked)
|
||||||
|
|
||||||
|
schedule = self._unclump_seam(schedule)
|
||||||
|
|
||||||
|
boosted = {p: w for p, w in weights.items() if w > 1}
|
||||||
|
logger.info(
|
||||||
|
"Vegas rotation weighted: %d slots for %d plugins (boosted: %s)",
|
||||||
|
len(schedule), len(ordered), boosted)
|
||||||
|
return schedule
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _unclump_seam(schedule: List[str]) -> List[str]:
|
||||||
|
"""Stop the heaviest plugin sitting on both ends of the cycle.
|
||||||
|
|
||||||
|
Smooth Weighted Round-Robin spaces repeats well *within* a pass, but
|
||||||
|
it schedules the heaviest item first and often last too. The strip
|
||||||
|
loops, so those two are neighbours: the one place the marquee shows
|
||||||
|
the same plugin twice running is the seam between cycles.
|
||||||
|
|
||||||
|
Rotating the list cannot fix this. Rotation preserves the cyclic order
|
||||||
|
exactly, so it only moves where the seam is drawn, not the adjacency
|
||||||
|
itself. The trailing entry has to be swapped with one from the middle
|
||||||
|
whose neighbours differ from it, which breaks the pair without
|
||||||
|
creating another.
|
||||||
|
|
||||||
|
Left alone when no such position exists -- a rotation short enough or
|
||||||
|
lopsided enough to have none is one where the plugin is unavoidably
|
||||||
|
adjacent to itself anyway.
|
||||||
|
"""
|
||||||
|
if len(schedule) < 3 or schedule[0] != schedule[-1]:
|
||||||
|
return schedule
|
||||||
|
|
||||||
|
repeated = schedule[-1]
|
||||||
|
size = len(schedule)
|
||||||
|
|
||||||
|
def cyclic_doubles(seq) -> int:
|
||||||
|
return sum(1 for i in range(size) if seq[i] == seq[(i + 1) % size])
|
||||||
|
|
||||||
|
def clearance(seq, value) -> int:
|
||||||
|
"""Smallest cyclic gap between appearances of `value`."""
|
||||||
|
at = [i for i, v in enumerate(seq) if v == value]
|
||||||
|
if len(at) < 2:
|
||||||
|
return size
|
||||||
|
return min(min((b - a) % size, (a - b) % size)
|
||||||
|
for i, a in enumerate(at) for b in at[i + 1:])
|
||||||
|
|
||||||
|
# Try each swap and judge the result, rather than reasoning about which
|
||||||
|
# neighbours the two moved elements will end up with. That reasoning is
|
||||||
|
# where the first version went wrong: it guarded the slot `repeated`
|
||||||
|
# moves into but not the one the displaced element lands in, so
|
||||||
|
# ['a','b','c','d','x','y','x','a'] came back ending ['x','x'] -- the
|
||||||
|
# seam duplicate traded for a fresh one.
|
||||||
|
best = None
|
||||||
|
best_clearance = -1
|
||||||
|
for j in range(1, size - 1):
|
||||||
|
candidate = list(schedule)
|
||||||
|
candidate[j], candidate[-1] = candidate[-1], candidate[j]
|
||||||
|
if cyclic_doubles(candidate):
|
||||||
|
continue
|
||||||
|
# Among the repairs that work, prefer the one that leaves the
|
||||||
|
# boosted plugin most evenly spread; taking the first that merely
|
||||||
|
# fits moved a repeat from a gap of 7 into a gap of 2.
|
||||||
|
spread = clearance(candidate, repeated)
|
||||||
|
if spread > best_clearance:
|
||||||
|
best, best_clearance = candidate, spread
|
||||||
|
|
||||||
|
# None exists when the value is unavoidably adjacent to itself -- a
|
||||||
|
# plugin holding most of the slots has to be. Schedule it as it is
|
||||||
|
# rather than refuse.
|
||||||
|
return best if best is not None else schedule
|
||||||
|
|
||||||
def _prefetch_content(self, count: int = 1) -> None:
|
def _prefetch_content(self, count: int = 1) -> None:
|
||||||
"""
|
"""
|
||||||
Prefetch content for upcoming plugins.
|
Prefetch content for upcoming plugins.
|
||||||
|
|||||||
@@ -30,16 +30,14 @@ def success_response(
|
|||||||
"""
|
"""
|
||||||
response_data = create_success_response(data, message, metadata)
|
response_data = create_success_response(data, message, metadata)
|
||||||
|
|
||||||
# Add request metadata if available
|
# Timing is merged into whatever the caller passed, without inventing a
|
||||||
if metadata is None:
|
# metadata block for responses that have neither.
|
||||||
metadata = {}
|
enriched = dict(metadata) if metadata is not None else {}
|
||||||
|
|
||||||
# Add timing if request start time is available
|
|
||||||
if hasattr(request, 'start_time'):
|
if hasattr(request, 'start_time'):
|
||||||
metadata['response_time_ms'] = int((time.time() - request.start_time) * 1000)
|
enriched['response_time_ms'] = int((time.time() - request.start_time) * 1000)
|
||||||
|
|
||||||
if metadata:
|
if metadata is not None or enriched:
|
||||||
response_data['metadata'] = metadata
|
response_data['metadata'] = enriched
|
||||||
|
|
||||||
return jsonify(response_data)
|
return jsonify(response_data)
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ Centralized error handling for web interface.
|
|||||||
Provides helpers for consistent error responses across API endpoints.
|
Provides helpers for consistent error responses across API endpoints.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
from flask import jsonify
|
from flask import jsonify
|
||||||
|
|
||||||
@@ -16,6 +17,97 @@ from src.logging_config import get_logger
|
|||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# Credentials that turn up inside exception text. A requests error quotes the
|
||||||
|
# URL it failed on, and plugins that authenticate by query string put their key
|
||||||
|
# there, so echoing an exception verbatim can hand out an API key. Redact the
|
||||||
|
# value, keep the parameter name -- knowing *which* credential was involved is
|
||||||
|
# part of the diagnosis.
|
||||||
|
_REDACT_CREDENTIAL = re.compile(
|
||||||
|
r'((?:api[_-]?key|access[_-]?token|auth|apikey|key|passwd|password|pwd|'
|
||||||
|
r'secret|sig|signature|token)["\']?\s*[=:]\s*["\']?)([^\s&"\'<>,}]+)',
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# `Authorization: <scheme> <credential>`. The scheme name is kept because it
|
||||||
|
# says which kind of credential failed; the credential goes. Any scheme
|
||||||
|
# matches, not a fixed list: ApiKey, Negotiate, NTLM, AWS4-HMAC-SHA256 and
|
||||||
|
# whatever a plugin's API invents next are all credentials, and a list would
|
||||||
|
# silently leak the ones nobody thought of. Not covered by the generic pattern
|
||||||
|
# above, whose value part stops at whitespace and so would keep the credential
|
||||||
|
# once a space follows the scheme.
|
||||||
|
_REDACT_AUTH_HEADER = re.compile(
|
||||||
|
r'((?:proxy-)?authorization["\']?\s*[=:]\s*["\']?\s*'
|
||||||
|
r'(?:[A-Za-z][\w.+-]*[ \t]+)?)' # optional scheme name, kept
|
||||||
|
r'([^\s,"\'<>}]+)', # the credential, redacted
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Credentials embedded in a URL: https://user:password@host. requests quotes
|
||||||
|
# the full URL in its exceptions, so this is a realistic leak. The username is
|
||||||
|
# kept -- it identifies which account failed without being the secret.
|
||||||
|
_REDACT_URL_USERINFO = re.compile(r'([a-z][a-z0-9+.-]*://[^/\s:@]+:)([^/\s@]+)(@)',
|
||||||
|
re.IGNORECASE)
|
||||||
|
|
||||||
|
# Long enough for an errno string with a path, short enough not to dump a
|
||||||
|
# parser's worth of context into a JSON field.
|
||||||
|
_MAX_DETAIL_LENGTH = 400
|
||||||
|
|
||||||
|
|
||||||
|
def describe_exception(exc: BaseException,
|
||||||
|
max_length: int = _MAX_DETAIL_LENGTH) -> str:
|
||||||
|
"""
|
||||||
|
One-line, safe-to-return description of an exception.
|
||||||
|
|
||||||
|
The generic "an error occurred; see logs for details" tells a user nothing
|
||||||
|
and, when the failure is bad enough, the logs are unreachable too: a device
|
||||||
|
whose storage was failing returned that message from every endpoint
|
||||||
|
*including* the log viewer, because journalctl could not be executed. The
|
||||||
|
underlying `[Errno 5] Input/output error` named the fault immediately.
|
||||||
|
|
||||||
|
Returns "TypeName: message", credentials redacted and length capped. The
|
||||||
|
type alone is worth carrying -- a bare PermissionError says more than any
|
||||||
|
generic sentence.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
exc: The exception to describe
|
||||||
|
max_length: Truncate beyond this many characters
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A single-line description, never empty
|
||||||
|
"""
|
||||||
|
message = str(exc).strip()
|
||||||
|
text = f"{type(exc).__name__}: {message}" if message else type(exc).__name__
|
||||||
|
return redact_text(text, max_length)
|
||||||
|
|
||||||
|
|
||||||
|
def redact_text(text: str, max_length: int = _MAX_DETAIL_LENGTH) -> str:
|
||||||
|
"""Make arbitrary text safe to hand back over HTTP.
|
||||||
|
|
||||||
|
Split out of describe_exception because exceptions are not the only thing
|
||||||
|
worth returning: a subprocess's stderr, or a message a helper script
|
||||||
|
printed, is just as useful to a user and just as capable of carrying a
|
||||||
|
token or a password in it.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: The text to redact
|
||||||
|
max_length: Truncate beyond this many characters
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A single line, credentials replaced, length capped.
|
||||||
|
"""
|
||||||
|
text = text or ''
|
||||||
|
# Order matters: the URL and header forms are more specific than the
|
||||||
|
# generic key=value pattern, which would otherwise chew the scheme.
|
||||||
|
text = _REDACT_URL_USERINFO.sub(r'\1<redacted>\3', text)
|
||||||
|
text = _REDACT_AUTH_HEADER.sub(r'\1<redacted>', text)
|
||||||
|
text = _REDACT_CREDENTIAL.sub(r'\1<redacted>', text)
|
||||||
|
# Collapse newlines/tabs so the detail stays one line in a JSON field.
|
||||||
|
text = ' '.join(text.split())
|
||||||
|
if len(text) > max_length:
|
||||||
|
text = text[:max_length - 1].rstrip() + '…'
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
def create_error_response(
|
def create_error_response(
|
||||||
error_code: ErrorCode,
|
error_code: ErrorCode,
|
||||||
message: str,
|
message: str,
|
||||||
@@ -69,13 +161,16 @@ def create_success_response(
|
|||||||
"status": "success"
|
"status": "success"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# All three use `is not None` rather than truthiness: "" and {} are
|
||||||
|
# values a caller chose to send, and dropping them silently would make
|
||||||
|
# the response shape depend on the data.
|
||||||
if data is not None:
|
if data is not None:
|
||||||
response["data"] = data
|
response["data"] = data
|
||||||
|
|
||||||
if message:
|
if message is not None:
|
||||||
response["message"] = message
|
response["message"] = message
|
||||||
|
|
||||||
if metadata:
|
if metadata is not None:
|
||||||
response["metadata"] = metadata
|
response["metadata"] = metadata
|
||||||
|
|
||||||
return response
|
return response
|
||||||
|
|||||||
@@ -89,7 +89,11 @@ class WebInterfaceError:
|
|||||||
self.category = category or self._infer_category(error_code)
|
self.category = category or self._infer_category(error_code)
|
||||||
self.details = details
|
self.details = details
|
||||||
self.context = context or {}
|
self.context = context or {}
|
||||||
self.suggested_fixes = suggested_fixes or self._get_default_suggestions(error_code)
|
# `is None`, not truthiness: an explicit [] means "this caller has
|
||||||
|
# no suggestions to offer", which the default list would override.
|
||||||
|
self.suggested_fixes = (
|
||||||
|
suggested_fixes if suggested_fixes is not None
|
||||||
|
else self._get_default_suggestions(error_code))
|
||||||
self.original_error = original_error
|
self.original_error = original_error
|
||||||
|
|
||||||
def _infer_category(self, error_code: ErrorCode) -> ErrorCategory:
|
def _infer_category(self, error_code: ErrorCode) -> ErrorCategory:
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ Provides functions for identifying, masking, separating, and filtering
|
|||||||
secret fields in plugin configurations based on JSON Schema x-secret markers.
|
secret fields in plugin configurations based on JSON Schema x-secret markers.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Any, Dict, Set, Tuple
|
from typing import Any, Dict, Optional, Set, Tuple
|
||||||
|
|
||||||
|
|
||||||
def find_secret_fields(properties: Dict[str, Any], prefix: str = '') -> Set[str]:
|
def find_secret_fields(properties: Dict[str, Any], prefix: str = '') -> Set[str]:
|
||||||
@@ -143,6 +143,12 @@ def mask_secret_fields(config: Dict[str, Any], schema_properties: Dict[str, Any]
|
|||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
#: What a masked secret looks like on the wire. Named because the write path
|
||||||
|
#: has to recognise it coming back: a client that renders the mask and posts
|
||||||
|
#: it unchanged must not store the mask as if it were the secret.
|
||||||
|
SECRET_MASK = '\u2022' * 8
|
||||||
|
|
||||||
|
|
||||||
def mask_all_secret_values(config: Dict[str, Any]) -> Dict[str, Any]:
|
def mask_all_secret_values(config: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
"""Blanket-mask every non-empty value in a secrets config dict.
|
"""Blanket-mask every non-empty value in a secrets config dict.
|
||||||
|
|
||||||
@@ -156,15 +162,25 @@ def mask_all_secret_values(config: Dict[str, Any]) -> Dict[str, Any]:
|
|||||||
Returns:
|
Returns:
|
||||||
A copy with all real values replaced by ``'••••••••'``.
|
A copy with all real values replaced by ``'••••••••'``.
|
||||||
"""
|
"""
|
||||||
masked: Dict[str, Any] = {}
|
return {k: _mask_value(v) for k, v in config.items()}
|
||||||
for k, v in config.items():
|
|
||||||
if isinstance(v, dict):
|
|
||||||
masked[k] = mask_all_secret_values(v)
|
def _mask_value(value: Any) -> Any:
|
||||||
elif v not in (None, '') and not (isinstance(v, str) and v.startswith('YOUR_')):
|
"""Mask one value, recursing through dicts and lists.
|
||||||
masked[k] = '••••••••'
|
|
||||||
else:
|
A list used to be masked as though it were a scalar, so
|
||||||
masked[k] = v
|
``accounts: [{"name": "a", "token": "..."}]`` came back as a single
|
||||||
return masked
|
``'••••••••'``. Nothing leaked, but the caller could no longer see how
|
||||||
|
many entries there were or any of their non-secret fields, and the raw
|
||||||
|
editor was shown a string where the file holds an array.
|
||||||
|
"""
|
||||||
|
if isinstance(value, dict):
|
||||||
|
return {k: _mask_value(v) for k, v in value.items()}
|
||||||
|
if isinstance(value, list):
|
||||||
|
return [_mask_value(item) for item in value]
|
||||||
|
if value in (None, '') or (isinstance(value, str) and value.startswith('YOUR_')):
|
||||||
|
return value
|
||||||
|
return SECRET_MASK
|
||||||
|
|
||||||
|
|
||||||
def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]:
|
def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
@@ -186,6 +202,133 @@ def remove_empty_secrets(secrets: Dict[str, Any]) -> Dict[str, Any]:
|
|||||||
nested = remove_empty_secrets(v)
|
nested = remove_empty_secrets(v)
|
||||||
if nested:
|
if nested:
|
||||||
result[k] = nested
|
result[k] = nested
|
||||||
|
elif isinstance(v, list):
|
||||||
|
# Lists used to fall through to the scalar branch below and be
|
||||||
|
# kept verbatim, blanks and all. Because lists merge by
|
||||||
|
# *replacement*, saving any unrelated setting then wrote
|
||||||
|
# [{"token": ""}, ...] straight over the stored list and
|
||||||
|
# destroyed every credential in it.
|
||||||
|
pruned = _prune_secret_list(v)
|
||||||
|
if pruned is not None:
|
||||||
|
result[k] = pruned
|
||||||
elif v is not None and not (isinstance(v, str) and v.strip() == ''):
|
elif v is not None and not (isinstance(v, str) and v.strip() == ''):
|
||||||
result[k] = v
|
result[k] = v
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _prune_secret_list(items: list) -> Optional[list]:
|
||||||
|
"""Strip blanks from inside a list of secrets, preserving every index.
|
||||||
|
|
||||||
|
The rest of the system treats a secrets list as *parallel* to the regular
|
||||||
|
one -- ``sec[i]`` holds the secret fields of item ``i``, and ``{}`` means
|
||||||
|
"item i has none" (see ConfigManager._strip_secrets_recursive). So an
|
||||||
|
emptied dict item stays ``{}``: putting ``None`` there makes that list stop
|
||||||
|
looking parallel, and the stripper then drops the whole key from the main
|
||||||
|
config, taking the non-secret fields with it.
|
||||||
|
|
||||||
|
A blank *scalar* becomes ``None``, meaning "no update at this index" --
|
||||||
|
:func:`merge_secrets` substitutes whatever is stored there. Returns
|
||||||
|
``None`` when nothing in the list carries a real value, so the caller drops
|
||||||
|
the key and leaves the stored list untouched.
|
||||||
|
"""
|
||||||
|
pruned: list = []
|
||||||
|
has_real_value = False
|
||||||
|
for item in items:
|
||||||
|
if isinstance(item, dict):
|
||||||
|
kept = remove_empty_secrets(item)
|
||||||
|
pruned.append(kept)
|
||||||
|
has_real_value = has_real_value or bool(kept)
|
||||||
|
elif isinstance(item, list):
|
||||||
|
sub = _prune_secret_list(item)
|
||||||
|
pruned.append(sub if sub is not None else [])
|
||||||
|
has_real_value = has_real_value or sub is not None
|
||||||
|
elif item is not None and not (isinstance(item, str) and item.strip() == ''):
|
||||||
|
pruned.append(item)
|
||||||
|
has_real_value = True
|
||||||
|
else:
|
||||||
|
pruned.append(None)
|
||||||
|
return pruned if has_real_value else None
|
||||||
|
|
||||||
|
|
||||||
|
def merge_secrets(stored: Any, incoming: Any) -> Any:
|
||||||
|
"""Merge submitted secrets over stored ones, element-wise inside lists.
|
||||||
|
|
||||||
|
``deep_merge`` replaces a list wholesale. For secrets that is destructive:
|
||||||
|
an incoming list that carries a real value for one entry and ``None`` for
|
||||||
|
the rest would drop the stored credentials of every other entry. Here a
|
||||||
|
list merges by index, and ``None`` means "keep what is stored".
|
||||||
|
|
||||||
|
Entries are matched by *position*, which is what the config form gives us
|
||||||
|
-- there is no schema-declared identity to key on, and it is the same
|
||||||
|
contract ConfigManager._strip_secrets_recursive already relies on. The
|
||||||
|
incoming list's length wins, so deleting an item deletes its secrets;
|
||||||
|
an item the client left blank keeps whatever is stored at that index.
|
||||||
|
"""
|
||||||
|
if isinstance(stored, dict) and isinstance(incoming, dict):
|
||||||
|
merged = dict(stored)
|
||||||
|
for key, value in incoming.items():
|
||||||
|
merged[key] = (merge_secrets(stored[key], value)
|
||||||
|
if key in stored else value)
|
||||||
|
return merged
|
||||||
|
if isinstance(stored, list) and isinstance(incoming, list):
|
||||||
|
# The incoming list sets the length -- the regular config's list is
|
||||||
|
# authoritative about how many items exist, and this one runs parallel
|
||||||
|
# to it. Removing an entry must therefore remove its secrets too.
|
||||||
|
merged_list = []
|
||||||
|
for index, item in enumerate(incoming):
|
||||||
|
stored_item = stored[index] if index < len(stored) else None
|
||||||
|
merged_list.append(stored_item if item is None
|
||||||
|
else merge_secrets(stored_item, item))
|
||||||
|
return merged_list
|
||||||
|
if incoming is None:
|
||||||
|
return stored
|
||||||
|
return incoming
|
||||||
|
|
||||||
|
|
||||||
|
def strip_masked_values(secrets: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
|
"""Remove values a client echoed back rather than changed.
|
||||||
|
|
||||||
|
The counterpart to :func:`mask_all_secret_values`. A client that GETs the
|
||||||
|
masked secrets, edits one field and POSTs the whole object back is sending
|
||||||
|
``SECRET_MASK`` for every field it did not touch. Storing those would
|
||||||
|
replace each untouched credential with eight bullet characters.
|
||||||
|
|
||||||
|
Drops the mask and, like :func:`remove_empty_secrets`, blank values -- so
|
||||||
|
the caller can merge the result onto what is already stored and have
|
||||||
|
"unchanged" mean unchanged. Empty nested dicts are pruned.
|
||||||
|
"""
|
||||||
|
result: Dict[str, Any] = {}
|
||||||
|
for k, v in secrets.items():
|
||||||
|
if isinstance(v, dict):
|
||||||
|
nested = strip_masked_values(v)
|
||||||
|
if nested:
|
||||||
|
result[k] = nested
|
||||||
|
elif isinstance(v, list):
|
||||||
|
# A list is merged by replacement, not element by element -- there
|
||||||
|
# is no identity to match entries on -- so a list that still holds
|
||||||
|
# a mask cannot be merged safely: keeping it would store bullets,
|
||||||
|
# and keeping the submitted entries alone would drop whichever the
|
||||||
|
# client did not send back. Dropping the key leaves the stored
|
||||||
|
# list untouched, which is what an untouched list should do.
|
||||||
|
#
|
||||||
|
# The consequence, deliberately: editing one secret inside a list
|
||||||
|
# through this endpoint requires sending real values for all of
|
||||||
|
# them. Sending some masks leaves the whole list as it was.
|
||||||
|
if not _contains_mask(v):
|
||||||
|
result[k] = v
|
||||||
|
elif v is None:
|
||||||
|
continue
|
||||||
|
elif isinstance(v, str) and (v.strip() == '' or v == SECRET_MASK):
|
||||||
|
continue
|
||||||
|
else:
|
||||||
|
result[k] = v
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _contains_mask(value: Any) -> bool:
|
||||||
|
"""True when a mask sentinel survives anywhere inside ``value``."""
|
||||||
|
if isinstance(value, dict):
|
||||||
|
return any(_contains_mask(v) for v in value.values())
|
||||||
|
if isinstance(value, list):
|
||||||
|
return any(_contains_mask(item) for item in value)
|
||||||
|
return value == SECRET_MASK
|
||||||
|
|||||||
@@ -43,10 +43,15 @@ def validate_image_url(url: str) -> Tuple[bool, Optional[str]]:
|
|||||||
if any(handler in url_lower for handler in ['onerror=', 'onload=', 'onclick=']):
|
if any(handler in url_lower for handler in ['onerror=', 'onload=', 'onclick=']):
|
||||||
return False, "Event handlers not allowed in URLs"
|
return False, "Event handlers not allowed in URLs"
|
||||||
|
|
||||||
|
# Reject directory traversal anywhere, not only in relative paths:
|
||||||
|
# http://host/../secret is as much a traversal attempt as /../secret.
|
||||||
|
if '..' in url:
|
||||||
|
return False, "Invalid path: directory traversal not allowed"
|
||||||
|
|
||||||
# Allow relative paths starting with /
|
# Allow relative paths starting with /
|
||||||
if url.startswith('/'):
|
if url.startswith('/'):
|
||||||
# Validate it's a safe relative path (no directory traversal)
|
# // would be a protocol-relative URL, not a local path
|
||||||
if '..' in url or url.startswith('//'):
|
if url.startswith('//'):
|
||||||
return False, "Invalid relative path"
|
return False, "Invalid relative path"
|
||||||
return True, None
|
return True, None
|
||||||
|
|
||||||
@@ -104,10 +109,11 @@ def validate_file_upload(filename: str, max_size_mb: int = 10,
|
|||||||
if '..' in filename or '/' in filename or '\\' in filename:
|
if '..' in filename or '/' in filename or '\\' in filename:
|
||||||
return False, "Filename contains invalid characters"
|
return False, "Filename contains invalid characters"
|
||||||
|
|
||||||
# Check extension if specified
|
# Check extension if specified. Both sides are lowercased: the caller's
|
||||||
|
# list is as likely to hold '.TTF' as the filename is.
|
||||||
if allowed_extensions:
|
if allowed_extensions:
|
||||||
file_ext = Path(filename).suffix.lower()
|
file_ext = Path(filename).suffix.lower()
|
||||||
if file_ext not in allowed_extensions:
|
if file_ext not in [ext.lower() for ext in allowed_extensions]:
|
||||||
return False, f"File extension must be one of: {', '.join(allowed_extensions)}"
|
return False, f"File extension must be one of: {', '.join(allowed_extensions)}"
|
||||||
|
|
||||||
return True, None
|
return True, None
|
||||||
@@ -147,7 +153,8 @@ def validate_numeric_range(value: float, min_val: Optional[float] = None,
|
|||||||
Returns:
|
Returns:
|
||||||
Tuple of (is_valid, error_message)
|
Tuple of (is_valid, error_message)
|
||||||
"""
|
"""
|
||||||
if not isinstance(value, (int, float)):
|
# bool is an int subclass, so True would otherwise validate as 1.
|
||||||
|
if not isinstance(value, (int, float)) or isinstance(value, bool):
|
||||||
return False, "Value must be a number"
|
return False, "Value must be a number"
|
||||||
|
|
||||||
if min_val is not None and value < min_val:
|
if min_val is not None and value < min_val:
|
||||||
@@ -183,7 +190,15 @@ def validate_string_length(text: str, min_length: Optional[int] = None,
|
|||||||
|
|
||||||
def sanitize_plugin_config(config: dict) -> dict:
|
def sanitize_plugin_config(config: dict) -> dict:
|
||||||
"""
|
"""
|
||||||
Sanitize plugin configuration input to prevent injection.
|
Restrict a plugin config to safe key names and value types.
|
||||||
|
|
||||||
|
Drops keys that are not plain identifiers and values that are not
|
||||||
|
JSON-ish scalars, lists, or dicts, recursing into the latter two.
|
||||||
|
|
||||||
|
String values are returned **unescaped**: output escaping is the
|
||||||
|
template layer's job, and escaping here would store the escaped form
|
||||||
|
in config.json. Do not read this function as XSS protection for
|
||||||
|
rendered output.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
config: Configuration dictionary
|
config: Configuration dictionary
|
||||||
|
|||||||
Regular → Executable
Regular → Executable
@@ -10,8 +10,8 @@ WorkingDirectory=__PROJECT_ROOT_DIR__
|
|||||||
ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/scripts/utils/wifi_monitor_daemon.py --interval 30
|
ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/scripts/utils/wifi_monitor_daemon.py --interval 30
|
||||||
Restart=on-failure
|
Restart=on-failure
|
||||||
RestartSec=10
|
RestartSec=10
|
||||||
StandardOutput=syslog
|
StandardOutput=journal
|
||||||
StandardError=syslog
|
StandardError=journal
|
||||||
SyslogIdentifier=ledmatrix-wifi-monitor
|
SyslogIdentifier=ledmatrix-wifi-monitor
|
||||||
|
|
||||||
[Install]
|
[Install]
|
||||||
|
|||||||
@@ -8,9 +8,38 @@ Type=simple
|
|||||||
User=root
|
User=root
|
||||||
WorkingDirectory=__PROJECT_ROOT_DIR__
|
WorkingDirectory=__PROJECT_ROOT_DIR__
|
||||||
Environment=PYTHONDONTWRITEBYTECODE=1
|
Environment=PYTHONDONTWRITEBYTECODE=1
|
||||||
|
# glibc gives each allocating thread its own malloc arena, up to 8 x CPU count,
|
||||||
|
# and an arena that has grown is never handed back to the OS. This process runs
|
||||||
|
# 9 threads on a 3-core Pi, so the ceiling is 24 arenas -- and a rig measured at
|
||||||
|
# 1030 MB resident held 23 large anonymous mappings on 64 MB-aligned addresses,
|
||||||
|
# 920 MB of them, while the live data it was actually holding (widest scroll
|
||||||
|
# strip seen: 35,746 x 64) accounts for roughly 15 MB. That gap is arena bloat,
|
||||||
|
# not leaked objects: RSS was flat across repeated sampling, not climbing.
|
||||||
|
#
|
||||||
|
# Capping the arenas trades a little allocator concurrency for a large amount of
|
||||||
|
# resident memory on a device that has neither to spare. 2 is the usual value;
|
||||||
|
# raise it if frame times regress.
|
||||||
|
Environment=MALLOC_ARENA_MAX=2
|
||||||
ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py
|
ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py
|
||||||
Restart=on-failure
|
# Restart=always, not on-failure: run.py exiting 0 (a clean shutdown path taken
|
||||||
|
# for a reason that no longer applies, e.g. a config reload) would otherwise leave
|
||||||
|
# the service stopped and the panel dark indefinitely, with systemd considering
|
||||||
|
# that a successful outcome and never bringing it back.
|
||||||
|
Restart=always
|
||||||
RestartSec=10
|
RestartSec=10
|
||||||
|
# Memory ceiling as a share of physical RAM, so one unit file suits a 512 MB
|
||||||
|
# Pi Zero 2 W and an 8 GB Pi 5 alike. This is a backstop, not a tuning knob: it
|
||||||
|
# turns "the board runs out of memory, stops being able to fork, and takes sshd
|
||||||
|
# and the panel down together until someone pulls the plug" into "this one
|
||||||
|
# service restarts".
|
||||||
|
#
|
||||||
|
# NOTE: Raspberry Pi firmware boots the kernel with cgroup_disable=memory, and
|
||||||
|
# systemd accepts this setting and then silently ignores it. Verify with:
|
||||||
|
# grep memory /sys/fs/cgroup/cgroup.controllers
|
||||||
|
# If that prints nothing, add "cgroup_enable=memory cgroup_memory=1" to
|
||||||
|
# /boot/firmware/cmdline.txt (all on line 1) and reboot. first_time_install.sh
|
||||||
|
# does this for you.
|
||||||
|
MemoryMax=85%
|
||||||
StandardOutput=journal
|
StandardOutput=journal
|
||||||
StandardError=journal
|
StandardError=journal
|
||||||
SyslogIdentifier=ledmatrix
|
SyslogIdentifier=ledmatrix
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
"""
|
||||||
|
Shared scaffolding for api_v3 blueprint tests.
|
||||||
|
|
||||||
|
Not a test module (the leading underscore keeps pytest from collecting
|
||||||
|
it). It is the pytest-fixture equivalent of ``_make_client()`` in
|
||||||
|
test_uninstall_and_reconcile_endpoint.py, which is unittest-style and
|
||||||
|
requires ``self.addCleanup``.
|
||||||
|
|
||||||
|
The api_v3 blueprint keeps its managers as attributes on a module-level
|
||||||
|
singleton, not in Flask app state, so replacing them with mocks leaks
|
||||||
|
into every later test that imports api_v3 unless the originals are put
|
||||||
|
back. ``api_v3_client`` snapshots and restores them around each test.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from flask import Flask
|
||||||
|
|
||||||
|
|
||||||
|
# Every manager attribute the blueprint reads. Anything missing here keeps
|
||||||
|
# whatever a previously-run test left on the singleton.
|
||||||
|
API_V3_MANAGER_ATTRS = (
|
||||||
|
'config_manager', 'plugin_manager', 'plugin_store_manager',
|
||||||
|
'plugin_state_manager', 'saved_repositories_manager', 'schema_manager',
|
||||||
|
'operation_queue', 'operation_history', 'cache_manager',
|
||||||
|
)
|
||||||
|
|
||||||
|
_SENTINEL = object()
|
||||||
|
|
||||||
|
|
||||||
|
def build_app(blueprint):
|
||||||
|
app = Flask(__name__)
|
||||||
|
app.config['TESTING'] = True
|
||||||
|
app.config['SECRET_KEY'] = 'test'
|
||||||
|
app.register_blueprint(blueprint, url_prefix='/api/v3')
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def api_v3_module():
|
||||||
|
"""The api_v3 module with every manager replaced by a MagicMock.
|
||||||
|
|
||||||
|
Restores the original attributes afterwards. Tests point individual
|
||||||
|
managers at real objects (a ConfigManager over tmp_path, say) or set
|
||||||
|
them to None to exercise the not-initialized branches.
|
||||||
|
"""
|
||||||
|
from web_interface.blueprints import api_v3 as module
|
||||||
|
|
||||||
|
originals = {
|
||||||
|
name: getattr(module.api_v3, name, _SENTINEL)
|
||||||
|
for name in API_V3_MANAGER_ATTRS
|
||||||
|
}
|
||||||
|
for name in API_V3_MANAGER_ATTRS:
|
||||||
|
setattr(module.api_v3, name, MagicMock())
|
||||||
|
# Default to the direct path; queue tests opt in explicitly.
|
||||||
|
module.api_v3.operation_queue = None
|
||||||
|
|
||||||
|
yield module
|
||||||
|
|
||||||
|
for name, original in originals.items():
|
||||||
|
if original is _SENTINEL:
|
||||||
|
if hasattr(module.api_v3, name):
|
||||||
|
try:
|
||||||
|
delattr(module.api_v3, name)
|
||||||
|
except AttributeError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
setattr(module.api_v3, name, original)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def api_v3_client(api_v3_module):
|
||||||
|
"""Flask test client wired to the mocked blueprint."""
|
||||||
|
return build_app(api_v3_module.api_v3).test_client()
|
||||||
@@ -0,0 +1,275 @@
|
|||||||
|
"""
|
||||||
|
Tests for src/common/api_helper.py (APIHelper).
|
||||||
|
|
||||||
|
Covers rate limiting, cached GETs, ESPN URL/cache-key construction,
|
||||||
|
session header defaults and per-call merging, the retry adapter, and the
|
||||||
|
fixed clear_cache() behavior (real CacheManager surface: clear_cache /
|
||||||
|
delete / list_cache_files, with safe no-ops elsewhere).
|
||||||
|
|
||||||
|
No real network: helper.session.get/post are always replaced with mocks.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import types
|
||||||
|
from unittest.mock import MagicMock, Mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import requests
|
||||||
|
from freezegun import freeze_time
|
||||||
|
|
||||||
|
import src.common.api_helper as api_helper_module
|
||||||
|
from src.common.api_helper import APIHelper
|
||||||
|
|
||||||
|
|
||||||
|
def _make_response(payload):
|
||||||
|
response = MagicMock()
|
||||||
|
response.json.return_value = payload
|
||||||
|
response.raise_for_status.return_value = None
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cache():
|
||||||
|
cache = MagicMock()
|
||||||
|
cache.get.return_value = None
|
||||||
|
return cache
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def helper(cache):
|
||||||
|
helper = APIHelper(cache_manager=cache)
|
||||||
|
# Default min interval is 1.0s and would really sleep between requests.
|
||||||
|
helper.set_rate_limit(0)
|
||||||
|
return helper
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Rate limiting
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestRateLimiting:
|
||||||
|
def test_sleeps_for_remaining_interval(self, helper, monkeypatch):
|
||||||
|
fake_time = MagicMock()
|
||||||
|
fake_time.time.side_effect = [102.0, 105.0]
|
||||||
|
monkeypatch.setattr(api_helper_module, 'time', fake_time)
|
||||||
|
|
||||||
|
helper.set_rate_limit(5)
|
||||||
|
helper._last_request_time = 100.0
|
||||||
|
helper._enforce_rate_limit()
|
||||||
|
|
||||||
|
# 2s elapsed of a 5s interval -> sleep the remaining 3s.
|
||||||
|
fake_time.sleep.assert_called_once()
|
||||||
|
assert fake_time.sleep.call_args[0][0] == pytest.approx(3.0)
|
||||||
|
assert helper._last_request_time == 105.0
|
||||||
|
|
||||||
|
def test_no_sleep_when_interval_elapsed(self, helper, monkeypatch):
|
||||||
|
fake_time = MagicMock()
|
||||||
|
fake_time.time.side_effect = [200.0, 201.0]
|
||||||
|
monkeypatch.setattr(api_helper_module, 'time', fake_time)
|
||||||
|
|
||||||
|
helper.set_rate_limit(5)
|
||||||
|
helper._last_request_time = 100.0
|
||||||
|
helper._enforce_rate_limit()
|
||||||
|
|
||||||
|
fake_time.sleep.assert_not_called()
|
||||||
|
assert helper._last_request_time == 201.0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# get()
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestGet:
|
||||||
|
def test_cache_hit_skips_request_and_rate_limit(self, helper, cache):
|
||||||
|
cache.get.return_value = {'cached': True}
|
||||||
|
helper.session.get = Mock()
|
||||||
|
rate_spy = Mock()
|
||||||
|
helper._enforce_rate_limit = rate_spy
|
||||||
|
|
||||||
|
result = helper.get('https://example.com/api', cache_key='k')
|
||||||
|
|
||||||
|
assert result == {'cached': True}
|
||||||
|
helper.session.get.assert_not_called()
|
||||||
|
rate_spy.assert_not_called()
|
||||||
|
|
||||||
|
def test_cache_miss_fetches_and_caches_without_ttl(self, helper, cache):
|
||||||
|
cache.get.return_value = None
|
||||||
|
helper.session.get = Mock(return_value=_make_response({'a': 1}))
|
||||||
|
|
||||||
|
result = helper.get('https://example.com/api', cache_key='k',
|
||||||
|
cache_ttl=999)
|
||||||
|
|
||||||
|
assert result == {'a': 1}
|
||||||
|
# Pin the ttl-dropped contract: CacheManager.set is called with
|
||||||
|
# (key, data) only — the cache_ttl argument is discarded.
|
||||||
|
cache.set.assert_called_once_with('k', {'a': 1})
|
||||||
|
|
||||||
|
def test_request_exception_returns_none_and_caches_nothing(
|
||||||
|
self, helper, cache):
|
||||||
|
helper.session.get = Mock(
|
||||||
|
side_effect=requests.exceptions.RequestException('boom'))
|
||||||
|
|
||||||
|
result = helper.get('https://example.com/api', cache_key='k')
|
||||||
|
|
||||||
|
assert result is None
|
||||||
|
cache.set.assert_not_called()
|
||||||
|
|
||||||
|
def test_timeout_zero_falls_back_to_default(self, helper):
|
||||||
|
# Quirk pin: `timeout or self.default_timeout` treats an explicit
|
||||||
|
# timeout=0 as falsy, so the default (30) is used instead.
|
||||||
|
helper.session.get = Mock(return_value=_make_response({}))
|
||||||
|
|
||||||
|
helper.get('https://example.com/api', timeout=0)
|
||||||
|
|
||||||
|
assert helper.session.get.call_args.kwargs['timeout'] == 30
|
||||||
|
|
||||||
|
def test_per_call_headers_merge_over_session_headers(self, helper):
|
||||||
|
helper.session.get = Mock(return_value=_make_response({}))
|
||||||
|
|
||||||
|
helper.get('https://example.com/api', headers={'X-Custom': 'yes'})
|
||||||
|
|
||||||
|
sent = helper.session.get.call_args.kwargs['headers']
|
||||||
|
# Merged, not replaced: session defaults survive alongside the
|
||||||
|
# per-call header.
|
||||||
|
assert sent['X-Custom'] == 'yes'
|
||||||
|
assert sent['User-Agent'] == (
|
||||||
|
'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)')
|
||||||
|
assert sent['Accept'] == 'application/json'
|
||||||
|
# The session's own headers are not polluted by the per-call ones.
|
||||||
|
assert 'X-Custom' not in helper.session.headers
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ESPN helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestEspnHelpers:
|
||||||
|
@freeze_time('2026-08-07')
|
||||||
|
def test_fetch_espn_scoreboard_url_params_and_cache_key(self, helper):
|
||||||
|
helper.get = Mock(return_value={'ok': 1})
|
||||||
|
|
||||||
|
result = helper.fetch_espn_scoreboard('football', 'nfl')
|
||||||
|
|
||||||
|
assert result == {'ok': 1}
|
||||||
|
helper.get.assert_called_once_with(
|
||||||
|
'https://site.api.espn.com/apis/site/v2/sports/football/nfl/scoreboard',
|
||||||
|
params={'dates': '20260807', 'limit': 1000},
|
||||||
|
cache_key='espn_football_nfl_20260807',
|
||||||
|
cache_ttl=300,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_fetch_espn_scoreboard_explicit_date(self, helper):
|
||||||
|
helper.get = Mock(return_value=None)
|
||||||
|
|
||||||
|
helper.fetch_espn_scoreboard('basketball', 'nba', date='20250115')
|
||||||
|
|
||||||
|
kwargs = helper.get.call_args.kwargs
|
||||||
|
assert kwargs['params'] == {'dates': '20250115', 'limit': 1000}
|
||||||
|
assert kwargs['cache_key'] == 'espn_basketball_nba_20250115'
|
||||||
|
|
||||||
|
def test_fetch_espn_standings_url_and_cache_key(self, helper):
|
||||||
|
helper.get = Mock(return_value={'ok': 1})
|
||||||
|
|
||||||
|
helper.fetch_espn_standings('football', 'nfl')
|
||||||
|
|
||||||
|
helper.get.assert_called_once_with(
|
||||||
|
'https://site.api.espn.com/apis/site/v2/sports/football/nfl/standings',
|
||||||
|
cache_key='espn_standings_football_nfl',
|
||||||
|
cache_ttl=3600,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_fetch_espn_rankings_url_and_cache_key(self, helper):
|
||||||
|
helper.get = Mock(return_value={'ok': 1})
|
||||||
|
|
||||||
|
helper.fetch_espn_rankings('football', 'college-football')
|
||||||
|
|
||||||
|
helper.get.assert_called_once_with(
|
||||||
|
'https://site.api.espn.com/apis/site/v2/sports/football/college-football/rankings',
|
||||||
|
cache_key='espn_rankings_football_college-football',
|
||||||
|
cache_ttl=3600,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Session setup
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestSessionSetup:
|
||||||
|
def test_user_agent_exact(self, helper):
|
||||||
|
# Regression guard: ESPN began 403ing other user agents; this exact
|
||||||
|
# string must be sent on every request.
|
||||||
|
assert helper.session.headers['User-Agent'] == (
|
||||||
|
'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)')
|
||||||
|
|
||||||
|
def test_retry_adapter_configuration(self):
|
||||||
|
helper = APIHelper(cache_manager=None, max_retries=7)
|
||||||
|
|
||||||
|
retries = helper.session.get_adapter('https://x').max_retries
|
||||||
|
assert retries.total == 7
|
||||||
|
assert {429, 500, 502, 503, 504} <= set(retries.status_forcelist)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# clear_cache (fixed behavior: real CacheManager surface)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestClearCache:
|
||||||
|
def test_no_pattern_uses_clear_cache_method(self):
|
||||||
|
manager = types.SimpleNamespace(clear_cache=Mock())
|
||||||
|
helper = APIHelper(cache_manager=manager)
|
||||||
|
helper.set_rate_limit(0)
|
||||||
|
|
||||||
|
helper.clear_cache()
|
||||||
|
|
||||||
|
manager.clear_cache.assert_called_once_with()
|
||||||
|
|
||||||
|
def test_no_pattern_falls_back_to_clear(self):
|
||||||
|
manager = types.SimpleNamespace(clear=Mock())
|
||||||
|
helper = APIHelper(cache_manager=manager)
|
||||||
|
helper.set_rate_limit(0)
|
||||||
|
|
||||||
|
helper.clear_cache()
|
||||||
|
|
||||||
|
manager.clear.assert_called_once_with()
|
||||||
|
|
||||||
|
def test_no_pattern_manager_without_any_clear_is_noop(self):
|
||||||
|
helper = APIHelper(cache_manager=object())
|
||||||
|
helper.set_rate_limit(0)
|
||||||
|
|
||||||
|
helper.clear_cache() # must not raise
|
||||||
|
|
||||||
|
def test_pattern_deletes_only_matching_keys(self):
|
||||||
|
manager = types.SimpleNamespace(
|
||||||
|
list_cache_files=Mock(return_value=[
|
||||||
|
{'key': 'espn_nfl_x'},
|
||||||
|
{'key': 'other'},
|
||||||
|
]),
|
||||||
|
delete=Mock(),
|
||||||
|
)
|
||||||
|
helper = APIHelper(cache_manager=manager)
|
||||||
|
helper.set_rate_limit(0)
|
||||||
|
|
||||||
|
helper.clear_cache(pattern='espn')
|
||||||
|
|
||||||
|
manager.delete.assert_called_once_with('espn_nfl_x')
|
||||||
|
|
||||||
|
def test_pattern_manager_without_list_cache_files_is_noop(self):
|
||||||
|
helper = APIHelper(cache_manager=object())
|
||||||
|
helper.set_rate_limit(0)
|
||||||
|
|
||||||
|
helper.clear_cache(pattern='espn') # must not raise
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# No cache manager
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestNoCacheManager:
|
||||||
|
def test_all_cache_operations_safe_without_manager(self):
|
||||||
|
helper = APIHelper(cache_manager=None)
|
||||||
|
helper.set_rate_limit(0)
|
||||||
|
|
||||||
|
assert helper.get_cache('k') is None
|
||||||
|
assert helper._get_from_cache('k') is None
|
||||||
|
assert helper.set_cache('k', {'a': 1}) is None
|
||||||
|
assert helper.clear_cache() is None
|
||||||
|
assert helper.clear_cache(pattern='espn') is None
|
||||||
@@ -0,0 +1,226 @@
|
|||||||
|
"""
|
||||||
|
Endpoint tests for POST /plugins/calendar/upload-credentials.
|
||||||
|
|
||||||
|
The endpoint takes an uploaded Google OAuth credentials file, writes it
|
||||||
|
into the calendar plugin's directory as credentials.json at mode 0600, and
|
||||||
|
copies any previous file aside first. It had no tests.
|
||||||
|
|
||||||
|
Regression coverage for two fixed bugs:
|
||||||
|
- The OAuth-shape check sat inside `except Exception: pass`, so a valid
|
||||||
|
JSON document that is not an object — a bare `42`, a list, a string —
|
||||||
|
raised TypeError on the membership test, was swallowed, and got saved
|
||||||
|
as credentials.json anyway.
|
||||||
|
- Each overwrite created a timestamped backup and nothing ever removed
|
||||||
|
them, so every re-upload left another complete copy of the user's OAuth
|
||||||
|
client credentials in the plugin directory, indefinitely.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
|
||||||
|
|
||||||
|
URL = "/api/v3/plugins/calendar/upload-credentials"
|
||||||
|
|
||||||
|
VALID_CREDENTIALS = {
|
||||||
|
"installed": {
|
||||||
|
"client_id": "abc.apps.googleusercontent.com",
|
||||||
|
"client_secret": "shh",
|
||||||
|
"redirect_uris": ["http://localhost"],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def plugin_dir(tmp_path, api_v3_module):
|
||||||
|
directory = tmp_path / "plugins" / "calendar"
|
||||||
|
directory.mkdir(parents=True)
|
||||||
|
api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(directory)
|
||||||
|
return directory
|
||||||
|
|
||||||
|
|
||||||
|
def upload(client, content, filename="credentials.json"):
|
||||||
|
# bytes are sent verbatim (to exercise malformed input); anything else
|
||||||
|
# is serialized, so None becomes the JSON literal null rather than an
|
||||||
|
# empty body.
|
||||||
|
payload = content if isinstance(content, bytes) else json.dumps(content).encode()
|
||||||
|
return client.post(
|
||||||
|
URL,
|
||||||
|
data={"file": (io.BytesIO(payload), filename)},
|
||||||
|
content_type="multipart/form-data",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def backups(plugin_dir):
|
||||||
|
return sorted(plugin_dir.glob("credentials.json.backup.*"))
|
||||||
|
|
||||||
|
|
||||||
|
class TestRequestValidation:
|
||||||
|
def test_no_file_part_is_a_400(self, api_v3_client, plugin_dir):
|
||||||
|
response = api_v3_client.post(URL, data={}, content_type="multipart/form-data")
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "No file provided" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_empty_filename_is_a_400(self, api_v3_client, plugin_dir):
|
||||||
|
response = upload(api_v3_client, VALID_CREDENTIALS, filename="")
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("filename", ["creds.txt", "creds.pem", "creds"])
|
||||||
|
def test_non_json_extension_is_a_400(self, api_v3_client, plugin_dir, filename):
|
||||||
|
response = upload(api_v3_client, VALID_CREDENTIALS, filename=filename)
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "JSON file" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_uppercase_json_extension_accepted(self, api_v3_client, plugin_dir):
|
||||||
|
assert upload(api_v3_client, VALID_CREDENTIALS,
|
||||||
|
filename="CREDENTIALS.JSON").status_code == 200
|
||||||
|
|
||||||
|
def test_oversized_file_is_a_400(self, api_v3_client, plugin_dir):
|
||||||
|
response = upload(api_v3_client, b"x" * (1024 * 1024 + 1))
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "1MB" in response.get_json()["message"]
|
||||||
|
assert not (plugin_dir / "credentials.json").exists()
|
||||||
|
|
||||||
|
def test_invalid_json_is_a_400(self, api_v3_client, plugin_dir):
|
||||||
|
response = upload(api_v3_client, b"{not json")
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "not valid JSON" in response.get_json()["message"]
|
||||||
|
assert not (plugin_dir / "credentials.json").exists()
|
||||||
|
|
||||||
|
def test_missing_plugin_directory_is_a_404(self, api_v3_client, api_v3_module, tmp_path):
|
||||||
|
api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(
|
||||||
|
tmp_path / "not-installed")
|
||||||
|
assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
class TestOAuthShapeValidation:
|
||||||
|
def test_installed_key_accepted(self, api_v3_client, plugin_dir):
|
||||||
|
assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 200
|
||||||
|
|
||||||
|
def test_web_key_accepted(self, api_v3_client, plugin_dir):
|
||||||
|
assert upload(api_v3_client, {"web": {"client_id": "x"}}).status_code == 200
|
||||||
|
|
||||||
|
def test_object_without_oauth_keys_is_a_400(self, api_v3_client, plugin_dir):
|
||||||
|
response = upload(api_v3_client, {"something": "else"})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "valid Google OAuth" in response.get_json()["message"]
|
||||||
|
assert not (plugin_dir / "credentials.json").exists()
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("content", [42, "a string", [1, 2, 3], True, None])
|
||||||
|
def test_valid_json_that_is_not_an_object_is_rejected(
|
||||||
|
self, api_v3_client, plugin_dir, content):
|
||||||
|
# Regression: `'installed' not in 42` raises TypeError, which the
|
||||||
|
# bare `except Exception: pass` swallowed — the file was then saved
|
||||||
|
# as credentials.json despite being unusable as credentials.
|
||||||
|
response = upload(api_v3_client, content)
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "valid Google OAuth" in response.get_json()["message"]
|
||||||
|
assert not (plugin_dir / "credentials.json").exists()
|
||||||
|
|
||||||
|
|
||||||
|
class TestSaving:
|
||||||
|
def test_file_written_with_contents_intact(self, api_v3_client, plugin_dir):
|
||||||
|
response = upload(api_v3_client, VALID_CREDENTIALS)
|
||||||
|
assert response.status_code == 200
|
||||||
|
saved = json.loads((plugin_dir / "credentials.json").read_text())
|
||||||
|
assert saved == VALID_CREDENTIALS
|
||||||
|
|
||||||
|
def test_response_reports_the_path(self, api_v3_client, plugin_dir):
|
||||||
|
body = upload(api_v3_client, VALID_CREDENTIALS).get_json()
|
||||||
|
assert body["path"].endswith("credentials.json")
|
||||||
|
|
||||||
|
def test_permissions_are_owner_only(self, api_v3_client, plugin_dir):
|
||||||
|
upload(api_v3_client, VALID_CREDENTIALS)
|
||||||
|
mode = stat.S_IMODE((plugin_dir / "credentials.json").stat().st_mode)
|
||||||
|
assert mode == 0o600
|
||||||
|
|
||||||
|
def test_first_upload_creates_no_backup(self, api_v3_client, plugin_dir):
|
||||||
|
upload(api_v3_client, VALID_CREDENTIALS)
|
||||||
|
assert backups(plugin_dir) == []
|
||||||
|
|
||||||
|
def test_overwrite_backs_up_the_previous_file(self, api_v3_client, plugin_dir):
|
||||||
|
(plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"old": 1}}))
|
||||||
|
upload(api_v3_client, VALID_CREDENTIALS)
|
||||||
|
assert len(backups(plugin_dir)) == 1
|
||||||
|
assert json.loads(backups(plugin_dir)[0].read_text()) == {"installed": {"old": 1}}
|
||||||
|
assert json.loads((plugin_dir / "credentials.json").read_text()) == VALID_CREDENTIALS
|
||||||
|
|
||||||
|
|
||||||
|
class TestBackupPruning:
|
||||||
|
def _seed(self, plugin_dir, count):
|
||||||
|
"""Create `count` backups with distinct, increasing mtimes."""
|
||||||
|
now = int(time.time())
|
||||||
|
for i in range(count):
|
||||||
|
path = plugin_dir / f"credentials.json.backup.{now - (count - i) * 10}"
|
||||||
|
path.write_text(json.dumps({"installed": {"gen": i}}))
|
||||||
|
os.utime(path, (now - (count - i) * 10, now - (count - i) * 10))
|
||||||
|
|
||||||
|
def test_old_backups_are_pruned(self, api_v3_client, plugin_dir):
|
||||||
|
# Regression: nothing ever removed these, so a plugin directory
|
||||||
|
# accumulated one full copy of the user's OAuth credentials per
|
||||||
|
# re-upload, forever.
|
||||||
|
(plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}}))
|
||||||
|
self._seed(plugin_dir, 7)
|
||||||
|
assert len(backups(plugin_dir)) == 7
|
||||||
|
|
||||||
|
upload(api_v3_client, VALID_CREDENTIALS)
|
||||||
|
assert len(backups(plugin_dir)) == 5
|
||||||
|
|
||||||
|
def test_the_newest_backups_are_the_ones_kept(self, api_v3_client, plugin_dir):
|
||||||
|
(plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}}))
|
||||||
|
self._seed(plugin_dir, 7)
|
||||||
|
|
||||||
|
upload(api_v3_client, VALID_CREDENTIALS)
|
||||||
|
remaining = backups(plugin_dir)
|
||||||
|
# The just-created backup (of "cur") plus the four newest seeds.
|
||||||
|
contents = [json.loads(p.read_text()) for p in remaining]
|
||||||
|
assert {"installed": {"cur": 1}} in contents
|
||||||
|
assert {"installed": {"gen": 0}} not in contents # oldest seed gone
|
||||||
|
|
||||||
|
def test_under_the_limit_nothing_is_removed(self, api_v3_client, plugin_dir):
|
||||||
|
(plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}}))
|
||||||
|
self._seed(plugin_dir, 2)
|
||||||
|
upload(api_v3_client, VALID_CREDENTIALS)
|
||||||
|
assert len(backups(plugin_dir)) == 3 # 2 seeded + 1 new
|
||||||
|
|
||||||
|
def test_repeated_uploads_stay_bounded(
|
||||||
|
self, api_v3_client, plugin_dir, api_v3_module, monkeypatch):
|
||||||
|
# The backup filename carries int(time.time()), so uploads inside
|
||||||
|
# the same second all write the same name and overwrite each other.
|
||||||
|
# Advance a fake clock a second per round — otherwise this never
|
||||||
|
# reaches six backups and the bound holds for the wrong reason.
|
||||||
|
clock = {"now": int(time.time())}
|
||||||
|
monkeypatch.setattr(
|
||||||
|
api_v3_module, "time", SimpleNamespace(time=lambda: clock["now"]))
|
||||||
|
for i in range(10):
|
||||||
|
clock["now"] += 1
|
||||||
|
upload(api_v3_client, {"installed": {"round": i}})
|
||||||
|
os.utime(plugin_dir / "credentials.json",
|
||||||
|
(clock["now"], clock["now"]))
|
||||||
|
remaining = backups(plugin_dir)
|
||||||
|
assert len(remaining) == 5
|
||||||
|
# And they are the five most recent rounds, not an arbitrary five.
|
||||||
|
kept = sorted(int(p.name.rsplit(".", 1)[1]) for p in remaining)
|
||||||
|
assert kept == [clock["now"] - 4 + i for i in range(5)]
|
||||||
|
|
||||||
|
def test_unremovable_backup_does_not_fail_the_upload(
|
||||||
|
self, api_v3_client, plugin_dir, monkeypatch):
|
||||||
|
(plugin_dir / "credentials.json").write_text(json.dumps({"installed": {"cur": 1}}))
|
||||||
|
self._seed(plugin_dir, 7)
|
||||||
|
|
||||||
|
def refuse(self):
|
||||||
|
raise OSError("read-only filesystem")
|
||||||
|
monkeypatch.setattr(Path, "unlink", refuse)
|
||||||
|
|
||||||
|
# Pruning is housekeeping; failing it must not lose the upload.
|
||||||
|
assert upload(api_v3_client, VALID_CREDENTIALS).status_code == 200
|
||||||
@@ -0,0 +1,302 @@
|
|||||||
|
"""
|
||||||
|
Endpoint tests for /plugins/authenticate/spotify and .../ytm.
|
||||||
|
|
||||||
|
The Spotify step-2 handler writes a Python wrapper script to a temp file
|
||||||
|
with the user's redirect URL embedded in it, then runs that file through
|
||||||
|
subprocess. That is the most dangerous shape in the blueprint and had no
|
||||||
|
tests: the URL is user input reaching generated source code.
|
||||||
|
|
||||||
|
The two endpoints are NOT symmetrical, despite the matching names. Only
|
||||||
|
Spotify has a two-step flow, a wrapper script, and a redirect_url; YTM
|
||||||
|
just runs its script directly.
|
||||||
|
|
||||||
|
Regression coverage for one fixed bug: the wrapper file was unlinked in
|
||||||
|
the success/failure branch and again in the TimeoutExpired handler, so
|
||||||
|
any other failure from subprocess.run — the interpreter missing, a fork
|
||||||
|
failure, an interrupted call — left a temp file containing the user's
|
||||||
|
redirect URL behind.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import ast
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def plugin_dir(tmp_path, api_v3_module):
|
||||||
|
"""A plugin directory containing both auth scripts."""
|
||||||
|
directory = tmp_path / "plugins" / "ledmatrix-music"
|
||||||
|
directory.mkdir(parents=True)
|
||||||
|
(directory / "authenticate_spotify.py").write_text("print('spotify')\n")
|
||||||
|
(directory / "authenticate_ytm.py").write_text("print('ytm')\n")
|
||||||
|
api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(directory)
|
||||||
|
return directory
|
||||||
|
|
||||||
|
|
||||||
|
def completed(returncode=0, stdout="ok", stderr=""):
|
||||||
|
return subprocess.CompletedProcess(
|
||||||
|
args=["python3"], returncode=returncode, stdout=stdout, stderr=stderr)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSpotifyPreconditions:
|
||||||
|
URL = "/api/v3/plugins/authenticate/spotify"
|
||||||
|
|
||||||
|
def test_missing_plugin_directory_is_404(self, api_v3_client, api_v3_module, tmp_path):
|
||||||
|
api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(
|
||||||
|
tmp_path / "not-installed")
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert response.get_json()["message"] == "Plugin not found"
|
||||||
|
|
||||||
|
def test_none_plugin_directory_is_404(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = None
|
||||||
|
assert api_v3_client.post(self.URL, json={}).status_code == 404
|
||||||
|
|
||||||
|
def test_missing_auth_script_is_404(self, api_v3_client, plugin_dir):
|
||||||
|
(plugin_dir / "authenticate_spotify.py").unlink()
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert "script not found" in response.get_json()["message"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestSpotifyStepTwo:
|
||||||
|
"""redirect_url present — the wrapper-script path."""
|
||||||
|
|
||||||
|
URL = "/api/v3/plugins/authenticate/spotify"
|
||||||
|
|
||||||
|
def test_success(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed(0, "done")):
|
||||||
|
response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.get_json()
|
||||||
|
assert body["status"] == "success"
|
||||||
|
assert body["output"] == "done"
|
||||||
|
|
||||||
|
def test_script_failure_is_a_400_with_combined_output(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed(1, "out", "err")):
|
||||||
|
response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.get_json()["output"] == "outerr"
|
||||||
|
|
||||||
|
def test_timeout_is_a_408(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run",
|
||||||
|
side_effect=subprocess.TimeoutExpired("python3", 120)):
|
||||||
|
response = api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"})
|
||||||
|
assert response.status_code == 408
|
||||||
|
assert "timed out" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_runs_a_list_argv_never_a_shell(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed()) as run:
|
||||||
|
api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"})
|
||||||
|
args, kwargs = run.call_args
|
||||||
|
assert isinstance(args[0], list)
|
||||||
|
assert args[0][0] == "python3"
|
||||||
|
assert kwargs.get("shell") in (None, False)
|
||||||
|
|
||||||
|
def test_timeout_is_bounded(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed()) as run:
|
||||||
|
api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"})
|
||||||
|
assert run.call_args.kwargs["timeout"] == 120
|
||||||
|
|
||||||
|
|
||||||
|
class TestSpotifyWrapperCleanup:
|
||||||
|
URL = "/api/v3/plugins/authenticate/spotify"
|
||||||
|
|
||||||
|
def _wrapper_paths_after(self, api_v3_client, run_mock):
|
||||||
|
"""Run the endpoint and return the wrapper path subprocess saw."""
|
||||||
|
seen = {}
|
||||||
|
|
||||||
|
def capture(args, **kwargs):
|
||||||
|
seen["path"] = args[1]
|
||||||
|
return run_mock(args, **kwargs)
|
||||||
|
|
||||||
|
with patch.object(subprocess, "run", side_effect=capture):
|
||||||
|
api_v3_client.post(self.URL, json={"redirect_url": "http://cb/?code=x"})
|
||||||
|
return seen["path"]
|
||||||
|
|
||||||
|
def test_removed_after_success(self, api_v3_client, plugin_dir):
|
||||||
|
path = self._wrapper_paths_after(api_v3_client, lambda *a, **kw: completed())
|
||||||
|
assert not os.path.exists(path)
|
||||||
|
|
||||||
|
def test_removed_after_script_failure(self, api_v3_client, plugin_dir):
|
||||||
|
path = self._wrapper_paths_after(
|
||||||
|
api_v3_client, lambda *a, **kw: completed(1, "out", "err"))
|
||||||
|
assert not os.path.exists(path)
|
||||||
|
|
||||||
|
def test_removed_after_timeout(self, api_v3_client, plugin_dir):
|
||||||
|
def raise_timeout(*a, **kw):
|
||||||
|
raise subprocess.TimeoutExpired("python3", 120)
|
||||||
|
path = self._wrapper_paths_after(api_v3_client, raise_timeout)
|
||||||
|
assert not os.path.exists(path)
|
||||||
|
|
||||||
|
def test_removed_when_subprocess_cannot_start(self, api_v3_client, plugin_dir):
|
||||||
|
# Regression: cleanup lived in the success/failure branch and in the
|
||||||
|
# TimeoutExpired handler only. An OSError from subprocess.run itself
|
||||||
|
# — no interpreter, fork failure — skipped both and left the wrapper,
|
||||||
|
# which contains the user's redirect URL, on disk.
|
||||||
|
def raise_oserror(*a, **kw):
|
||||||
|
raise OSError("[Errno 12] Cannot allocate memory")
|
||||||
|
path = self._wrapper_paths_after(api_v3_client, raise_oserror)
|
||||||
|
assert not os.path.exists(path)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSpotifyRedirectUrlIsNotInjectable:
|
||||||
|
"""The wrapper embeds redirect_url into generated Python source."""
|
||||||
|
|
||||||
|
URL = "/api/v3/plugins/authenticate/spotify"
|
||||||
|
|
||||||
|
ADVERSARIAL = [
|
||||||
|
'''http://cb/?code=x"''',
|
||||||
|
"""http://cb/?code=x'""",
|
||||||
|
'http://cb/?code=x\\',
|
||||||
|
'http://cb/?code=x\nimport os; os.system("id")',
|
||||||
|
'http://cb/?code=x"""\nimport os\n"""',
|
||||||
|
"http://cb/?code=x'''",
|
||||||
|
'http://cb/?code=x\\"\\n',
|
||||||
|
'"; import os; os.system("id"); "',
|
||||||
|
]
|
||||||
|
|
||||||
|
def _wrapper_source(self, api_v3_client, redirect_url):
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def capture(args, **kwargs):
|
||||||
|
captured["source"] = Path(args[1]).read_text()
|
||||||
|
return completed()
|
||||||
|
|
||||||
|
with patch.object(subprocess, "run", side_effect=capture):
|
||||||
|
api_v3_client.post(self.URL, json={"redirect_url": redirect_url})
|
||||||
|
return captured["source"]
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("redirect_url", ADVERSARIAL)
|
||||||
|
def test_wrapper_is_still_valid_python(self, api_v3_client, plugin_dir, redirect_url):
|
||||||
|
# If escaping failed, the generated file would not parse at all.
|
||||||
|
source = self._wrapper_source(api_v3_client, redirect_url)
|
||||||
|
ast.parse(source)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("redirect_url", ADVERSARIAL)
|
||||||
|
def test_url_survives_as_one_string_literal(
|
||||||
|
self, api_v3_client, plugin_dir, redirect_url):
|
||||||
|
# Stronger than "it parses": the URL must still be a single string
|
||||||
|
# assigned to redirect_url, not code that escaped into statements.
|
||||||
|
source = self._wrapper_source(api_v3_client, redirect_url)
|
||||||
|
tree = ast.parse(source)
|
||||||
|
assigned = [
|
||||||
|
node.value.value for node in ast.walk(tree)
|
||||||
|
if isinstance(node, ast.Assign)
|
||||||
|
and isinstance(node.value, ast.Constant)
|
||||||
|
and any(getattr(t, "id", None) == "redirect_url" for t in node.targets)
|
||||||
|
]
|
||||||
|
assert assigned == [redirect_url.strip()]
|
||||||
|
|
||||||
|
def test_injected_call_does_not_become_a_statement(self, api_v3_client, plugin_dir):
|
||||||
|
source = self._wrapper_source(
|
||||||
|
api_v3_client, 'http://cb/\nimport os; os.system("id")')
|
||||||
|
tree = ast.parse(source)
|
||||||
|
imported = {
|
||||||
|
alias.name for node in ast.walk(tree)
|
||||||
|
if isinstance(node, ast.Import) for alias in node.names
|
||||||
|
}
|
||||||
|
# The wrapper legitimately imports sys, subprocess and os; what it
|
||||||
|
# must not gain is a *call* smuggled in through the URL.
|
||||||
|
calls = [
|
||||||
|
node for node in ast.walk(tree)
|
||||||
|
if isinstance(node, ast.Call)
|
||||||
|
and isinstance(node.func, ast.Attribute)
|
||||||
|
and node.func.attr == "system"
|
||||||
|
]
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestSpotifyStepOne:
|
||||||
|
"""No redirect_url — the OAuth-URL path, which imports the script."""
|
||||||
|
|
||||||
|
URL = "/api/v3/plugins/authenticate/spotify"
|
||||||
|
|
||||||
|
def test_script_without_credentials_helper_is_an_error(
|
||||||
|
self, api_v3_client, plugin_dir):
|
||||||
|
# The stub script defines neither get_auth_url nor
|
||||||
|
# load_spotify_credentials, so no URL can be produced.
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code in (400, 500)
|
||||||
|
assert response.get_json()["status"] == "error"
|
||||||
|
|
||||||
|
def test_unusable_credentials_do_not_leak_into_the_response(
|
||||||
|
self, api_v3_client, plugin_dir):
|
||||||
|
(plugin_dir / "authenticate_spotify.py").write_text(
|
||||||
|
"def load_spotify_credentials():\n"
|
||||||
|
" return ('id-abc', 'super-secret-value', None)\n"
|
||||||
|
)
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert "super-secret-value" not in response.get_data(as_text=True)
|
||||||
|
|
||||||
|
def test_script_raising_on_import_is_handled(self, api_v3_client, plugin_dir):
|
||||||
|
(plugin_dir / "authenticate_spotify.py").write_text("raise RuntimeError('boom')\n")
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 500
|
||||||
|
assert response.get_json()["status"] == "error"
|
||||||
|
|
||||||
|
def test_bodyless_post_reaches_step_one(self, api_v3_client, plugin_dir):
|
||||||
|
# Covered by the silent=True fix: previously a 500 from body parsing.
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
assert response.status_code in (400, 500)
|
||||||
|
assert response.get_json()["status"] == "error"
|
||||||
|
|
||||||
|
def test_whitespace_redirect_url_is_treated_as_absent(
|
||||||
|
self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed()) as run:
|
||||||
|
api_v3_client.post(self.URL, json={"redirect_url": " "})
|
||||||
|
# Step 2 never runs, so no wrapper is executed.
|
||||||
|
run.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
class TestYouTubeMusic:
|
||||||
|
"""No wrapper script and no redirect_url — deliberately not symmetric."""
|
||||||
|
|
||||||
|
URL = "/api/v3/plugins/authenticate/ytm"
|
||||||
|
|
||||||
|
def test_missing_plugin_directory_is_404(self, api_v3_client, api_v3_module, tmp_path):
|
||||||
|
api_v3_module.api_v3.plugin_manager.get_plugin_directory.return_value = str(
|
||||||
|
tmp_path / "not-installed")
|
||||||
|
assert api_v3_client.post(self.URL).status_code == 404
|
||||||
|
|
||||||
|
def test_missing_script_is_404(self, api_v3_client, plugin_dir):
|
||||||
|
(plugin_dir / "authenticate_ytm.py").unlink()
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
assert response.status_code == 404
|
||||||
|
assert "script not found" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_success(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed(0, "authorized")):
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.get_json()["output"] == "authorized"
|
||||||
|
|
||||||
|
def test_failure_is_a_400_with_combined_output(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed(1, "out", "err")):
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.get_json()["output"] == "outerr"
|
||||||
|
|
||||||
|
def test_timeout_is_a_408(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run",
|
||||||
|
side_effect=subprocess.TimeoutExpired("python3", 60)):
|
||||||
|
assert api_v3_client.post(self.URL).status_code == 408
|
||||||
|
|
||||||
|
def test_runs_the_script_directly_without_a_shell(self, api_v3_client, plugin_dir):
|
||||||
|
with patch.object(subprocess, "run", return_value=completed()) as run:
|
||||||
|
api_v3_client.post(self.URL)
|
||||||
|
args, kwargs = run.call_args
|
||||||
|
assert args[0][0] == "python3"
|
||||||
|
assert args[0][1].endswith("authenticate_ytm.py")
|
||||||
|
assert kwargs.get("shell") in (None, False)
|
||||||
|
assert kwargs["timeout"] == 60
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
"""Non-finite JSON numbers must be rejected, not raise.
|
||||||
|
|
||||||
|
json.loads accepts Infinity/-Infinity/NaN by default (they are not valid JSON,
|
||||||
|
but Python's parser emits them) and Flask's get_json passes them straight
|
||||||
|
through. int(float('inf')) raises OverflowError, which is neither ValueError
|
||||||
|
nor TypeError -- so validation blocks that carefully caught those let it
|
||||||
|
through and Flask turned it into a 500.
|
||||||
|
|
||||||
|
The damage was not the status code. /config/dim-schedule answered with
|
||||||
|
CONFIG_SAVE_FAILED and suggested "Check file permissions on config directory"
|
||||||
|
and "Check available disk space" for what was actually an invalid number.
|
||||||
|
|
||||||
|
NaN already returned 400 (int(nan) raises ValueError), which is why this only
|
||||||
|
showed up for the infinities.
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
|
||||||
|
|
||||||
|
|
||||||
|
#: (route, field) that returned 500 before OverflowError was caught. Both
|
||||||
|
#: infinity signs are exercised: int() raises OverflowError for either, but
|
||||||
|
#: only one of them was in the original report, and a guard that special-cased
|
||||||
|
#: the sign would pass a one-sided test.
|
||||||
|
NON_FINITE_ROUTES = [
|
||||||
|
('/api/v3/config/dim-schedule', 'dim_brightness'),
|
||||||
|
('/api/v3/errors/clear', 'max_age_hours'),
|
||||||
|
('/api/v3/config/main', 'multiplexing'),
|
||||||
|
('/api/v3/config/main', 'row_address_type'),
|
||||||
|
]
|
||||||
|
NON_FINITE_CASES = [
|
||||||
|
(route, '{"%s": %s}' % (field, literal))
|
||||||
|
for route, field in NON_FINITE_ROUTES
|
||||||
|
for literal in ('Infinity', '-Infinity')
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("route,body", NON_FINITE_CASES)
|
||||||
|
def test_infinity_is_a_client_error_not_a_server_error(api_v3_client, route, body):
|
||||||
|
"""Exactly 400, not merely "some 4xx".
|
||||||
|
|
||||||
|
Accepting any 4xx would let a 404 pass, so renaming one of these routes
|
||||||
|
would leave the test green while testing nothing -- the failure mode this
|
||||||
|
whole file exists to catch.
|
||||||
|
"""
|
||||||
|
response = api_v3_client.post(route, data=body, content_type='application/json')
|
||||||
|
assert response.status_code == 400, (
|
||||||
|
f"{route} with {body} answered {response.status_code}; expected 400"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("route,body", [
|
||||||
|
('/api/v3/config/dim-schedule', '{"dim_brightness": NaN}'),
|
||||||
|
('/api/v3/errors/clear', '{"max_age_hours": NaN}'),
|
||||||
|
])
|
||||||
|
def test_nan_is_also_a_client_error(api_v3_client, route, body):
|
||||||
|
"""int(nan) raises ValueError so this path already worked -- pinned so a
|
||||||
|
refactor that narrows the except tuple cannot quietly break it."""
|
||||||
|
response = api_v3_client.post(route, data=body, content_type='application/json')
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_valid_number_is_accepted(api_v3_client, api_v3_module, monkeypatch):
|
||||||
|
"""Prove the widened except did not start swallowing ordinary input.
|
||||||
|
|
||||||
|
Asserting "not a 400" would not show that: the mocked save path fails for
|
||||||
|
any input, so the assertion would hold even if validation had rejected the
|
||||||
|
value. Give load_config a real dict and stub the atomic save, and the
|
||||||
|
endpoint reaches its success response -- which only happens if 30 passed
|
||||||
|
validation.
|
||||||
|
"""
|
||||||
|
api_v3_module.api_v3.config_manager.load_config.return_value = {}
|
||||||
|
monkeypatch.setattr(api_v3_module, '_save_config_atomic',
|
||||||
|
lambda *a, **k: (True, ''))
|
||||||
|
response = api_v3_client.post(
|
||||||
|
'/api/v3/config/dim-schedule',
|
||||||
|
data='{"dim_brightness": 30}',
|
||||||
|
content_type='application/json',
|
||||||
|
)
|
||||||
|
assert response.status_code == 200, response.get_data(as_text=True)[:200]
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
"""
|
||||||
|
Regression tests: POST endpoints whose body is optional must accept a
|
||||||
|
request that has no body at all.
|
||||||
|
|
||||||
|
Six handlers in api_v3 read their body as ``request.get_json() or {}``.
|
||||||
|
The ``or {}`` states the intent plainly — every field is optional, so a
|
||||||
|
bodyless POST should fall back to defaults. But ``get_json()`` without
|
||||||
|
``silent=True`` raises ``UnsupportedMediaType`` when the request carries
|
||||||
|
no JSON Content-Type, and it raises *before* ``or {}`` is evaluated. Each
|
||||||
|
handler's catch-all then turned that into a 500.
|
||||||
|
|
||||||
|
So the natural way to call these endpoints — a POST with no body, which
|
||||||
|
is what curl, a fetch() without options, and most HTTP clients send by
|
||||||
|
default — failed on every one of them. The shipped UI always sends a JSON
|
||||||
|
object, which is why this went unnoticed.
|
||||||
|
|
||||||
|
This file covers the endpoints whose bodyless behaviour is not already
|
||||||
|
tested in their own suite.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
|
||||||
|
|
||||||
|
|
||||||
|
class TestOnDemandStart:
|
||||||
|
URL = "/api/v3/display/on-demand/start"
|
||||||
|
|
||||||
|
def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module):
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
# The endpoint may still reject the request on its own terms (no
|
||||||
|
# plugin_id, nothing to display); what it must not do is fail with
|
||||||
|
# a 500 raised out of body parsing.
|
||||||
|
assert response.status_code != 500
|
||||||
|
|
||||||
|
def test_json_body_still_works(self, api_v3_client, api_v3_module):
|
||||||
|
assert api_v3_client.post(self.URL, json={}).status_code != 500
|
||||||
|
|
||||||
|
|
||||||
|
class TestResetPluginConfig:
|
||||||
|
URL = "/api/v3/plugins/config/reset"
|
||||||
|
|
||||||
|
def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module):
|
||||||
|
assert api_v3_client.post(self.URL).status_code != 500
|
||||||
|
|
||||||
|
def test_json_body_still_works(self, api_v3_client, api_v3_module):
|
||||||
|
assert api_v3_client.post(self.URL, json={}).status_code != 500
|
||||||
|
|
||||||
|
|
||||||
|
class TestDeleteOfTheDayJson:
|
||||||
|
URL = "/api/v3/plugins/of-the-day/json/delete"
|
||||||
|
|
||||||
|
def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module):
|
||||||
|
assert api_v3_client.post(self.URL).status_code != 500
|
||||||
|
|
||||||
|
def test_json_body_still_works(self, api_v3_client, api_v3_module):
|
||||||
|
assert api_v3_client.post(self.URL, json={}).status_code != 500
|
||||||
|
|
||||||
|
|
||||||
|
class TestPluginLimits:
|
||||||
|
URL = "/api/v3/plugins/clock/limits"
|
||||||
|
|
||||||
|
def test_bodyless_post_is_not_a_server_error(self, api_v3_client, api_v3_module):
|
||||||
|
assert api_v3_client.post(self.URL).status_code != 500
|
||||||
|
|
||||||
|
|
||||||
|
class TestMissingBodyGivesTheDeclaredError:
|
||||||
|
"""Handlers that answer "No data provided" must actually be able to.
|
||||||
|
|
||||||
|
A second group of handlers reads `data = request.get_json()` and then
|
||||||
|
guards with `if not data: return 400`. That guard is unreachable for a
|
||||||
|
request with no JSON body, because get_json() raises first — so the
|
||||||
|
caller got a 500 "an error occurred; see logs for details" instead of
|
||||||
|
the 400 the handler plainly intends to send.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("url", [
|
||||||
|
"/api/v3/plugins/install",
|
||||||
|
"/api/v3/plugins/install-from-url",
|
||||||
|
"/api/v3/plugins/registry-from-url",
|
||||||
|
"/api/v3/config/raw/main",
|
||||||
|
"/api/v3/config/raw/secrets",
|
||||||
|
"/api/v3/cache/delete",
|
||||||
|
])
|
||||||
|
def test_bodyless_post_gets_a_400_not_a_500(self, api_v3_client, api_v3_module, url):
|
||||||
|
response = api_v3_client.post(url)
|
||||||
|
assert response.status_code == 400, (
|
||||||
|
f"{url} answered {response.status_code}: "
|
||||||
|
f"{response.get_data(as_text=True)[:200]}")
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("url", [
|
||||||
|
"/api/v3/plugins/install",
|
||||||
|
"/api/v3/config/raw/main",
|
||||||
|
])
|
||||||
|
def test_malformed_json_gets_a_400_not_a_500(self, api_v3_client, api_v3_module, url):
|
||||||
|
response = api_v3_client.post(
|
||||||
|
url, data="{not json", content_type="application/json")
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
class TestNoBodyReadContradictsItsOwnGuard:
|
||||||
|
SOURCE = Path(__file__).parent.parent / "web_interface/blueprints/api_v3.py"
|
||||||
|
|
||||||
|
def test_no_or_default_read_is_unguarded(self):
|
||||||
|
"""`get_json() or <default>` is a contradiction without silent=True.
|
||||||
|
|
||||||
|
Writing `or {}` declares the body optional; omitting silent=True
|
||||||
|
means the call raises before the default can apply.
|
||||||
|
"""
|
||||||
|
offenders = [
|
||||||
|
line.strip() for line in self.SOURCE.read_text().splitlines()
|
||||||
|
if "request.get_json()" in line and " or " in line
|
||||||
|
]
|
||||||
|
assert offenders == [], (
|
||||||
|
"these reads declare a default but raise before reaching it; "
|
||||||
|
f"use get_json(silent=True): {offenders}")
|
||||||
|
|
||||||
|
def test_no_not_data_guard_is_unreachable(self):
|
||||||
|
"""A `if not data:` guard needs a read that can actually return None."""
|
||||||
|
lines = self.SOURCE.read_text().splitlines()
|
||||||
|
offenders = []
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
if re.search(r"=\s*request\.get_json\(\)\s*$", line):
|
||||||
|
window = "\n".join(lines[i + 1:i + 3])
|
||||||
|
if re.search(r"if\s+(not\s+data\b|data\s+is\s+None)", window):
|
||||||
|
offenders.append(f"line {i + 1}: {line.strip()}")
|
||||||
|
assert offenders == [], (
|
||||||
|
"these handlers guard on a missing body but raise before the "
|
||||||
|
f"guard runs; use get_json(silent=True): {offenders}")
|
||||||
@@ -0,0 +1,302 @@
|
|||||||
|
"""
|
||||||
|
Endpoint tests for POST /plugins/install and POST /plugins/install-from-url.
|
||||||
|
|
||||||
|
Both were only ever tested at the PluginStoreManager layer, so the route
|
||||||
|
logic — the queue-vs-direct branch, schema invalidation, plugin discovery,
|
||||||
|
state and history recording — was unexercised.
|
||||||
|
|
||||||
|
/plugins/install carries the same install logic twice: once inside the
|
||||||
|
operation-queue callback and once in the direct fallback. The paired
|
||||||
|
tests below assert both branches produce the same side effects, so the
|
||||||
|
duplication cannot quietly drift.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
|
||||||
|
|
||||||
|
INSTALL = "/api/v3/plugins/install"
|
||||||
|
FROM_URL = "/api/v3/plugins/install-from-url"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def queued(api_v3_module):
|
||||||
|
"""Enable the operation queue and run its callback synchronously."""
|
||||||
|
queue = MagicMock()
|
||||||
|
|
||||||
|
def enqueue(operation_type, plugin_id, operation_callback=None):
|
||||||
|
queue.callback_result = operation_callback(MagicMock())
|
||||||
|
return "op-123"
|
||||||
|
|
||||||
|
queue.enqueue_operation.side_effect = enqueue
|
||||||
|
api_v3_module.api_v3.operation_queue = queue
|
||||||
|
return queue
|
||||||
|
|
||||||
|
|
||||||
|
def side_effects(module):
|
||||||
|
"""The manager calls a successful install is expected to make."""
|
||||||
|
api = module.api_v3
|
||||||
|
return {
|
||||||
|
"schema_invalidated": api.schema_manager.invalidate_cache.call_args_list,
|
||||||
|
"discovered": api.plugin_manager.discover_plugins.call_count,
|
||||||
|
"loaded": api.plugin_manager.load_plugin.call_args_list,
|
||||||
|
"state_set": api.plugin_state_manager.set_plugin_installed.call_args_list,
|
||||||
|
"history": api.operation_history.record_operation.call_args_list,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class TestInstallValidation:
|
||||||
|
def test_uninitialized_store_manager_is_a_500(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager = None
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert response.status_code == 500
|
||||||
|
assert "not initialized" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_missing_plugin_id_is_a_400(self, api_v3_client, api_v3_module):
|
||||||
|
response = api_v3_client.post(INSTALL, json={})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "plugin_id required" in response.get_json()["message"]
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.assert_not_called()
|
||||||
|
|
||||||
|
def test_empty_body_is_a_400(self, api_v3_client, api_v3_module):
|
||||||
|
assert api_v3_client.post(INSTALL, json=None).status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
class TestInstallDirectPath:
|
||||||
|
"""operation_queue is None — the fallback branch."""
|
||||||
|
|
||||||
|
def test_success(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.get_json()["status"] == "success"
|
||||||
|
|
||||||
|
def test_success_side_effects(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
effects = side_effects(api_v3_module)
|
||||||
|
assert effects["schema_invalidated"] == [(("clock",), {})]
|
||||||
|
assert effects["discovered"] == 1
|
||||||
|
assert effects["loaded"] == [(("clock",), {})]
|
||||||
|
assert effects["state_set"] == [(("clock",), {})]
|
||||||
|
assert effects["history"][0].kwargs["status"] == "success"
|
||||||
|
|
||||||
|
def test_branch_forwarded_to_the_manager(self, api_v3_client, api_v3_module):
|
||||||
|
manager = api_v3_module.api_v3.plugin_store_manager
|
||||||
|
manager.install_plugin.return_value = True
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"})
|
||||||
|
manager.install_plugin.assert_called_once_with("clock", branch="dev")
|
||||||
|
|
||||||
|
def test_branch_named_in_the_message(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"})
|
||||||
|
assert "(branch: dev)" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_failure_is_a_500(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert response.status_code == 500
|
||||||
|
assert "Failed to install" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_failure_mentions_missing_registry_entry(self, api_v3_client, api_v3_module):
|
||||||
|
manager = api_v3_module.api_v3.plugin_store_manager
|
||||||
|
manager.install_plugin.return_value = False
|
||||||
|
manager.get_plugin_info.return_value = None
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "ghost"})
|
||||||
|
assert "not found in registry" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_failure_omits_registry_note_when_plugin_is_known(
|
||||||
|
self, api_v3_client, api_v3_module):
|
||||||
|
manager = api_v3_module.api_v3.plugin_store_manager
|
||||||
|
manager.install_plugin.return_value = False
|
||||||
|
manager.get_plugin_info.return_value = {"id": "clock"}
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert "not found in registry" not in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_failure_recorded_in_history(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
record = api_v3_module.api_v3.operation_history.record_operation.call_args
|
||||||
|
assert record.kwargs["status"] == "failed"
|
||||||
|
|
||||||
|
def test_no_side_effects_on_failure(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
effects = side_effects(api_v3_module)
|
||||||
|
assert effects["schema_invalidated"] == []
|
||||||
|
assert effects["loaded"] == []
|
||||||
|
assert effects["state_set"] == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestInstallQueuedPath:
|
||||||
|
"""operation_queue present — the callback branch."""
|
||||||
|
|
||||||
|
def test_returns_an_operation_id(self, api_v3_client, api_v3_module, queued):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.get_json()["data"]["operation_id"] == "op-123"
|
||||||
|
|
||||||
|
def test_message_says_queued(self, api_v3_client, api_v3_module, queued):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert "queued" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_callback_success_side_effects(self, api_v3_client, api_v3_module, queued):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
effects = side_effects(api_v3_module)
|
||||||
|
assert effects["schema_invalidated"] == [(("clock",), {})]
|
||||||
|
assert effects["discovered"] == 1
|
||||||
|
assert effects["loaded"] == [(("clock",), {})]
|
||||||
|
assert effects["state_set"] == [(("clock",), {})]
|
||||||
|
assert effects["history"][0].kwargs["status"] == "success"
|
||||||
|
|
||||||
|
def test_callback_reports_success(self, api_v3_client, api_v3_module, queued):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert queued.callback_result["success"] is True
|
||||||
|
|
||||||
|
def test_callback_failure_raises_for_the_queue(self, api_v3_client, api_v3_module, queued):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False
|
||||||
|
# The callback signals failure by raising, so the queue can mark the
|
||||||
|
# operation failed; the route's catch-all turns it into a 500.
|
||||||
|
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
assert response.status_code == 500
|
||||||
|
|
||||||
|
def test_callback_failure_recorded_in_history(self, api_v3_client, api_v3_module, queued):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = False
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
|
||||||
|
record = api_v3_module.api_v3.operation_history.record_operation.call_args
|
||||||
|
assert record.kwargs["status"] == "failed"
|
||||||
|
|
||||||
|
def test_branch_forwarded_from_the_callback(self, api_v3_client, api_v3_module, queued):
|
||||||
|
manager = api_v3_module.api_v3.plugin_store_manager
|
||||||
|
manager.install_plugin.return_value = True
|
||||||
|
api_v3_client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"})
|
||||||
|
manager.install_plugin.assert_called_once_with("clock", branch="dev")
|
||||||
|
|
||||||
|
|
||||||
|
class TestInstallPathsAgree:
|
||||||
|
"""The queue callback and the direct fallback duplicate the same logic."""
|
||||||
|
|
||||||
|
def _run(self, client, module, install_ok, queue):
|
||||||
|
module.api_v3.plugin_store_manager.install_plugin.return_value = install_ok
|
||||||
|
client.post(INSTALL, json={"plugin_id": "clock", "branch": "dev"})
|
||||||
|
return side_effects(module)
|
||||||
|
|
||||||
|
def test_success_side_effects_match(self, api_v3_client, api_v3_module):
|
||||||
|
direct = self._run(api_v3_client, api_v3_module, True, None)
|
||||||
|
|
||||||
|
# Reset and re-run through the queue.
|
||||||
|
for mock in (api_v3_module.api_v3.schema_manager,
|
||||||
|
api_v3_module.api_v3.plugin_manager,
|
||||||
|
api_v3_module.api_v3.plugin_state_manager,
|
||||||
|
api_v3_module.api_v3.operation_history):
|
||||||
|
mock.reset_mock()
|
||||||
|
queue = MagicMock()
|
||||||
|
queue.enqueue_operation.side_effect = (
|
||||||
|
lambda t, p, operation_callback=None: operation_callback(MagicMock()) and "op")
|
||||||
|
api_v3_module.api_v3.operation_queue = queue
|
||||||
|
queued = self._run(api_v3_client, api_v3_module, True, queue)
|
||||||
|
|
||||||
|
assert direct["schema_invalidated"] == queued["schema_invalidated"]
|
||||||
|
assert direct["discovered"] == queued["discovered"]
|
||||||
|
assert direct["loaded"] == queued["loaded"]
|
||||||
|
assert direct["state_set"] == queued["state_set"]
|
||||||
|
assert (direct["history"][0].kwargs["status"]
|
||||||
|
== queued["history"][0].kwargs["status"])
|
||||||
|
assert (direct["history"][0].kwargs["details"]
|
||||||
|
== queued["history"][0].kwargs["details"])
|
||||||
|
|
||||||
|
def test_only_the_message_wording_differs(self, api_v3_client, api_v3_module):
|
||||||
|
# Characterized: the direct path says "Plugin installed
|
||||||
|
# successfully" while the queue callback says "Plugin clock
|
||||||
|
# installed successfully". Cosmetic, and the queue's text is
|
||||||
|
# internal to the operation record rather than the HTTP response.
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_plugin.return_value = True
|
||||||
|
direct = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}).get_json()
|
||||||
|
assert direct["message"] == "Plugin installed successfully"
|
||||||
|
|
||||||
|
|
||||||
|
class TestInstallFromUrl:
|
||||||
|
def test_uninitialized_store_manager_is_a_500(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager = None
|
||||||
|
assert api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).status_code == 500
|
||||||
|
|
||||||
|
def test_missing_repo_url_is_a_400(self, api_v3_client, api_v3_module):
|
||||||
|
response = api_v3_client.post(FROM_URL, json={})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "repo_url required" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_success(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = {
|
||||||
|
"success": True, "plugin_id": "clock", "name": "Clock"}
|
||||||
|
response = api_v3_client.post(FROM_URL, json={"repo_url": "https://github.com/o/r"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.get_json()
|
||||||
|
assert body["plugin_id"] == "clock"
|
||||||
|
assert body["name"] == "Clock"
|
||||||
|
|
||||||
|
def test_all_optional_arguments_forwarded(self, api_v3_client, api_v3_module):
|
||||||
|
manager = api_v3_module.api_v3.plugin_store_manager
|
||||||
|
manager.install_from_url.return_value = {"success": True, "plugin_id": "clock"}
|
||||||
|
api_v3_client.post(FROM_URL, json={
|
||||||
|
"repo_url": " https://github.com/o/r ",
|
||||||
|
"plugin_id": "clock",
|
||||||
|
"plugin_path": "plugins/clock",
|
||||||
|
"branch": "dev",
|
||||||
|
})
|
||||||
|
manager.install_from_url.assert_called_once_with(
|
||||||
|
repo_url="https://github.com/o/r",
|
||||||
|
plugin_id="clock",
|
||||||
|
plugin_path="plugins/clock",
|
||||||
|
branch="dev",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_success_invalidates_schema_and_loads_plugin(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = {
|
||||||
|
"success": True, "plugin_id": "clock"}
|
||||||
|
api_v3_client.post(FROM_URL, json={"repo_url": "http://x"})
|
||||||
|
api_v3_module.api_v3.schema_manager.invalidate_cache.assert_called_once_with("clock")
|
||||||
|
api_v3_module.api_v3.plugin_manager.load_plugin.assert_called_once_with("clock")
|
||||||
|
|
||||||
|
def test_success_without_plugin_id_skips_discovery(self, api_v3_client, api_v3_module):
|
||||||
|
# install_from_url can succeed without naming the plugin; there is
|
||||||
|
# then nothing to invalidate or load.
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = {
|
||||||
|
"success": True, "plugin_id": None}
|
||||||
|
api_v3_client.post(FROM_URL, json={"repo_url": "http://x"})
|
||||||
|
api_v3_module.api_v3.schema_manager.invalidate_cache.assert_not_called()
|
||||||
|
api_v3_module.api_v3.plugin_manager.load_plugin.assert_not_called()
|
||||||
|
|
||||||
|
def test_branch_from_result_included(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = {
|
||||||
|
"success": True, "plugin_id": "clock", "branch": "dev"}
|
||||||
|
body = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).get_json()
|
||||||
|
assert body["branch"] == "dev"
|
||||||
|
assert "(branch: dev)" in body["message"]
|
||||||
|
|
||||||
|
def test_failure_reports_the_managers_error(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = {
|
||||||
|
"success": False, "error": "repo not found"}
|
||||||
|
response = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"})
|
||||||
|
assert response.status_code == 500
|
||||||
|
assert response.get_json()["message"] == "repo not found"
|
||||||
|
|
||||||
|
def test_failure_without_error_uses_fallback_text(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_from_url.return_value = {
|
||||||
|
"success": False}
|
||||||
|
response = api_v3_client.post(FROM_URL, json={"repo_url": "http://x"})
|
||||||
|
assert "Failed to install plugin from URL" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_manager_exception_is_a_500(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.install_from_url.side_effect = (
|
||||||
|
RuntimeError("boom"))
|
||||||
|
assert api_v3_client.post(FROM_URL, json={"repo_url": "http://x"}).status_code == 500
|
||||||
@@ -0,0 +1,179 @@
|
|||||||
|
"""
|
||||||
|
Endpoint tests for the plugin-registry routes in api_v3:
|
||||||
|
POST /plugins/store/refresh and POST /plugins/registry-from-url.
|
||||||
|
|
||||||
|
Both reach out to the network through PluginStoreManager (mocked here) and
|
||||||
|
had no endpoint-level coverage; registry-from-url in particular takes a
|
||||||
|
user-supplied URL and hands it straight to the manager.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
|
||||||
|
|
||||||
|
|
||||||
|
class TestRefreshPluginStore:
|
||||||
|
URL = "/api/v3/plugins/store/refresh"
|
||||||
|
|
||||||
|
def test_uninitialized_manager_is_a_500(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager = None
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 500
|
||||||
|
assert "not initialized" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_success_reports_plugin_count(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {
|
||||||
|
"plugins": [{"id": "a"}, {"id": "b"}, {"id": "c"}]}
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.get_json()["plugin_count"] == 3
|
||||||
|
|
||||||
|
def test_forces_a_refresh_rather_than_using_cache(self, api_v3_client, api_v3_module):
|
||||||
|
manager = api_v3_module.api_v3.plugin_store_manager
|
||||||
|
manager.fetch_registry.return_value = {"plugins": []}
|
||||||
|
api_v3_client.post(self.URL, json={})
|
||||||
|
manager.fetch_registry.assert_called_once_with(force_refresh=True)
|
||||||
|
|
||||||
|
def test_empty_registry_reports_zero(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {}
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.get_json()["plugin_count"] == 0
|
||||||
|
|
||||||
|
def test_no_body_is_accepted(self, api_v3_client, api_v3_module):
|
||||||
|
# Regression: `request.get_json() or {}` says a missing body is
|
||||||
|
# fine, but get_json() raises UnsupportedMediaType before `or {}`
|
||||||
|
# is reached, so a bodyless POST — the natural way to call a
|
||||||
|
# refresh endpoint — came back 500.
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []}
|
||||||
|
assert api_v3_client.post(self.URL).status_code == 200
|
||||||
|
|
||||||
|
def test_body_without_json_content_type_is_accepted(
|
||||||
|
self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []}
|
||||||
|
response = api_v3_client.post(self.URL, data="", content_type="text/plain")
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
def test_malformed_json_body_falls_back_to_defaults(
|
||||||
|
self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []}
|
||||||
|
response = api_v3_client.post(
|
||||||
|
self.URL, data="{not json", content_type="application/json")
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("key", ["fetch_commit_info", "fetch_latest_versions"])
|
||||||
|
def test_either_commit_info_key_extends_the_message(
|
||||||
|
self, api_v3_client, api_v3_module, key):
|
||||||
|
# fetch_latest_versions is the older spelling; both must work.
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []}
|
||||||
|
response = api_v3_client.post(self.URL, json={key: True})
|
||||||
|
assert "commit metadata" in response.get_json()["message"]
|
||||||
|
|
||||||
|
def test_message_stays_plain_without_the_flag(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.return_value = {"plugins": []}
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.get_json()["message"] == "Plugin store refreshed"
|
||||||
|
|
||||||
|
def test_network_failure_is_a_500(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.side_effect = (
|
||||||
|
ConnectionError("github unreachable"))
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 500
|
||||||
|
assert response.get_json()["message"] == "An error occurred; see logs for details"
|
||||||
|
|
||||||
|
def test_failure_body_carries_no_traceback_or_paths(
|
||||||
|
self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry.side_effect = (
|
||||||
|
RuntimeError("failed at /home/user/LEDMatrix/src/secret.py line 42"))
|
||||||
|
body = api_v3_client.post(self.URL, json={}).get_json()
|
||||||
|
assert "Traceback" not in str(body)
|
||||||
|
# `details` is describe_exception output: one line, type-named,
|
||||||
|
# credential-redacted. It may quote the message, but never a stack.
|
||||||
|
assert body["details"].startswith("RuntimeError:")
|
||||||
|
assert "\n" not in body["details"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestRegistryFromUrl:
|
||||||
|
URL = "/api/v3/plugins/registry-from-url"
|
||||||
|
|
||||||
|
def test_uninitialized_manager_is_a_500(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager = None
|
||||||
|
response = api_v3_client.post(self.URL, json={"repo_url": "http://x"})
|
||||||
|
assert response.status_code == 500
|
||||||
|
|
||||||
|
def test_missing_repo_url_is_a_400(self, api_v3_client, api_v3_module):
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "repo_url required" in response.get_json()["message"]
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called()
|
||||||
|
|
||||||
|
def test_success_returns_the_plugin_list(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = {
|
||||||
|
"plugins": [{"id": "clock"}]}
|
||||||
|
response = api_v3_client.post(
|
||||||
|
self.URL, json={"repo_url": "https://github.com/o/r"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.get_json()
|
||||||
|
assert body["plugins"] == [{"id": "clock"}]
|
||||||
|
assert body["registry_url"] == "https://github.com/o/r"
|
||||||
|
|
||||||
|
def test_url_is_trimmed_before_use(self, api_v3_client, api_v3_module):
|
||||||
|
manager = api_v3_module.api_v3.plugin_store_manager
|
||||||
|
manager.fetch_registry_from_url.return_value = {"plugins": []}
|
||||||
|
api_v3_client.post(self.URL, json={"repo_url": " https://github.com/o/r "})
|
||||||
|
manager.fetch_registry_from_url.assert_called_once_with("https://github.com/o/r")
|
||||||
|
|
||||||
|
def test_registry_without_plugins_key_returns_empty_list(
|
||||||
|
self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = {
|
||||||
|
"other": 1}
|
||||||
|
response = api_v3_client.post(self.URL, json={"repo_url": "http://x"})
|
||||||
|
assert response.get_json()["plugins"] == []
|
||||||
|
|
||||||
|
def test_no_registry_found_is_a_400(self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = None
|
||||||
|
response = api_v3_client.post(self.URL, json={"repo_url": "http://x/not-a-registry"})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "Failed to fetch registry" in response.get_json()["message"]
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("url", [
|
||||||
|
"not a url",
|
||||||
|
"javascript:alert(1)",
|
||||||
|
"file:///etc/passwd",
|
||||||
|
"http://localhost:8080/admin",
|
||||||
|
])
|
||||||
|
def test_unusable_urls_fail_cleanly(self, api_v3_client, api_v3_module, url):
|
||||||
|
# Characterization: the handler performs no URL validation of its
|
||||||
|
# own — whatever the manager makes of the URL decides the outcome.
|
||||||
|
# What is pinned here is that a rejected URL produces a clean 400
|
||||||
|
# rather than a traceback or a 500.
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.return_value = None
|
||||||
|
response = api_v3_client.post(self.URL, json={"repo_url": url})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "Traceback" not in str(response.get_json())
|
||||||
|
|
||||||
|
def test_fetch_exception_is_a_500_without_internals(
|
||||||
|
self, api_v3_client, api_v3_module):
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.side_effect = (
|
||||||
|
ValueError("parse failed in /srv/app/internal.py"))
|
||||||
|
response = api_v3_client.post(self.URL, json={"repo_url": "http://x"})
|
||||||
|
assert response.status_code == 500
|
||||||
|
body = response.get_json()
|
||||||
|
assert body["message"] == "An error occurred; see logs for details"
|
||||||
|
assert "Traceback" not in str(body)
|
||||||
|
|
||||||
|
def test_non_string_repo_url_is_rejected(self, api_v3_client, api_v3_module):
|
||||||
|
# Regression: .strip() on a non-string raised, and the catch-all
|
||||||
|
# reported the caller's own mistake as a server fault.
|
||||||
|
response = api_v3_client.post(self.URL, json={"repo_url": 12345})
|
||||||
|
assert response.status_code == 400
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called()
|
||||||
|
|
||||||
|
def test_blank_repo_url_is_rejected(self, api_v3_client, api_v3_module):
|
||||||
|
response = api_v3_client.post(self.URL, json={"repo_url": " "})
|
||||||
|
assert response.status_code == 400
|
||||||
|
api_v3_module.api_v3.plugin_store_manager.fetch_registry_from_url.assert_not_called()
|
||||||
@@ -0,0 +1,240 @@
|
|||||||
|
"""
|
||||||
|
Endpoint tests for the /wifi/* routes in api_v3.
|
||||||
|
|
||||||
|
These routes drive the host's actual networking — connecting, dropping a
|
||||||
|
connection, switching the radio off — and had no endpoint-level tests at
|
||||||
|
all. WiFiManager is mocked throughout; nothing here may touch real
|
||||||
|
networking.
|
||||||
|
|
||||||
|
Each handler does `from src.wifi_manager import WiFiManager` inside the
|
||||||
|
function body, so the patch target is the class at its definition site.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
|
|
||||||
|
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def wifi_manager():
|
||||||
|
"""Patch WiFiManager where it is defined; yield the instance mock."""
|
||||||
|
with patch("src.wifi_manager.WiFiManager") as cls:
|
||||||
|
instance = MagicMock()
|
||||||
|
cls.return_value = instance
|
||||||
|
yield instance
|
||||||
|
|
||||||
|
|
||||||
|
class TestConnect:
|
||||||
|
URL = "/api/v3/wifi/connect"
|
||||||
|
|
||||||
|
def test_success(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.connect_to_network.return_value = (True, "Connected to HomeNet")
|
||||||
|
response = api_v3_client.post(self.URL, json={"ssid": "HomeNet", "password": "pw"})
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.get_json()["message"] == "Connected to HomeNet"
|
||||||
|
wifi_manager.connect_to_network.assert_called_once_with("HomeNet", "pw")
|
||||||
|
|
||||||
|
def test_missing_body_rejected(self, api_v3_client, wifi_manager):
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 400
|
||||||
|
wifi_manager.connect_to_network.assert_not_called()
|
||||||
|
|
||||||
|
def test_missing_ssid_rejected(self, api_v3_client, wifi_manager):
|
||||||
|
response = api_v3_client.post(self.URL, json={"password": "pw"})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "SSID is required" in response.get_json()["message"]
|
||||||
|
wifi_manager.connect_to_network.assert_not_called()
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("ssid", ["", " ", "\t"])
|
||||||
|
def test_blank_ssid_rejected(self, api_v3_client, wifi_manager, ssid):
|
||||||
|
response = api_v3_client.post(self.URL, json={"ssid": ssid})
|
||||||
|
assert response.status_code == 400
|
||||||
|
wifi_manager.connect_to_network.assert_not_called()
|
||||||
|
|
||||||
|
def test_ssid_is_trimmed(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.connect_to_network.return_value = (True, "ok")
|
||||||
|
api_v3_client.post(self.URL, json={"ssid": " HomeNet "})
|
||||||
|
wifi_manager.connect_to_network.assert_called_once_with("HomeNet", "")
|
||||||
|
|
||||||
|
def test_missing_password_becomes_empty_string(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.connect_to_network.return_value = (True, "ok")
|
||||||
|
api_v3_client.post(self.URL, json={"ssid": "OpenNet"})
|
||||||
|
wifi_manager.connect_to_network.assert_called_once_with("OpenNet", "")
|
||||||
|
|
||||||
|
def test_null_password_becomes_empty_string(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.connect_to_network.return_value = (True, "ok")
|
||||||
|
api_v3_client.post(self.URL, json={"ssid": "OpenNet", "password": None})
|
||||||
|
wifi_manager.connect_to_network.assert_called_once_with("OpenNet", "")
|
||||||
|
|
||||||
|
def test_failure_reports_the_managers_reason(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.connect_to_network.return_value = (False, "Bad password")
|
||||||
|
response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.get_json()["message"] == "Bad password"
|
||||||
|
|
||||||
|
def test_failure_without_reason_uses_fallback_text(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.connect_to_network.return_value = (False, None)
|
||||||
|
response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.get_json()["message"] == "Failed to connect to network"
|
||||||
|
|
||||||
|
def test_manager_exception_is_a_500_without_leaking_internals(
|
||||||
|
self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.connect_to_network.side_effect = RuntimeError(
|
||||||
|
"/usr/lib/secret/path blew up")
|
||||||
|
response = api_v3_client.post(self.URL, json={"ssid": "HomeNet"})
|
||||||
|
assert response.status_code == 500
|
||||||
|
body = response.get_json()
|
||||||
|
assert body["message"] == "An error occurred; see logs for details"
|
||||||
|
# `details` comes from describe_exception, which is deliberately
|
||||||
|
# safe to return (redacted, capped) — it names the type.
|
||||||
|
assert "RuntimeError" in body["details"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestDisconnect:
|
||||||
|
URL = "/api/v3/wifi/disconnect"
|
||||||
|
|
||||||
|
def test_success(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.disconnect_from_network.return_value = (True, "Disconnected")
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.get_json()["message"] == "Disconnected"
|
||||||
|
|
||||||
|
def test_failure(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.disconnect_from_network.return_value = (False, "Not connected")
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.get_json()["message"] == "Not connected"
|
||||||
|
|
||||||
|
def test_failure_without_reason_uses_fallback(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.disconnect_from_network.return_value = (False, "")
|
||||||
|
response = api_v3_client.post(self.URL)
|
||||||
|
assert response.get_json()["message"] == "Failed to disconnect from network"
|
||||||
|
|
||||||
|
def test_exception_is_a_500(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.disconnect_from_network.side_effect = OSError("nmcli missing")
|
||||||
|
assert api_v3_client.post(self.URL).status_code == 500
|
||||||
|
|
||||||
|
|
||||||
|
class TestApMode:
|
||||||
|
ENABLE = "/api/v3/wifi/ap/enable"
|
||||||
|
DISABLE = "/api/v3/wifi/ap/disable"
|
||||||
|
|
||||||
|
def test_enable_success(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.enable_ap_mode.return_value = (True, "AP enabled")
|
||||||
|
response = api_v3_client.post(self.ENABLE, json={})
|
||||||
|
assert response.status_code == 200
|
||||||
|
wifi_manager.enable_ap_mode.assert_called_once_with(force=False)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("raw,expected", [
|
||||||
|
(True, True), (False, False),
|
||||||
|
("true", True), ("TRUE", True), ("1", True),
|
||||||
|
("false", False), ("no", False), ("yes", False),
|
||||||
|
(1, False), # only real True or the listed strings count
|
||||||
|
])
|
||||||
|
def test_force_coercion(self, api_v3_client, wifi_manager, raw, expected):
|
||||||
|
wifi_manager.enable_ap_mode.return_value = (True, "ok")
|
||||||
|
api_v3_client.post(self.ENABLE, json={"force": raw})
|
||||||
|
wifi_manager.enable_ap_mode.assert_called_once_with(force=expected)
|
||||||
|
|
||||||
|
def test_enable_without_body(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.enable_ap_mode.return_value = (True, "ok")
|
||||||
|
assert api_v3_client.post(self.ENABLE).status_code == 200
|
||||||
|
|
||||||
|
def test_enable_failure(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.enable_ap_mode.return_value = (False, "hostapd missing")
|
||||||
|
response = api_v3_client.post(self.ENABLE, json={})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert response.get_json()["message"] == "hostapd missing"
|
||||||
|
|
||||||
|
def test_disable_success(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.disable_ap_mode.return_value = (True, "AP disabled")
|
||||||
|
assert api_v3_client.post(self.DISABLE).status_code == 200
|
||||||
|
|
||||||
|
def test_disable_failure(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.disable_ap_mode.return_value = (False, "not running")
|
||||||
|
assert api_v3_client.post(self.DISABLE).status_code == 400
|
||||||
|
|
||||||
|
def test_enable_exception_is_a_500(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.enable_ap_mode.side_effect = RuntimeError("boom")
|
||||||
|
assert api_v3_client.post(self.ENABLE, json={}).status_code == 500
|
||||||
|
|
||||||
|
|
||||||
|
class TestRadio:
|
||||||
|
URL = "/api/v3/wifi/radio"
|
||||||
|
|
||||||
|
def test_get_state(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.get_wifi_radio_state.return_value = {
|
||||||
|
"enabled": True, "ethernet_connected": False}
|
||||||
|
response = api_v3_client.get(self.URL)
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.get_json()["data"]["enabled"] is True
|
||||||
|
|
||||||
|
def test_get_state_exception_is_a_500(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.get_wifi_radio_state.side_effect = OSError("rfkill missing")
|
||||||
|
assert api_v3_client.get(self.URL).status_code == 500
|
||||||
|
|
||||||
|
def test_enabled_is_required(self, api_v3_client, wifi_manager):
|
||||||
|
response = api_v3_client.post(self.URL, json={})
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "enabled is required" in response.get_json()["message"]
|
||||||
|
wifi_manager.set_wifi_radio.assert_not_called()
|
||||||
|
|
||||||
|
def test_enable_success(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.set_wifi_radio.return_value = (True, "Radio on", None)
|
||||||
|
wifi_manager.get_wifi_radio_state.return_value = {"enabled": True}
|
||||||
|
response = api_v3_client.post(self.URL, json={"enabled": True})
|
||||||
|
assert response.status_code == 200
|
||||||
|
wifi_manager.set_wifi_radio.assert_called_once_with(True, force=False)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("raw,expected", [
|
||||||
|
(True, True), ("true", True), ("1", True), ("yes", True),
|
||||||
|
(False, False), ("false", False), ("off", False), (0, False),
|
||||||
|
])
|
||||||
|
def test_enabled_coercion_is_string_aware(
|
||||||
|
self, api_v3_client, wifi_manager, raw, expected):
|
||||||
|
# bool("false") is True, so the endpoint parses strings explicitly
|
||||||
|
# rather than trusting truthiness — it is a public contract, not
|
||||||
|
# only the shipped UI which always sends real JSON booleans.
|
||||||
|
wifi_manager.set_wifi_radio.return_value = (True, "ok", None)
|
||||||
|
wifi_manager.get_wifi_radio_state.return_value = {}
|
||||||
|
api_v3_client.post(self.URL, json={"enabled": raw})
|
||||||
|
wifi_manager.set_wifi_radio.assert_called_once_with(expected, force=False)
|
||||||
|
|
||||||
|
def test_force_passed_through(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.set_wifi_radio.return_value = (True, "ok", None)
|
||||||
|
wifi_manager.get_wifi_radio_state.return_value = {}
|
||||||
|
api_v3_client.post(self.URL, json={"enabled": False, "force": "true"})
|
||||||
|
wifi_manager.set_wifi_radio.assert_called_once_with(False, force=True)
|
||||||
|
|
||||||
|
def test_refusal_reports_reason(self, api_v3_client, wifi_manager):
|
||||||
|
# Disabling the radio without Ethernet would lock the user out of
|
||||||
|
# this very interface, so the manager can refuse with a reason.
|
||||||
|
wifi_manager.set_wifi_radio.return_value = (
|
||||||
|
False, "Refusing: no wired fallback", "no_ethernet")
|
||||||
|
response = api_v3_client.post(self.URL, json={"enabled": False})
|
||||||
|
assert response.status_code == 400
|
||||||
|
body = response.get_json()
|
||||||
|
assert body["reason"] == "no_ethernet"
|
||||||
|
assert "Refusing" in body["message"]
|
||||||
|
|
||||||
|
def test_exception_is_a_500(self, api_v3_client, wifi_manager):
|
||||||
|
wifi_manager.set_wifi_radio.side_effect = RuntimeError("boom")
|
||||||
|
assert api_v3_client.post(self.URL, json={"enabled": True}).status_code == 500
|
||||||
|
|
||||||
|
|
||||||
|
class TestNoRealNetworking:
|
||||||
|
def test_wifi_manager_is_never_constructed_for_real(self, api_v3_client):
|
||||||
|
# Guard against a future refactor moving the import to module level,
|
||||||
|
# where the fixture's patch of the definition site would stop
|
||||||
|
# applying and the tests would start driving real networking.
|
||||||
|
with patch("src.wifi_manager.WiFiManager") as cls:
|
||||||
|
cls.return_value.disconnect_from_network.return_value = (True, "ok")
|
||||||
|
api_v3_client.post("/api/v3/wifi/disconnect")
|
||||||
|
assert cls.called
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
"""web_interface/app.py resolves plugins_dir from config at import time.
|
||||||
|
|
||||||
|
`project_root` used to be assigned only inside the `else` branch (the
|
||||||
|
relative-path case). SchemaManager and composer_bp are wired up with
|
||||||
|
`project_root` further down the same module, so a device configured with an
|
||||||
|
*absolute* `plugin_system.plugins_directory` (a supported value -- see
|
||||||
|
CLAUDE.md) hit `UnboundLocalError` importing web_interface.app at all, taking
|
||||||
|
the whole web UI down.
|
||||||
|
|
||||||
|
app.py isn't imported directly by the test suite -- it has real side effects
|
||||||
|
(managers, background state) at import time -- so this extracts the exact
|
||||||
|
resolution block by its stable comment markers and executes it in isolation,
|
||||||
|
the same way the rest of the suite validates generated/templated code rather
|
||||||
|
than re-implementing it.
|
||||||
|
"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
APP_PY = Path(__file__).resolve().parent.parent / "web_interface/app.py"
|
||||||
|
|
||||||
|
START_MARKER = "# Resolve plugin directory - handle both absolute and relative paths\n"
|
||||||
|
END_MARKER = "\nplugin_manager = PluginManager("
|
||||||
|
|
||||||
|
|
||||||
|
def _resolution_block() -> str:
|
||||||
|
src = APP_PY.read_text()
|
||||||
|
start = src.index(START_MARKER)
|
||||||
|
end = src.index(END_MARKER, start)
|
||||||
|
return src[start:end]
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolution_block_still_matches_expected_markers():
|
||||||
|
"""If app.py is restructured enough that these markers move, the exec
|
||||||
|
below would silently test nothing -- fail loudly instead."""
|
||||||
|
block = _resolution_block()
|
||||||
|
assert "plugins_dir" in block and "os.path.isabs" in block
|
||||||
|
|
||||||
|
|
||||||
|
def test_absolute_plugins_directory_does_not_raise_unboundlocalerror():
|
||||||
|
import os
|
||||||
|
ns = {"os": os, "Path": Path, "__file__": str(APP_PY),
|
||||||
|
"plugins_dir_name": "/opt/ledmatrix-plugins"}
|
||||||
|
exec(compile(_resolution_block(), str(APP_PY), "exec"), ns) # noqa: S102
|
||||||
|
assert ns["project_root"] == APP_PY.parent.parent
|
||||||
|
assert ns["plugins_dir"] == Path("/opt/ledmatrix-plugins")
|
||||||
|
|
||||||
|
|
||||||
|
def test_relative_plugins_directory_still_resolves_under_project_root():
|
||||||
|
import os
|
||||||
|
ns = {"os": os, "Path": Path, "__file__": str(APP_PY),
|
||||||
|
"plugins_dir_name": "plugin-repos"}
|
||||||
|
exec(compile(_resolution_block(), str(APP_PY), "exec"), ns) # noqa: S102
|
||||||
|
assert ns["plugins_dir"] == ns["project_root"] / "plugin-repos"
|
||||||
@@ -0,0 +1,450 @@
|
|||||||
|
"""A second request for a key already being fetched must join, not duplicate.
|
||||||
|
|
||||||
|
request_id embeds a millisecond timestamp and active_requests is keyed by it,
|
||||||
|
so every submit looked new and nothing compared what was actually being
|
||||||
|
fetched. On a real board the season-schedule cache_key is requested by both
|
||||||
|
the Recent and the Upcoming manager: they miss the cache in the same
|
||||||
|
millisecond and each start a full download and parse of the same payload.
|
||||||
|
Measured on a running board, 138 background fetches in 24 hours arriving in
|
||||||
|
pairs at identical timestamps -- half of them redundant.
|
||||||
|
|
||||||
|
The cost of a duplicate is a second download, a second JSON parse (the
|
||||||
|
expensive part on a Pi), and a second parsed copy resident at the same time.
|
||||||
|
Schedules on that board run from 256KB to 20MB. It also consumes a second of
|
||||||
|
the three executor slots with identical work, which is what makes two large
|
||||||
|
parses peak simultaneously.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from unittest.mock import MagicMock, Mock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from src.background_data_service import BackgroundDataService, FetchStatus
|
||||||
|
|
||||||
|
|
||||||
|
PAYLOAD = {"events": [{"id": f"g{i}"} for i in range(20)]}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cache():
|
||||||
|
m = MagicMock()
|
||||||
|
m.get.return_value = None # always a miss: force the fetch path
|
||||||
|
m.set.return_value = None
|
||||||
|
return m
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def service(cache):
|
||||||
|
svc = BackgroundDataService(cache, max_workers=3, request_timeout=5)
|
||||||
|
yield svc
|
||||||
|
svc.shutdown(wait=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _resp():
|
||||||
|
r = Mock()
|
||||||
|
r.json.return_value = PAYLOAD
|
||||||
|
r.raise_for_status.return_value = None
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
def _wait(service, req_id, timeout=5):
|
||||||
|
deadline = time.time() + timeout
|
||||||
|
while not service.is_request_complete(req_id) and time.time() < deadline:
|
||||||
|
time.sleep(0.02)
|
||||||
|
|
||||||
|
|
||||||
|
class _BlockingSession:
|
||||||
|
"""Holds the first fetch open so a second can be submitted mid-flight."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.calls = 0
|
||||||
|
self.release = threading.Event()
|
||||||
|
self.started = threading.Event()
|
||||||
|
|
||||||
|
def get(self, *a, **k):
|
||||||
|
self.calls += 1
|
||||||
|
self.started.set()
|
||||||
|
self.release.wait(timeout=5)
|
||||||
|
return _resp()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_second_submit_for_the_same_key_does_not_fetch_twice(service):
|
||||||
|
session = _BlockingSession()
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="nba_2026",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
|
||||||
|
second = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="nba_2026",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
|
||||||
|
assert second == first, "the joiner should share the in-flight request id"
|
||||||
|
session.release.set()
|
||||||
|
_wait(service, first)
|
||||||
|
|
||||||
|
assert session.calls == 1, f"the payload was fetched {session.calls} times"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_joiner_still_gets_its_callback(service):
|
||||||
|
session = _BlockingSession()
|
||||||
|
seen = []
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: seen.append("first"), max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
joined = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: seen.append("second"), max_retries=0)
|
||||||
|
# Assert the coalescing happened, otherwise this passes trivially:
|
||||||
|
# two independent requests would each fire their own callback and the
|
||||||
|
# test would say nothing about the joined path.
|
||||||
|
assert joined == first
|
||||||
|
session.release.set()
|
||||||
|
_wait(service, first)
|
||||||
|
|
||||||
|
deadline = time.time() + 5
|
||||||
|
while len(seen) < 2 and time.time() < deadline:
|
||||||
|
time.sleep(0.02)
|
||||||
|
assert sorted(seen) == ["first", "second"], (
|
||||||
|
f"both submitters must be called back, got {seen}")
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_callback_raising_does_not_silence_the_other(service):
|
||||||
|
session = _BlockingSession()
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
def boom(result):
|
||||||
|
raise RuntimeError("consumer blew up")
|
||||||
|
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=boom, max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
joined = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: seen.append("survivor"), max_retries=0)
|
||||||
|
# Same reason: without coalescing these are separate requests and
|
||||||
|
# neither callback can affect the other.
|
||||||
|
assert joined == first
|
||||||
|
session.release.set()
|
||||||
|
_wait(service, first)
|
||||||
|
|
||||||
|
deadline = time.time() + 5
|
||||||
|
while not seen and time.time() < deadline:
|
||||||
|
time.sleep(0.02)
|
||||||
|
assert seen == ["survivor"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_different_keys_are_not_coalesced(service):
|
||||||
|
session = _BlockingSession()
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
a = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/a", cache_key="key_a",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
b = service.submit_fetch_request(
|
||||||
|
sport="nhl", year=2026, url="https://x/b", cache_key="key_b",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
assert a != b, "different cache keys must not share a request"
|
||||||
|
session.release.set()
|
||||||
|
_wait(service, a)
|
||||||
|
_wait(service, b)
|
||||||
|
assert session.calls == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_later_submit_after_completion_fetches_again(service):
|
||||||
|
"""Dedupe is for concurrent requests only, not a second cache layer."""
|
||||||
|
with patch.object(service.session, "get", side_effect=[_resp(), _resp()]) as get:
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
_wait(service, first)
|
||||||
|
second = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
_wait(service, second)
|
||||||
|
assert first != second
|
||||||
|
assert get.call_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_cancelling_releases_the_key(service):
|
||||||
|
"""A cancelled request must not wedge its key against future fetches."""
|
||||||
|
session = _BlockingSession()
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
service.cancel_request(first)
|
||||||
|
assert "k" not in service._inflight_by_cache_key
|
||||||
|
session.release.set()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_stranded_index_entry_cannot_wedge_a_key(service):
|
||||||
|
"""Defensive: the request is looked up, not trusted from the id alone."""
|
||||||
|
service._inflight_by_cache_key["ghost"] = "no_such_request"
|
||||||
|
with patch.object(service.session, "get", return_value=_resp()):
|
||||||
|
req = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="ghost",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
_wait(service, req)
|
||||||
|
assert service.get_result(req).success is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_deduplicated_count_is_reported(service):
|
||||||
|
session = _BlockingSession()
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
max_retries=0)
|
||||||
|
session.release.set()
|
||||||
|
_wait(service, first)
|
||||||
|
assert service.get_statistics().get("deduplicated_requests") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_cancelled_worker_cannot_overwrite_its_replacement(service, cache):
|
||||||
|
"""Cancelling frees the key, so a replacement may already own it.
|
||||||
|
|
||||||
|
The worker cannot abort an HTTP call in flight, so when the cancelled one
|
||||||
|
finally returns it must discard its response rather than write it. Without
|
||||||
|
that, the sequence is: cancel A, submit B for the same key, B fetches and
|
||||||
|
caches fresh data, A returns and overwrites it with the response nobody
|
||||||
|
wanted -- and calls A's callbacks too.
|
||||||
|
"""
|
||||||
|
slow = _BlockingSession()
|
||||||
|
stale = {"events": [{"id": "STALE"}]}
|
||||||
|
slow_resp = Mock()
|
||||||
|
slow_resp.json.return_value = stale
|
||||||
|
slow_resp.raise_for_status.return_value = None
|
||||||
|
|
||||||
|
def blocked_get(*a, **k):
|
||||||
|
slow.calls += 1
|
||||||
|
slow.started.set()
|
||||||
|
slow.release.wait(timeout=5)
|
||||||
|
return slow_resp
|
||||||
|
|
||||||
|
called = []
|
||||||
|
with patch.object(service.session, "get", side_effect=blocked_get):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: called.append("cancelled_one"), max_retries=0)
|
||||||
|
assert slow.started.wait(timeout=5)
|
||||||
|
|
||||||
|
service.cancel_request(first)
|
||||||
|
assert "k" not in service._inflight_by_cache_key
|
||||||
|
|
||||||
|
# The replacement writes the fresh value while the cancelled fetch is held.
|
||||||
|
fresh = {"events": [{"id": "FRESH"}]}
|
||||||
|
fresh_resp = Mock()
|
||||||
|
fresh_resp.json.return_value = fresh
|
||||||
|
fresh_resp.raise_for_status.return_value = None
|
||||||
|
with patch.object(service.session, "get", return_value=fresh_resp):
|
||||||
|
second = service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s", cache_key="k",
|
||||||
|
callback=lambda r: called.append("replacement"), max_retries=0)
|
||||||
|
_wait(service, second)
|
||||||
|
|
||||||
|
assert cache.set.call_args[0][1] == fresh, "replacement must own the cache"
|
||||||
|
|
||||||
|
# Now let the cancelled fetch finish. It must write nothing and call nobody.
|
||||||
|
# Wait for the worker to actually finish rather than sleeping: a fixed
|
||||||
|
# sleep is a race under load, and a slow worker would make this pass for
|
||||||
|
# the wrong reason. A cancelled request is still filed in
|
||||||
|
# completed_requests, so that is the signal it has run to completion.
|
||||||
|
writes_before = cache.set.call_count
|
||||||
|
slow.release.set()
|
||||||
|
deadline = time.time() + 5
|
||||||
|
while first not in service.completed_requests and time.time() < deadline:
|
||||||
|
time.sleep(0.02)
|
||||||
|
assert first in service.completed_requests, "cancelled worker never finished"
|
||||||
|
|
||||||
|
assert cache.set.call_count == writes_before, (
|
||||||
|
"the cancelled worker wrote to the cache after its replacement")
|
||||||
|
assert cache.set.call_args[0][1] == fresh, "stale data overwrote fresh"
|
||||||
|
assert "cancelled_one" not in called, (
|
||||||
|
"a cancelled request must not deliver callbacks")
|
||||||
|
|
||||||
|
|
||||||
|
def test_request_ids_are_unique_within_a_millisecond(service):
|
||||||
|
"""request_id was sport_year_milliseconds, which collides.
|
||||||
|
|
||||||
|
Two submits inside the same millisecond produced the SAME id, so one
|
||||||
|
silently replaced the other in active_requests and completed_requests.
|
||||||
|
Dedupe hands this id back to every joiner as their handle for
|
||||||
|
get_result(), so uniqueness is now load-bearing rather than incidental.
|
||||||
|
"""
|
||||||
|
# Stub the executor rather than the session: this is about what submit
|
||||||
|
# hands back, and letting 50 workers loose would outlive the patch and
|
||||||
|
# make real network calls.
|
||||||
|
with patch.object(service.executor, "submit"):
|
||||||
|
ids = [
|
||||||
|
service.submit_fetch_request(
|
||||||
|
sport="nba", year=2026, url="https://x/s",
|
||||||
|
cache_key=f"key_{i}", # distinct keys: no dedupe
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
for i in range(50)
|
||||||
|
]
|
||||||
|
assert len(set(ids)) == len(ids), "request ids collided"
|
||||||
|
|
||||||
|
|
||||||
|
# --- cancellation must be terminal ------------------------------------------
|
||||||
|
#
|
||||||
|
# Cancelling used to be advisory: three separate paths wrote request.status
|
||||||
|
# without checking whether the request had already been cancelled, so a cancel
|
||||||
|
# could be silently undone and the work it was meant to stop went ahead.
|
||||||
|
|
||||||
|
URL = "http://example.invalid/scores"
|
||||||
|
KEY = "sched_nfl_2025"
|
||||||
|
|
||||||
|
|
||||||
|
class _CountingSession:
|
||||||
|
"""Records whether an HTTP fetch was ever attempted."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.calls = 0
|
||||||
|
|
||||||
|
def get(self, *a, **k):
|
||||||
|
self.calls += 1
|
||||||
|
return _resp()
|
||||||
|
|
||||||
|
|
||||||
|
class _BlockingFailingSession(_CountingSession):
|
||||||
|
"""Holds the fetch open, then fails it.
|
||||||
|
|
||||||
|
Cancelling while the worker is parked inside the HTTP call is the only way
|
||||||
|
to reach the exception handler as a cancelled request. Cancel it before
|
||||||
|
the call starts and the worker returns at the pre-start branch instead,
|
||||||
|
which would leave the except path untested.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self.started = threading.Event()
|
||||||
|
self.release = threading.Event()
|
||||||
|
|
||||||
|
def get(self, *a, **k):
|
||||||
|
self.calls += 1
|
||||||
|
self.started.set()
|
||||||
|
self.release.wait(timeout=5)
|
||||||
|
raise requests.RequestException("connection reset")
|
||||||
|
|
||||||
|
|
||||||
|
def _fill_every_worker_slot(service, slots=3):
|
||||||
|
"""Occupy the pool so the next submit is queued rather than started.
|
||||||
|
|
||||||
|
This is what makes "cancel before the worker runs" deterministic instead
|
||||||
|
of a race the test would win only sometimes. Returns the gate that
|
||||||
|
releases the pool.
|
||||||
|
"""
|
||||||
|
gate = threading.Event()
|
||||||
|
for _ in range(slots):
|
||||||
|
service.executor.submit(gate.wait, 5)
|
||||||
|
return gate
|
||||||
|
|
||||||
|
|
||||||
|
def test_cancelling_before_the_worker_starts_stops_the_fetch(service, cache):
|
||||||
|
"""The queued worker must honour a cancel, not overwrite it with IN_PROGRESS.
|
||||||
|
|
||||||
|
Between submit and the worker picking the job up, the request sits in the
|
||||||
|
executor queue. Cancelling there is the cheapest possible cancel -- nothing
|
||||||
|
has been downloaded yet -- and it was the one that did not work.
|
||||||
|
"""
|
||||||
|
gate = _fill_every_worker_slot(service)
|
||||||
|
session = _CountingSession()
|
||||||
|
delivered = []
|
||||||
|
|
||||||
|
with patch.object(service, 'session', session):
|
||||||
|
rid = service.submit_fetch_request(
|
||||||
|
"nfl", 2025, URL, KEY, max_retries=0, callback=delivered.append
|
||||||
|
)
|
||||||
|
assert service.cancel_request(rid) is True
|
||||||
|
gate.set() # let the queued worker run
|
||||||
|
_wait(service, rid)
|
||||||
|
|
||||||
|
assert session.calls == 0, (
|
||||||
|
"cancelled before it started, yet the worker still downloaded the payload"
|
||||||
|
)
|
||||||
|
assert cache.set.call_count == 0, "a cancelled request wrote to the cache"
|
||||||
|
assert delivered == [], "a cancelled request invoked its callbacks"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_cancel_during_the_commit_is_refused(service, cache):
|
||||||
|
"""Once the worker has claimed the commit, cancelling is too late.
|
||||||
|
|
||||||
|
The claim and the cancelled-check happen in one critical section, so a
|
||||||
|
cancel arriving after it cannot retroactively abandon data already on its
|
||||||
|
way to the cache. Letting it through stranded every joiner: the payload
|
||||||
|
landed in the cache but the callbacks were suppressed, so a manager that
|
||||||
|
joined this fetch waited for a call that never came.
|
||||||
|
"""
|
||||||
|
gate = _fill_every_worker_slot(service)
|
||||||
|
late = {}
|
||||||
|
delivered = []
|
||||||
|
|
||||||
|
def cancel_mid_write(key, data, *a, **k):
|
||||||
|
late['returned'] = service.cancel_request(late['rid'])
|
||||||
|
|
||||||
|
cache.set.side_effect = cancel_mid_write
|
||||||
|
|
||||||
|
# Read the payload inside the callback. The service releases result.data
|
||||||
|
# once every callback has been delivered, so inspecting the FetchResult
|
||||||
|
# afterwards sees the released object, not what the caller was handed.
|
||||||
|
def record(result):
|
||||||
|
delivered.append((result.success, result.data))
|
||||||
|
|
||||||
|
with patch.object(service, 'session', _CountingSession()):
|
||||||
|
late['rid'] = service.submit_fetch_request(
|
||||||
|
"nfl", 2025, URL, KEY, max_retries=0, callback=record
|
||||||
|
)
|
||||||
|
gate.set() # only now can the worker reach the commit
|
||||||
|
_wait(service, late['rid'])
|
||||||
|
|
||||||
|
assert late.get('returned') is False, (
|
||||||
|
"cancelled a request that had already committed"
|
||||||
|
)
|
||||||
|
assert cache.set.call_count == 1, "the commit itself was lost"
|
||||||
|
assert delivered == [(True, PAYLOAD)], (
|
||||||
|
"data reached the cache but the callbacks were suppressed -- "
|
||||||
|
"every joined submitter is left waiting forever"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_failure_after_cancelling_stays_cancelled(service, cache):
|
||||||
|
"""A cancelled request that then errors must not resurface as FAILED.
|
||||||
|
|
||||||
|
The except path overwrote CANCELLED with FAILED, and the callback gate in
|
||||||
|
the finally block only suppresses callbacks for CANCELLED -- so cancelling
|
||||||
|
a request that was about to time out delivered a spurious error callback.
|
||||||
|
"""
|
||||||
|
session = _BlockingFailingSession()
|
||||||
|
delivered = []
|
||||||
|
|
||||||
|
with patch.object(service, 'session', session):
|
||||||
|
rid = service.submit_fetch_request(
|
||||||
|
"nfl", 2025, URL, KEY, max_retries=0, callback=delivered.append
|
||||||
|
)
|
||||||
|
# Assert the worker is inside the HTTP call before cancelling,
|
||||||
|
# otherwise this silently degrades into the pre-start case and the
|
||||||
|
# exception handler is never exercised.
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
assert service.cancel_request(rid) is True
|
||||||
|
session.release.set()
|
||||||
|
_wait(service, rid)
|
||||||
|
|
||||||
|
assert session.calls == 1, "the fetch never started, so nothing could fail"
|
||||||
|
|
||||||
|
assert delivered == [], "a cancelled request delivered a failure callback"
|
||||||
|
assert service.get_request_status(rid) is FetchStatus.CANCELLED, (
|
||||||
|
"a cancelled request that then errored was reported as FAILED"
|
||||||
|
)
|
||||||
@@ -0,0 +1,280 @@
|
|||||||
|
"""A delivered fetch payload must not stay resident on the stored result.
|
||||||
|
|
||||||
|
BackgroundDataService kept the fetched body on the FetchResult it filed in
|
||||||
|
`completed_requests`, which is swept only hourly and capped at 500 entries by
|
||||||
|
count. For status records that is free; for a season schedule it is not. NCAA
|
||||||
|
football's 2026 schedule is 946 games, and on a 1GB Pi 3B+ the parsed payload
|
||||||
|
measured ~90MB -- a tenth of the board's memory, pinned for an hour after the
|
||||||
|
consumer had already been handed it.
|
||||||
|
|
||||||
|
The cache-hit path was the worse of the two. It runs once per update interval
|
||||||
|
per sport, mints a fresh request_id each time, and hands back whatever the
|
||||||
|
cache returns -- so a memory-tier miss (the tier is capped at 150 entries)
|
||||||
|
re-parses the payload from disk into a genuinely new object. Those accumulate
|
||||||
|
as separate copies rather than shared references, which is the staircase seen
|
||||||
|
in the field: RSS stepping up ~90MB per sport as seasons loaded and never
|
||||||
|
coming back down.
|
||||||
|
|
||||||
|
Releasing is safe because the payload is written to the cache under the
|
||||||
|
request's cache_key before the result is built, and that is where consumers
|
||||||
|
read it from -- the callback is handed the object directly and the plugins use
|
||||||
|
it only in passing before reading the cache back.
|
||||||
|
|
||||||
|
Requests submitted *without* a callback keep their payload: polling
|
||||||
|
get_result() is then the only way to collect it, so releasing would break that
|
||||||
|
contract.
|
||||||
|
|
||||||
|
And the release must happen after EVERY callback, not after each one. Callers
|
||||||
|
that joined an in-flight fetch share a single FetchResult, so releasing per
|
||||||
|
delivery strips the payload out from under everyone still queued -- see
|
||||||
|
TestJoinersAllGetTheData.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import MagicMock, Mock, patch
|
||||||
|
|
||||||
|
from src.background_data_service import BackgroundDataService
|
||||||
|
|
||||||
|
|
||||||
|
PAYLOAD = {"events": [{"id": f"g{i}"} for i in range(50)]}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cache():
|
||||||
|
m = MagicMock()
|
||||||
|
m.get.return_value = None
|
||||||
|
m.set.return_value = None
|
||||||
|
return m
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def service(cache):
|
||||||
|
svc = BackgroundDataService(cache, max_workers=2, request_timeout=5)
|
||||||
|
yield svc
|
||||||
|
svc.shutdown(wait=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait(service, req_id, timeout=5):
|
||||||
|
"""Wait for the result to be FILED.
|
||||||
|
|
||||||
|
Enough for anything that is true by the time the worker stores the result:
|
||||||
|
its success flag, its error, the cache write that happened during the
|
||||||
|
fetch.
|
||||||
|
"""
|
||||||
|
deadline = time.time() + timeout
|
||||||
|
while not service.is_request_complete(req_id) and time.time() < deadline:
|
||||||
|
time.sleep(0.02)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_release(service, req_id, timeout=5):
|
||||||
|
"""Wait for the payload to be RELEASED, which is strictly later.
|
||||||
|
|
||||||
|
The worker files the result, then runs the callback, then releases. So
|
||||||
|
is_request_complete() goes true while the callback still has not run --
|
||||||
|
waiting on it alone leaves a window in which `seen` is empty and the
|
||||||
|
payload is still resident, and the assertions race the worker. It passes
|
||||||
|
in practice only because a one-line callback usually beats the 20ms poll.
|
||||||
|
|
||||||
|
Release happens after the callback returns, so a released payload also
|
||||||
|
means the callback has finished: one wait covers both.
|
||||||
|
"""
|
||||||
|
deadline = time.time() + timeout
|
||||||
|
while time.time() < deadline:
|
||||||
|
result = service.get_result(req_id)
|
||||||
|
if result is not None and result.data is None:
|
||||||
|
return
|
||||||
|
time.sleep(0.02)
|
||||||
|
raise AssertionError(
|
||||||
|
f"payload for {req_id} was never released (callback may not have run)")
|
||||||
|
|
||||||
|
|
||||||
|
def _resp():
|
||||||
|
r = Mock()
|
||||||
|
r.json.return_value = PAYLOAD
|
||||||
|
r.raise_for_status.return_value = None
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
class TestFetchPath:
|
||||||
|
def test_callback_receives_the_payload_then_it_is_released(self, service, cache):
|
||||||
|
seen = {}
|
||||||
|
|
||||||
|
def callback(result):
|
||||||
|
# The consumer's one look at the data happens here.
|
||||||
|
seen['events'] = len(result.data['events'])
|
||||||
|
|
||||||
|
with patch.object(service.session, "get", return_value=_resp()):
|
||||||
|
req_id = service.submit_fetch_request(
|
||||||
|
sport="ncaa_fb", year=2026, url="https://example.com/s",
|
||||||
|
cache_key="ncaa_fb_2026", callback=callback, max_retries=0,
|
||||||
|
)
|
||||||
|
_wait_for_release(service, req_id)
|
||||||
|
|
||||||
|
assert seen['events'] == 50, "callback must still be handed the payload"
|
||||||
|
|
||||||
|
stored = service.get_result(req_id)
|
||||||
|
assert stored is not None
|
||||||
|
assert stored.success is True
|
||||||
|
assert stored.data is None, "payload must not stay on the stored result"
|
||||||
|
|
||||||
|
def test_nothing_is_lost_the_cache_holds_it(self, service, cache):
|
||||||
|
with patch.object(service.session, "get", return_value=_resp()):
|
||||||
|
req_id = service.submit_fetch_request(
|
||||||
|
sport="ncaa_fb", year=2026, url="https://example.com/s",
|
||||||
|
cache_key="ncaa_fb_2026", callback=lambda r: None, max_retries=0,
|
||||||
|
)
|
||||||
|
_wait(service, req_id)
|
||||||
|
|
||||||
|
cache.set.assert_called_once()
|
||||||
|
key, written = cache.set.call_args[0][:2]
|
||||||
|
assert key == "ncaa_fb_2026"
|
||||||
|
assert written == PAYLOAD, "the payload must be persisted before release"
|
||||||
|
|
||||||
|
def test_without_a_callback_the_payload_is_kept(self, service, cache):
|
||||||
|
# Polling get_result() is then the only delivery mechanism.
|
||||||
|
with patch.object(service.session, "get", return_value=_resp()):
|
||||||
|
req_id = service.submit_fetch_request(
|
||||||
|
sport="nfl", year=2026, url="https://example.com/s",
|
||||||
|
cache_key="nfl_2026", max_retries=0,
|
||||||
|
)
|
||||||
|
_wait(service, req_id)
|
||||||
|
|
||||||
|
assert service.get_result(req_id).data == PAYLOAD
|
||||||
|
|
||||||
|
def test_a_failed_fetch_still_records_its_error(self, service, cache):
|
||||||
|
with patch.object(service.session, "get", side_effect=Exception("boom")):
|
||||||
|
req_id = service.submit_fetch_request(
|
||||||
|
sport="nfl", year=2026, url="https://example.com/s",
|
||||||
|
cache_key="nfl_2026", callback=lambda r: None, max_retries=0,
|
||||||
|
)
|
||||||
|
_wait(service, req_id)
|
||||||
|
|
||||||
|
stored = service.get_result(req_id)
|
||||||
|
assert stored.success is False
|
||||||
|
assert stored.error is not None
|
||||||
|
|
||||||
|
|
||||||
|
class TestCacheHitPath:
|
||||||
|
def test_cache_hit_releases_after_the_callback(self, service, cache):
|
||||||
|
cache.get.return_value = PAYLOAD
|
||||||
|
seen = {}
|
||||||
|
|
||||||
|
req_id = service.submit_fetch_request(
|
||||||
|
sport="ncaa_fb", year=2026, url="https://example.com/s",
|
||||||
|
cache_key="ncaa_fb_2026",
|
||||||
|
callback=lambda r: seen.update(events=len(r.data['events'])),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert seen['events'] == 50
|
||||||
|
assert service.get_result(req_id).data is None
|
||||||
|
|
||||||
|
def test_repeated_cache_hits_do_not_accumulate_payloads(self, service, cache):
|
||||||
|
# The staircase: one entry per update interval per sport, each one
|
||||||
|
# potentially a freshly parsed copy after a memory-tier miss.
|
||||||
|
cache.get.return_value = PAYLOAD
|
||||||
|
|
||||||
|
for _ in range(25):
|
||||||
|
service.submit_fetch_request(
|
||||||
|
sport="ncaa_fb", year=2026, url="https://example.com/s",
|
||||||
|
cache_key="ncaa_fb_2026", callback=lambda r: None,
|
||||||
|
)
|
||||||
|
|
||||||
|
retained = [r for r in service.completed_requests.values() if r.data is not None]
|
||||||
|
assert retained == [], f"{len(retained)} payloads still resident"
|
||||||
|
|
||||||
|
def test_cache_hit_without_a_callback_is_unchanged(self, service, cache):
|
||||||
|
cache.get.return_value = PAYLOAD
|
||||||
|
req_id = service.submit_fetch_request(
|
||||||
|
sport="nfl", year=2026, url="https://example.com/s",
|
||||||
|
cache_key="nfl_2026",
|
||||||
|
)
|
||||||
|
assert service.get_result(req_id).data == PAYLOAD
|
||||||
|
|
||||||
|
|
||||||
|
class TestJoinersAllGetTheData:
|
||||||
|
"""Deduplicated callers share one FetchResult; releasing between them
|
||||||
|
empties it for the rest.
|
||||||
|
|
||||||
|
Not a corner case. A sport's recent, upcoming and live managers all ask for
|
||||||
|
the same season schedule, so the second and third are joiners on almost
|
||||||
|
every cycle. Releasing inside the delivery loop handed the payload to
|
||||||
|
whichever ran first and gave the others `result.data is None`.
|
||||||
|
|
||||||
|
The consequence was worse than a quiet degradation, because consumers do
|
||||||
|
`result.data.get('events')`: they raised AttributeError, the delivery loop
|
||||||
|
caught it, and the whole failure surfaced as a single
|
||||||
|
"Error in callback for request ..." line while that manager silently never
|
||||||
|
received its schedule.
|
||||||
|
"""
|
||||||
|
|
||||||
|
class _BlockingSession:
|
||||||
|
"""Holds the fetch open so a second submit lands while in flight."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.release = threading.Event()
|
||||||
|
self.started = threading.Event()
|
||||||
|
|
||||||
|
def get(self, *a, **k):
|
||||||
|
self.started.set()
|
||||||
|
self.release.wait(timeout=5)
|
||||||
|
return _resp()
|
||||||
|
|
||||||
|
def test_every_joiner_is_handed_the_payload(self, service):
|
||||||
|
session = self._BlockingSession()
|
||||||
|
seen = {}
|
||||||
|
|
||||||
|
def record(name):
|
||||||
|
# Read it the way the sport managers do. `result.data['events']`
|
||||||
|
# would raise TypeError on None; `.get` raises AttributeError,
|
||||||
|
# which is the error actually seen in the field.
|
||||||
|
def cb(result):
|
||||||
|
seen[name] = result.data.get('events') if result.data else None
|
||||||
|
return cb
|
||||||
|
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026",
|
||||||
|
callback=record("first"), max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
|
||||||
|
joined = service.submit_fetch_request(
|
||||||
|
sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026",
|
||||||
|
callback=record("second"), max_retries=0)
|
||||||
|
# Without coalescing these are two independent fetches that each
|
||||||
|
# own their result, and the test proves nothing about sharing.
|
||||||
|
assert joined == first, "the joiner should share the in-flight id"
|
||||||
|
|
||||||
|
session.release.set()
|
||||||
|
_wait(service, first)
|
||||||
|
|
||||||
|
deadline = time.time() + 5
|
||||||
|
while len(seen) < 2 and time.time() < deadline:
|
||||||
|
time.sleep(0.02)
|
||||||
|
|
||||||
|
assert set(seen) == {"first", "second"}, f"both must be called, got {seen}"
|
||||||
|
for name, events in seen.items():
|
||||||
|
assert events is not None, (
|
||||||
|
f"{name!r} was handed a released payload: the result was "
|
||||||
|
f"emptied before every callback had been delivered")
|
||||||
|
assert len(events) == 50, f"{name!r} got {events!r}"
|
||||||
|
|
||||||
|
def test_the_payload_is_still_released_once_they_have_all_had_it(self, service):
|
||||||
|
"""The memory fix must survive the ordering fix."""
|
||||||
|
session = self._BlockingSession()
|
||||||
|
|
||||||
|
with patch.object(service, "session", session):
|
||||||
|
first = service.submit_fetch_request(
|
||||||
|
sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
assert session.started.wait(timeout=5)
|
||||||
|
service.submit_fetch_request(
|
||||||
|
sport="nhl", year=2026, url="https://x/s", cache_key="nhl_2026",
|
||||||
|
callback=lambda r: None, max_retries=0)
|
||||||
|
session.release.set()
|
||||||
|
_wait_for_release(service, first)
|
||||||
|
|
||||||
|
stored = service.get_result(first)
|
||||||
|
assert stored is not None and stored.data is None, (
|
||||||
|
"the payload must still be dropped once every callback has run")
|
||||||
@@ -0,0 +1,365 @@
|
|||||||
|
"""
|
||||||
|
Tests for src/base_odds_manager.py (BaseOddsManager).
|
||||||
|
|
||||||
|
Covers get_odds validation/caching/URL construction, the null-safe
|
||||||
|
_extract_espn_data fix (ESPN sends explicit JSON nulls for absent sides),
|
||||||
|
the no_odds sentinel, stale-cache fallback on request failure,
|
||||||
|
is_odds_available's ML-blind truth table, the fixed format_odds_summary
|
||||||
|
gate (money-line-only odds now format), get_odds_for_games, and
|
||||||
|
configuration loading.
|
||||||
|
|
||||||
|
No real network: requests.Session.get is always patched. The odds path sends
|
||||||
|
its requests through a session so it can identify itself to ESPN, so patching
|
||||||
|
the module-level requests.get would no longer intercept anything.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from src.base_odds_manager import BaseOddsManager
|
||||||
|
|
||||||
|
|
||||||
|
FULL_ITEM = {
|
||||||
|
'details': 'DAL -3.5',
|
||||||
|
'overUnder': 47.5,
|
||||||
|
'spread': -3.5,
|
||||||
|
'homeTeamOdds': {'moneyLine': -150, 'current': {'pointSpread': {'value': -3.5}}},
|
||||||
|
'awayTeamOdds': {'moneyLine': 130, 'current': {'pointSpread': {'value': 3.5}}},
|
||||||
|
}
|
||||||
|
|
||||||
|
FULL_EXTRACTED = {
|
||||||
|
'details': 'DAL -3.5',
|
||||||
|
'over_under': 47.5,
|
||||||
|
'spread': -3.5,
|
||||||
|
'home_team_odds': {'money_line': -150, 'spread_odds': -3.5},
|
||||||
|
'away_team_odds': {'money_line': 130, 'spread_odds': 3.5},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _make_response(payload):
|
||||||
|
response = MagicMock()
|
||||||
|
response.json.return_value = payload
|
||||||
|
response.raise_for_status.return_value = None
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cache_manager():
|
||||||
|
cm = MagicMock()
|
||||||
|
# A bare MagicMock returns truthy Mocks from every call, so every
|
||||||
|
# get_odds() would look like a cache hit. Explicitly wire a miss.
|
||||||
|
cm.get_with_auto_strategy.return_value = None
|
||||||
|
return cm
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def manager(cache_manager):
|
||||||
|
return BaseOddsManager(cache_manager)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def mock_get():
|
||||||
|
with patch('src.base_odds_manager.requests.Session.get') as m:
|
||||||
|
m.return_value = _make_response({'items': [dict(FULL_ITEM)]})
|
||||||
|
yield m
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# get_odds
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestGetOdds:
|
||||||
|
def test_none_sport_raises(self, manager):
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
manager.get_odds(None, 'nfl', '1')
|
||||||
|
|
||||||
|
def test_none_league_raises(self, manager):
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
manager.get_odds('football', None, '1')
|
||||||
|
|
||||||
|
def test_cache_key_and_url(self, manager, cache_manager, mock_get):
|
||||||
|
manager.get_odds('football', 'nfl', '401')
|
||||||
|
|
||||||
|
cache_manager.get_with_auto_strategy.assert_called_once_with(
|
||||||
|
'odds_espn_football_nfl_401')
|
||||||
|
url = mock_get.call_args[0][0]
|
||||||
|
# Event id appears twice: /events/<id>/competitions/<id>/odds
|
||||||
|
assert '/events/401/competitions/401/odds' in url
|
||||||
|
assert url == ('https://sports.core.api.espn.com/v2/sports/football/'
|
||||||
|
'leagues/nfl/events/401/competitions/401/odds')
|
||||||
|
# The number matters less than the property: a single stalled request
|
||||||
|
# must not be able to consume the plugin executor's 30s operation
|
||||||
|
# budget, since odds are fetched per live game inside update().
|
||||||
|
assert mock_get.call_args.kwargs['timeout'] == 5
|
||||||
|
assert mock_get.call_args.kwargs['timeout'] < 30
|
||||||
|
|
||||||
|
def test_ncaa_fb_maps_to_college_football(self, manager, mock_get):
|
||||||
|
manager.get_odds('football', 'ncaa_fb', '401')
|
||||||
|
|
||||||
|
url = mock_get.call_args[0][0]
|
||||||
|
assert '/leagues/college-football/' in url
|
||||||
|
|
||||||
|
def test_unknown_league_passes_through(self, manager, mock_get):
|
||||||
|
manager.get_odds('football', 'xfl', '401')
|
||||||
|
|
||||||
|
assert '/leagues/xfl/' in mock_get.call_args[0][0]
|
||||||
|
|
||||||
|
def test_cache_hit_skips_http(self, manager, cache_manager, mock_get):
|
||||||
|
cache_manager.get_with_auto_strategy.return_value = {'spread': -3.0}
|
||||||
|
|
||||||
|
result = manager.get_odds('football', 'nfl', '401')
|
||||||
|
|
||||||
|
assert result == {'spread': -3.0}
|
||||||
|
mock_get.assert_not_called()
|
||||||
|
|
||||||
|
def test_cached_no_odds_sentinel_returned_verbatim(
|
||||||
|
self, manager, cache_manager, mock_get):
|
||||||
|
cache_manager.get_with_auto_strategy.return_value = {'no_odds': True}
|
||||||
|
|
||||||
|
result = manager.get_odds('football', 'nfl', '401')
|
||||||
|
|
||||||
|
assert result == {'no_odds': True}
|
||||||
|
mock_get.assert_not_called()
|
||||||
|
assert manager.is_odds_available(result) is False
|
||||||
|
|
||||||
|
def test_success_caches_extracted_data_with_interval_ttl(
|
||||||
|
self, manager, cache_manager, mock_get):
|
||||||
|
result = manager.get_odds('football', 'nfl', '401',
|
||||||
|
update_interval_seconds=100)
|
||||||
|
|
||||||
|
assert result == FULL_EXTRACTED
|
||||||
|
cache_manager.set.assert_called_once_with(
|
||||||
|
'odds_espn_football_nfl_401', FULL_EXTRACTED, ttl=100)
|
||||||
|
|
||||||
|
def test_no_odds_caches_sentinel(self, manager, cache_manager, mock_get):
|
||||||
|
mock_get.return_value = _make_response({'count': 0, 'items': []})
|
||||||
|
|
||||||
|
result = manager.get_odds('football', 'nfl', '401')
|
||||||
|
|
||||||
|
assert result is None
|
||||||
|
cache_manager.set.assert_called_once_with(
|
||||||
|
'odds_espn_football_nfl_401', {'no_odds': True}, ttl=3600)
|
||||||
|
|
||||||
|
def test_zero_interval_falls_back_to_default(
|
||||||
|
self, manager, cache_manager, mock_get):
|
||||||
|
# Quirk pin: `update_interval_seconds or self.update_interval`
|
||||||
|
# treats an explicit 0 as falsy, so the 3600 default wins.
|
||||||
|
manager.get_odds('football', 'nfl', '401', update_interval_seconds=0)
|
||||||
|
|
||||||
|
assert cache_manager.set.call_args.kwargs['ttl'] == 3600
|
||||||
|
|
||||||
|
def test_request_exception_falls_back_to_stale_cache(
|
||||||
|
self, manager, cache_manager, mock_get):
|
||||||
|
cache_manager.get_with_auto_strategy.side_effect = [
|
||||||
|
None, {'stale': True}]
|
||||||
|
mock_get.side_effect = requests.exceptions.RequestException('boom')
|
||||||
|
|
||||||
|
result = manager.get_odds('football', 'nfl', '401')
|
||||||
|
|
||||||
|
assert result == {'stale': True}
|
||||||
|
assert cache_manager.get_with_auto_strategy.call_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _extract_espn_data
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestExtractEspnData:
|
||||||
|
def test_full_item_extracts_all_fields(self, manager):
|
||||||
|
result = manager._extract_espn_data({'items': [dict(FULL_ITEM)]})
|
||||||
|
assert result == FULL_EXTRACTED
|
||||||
|
|
||||||
|
def test_explicit_nulls_do_not_raise(self, manager):
|
||||||
|
# Post-fix: ESPN sends explicit JSON nulls for absent sides
|
||||||
|
# ("homeTeamOdds": null, "current": null); extraction must not
|
||||||
|
# raise and yields None fields.
|
||||||
|
payload = {'items': [{
|
||||||
|
'homeTeamOdds': None,
|
||||||
|
'awayTeamOdds': {'moneyLine': 150, 'current': None},
|
||||||
|
}]}
|
||||||
|
|
||||||
|
result = manager._extract_espn_data(payload)
|
||||||
|
|
||||||
|
assert result is not None
|
||||||
|
assert result['home_team_odds']['money_line'] is None
|
||||||
|
assert result['home_team_odds']['spread_odds'] is None
|
||||||
|
assert result['away_team_odds']['money_line'] == 150
|
||||||
|
assert result['away_team_odds']['spread_odds'] is None
|
||||||
|
|
||||||
|
def test_valid_empty_response_returns_none(self, manager):
|
||||||
|
assert manager._extract_espn_data({'count': 0, 'items': []}) is None
|
||||||
|
|
||||||
|
def test_unexpected_structure_returns_none(self, manager):
|
||||||
|
assert manager._extract_espn_data({'unexpected': True}) is None
|
||||||
|
|
||||||
|
def test_item_without_odds_fields_cached_as_data_not_sentinel(
|
||||||
|
self, manager, cache_manager, mock_get):
|
||||||
|
# Characterization pin: an item with no odds fields still extracts
|
||||||
|
# to a truthy dict of all-None values, so get_odds caches it as
|
||||||
|
# real data (NOT the no_odds sentinel) — but is_odds_available
|
||||||
|
# correctly reports False for it.
|
||||||
|
mock_get.return_value = _make_response({'items': [{}]})
|
||||||
|
|
||||||
|
result = manager.get_odds('football', 'nfl', '401')
|
||||||
|
|
||||||
|
assert result == {
|
||||||
|
'details': None,
|
||||||
|
'over_under': None,
|
||||||
|
'spread': None,
|
||||||
|
'home_team_odds': {'money_line': None, 'spread_odds': None},
|
||||||
|
'away_team_odds': {'money_line': None, 'spread_odds': None},
|
||||||
|
}
|
||||||
|
cache_manager.set.assert_called_once_with(
|
||||||
|
'odds_espn_football_nfl_401', result, ttl=3600)
|
||||||
|
assert manager.is_odds_available(result) is False
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# is_odds_available
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestIsOddsAvailable:
|
||||||
|
def test_none_is_false(self, manager):
|
||||||
|
assert manager.is_odds_available(None) is False
|
||||||
|
|
||||||
|
def test_empty_dict_is_false(self, manager):
|
||||||
|
assert manager.is_odds_available({}) is False
|
||||||
|
|
||||||
|
def test_no_odds_sentinel_is_false(self, manager):
|
||||||
|
assert manager.is_odds_available({'no_odds': True}) is False
|
||||||
|
|
||||||
|
def test_spread_is_true(self, manager):
|
||||||
|
assert manager.is_odds_available({'spread': -3.5}) is True
|
||||||
|
|
||||||
|
def test_over_under_is_true(self, manager):
|
||||||
|
assert manager.is_odds_available({'over_under': 47.5}) is True
|
||||||
|
|
||||||
|
def test_nested_home_spread_odds_is_true(self, manager):
|
||||||
|
assert manager.is_odds_available(
|
||||||
|
{'home_team_odds': {'spread_odds': -3.5}}) is True
|
||||||
|
|
||||||
|
def test_nested_away_spread_odds_is_true(self, manager):
|
||||||
|
assert manager.is_odds_available(
|
||||||
|
{'away_team_odds': {'spread_odds': 3.5}}) is True
|
||||||
|
|
||||||
|
def test_moneyline_only_is_false(self, manager):
|
||||||
|
# Pinned ML-blind contract: is_odds_available ignores money lines
|
||||||
|
# (its callers decide whether to render an odds widget). Note that
|
||||||
|
# format_odds_summary deliberately uses a DIFFERENT gate — it will
|
||||||
|
# still format money-line-only odds (see TestFormatOddsSummary).
|
||||||
|
ml_only = {
|
||||||
|
'home_team_odds': {'money_line': -120},
|
||||||
|
'away_team_odds': {'money_line': 100},
|
||||||
|
}
|
||||||
|
assert manager.is_odds_available(ml_only) is False
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# format_odds_summary (fixed gate: empty / no_odds only)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestFormatOddsSummary:
|
||||||
|
def test_moneyline_only_formats(self, manager):
|
||||||
|
result = manager.format_odds_summary({
|
||||||
|
'home_team_odds': {'money_line': -120},
|
||||||
|
'away_team_odds': {'money_line': 100},
|
||||||
|
})
|
||||||
|
assert result == 'Home ML: -120 | Away ML: 100'
|
||||||
|
|
||||||
|
def test_full_data_formats_all_parts(self, manager):
|
||||||
|
result = manager.format_odds_summary(FULL_EXTRACTED)
|
||||||
|
assert result == 'Spread: -3.5 | O/U: 47.5 | Home ML: -150 | Away ML: 130'
|
||||||
|
|
||||||
|
def test_none_is_no_odds(self, manager):
|
||||||
|
assert manager.format_odds_summary(None) == 'No odds available'
|
||||||
|
|
||||||
|
def test_empty_dict_is_no_odds(self, manager):
|
||||||
|
assert manager.format_odds_summary({}) == 'No odds available'
|
||||||
|
|
||||||
|
def test_no_odds_sentinel_is_no_odds(self, manager):
|
||||||
|
assert manager.format_odds_summary(
|
||||||
|
{'no_odds': True}) == 'No odds available'
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# get_odds_for_games
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestGetOddsForGames:
|
||||||
|
def test_missing_fields_get_none_odds_without_http(self, manager, mock_get):
|
||||||
|
games = [
|
||||||
|
{'sport': 'football'},
|
||||||
|
{'league': 'nfl'},
|
||||||
|
{'id': '9'},
|
||||||
|
{},
|
||||||
|
]
|
||||||
|
|
||||||
|
result = manager.get_odds_for_games(games)
|
||||||
|
|
||||||
|
assert all(g['odds'] is None for g in result)
|
||||||
|
mock_get.assert_not_called()
|
||||||
|
|
||||||
|
def test_per_game_exception_continues_loop(self, manager, monkeypatch):
|
||||||
|
def fake_get_odds(sport, league, event_id,
|
||||||
|
update_interval_seconds=None):
|
||||||
|
if event_id == 'bad':
|
||||||
|
raise RuntimeError('boom')
|
||||||
|
return {'spread': -1.0}
|
||||||
|
|
||||||
|
monkeypatch.setattr(manager, 'get_odds', fake_get_odds)
|
||||||
|
games = [
|
||||||
|
{'sport': 'football', 'league': 'nfl', 'id': 'bad'},
|
||||||
|
{'sport': 'football', 'league': 'nfl', 'id': 'ok'},
|
||||||
|
]
|
||||||
|
|
||||||
|
result = manager.get_odds_for_games(games)
|
||||||
|
|
||||||
|
assert len(result) == 2
|
||||||
|
assert result[0]['odds'] is None
|
||||||
|
assert result[1]['odds'] == {'spread': -1.0}
|
||||||
|
|
||||||
|
def test_input_dicts_mutated_in_place_and_returned(self, manager, mock_get):
|
||||||
|
# Pin: get_odds_for_games mutates the caller's game dicts in place
|
||||||
|
# and returns the same objects, not copies.
|
||||||
|
game = {'sport': 'football', 'league': 'nfl', 'id': '401'}
|
||||||
|
|
||||||
|
result = manager.get_odds_for_games([game])
|
||||||
|
|
||||||
|
assert result[0] is game
|
||||||
|
assert game['odds'] == FULL_EXTRACTED
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# _load_configuration
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TestLoadConfiguration:
|
||||||
|
def test_loads_values_from_config(self, cache_manager):
|
||||||
|
config_manager = MagicMock()
|
||||||
|
config_manager.get_config.return_value = {
|
||||||
|
'base_odds_manager': {
|
||||||
|
'update_interval': 100,
|
||||||
|
'timeout': 5,
|
||||||
|
'cache_ttl': 42,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
manager = BaseOddsManager(cache_manager, config_manager=config_manager)
|
||||||
|
|
||||||
|
assert manager.update_interval == 100
|
||||||
|
# Key/attr mismatch pin: the config key is 'timeout' but the
|
||||||
|
# attribute is request_timeout.
|
||||||
|
assert manager.request_timeout == 5
|
||||||
|
assert manager.cache_ttl == 42
|
||||||
|
|
||||||
|
def test_get_config_raising_keeps_defaults(self, cache_manager):
|
||||||
|
config_manager = MagicMock()
|
||||||
|
config_manager.get_config.side_effect = RuntimeError('boom')
|
||||||
|
|
||||||
|
manager = BaseOddsManager(cache_manager, config_manager=config_manager)
|
||||||
|
|
||||||
|
assert manager.update_interval == 3600
|
||||||
|
assert manager.request_timeout == 5
|
||||||
|
assert manager.cache_ttl == 1800
|
||||||
@@ -78,10 +78,20 @@ class TestInstanceVariable:
|
|||||||
instance_duration=[30])
|
instance_duration=[30])
|
||||||
assert plugin.get_display_duration() == 20.0
|
assert plugin.get_display_duration() == 20.0
|
||||||
|
|
||||||
def test_bool_true_is_one_second(self):
|
def test_bool_true_falls_through_like_any_non_number(self):
|
||||||
# Characterized quirk: bool is an int subclass, so display_duration =
|
# bool is an int subclass, but a boolean is not a duration: True
|
||||||
# True passes the isinstance((int, float)) branch and returns 1.0.
|
# must NOT read as 1 second — it falls through to config/default.
|
||||||
assert make_plugin(instance_duration=True).get_display_duration() == 1.0
|
assert make_plugin(instance_duration=True).get_display_duration() == 15.0
|
||||||
|
|
||||||
|
def test_bool_true_falls_through_to_config(self):
|
||||||
|
plugin = make_plugin(config={"display_duration": 20},
|
||||||
|
instance_duration=True)
|
||||||
|
assert plugin.get_display_duration() == 20.0
|
||||||
|
|
||||||
|
def test_bool_false_still_falls_through(self):
|
||||||
|
plugin = make_plugin(config={"display_duration": 20},
|
||||||
|
instance_duration=False)
|
||||||
|
assert plugin.get_display_duration() == 20.0
|
||||||
|
|
||||||
|
|
||||||
class TestConfigFallback:
|
class TestConfigFallback:
|
||||||
@@ -108,3 +118,29 @@ class TestConfigFallback:
|
|||||||
|
|
||||||
def test_config_none_uses_default(self):
|
def test_config_none_uses_default(self):
|
||||||
assert make_plugin({"display_duration": None}).get_display_duration() == 15.0
|
assert make_plugin({"display_duration": None}).get_display_duration() == 15.0
|
||||||
|
|
||||||
|
def test_config_bool_uses_default(self):
|
||||||
|
assert make_plugin({"display_duration": True}).get_display_duration() == 15.0
|
||||||
|
assert make_plugin({"display_duration": False}).get_display_duration() == 15.0
|
||||||
|
|
||||||
|
|
||||||
|
class TestValidateConfigDuration:
|
||||||
|
# validate_config must agree with get_display_duration about what a
|
||||||
|
# valid duration is — a config it accepts must not then be rejected
|
||||||
|
# (or silently defaulted) when the duration is actually read.
|
||||||
|
|
||||||
|
def test_positive_number_valid(self):
|
||||||
|
assert make_plugin({"display_duration": 20}).validate_config() is True
|
||||||
|
|
||||||
|
def test_zero_and_negative_invalid(self):
|
||||||
|
assert make_plugin({"display_duration": 0}).validate_config() is False
|
||||||
|
assert make_plugin({"display_duration": -5}).validate_config() is False
|
||||||
|
|
||||||
|
def test_bool_invalid(self):
|
||||||
|
# bool is an int subclass; True would otherwise pass as "positive
|
||||||
|
# number" here while get_display_duration rejects it.
|
||||||
|
assert make_plugin({"display_duration": True}).validate_config() is False
|
||||||
|
assert make_plugin({"display_duration": False}).validate_config() is False
|
||||||
|
|
||||||
|
def test_missing_duration_valid(self):
|
||||||
|
assert make_plugin({}).validate_config() is True
|
||||||
|
|||||||
@@ -0,0 +1,146 @@
|
|||||||
|
"""Tests that one cache directory gets one cleanup thread per process.
|
||||||
|
|
||||||
|
The sweep lists a directory and deletes from it, so a second thread over the
|
||||||
|
same directory only duplicates the scan. Nothing enforced that: every
|
||||||
|
CacheManager started its own, and since the loop closes over `self`, a
|
||||||
|
discarded manager could never be collected -- its thread stayed alive and
|
||||||
|
re-scanned the same directory every 24 hours for the life of the process.
|
||||||
|
|
||||||
|
On the dev rig a display process carried three, for one cache directory:
|
||||||
|
|
||||||
|
14:22:59.954 display_controller (the real one)
|
||||||
|
14:22:59.973 startup validation, run 1 (discarded)
|
||||||
|
14:23:01.055 startup validation, run 2 (discarded)
|
||||||
|
|
||||||
|
Startup validation runs twice and built a throwaway manager each time, purely
|
||||||
|
to read a directory path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import threading
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.cache_manager import CacheManager
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clean_registry():
|
||||||
|
CacheManager._cleanup_owners.clear()
|
||||||
|
yield
|
||||||
|
for owner in list(CacheManager._cleanup_owners.values()):
|
||||||
|
owner.stop_cleanup_thread()
|
||||||
|
CacheManager._cleanup_owners.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def _live_cleanup_threads():
|
||||||
|
return [t for t in threading.enumerate()
|
||||||
|
if t.name == 'DiskCacheCleanup' and t.is_alive()]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def manager(tmp_path, monkeypatch):
|
||||||
|
"""A CacheManager pinned to a temp dir, so tests never touch the real one."""
|
||||||
|
monkeypatch.setattr(CacheManager, '_get_writable_cache_dir',
|
||||||
|
lambda self: str(tmp_path))
|
||||||
|
return CacheManager
|
||||||
|
|
||||||
|
|
||||||
|
class TestOneThreadPerDirectory:
|
||||||
|
def test_a_single_manager_starts_one(self, manager):
|
||||||
|
before = len(_live_cleanup_threads())
|
||||||
|
m = manager()
|
||||||
|
try:
|
||||||
|
assert len(_live_cleanup_threads()) == before + 1
|
||||||
|
finally:
|
||||||
|
m.stop_cleanup_thread()
|
||||||
|
|
||||||
|
def test_three_managers_still_start_one(self, manager):
|
||||||
|
# Exactly the rig's shape: the real manager plus two throwaways.
|
||||||
|
before = len(_live_cleanup_threads())
|
||||||
|
managers = [manager() for _ in range(3)]
|
||||||
|
try:
|
||||||
|
assert len(_live_cleanup_threads()) == before + 1
|
||||||
|
finally:
|
||||||
|
for m in managers:
|
||||||
|
m.stop_cleanup_thread()
|
||||||
|
|
||||||
|
def test_the_first_one_owns_it(self, manager):
|
||||||
|
first, second = manager(), manager()
|
||||||
|
try:
|
||||||
|
assert CacheManager._cleanup_owners[first.cache_dir] is first
|
||||||
|
assert second._cleanup_thread is None
|
||||||
|
finally:
|
||||||
|
first.stop_cleanup_thread()
|
||||||
|
second.stop_cleanup_thread()
|
||||||
|
|
||||||
|
def test_the_survivor_can_take_over(self, manager):
|
||||||
|
first = manager()
|
||||||
|
first.stop_cleanup_thread()
|
||||||
|
assert not _live_cleanup_threads()
|
||||||
|
|
||||||
|
second = manager()
|
||||||
|
try:
|
||||||
|
# Ownership was released, so the directory is swept again rather
|
||||||
|
# than being left permanently unclaimed by a dead owner.
|
||||||
|
assert len(_live_cleanup_threads()) == 1
|
||||||
|
assert CacheManager._cleanup_owners[second.cache_dir] is second
|
||||||
|
finally:
|
||||||
|
second.stop_cleanup_thread()
|
||||||
|
|
||||||
|
def test_stopping_a_non_owner_does_not_unclaim_the_directory(self, manager):
|
||||||
|
first, second = manager(), manager()
|
||||||
|
try:
|
||||||
|
second.stop_cleanup_thread() # never owned it
|
||||||
|
assert CacheManager._cleanup_owners[first.cache_dir] is first
|
||||||
|
assert len(_live_cleanup_threads()) == 1
|
||||||
|
finally:
|
||||||
|
first.stop_cleanup_thread()
|
||||||
|
|
||||||
|
def test_separate_directories_get_separate_threads(self, tmp_path, monkeypatch):
|
||||||
|
a, b = tmp_path / 'a', tmp_path / 'b'
|
||||||
|
a.mkdir()
|
||||||
|
b.mkdir()
|
||||||
|
dirs = iter([str(a), str(b)])
|
||||||
|
monkeypatch.setattr(CacheManager, '_get_writable_cache_dir',
|
||||||
|
lambda self: next(dirs))
|
||||||
|
first, second = CacheManager(), CacheManager()
|
||||||
|
try:
|
||||||
|
assert first.cache_dir != second.cache_dir
|
||||||
|
assert len(_live_cleanup_threads()) == 2
|
||||||
|
finally:
|
||||||
|
first.stop_cleanup_thread()
|
||||||
|
second.stop_cleanup_thread()
|
||||||
|
|
||||||
|
def test_no_thread_leaks_across_many_constructions(self, manager):
|
||||||
|
before = len(_live_cleanup_threads())
|
||||||
|
made = [manager() for _ in range(12)]
|
||||||
|
try:
|
||||||
|
assert len(_live_cleanup_threads()) == before + 1
|
||||||
|
finally:
|
||||||
|
for m in made:
|
||||||
|
m.stop_cleanup_thread()
|
||||||
|
assert len(_live_cleanup_threads()) == before
|
||||||
|
|
||||||
|
|
||||||
|
class TestValidatorDoesNotBuildItsOwn:
|
||||||
|
def test_it_uses_the_cache_manager_it_is_given(self, manager):
|
||||||
|
from src.startup_validator import StartupValidator
|
||||||
|
|
||||||
|
shared = manager()
|
||||||
|
try:
|
||||||
|
before = len(_live_cleanup_threads())
|
||||||
|
v = StartupValidator(config_manager=object(), cache_manager=shared)
|
||||||
|
v._validate_cache_directory()
|
||||||
|
assert len(_live_cleanup_threads()) == before, (
|
||||||
|
"validation started another cleanup thread")
|
||||||
|
finally:
|
||||||
|
shared.stop_cleanup_thread()
|
||||||
|
|
||||||
|
def test_without_one_it_cleans_up_after_itself(self, manager):
|
||||||
|
from src.startup_validator import StartupValidator
|
||||||
|
|
||||||
|
before = len(_live_cleanup_threads())
|
||||||
|
v = StartupValidator(config_manager=object())
|
||||||
|
v._validate_cache_directory()
|
||||||
|
assert len(_live_cleanup_threads()) == before, (
|
||||||
|
"the fallback manager left its cleanup thread running")
|
||||||
@@ -458,3 +458,26 @@ class TestDiskCacheWriteEconomy:
|
|||||||
cache = DiskCache(cache_dir=str(tmp_path))
|
cache = DiskCache(cache_dir=str(tmp_path))
|
||||||
cache.set("k", {"when": datetime(2026, 7, 12, 10, 30)})
|
cache.set("k", {"when": datetime(2026, 7, 12, 10, 30)})
|
||||||
assert cache.get("k") == {"when": "2026-07-12T10:30:00"}
|
assert cache.get("k") == {"when": "2026-07-12T10:30:00"}
|
||||||
|
|
||||||
|
|
||||||
|
# --- the ceiling has to hold between cleanup sweeps ---------------------------
|
||||||
|
|
||||||
|
def test_memory_cache_enforces_ceiling_on_every_write():
|
||||||
|
"""_cleanup_memory_cache only runs every cleanup_interval seconds (300 by
|
||||||
|
default). If set() accepted entries without bound in between, a burst could
|
||||||
|
take the cache far past max_size -- which is the unbounded growth the limit
|
||||||
|
exists to prevent, and on a 1GB board the difference between a bounded cache
|
||||||
|
and a Pi that cannot fork.
|
||||||
|
"""
|
||||||
|
from src.cache.memory_cache import MemoryCache
|
||||||
|
|
||||||
|
cache = MemoryCache(max_size=150, cleanup_interval=300.0)
|
||||||
|
for i in range(1000):
|
||||||
|
cache.set(f"k{i}", {"v": i})
|
||||||
|
|
||||||
|
assert len(cache._cache) <= 150
|
||||||
|
# The timestamp map has to be evicted alongside the values, or it becomes
|
||||||
|
# the leak instead.
|
||||||
|
assert len(cache._timestamps) <= 150
|
||||||
|
assert cache.get("k999") is not None, "the newest write must survive"
|
||||||
|
assert cache.get("k0") is None, "the oldest must be the one evicted"
|
||||||
|
|||||||
@@ -0,0 +1,188 @@
|
|||||||
|
"""Tests that abandoned cache temp files get collected.
|
||||||
|
|
||||||
|
DiskCache.set() writes through tempfile.mkstemp and os.replace, removing its
|
||||||
|
own temp file in a finally. That covers a failed write, but not a process that
|
||||||
|
dies between the two -- a SIGKILL, a lost restart race, a power cut, all
|
||||||
|
ordinary on a Pi. Nothing collected what was left behind: the temp names are
|
||||||
|
".<key>.json.<random>", and the expiry sweep only listed names ending in
|
||||||
|
.json, so they accumulated for as long as the card had been in service.
|
||||||
|
|
||||||
|
Measured on a live rig before this fix: 76 orphans totalling 1,050 MB -- 81%
|
||||||
|
of the entire cache directory -- the oldest six months old.
|
||||||
|
|
||||||
|
The predicate that decides what to delete is tested harder than the sweep
|
||||||
|
itself, because a false positive here destroys real data.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.cache.disk_cache import DiskCache, _ORPHAN_TEMP_MAX_AGE_SECONDS
|
||||||
|
|
||||||
|
|
||||||
|
class FakeStrategy:
|
||||||
|
@staticmethod
|
||||||
|
def get_data_type_from_key(key):
|
||||||
|
return 'default'
|
||||||
|
|
||||||
|
|
||||||
|
POLICIES = {'default': 30}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cache(tmp_path):
|
||||||
|
return DiskCache(str(tmp_path))
|
||||||
|
|
||||||
|
|
||||||
|
def _age(path, seconds):
|
||||||
|
old = time.time() - seconds
|
||||||
|
os.utime(path, (old, old))
|
||||||
|
|
||||||
|
|
||||||
|
def _write(tmp_path, name, body='{}'):
|
||||||
|
p = tmp_path / name
|
||||||
|
p.write_text(body, encoding='utf-8')
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
class TestWhatCountsAsAnOrphan:
|
||||||
|
@pytest.mark.parametrize('name', [
|
||||||
|
'.weather.json.a1b2c3d4',
|
||||||
|
'.odds_espn_football_nfl_401.json.xyz00000',
|
||||||
|
'.a.json.b',
|
||||||
|
])
|
||||||
|
def test_our_temp_files_are_orphans(self, name):
|
||||||
|
assert DiskCache._is_orphaned_temp(name)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('name', [
|
||||||
|
'weather.json', # real data
|
||||||
|
'.weather.json', # a dotted key that completed
|
||||||
|
'.gitignore', # not ours
|
||||||
|
'.hidden', # not ours
|
||||||
|
'weather.json.bak', # no leading dot: someone else's
|
||||||
|
'.json.abc', # no key between the dot and .json.
|
||||||
|
'.weather.json.', # no random component
|
||||||
|
'notes.txt',
|
||||||
|
])
|
||||||
|
def test_everything_else_is_left_alone(self, name):
|
||||||
|
assert not DiskCache._is_orphaned_temp(name)
|
||||||
|
|
||||||
|
def test_the_names_set_actually_creates_are_matched(self, cache, tmp_path):
|
||||||
|
"""Guard against the predicate and the writer drifting apart."""
|
||||||
|
created = []
|
||||||
|
real = os.replace
|
||||||
|
|
||||||
|
def capture(src, dst):
|
||||||
|
created.append(os.path.basename(src))
|
||||||
|
return real(src, dst)
|
||||||
|
|
||||||
|
import src.cache.disk_cache as mod
|
||||||
|
mod.os.replace = capture
|
||||||
|
try:
|
||||||
|
cache.set('weather', {'v': 1})
|
||||||
|
finally:
|
||||||
|
mod.os.replace = real
|
||||||
|
|
||||||
|
assert created, "set() did not go through the temp-file path"
|
||||||
|
assert all(DiskCache._is_orphaned_temp(n) for n in created), created
|
||||||
|
|
||||||
|
|
||||||
|
class TestTheSweep:
|
||||||
|
def test_an_old_orphan_is_removed(self, cache, tmp_path):
|
||||||
|
p = _write(tmp_path, '.weather.json.a1b2c3d4', 'x' * 5000)
|
||||||
|
_age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60)
|
||||||
|
|
||||||
|
stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
|
||||||
|
assert not p.exists()
|
||||||
|
assert stats['orphan_temp_files_deleted'] == 1
|
||||||
|
assert stats['space_freed_bytes'] >= 5000
|
||||||
|
|
||||||
|
def test_an_in_flight_write_is_not_snatched_away(self, cache, tmp_path):
|
||||||
|
# The whole risk of this sweep: deleting a temp file another thread is
|
||||||
|
# about to os.replace into place.
|
||||||
|
p = _write(tmp_path, '.weather.json.inflight')
|
||||||
|
|
||||||
|
cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
|
||||||
|
assert p.exists()
|
||||||
|
|
||||||
|
def test_real_cache_files_survive(self, cache, tmp_path):
|
||||||
|
fresh = _write(tmp_path, 'weather.json')
|
||||||
|
dotted = _write(tmp_path, '.weather.json')
|
||||||
|
_age(dotted, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60)
|
||||||
|
|
||||||
|
cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
|
||||||
|
assert fresh.exists()
|
||||||
|
assert dotted.exists(), "a completed .json was treated as a temp file"
|
||||||
|
|
||||||
|
def test_unrelated_dotfiles_survive(self, cache, tmp_path):
|
||||||
|
keep = _write(tmp_path, '.gitignore')
|
||||||
|
_age(keep, 400 * 86400)
|
||||||
|
|
||||||
|
cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
|
||||||
|
assert keep.exists()
|
||||||
|
|
||||||
|
def test_expiry_still_works_alongside_it(self, cache, tmp_path):
|
||||||
|
stale = _write(tmp_path, 'old.json')
|
||||||
|
_age(stale, 40 * 86400) # past the 30-day default
|
||||||
|
orphan = _write(tmp_path, '.old.json.zz999999')
|
||||||
|
_age(orphan, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60)
|
||||||
|
|
||||||
|
stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
|
||||||
|
assert not stale.exists()
|
||||||
|
assert not orphan.exists()
|
||||||
|
assert stats['files_deleted'] == 2
|
||||||
|
assert stats['orphan_temp_files_deleted'] == 1
|
||||||
|
|
||||||
|
def test_the_rig_scenario(self, cache, tmp_path):
|
||||||
|
"""76 orphans of assorted ages, none of them reachable before."""
|
||||||
|
for i in range(76):
|
||||||
|
p = _write(tmp_path, '.sched_%d.json.r%06d' % (i, i), 'x' * 1000)
|
||||||
|
_age(p, (i + 2) * 86400)
|
||||||
|
keep = _write(tmp_path, 'sched.json')
|
||||||
|
|
||||||
|
stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
|
||||||
|
assert stats['orphan_temp_files_deleted'] == 76
|
||||||
|
assert keep.exists()
|
||||||
|
assert not list(tmp_path.glob('.sched_*'))
|
||||||
|
# The summary line is "<deleted>/<scanned>", so an orphan that is
|
||||||
|
# deleted but never counted as scanned renders as "76/1".
|
||||||
|
assert stats['files_scanned'] == 77
|
||||||
|
assert stats['files_deleted'] <= stats['files_scanned']
|
||||||
|
|
||||||
|
def test_deleted_never_exceeds_scanned(self, cache, tmp_path):
|
||||||
|
p = _write(tmp_path, '.only.json.a1b2c3d4')
|
||||||
|
_age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60)
|
||||||
|
|
||||||
|
stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
|
||||||
|
assert stats['files_deleted'] == 1
|
||||||
|
assert stats['files_scanned'] == 1
|
||||||
|
|
||||||
|
def test_a_missing_file_mid_sweep_is_not_an_error(self, cache, tmp_path):
|
||||||
|
p = _write(tmp_path, '.weather.json.a1b2c3d4')
|
||||||
|
_age(p, _ORPHAN_TEMP_MAX_AGE_SECONDS + 60)
|
||||||
|
|
||||||
|
import src.cache.disk_cache as mod
|
||||||
|
real = mod.os.path.getsize
|
||||||
|
|
||||||
|
def vanish(path):
|
||||||
|
if path.endswith('.a1b2c3d4'):
|
||||||
|
os.remove(path)
|
||||||
|
raise FileNotFoundError(path)
|
||||||
|
return real(path)
|
||||||
|
|
||||||
|
mod.os.path.getsize = vanish
|
||||||
|
try:
|
||||||
|
stats = cache.cleanup_expired_files(FakeStrategy(), POLICIES)
|
||||||
|
finally:
|
||||||
|
mod.os.path.getsize = real
|
||||||
|
|
||||||
|
assert stats['errors'] == 0
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
"""Tests that a per-entry ttl actually controls expiry.
|
||||||
|
|
||||||
|
Regression under test: `CacheManager.set(key, data, ttl=...)` stored the value
|
||||||
|
and no read path ever consulted it. Expiry came from a `max_age` inferred from
|
||||||
|
substrings in the key ("live", "odds", "stock"), so every caller passing `ttl=`
|
||||||
|
-- 48 sites across the plugins and 4 in the core -- was writing a number that
|
||||||
|
did nothing. The old docstring admitted as much: "stored for compatibility but
|
||||||
|
expiration is still controlled via max_age when reading".
|
||||||
|
|
||||||
|
Measured against a real device's cache (8,873 entries carrying a ttl), the
|
||||||
|
inferred value and the intended one disagreed almost everywhere:
|
||||||
|
|
||||||
|
stocks max_age 600 vs ttl 1800 4903 entries
|
||||||
|
news max_age 3600 vs ttl 600 1770 entries
|
||||||
|
odds max_age 1800 vs ttl 3600 1301 entries
|
||||||
|
images max_age 300 vs ttl 2592000 20 entries
|
||||||
|
|
||||||
|
No `sports_live` entry carries a ttl, so live scores keep their inferred
|
||||||
|
30-second freshness either way.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.cache.memory_cache import MemoryCache
|
||||||
|
from src.cache.disk_cache import DiskCache
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def disk(tmp_path):
|
||||||
|
return DiskCache(cache_dir=str(tmp_path))
|
||||||
|
|
||||||
|
|
||||||
|
def _record(ttl=None, age=0.0):
|
||||||
|
rec = {"data": {"v": 1}, "timestamp": time.time() - age}
|
||||||
|
if ttl is not None:
|
||||||
|
rec["ttl"] = ttl
|
||||||
|
return rec
|
||||||
|
|
||||||
|
|
||||||
|
class TestDiskCacheHonoursTtl:
|
||||||
|
def test_ttl_longer_than_max_age_keeps_the_entry(self, disk):
|
||||||
|
# The odds case: written wanting an hour, expired at 30 minutes.
|
||||||
|
disk.set("odds_espn_football_nfl_401", _record(ttl=3600, age=1900))
|
||||||
|
assert disk.get("odds_espn_football_nfl_401", max_age=1800) is not None
|
||||||
|
|
||||||
|
def test_ttl_shorter_than_max_age_expires_the_entry(self, disk):
|
||||||
|
# The news case: written wanting 10 minutes, kept for an hour.
|
||||||
|
disk.set("news_NHL_1", _record(ttl=600, age=900))
|
||||||
|
assert disk.get("news_NHL_1", max_age=3600) is None
|
||||||
|
|
||||||
|
def test_without_a_ttl_max_age_still_applies(self, disk):
|
||||||
|
disk.set("plain_key", _record(age=400))
|
||||||
|
assert disk.get("plain_key", max_age=300) is None
|
||||||
|
disk.set("plain_key2", _record(age=100))
|
||||||
|
assert disk.get("plain_key2", max_age=300) is not None
|
||||||
|
|
||||||
|
def test_a_fresh_entry_within_its_ttl_survives(self, disk):
|
||||||
|
disk.set("k", _record(ttl=600, age=10))
|
||||||
|
assert disk.get("k", max_age=30) is not None
|
||||||
|
|
||||||
|
def test_ttl_zero_expires_immediately(self, disk):
|
||||||
|
# 0 means zero seconds, not "forever" -- max_age=None is how a caller
|
||||||
|
# asks for no expiry.
|
||||||
|
disk.set("k", _record(ttl=0, age=1))
|
||||||
|
assert disk.get("k", max_age=99999) is None
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bad", ["600", None, True, False, -5, {"a": 1}])
|
||||||
|
def test_a_nonsense_ttl_falls_back_to_max_age(self, disk, bad):
|
||||||
|
# Including bools: True is an int in Python and must not become a 1s ttl.
|
||||||
|
rec = _record(age=400)
|
||||||
|
rec["ttl"] = bad
|
||||||
|
disk.set("k_%s" % type(bad).__name__, rec)
|
||||||
|
assert disk.get("k_%s" % type(bad).__name__, max_age=300) is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestMemoryCacheHonoursTtl:
|
||||||
|
def test_ttl_longer_than_max_age_keeps_the_entry(self):
|
||||||
|
m = MemoryCache()
|
||||||
|
m.set("k", _record(ttl=3600))
|
||||||
|
m._timestamps["k"] = time.time() - 1900
|
||||||
|
assert m.get("k", max_age=1800) is not None
|
||||||
|
|
||||||
|
def test_ttl_shorter_than_max_age_expires_the_entry(self):
|
||||||
|
m = MemoryCache()
|
||||||
|
m.set("k", _record(ttl=600))
|
||||||
|
m._timestamps["k"] = time.time() - 900
|
||||||
|
assert m.get("k", max_age=3600) is None
|
||||||
|
|
||||||
|
def test_without_a_ttl_max_age_still_applies(self):
|
||||||
|
m = MemoryCache()
|
||||||
|
m.set("k", _record())
|
||||||
|
m._timestamps["k"] = time.time() - 400
|
||||||
|
assert m.get("k", max_age=300) is None
|
||||||
|
|
||||||
|
def test_both_layers_agree(self, tmp_path):
|
||||||
|
"""A record must not be live in one layer and expired in the other."""
|
||||||
|
rec = _record(ttl=3600, age=1900)
|
||||||
|
d = DiskCache(cache_dir=str(tmp_path))
|
||||||
|
d.set("k", rec)
|
||||||
|
m = MemoryCache()
|
||||||
|
m.set("k", rec)
|
||||||
|
m._timestamps["k"] = rec["timestamp"]
|
||||||
|
assert (d.get("k", max_age=1800) is not None) == (m.get("k", max_age=1800) is not None)
|
||||||
|
|
||||||
|
|
||||||
|
class TestEndToEnd:
|
||||||
|
def test_set_then_get_respects_the_ttl(self, tmp_path, monkeypatch):
|
||||||
|
"""The behaviour a caller of CacheManager.set(ttl=...) expects."""
|
||||||
|
from src.cache_manager import CacheManager
|
||||||
|
|
||||||
|
cm = CacheManager()
|
||||||
|
cm._disk_cache_component = DiskCache(cache_dir=str(tmp_path))
|
||||||
|
cm._memory_cache_component = MemoryCache()
|
||||||
|
|
||||||
|
cm.set("odds_espn_football_nfl_401", {"spread": 6.5}, ttl=3600)
|
||||||
|
|
||||||
|
# Age the stored record past the inferred max_age for odds (1800s) but
|
||||||
|
# within the ttl the caller asked for.
|
||||||
|
path = cm._disk_cache_component.get_cache_path("odds_espn_football_nfl_401")
|
||||||
|
import json
|
||||||
|
rec = json.load(open(path))
|
||||||
|
rec["timestamp"] = time.time() - 1900
|
||||||
|
json.dump(rec, open(path, "w"))
|
||||||
|
cm._memory_cache_component.clear() if hasattr(
|
||||||
|
cm._memory_cache_component, "clear") else None
|
||||||
|
|
||||||
|
got = cm.get_with_auto_strategy("odds_espn_football_nfl_401")
|
||||||
|
assert got is not None, "the ttl the caller asked for was ignored"
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
"""A checkbox group must not post back options it cannot show.
|
||||||
|
|
||||||
|
The enum that lets the widget draw checkboxes is also what validates the
|
||||||
|
saved value. When a league retires a team code -- OAK for the Athletics, ARI
|
||||||
|
for the Coyotes -- or a schema drops an option, a config that still holds the
|
||||||
|
old value has nothing to render for it. The value stayed in the hidden
|
||||||
|
``_data`` input regardless, because that input is seeded from the stored array
|
||||||
|
and only rebuilt by ``updateCheckboxGroupData()`` on change. Editing any other
|
||||||
|
field on that plugin therefore posted the stale value back, the schema
|
||||||
|
rejected it, and the save endpoint returned 400
|
||||||
|
``CONFIG_VALIDATION_FAILED`` -- so the whole plugin became uneditable until
|
||||||
|
the user worked out which invisible entry was at fault.
|
||||||
|
|
||||||
|
Runtime was never affected: plugin loading treats schema violations as
|
||||||
|
warn/degrade, and the stale code already matched no team. Only the web UI
|
||||||
|
blocked.
|
||||||
|
|
||||||
|
These tests render the checkbox-group block lifted *out of the shipped
|
||||||
|
template*, following test_enum_option_labels.py, so they exercise the
|
||||||
|
production expression rather than a copy that could drift from it.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from jinja2 import DictLoader, Environment
|
||||||
|
|
||||||
|
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
CONFIG_FORM = (PROJECT_ROOT / 'web_interface' / 'templates' / 'v3' / 'partials'
|
||||||
|
/ 'plugin_config.html')
|
||||||
|
|
||||||
|
# The checkbox-group branch: from its `{% elif %}` guard through the sentinel
|
||||||
|
# hidden input that closes it. Anchored on the guard so the match cannot run on
|
||||||
|
# into a neighbouring widget branch.
|
||||||
|
BLOCK_RE = re.compile(
|
||||||
|
r"\{%\s*elif x_widget == 'checkbox-group'\s*%\}(.*?)"
|
||||||
|
r"<input type=\"hidden\" name=\"\{\{ full_key \}\}\[\]\" value=\"\">",
|
||||||
|
re.S,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _shipped_block() -> str:
|
||||||
|
"""Return the live checkbox-group block lifted from plugin_config.html."""
|
||||||
|
source = CONFIG_FORM.read_text(encoding='utf-8')
|
||||||
|
match = BLOCK_RE.search(source)
|
||||||
|
assert match, (
|
||||||
|
'could not find the checkbox-group block in plugin_config.html — the '
|
||||||
|
'template changed shape and this guard needs updating'
|
||||||
|
)
|
||||||
|
block = match.group(1)
|
||||||
|
assert 'data-option-value' in block, 'extracted the wrong branch'
|
||||||
|
assert '{% elif' not in block, 'extraction ran past the checkbox-group branch'
|
||||||
|
return block
|
||||||
|
|
||||||
|
|
||||||
|
def _render(prop: dict, value=None) -> str:
|
||||||
|
env = Environment(loader=DictLoader({'f': _shipped_block()}), autoescape=True)
|
||||||
|
return env.get_template('f').render(
|
||||||
|
prop=prop, value=value, field_id='fid', full_key='k'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _submitted(html: str) -> list:
|
||||||
|
"""The array the form will actually post: the hidden _data input."""
|
||||||
|
match = re.search(r'id="fid_data"[^>]*\svalue=\'([^\']*)\'', html)
|
||||||
|
assert match, f'hidden _data input not found in:\n{html}'
|
||||||
|
return json.loads(match.group(1).replace(''', "'"))
|
||||||
|
|
||||||
|
|
||||||
|
def _checked(html: str) -> list:
|
||||||
|
return re.findall(r'data-option-value="([^"]+)"[^>]*checked', html)
|
||||||
|
|
||||||
|
|
||||||
|
MLB = {'type': 'array', 'items': {'type': 'string', 'enum': ['NYY', 'BOS', 'ATH']},
|
||||||
|
'x-widget': 'checkbox-group'}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_retired_code_is_not_posted_back() -> None:
|
||||||
|
"""The regression: OAK became ATH, and OAK used to ride along on save."""
|
||||||
|
html = _render(MLB, ['NYY', 'OAK'])
|
||||||
|
assert _submitted(html) == ['NYY'], 'stale value would still be submitted'
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_dropped_value_is_named_rather_than_vanishing() -> None:
|
||||||
|
html = _render(MLB, ['NYY', 'OAK'])
|
||||||
|
assert 'OAK' in html
|
||||||
|
assert 'data-stale-options' in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_valid_values_are_untouched_and_still_checked() -> None:
|
||||||
|
html = _render(MLB, ['NYY', 'ATH'])
|
||||||
|
assert _submitted(html) == ['NYY', 'ATH']
|
||||||
|
assert sorted(_checked(html)) == ['ATH', 'NYY']
|
||||||
|
assert 'data-stale-options' not in html
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_all_stale_selection_clears_rather_than_blocking() -> None:
|
||||||
|
html = _render(MLB, ['OAK', 'SD'])
|
||||||
|
assert _submitted(html) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_empty_enum_leaves_the_value_alone() -> None:
|
||||||
|
"""No options means nothing to validate against — filtering would wipe it."""
|
||||||
|
prop = {'type': 'array', 'items': {'type': 'string'}, 'x-widget': 'checkbox-group'}
|
||||||
|
html = _render(prop, ['ANYTHING', 'GOES'])
|
||||||
|
assert _submitted(html) == ['ANYTHING', 'GOES']
|
||||||
|
|
||||||
|
|
||||||
|
def test_unset_value_falls_back_to_the_default() -> None:
|
||||||
|
prop = dict(MLB, default=['BOS'])
|
||||||
|
html = _render(prop, None)
|
||||||
|
assert _submitted(html) == ['BOS']
|
||||||
|
assert _checked(html) == ['BOS']
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
"""composer_bp must actually be registered on the Flask app.
|
||||||
|
|
||||||
|
web_interface/blueprints/composer.py defines a full blueprint -- the
|
||||||
|
/composer/ page, /composer/api/generate, /composer/api/install, etc. -- but
|
||||||
|
nothing in web_interface/app.py imported or registered it. pages_v3 and
|
||||||
|
api_v3 both go through the same "import, wire managers, register_blueprint"
|
||||||
|
sequence; composer_bp had none of the three. The routes existed and worked
|
||||||
|
in isolation (every other composer test builds its own minimal Flask app
|
||||||
|
and registers composer_bp directly), but in the real running app every
|
||||||
|
composer URL -- including the page itself -- was a 404, and the frontend's
|
||||||
|
hardcoded '/composer/api/...' fetches (composer-app.js) had nothing to
|
||||||
|
reach.
|
||||||
|
|
||||||
|
app.py has import-time side effects (it loads config, builds managers, and
|
||||||
|
in some configurations starts background threads), so this scans the source
|
||||||
|
text rather than importing the module -- the same approach already used for
|
||||||
|
test_composer_js_contracts.py's structural checks.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
APP_PY = Path(__file__).resolve().parent.parent / "web_interface" / "app.py"
|
||||||
|
|
||||||
|
|
||||||
|
def _source() -> str:
|
||||||
|
return APP_PY.read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_composer_blueprint_is_imported():
|
||||||
|
src = _source()
|
||||||
|
assert re.search(
|
||||||
|
r"from web_interface\.blueprints\.composer import composer_bp", src
|
||||||
|
), "app.py never imports composer_bp"
|
||||||
|
|
||||||
|
|
||||||
|
def test_composer_blueprint_is_registered_under_composer_prefix():
|
||||||
|
src = _source()
|
||||||
|
match = re.search(
|
||||||
|
r"app\.register_blueprint\(\s*composer_bp\s*,\s*url_prefix\s*=\s*"
|
||||||
|
r"['\"](?P<prefix>[^'\"]*)['\"]",
|
||||||
|
src,
|
||||||
|
)
|
||||||
|
assert match, "app.py never calls app.register_blueprint(composer_bp, ...)"
|
||||||
|
# composer.html and composer-app.js hardcode '/composer/api/...' and
|
||||||
|
# '/composer/api/fonts/...' -- any other prefix serves a page whose own
|
||||||
|
# fetches and @font-face all 404.
|
||||||
|
assert match.group("prefix") == "/composer"
|
||||||
|
|
||||||
|
|
||||||
|
def test_composer_blueprint_managers_are_wired_before_registration():
|
||||||
|
"""Registered with no managers set means every route 503s or crashes.
|
||||||
|
|
||||||
|
composer.py initialises config_manager/plugin_manager/plugins_dir/
|
||||||
|
project_root to None at import time specifically so app.py can set them
|
||||||
|
-- the same pattern api_v3 and pages_v3 use.
|
||||||
|
"""
|
||||||
|
src = _source()
|
||||||
|
register_match = re.search(r"app\.register_blueprint\(\s*composer_bp", src)
|
||||||
|
assert register_match, "app.py never registers composer_bp"
|
||||||
|
before_registration = src[: register_match.start()]
|
||||||
|
for attr in ("config_manager", "plugin_manager", "plugins_dir", "project_root"):
|
||||||
|
assert re.search(rf"composer_bp\.{attr}\s*=", before_registration), (
|
||||||
|
f"composer_bp.{attr} is never set before registration"
|
||||||
|
)
|
||||||
@@ -0,0 +1,323 @@
|
|||||||
|
"""The composer generates Python that the plugin loader imports and executes.
|
||||||
|
|
||||||
|
/api/install writes the generated manager.py into plugins_dir and the loader
|
||||||
|
imports it, so anything the payload can splice into that source runs on the
|
||||||
|
device. The ast.parse check in _generate_plugin_files rejects only *invalid*
|
||||||
|
syntax -- an injected `import os` is perfectly valid and passed it.
|
||||||
|
|
||||||
|
Two ways in, both confirmed against the code before it was fixed:
|
||||||
|
|
||||||
|
metadata.name = a name containing a triple-quote, a newline, then
|
||||||
|
`import os; PWNED = os.getuid()`, then another triple-quote
|
||||||
|
-> closes the module docstring; the rest became module-level statements
|
||||||
|
(spelled out rather than shown literally -- writing the payload into
|
||||||
|
this docstring closes *this* file's docstring, which is the bug)
|
||||||
|
|
||||||
|
element x = '0 or __import__("os").system("id")'
|
||||||
|
-> f-string interpolated it verbatim: x=0 or __import__("os").system("id")
|
||||||
|
"""
|
||||||
|
import ast
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||||
|
|
||||||
|
from web_interface.blueprints import composer as C # noqa: E402
|
||||||
|
|
||||||
|
BASE_META = {"id": "test-plugin", "name": "Clock", "author": "a",
|
||||||
|
"version": "1.0.0", "description": "d"}
|
||||||
|
|
||||||
|
#: Values that terminate a Python expression and start a new statement.
|
||||||
|
EXPR_PAYLOADS = [
|
||||||
|
'0 or __import__("os").system("id")',
|
||||||
|
'0);import os;os.system("id");(',
|
||||||
|
'__import__("subprocess").run(["id"])',
|
||||||
|
"0 if False else exec('x=1')",
|
||||||
|
"1e999", "nan", "0x41", "0__0",
|
||||||
|
]
|
||||||
|
|
||||||
|
#: Values that close a string literal in the generated source.
|
||||||
|
LITERAL_PAYLOADS = [
|
||||||
|
'Clock"""\nimport os; PWNED = os.getuid()\n"""',
|
||||||
|
"Clock'''\nimport os\n'''",
|
||||||
|
'Clock" + __import__("os").system("id") + "',
|
||||||
|
"Clock\\", "Clock\nimport os",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _payload(**over):
|
||||||
|
# dataModel.configVars is the key _generate_plugin_files reads; "config_vars"
|
||||||
|
# was never looked at, so anything passed through it tested nothing.
|
||||||
|
p = {"metadata": dict(BASE_META), "elements": [],
|
||||||
|
"dataModel": {"configVars": over.pop("config_vars", [])}}
|
||||||
|
p["metadata"].update(over.pop("metadata", {}))
|
||||||
|
p.update(over)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def _generated(payload):
|
||||||
|
return C._generate_plugin_files(payload)["manager.py"]
|
||||||
|
|
||||||
|
|
||||||
|
def _module_level_code(src):
|
||||||
|
"""Statements at module level that are not the docstring/imports/classes."""
|
||||||
|
tree = ast.parse(src)
|
||||||
|
out = []
|
||||||
|
for node in tree.body:
|
||||||
|
if isinstance(node, (ast.ClassDef, ast.FunctionDef, ast.ImportFrom, ast.Import)):
|
||||||
|
continue
|
||||||
|
if isinstance(node, ast.Expr) and isinstance(node.value, ast.Constant):
|
||||||
|
continue # the docstring
|
||||||
|
out.append(ast.unparse(node))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("payload", LITERAL_PAYLOADS)
|
||||||
|
def test_a_name_that_breaks_out_of_a_literal_is_refused(payload):
|
||||||
|
with pytest.raises(C.ComposerInputError):
|
||||||
|
_generated(_payload(metadata={"name": payload}))
|
||||||
|
|
||||||
|
|
||||||
|
#: Types with a drawing branch in manager.py.j2. An injection test using any
|
||||||
|
#: other type proves nothing: _preprocess_elements drops it, so its values
|
||||||
|
#: never reach the generated source and every assertion passes trivially.
|
||||||
|
#: This test previously used "line", which has never had a branch.
|
||||||
|
RENDERED_GEOMETRY_CASES = [
|
||||||
|
("rectangle", {"x": 0, "y": 0, "width": 10, "height": 8}),
|
||||||
|
("arc", {"x": 0, "y": 0, "width": 24, "height": 24}),
|
||||||
|
("ellipse", {"x": 0, "y": 0, "width": 24, "height": 12}),
|
||||||
|
("rounded_rectangle", {"x": 0, "y": 0, "width": 24, "height": 10}),
|
||||||
|
("gauge", {"x": 0, "y": 0, "width": 32, "height": 32}),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("etype,base", RENDERED_GEOMETRY_CASES)
|
||||||
|
@pytest.mark.parametrize("evil", EXPR_PAYLOADS)
|
||||||
|
@pytest.mark.parametrize("field", ["x", "y", "width", "height"])
|
||||||
|
def test_a_non_numeric_geometry_value_cannot_reach_the_source(etype, base, evil, field):
|
||||||
|
"""width/height were interpolated raw into the generated source.
|
||||||
|
|
||||||
|
p['x2_expr'] = f"({x_expr}) + {w}" with w straight off the payload, so a
|
||||||
|
rectangle with width='0 or __import__("os").system("id")' produced
|
||||||
|
|
||||||
|
[0, 0, (0) + 0 or __import__("os").system("id"), (0) + 8],
|
||||||
|
|
||||||
|
in a manager.py that /api/install writes to disk and the loader imports.
|
||||||
|
"""
|
||||||
|
el = {"type": etype, "id": "e1", **base}
|
||||||
|
el[field] = evil
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert "__import__" not in src, f"{etype}.{field}={evil!r} reached the generated source"
|
||||||
|
assert "os.system" not in src
|
||||||
|
assert not _module_level_code(src), \
|
||||||
|
f"{etype}.{field}={evil!r} produced module-level statements: {_module_level_code(src)}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_injection_case_uses_a_type_that_actually_renders():
|
||||||
|
"""Guards against the whole suite quietly going vacuous again.
|
||||||
|
|
||||||
|
An element type with no template branch is dropped before generation, so
|
||||||
|
an injection test written against one asserts nothing and still passes.
|
||||||
|
"""
|
||||||
|
used = {etype for etype, _ in RENDERED_GEOMETRY_CASES}
|
||||||
|
missing = used - set(C._RENDERABLE_ELEMENT_TYPES)
|
||||||
|
assert not missing, f"injection tests use non-rendering types: {sorted(missing)}"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("evil", EXPR_PAYLOADS)
|
||||||
|
@pytest.mark.parametrize("channel", ["r", "g", "b"])
|
||||||
|
def test_a_non_numeric_colour_channel_cannot_reach_the_source(evil, channel):
|
||||||
|
el = {"type": "text", "id": "t1", "x": 0, "y": 0, "text": "hi",
|
||||||
|
"font": "press_start", "r": 255, "g": 255, "b": 255}
|
||||||
|
el[channel] = evil
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert "__import__" not in src and "os.system" not in src
|
||||||
|
assert not _module_level_code(src)
|
||||||
|
|
||||||
|
|
||||||
|
def test_colour_channels_are_clamped_to_a_byte():
|
||||||
|
el = {"type": "text", "id": "t1", "x": 0, "y": 0, "text": "hi",
|
||||||
|
"font": "press_start", "r": 99999, "g": -5, "b": 128}
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert "(255, 0, 128)" in src, "channels were not clamped to 0-255"
|
||||||
|
|
||||||
|
|
||||||
|
#: fillR/fillG/fillB (and outR/G/B, bgR/G/B, trackR/G/B) go through
|
||||||
|
#: _as_fill_filter -> _as_rgb_filter, a separate path from _rgb_expr above.
|
||||||
|
#: It used raw int() until it was found to raise ValueError on a non-numeric
|
||||||
|
#: channel instead of clamping like every other coerced value in this module.
|
||||||
|
@pytest.mark.parametrize("evil", EXPR_PAYLOADS)
|
||||||
|
@pytest.mark.parametrize("channel", ["fillR", "fillG", "fillB"])
|
||||||
|
def test_a_non_numeric_fill_channel_cannot_reach_the_source(evil, channel):
|
||||||
|
el = {"type": "rectangle", "id": "r1", "x": 0, "y": 0, "width": 10, "height": 8,
|
||||||
|
"fillR": 0, "fillG": 0, "fillB": 128}
|
||||||
|
el[channel] = evil
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert "__import__" not in src and "os.system" not in src
|
||||||
|
assert not _module_level_code(src)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fill_channels_are_clamped_to_a_byte():
|
||||||
|
el = {"type": "rectangle", "id": "r1", "x": 0, "y": 0, "width": 10, "height": 8,
|
||||||
|
"fillR": 99999, "fillG": -5, "fillB": 128}
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert "(255, 0, 128)" in src, "fill channels were not clamped to 0-255"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_generated_module_still_has_no_top_level_statements():
|
||||||
|
"""The clean case: a normal payload produces only imports and a class."""
|
||||||
|
el = {"type": "text", "id": "t1", "x": 4, "y": 4, "text": "hi",
|
||||||
|
"font": "press_start", "r": 1, "g": 2, "b": 3}
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert not _module_level_code(src)
|
||||||
|
assert "(1, 2, 3)" in src
|
||||||
|
|
||||||
|
|
||||||
|
# --- config variable keys ---------------------------------------------------
|
||||||
|
|
||||||
|
def _with_key(key):
|
||||||
|
return {"metadata": dict(BASE_META), "elements": [],
|
||||||
|
"dataModel": {"configVars": [{"key": key, "type": "string",
|
||||||
|
"default": "x", "label": "L"}]}}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("key", ["class", "def", "import", "None", "True",
|
||||||
|
"lambda", "pass", "match", "case"])
|
||||||
|
def test_a_keyword_config_key_is_named_in_the_error(key):
|
||||||
|
"""ast.parse already rejected these, but as an unhelpful line number.
|
||||||
|
|
||||||
|
"Generated code has a syntax error: invalid syntax (line 17)" tells the
|
||||||
|
user nothing about which field to fix.
|
||||||
|
"""
|
||||||
|
with pytest.raises(C.ComposerInputError) as exc:
|
||||||
|
_generated(_with_key(key))
|
||||||
|
assert key in str(exc.value) and "keyword" in str(exc.value).lower()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("key", ["config", "logger", "display_manager",
|
||||||
|
"cache_manager", "plugin_id", "enabled",
|
||||||
|
"self", "update", "display"])
|
||||||
|
def test_a_reserved_attribute_config_key_is_refused(key):
|
||||||
|
"""These generate *valid* Python that silently clobbers plugin state.
|
||||||
|
|
||||||
|
The worst is `config`: the assignment lands right after super().__init__(),
|
||||||
|
so `self.config = config.get("config", "x")` replaces the plugin's config
|
||||||
|
dict with a string and every later self.config.get(...) fails at runtime.
|
||||||
|
"""
|
||||||
|
with pytest.raises(C.ComposerInputError) as exc:
|
||||||
|
_generated(_with_key(key))
|
||||||
|
assert key in str(exc.value) and "reserved" in str(exc.value).lower()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("key", ["brightness", "my_var", "_private", "x1",
|
||||||
|
"update_interval_seconds"])
|
||||||
|
def test_ordinary_config_keys_are_still_accepted(key):
|
||||||
|
src = _generated(_with_key(key))
|
||||||
|
assert f"self.{key} = config.get(" in src
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_generated_config_assignment_does_not_precede_super_init():
|
||||||
|
"""Guards the reasoning behind the reserved list, not just the list."""
|
||||||
|
src = _generated(_with_key("brightness"))
|
||||||
|
body = src.splitlines()
|
||||||
|
super_at = next(i for i, line in enumerate(body) if "super().__init__(" in line)
|
||||||
|
assign_at = next(i for i, line in enumerate(body)
|
||||||
|
if "self.brightness = config.get(" in line)
|
||||||
|
assert assign_at > super_at, (
|
||||||
|
"config vars are assigned before super().__init__(); the reserved-name "
|
||||||
|
"list assumes they land after it")
|
||||||
|
|
||||||
|
|
||||||
|
# --- optional keys ----------------------------------------------------------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("el_type,missing", [
|
||||||
|
("text", "text"), ("text", "text2"), ("clock", "format"),
|
||||||
|
])
|
||||||
|
def test_an_element_missing_an_optional_key_does_not_500(el_type, missing):
|
||||||
|
"""`p` is a copy of the raw element, so an absent key stays absent.
|
||||||
|
|
||||||
|
The defaults were applied to locals only, so manager.py.j2 rendered
|
||||||
|
`{{ el.text | tojson }}` over a jinja2.Undefined and tojson raised
|
||||||
|
TypeError -- which no handler catches, making a missing key a 500 rather
|
||||||
|
than a validation error or a sensible default.
|
||||||
|
"""
|
||||||
|
el = {"type": el_type, "id": "e1", "x": 0, "y": 0, "font": "press_start"}
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
ast.parse(src) # must still be valid Python
|
||||||
|
assert "Undefined" not in src
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_clock_without_a_format_uses_the_documented_default():
|
||||||
|
el = {"type": "clock", "id": "c1", "x": 0, "y": 0, "font": "press_start"}
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert '"%H:%M"' in src, "the %H:%M default did not reach the generated source"
|
||||||
|
|
||||||
|
|
||||||
|
#: (element type, channel key, base element) for colour channels that were
|
||||||
|
#: interpolated raw rather than through _rgb_expr/_safe_int. Prefixed channels
|
||||||
|
#: (emptyR/G/B, labelR/G/B) were the ones the original r/g/b test never reached.
|
||||||
|
RAW_COLOUR_CASES = [
|
||||||
|
("progress_bar", "r", {"x": 0, "y": 0}),
|
||||||
|
("progress_bar", "g", {"x": 0, "y": 0}),
|
||||||
|
("pips", "b", {"x": 0, "y": 0}),
|
||||||
|
("pips", "emptyR", {"x": 0, "y": 0}),
|
||||||
|
("pips", "emptyG", {"x": 0, "y": 0}),
|
||||||
|
("sparkline", "r", {"x": 0, "y": 0}),
|
||||||
|
("gauge", "labelR", {"x": 0, "y": 0, "width": 32, "height": 32}),
|
||||||
|
("gauge", "labelB", {"x": 0, "y": 0, "width": 32, "height": 32}),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("etype,channel,base", RAW_COLOUR_CASES)
|
||||||
|
@pytest.mark.parametrize("evil", EXPR_PAYLOADS)
|
||||||
|
def test_a_prefixed_colour_channel_cannot_reach_the_source(etype, channel, base, evil):
|
||||||
|
"""Five tuples were built with f"({el.get('r', 100)}, ...)" -- no coercion.
|
||||||
|
|
||||||
|
The pre-existing colour test only covered r/g/b on a text element, so the
|
||||||
|
prefixed channels and the four other types were never exercised.
|
||||||
|
"""
|
||||||
|
el = {"type": etype, "id": "e1", **base}
|
||||||
|
el[channel] = evil
|
||||||
|
src = _generated(_payload(elements=[el]))
|
||||||
|
assert "__import__" not in src, f"{etype}.{channel}={evil!r} reached the source"
|
||||||
|
assert "os.system" not in src
|
||||||
|
assert not _module_level_code(src)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("value", [float("inf"), float("-inf"), float("nan")])
|
||||||
|
@pytest.mark.parametrize("field", ["x", "y", "width", "height"])
|
||||||
|
def test_a_non_finite_dimension_does_not_escape_as_an_unhandled_error(field, value):
|
||||||
|
"""json.loads accepts Infinity/NaN and Flask passes them through, so a
|
||||||
|
payload can hand _safe_int a non-finite float. int(inf) raises
|
||||||
|
OverflowError -- neither ValueError nor ComposerInputError -- so it escaped
|
||||||
|
both handlers and surfaced as a 500 with a traceback instead of a 422."""
|
||||||
|
el = {"type": "rectangle", "id": "r1", "x": 0, "y": 0, "width": 10, "height": 8}
|
||||||
|
el[field] = value
|
||||||
|
src = _generated(_payload(elements=[el])) # must not raise
|
||||||
|
# A non-finite value must be replaced by the default, not spelled into the
|
||||||
|
# source. Word-boundary match: "info" in self.logger.info contains "inf".
|
||||||
|
assert not re.search(r"\b(inf|nan|Infinity|NaN)\b", src), \
|
||||||
|
f"{field}={value!r} leaked a non-finite literal into the source"
|
||||||
|
assert not _module_level_code(src)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bad_id", [
|
||||||
|
'x = __import__("os").system("id") #',
|
||||||
|
"x\nimport os\n_y",
|
||||||
|
"x[0]",
|
||||||
|
"",
|
||||||
|
"a" * 200,
|
||||||
|
])
|
||||||
|
def test_a_marquee_id_cannot_become_code(bad_id):
|
||||||
|
"""data_key is spliced UNQUOTED into variable names
|
||||||
|
(_{{ data_key }}_text = ...), so a non-identifier id landed in the source
|
||||||
|
as code. ast.parse caught it, but the caller then got an opaque
|
||||||
|
"Generated code has a syntax error" rather than being told the id is bad."""
|
||||||
|
el = {"type": "marquee", "id": bad_id, "x": 0, "y": 0, "text": "hi"}
|
||||||
|
src = _generated(_payload(elements=[el])) # must not raise
|
||||||
|
assert "__import__(" not in src
|
||||||
|
assert "os.system(" not in src
|
||||||
|
assert not _module_level_code(src)
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
"""An element the template cannot draw must not produce an empty `if` block.
|
||||||
|
|
||||||
|
manager.py.j2 wraps each element in `if width >= N:` (breakpoint) and/or
|
||||||
|
`if int(time.time() * 2) % 2:` (blink), and the body comes from the per-type
|
||||||
|
branches. A type with no branch contributed nothing, so the wrapper opened a
|
||||||
|
block with no statements in it. ast.parse in _generate_plugin_files then
|
||||||
|
failed and the caller was told only:
|
||||||
|
|
||||||
|
Generated code has a syntax error: expected an indented block after
|
||||||
|
'if' statement on line 49
|
||||||
|
|
||||||
|
which names a line of generated source the user never sees. Confirmed against
|
||||||
|
the code before the fix with a `group` element carrying minWidth.
|
||||||
|
|
||||||
|
Two defences, both covered here: _preprocess_elements drops types the template
|
||||||
|
has no branch for, and the template emits a `pass` fallback so a type added to
|
||||||
|
the canvas before its branch exists degrades to a no-op instead of a broken
|
||||||
|
plugin.
|
||||||
|
"""
|
||||||
|
import ast
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||||
|
|
||||||
|
from web_interface.blueprints import composer as C # noqa: E402
|
||||||
|
|
||||||
|
TEMPLATE = (Path(__file__).resolve().parent.parent
|
||||||
|
/ "web_interface/templates/v3/composer/manager.py.j2")
|
||||||
|
|
||||||
|
BASE_META = {"id": "test-plugin", "name": "Clock", "author": "a",
|
||||||
|
"version": "1.0.0", "description": "d"}
|
||||||
|
|
||||||
|
|
||||||
|
def generate(element):
|
||||||
|
return C._generate_plugin_files({
|
||||||
|
"metadata": BASE_META,
|
||||||
|
"elements": [element],
|
||||||
|
"dataModel": {"configVars": []},
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("wrapper", [
|
||||||
|
{"minWidth": 64}, # breakpoint block
|
||||||
|
{"blink": True}, # blink block
|
||||||
|
{"minWidth": 64, "blink": True}, # both, nested
|
||||||
|
])
|
||||||
|
@pytest.mark.parametrize("etype", ["group", "widget_9000", "section"])
|
||||||
|
def test_undrawable_element_does_not_break_generation(etype, wrapper):
|
||||||
|
element = {"type": etype, "x": 0, "y": 0, "color": "#ffffff", **wrapper}
|
||||||
|
files = generate(element) # must not raise ComposerInputError
|
||||||
|
assert "manager.py" in files
|
||||||
|
|
||||||
|
|
||||||
|
def test_drawable_element_still_renders_inside_a_breakpoint():
|
||||||
|
files = generate({"type": "text", "text": "hi", "x": 0, "y": 0,
|
||||||
|
"minWidth": 64, "color": "#ffffff"})
|
||||||
|
src = files["manager.py"]
|
||||||
|
assert "if width >= 64:" in src
|
||||||
|
assert "draw_text" in src
|
||||||
|
|
||||||
|
|
||||||
|
def test_renderable_types_match_the_template_branches():
|
||||||
|
"""The constant and the template must agree.
|
||||||
|
|
||||||
|
A type listed in the constant with no branch emits an empty block (the bug
|
||||||
|
above); a type with a branch but missing from the constant is silently
|
||||||
|
dropped from every generated plugin. Neither is visible without this check.
|
||||||
|
"""
|
||||||
|
branches = set(re.findall(r"el\.type == '([a-z_]+)'", TEMPLATE.read_text()))
|
||||||
|
assert branches == set(C._RENDERABLE_ELEMENT_TYPES)
|
||||||
|
|
||||||
|
|
||||||
|
def test_template_closes_the_branch_chain_with_a_fallback():
|
||||||
|
"""Belt and braces: even if the constant drifts, no empty block escapes."""
|
||||||
|
text = TEMPLATE.read_text()
|
||||||
|
assert "{% else %}" in text
|
||||||
|
assert "pass # element type" in text
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("wrapper", [
|
||||||
|
{"minWidth": 64},
|
||||||
|
{"blink": True},
|
||||||
|
{"minWidth": 64, "blink": True},
|
||||||
|
])
|
||||||
|
def test_dynamic_text_with_non_config_binding_does_not_break_generation(wrapper):
|
||||||
|
"""dynamic_text only renders a body for binding.source == 'config'.
|
||||||
|
|
||||||
|
_preprocess_elements accepts any string as binding.source (it just
|
||||||
|
defaults a missing one to 'config'), so a 'live' or 'sensor' source --
|
||||||
|
anything a client sends that isn't literally 'config' -- hit the same
|
||||||
|
empty-if-block bug as an undrawable element type, just one level deeper:
|
||||||
|
the branch is taken, but its own inner `if` produced nothing.
|
||||||
|
"""
|
||||||
|
element = {"type": "dynamic_text", "x": 0, "y": 0, "color": "#ffffff",
|
||||||
|
"binding": {"source": "live", "key": "temperature"}, **wrapper}
|
||||||
|
files = generate(element) # must not raise ComposerInputError
|
||||||
|
assert "manager.py" in files
|
||||||
|
assert "non-config dynamic_text binding draws nothing" in files["manager.py"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_binding_source_cannot_break_out_of_the_comment_it_lands_in():
|
||||||
|
"""binding.source used to be interpolated straight into a Python comment
|
||||||
|
(`pass # dynamic_text binding_source "{{ el.binding_source }}" draws
|
||||||
|
nothing`) with no escaping. A source string carrying a newline closed the
|
||||||
|
comment, and text on the following line(s), indented to match, became a
|
||||||
|
real statement in the generated plugin -- CWE-94, and live once
|
||||||
|
composer_bp was registered (confirmed: this exact payload produces a
|
||||||
|
manager.py containing a clean, ast.parse-valid `import os` against the
|
||||||
|
pre-fix template). The comment is now a fixed literal that never
|
||||||
|
interpolates the value at all.
|
||||||
|
"""
|
||||||
|
payload = "foo\n import os\n os.system('id') # "
|
||||||
|
element = {"type": "dynamic_text", "x": 0, "y": 0, "color": "#ffffff",
|
||||||
|
"binding": {"source": payload, "key": "temperature"}}
|
||||||
|
files = generate(element)
|
||||||
|
src = files["manager.py"]
|
||||||
|
assert "import os" not in src
|
||||||
|
assert "os.system" not in src
|
||||||
|
assert "non-config dynamic_text binding draws nothing" in src
|
||||||
|
ast.parse(src) # belt and braces: generate() already enforces this
|
||||||
@@ -0,0 +1,263 @@
|
|||||||
|
"""Structural checks on the composer's JavaScript.
|
||||||
|
|
||||||
|
There is no JS test runner in this repo, but three defects here are structural
|
||||||
|
enough to assert on the parse tree, and each was a real bug:
|
||||||
|
|
||||||
|
- Stroke widths inside _drawElement were left in canvas pixels while the
|
||||||
|
geometry around them scaled by SCALE, so at SCALE>1 every outline rendered
|
||||||
|
thinner than one LED pixel and the preview stopped matching the panel.
|
||||||
|
- The `line` branch drew raw el.x0/el.y0/el.x1/el.y1, ignoring the anchor that
|
||||||
|
every other element type honours, so setting xAnchor moved everything except
|
||||||
|
lines -- and getBoundingBox had the same omission, leaving the hit box behind.
|
||||||
|
- Four methods mutated exactly what _snapshot() serialises (metadata,
|
||||||
|
currentPreset) without calling it. _snapshot is the only caller of
|
||||||
|
_debouncedAutosave, so those changes were lost on reload and could not be
|
||||||
|
undone.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
tree_sitter = pytest.importorskip("tree_sitter")
|
||||||
|
tree_sitter_javascript = pytest.importorskip("tree_sitter_javascript")
|
||||||
|
|
||||||
|
JS_DIR = Path(__file__).resolve().parent.parent / "web_interface/static/v3/js/composer"
|
||||||
|
CANVAS = JS_DIR / "composer-canvas.js"
|
||||||
|
APP = JS_DIR / "composer-app.js"
|
||||||
|
|
||||||
|
|
||||||
|
def _function_source(path: Path, name: str) -> str:
|
||||||
|
"""Return the source of a top-level function declaration by name."""
|
||||||
|
src = path.read_bytes()
|
||||||
|
lang = tree_sitter.Language(tree_sitter_javascript.language())
|
||||||
|
tree = tree_sitter.Parser(lang).parse(src)
|
||||||
|
found = []
|
||||||
|
|
||||||
|
def walk(node):
|
||||||
|
if node.type == "function_declaration":
|
||||||
|
ident = node.child_by_field_name("name")
|
||||||
|
if ident is not None and src[ident.start_byte:ident.end_byte].decode() == name:
|
||||||
|
found.append(src[node.start_byte:node.end_byte].decode())
|
||||||
|
for c in node.children:
|
||||||
|
walk(c)
|
||||||
|
|
||||||
|
walk(tree.root_node)
|
||||||
|
assert found, f"{name} not found in {path.name}"
|
||||||
|
return found[0]
|
||||||
|
|
||||||
|
|
||||||
|
def _method_source(path: Path, name: str) -> str:
|
||||||
|
"""Return the source of a top-level object method by name."""
|
||||||
|
src = path.read_bytes()
|
||||||
|
lang = tree_sitter.Language(tree_sitter_javascript.language())
|
||||||
|
tree = tree_sitter.Parser(lang).parse(src)
|
||||||
|
found = []
|
||||||
|
|
||||||
|
def walk(node):
|
||||||
|
if node.type == "method_definition":
|
||||||
|
ident = node.child_by_field_name("name")
|
||||||
|
if ident is not None and src[ident.start_byte:ident.end_byte].decode() == name:
|
||||||
|
found.append(src[node.start_byte:node.end_byte].decode())
|
||||||
|
for c in node.children:
|
||||||
|
walk(c)
|
||||||
|
|
||||||
|
walk(tree.root_node)
|
||||||
|
assert found, f"{name} not found in {path.name}"
|
||||||
|
return found[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_both_files_parse():
|
||||||
|
lang = tree_sitter.Language(tree_sitter_javascript.language())
|
||||||
|
parser = tree_sitter.Parser(lang)
|
||||||
|
for path in (CANVAS, APP):
|
||||||
|
tree = parser.parse(path.read_bytes())
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
def walk(node):
|
||||||
|
if node.type == "ERROR" or node.is_missing:
|
||||||
|
errors.append(node.start_point[0] + 1)
|
||||||
|
for c in node.children:
|
||||||
|
walk(c)
|
||||||
|
|
||||||
|
walk(tree.root_node)
|
||||||
|
assert not errors, f"{path.name} has parse errors at lines {errors}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_element_strokes_scale_with_scale():
|
||||||
|
"""No bare `ctx.lineWidth = 1` inside _drawElement.
|
||||||
|
|
||||||
|
Selection handles and the grid are drawn in canvas pixels deliberately and
|
||||||
|
live in other functions, so this is scoped to the element drawing routine.
|
||||||
|
"""
|
||||||
|
body = _function_source(CANVAS, "_drawElement")
|
||||||
|
offenders = re.findall(r"ctx\.lineWidth\s*=\s*1\s*;", body)
|
||||||
|
assert not offenders, f"{len(offenders)} unscaled stroke width(s) in _drawElement"
|
||||||
|
|
||||||
|
|
||||||
|
def test_line_branch_applies_the_anchor_offset():
|
||||||
|
"""Scoped to _drawElement.
|
||||||
|
|
||||||
|
getBoundingBox has its own `case 'line': {` and appears first in the file,
|
||||||
|
so searching the whole text found *that* branch -- this assertion passed
|
||||||
|
with the draw branch's anchor offset removed. Verified: stripping it and
|
||||||
|
re-running gave 11/11 green.
|
||||||
|
"""
|
||||||
|
body = _function_source(CANVAS, "_drawElement")
|
||||||
|
line_branch = body[body.index("case 'line': {"):]
|
||||||
|
line_branch = line_branch[:line_branch.index("case 'divider'")]
|
||||||
|
assert "ax - el.x0" in line_branch and "ay - el.y0" in line_branch, \
|
||||||
|
"line drawing ignores xAnchor/yAnchor"
|
||||||
|
assert "moveTo(el.x0 * s" not in line_branch, \
|
||||||
|
"line still drawn from unanchored endpoints"
|
||||||
|
|
||||||
|
|
||||||
|
def test_line_bounding_box_applies_the_anchor_offset():
|
||||||
|
"""The companion to the above: scoped to getBoundingBox specifically, so
|
||||||
|
the two tests cannot both be satisfied by the same branch."""
|
||||||
|
body = _function_source(CANVAS, "getBoundingBox")
|
||||||
|
box = body[body.index("case 'line'"):]
|
||||||
|
box = box[:box.index("case 'divider'")]
|
||||||
|
assert "ax - el.x0" in box, "line bounding box ignores the anchor"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("method", [
|
||||||
|
"onBgColorChange", # mutates metadata.bgColor
|
||||||
|
"setCustomSize", # mutates currentPreset / MATRIX_W / MATRIX_H
|
||||||
|
"changePreset", # mutates currentPreset / MATRIX_W / MATRIX_H
|
||||||
|
"applyPresetLabel", # same, for sizes not in DISPLAY_PRESETS
|
||||||
|
"onColorChange", # the one that was already fixed — keeps it fixed
|
||||||
|
])
|
||||||
|
def test_state_mutations_take_a_snapshot(method):
|
||||||
|
body = _method_source(APP, method)
|
||||||
|
assert "_snapshot()" in body, \
|
||||||
|
f"{method} changes snapshotted state without calling _snapshot()"
|
||||||
|
assert "isDirty = true" in body, f"{method} does not mark the design dirty"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("method", ["changePreset", "applyPresetLabel"])
|
||||||
|
def test_restore_path_stays_snapshot_free(method):
|
||||||
|
"""_applyState and loadTemplate call these with {silent: true} while
|
||||||
|
restoring; snapshotting there would push restore steps onto the undo stack
|
||||||
|
and re-autosave the state just loaded."""
|
||||||
|
body = _method_source(APP, method)
|
||||||
|
assert "opts.silent" in body, f"{method} lost its silent guard"
|
||||||
|
snap = body.index("_snapshot()")
|
||||||
|
guard = body.index("!opts.silent")
|
||||||
|
assert guard < snap, f"{method} snapshots outside the !opts.silent guard"
|
||||||
|
|
||||||
|
|
||||||
|
TEMPLATE_HTML = (Path(__file__).resolve().parent.parent
|
||||||
|
/ "web_interface/templates/v3/composer.html")
|
||||||
|
|
||||||
|
#: The six toolbar buttons and the wrapper each must call.
|
||||||
|
ALIGN_BUTTONS = ["alignLeft", "alignHCenter", "alignRight",
|
||||||
|
"alignTop", "alignVCenter", "alignBottom"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_alignment_buttons_use_the_anchor_clearing_path():
|
||||||
|
"""Two alignment implementations existed and the toolbar used the wrong one.
|
||||||
|
|
||||||
|
The legacy alignElement(dir) set el.x/el.y but left xAnchor/yAnchor in
|
||||||
|
place. resolveAnchor turns anchor='right' into `dim - val`, so "align left"
|
||||||
|
(el.x = 0) resolved to x = MATRIX_W -- the element jumped to the far right
|
||||||
|
edge instead. _alignElement clears the anchor first, so the stored value is
|
||||||
|
absolute, and it also updates el.x0/el.y0 so lines actually move.
|
||||||
|
"""
|
||||||
|
html = TEMPLATE_HTML.read_text()
|
||||||
|
for wrapper in ALIGN_BUTTONS:
|
||||||
|
assert f"{wrapper}()" in html, f"toolbar does not call {wrapper}()"
|
||||||
|
assert not re.search(r"[^_]alignElement\(", html), \
|
||||||
|
"toolbar still calls the legacy alignElement()"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_legacy_alignelement_is_gone():
|
||||||
|
"""Leaving it in place invites the toolbar drifting back to it."""
|
||||||
|
src = APP.read_text()
|
||||||
|
assert not re.search(r"^\s{4}alignElement\(dir\)", src, re.M), \
|
||||||
|
"legacy alignElement(dir) still defined"
|
||||||
|
|
||||||
|
|
||||||
|
def test_align_clears_the_anchor_and_moves_line_endpoints():
|
||||||
|
body = _method_source(APP, "_alignElement")
|
||||||
|
assert "xAnchor = null" in body and "yAnchor = null" in body, \
|
||||||
|
"_alignElement no longer clears the anchor, so aligning an anchored " \
|
||||||
|
"element resolves to the wrong edge"
|
||||||
|
assert "el.x0" in body and "el.y0" in body, \
|
||||||
|
"_alignElement no longer moves line endpoints"
|
||||||
|
|
||||||
|
|
||||||
|
def test_align_translates_both_line_endpoints_not_just_the_start():
|
||||||
|
"""Setting only x0 (or y0) left x1/y1 behind, so aligning a line changed
|
||||||
|
its shape instead of moving it -- e.g. a line from x0=20 to x1=50 aligned
|
||||||
|
right became x0=98, x1=50, stretching rather than translating it. Both
|
||||||
|
endpoints must move by the same delta."""
|
||||||
|
body = _method_source(APP, "_alignElement")
|
||||||
|
x_branch = body[body.index("if (axis === 'x')"):body.index("} else {")]
|
||||||
|
y_branch = body[body.index("} else {"):]
|
||||||
|
assert "el.x1" in x_branch, \
|
||||||
|
"_alignElement moves x0 but not x1 -- a line's shape changes, not its position"
|
||||||
|
assert "el.y1" in y_branch, \
|
||||||
|
"_alignElement moves y0 but not y1 -- a line's shape changes, not its position"
|
||||||
|
|
||||||
|
|
||||||
|
def test_divider_stroke_is_centered_in_led_pixels():
|
||||||
|
"""A 0.5 canvas-pixel offset (correct only at SCALE=1) was applied after
|
||||||
|
scaling instead of before it, so at SCALE>1 the stroke bled into the
|
||||||
|
preceding LED row/column instead of straddling its own."""
|
||||||
|
body = _function_source(CANVAS, "_drawElement")
|
||||||
|
divider_branch = body[body.index("case 'divider'"):]
|
||||||
|
divider_branch = divider_branch[:divider_branch.index("case 'pips'")]
|
||||||
|
assert "ay * s + 0.5" not in divider_branch, \
|
||||||
|
"divider still offsets by 0.5 canvas pixels after scaling"
|
||||||
|
assert "ax * s + 0.5" not in divider_branch, \
|
||||||
|
"divider still offsets by 0.5 canvas pixels after scaling"
|
||||||
|
assert "(ay + 0.5) * s" in divider_branch and "(ax + 0.5) * s" in divider_branch, \
|
||||||
|
"divider stroke is not centered within its LED pixel"
|
||||||
|
|
||||||
|
|
||||||
|
def test_gauge_radii_are_clamped_to_zero():
|
||||||
|
"""An imported design can carry a small gauge with a wide lineWidth --
|
||||||
|
width=1, height=1, lineWidth=3 sends a negative radius into
|
||||||
|
ctx.ellipse(), which throws IndexSizeError and aborts render() for every
|
||||||
|
element still to be drawn, not just the gauge."""
|
||||||
|
body = _function_source(CANVAS, "_drawElement")
|
||||||
|
gauge_branch = body[body.index("case 'gauge'"):]
|
||||||
|
assert re.search(r"Math\.max\(0,\s*rx\s*-\s*lwPx\s*/\s*2\)", gauge_branch), \
|
||||||
|
"gauge x-radius is not clamped to zero"
|
||||||
|
assert re.search(r"Math\.max\(0,\s*ry\s*-\s*lwPx\s*/\s*2\)", gauge_branch), \
|
||||||
|
"gauge y-radius is not clamped to zero"
|
||||||
|
assert "rx - lwPx / 2" not in re.sub(r"Math\.max\(0,\s*rx\s*-\s*lwPx\s*/\s*2\)", "", gauge_branch), \
|
||||||
|
"an unclamped gauge radius is still passed to ctx.ellipse()"
|
||||||
|
|
||||||
|
|
||||||
|
#: ELEMENT_DEFAULTS in composer-canvas.js gives exactly these types a
|
||||||
|
#: `binding` object; verified against that file rather than assumed.
|
||||||
|
BOUND_TYPES = ["dynamic_text", "progress_bar", "countdown", "pips", "sparkline", "gauge"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_bound_types_constant_lists_every_bound_element_type():
|
||||||
|
"""ELEMENT_DEFAULTS in composer-canvas.js gives dynamic_text, progress_bar,
|
||||||
|
countdown, pips, sparkline and gauge a `binding` object. If a type is added
|
||||||
|
to (or removed from) that list without updating APP's BOUND_TYPES, the
|
||||||
|
checks below drift out of sync silently -- this pins the two together."""
|
||||||
|
src = APP.read_text()
|
||||||
|
match = re.search(r"const BOUND_TYPES = \[([^\]]*)\];", src)
|
||||||
|
assert match, "BOUND_TYPES constant not found in composer-app.js"
|
||||||
|
declared = {t.strip().strip("'\"") for t in match.group(1).split(",") if t.strip()}
|
||||||
|
assert declared == set(BOUND_TYPES), \
|
||||||
|
f"BOUND_TYPES {declared} does not match the bound element types {set(BOUND_TYPES)}"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("method", ["_isBound", "removeConfigVar", "_validateBeforeExport"])
|
||||||
|
def test_binding_checks_cover_every_bound_element_type(method):
|
||||||
|
"""Only dynamic_text and progress_bar were checked, so a countdown, pips,
|
||||||
|
sparkline or gauge element with an empty binding passed export validation
|
||||||
|
silently, and deleting a config var still used by one of them gave no
|
||||||
|
warning -- the plugin would read a now-missing key at runtime with no
|
||||||
|
indication why."""
|
||||||
|
body = _method_source(APP, method)
|
||||||
|
assert "BOUND_TYPES.includes(e.type)" in body, \
|
||||||
|
f"{method} does not check every bound element type via BOUND_TYPES"
|
||||||
|
assert "e.type === 'dynamic_text'" not in body, \
|
||||||
|
f"{method} still hardcodes only dynamic_text instead of BOUND_TYPES"
|
||||||
@@ -0,0 +1,225 @@
|
|||||||
|
"""A composer plugin id must never resolve outside the plugins directory.
|
||||||
|
|
||||||
|
CodeQL reported sixteen high-severity py/path-injection alerts against
|
||||||
|
web_interface/blueprints/composer.py: a request-supplied plugin_id reaching
|
||||||
|
Path(plugins_dir) / plugin_id, which is then created, written to, deleted
|
||||||
|
(shutil.rmtree) and read back.
|
||||||
|
|
||||||
|
The id was already validated by an anchored regex, so every traversal payload
|
||||||
|
was in fact rejected. What was missing was the guarantee living *with* the path
|
||||||
|
building rather than in a regex several hundred lines away -- loosen that regex
|
||||||
|
later and the traversal opens silently, with nothing at the filesystem boundary
|
||||||
|
to catch it. _plugin_dir() closes that, and is the form static analysis can see.
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||||
|
|
||||||
|
from web_interface.blueprints import composer as C # noqa: E402
|
||||||
|
|
||||||
|
#: Anything that has ever been used to climb out of a directory.
|
||||||
|
TRAVERSAL = [
|
||||||
|
"../../etc/passwd", "..", ".", "a/../../etc", "good/../../..",
|
||||||
|
"/etc/passwd", "//etc/passwd", "a\\..\\..", "a%2f..%2f..",
|
||||||
|
"....//....//etc", "a/./../../etc", "~", "~root",
|
||||||
|
"plugin/../../../../../../etc/shadow",
|
||||||
|
]
|
||||||
|
|
||||||
|
#: Rejected for shape, not traversal -- but rejected all the same.
|
||||||
|
MALFORMED = ["", "A-upper", "1-leading-digit", "-leading-dash", "has_underscore",
|
||||||
|
"has space", "has.dot", "a" * 64, "plugin\n", "plugin\n../../etc",
|
||||||
|
"\n", "plug\x00in"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def plugins_dir(tmp_path, monkeypatch):
|
||||||
|
base = tmp_path / "plugin-repos"
|
||||||
|
base.mkdir()
|
||||||
|
monkeypatch.setattr(C.composer_bp, "plugins_dir", str(base), raising=False)
|
||||||
|
return base
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("payload", TRAVERSAL)
|
||||||
|
def test_traversal_payloads_are_refused(plugins_dir, payload):
|
||||||
|
assert C._plugin_dir(payload) is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("payload", MALFORMED)
|
||||||
|
def test_malformed_ids_are_refused(plugins_dir, payload):
|
||||||
|
assert C._plugin_dir(payload) is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("payload", ["a", "my-plugin", "x9", "a" * 63])
|
||||||
|
def test_valid_ids_resolve_inside_the_base(plugins_dir, payload):
|
||||||
|
resolved = C._plugin_dir(payload)
|
||||||
|
assert resolved is not None, f"{payload!r} was rejected but is valid"
|
||||||
|
assert resolved.parent == plugins_dir.resolve(), (
|
||||||
|
f"{payload!r} resolved to {resolved}, outside {plugins_dir}")
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_payload_can_escape_even_if_the_regex_is_loosened(plugins_dir, monkeypatch):
|
||||||
|
"""The containment check must stand on its own.
|
||||||
|
|
||||||
|
This is the whole point of resolving at the filesystem boundary: if the id
|
||||||
|
pattern is ever relaxed, traversal must still be impossible. Replace the
|
||||||
|
regex with one that permits slashes and dots, then re-run the payloads.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
monkeypatch.setattr(C, "_PLUGIN_ID_RE", re.compile(r"\A[\w./\\~-]+\Z"))
|
||||||
|
import os
|
||||||
|
escaped = []
|
||||||
|
base = os.path.realpath(str(plugins_dir))
|
||||||
|
for payload in TRAVERSAL:
|
||||||
|
resolved = C._plugin_dir(payload)
|
||||||
|
if resolved is None:
|
||||||
|
continue
|
||||||
|
real = os.path.realpath(str(resolved))
|
||||||
|
if real != base and os.path.commonpath([base, real]) != base:
|
||||||
|
escaped.append((payload, real))
|
||||||
|
assert not escaped, f"these escaped the base with a loosened regex: {escaped}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_sibling_directory_with_a_shared_prefix_is_not_inside(tmp_path, monkeypatch):
|
||||||
|
"""commonpath, not startswith.
|
||||||
|
|
||||||
|
"/x/plugins-evil" starts with "/x/plugins" but is a different directory, so
|
||||||
|
a prefix test would accept it.
|
||||||
|
"""
|
||||||
|
base = tmp_path / "plugins"
|
||||||
|
base.mkdir()
|
||||||
|
(tmp_path / "plugins-evil").mkdir()
|
||||||
|
monkeypatch.setattr(C.composer_bp, "plugins_dir", str(base), raising=False)
|
||||||
|
import re
|
||||||
|
# Neutralise the two layers in front so this exercises the containment
|
||||||
|
# check itself; otherwise secure_filename rejects the payload first and a
|
||||||
|
# startswith regression would go unnoticed here.
|
||||||
|
monkeypatch.setattr(C, "_PLUGIN_ID_RE", re.compile(r"\A[\w./\\~-]+\Z"))
|
||||||
|
monkeypatch.setattr(C, "secure_filename", lambda v: v)
|
||||||
|
assert C._plugin_dir("../plugins-evil") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_containment_still_holds_if_the_sanitiser_is_defeated(plugins_dir, monkeypatch):
|
||||||
|
"""Each layer is tested on its own, not just the stack.
|
||||||
|
|
||||||
|
secure_filename's equality guard rejects every traversal payload before the
|
||||||
|
containment check sees it, so removing containment does not fail the other
|
||||||
|
tests -- which would make it look load-bearing when it is not. Neutralise
|
||||||
|
the regex *and* the sanitiser, and the realpath/commonpath check must still
|
||||||
|
refuse everything on its own.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
monkeypatch.setattr(C, "_PLUGIN_ID_RE", re.compile(r"\A[\w./\\~-]+\Z"))
|
||||||
|
monkeypatch.setattr(C, "secure_filename", lambda v: v)
|
||||||
|
import os
|
||||||
|
base = os.path.realpath(str(plugins_dir))
|
||||||
|
escaped = []
|
||||||
|
for payload in TRAVERSAL:
|
||||||
|
resolved = C._plugin_dir(payload)
|
||||||
|
if resolved is None:
|
||||||
|
continue
|
||||||
|
real = os.path.realpath(str(resolved))
|
||||||
|
# Inside the base is fine -- "...." and "~" are ordinary directory
|
||||||
|
# names on Linux, so they are not escapes. What must never happen is
|
||||||
|
# landing outside the base, or on the base itself: install() rmtrees
|
||||||
|
# its target, so the plugins root resolving to a "plugin" would wipe
|
||||||
|
# every installed plugin.
|
||||||
|
if real == base or os.path.commonpath([base, real]) != base:
|
||||||
|
escaped.append((payload, real))
|
||||||
|
assert not escaped, f"containment alone let these through: {escaped}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_secure_filename_never_rewrites_an_accepted_id(plugins_dir):
|
||||||
|
"""The sanitiser must be a no-op on everything the regex accepts.
|
||||||
|
|
||||||
|
If secure_filename ever altered an accepted id, _plugin_dir would resolve
|
||||||
|
to a *different* plugin's directory than the caller asked for -- a silent
|
||||||
|
redirect, which is worse than a refusal. The guard turns that into a
|
||||||
|
refusal; this proves the guard never has to fire in practice.
|
||||||
|
"""
|
||||||
|
import random
|
||||||
|
from werkzeug.utils import secure_filename
|
||||||
|
random.seed(1)
|
||||||
|
alphabet = "abcdefghijklmnopqrstuvwxyz0123456789-"
|
||||||
|
altered = []
|
||||||
|
for _ in range(2000):
|
||||||
|
n = random.randint(1, 63)
|
||||||
|
cand = random.choice("abcdefghijklmnopqrstuvwxyz") + "".join(
|
||||||
|
random.choice(alphabet) for _ in range(n - 1))
|
||||||
|
if C._PLUGIN_ID_RE.match(cand) and secure_filename(cand) != cand:
|
||||||
|
altered.append((cand, secure_filename(cand)))
|
||||||
|
assert not altered, f"secure_filename rewrote accepted ids: {altered[:5]}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_trailing_newline_is_not_a_valid_id():
|
||||||
|
r"""Python's `$` also matches before a trailing newline, so the original
|
||||||
|
`^...$` accepted "myplugin\n" and would have created a directory whose
|
||||||
|
name ends in one. \Z does not."""
|
||||||
|
assert C._PLUGIN_ID_RE.match("myplugin") is not None
|
||||||
|
assert C._PLUGIN_ID_RE.match("myplugin\n") is None
|
||||||
|
|
||||||
|
|
||||||
|
# --- font serving -----------------------------------------------------------
|
||||||
|
|
||||||
|
FONT_TRAVERSAL = [
|
||||||
|
"../../../etc/passwd", "../config/config.json", "..%2f..%2fetc%2fpasswd",
|
||||||
|
"PressStart2P-Regular.ttf/../../../etc/passwd", "/etc/passwd", "",
|
||||||
|
"PressStart2P-Regular.TTF", # case differs -> not the allowlisted name
|
||||||
|
"PressStart2P-Regular.ttf ", # trailing space
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve_font_refuses_a_file_that_exists_but_is_not_allowlisted(monkeypatch, tmp_path):
|
||||||
|
"""The allowlist must be what refuses it, not a missing file.
|
||||||
|
|
||||||
|
Asserting 404 on traversal payloads proves nothing here: Flask's router
|
||||||
|
will not match a path segment containing '/', and everything else 404s
|
||||||
|
simply because no such file exists. Put a real, readable file next to the
|
||||||
|
fonts and confirm it is still refused -- that is the allowlist working.
|
||||||
|
"""
|
||||||
|
fonts = tmp_path / "assets" / "fonts"
|
||||||
|
fonts.mkdir(parents=True)
|
||||||
|
(fonts / "id_rsa.ttf").write_bytes(b"PRIVATE KEY")
|
||||||
|
monkeypatch.setattr(C.composer_bp, "project_root", str(tmp_path), raising=False)
|
||||||
|
app = __import__("flask").Flask(__name__)
|
||||||
|
app.register_blueprint(C.composer_bp)
|
||||||
|
with app.test_client() as client:
|
||||||
|
resp = client.get("/api/fonts/id_rsa.ttf")
|
||||||
|
assert resp.status_code == 404, (
|
||||||
|
"a readable non-allowlisted file was served; the allowlist is not gating")
|
||||||
|
assert b"PRIVATE KEY" not in resp.data
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("payload", FONT_TRAVERSAL)
|
||||||
|
def test_serve_font_refuses_anything_not_allowlisted(payload, monkeypatch, tmp_path):
|
||||||
|
"""The name reaching the filesystem must come from the allowlist constant.
|
||||||
|
|
||||||
|
_ALLOWED_FONTS gates this endpoint, so nothing here was ever exploitable.
|
||||||
|
Building the path from the matched constant rather than the request value
|
||||||
|
is what makes that provable -- and it is why CodeQL reported two
|
||||||
|
high-severity py/path-injection alerts on an endpoint that was already
|
||||||
|
safe.
|
||||||
|
"""
|
||||||
|
monkeypatch.setattr(C.composer_bp, "project_root", str(tmp_path), raising=False)
|
||||||
|
app = C.composer_bp.name and __import__("flask").Flask(__name__)
|
||||||
|
app.register_blueprint(C.composer_bp)
|
||||||
|
with app.test_client() as client:
|
||||||
|
resp = client.get(f"/api/fonts/{payload}")
|
||||||
|
assert resp.status_code in (404, 405, 308), (
|
||||||
|
f"{payload!r} was not refused (status {resp.status_code})")
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve_font_still_serves_each_allowlisted_font(monkeypatch, tmp_path):
|
||||||
|
fonts = tmp_path / "assets" / "fonts"
|
||||||
|
fonts.mkdir(parents=True)
|
||||||
|
monkeypatch.setattr(C.composer_bp, "project_root", str(tmp_path), raising=False)
|
||||||
|
app = __import__("flask").Flask(__name__)
|
||||||
|
app.register_blueprint(C.composer_bp)
|
||||||
|
for name in C._ALLOWED_FONTS:
|
||||||
|
(fonts / name).write_bytes(b"\x00\x01ttf")
|
||||||
|
with app.test_client() as client:
|
||||||
|
resp = client.get(f"/api/fonts/{name}")
|
||||||
|
assert resp.status_code == 200, f"{name} should be served, got {resp.status_code}"
|
||||||
@@ -0,0 +1,253 @@
|
|||||||
|
"""
|
||||||
|
Tests for src/common/config_helper.py — pins the ConfigHelper contract.
|
||||||
|
|
||||||
|
Covers: load/save round trips (missing/malformed files return {} rather
|
||||||
|
than raising, non-ASCII preserved via ensure_ascii=False, top-level JSON
|
||||||
|
lists returned as-is), dot-notation get/set including the silent-failure
|
||||||
|
contract when an intermediate key holds a non-dict, merge_configs deep
|
||||||
|
semantics with NO aliasing of the base config (the fixed bug — the old
|
||||||
|
shallow copy let mutations of the merged result leak into base's nested
|
||||||
|
dicts), simplified schema validation including the caught-TypeError path
|
||||||
|
when a schema 'type' is given as a string, plugin config key conventions
|
||||||
|
('{plugin_id}_config', enabled defaults True), and required-key checks
|
||||||
|
where a key present with value None counts as present.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.common.config_helper import ConfigHelper
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def helper():
|
||||||
|
return ConfigHelper()
|
||||||
|
|
||||||
|
|
||||||
|
class TestLoadConfig:
|
||||||
|
def test_missing_file_returns_empty_dict(self, helper, tmp_path):
|
||||||
|
assert helper.load_config(tmp_path / "nope.json") == {}
|
||||||
|
|
||||||
|
def test_malformed_json_returns_empty_dict(self, helper, tmp_path):
|
||||||
|
path = tmp_path / "bad.json"
|
||||||
|
path.write_text("{ this is not json", encoding="utf-8")
|
||||||
|
assert helper.load_config(path) == {}
|
||||||
|
|
||||||
|
def test_top_level_list_returned_as_is(self, helper, tmp_path):
|
||||||
|
# load_config does not enforce a dict shape: a JSON list comes
|
||||||
|
# straight back. Pinned as a characterization of current behavior.
|
||||||
|
path = tmp_path / "list.json"
|
||||||
|
path.write_text("[1, 2, 3]", encoding="utf-8")
|
||||||
|
assert helper.load_config(path) == [1, 2, 3]
|
||||||
|
|
||||||
|
|
||||||
|
class TestSaveConfig:
|
||||||
|
def test_round_trip(self, helper, tmp_path):
|
||||||
|
path = tmp_path / "config.json"
|
||||||
|
config = {'display': {'hardware': {'rows': 32}}, 'timezone': 'UTC'}
|
||||||
|
assert helper.save_config(config, path) is True
|
||||||
|
assert helper.load_config(path) == config
|
||||||
|
|
||||||
|
def test_creates_parent_directories(self, helper, tmp_path):
|
||||||
|
path = tmp_path / "deep" / "nested" / "config.json"
|
||||||
|
assert helper.save_config({'a': 1}, path) is True
|
||||||
|
assert path.exists()
|
||||||
|
assert helper.load_config(path) == {'a': 1}
|
||||||
|
|
||||||
|
def test_non_ascii_survives_round_trip(self, helper, tmp_path):
|
||||||
|
path = tmp_path / "config.json"
|
||||||
|
config = {'city': 'Zürich', 'note': 'météo ☀'}
|
||||||
|
assert helper.save_config(config, path) is True
|
||||||
|
assert helper.load_config(path) == config
|
||||||
|
# ensure_ascii=False: characters are written raw, not \u-escaped
|
||||||
|
assert 'Zürich' in path.read_text(encoding='utf-8')
|
||||||
|
|
||||||
|
def test_directory_path_returns_false_not_raise(self, helper, tmp_path):
|
||||||
|
assert helper.save_config({'a': 1}, tmp_path) is False
|
||||||
|
|
||||||
|
|
||||||
|
class TestGetConfigValue:
|
||||||
|
def test_dot_notation_hit(self, helper):
|
||||||
|
config = {'display': {'hardware': {'rows': 32}}}
|
||||||
|
assert helper.get_config_value(config, 'display.hardware.rows') == 32
|
||||||
|
|
||||||
|
def test_missing_returns_default(self, helper):
|
||||||
|
sentinel = object()
|
||||||
|
assert helper.get_config_value({}, 'display.rows', default=sentinel) is sentinel
|
||||||
|
|
||||||
|
def test_intermediate_non_dict_returns_default(self, helper):
|
||||||
|
config = {'display': 'not-a-dict'}
|
||||||
|
assert helper.get_config_value(config, 'display.hardware.rows', default=64) == 64
|
||||||
|
|
||||||
|
def test_required_missing_raises_keyerror(self, helper):
|
||||||
|
with pytest.raises(KeyError):
|
||||||
|
helper.get_config_value({}, 'display.rows', required=True)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSetConfigValue:
|
||||||
|
def test_sets_top_level(self, helper):
|
||||||
|
config = {}
|
||||||
|
helper.set_config_value(config, 'timezone', 'UTC')
|
||||||
|
assert config == {'timezone': 'UTC'}
|
||||||
|
|
||||||
|
def test_auto_creates_intermediates(self, helper):
|
||||||
|
config = {}
|
||||||
|
helper.set_config_value(config, 'display.hardware.rows', 32)
|
||||||
|
assert config == {'display': {'hardware': {'rows': 32}}}
|
||||||
|
|
||||||
|
def test_silent_failure_on_non_dict_intermediate(self, helper):
|
||||||
|
# 'a' exists but holds an int; the assignment attempt raises
|
||||||
|
# TypeError internally, which set_config_value swallows and logs.
|
||||||
|
# The config is left unchanged — pinned silent-failure contract.
|
||||||
|
config = {'a': 5}
|
||||||
|
helper.set_config_value(config, 'a.b', 1)
|
||||||
|
assert config == {'a': 5}
|
||||||
|
|
||||||
|
|
||||||
|
class TestMergeConfigs:
|
||||||
|
def test_nested_dicts_merge_recursively(self, helper):
|
||||||
|
base = {'display': {'rows': 32, 'cols': 64}, 'timezone': 'UTC'}
|
||||||
|
override = {'display': {'cols': 128, 'brightness': 90}}
|
||||||
|
merged = helper.merge_configs(base, override)
|
||||||
|
assert merged == {
|
||||||
|
'display': {'rows': 32, 'cols': 128, 'brightness': 90},
|
||||||
|
'timezone': 'UTC',
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_scalar_override_wins_over_dict(self, helper):
|
||||||
|
merged = helper.merge_configs({'display': {'rows': 32}}, {'display': 7})
|
||||||
|
assert merged['display'] == 7
|
||||||
|
|
||||||
|
def test_dict_override_wins_over_scalar(self, helper):
|
||||||
|
merged = helper.merge_configs({'display': 7}, {'display': {'rows': 32}})
|
||||||
|
assert merged['display'] == {'rows': 32}
|
||||||
|
|
||||||
|
def test_no_aliasing_of_base(self, helper):
|
||||||
|
# Post-fix: merge deep-copies base, so mutating the result never
|
||||||
|
# leaks back into the caller's base config.
|
||||||
|
base = {'display': {'x': 1}}
|
||||||
|
merged = helper.merge_configs(base, {})
|
||||||
|
assert merged['display'] is not base['display']
|
||||||
|
merged['display']['x'] = 99
|
||||||
|
assert base['display']['x'] == 1
|
||||||
|
|
||||||
|
def test_inputs_unchanged(self, helper):
|
||||||
|
base = {'a': {'b': 1}}
|
||||||
|
override = {'a': {'c': 2}}
|
||||||
|
helper.merge_configs(base, override)
|
||||||
|
assert base == {'a': {'b': 1}}
|
||||||
|
assert override == {'a': {'c': 2}}
|
||||||
|
|
||||||
|
def test_no_aliasing_of_override_values(self, helper):
|
||||||
|
# The non-recursive branch must deep-copy the override value too:
|
||||||
|
# mutating a merged-in list or dict must not reach back into
|
||||||
|
# override_config.
|
||||||
|
override = {'teams': ['A', 'B'], 'nested': {'x': [1]}}
|
||||||
|
merged = helper.merge_configs({}, override)
|
||||||
|
merged['teams'].append('C')
|
||||||
|
merged['nested']['x'].append(2)
|
||||||
|
assert override == {'teams': ['A', 'B'], 'nested': {'x': [1]}}
|
||||||
|
|
||||||
|
|
||||||
|
class TestValidateConfig:
|
||||||
|
def test_no_schema_dict_is_valid(self, helper):
|
||||||
|
assert helper.validate_config({'a': 1}) is True
|
||||||
|
|
||||||
|
def test_no_schema_list_is_invalid(self, helper):
|
||||||
|
assert helper.validate_config([1, 2]) is False
|
||||||
|
|
||||||
|
def test_required_key_missing_is_invalid(self, helper):
|
||||||
|
schema = {'rows': {'required': True, 'type': int}}
|
||||||
|
assert helper.validate_config({}, schema) is False
|
||||||
|
|
||||||
|
def test_optional_key_missing_is_valid(self, helper):
|
||||||
|
schema = {'rows': {'required': False, 'type': int}}
|
||||||
|
assert helper.validate_config({}, schema) is True
|
||||||
|
|
||||||
|
def test_wrong_type_is_invalid(self, helper):
|
||||||
|
schema = {'rows': {'type': int}}
|
||||||
|
assert helper.validate_config({'rows': 'thirty-two'}, schema) is False
|
||||||
|
assert helper.validate_config({'rows': 32}, schema) is True
|
||||||
|
|
||||||
|
def test_allowed_values_violation_is_invalid(self, helper):
|
||||||
|
schema = {'mode': {'allowed_values': ['clock', 'weather']}}
|
||||||
|
assert helper.validate_config({'mode': 'stocks'}, schema) is False
|
||||||
|
assert helper.validate_config({'mode': 'clock'}, schema) is True
|
||||||
|
|
||||||
|
def test_string_type_in_schema_is_invalid_via_typeerror(self, helper):
|
||||||
|
# 'type' given as the STRING "int" makes isinstance() raise
|
||||||
|
# TypeError; validate_config catches it and returns False rather
|
||||||
|
# than raising. Pinned characterization.
|
||||||
|
schema = {'rows': {'type': 'int'}}
|
||||||
|
assert helper.validate_config({'rows': 32}, schema) is False
|
||||||
|
|
||||||
|
|
||||||
|
class TestPluginConfigHelpers:
|
||||||
|
def test_get_plugin_config_uses_suffixed_key(self, helper):
|
||||||
|
plugin_cfg = {'enabled': True, 'display_duration': 30}
|
||||||
|
assert helper.get_plugin_config({'clock_config': plugin_cfg}, 'clock') == plugin_cfg
|
||||||
|
|
||||||
|
def test_get_plugin_config_bare_id_key_not_found(self, helper):
|
||||||
|
# Only '{plugin_id}_config' is consulted — a bare 'clock' section
|
||||||
|
# is invisible to this helper. Pinned key contract.
|
||||||
|
assert helper.get_plugin_config({'clock': {'enabled': True}}, 'clock') == {}
|
||||||
|
|
||||||
|
def test_create_default_config_wraps_in_suffixed_key(self, helper):
|
||||||
|
defaults = {'enabled': True}
|
||||||
|
assert helper.create_default_config('clock', defaults) == {'clock_config': defaults}
|
||||||
|
|
||||||
|
def test_is_plugin_enabled_defaults_true_for_unknown(self, helper):
|
||||||
|
assert helper.is_plugin_enabled({}, 'clock') is True
|
||||||
|
|
||||||
|
def test_is_plugin_enabled_false_when_disabled(self, helper):
|
||||||
|
config = {'clock_config': {'enabled': False}}
|
||||||
|
assert helper.is_plugin_enabled(config, 'clock') is False
|
||||||
|
|
||||||
|
def test_is_plugin_enabled_ignores_bare_id_key(self, helper):
|
||||||
|
# Disabled under the wrong key -> still reported enabled (default).
|
||||||
|
config = {'clock': {'enabled': False}}
|
||||||
|
assert helper.is_plugin_enabled(config, 'clock') is True
|
||||||
|
|
||||||
|
|
||||||
|
class TestSportsAndDisplayHelpers:
|
||||||
|
def test_get_display_config(self, helper):
|
||||||
|
display = {'hardware': {'rows': 32}}
|
||||||
|
assert helper.get_display_config({'display': display}) == display
|
||||||
|
assert helper.get_display_config({}) == {}
|
||||||
|
|
||||||
|
def test_get_sports_config_uses_scoreboard_suffix(self, helper):
|
||||||
|
sport_cfg = {'favorite_teams': ['TB']}
|
||||||
|
config = {'football_scoreboard': sport_cfg}
|
||||||
|
assert helper.get_sports_config(config, 'football') == sport_cfg
|
||||||
|
assert helper.get_sports_config(config, 'hockey') == {}
|
||||||
|
|
||||||
|
def test_get_favorite_teams(self, helper):
|
||||||
|
config = {'football_scoreboard': {'favorite_teams': ['TB', 'DAL']}}
|
||||||
|
assert helper.get_favorite_teams(config, 'football') == ['TB', 'DAL']
|
||||||
|
assert helper.get_favorite_teams({}, 'football') == []
|
||||||
|
|
||||||
|
def test_get_display_modes(self, helper):
|
||||||
|
modes = {'live': True, 'recent': False}
|
||||||
|
config = {'football_scoreboard': {'display_modes': modes}}
|
||||||
|
assert helper.get_display_modes(config, 'football') == modes
|
||||||
|
assert helper.get_display_modes({}, 'football') == {}
|
||||||
|
|
||||||
|
|
||||||
|
class TestValidateRequiredKeys:
|
||||||
|
def test_returns_missing_subset(self, helper):
|
||||||
|
config = {'a': 1, 'c': {'d': 2}}
|
||||||
|
missing = helper.validate_required_keys(config, ['a', 'b', 'c.d', 'c.e'])
|
||||||
|
assert missing == ['b', 'c.e']
|
||||||
|
|
||||||
|
def test_dot_notation_present(self, helper):
|
||||||
|
config = {'display': {'hardware': {'rows': 32}}}
|
||||||
|
assert helper.validate_required_keys(config, ['display.hardware.rows']) == []
|
||||||
|
|
||||||
|
def test_empty_requirements(self, helper):
|
||||||
|
assert helper.validate_required_keys({'a': 1}, []) == []
|
||||||
|
|
||||||
|
def test_present_with_none_counts_as_present(self, helper):
|
||||||
|
# _has_key checks key membership, not truthiness — a key set to
|
||||||
|
# None is NOT reported missing. Pinned semantics.
|
||||||
|
assert helper.validate_required_keys({'a': None}, ['a']) == []
|
||||||
@@ -0,0 +1,143 @@
|
|||||||
|
"""GET /config/main must not hand out credentials.
|
||||||
|
|
||||||
|
The endpoint returned the raw config to anyone who could reach the port, and
|
||||||
|
this web interface has no authentication of any kind. Measured against a live
|
||||||
|
rig, an unauthenticated request returned:
|
||||||
|
|
||||||
|
github.api_token 40 chars
|
||||||
|
incoming-packages.ha_token 183 chars
|
||||||
|
jellyfin-now-playing.api_key 32 chars
|
||||||
|
ledmatrix-weather.api_key 32 chars
|
||||||
|
on-air.mqtt_password 8 chars
|
||||||
|
youtube.api_key 20 chars
|
||||||
|
youtube-stats.api_key 39 chars
|
||||||
|
|
||||||
|
A GitHub token and a Home Assistant long-lived token among them.
|
||||||
|
|
||||||
|
The x-secret masking the plugin config endpoints use does not apply here: this
|
||||||
|
endpoint never consults a schema, and core keys such as github.api_token have
|
||||||
|
no schema to carry the marker. Several of those fields *are* tagged x-secret in
|
||||||
|
their plugin's schema and were still returned in full, which is what makes the
|
||||||
|
schema route the wrong one to rely on for this endpoint.
|
||||||
|
|
||||||
|
Matching on field name is blunt. For a whole-config dump it is the right
|
||||||
|
default: anything named like a credential should not leave the process, and a
|
||||||
|
new plugin that adds a differently-shaped secret is covered without anyone
|
||||||
|
remembering to tag it.
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from web_interface.blueprints.api_v3 import (
|
||||||
|
_looks_like_a_credential,
|
||||||
|
_redact_credentials,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("name", [
|
||||||
|
"password", "mqtt_password", "opensky_password", "passwd",
|
||||||
|
"api_key", "apikey", "API_KEY", "flightaware_api_key",
|
||||||
|
"token", "ha_token", "api_token", "access_token",
|
||||||
|
"secret", "client_secret", "spotify_client_secret",
|
||||||
|
"access_key", "private_key",
|
||||||
|
])
|
||||||
|
def test_credential_names_are_recognised(name):
|
||||||
|
assert _looks_like_a_credential(name)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("name", [
|
||||||
|
"timezone", "city", "brightness", "enabled", "update_interval",
|
||||||
|
"favorite_teams", "display_duration", "keyword",
|
||||||
|
])
|
||||||
|
def test_ordinary_names_are_left_alone(name):
|
||||||
|
assert not _looks_like_a_credential(name)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_measured_leak_is_closed():
|
||||||
|
"""The exact shape taken off the rig."""
|
||||||
|
config = {
|
||||||
|
"github": {"api_token": "ghp_" + "x" * 36},
|
||||||
|
"incoming-packages": {"ha_token": "y" * 183, "enabled": True},
|
||||||
|
"jellyfin-now-playing": {"api_key": "z" * 32},
|
||||||
|
"on-air": {"mqtt_password": "hunter22"},
|
||||||
|
"youtube": {"api_key": "k" * 20},
|
||||||
|
"timezone": "America/New_York",
|
||||||
|
}
|
||||||
|
out = _redact_credentials(config)
|
||||||
|
assert out["github"]["api_token"] == ""
|
||||||
|
assert out["incoming-packages"]["ha_token"] == ""
|
||||||
|
assert out["jellyfin-now-playing"]["api_key"] == ""
|
||||||
|
assert out["on-air"]["mqtt_password"] == ""
|
||||||
|
assert out["youtube"]["api_key"] == ""
|
||||||
|
# Everything else survives, or the config editor breaks.
|
||||||
|
assert out["timezone"] == "America/New_York"
|
||||||
|
assert out["incoming-packages"]["enabled"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_nested_and_listed_credentials_are_reached():
|
||||||
|
config = {"a": {"b": {"c": {"password": "p"}}},
|
||||||
|
"feeds": [{"name": "x", "api_key": "k"}, {"name": "y"}]}
|
||||||
|
out = _redact_credentials(config)
|
||||||
|
assert out["a"]["b"]["c"]["password"] == ""
|
||||||
|
assert out["feeds"][0]["api_key"] == ""
|
||||||
|
assert out["feeds"][0]["name"] == "x"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_original_is_not_mutated():
|
||||||
|
"""The caller holds the live config; redaction must not edit it in place."""
|
||||||
|
config = {"github": {"api_token": "keepme"}}
|
||||||
|
_redact_credentials(config)
|
||||||
|
assert config["github"]["api_token"] == "keepme"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_credential_shaped_container_is_still_walked():
|
||||||
|
"""`secrets: {...}` is a section name, not a value to blank."""
|
||||||
|
config = {"secrets": {"api_key": "k", "note": "keep"}}
|
||||||
|
out = _redact_credentials(config)
|
||||||
|
assert out["secrets"]["api_key"] == ""
|
||||||
|
assert out["secrets"]["note"] == "keep"
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_dict_input_passes_through():
|
||||||
|
assert _redact_credentials("plain") == "plain"
|
||||||
|
assert _redact_credentials(7) == 7
|
||||||
|
assert _redact_credentials(None) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_endpoint_itself_redacts():
|
||||||
|
"""Through the view function, not the helper.
|
||||||
|
|
||||||
|
The helper tests above all passed with the route still returning
|
||||||
|
`config` -- reverting the one line that calls the redactor changed
|
||||||
|
nothing, because nothing exercised the route. A property asserted on a
|
||||||
|
helper is not a property asserted on the endpoint, and it is the endpoint
|
||||||
|
that is exposed to the network.
|
||||||
|
"""
|
||||||
|
import json as _json
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import flask
|
||||||
|
|
||||||
|
from web_interface.blueprints import api_v3 as mod
|
||||||
|
|
||||||
|
raw = {"github": {"api_token": "ghp_secret_value"},
|
||||||
|
"timezone": "America/New_York"}
|
||||||
|
|
||||||
|
manager = MagicMock()
|
||||||
|
manager.load_config.return_value = raw
|
||||||
|
previous = getattr(mod.api_v3, "config_manager", None)
|
||||||
|
mod.api_v3.config_manager = manager
|
||||||
|
|
||||||
|
app = flask.Flask(__name__)
|
||||||
|
try:
|
||||||
|
with app.test_request_context("/config/main"):
|
||||||
|
response = mod.get_main_config()
|
||||||
|
payload = response.get_json() if hasattr(response, "get_json") else _json.loads(response[0].data)
|
||||||
|
finally:
|
||||||
|
mod.api_v3.config_manager = previous
|
||||||
|
|
||||||
|
data = payload["data"]
|
||||||
|
assert data["github"]["api_token"] == "", (
|
||||||
|
"the endpoint returned the token; the redactor is not wired in")
|
||||||
|
assert data["timezone"] == "America/New_York"
|
||||||
|
# And the config the manager handed over is untouched.
|
||||||
|
assert raw["github"]["api_token"] == "ghp_secret_value"
|
||||||
@@ -177,3 +177,159 @@ class TestLoadFastPath:
|
|||||||
os.utime(config_file, ns=(1_000_000_000, 1_000_000_000))
|
os.utime(config_file, ns=(1_000_000_000, 1_000_000_000))
|
||||||
|
|
||||||
assert manager.load_config()["timezone"] == "AAA" # stale, by design
|
assert manager.load_config()["timezone"] == "AAA" # stale, by design
|
||||||
|
|
||||||
|
|
||||||
|
class TestArraySecretStripAndMerge:
|
||||||
|
"""Array-item secrets round-trip (parallel-placeholder lists).
|
||||||
|
|
||||||
|
secret_helpers.separate_secrets emits array secrets as a list parallel
|
||||||
|
to the regular list, with {} for items that carry no secrets. Strip
|
||||||
|
must remove the secret fields from config.json while preserving item
|
||||||
|
indices; load must merge them back into the right items. The regular
|
||||||
|
list's length is authoritative in both directions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_strip_removes_array_item_secrets_keeps_indices(self, tmp_path):
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
data = {"plugin": {"accounts": [
|
||||||
|
{"name": "a", "token": "ta"},
|
||||||
|
{"name": "b"},
|
||||||
|
]}}
|
||||||
|
secrets = {"plugin": {"accounts": [{"token": "ta"}, {}]}}
|
||||||
|
stripped = manager._strip_secrets_recursive(data, secrets)
|
||||||
|
assert stripped == {"plugin": {"accounts": [{"name": "a"}, {"name": "b"}]}}
|
||||||
|
|
||||||
|
def test_strip_keeps_all_placeholder_items(self, tmp_path):
|
||||||
|
# Even when every item strips to nothing extra, the list survives
|
||||||
|
# with its indices — required for merge-on-load alignment.
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
data = {"accounts": [{"token": "t1"}, {"token": "t2"}]}
|
||||||
|
secrets = {"accounts": [{"token": "t1"}, {"token": "t2"}]}
|
||||||
|
stripped = manager._strip_secrets_recursive(data, secrets)
|
||||||
|
assert stripped == {"accounts": [{}, {}]}
|
||||||
|
|
||||||
|
def test_strip_whole_scalar_array_secret_drops_key(self, tmp_path):
|
||||||
|
# A list of secret scalars is a whole-key secret, not the parallel
|
||||||
|
# shape — the key must vanish from config.json entirely.
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
data = {"recovery_codes": ["a", "b"], "city": "Austin"}
|
||||||
|
secrets = {"recovery_codes": ["a", "b"]}
|
||||||
|
stripped = manager._strip_secrets_recursive(data, secrets)
|
||||||
|
assert stripped == {"city": "Austin"}
|
||||||
|
|
||||||
|
def test_strip_shape_mismatch_drops_key(self, tmp_path):
|
||||||
|
# Conservative contract: if the shapes disagree, never leak.
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
data = {"accounts": {"name": "not-a-list"}}
|
||||||
|
secrets = {"accounts": [{"token": "t"}]}
|
||||||
|
stripped = manager._strip_secrets_recursive(data, secrets)
|
||||||
|
assert stripped == {}
|
||||||
|
|
||||||
|
def test_strip_ignores_extra_secrets_entries(self, tmp_path):
|
||||||
|
# Regular list length is authoritative: a user deleted an item.
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
data = {"accounts": [{"name": "a", "token": "ta"}]}
|
||||||
|
secrets = {"accounts": [{"token": "ta"}, {"token": "tb"}]}
|
||||||
|
stripped = manager._strip_secrets_recursive(data, secrets)
|
||||||
|
assert stripped == {"accounts": [{"name": "a"}]}
|
||||||
|
|
||||||
|
def test_merge_restores_array_item_secrets(self, tmp_path):
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
target = {"accounts": [{"name": "a"}, {"name": "b"}]}
|
||||||
|
manager._deep_merge(target, {"accounts": [{"token": "ta"}, {}]})
|
||||||
|
assert target == {"accounts": [
|
||||||
|
{"name": "a", "token": "ta"},
|
||||||
|
{"name": "b"},
|
||||||
|
]}
|
||||||
|
|
||||||
|
def test_merge_ignores_extra_secrets_entries_with_warning(self, tmp_path, caplog):
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
target = {"accounts": [{"name": "a"}]}
|
||||||
|
with caplog.at_level("WARNING"):
|
||||||
|
manager._deep_merge(
|
||||||
|
target, {"accounts": [{"token": "ta"}, {"token": "ghost"}]})
|
||||||
|
assert target == {"accounts": [{"name": "a", "token": "ta"}]}
|
||||||
|
assert any("longer than the config list" in r.message for r in caplog.records)
|
||||||
|
|
||||||
|
def test_merge_non_dict_item_replaced_by_secret(self, tmp_path):
|
||||||
|
# Shape drift inside the list: the secret wins for that index.
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
target = {"accounts": ["oddball", {"name": "b"}]}
|
||||||
|
manager._deep_merge(target, {"accounts": [{"token": "ta"}, {}]})
|
||||||
|
assert target == {"accounts": [{"token": "ta"}, {"name": "b"}]}
|
||||||
|
|
||||||
|
def test_merge_whole_scalar_array_still_replaces(self, tmp_path):
|
||||||
|
# Legacy behavior preserved: a non-parallel list replaces wholesale.
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
target = {"recovery_codes": ["old"]}
|
||||||
|
manager._deep_merge(target, {"recovery_codes": ["new1", "new2"]})
|
||||||
|
assert target == {"recovery_codes": ["new1", "new2"]}
|
||||||
|
|
||||||
|
def test_full_save_load_round_trip(self, tmp_path):
|
||||||
|
# End to end on real files: save strips array secrets out of
|
||||||
|
# config.json; load merges them back into the right items.
|
||||||
|
manager = make_manager(
|
||||||
|
tmp_path,
|
||||||
|
config={"plugin": {"accounts": [
|
||||||
|
{"name": "a", "token": "s3cret-a"},
|
||||||
|
{"name": "b", "token": "s3cret-b"},
|
||||||
|
]}},
|
||||||
|
secrets={"plugin": {"accounts": [
|
||||||
|
{"token": "s3cret-a"}, {"token": "s3cret-b"},
|
||||||
|
]}},
|
||||||
|
)
|
||||||
|
loaded = manager.load_config()
|
||||||
|
assert loaded["plugin"]["accounts"][0]["token"] == "s3cret-a"
|
||||||
|
|
||||||
|
manager.save_config(loaded)
|
||||||
|
|
||||||
|
raw = (tmp_path / "config.json").read_text()
|
||||||
|
assert "s3cret" not in raw
|
||||||
|
on_disk = json.loads(raw)
|
||||||
|
assert on_disk["plugin"]["accounts"] == [{"name": "a"}, {"name": "b"}]
|
||||||
|
|
||||||
|
# A fresh manager (constructed directly — make_manager would
|
||||||
|
# overwrite the just-saved config.json) re-merges from the secrets
|
||||||
|
# file on load.
|
||||||
|
fresh = ConfigManager(config_path=str(tmp_path / "config.json"),
|
||||||
|
secrets_path=str(tmp_path / "config_secrets.json"))
|
||||||
|
fresh.template_path = str(tmp_path / "no-template.json")
|
||||||
|
reloaded = fresh.load_config()
|
||||||
|
assert reloaded["plugin"]["accounts"] == [
|
||||||
|
{"name": "a", "token": "s3cret-a"},
|
||||||
|
{"name": "b", "token": "s3cret-b"},
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_whole_item_secret_list_never_leaks_values(self, tmp_path):
|
||||||
|
# When the ENTIRE array item is secret (schema marks both key[]
|
||||||
|
# and key[].field), separate_secrets stores the full item dicts in
|
||||||
|
# the secrets file. That shape also matches the parallel-list
|
||||||
|
# discriminator — which is safe: strip drops every leaf key that
|
||||||
|
# appears in the secret item, so only empty {} skeletons (item
|
||||||
|
# count, no values) can reach config.json, and merge-on-load
|
||||||
|
# restores the full items from those skeletons.
|
||||||
|
from src.web_interface.secret_helpers import (
|
||||||
|
find_secret_fields, separate_secrets)
|
||||||
|
schema_props = {"accounts": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {"type": "object", "x-secret": True, "properties": {
|
||||||
|
"id": {"type": "string"},
|
||||||
|
"token": {"type": "string", "x-secret": True},
|
||||||
|
}},
|
||||||
|
}}
|
||||||
|
paths = find_secret_fields(schema_props)
|
||||||
|
assert paths == {"accounts[]", "accounts[].token"}
|
||||||
|
full = {"accounts": [{"id": "i1", "token": "s3cret-a"},
|
||||||
|
{"id": "i2", "token": "s3cret-b"}]}
|
||||||
|
_, secrets = separate_secrets(full, paths)
|
||||||
|
assert secrets == full # whole items are secret
|
||||||
|
|
||||||
|
manager = make_manager(tmp_path)
|
||||||
|
stripped = manager._strip_secrets_recursive(full, secrets)
|
||||||
|
assert stripped == {"accounts": [{}, {}]}
|
||||||
|
|
||||||
|
raw = json.dumps(stripped)
|
||||||
|
assert "s3cret" not in raw and "i1" not in raw
|
||||||
|
|
||||||
|
manager._deep_merge(stripped, secrets)
|
||||||
|
assert stripped == full # round trip restores the items
|
||||||
|
|||||||
@@ -0,0 +1,122 @@
|
|||||||
|
"""The gate must read the core version from disk, not from its own import.
|
||||||
|
|
||||||
|
`from src import __version__` binds whatever the process loaded at start. The
|
||||||
|
web UI is a long-lived service of its own (ledmatrix-web.service), and updating
|
||||||
|
the core replaces files on disk without restarting it -- the update route says
|
||||||
|
so and asks the user to restart, but its prompt named only the *display*
|
||||||
|
service, so a user who followed it left the web process holding the old number.
|
||||||
|
|
||||||
|
The plugin store's gate lives in that web process, so being stale by exactly
|
||||||
|
one release is the case that bites: every plugin flooring on the release you
|
||||||
|
just installed is refused, with a message blaming a core version that is
|
||||||
|
already correct on disk.
|
||||||
|
|
||||||
|
Observed on hardware: after updating a rig to 3.3.0 and restarting only the
|
||||||
|
display service, all eight sports scoreboards were refused with
|
||||||
|
"supports LEDMatrix >=3.3.0, but this system is running 3.2.0" while
|
||||||
|
src/__init__.py on that machine read 3.3.0.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from src.plugin_system import compatibility
|
||||||
|
|
||||||
|
|
||||||
|
class TestCurrentCoreVersion:
|
||||||
|
def test_it_reads_the_file_rather_than_the_imported_value(self, monkeypatch, tmp_path):
|
||||||
|
# Simulate a process whose import predates the update: the module
|
||||||
|
# object says 3.2.0 while the file on disk says 3.3.0.
|
||||||
|
import src
|
||||||
|
monkeypatch.setattr(src, "__version__", "3.2.0")
|
||||||
|
fake = tmp_path / "__init__.py"
|
||||||
|
fake.write_text('__version__ = "3.3.0"\n', encoding="utf-8")
|
||||||
|
monkeypatch.setattr(compatibility, "_VERSION_FILE", fake)
|
||||||
|
assert compatibility.current_core_version() == "3.3.0"
|
||||||
|
|
||||||
|
def test_it_matches_the_real_file_by_default(self):
|
||||||
|
import src
|
||||||
|
assert compatibility.current_core_version() == src.__version__
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("body", [
|
||||||
|
"__version__ = '3.4.1'\n",
|
||||||
|
'__version__="3.4.1"\n',
|
||||||
|
'"""doc"""\n\n__version__ = "3.4.1" # trailing comment\n',
|
||||||
|
])
|
||||||
|
def test_it_tolerates_the_ways_that_line_gets_written(self, monkeypatch, tmp_path, body):
|
||||||
|
fake = tmp_path / "__init__.py"
|
||||||
|
fake.write_text(body, encoding="utf-8")
|
||||||
|
monkeypatch.setattr(compatibility, "_VERSION_FILE", fake)
|
||||||
|
assert compatibility.current_core_version() == "3.4.1"
|
||||||
|
|
||||||
|
def test_a_missing_file_falls_back_to_the_import(self, monkeypatch, tmp_path):
|
||||||
|
# Never worse than before: an unreadable file returns what the old
|
||||||
|
# code would have returned.
|
||||||
|
import src
|
||||||
|
monkeypatch.setattr(src, "__version__", "3.2.0")
|
||||||
|
monkeypatch.setattr(compatibility, "_VERSION_FILE", tmp_path / "gone.py")
|
||||||
|
assert compatibility.current_core_version() == "3.2.0"
|
||||||
|
|
||||||
|
def test_a_file_without_the_line_falls_back(self, monkeypatch, tmp_path):
|
||||||
|
import src
|
||||||
|
monkeypatch.setattr(src, "__version__", "3.2.0")
|
||||||
|
fake = tmp_path / "__init__.py"
|
||||||
|
fake.write_text("# no version here\n", encoding="utf-8")
|
||||||
|
monkeypatch.setattr(compatibility, "_VERSION_FILE", fake)
|
||||||
|
assert compatibility.current_core_version() == "3.2.0"
|
||||||
|
|
||||||
|
def test_it_never_raises(self, monkeypatch, tmp_path):
|
||||||
|
# This runs on the install path; an exception here would surface as a
|
||||||
|
# failed update rather than a version mismatch.
|
||||||
|
bad = tmp_path / "__init__.py"
|
||||||
|
bad.write_bytes(b"\xff\xfe\x00 not utf-8 \xff")
|
||||||
|
monkeypatch.setattr(compatibility, "_VERSION_FILE", bad)
|
||||||
|
assert isinstance(compatibility.current_core_version(), str)
|
||||||
|
|
||||||
|
|
||||||
|
class TestTheBugItFixes:
|
||||||
|
def test_a_stale_import_no_longer_refuses_a_compatible_plugin(self, monkeypatch, tmp_path):
|
||||||
|
"""The exact hardware failure, as a test."""
|
||||||
|
import src
|
||||||
|
monkeypatch.setattr(src, "__version__", "3.2.0") # what the process holds
|
||||||
|
fake = tmp_path / "__init__.py"
|
||||||
|
fake.write_text('__version__ = "3.3.0"\n', encoding="utf-8") # what is on disk
|
||||||
|
monkeypatch.setattr(compatibility, "_VERSION_FILE", fake)
|
||||||
|
|
||||||
|
manifest = {"name": "Hockey Scoreboard", "min_ledmatrix_version": "3.3.0"}
|
||||||
|
|
||||||
|
stale_ok, _ = compatibility.check(manifest, src.__version__)
|
||||||
|
assert stale_ok is False, "precondition: the stale value is what refused it"
|
||||||
|
|
||||||
|
fresh_ok, reason = compatibility.check(
|
||||||
|
manifest, compatibility.current_core_version())
|
||||||
|
assert fresh_ok is True, f"the disk version must allow it, got: {reason}"
|
||||||
|
|
||||||
|
def test_it_still_refuses_when_the_core_really_is_too_old(self, monkeypatch, tmp_path):
|
||||||
|
# The gate must not become permissive: a genuinely old core still says no.
|
||||||
|
fake = tmp_path / "__init__.py"
|
||||||
|
fake.write_text('__version__ = "3.2.0"\n', encoding="utf-8")
|
||||||
|
monkeypatch.setattr(compatibility, "_VERSION_FILE", fake)
|
||||||
|
ok, reason = compatibility.check(
|
||||||
|
{"name": "Hockey", "min_ledmatrix_version": "3.3.0"},
|
||||||
|
compatibility.current_core_version())
|
||||||
|
assert ok is False
|
||||||
|
assert "3.2.0" in (reason or "")
|
||||||
|
|
||||||
|
|
||||||
|
class TestCallSites:
|
||||||
|
@pytest.mark.parametrize("module", [
|
||||||
|
"src.plugin_system.store_manager",
|
||||||
|
"src.plugin_system.plugin_loader",
|
||||||
|
])
|
||||||
|
def test_no_gate_binds_the_version_at_import(self, module):
|
||||||
|
"""Catch a future call site reintroducing the stale read."""
|
||||||
|
import inspect
|
||||||
|
mod = importlib.import_module(module)
|
||||||
|
source = inspect.getsource(mod)
|
||||||
|
assert "from src import __version__ as core_version" not in source, (
|
||||||
|
f"{module} binds __version__ at import; use "
|
||||||
|
f"compatibility.current_core_version() so a long-lived process "
|
||||||
|
f"sees a core update.")
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user