Compare commits

...
Author SHA1 Message Date
ChuckandClaude Opus 5.5 b210fb6083 feat(fetch): count bytes on the wire as well as decoded
fetch-stats reported only `bytes`, len(response.content), and that read as
the download volume. ESPN gzips every scoreboard, so it overstated real
traffic about 14x: a college football Saturday is 865 KB decoded, 63 KB on
the wire. Every counter set now carries `wire_bytes`, read from urllib3's
count of raw bytes taken off the socket (decoded size when there is no
urllib3 response behind it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 17:19:43 -04:00
ChuckandClaude Opus 5.5 a74b5a2f0f fix(espn): fetch a window's edge months whole, cap chunk requests per process (#751)
- fetch_espn_date_chunks() asks for a window's partial edge month whole when the window covers ESPN_MONTH_COVER_MIN_DAYS (7) or more of its days, trimmed to the window by US Eastern start date. New espn_request_chunks().
- Chunk requests share one process-wide cap of ESPN_CHUNK_WORKERS (6) in flight.
- A new process starts as if a range had just been rejected, so it no longer spends a doomed 400 per window at start.
- Also: _eastern_zone() without try/except/pass (Codacy), and test_on_demand_live_and_restore reads the last on-demand state write rather than the last cache write (the font-usage publisher raced it; main CI had failed on it since #748).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 15:46:15 -04:00
ChuckandClaude Opus 5.5 57d7df6705 fix(web): schedule saves, restart_required, health and current-status agree with the rig (#750)
- POST /api/v3/config/schedule and /config/dim-schedule accept a disabled per-day schedule with every day off, and keep an off day's times.
- POST /api/v3/config/main answers restart_required only when the save changed a setting the running display does not apply live.
- GET /api/v3/health reports degraded with checks.display_loop.status stopped when the display service is stopped.
- GET /api/v3/display/current-status answers unknown (null fields) after the display stops instead of the cached last state. New web_interface.display_state.display_gone().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 15:07:19 -04:00
ChuckandClaude Opus 5.5 e09e251553 fix(display): a frame the preview throttle skipped still reaches the snapshot (#752)
A screen that draws its card once and holds it no longer leaves the web preview black: DisplayManager remembers a changed frame the snapshot throttle skipped, and the render loop writes it (write_owed_snapshot(), called from _display_once) once the interval has passed. A failed owed write stays owed and is retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 14:41:47 -04:00
ChuckandClaude Opus 5.5 7026eeb156 fix(on-demand): show a named live mode; end a session that cannot resume (#748)
On-demand: a mode requested by name is shown first (even a quiet live mode); a session that can't resume after a restart, or whose plugin system failed to start, ends with status restore-failed instead of staying dead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 13:25:12 -04:00
ChuckandClaude Opus 5.5 2236ff3081 fix(web-ui): MQTT password without TLS, Overview poll that never stopped, brightness slider error, token form left dirty (#745)
* fix(web-ui): let the MQTT bridge form save a password without TLS

PUT /api/v3/integrations/mqtt-bridge/config refuses a stored password
while mqtt_tls is off unless allow_insecure_mqtt is set (the CWE-319
guard in api_v3/misc.py). The Tools tab form neither rendered a control
for that flag nor sent it, so a password-protected broker on a LAN
without TLS could never be saved from the UI, and once such a password
was in bridge_config.json every later save from the form was refused.

The form now shows "Allow without TLS (trusted network)" while "Use
TLS" is unchecked, prefilled from the GET's config.allow_insecure_mqtt,
and mqttBody() sends its state as allow_insecure_mqtt. The box is off
until the user ticks it, so the server's guard still refuses a
cleartext password by default.

Tests: the Tools DOM suite checks the control, its show/hide with the
TLS box, the prefill and the value saved; a Flask test pins that the
GET reports the opt-in (false until saved on).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web-ui): stop the Overview reconciliation poll from running forever

The reconciliation banner script in partials/overview.html re-asked
/api/v3/plugins/reconciliation-status every 2 s until the answer said
done, with no limit. The route answers done: false whenever
ledmatrix_reconciliation.json is missing or unreadable, which happens
when _run_startup_reconciliation raises before writing it or when /tmp
is cleaned under a long-running web service (reconciliation runs once
per process). The browser then sent that request every 2 s for as long
as the page stayed open, on every tab, since the poll was never tied to
the Overview being visible.

The poll now gives up after 30 tries (a minute) and runs only while the
Overview is the active, visible tab, registered with LEDVisibility under
its own key like the other partials' pollers. Dismissing the banner
ends it too.

Test: test/js/unit/test_overview_reconciliation_poll.js runs the shipped
script in a vm with fake timers and fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web-ui): drop the Display tab's lookup of a removed brightness label

The brightness slider's input handler in partials/display.html set the
text of both #brightness-value and #brightness-display. #387
(978a03b42) removed the "LED brightness: N%" line that carried
#brightness-display, so getElementById returned null and every step of
the slider threw "Cannot set properties of null" into the console. The
visible label still updated, because it is written first.

The dead lookup is removed.

Test: test/js/unit/test_display_partial_ids.js checks every literal
getElementById() in the partial's inline scripts against the ids its
markup renders, and runs the shipped script in a vm to move the slider.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web-ui): a created API token leaves the General tab's form clean

app.js marks a form data-dirty on any input inside it and removes the
mark only after a successful htmx request; its beforeunload handler
asks "Leave site?" while a visible form is still dirty. The API token
form in partials/general.html posts through window.webLogin.createToken
with fetch, so the mark survived the token being created and a reload
of the page with the General tab open prompted about a change that had
already been saved.

createToken now removes data-dirty after a successful create, next to
the form.reset() it already did. A refused request keeps the mark.

Test: test/js/unit/test_general_web_login_token.js runs the shipped
script in a vm with a fake fetch and DOM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(js): match <script> tags the way CodeQL's tag-filter rule expects

The three new suites pull the inline scripts out of their partials with
/<script>([\s\S]*?)<\/script>/g. CodeQL flags that shape as a bad HTML
filtering regexp (js/bad-tag-filter: misses upper case and tags with
attributes or whitespace), four high alerts that blocked the PR. These are
our own templates read by tests, not user input, but the stricter pattern
costs nothing: /<script\b[^>]*>(...)<\/script[^>]*>/gi, as
test_html_escaping.js already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(js): slice the Display partial's markup around its scripts

CodeQL read the script-stripping replace() as an incomplete HTML sanitizer
(js/incomplete-multi-character-sanitization). The test only reads our own
template, but slicing between the matched blocks gives the same markup
without the pattern.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 22:30:51 -04:00
ChuckandClaude Opus 5.5 5ad5e9aa59 fix(web): plugin action params, refused on-demand starts, pending-operation 500, double-click 409, binary static files (#744)
* fix(web): pass plugin action params to the wrapper on stdin

POST /api/v3/plugins/action runs a plugin's script through a generated
Python wrapper, and the params went into that wrapper's source as
`params = <json.dumps(params)>`. JSON true, false and null are undefined
names in Python, so any params holding one made the wrapper die with a
NameError before the script ran, and the route answered "Action failed".
The plugin file manager's category toggle sends {"category_name": ...,
"enabled": true}, so of-the-day's category toggle failed every time.

The wrapper now reads the params from its own stdin (json.loads) and the
route passes them there; nothing taken from the request is written into
the generated source any more. The script's side is unchanged: the same
json.dumps(params) on its stdin, LEDMATRIX_ROOT set, stdout parsed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a refused on-demand start leaves no request in the mailbox

POST /api/v3/display/on-demand/start delivered the request (control
socket, else the file mailbox) before it checked the display service.
With the service stopped the socket is absent, so the request went to the
mailbox; the route then answered 400 "Display service is not running"
when start_service was off, or 500 "Failed to start display service" when
the start failed. The display reads that mailbox with max_age=3600 and
never checks a request's timestamp, so the next time it was started it
ran the refused request, pinned if asked.

The service is now checked before anything is delivered, and nothing is
posted when start_service is off and the service is down. When the start
itself fails, the request is withdrawn from the mailbox, but only while
the mailbox still holds this request_id (the compare-before-delete the
display's _consume_on_demand_request uses), so a newer request posted in
the meantime is left for the display.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a pending plugin operation's status no longer answers 500

PluginOperationQueue.enqueue_operation stores the operation's callback in
operation.parameters['_callback'], and the worker pops it only when it
runs the operation. PluginOperation.to_dict() returned parameters as they
were, so GET /api/v3/plugins/operation/<id> for an operation still
waiting in the queue (an install queued behind another plugin's) handed
jsonify a function and answered 500 "A system error occurred" on every
poll until the worker reached it.

to_dict() now leaves out parameters whose name starts with "_". The
operation itself keeps its callback for the worker; every other field of
the answer, and the operation-history records (a different class), are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a second install or uninstall of a busy plugin is a 409

PluginOperationQueue.enqueue_operation raises ValueError when the plugin
already has an operation waiting or running. /plugins/install did not
catch it, so a double-clicked Install (the button is never disabled)
answered 500 "An error occurred; see logs for details" from the
blueprint's catch-all while the first install carried on.
/plugins/uninstall caught it in its own catch-all: a 500 "Failed to
uninstall plugin", plus an "uninstall failed" operation-history record
for an uninstall that never started.

Both routes now enqueue through _enqueue_or_conflict, which turns the
queue's refusal into a 409 PLUGIN_OPERATION_CONFLICT naming the plugin,
and records nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): serve binary plugin static files instead of a 500

GET /api/v3/plugins/<plugin_id>/static/<path> read every file with
open(..., 'r', encoding='utf-8') and returned the decoded text, so any
binary file -- a plugin icon or preview image, which is what the REST API
reference says the route is for -- raised UnicodeDecodeError and answered
500.

The file is now sent with send_file, as bytes. HTML, JavaScript, CSS and
JSON keep the content types the route always set, and other text keeps
text/plain; anything else gets the type mimetypes knows it by (image/png
for a .png). The plugin id and path validation and the resolve_under
containment check are untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a socket-acknowledged on-demand start is a success

cdaeb385 checked the systemd unit before delivering the on-demand
request, so a display run by hand or in the emulator (no active unit)
with start_service off now got nothing, where before the request went
over the control socket and took effect behind a 400. A socket
acknowledgement is the display itself saying it is running and has the
request queued, so it is the better witness than systemd.

The request is delivered first again. When the display acknowledged it
over the socket, the route answers success without consulting systemd for
the "not running" 400 and without starting the unit (with start_service
on it tried to start a second display beside the one that answered); the
service is still reported the way _ensure_display_service_running reports
a running one. When it went to the mailbox, the 400 (service down,
start_service off) and the failed-start 500 both withdraw this request_id
from the mailbox, leaving a newer request alone, so neither refusal runs
later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 22:30:39 -04:00
ChuckandClaude Opus 5.5 8a0cce1aaf fix(web): mask the Config Editor's secrets; keep disabled plugins' rotation slot and Vegas exclusion; restore only missing plugins (#743)
* fix(web): mask the Config Editor's secrets like GET /config/secrets

The Config Editor tab (/partials/raw-json) filled its config_secrets.json
editor with the file as it is on disk. GET /api/v3/config/secrets masks every
value because the interface is reachable without a login by default, but
this page handed the same credentials (GitHub token, Home Assistant token,
plugin API keys) to anyone who loaded it. The masked-save path in
save_raw_secrets_config was written for a masked editor and never got one.

_load_raw_json_partial now masks the section with mask_all_secret_values
after strip_auth_section, exactly as the GET does. Saving it back is safe:
save_raw_secrets_config drops the masks (strip_masked_values) and merges the
rest onto the stored file (deep_merge), so an untouched secret stays as it
is and a replaced mask is the only value that changes.

The config.json editor is left as it is. Its save (save_raw_main_config)
writes the posted object verbatim, with no mask stripping or merge, so a
masked main editor would write the bullets over any credential it holds.
Masking it needs a merge-on-save of its own first.

Tests: TestConfigEditorRoundTrip renders the partial over a real
ConfigManager, checks no real value is in the editor, and posts the editor
back unchanged (the file is identical) and with one mask replaced (only that
value changes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): keep disabled plugins in the saved rotation order and Vegas exclusions

PluginOrderList draws one row per enabled plugin and, once drawn, rewrites
its hidden inputs (plugin_rotation_order, vegas_plugin_order,
vegas_excluded_plugins) from those rows. A disabled plugin has no row, so
merely opening the Display or Rotation & Durations tab took it out of the
inputs, and the next save of that form stored the lists without it. Exclude
Clock from Vegas, disable it, change the brightness, re-enable it: Clock was
scrolling in Vegas again and had moved to the end of the rotation.

syncInputs now keeps the saved ids that have no row. In the order, each one
keeps its saved slot and the rows fill the other slots in their current
order, with rows not in the saved order last, as before. In the exclusions
they follow the unchecked rows. Only string ids are carried over, once each:
/config/main refuses a list holding anything else, which would block every
later save of the tab.

Tests: test/js/unit/test_plugin_order_list.js runs the shipped widget in a vm
with a fake DOM (draw, reorder, include/exclude, the rotation list, junk ids)
and is in run_all.js and the README. The durations DOM suite now reads only
its own rows' ids from the input, since a rig's saved order can hold others.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a restore reinstalls only the plugins that are missing

POST /backup/restore with reinstall_plugins (the "Reinstall missing plugins"
box) passed every plugin in the backup's plugins.json to
install_plugin(). That replaces an installed copy with a fresh download, so
a restore onto the same device re-downloaded every plugin inside the
request. A plugin installed from its own URL is not in the registry, so its
install returned False, plugins_failed set success to False, and the restore
answered 500 "Restore incomplete ... plugins not reinstalled: <id>" (shown
as "Restore failed") with the plugin still installed and the config
restored.

Each plugin is now looked up first with the store's _existing_install, the
same lookup install_plugin makes to decide a copy exists: the id, or an id
the registry proves is the same plugin (aliases, the plugin_path name), and
never a bare ledmatrix-<id> folder (#686). One that is installed is recorded
in result.skipped as "plugin:<id> (installed)", which the page lists under
Skipped; a missing one is installed as before. The list_installed_plugins
docstring said every listed plugin is reinstalled and now says otherwise.

Tests: TestInstalledPluginsAreNotReinstalled, with a mocked store (installed
skipped, missing installed; an installed plugin the store can't install is
not a failure) and with a real PluginStoreManager (a registry alias and a
third-party install are skipped, a missing plugin installed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): /config/main answers malformed JSON with a 400

save_main_config read a JSON body with request.get_json(), which raises
Werkzeug's BadRequest for a body that does not parse (or an empty one sent as
application/json). That happened inside the handler's try, so the
catch-all answered 500 CONFIG_SAVE_FAILED with "Check file permissions on
config directory" among its suggested fixes and logged a traceback at
ERROR, for what was the caller's mistake.

It now reads with get_json(silent=True), as save_raw_main_config does, and
answers a sent-but-unparseable body with the same 400
{"status": "error", "message": "Invalid JSON in request body"}. An empty
JSON body falls through to the existing 400 "No data provided". The change
is limited to the lines that read the body.

Tests: TestMalformedBody in test_api_v3_partial_main_save.py (the 400 and its
shape, identical to /config/raw/main's, and nothing saved; the empty body).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a restore that brings back fonts clears the font catalog cache

GET /api/v3/fonts/catalog caches its answer as fonts_catalog for five
minutes. Font upload and delete clear that entry (fonts.py), but
POST /backup/restore copies user fonts into assets/fonts without touching
it, so restored fonts were missing from the Fonts tab and every font picker
until the cache expired.

backup_restore now clears fonts_catalog when the result lists restored fonts
(restore_backup records them as "fonts (<count>)"). A restore that restored
no fonts leaves the cache alone.

Tests: TestFontsCatalogCache in test_api_v3_backup_restore.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): drop uninstalled plugins from the carried-over order and exclusions

2b34f254 made the plugin order list keep every saved id that has no row,
so a disabled plugin keeps its rotation slot and Vegas exclusion. That
also kept the ids of plugins that have since been uninstalled: they stayed
in plugin_rotation_order and vegas_excluded_plugins for good, where before
the next save of the tab dropped them.

The widget already fetches /api/v3/plugins/installed, every installed plugin
with its enabled flag, and draws only the enabled ones. It now keeps that
response's full id set and carries over only saved ids that are installed
but have no row (disabled). An id outside the set is dropped, as before.
With no list, nothing is dropped: a failed request draws no rows and leaves
the inputs as saved, and the carry-over keeps everything if the set was
never filled.

Tests: test/js/unit/test_plugin_order_list.js adds a disabled plugin kept
while an uninstalled one is dropped (order and exclusions; fails on
2b34f254), and a failed plugin list leaving both inputs as saved. The
CHANGELOG bullet and the README row say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(js): register the order-list suite apart from other branches' suites

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 22:30:28 -04:00
ChuckandClaude Opus 5.5 0b039c875f fix(web): plugin settings endpoints - a refused save no longer leaks into config.json; GET masks secrets (#742)
* fix(config): load_config hands each caller a private copy

ConfigManager.load_config() returned its cached self.config itself (the
mtime fast path from #410 kept the full path's aliasing). Web handlers
edit what they load and then validate: the plugin form save applies the
posted fields to the loaded section (a shallow .copy(), so nested dicts
were the cache's own), and save_main_config sets its checkboxes before
it checks auto_update_channel. When the save was refused, the edit
stayed in the cache the fast path serves, and the next save of any
other setting wrote it to config.json: the refused value, and a nested
secret typed into the same form (mqtt.password, league.espn_s2,
flightaware.api_key) in plain text, since it never reached
config_secrets.json to be stripped. The form also reloaded showing the
refused values.

load_config() now returns a private copy on both paths, and
save_config/save_config_atomic keep a copy of what they were given, so
nothing a caller edits reaches the cache unless it is saved. Fixing it
here rather than in each handler covers every route that edits before it
validates. No caller relies on editing the cache without saving: every
src/ and web_interface/ caller either reads, or saves the dict it
edited. get_config() still returns the live dict for the display
process's readers.

The copy is a pickle round trip: on a Pi 4 with its real 64 KiB config,
2.0 ms against 6.9 ms for copy.deepcopy (json round trip 3.4 ms). Two
tests asserted the aliasing itself and now assert a copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): GET /plugins/config masks secrets and refuses core sections

The route returned the plugin's section as load_config() has it, with
config_secrets.json merged in: API keys and tokens went out in plain
text. #276 masked them here; #330's rewrite of the route dropped it,
while the settings page and GET /config/secrets kept masking. It also
took any plugin_id, so ?plugin_id=web_auth returned the login's
cookie-signing key and password hash, and ?plugin_id=github the Plugin
Store token, which GET /config/main strips and redacts.

The route now refuses what _non_plugin_id_error refuses for reset and
uninstall (core sections, malformed ids) with a 400, and blanks x-secret
fields with mask_secret_fields after the defaults merge, as the page
does. A plugin with no schema has its credential-named fields blanked by
_redact_credentials, as GET /config/main does. Blank rather than the
bullets of GET /config/secrets: the save drops a blank secret as
"unchanged" (remove_empty_secrets) but would store the bullets, so the
response must post back as it came. Tested: GET, then POST the response
unchanged, keeps every stored secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): parse a table row's cells against the list's item schema

An array of objects drawn as a table posts each cell as
"cities.0.timezone". _get_schema_property stopped at "cities" (an array,
not an object with properties), so _parse_form_value_with_schema got no
schema for the cell and guessed: a blank optional text cell became None
and a text cell holding digits became an int. Validation refused both,
so every save of the page failed for as long as such a row existed --
geochron's city without a timezone, a countdown named "2027". A secret
cell is always drawn blank, so a plugin with secrets in its rows could
not be saved from the form at all.

The lookup now steps from an index segment into the array's items: to
the item schema itself for "color.2", into its properties for a row
cell. Number, boolean and required cells convert as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a blank secret field saves as "unchanged", required or not

The settings page draws a stored secret blank (mask_secret_fields) and
posts the blank back. _parse_form_value_with_schema turned a blank
optional string into "" -- which the save drops as unchanged
(remove_empty_secrets) -- but a blank required one into None. For a
secret that is required with no default (youtube-stats' api_key) that
None failed validation, so every save of the page was refused until the
key was typed in again.

A blank text secret (x-secret, type string) now parses to "", whatever
its required list says; a list or object secret keeps getting [] or {},
which the save drops the same way. Not _SKIP_FIELD: skipping keeps the
value load_config() merged in, and the save would then write it back to
config_secrets.json -- after a secret change the cached section can
still hold the old one, so that write reverted it. A test covers that
sequence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): POST /plugins/config refuses core sections and malformed ids

Reset and uninstall check the plugin id with _non_plugin_id_error; the
save did not. {"plugin_id": "display", "config": {...}} found no schema,
so nothing was validated or filtered, and the body was merged into the
core display section along with "enabled": true -- rows: "banana"
included. A plugin_id that was not a string (a list, an object, a number)
reached config.get() or the schema lookup, raised TypeError, and came
back as a 500.

Both the JSON and the form path now call _non_plugin_id_error first and
answer its 400.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a text field keeps "true", "[1, 2]" and "{}" as typed

_parse_form_value_with_schema guessed before it consulted the schema:
"true"/"false" became booleans, and a value starting with "[" or "{"
that parsed as JSON became a list or object, whatever the field's type.
A text setting holding "true", "False", "[1, 2]" or "{}" was then
refused by validation ("Expected type string, got bool"), and the save
with it.

A field whose schema type is string, or string-or-null, now returns the
posted text as it came. Every other type goes through the conversions as
before; numbers in text fields were already left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(config): copy the cached config without pickle

_private_copy was a pickle round trip. It only ever unpickled bytes it had
just made from our own dict, so nothing untrusted reached it, but it put
pickle in the config path and Codacy failed the PR for it (B301/B403).
The config is JSON data, so copying its dicts and lists is a full copy;
every other value is immutable. Measured on ledpi (Pi 4) with its real
60 KiB config: 2.11 ms, against 1.92 ms for pickle and 6.75 ms for
copy.deepcopy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): describe the config copy without pickle

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 22:30:16 -04:00
59 changed files with 3811 additions and 145 deletions
+249
View File
@@ -19,6 +19,56 @@ accepts both, but the store flags the old spelling as deprecated
## Unreleased
### Fixed
- The web preview and `/api/v3/display/current` no longer stay black for a
whole screen that draws its card once and then holds it. The snapshot is
written from `update_display()` at most once per write interval, so a frame
pushed inside that interval was skipped and left for the next
`update_display()` -- which such a screen never makes. Soccer's
recent/upcoming cards skip redundant redraws, and the first one after an
on-demand start lands a few milliseconds after the start's clear wrote a
black frame: on ledpi the preview showed 0 lit pixels for the whole 15 s
while the panel showed the card. `DisplayManager` now remembers a skipped
changed frame, and the render loop writes it (`write_owed_snapshot()`)
once the interval has passed. The cadence is unchanged, and nothing extra
runs when no frame is owed.
### ESPN date-range fetches: fewer requests, fewer at once
A soccer board (8 leagues, ESPN rejecting `dates=` ranges) logged ~90
`NameResolutionError` lines and an `update() timed out` at every start on a
Pi: each league's fortnight-either-side window was 29 day requests, fetched
by several managers at once, ~40 in flight. Measured against live ESPN with
soccer-scoreboard 2.39.2, alternating runs: **~450 requests per start, peak
~45 in flight, ~75 DNS lookups -> 46 requests, peak 13, ~30 lookups**.
- `fetch_espn_date_chunks()` asks for a window's partial edge month whole
when the window covers `ESPN_MONTH_COVER_MIN_DAYS` (7) or more of its days,
and trims the answer to the window's days by each event's US Eastern start
date -- the day ESPN's `dates=YYYYMMDD` means (417 of 417 live soccer
events matched). A 29-day window spanning two months is 2 requests instead
of 29. Short windows (a live poll's 1-2 days) stay day by day. A trimmed
month that comes back at the 500-event cap re-asks only the window's days.
An event with no readable date is kept. New: `espn_request_chunks()`.
- Chunk requests share one process-wide cap of `ESPN_CHUNK_WORKERS` (6) in
flight, across every window being fetched, instead of six per window.
- A new process starts as if a range had just been rejected, so it no longer
spends one doomed 400 per window at every start (eleven at once from a
soccer board); the range is still retried `RANGE_RETRY_SECONDS` in.
### Fetch stats: bytes on the wire, not just decoded
`GET /api/v3/plugins/fetch-stats` reported only `bytes`, the decoded body
size, and that read as the download volume. ESPN gzips every scoreboard, so
it overstated what crossed the network about 14x: a college football
Saturday's scoreboard is 865 KB decoded and 63 KB on the wire, and ledpi's
"643 MB in 6 hours" of football was ~47 MB of actual traffic. Every counter
set (totals, per plugin, per host) now has `wire_bytes` too, read from
urllib3's count of the raw bytes it took off the socket. A response with no
urllib3 response behind it is counted at its decoded size. `bytes` keeps its
meaning.
### Cheap per-frame and per-fetch savings
- `BaseOddsManager.get_odds()` no longer pretty-prints every odds response
@@ -505,6 +555,54 @@ policies are unchanged.
stored `ttl` was stretched the same way. A memory hit is now also checked against
the record's own timestamp, and a stale one falls through to disk, which
returns a newer write if there is one.
- An on-demand request that names a `*_live` mode now shows that mode. On
ledpi, `{"plugin_id": "football-scoreboard", "mode": "ncaa_fb_live"}` with
15 college games on answered 200 and showed `nfl_recent`. The session's
mode list kept a live mode only when the plugin's `has_live_content()`
said so. That method answers the live-priority question, and the sports
plugins answer it for favourite teams only. A mode the request names
(not one resolved from a bare plugin id) now leads the session, with the
plugin's other modes after it. If it has nothing to draw, the session
moves on to the next of those modes, like any empty on-demand mode. The
name is saved with the session (`named_mode` in
`display_on_demand_config`), so a restart resumes on it.
- A restart during an on-demand session whose plugin then fails to load no
longer leaves a session with no modes. On ledpi, `clock-simple` failed
config validation after a crash. The display logged `No valid display
modes found for on-demand plugin 'clock-simple' after restoration` and
kept reporting the session as active until its first pass ended it as
`idle`. The cached request stayed behind for the next restart. The session
now ends at startup with status `error` and error `restore-failed`, which
`/display/on-demand/status` reports, and the cached request is dropped. The
same applies when the plugin system itself fails to start.
- `POST /api/v3/config/schedule` and `/config/dim-schedule` accept a
disabled per-day schedule with every day off. That is the shape
`config.template.json` ships, so posting back what GET returned on a fresh
install answered 400 "At least one day must be enabled". An enabled per-day
schedule still needs a day on. A day that is off now keeps the times it
was posted with (the schedule picker sends them). Before, saving dropped
them, so turning the day back on showed the defaults.
- `POST /api/v3/config/main` answers `restart_required: true` only when the
save changed a setting the running display does not apply by itself.
Brightness (`brightness.set` and the config watcher), the per-mode
durations and plugin sections are applied live. A brightness-only save,
such as the MQTT bridge's slider, or a save that changed nothing, no longer
shows the restart banner. Hardware, rotation order, timezone and every
other setting still ask for the restart.
- `GET /api/v3/health` reports `degraded` when the display service is
stopped. Before, only the sub-checks changed, and the overall status stayed
`healthy` for as long as the last preview frame was under 60 s old.
`checks.display_loop.status` is now `stopped` when three things agree:
systemd says the service is not active, the control socket does not
answer, and there is no live heartbeat. Where the platform has no socket
(Windows) or it is switched off, nothing changes.
- `GET /api/v3/display/current-status` no longer reports the stopped
display's last state (`is_display_active: true`) from the cache for up to
120 s. When the control socket does not answer and the render loop's
heartbeat is absent, stale, or from a process that is gone (#726's rules),
the answer is unknown, with every field `null`. A display that still beats
without a socket, Windows and a socket switched off read the cache as
before. New `web_interface.display_state.display_gone()`.
- The garbage-collection timer (`GcMonitor`, above) no longer prints
`Exception ignored while calling GC callback ... 'NoneType' object has no
attribute 'perf_counter'` when the display service or a test run exits.
@@ -618,6 +716,57 @@ policies are unchanged.
the plugin leaves rotation until the cooldown ends, the same as a raising
`update()`. The display still moves straight on to the next mode. A hung
`display()` is still recorded once, as a hang.
- A plugin settings save that failed validation no longer leaks into the next
save. `ConfigManager.load_config()` returned its cached config itself (the
fast path from #410), so the form save's edits went into the cache before
validation ran, and a refused save left them there. The next save of any
other setting (another plugin's, a plugin toggle, the schedule) wrote them
to config.json: the refused value, and a nested secret typed into the same
form (`mqtt.password`, `league.espn_s2`, `flightaware.api_key`) in plain
text, because it had never reached config_secrets.json to be stripped.
The form also reloaded showing the refused values. `load_config()` now
returns a private copy, and the saves keep one, so nothing a caller edits
reaches the cache unless it is saved. The copy duplicates only the dicts
and lists (every other JSON value is immutable): 2.1 ms for a real 60 KiB
config on a Pi 4, against 6.8 ms for `copy.deepcopy`.
- `GET /api/v3/plugins/config` no longer returns secrets. It sent back the
plugin's section with config_secrets.json merged in, API keys and tokens
in plain text: the masking #276 added was dropped in #330. It also took
any id, so `?plugin_id=web_auth` returned the login's cookie-signing key
and password hash and `?plugin_id=github` the Plugin Store token. Secret
fields now come back blank, as the settings page renders them, and a
plugin with no schema has its credential-named fields blanked, as
`GET /config/main` does. Blank rather than the `••••••••` of
`GET /config/secrets`, because the save reads a blank secret as
"unchanged", so a client can post the response back without erasing
one. Core sections and malformed ids get a 400, as they already did from
reset and uninstall.
- Plugin settings with a table (a list of rows, such as geochron's cities
or the countdowns) save again when a text cell is blank or holds only
digits. A row posts its cells as `cities.0.timezone`, and the schema
lookup stopped at the list, so each cell was parsed with no schema: a
blank optional text cell became null, and a name like "2027" became a
number. Either failed validation, and every save of the page failed for
as long as the row existed. A plugin with a secret in its rows could not
be saved from the page at all, since the secret cell is drawn blank. The
lookup now steps from the index into the list's item schema.
- A plugin whose API key is required and has no default (youtube-stats)
can be saved from its settings page without typing the key in again. The
page draws a stored secret blank and posts the blank back; for a required
secret the save read that blank as null, failed validation, and refused
every save of the page. A blank secret field now means "unchanged", as it
already did for an optional one.
- `POST /api/v3/plugins/config` refuses a core section or a malformed
plugin id with a 400, as reset and uninstall already did.
`{"plugin_id": "display", ...}` merged unvalidated values into the core
display section (and added `"enabled": true` to it), and an id that was
not a string answered with a 500.
- A plugin text setting saves what was typed when that looks like a
boolean or JSON. The form save tried `true`/`false` and `[...]`/`{...}`
before it looked at the schema, so a text field holding "true", "False",
"[1, 2]" or "{}" was stored as a boolean, list or object, and the save
failed validation. Text fields, nullable ones included, are now taken as
typed; other types convert as before.
- A WiFi notice (such as "Connected to HomeNet" or "AP mode on") now shows
within about a second of being posted. It was only checked between
screens, so a 5 s notice posted during a 20 s screen expired before that
@@ -626,6 +775,42 @@ policies are unchanged.
notice is what shows next, and Vegas resumes after it; before, a rotation
screen showed instead and the notice expired behind it. An active
on-demand session still holds the panel until it ends.
- The Config Editor tab no longer shows API keys and tokens in plain
text. Its `config_secrets.json` editor (`/partials/raw-json`) was filled
with the file as it is on disk, so while the web login is off (the
default) anyone who could reach the port could read every credential,
although `GET /api/v3/config/secrets` masks them. The editor now shows the
same masked values. Saving it unchanged changes nothing, because the save
drops the masks and merges onto the stored file; to change a secret,
replace its mask. A list of secrets still needs every entry's real value
to be changed. The `config.json` editor is unchanged: its save writes the
file as given, so a mask there would be stored.
- A disabled plugin keeps its place in the rotation order and its Vegas
exclusion when the Display or Rotation & Durations tab is saved. The order
lists show enabled plugins only and rewrite their hidden inputs from those
rows as soon as they are drawn, so any save of either tab stored the lists
without the disabled plugin. Once re-enabled, it came back at the end of
the rotation and scrolling in Vegas again. A disabled plugin's saved id
now stays in its saved place (`widgets/plugin-order-list.js`); the id of
a plugin that is no longer installed is still dropped.
- Restoring a backup with "Reinstall missing plugins" installs only the
plugins that are missing. Every plugin the backup listed was sent to the
store's install, which replaces an installed copy with a fresh download,
so a restore onto the same device re-downloaded all of them in one
request. A plugin installed from its own URL is not in the registry, so
its "reinstall" failed and the restore answered "Restore failed" while
the plugin sat there installed. An installed plugin, found by the store's
own lookup (registry aliases included), is now listed under Skipped as
`plugin:<id> (installed)`.
- `POST /api/v3/config/main` answers a JSON body that does not parse with
400 `Invalid JSON in request body`, as `/config/raw/main` does, and an
empty JSON body with 400 `No data provided`. Both were a 500
`CONFIG_SAVE_FAILED` suggesting file permissions and disk space, with a
traceback logged at ERROR: `get_json()` raised inside the handler's
catch-all.
- Fonts restored from a backup show up in the Fonts tab and the font
pickers straight away. The font catalog is cached for five minutes, and
upload and delete cleared it but a restore did not.
- A game that goes live now takes over the panel within about a second.
Live priority was only checked between screens, so a game that went live
during a 30 s screen waited for that screen to end. The frame loops and the
@@ -635,6 +820,45 @@ policies are unchanged.
screen showed first and the game came after it. Each check also asks each
plugin `has_live_content()` once, where a plugin registered under several
modes used to be asked once per mode.
- A plugin action whose params hold `true`, `false` or `null` runs again.
`/api/v3/plugins/action` wrote the params into the source of the wrapper
that runs the plugin's script, and those JSON words are not Python, so the
wrapper stopped with a NameError and the action answered "Action failed".
The plugin file manager's category toggle sends `"enabled": true`, so
turning a category on or off in of-the-day always failed. The params now
reach the wrapper on its stdin; the script still receives them as JSON on
its own stdin, as before.
- An on-demand request that `/api/v3/display/on-demand/start` refuses no
longer runs later. With the display stopped the request goes to the
display's mailbox, and the display reads that mailbox for an hour without
looking at a request's age. So with "Start display service" unticked, the
answer was "Display service is not running", yet the next time the
display was started it ran that plugin, pinned if the request said so.
The same happened after "Failed to start display service". On either
refusal the route now takes its request back out of the mailbox, unless a
newer one has replaced it. A request the display acknowledges over the
control socket is now a success whatever systemd reports: a display run
by hand or in the emulator was told "not running" for a request it had
already taken, and with "Start display service" ticked the route tried to
start the service beside it.
- `/api/v3/plugins/operation/<id>` reports a queued operation as `pending`
instead of answering 500. The queue keeps an operation's callback among
its parameters until it runs, and the status route tried to send that
function as JSON. An install queued behind another plugin's install
failed every status poll until the first one finished. Parameters whose
name starts with `_` are internal and are no longer in the answer.
- A second click on Install while that plugin is still installing, or an
Uninstall during its install, now answers 409 "already has an install,
update or uninstall in progress" instead of 500 "An error occurred". The
first operation carried on either way. The uninstall route also stopped
recording a failed uninstall in the operation history for an uninstall
that never started.
- `/api/v3/plugins/<plugin_id>/static/<path>` serves images and other
binary files. It opened every file as UTF-8 text, so a plugin's icon or
preview image answered 500 `UnicodeDecodeError`. Files are now sent as
they are on disk, an image with its own content type; HTML, JavaScript,
CSS, JSON and other text keep the types they had. The path checks are
unchanged.
- The display schedule turns the panel off at exactly the end time. A window
now runs from its start time up to, but not including, its end time: with
07:00-23:00 the panel is on at 07:00 and off at 23:00. Before, the end
@@ -642,6 +866,31 @@ policies are unchanged.
the panel went off at 23:00 or at 23:01 depending on when in the minute
that check ran. Windows that cross midnight and per-day schedules follow
the same rule, and so does the dim schedule.
- The MQTT bridge settings on the Tools tab can save a broker password with
TLS off. The server refuses that unless `allow_insecure_mqtt` is set, and
the form had no way to set it, so a password-protected broker on a home
network without TLS could not be saved from the web UI, and once such a
password was stored every later save failed too. While "Use TLS" is
unchecked the form now shows "Allow without TLS (trusted network)",
prefilled from the saved settings. It is off until ticked, so the server
still refuses a cleartext password by default.
- The Overview's plugin-config warning check stops polling. It asked
`/api/v3/plugins/reconciliation-status` every 2 s until startup
reconciliation reported done, and the route reports not done whenever its
status file is missing: reconciliation raised before writing it, or /tmp
was cleaned under a long-running web service. The page then sent that
request every 2 s for as long as it stayed open, whichever tab was showing.
It now gives up after a minute and only polls while the Overview is on
screen.
- Moving the Brightness slider on the Display tab no longer throws an error
in the browser console on every step. Its handler also updated a "LED
brightness" line that was removed from the page in #387; the lookup is
gone.
- Creating an API token on the General tab no longer leaves the page asking
"Leave site?" on reload. The unsaved-changes guard marks a form when you
type in it and clears the mark only after an htmx save, and the token form
saves with a plain request, so it stayed marked after the token was
created. It is cleared once the token is saved.
- An on-demand session that ends during scheduled-off hours, by expiring or
being stopped, blanks the panel within about a second. It used to stay on
until the next minute, because the once-a-minute schedule check had
+26 -12
View File
@@ -159,14 +159,16 @@ there an unchecked checkbox — which the browser omits — is saved as
}
```
`restart_required` is always true here: display hardware, rotation,
durations and general settings take effect when the display restarts, and
the web UI shows its restart banner on the flag. (Plugin sections saved
through this route reach the running plugin live, like
`POST /plugins/config`.)
`restart_required` is true when the save changed a setting that takes
effect when the display restarts: display hardware, rotation order,
timezone, general settings and the rest. The web UI shows its restart banner
on the flag. It is false when the save changed only what the running display
applies by itself, or nothing: `brightness`, the per-mode durations
(`duration__<mode>`, `display.display_durations`) and plugin sections, which
reach the running plugin live, like `POST /plugins/config`.
A saved `brightness` is the exception: it reaches the panel without a
restart. The route also sends it to the running display over the control
A saved `brightness` reaches the panel without a restart. The route also
sends it to the running display over the control
socket (`brightness.set`), which puts it on the panel at once, and the
response adds `"brightness_transport": "socket"`. Otherwise it is
`"config"`, with `brightness_socket_error` giving the reason, and the
@@ -246,7 +248,10 @@ Replace the schedule configuration.
```
A day whose `<day>_enabled` key is absent counts as enabled, with default
times `07:00`-`23:00`. At least one day must be enabled.
times `07:00`-`23:00`. An enabled schedule needs at least one day enabled; a
disabled one (`"enabled": false`) may have every day off, as
`config.template.json` ships it. A day that is off keeps the times sent for
it, when they are valid `HH:MM`.
**Response**:
```json
@@ -343,7 +348,11 @@ control socket ([IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)), and `cache`
when it came from the `display_current_state` cache key (no socket: the
display is stopped or older, or this is Windows). A display whose render
loop has not refreshed its state for 120 seconds is reported with every
field `null`, either way.
field `null`, either way. So is a stopped display: when the socket does not
answer and the render loop's heartbeat
(`/run/ledmatrix/display-heartbeat.json`) is absent, stale or from a process
that is gone, the cache's last entry is not used. A display still beating
without a socket, Windows, or a socket switched off reads the cache.
### List Display Modes
@@ -2281,7 +2290,11 @@ display snapshot. `data.status` is `healthy` or `degraded`, with
(with `heartbeat_age_seconds`), `stalled` (no heartbeat for 60s: the panel is
frozen even if the service is active; the status turns `degraded`), or
`not_reported` when the display writes none (not started yet, the dev server,
Windows), which does not affect the status. Its `source` is `socket` when the
Windows), which does not affect the status, or `stopped` (with `source:
"service"`) when the display service is not active, the control socket does
not answer and there is no live heartbeat; the status then turns
`degraded`. A platform with no control socket (Windows) or a socket switched
off never reports `stopped`. Its `source` is `socket` when the
age came from the display's state stream over the control socket (measured
in memory by the display) and `heartbeat_file` when it came from
`/run/ledmatrix/display-heartbeat.json`.
@@ -2344,8 +2357,9 @@ Replace the dim schedule. `dim_brightness` is 0-100 (default 30). In
`per-day` mode the days can be sent either as the `days` object that GET
returns, or as the web form's flat fields (`monday_enabled`,
`monday_start`, `monday_end`, ...). A day that is not sent counts as
enabled with default times `20:00`-`07:00`; at least one day must be
enabled.
enabled with default times `20:00`-`07:00`. As for the schedule above, an
enabled dim schedule needs at least one day enabled and a disabled one may
have every day off.
---
+3 -2
View File
@@ -181,13 +181,14 @@ that the harness patches in today.
- dynamic duration (cycle complete, plugin cap, global cap)
- live priority taking over and handing back; live round-robin
- on-demand start/stop/expiry; pinned on-demand; a session resumed after
a restart
a restart, and one that cannot resume (its plugin did not load); a
request naming a live mode the plugin's live check would drop
- schedule off and dim, with an on-demand override during downtime
- WiFi notice; sync follower
- Vegas, with and without `live_in_ticker`
- Each trace row is `[start, mode, duration, exit_reason, frames,
force_clear]`. The exit reason is the event that decided what came next.
- All 16 tests run in under a second. The goldens were generated from
- All 18 tests run in under a second. The goldens were generated from
main's `run()` before any code moved.
- Vegas uses `FakeVegas`, which implements only the contract the controller
depends on: `run_iteration()` returns True after its duration and False
+3 -2
View File
@@ -213,8 +213,9 @@ def list_installed_plugins(project_root: Path) -> List[Dict[str, Any]]:
The plugins are the ``manifest.json`` files in the configured plugin
directory (see :func:`_plugins_directory`), with the manifest's version;
``enabled`` is config.json's flag by the display's rule (a missing flag
is disabled). A restore reinstalls every listed plugin and takes enabled
state from the restored config.json, so ``enabled`` is informational.
is disabled). A restore installs each listed plugin that is missing and
takes enabled state from the restored config.json, so ``enabled`` is
informational.
``data/plugin_state.json`` is not read: it only ever repeated config's
enabled flags and the manifests' versions, and is retired (nothing
+5 -2
View File
@@ -109,8 +109,11 @@ and the plugin test harness all use it. Most plugins get BDF text through
[`espn_dates.py`](espn_dates.py). ESPN's site API rejects `dates=` ranges
and truncates results when `limit` is above 500. `fetch_espn_scoreboard()`
splits a range into month and day requests ESPN accepts and merges the
results; `espn_date_chunks()`, `fetch_espn_date_chunks()`,
`clamp_espn_limit()` and `merge_scoreboard_payloads()` are the pieces.
results; `espn_date_chunks()`, `espn_request_chunks()`,
`fetch_espn_date_chunks()`, `clamp_espn_limit()` and
`merge_scoreboard_payloads()` are the pieces. A window's partial edge months
are asked whole and trimmed to its days (US Eastern), and chunk requests share
one process-wide cap of `ESPN_CHUNK_WORKERS` in flight.
Every request goes through [`fetch_service`](#fetch_service), the chunks
counted against the plugin that asked. Scoreboard plugins also bundle a copy
for older cores.
+189 -37
View File
@@ -26,10 +26,30 @@ A month can hold more than 500 events (college baseball's March does), and
ESPN answers that with exactly ``limit`` events and no hint that more exist. A
month chunk that comes back full is therefore re-asked day by day.
A window's *partial* edge months are asked for whole, too, once the window
covers ``ESPN_MONTH_COVER_MIN_DAYS`` or more of their days, and the answer is
trimmed back to the window's days. A scoreboard's default fortnight either side
of today (29 days, two partial months) was 29 day requests per league; it is
now 2. Trimming needs ESPN's "game day", which is the event's start in US
Eastern time -- checked against the live API on 2026-10-03: 417 of 417 soccer
events across five leagues and three months (one of them spanning the end of
daylight saving) came back from exactly the day query their Eastern date
names. A short window (a live poll's one or two days) stays day by day, so it
never downloads a whole month to read a day of it.
Chunk requests share one process-wide budget of ``ESPN_CHUNK_WORKERS`` in
flight, however many windows are being fetched at once. Each window used to get
its own six, so a scoreboard starting eight leagues -- each with a recent and
an upcoming manager -- had ~40 requests in flight, every one beyond a session's
pool a new connection and a new DNS lookup. On a Pi whose resolver could not
keep up, that was ~90 ``NameResolutionError`` lines within a minute of every
start.
Once a range has been rejected, later ranges skip straight to chunks for
``RANGE_RETRY_SECONDS`` instead of spending a doomed request first -- live
scoreboards ask every 30 seconds. After that the range is tried again, so the
workaround retires itself if ESPN reverts.
workaround retires itself if ESPN reverts. A process starts inside that
period, as if a range had just been rejected.
ONE CACHE KEY PER SCOREBOARD
----------------------------
@@ -55,7 +75,7 @@ import re
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from datetime import date, datetime, timedelta
from datetime import date, datetime, timedelta, tzinfo
from functools import partial
from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, cast
@@ -100,16 +120,57 @@ RANGE_RETRY_SECONDS = 6 * 60 * 60
# pool_maxsize of 10 so the shared Session never has to discard connections.
ESPN_CHUNK_WORKERS = 6
#: An edge month the window covers at least this many days of is asked for
#: whole and trimmed, instead of one request per day (see module docstring).
#: Below it the days are cheaper than the month: a whole month is two to
#: three times the bytes of the half of it a fortnight window holds.
ESPN_MONTH_COVER_MIN_DAYS = 7
# Every chunk request in the process holds one of these while it is in flight
# -- the cap is per process, not per window (see module docstring).
_chunk_slots = threading.BoundedSemaphore(ESPN_CHUNK_WORKERS)
def _eastern_zone() -> Optional[tzinfo]:
"""US Eastern, the zone ESPN's ``dates=YYYYMMDD`` means, or None when
this Python has no time zone data (no edge month is trimmed then)."""
zone: Optional[tzinfo] = None
try:
from zoneinfo import ZoneInfo
zone = ZoneInfo("America/New_York")
except Exception: # noqa: BLE001 - no zoneinfo module or no tz database
zone = None
if zone is not None:
return zone
try:
import pytz
return cast(tzinfo, pytz.timezone("America/New_York"))
except Exception: # noqa: BLE001
return None
_EASTERN = _eastern_zone()
# What _fetch_one_chunk returns for a month that came back at the cap.
_CAPPED: Any = object()
_range_lock = threading.Lock()
_ranges_rejected_until = 0.0
# A process starts out assuming ranges are still rejected, as they have been
# since 2026-09-15, and tries one again RANGE_RETRY_SECONDS in. Starting
# from "unknown" cost one doomed range request per window at every start --
# eleven 400s at once from a soccer board, each fetching before any had
# answered -- to learn what every start learns.
_ranges_rejected_until = time.monotonic() + RANGE_RETRY_SECONDS
__all__ = [
"ESPN_MAX_LIMIT",
"ESPN_CHUNK_WORKERS",
"ESPN_MONTH_COVER_MIN_DAYS",
"RANGE_RETRY_SECONDS",
"clamp_espn_limit",
"parse_espn_date_range",
"espn_date_chunks",
"espn_request_chunks",
"merge_scoreboard_payloads",
"fetch_espn_date_chunks",
"fetch_espn_scoreboard",
@@ -220,6 +281,79 @@ def espn_date_chunks(start: date, end: date) -> List[str]:
return chunks
def espn_request_chunks(
start: date,
end: date,
month_cover_min_days: Optional[int] = None,
) -> List[Tuple[str, Optional[Tuple[date, date]]]]:
"""The requests that fetch ``[start, end]``, as ``(dates, trim)`` pairs.
:func:`espn_date_chunks`, except that a partial edge month with
``month_cover_min_days`` (default ``ESPN_MONTH_COVER_MIN_DAYS``) or more
of its days in the window becomes one ``YYYYMM`` request whose ``trim``
is the first and last of those days: its events that start outside them
(US Eastern) are dropped. ``trim`` is None for every other request.
Without time zone data nothing can be trimmed, so the edge days stay day
requests.
"""
if month_cover_min_days is None:
month_cover_min_days = ESPN_MONTH_COVER_MIN_DAYS
planned: List[Tuple[str, Optional[Tuple[date, date]]]] = []
run: List[str] = []
def flush() -> None:
if (_EASTERN is not None and month_cover_min_days > 0
and len(run) >= month_cover_min_days):
planned.append((run[0][:6], (_parse_day(run[0]), _parse_day(run[-1]))))
else:
planned.extend((day, None) for day in run)
run.clear()
for chunk in espn_date_chunks(start, end):
if run and (len(chunk) != 8 or chunk[:6] != run[0][:6]):
flush()
if len(chunk) == 8:
run.append(chunk)
else:
planned.append((chunk, None))
flush()
return planned
def _parse_day(text: str) -> date:
return date(int(text[:4]), int(text[4:6]), int(text[6:8]))
def _eastern_day(stamp: Any) -> Optional[date]:
"""The US Eastern date of an ESPN event ``date`` ("2026-10-10T11:30Z"),
or None when it cannot be read."""
if not isinstance(stamp, str) or _EASTERN is None:
return None
try:
moment = datetime.fromisoformat(stamp.strip().replace("Z", "+00:00"))
except ValueError:
return None
if moment.tzinfo is None:
return None
return moment.astimezone(_EASTERN).date()
def _trim_to_days(payload: Any, first: date, last: date) -> Any:
"""Drop the events of a month payload that start outside ``[first, last]``
(US Eastern). An event whose date cannot be read is kept: its day query
might well have returned it, and a game is never dropped on a guess.
"""
if not isinstance(payload, dict) or not isinstance(payload.get("events"), list):
return payload
kept = []
for event in payload["events"]:
day = _eastern_day(event.get("date")) if isinstance(event, dict) else None
if day is None or first <= day <= last:
kept.append(event)
payload["events"] = kept
return payload
def merge_scoreboard_payloads(payloads: List[Any]) -> Dict[str, Any]:
"""Fold chunk responses into one scoreboard payload.
@@ -250,37 +384,54 @@ def merge_scoreboard_payloads(payloads: List[Any]) -> Dict[str, Any]:
def _fetch_one_chunk(
session, url: str, params: Dict[str, Any], headers, timeout, logger, chunk: str,
cache_max_age: Optional[float] = None,
) -> Optional[Dict[str, Any]]:
trims: Optional[Dict[str, Tuple[date, date]]] = None,
) -> Any:
"""GET a single ``dates=`` chunk, or None when it failed.
One bad chunk must not sink the rest of the season, so every error is
logged and swallowed here rather than raised to the gather below.
A month that comes back at the cap is truncated: it returns ``_CAPPED``,
its payload dropped here before it is ever held beside the others. A
month in ``trims`` loses its events outside the days given there.
The request holds one of the process-wide ``_chunk_slots`` while it runs.
"""
try:
response = fetch_get(
session,
url,
params=dict(params, dates=chunk, limit=ESPN_MAX_LIMIT),
headers=headers,
timeout=timeout,
**_memo_kwargs(cache_max_age),
)
response.raise_for_status()
return cast(Optional[Dict[str, Any]], response_json(response))
with _chunk_slots:
response = fetch_get(
session,
url,
params=dict(params, dates=chunk, limit=ESPN_MAX_LIMIT),
headers=headers,
timeout=timeout,
**_memo_kwargs(cache_max_age),
)
response.raise_for_status()
payload = response_json(response)
except Exception as exc: # noqa: BLE001 - see docstring
if logger:
logger.warning("ESPN chunk %s failed, skipping it: %s", chunk, exc)
return None
if len(chunk) == 6 and isinstance(payload, dict):
if len(payload.get("events") or []) >= ESPN_MAX_LIMIT:
return _CAPPED
trim = (trims or {}).get(chunk)
if trim is not None:
payload = _trim_to_days(payload, *trim)
return payload
def _fetch_chunks(
session, url: str, params: Dict[str, Any], headers, timeout, logger,
chunks: List[str], cache_max_age: Optional[float] = None,
) -> List[Optional[Dict[str, Any]]]:
trims: Optional[Dict[str, Tuple[date, date]]] = None,
) -> List[Any]:
"""Fetch every chunk, returning payloads positionally aligned with ``chunks``.
Requests go out ``ESPN_CHUNK_WORKERS`` at a time because a cold season is
over a hundred of them. The order they come back in is not significant --
over a hundred of them -- and no more than that across every window the
process is fetching, which ``_fetch_one_chunk``'s slot enforces. The order they come back in is not significant --
callers keep ``chunks`` order from the returned list -- but it does mean
the session is shared across threads, which is why this only ever issues
GETs and never touches session state.
@@ -293,7 +444,7 @@ def _fetch_chunks(
return []
fetch = partial(
_fetch_one_chunk, session, url, params, headers, timeout, logger,
cache_max_age=cache_max_age,
cache_max_age=cache_max_age, trims=trims,
)
if len(chunks) == 1:
return [fetch(chunks[0])]
@@ -340,7 +491,9 @@ def fetch_espn_date_chunks(
if span is None:
return None
chunks = espn_date_chunks(*span)
planned = espn_request_chunks(*span)
chunks = [chunk for chunk, _ in planned]
trims = {chunk: trim for chunk, trim in planned if trim is not None}
if logger:
logger.debug(
"Fetching ESPN date range %s as %d month/day chunks",
@@ -349,32 +502,31 @@ def fetch_espn_date_chunks(
results = _fetch_chunks(
session, url, params, headers, timeout, logger, chunks, cache_max_age,
trims,
)
attempted = len(chunks)
# A month that came back at the cap is truncated; its days replace it in
# place, so merged events stay in chunk order however the requests raced.
# A month that came back at the cap is truncated; its days (only the
# window's, for a trimmed edge month) replace it in place, so merged
# events stay in chunk order however the requests raced. Its payload was
# already dropped in the worker: a capped college-baseball month is ~2MB
# of parsed JSON, and holding four of them through ~120 day requests added
# ~25MB to the peak -- more than the concurrency itself. Low-memory boards
# (docs/LOW_MEMORY_BOARDS.md) have under 200MB of headroom.
slots: List[Any] = results
capped: Dict[int, List[str]] = {}
for index, chunk in enumerate(chunks):
payload = slots[index]
if payload is None or len(chunk) != 6:
if slots[index] is not _CAPPED:
continue
events = payload.get("events") if isinstance(payload, dict) else None
if len(events or []) >= ESPN_MAX_LIMIT:
if logger:
logger.info(
"ESPN month %s hit the %d-event cap; re-asking it day by day",
chunk, ESPN_MAX_LIMIT,
)
capped[index] = _days_of_month(chunk)
# Drop the truncated month now rather than after its days arrive:
# a capped college-baseball month is ~2MB of parsed JSON, and
# holding four of them through ~120 day requests added ~25MB to
# the peak -- more than the concurrency itself. Low-memory boards
# (docs/LOW_MEMORY_BOARDS.md) have under 200MB of headroom.
slots[index] = None
payload = events = None
if logger:
logger.info(
"ESPN month %s hit the %d-event cap; re-asking it day by day",
chunk, ESPN_MAX_LIMIT,
)
trim = trims.get(chunk)
capped[index] = (_days_of_month(chunk) if trim is None
else espn_date_chunks(*trim))
slots[index] = None
if capped:
days = [day for index in sorted(capped) for day in capped[index]]
+30 -1
View File
@@ -59,7 +59,10 @@ says how old with ``cache_max_age`` (``fetch_get(..., cache_max_age=ttl)``;
Identical means what the validator store keys on: URL, query, effective
headers and, for a session with cookies or auth, the session.
**Counters.** Requests, merged requests, bytes, 304s, errors, HTTP errors,
**Counters.** Requests, merged requests, bytes (``bytes`` decoded, as the
caller reads them; ``wire_bytes`` as they crossed the network, which is
what a metered connection pays for -- ESPN gzips, so the two differ ~14x),
304s, errors, HTTP errors,
adapter retries, throttled requests and seconds waited, plus requests
answered without the network: ``memo_hits`` (the response cache) and
``cache_hits`` / ``legacy_cache_hits`` (a shared ESPN scoreboard cache entry,
@@ -201,6 +204,7 @@ _COUNTER_FIELDS = (
"throttled", # requests that waited for a host budget
"overruns", # requests that went after max_wait_seconds anyway
"bytes", # decoded response body bytes received
"wire_bytes", # body bytes as they came off the socket (still compressed)
"wait_seconds", # time spent waiting for host budgets
"memo_hits", # answered from the response cache (max-age); nothing sent
"cache_hits", # scoreboard fetches answered from a shared ESPN cache entry
@@ -616,6 +620,30 @@ def _body_of(response: Any) -> Optional[bytes]:
return content if isinstance(content, bytes) else None
def _wire_bytes_of(response: Any, body: Optional[bytes]) -> int:
"""How many body bytes came off the socket for ``response``: the
compressed size when the server sent gzip, which ESPN does for every
scoreboard (63 KB on the wire for an 865 KB college football Saturday).
urllib3's ``HTTPResponse.tell()`` counts the raw bytes read before
decoding. A response without one (a test double, an adapter that is not
urllib3) or one whose body was not read is counted at its decoded size,
or as 0, so the counter never claims less than it can prove.
"""
if body is None:
return 0
raw = getattr(response, "raw", None)
tell = getattr(raw, "tell", None)
if callable(tell):
try:
read = tell()
except Exception:
read = None
if isinstance(read, int) and not isinstance(read, bool) and read > 0:
return read
return len(body)
def _retries_of(response: Any) -> int:
raw = getattr(response, "raw", None)
retries = getattr(raw, "retries", None)
@@ -1117,6 +1145,7 @@ class FetchService:
http_errors=int(status is not None and status >= 400),
retries=_retries_of(response),
bytes=len(body) if body is not None else 0,
wire_bytes=_wire_bytes_of(response, body),
throttled=int(waited > 0), overruns=int(overrun),
wait_seconds=waited)
except Exception:
+43 -10
View File
@@ -46,6 +46,35 @@ from src.common.permission_utils import (
get_config_dir_mode
)
def _private_copy(config: Dict[str, Any]) -> Dict[str, Any]:
"""A deep copy of ``config`` that shares nothing with it.
load_config() hands one out per call, and the saves keep one, so the
cached config is never an object a caller holds. A web handler edits what
it loaded, validates, and may refuse the save; when the cache was that
same object, the refused edit stayed in it, and the next save of any
other setting wrote it to config.json -- a nested secret included, in
plain text, since it had never reached config_secrets.json to be
stripped.
The config is JSON data, so only its dicts and lists need copying; every
other value in it is immutable. On a Pi 4 with a real 60 KiB config this
takes 2.1 ms against copy.deepcopy's 6.8 ms, on a path ~30 handlers call
(a pickle round trip is no faster, 1.9 ms, and brings pickle into the
config path for nothing).
"""
return _copy_containers(config)
def _copy_containers(value: Any) -> Any:
if isinstance(value, dict):
return {key: _copy_containers(item) for key, item in value.items()}
if isinstance(value, list):
return [_copy_containers(item) for item in value]
return value
class ConfigManager:
"""
Reads and writes the main application configuration files.
@@ -126,9 +155,10 @@ class ConfigManager:
validate_after_write=validate_after_write
)
# Update in-memory config if save was successful
# Update in-memory config if save was successful. A copy: the caller
# still holds new_config_data (see _private_copy).
if result.status == SaveResultStatus.SUCCESS:
self.config = new_config_data
self.config = _private_copy(new_config_data)
# In-memory config now matches what was just written, so the
# load_config fast path may return it. It still carries the
# merged secrets that were stripped on disk; that matches a full
@@ -208,14 +238,16 @@ class ConfigManager:
Fast path: when config.json, config_secrets.json and the template
are all unchanged since the last successful load (mtime_ns + size),
the already-parsed self.config is returned without touching the
files — same aliasing semantics as the full path, which also
returns self.config.
a copy of the already-parsed self.config is returned without
touching the files.
Either way the caller gets its own copy (see _private_copy): editing
it changes nothing here until it is saved.
"""
try:
current_sig = self._files_signature()
if self.config and self._loaded_sig == current_sig:
return self.config
return _private_copy(self.config)
# Check if config file exists, if not create from template
if not os.path.exists(self.config_path):
@@ -249,8 +281,8 @@ class ConfigManager:
# Signature taken AFTER load + migration (migration may write the
# config back), so it reflects exactly what was read/written.
self._loaded_sig = self._files_signature()
return self.config
return _private_copy(self.config)
except FileNotFoundError as e:
# Only config.json can get here: a missing or unreadable secrets
# file is handled where it is read.
@@ -355,8 +387,9 @@ class ConfigManager:
try:
atomic_write_json(self.config_path, config_to_write)
# Update the in-memory config to the new state (which includes secrets for runtime)
self.config = new_config_data
# Update the in-memory config to the new state (which includes
# secrets for runtime), as a copy -- see _private_copy
self.config = _private_copy(new_config_data)
self._loaded_sig = self._files_signature()
self.logger.info(f"Configuration successfully saved to {os.path.abspath(self.config_path)}")
if secrets_content:
+65 -5
View File
@@ -368,6 +368,10 @@ class DisplayController:
self.on_demand_last_error: Optional[str] = None
self.on_demand_last_event: Optional[str] = None
self.on_demand_schedule_override = False
# The mode the request named, when it named one (not a mode resolved
# from a bare plugin id). Shown even when the plugin's live checks
# would leave it out of the session (_on_demand_modes_for_plugin).
self._on_demand_named_mode: Optional[str] = None
# Plugins that are disabled in config and loaded only because an
# on-demand request named them. The main loop unloads each one once
# on-demand has moved off it (_release_on_demand_plugins).
@@ -561,6 +565,17 @@ class DisplayController:
except Exception: # pylint: disable=broad-except
logger.exception("Plugin system initialization failed")
self.plugin_manager = None
# A restored session has no plugin to resume on. It may have been
# read already (on_demand_active) or not yet, if initialization
# failed before the restore ran; either way, end it visibly.
try:
cached_session = self.cache_manager.get('display_on_demand_config',
max_age=3600)
except Exception: # pylint: disable=broad-except
cached_session = None
if self.on_demand_active or cached_session:
self.cache_manager.clear_cache('display_on_demand_config')
self._set_on_demand_error('restore-failed')
# Its state machine no longer describes what runs; let the last
# snapshot go stale (readers then say unknown) rather than keep
# refreshing it.
@@ -1215,6 +1230,13 @@ class DisplayController:
note = getattr(self.plugin_manager, 'note_display_duration', None)
if note is not None and plugin_id:
note(plugin_id, time.monotonic() - started)
# A screen that drew once and holds makes no more
# update_display() calls, so a frame the preview throttle
# skipped would otherwise never reach the snapshot.
write_owed = getattr(getattr(self, 'display_manager', None),
'write_owed_snapshot', None)
if write_owed is not None:
write_owed()
def _health_tracker(self):
"""The plugin circuit breaker, or None when it is not enabled."""
@@ -1665,6 +1687,7 @@ class DisplayController:
self.on_demand_expires_at = None
self.on_demand_pinned = False
self.on_demand_schedule_override = False
self._on_demand_named_mode = None
# While the session ran, _evaluate_schedule may have forced
# is_display_active on over a scheduled-off answer. Drop the minute
# gate so the next _check_schedule recomputes it; otherwise the panel
@@ -1831,6 +1854,7 @@ class DisplayController:
self.on_demand_pinned = on_demand_config.get('pinned', False)
self.on_demand_requested_at = on_demand_config.get('requested_at')
self.on_demand_expires_at = on_demand_config.get('expires_at')
self._on_demand_named_mode = on_demand_config.get('named_mode')
self.on_demand_status = 'active'
self.on_demand_schedule_override = True
logger.info("On-demand mode detected during initialization: resuming on plugin '%s'; "
@@ -2317,13 +2341,25 @@ class DisplayController:
return modes[0]
return plugin_id
def _on_demand_modes_for_plugin(self, plugin_id: str) -> List[str]:
def _on_demand_modes_for_plugin(self, plugin_id: str,
named_mode: Optional[str] = None) -> List[str]:
"""Every loaded display mode belonging to `plugin_id`, in rotation order.
Live modes that actually have content lead, then the rest, then live
modes with nothing to show -- so an on-demand request for a sports
plugin opens on a game in progress rather than an empty live screen.
Returns an empty list when the plugin has no loaded modes.
`named_mode` is a mode the request asked for by name. It is always
in the list, first when the checks below would have dropped it.
Those checks ask has_live_content(), which is the live-priority
question -- "should this plugin take the panel from the rotation?"
-- and the sports plugins answer it for favourite teams only. Asking
for ncaa_fb_live with fifteen games on and no favourite playing got
a 200 and nfl_recent on the panel. The plugin's display() is what
knows whether the mode has anything to draw; when it has not, the
session moves to the plugin's next mode like any empty on-demand
mode.
"""
plugin_modes = self.plugin_display_modes.get(plugin_id, [])
if not plugin_modes:
@@ -2364,6 +2400,15 @@ class DisplayController:
# Only live modes available but no content - use them anyway
ordered_modes = live_modes
if named_mode and named_mode in available_plugin_modes:
# The named mode leads whether or not the live check kept it: a
# second live mode with content is already in the list, but behind
# the first, so the session would rotate away before reaching it.
if named_mode not in ordered_modes:
logger.info("On-demand: showing %s as requested; plugin '%s' reports no "
"live-priority content for it", named_mode, plugin_id)
ordered_modes = [named_mode] + [m for m in ordered_modes if m != named_mode]
return ordered_modes
def _apply_on_demand_pin(self, ordered_modes: List[str], resolved_mode: Optional[str],
@@ -2392,10 +2437,20 @@ class DisplayController:
plugin_id = self.on_demand_plugin_id
ordered_modes = self._on_demand_modes_for_plugin(plugin_id)
ordered_modes = self._on_demand_modes_for_plugin(plugin_id, self._on_demand_named_mode)
if not ordered_modes:
logger.warning("No valid display modes found for on-demand plugin '%s' after restoration", plugin_id)
self.on_demand_modes = []
# The plugin did not load this time (seen on a rig: its config
# failed validation after the crash that caused the restart), so
# there is nothing to resume. Leaving the session active with no
# modes published it as active for a plugin that was not running
# until the first pass ended it as an ordinary 'idle', and kept
# the cached request for the next restart to trip over. End it
# as a failure the status endpoint reports, and drop the cache.
logger.error("On-demand session for plugin '%s' cannot resume after the "
"restart: the plugin has no loaded display modes (did it "
"fail to load?); ending it", plugin_id)
self.cache_manager.clear_cache('display_on_demand_config')
self._set_on_demand_error('restore-failed')
return
# A restart must not silently un-pin: the pin is part of the request
@@ -2596,7 +2651,10 @@ class DisplayController:
if resolved_mode in self.available_modes:
self.current_mode_index = self.available_modes.index(resolved_mode)
ordered_modes = self._on_demand_modes_for_plugin(resolved_plugin_id)
# Named: the request gave this mode itself, rather than a plugin id
# (or a mode the plugin doesn't have) that resolved to a default.
named_mode = resolved_mode if mode == resolved_mode else None
ordered_modes = self._on_demand_modes_for_plugin(resolved_plugin_id, named_mode)
if not ordered_modes:
logger.error("No valid display modes found for plugin '%s'", resolved_plugin_id)
self._set_on_demand_error("no-modes")
@@ -2613,6 +2671,7 @@ class DisplayController:
self.on_demand_requested_at = now
self.on_demand_expires_at = (now + duration) if duration else None
self.on_demand_pinned = pinned
self._on_demand_named_mode = named_mode
self.on_demand_status = 'active'
self.on_demand_last_error = None
self.on_demand_last_event = 'started'
@@ -2641,6 +2700,7 @@ class DisplayController:
'mode': resolved_mode,
'duration': duration,
'pinned': pinned,
'named_mode': named_mode,
'requested_at': now,
'expires_at': self.on_demand_expires_at
}
+52 -2
View File
@@ -317,6 +317,11 @@ class DisplayManager:
# is handed to the writer; this only once it has been saved, so an
# mtime touch never vouches for a frame still waiting to be written.
self._saved_snapshot_digest: Optional[int] = None
# A changed frame reached _write_snapshot_if_due() inside the write
# interval and was skipped. Nothing writes it unless update_display()
# runs again, and a screen that draws once and holds never calls it
# again -- see write_owed_snapshot().
self._snapshot_owed = False
self._snapshot_dir_prepared = False
# Background writer used mid-scroll; see _write_snapshot_if_due.
self._snapshot_cond = threading.Condition()
@@ -1788,9 +1793,10 @@ class DisplayManager:
if frame_checksum is not None:
digest = frame_checksum
frame_changed = digest != self._last_snapshot_digest
action = snapshot_policy.decide(
now, self._last_snapshot_ts, self._last_snapshot_touch_ts,
viewer_fresh, digest != self._last_snapshot_digest)
viewer_fresh, frame_changed)
else:
# Ask as if the frame had changed before paying to find out.
# decide() is monotone in frame_changed -- a SKIP for a
@@ -1802,22 +1808,36 @@ class DisplayManager:
now, self._last_snapshot_ts, self._last_snapshot_touch_ts,
viewer_fresh, True)
if action is snapshot_policy.SnapshotAction.SKIP:
# Not hashed, so not known to be unchanged: owed until a
# later look finds it written or unchanged.
self._snapshot_owed = True
return
digest = zlib.adler32(self.image.tobytes())
if digest == self._last_snapshot_digest:
frame_changed = digest != self._last_snapshot_digest
if not frame_changed:
# Unchanged after all: the decision an unchanged frame gets.
action = snapshot_policy.decide(
now, self._last_snapshot_ts,
self._last_snapshot_touch_ts, viewer_fresh, False)
if action is snapshot_policy.SnapshotAction.SKIP:
# A changed frame inside the write interval stays owed: the
# next update_display() would write it, but a static screen
# may not make one -- write_owed_snapshot() covers that.
self._snapshot_owed = frame_changed
return
if (action is snapshot_policy.SnapshotAction.TOUCH
and self._saved_snapshot_digest == digest):
# mtime bump only: keeps the health check (snapshot age)
# green without paying for a PNG encode of an unchanged frame
# (this frame is already on disk, so nothing is owed).
self._snapshot_owed = False
os.utime(self._snapshot_path, None)
self._last_snapshot_touch_ts = now
return
# Owed until the write below succeeds: if it raises, the frame
# stays owed and write_owed_snapshot() retries it, rather than a
# held screen leaving the preview stale after one failed write.
self._snapshot_owed = True
# (A TOUCH for a frame that isn't on disk yet -- still queued, or
# its write failed -- is written instead: touching would make the
# older file on disk look current.)
@@ -1842,9 +1862,39 @@ class DisplayManager:
self._last_snapshot_ts = now
self._last_snapshot_touch_ts = now
self._last_snapshot_digest = digest
self._snapshot_owed = False
except Exception as e:
self._log_snapshot_failure(e)
def write_owed_snapshot(self) -> None:
"""Write a frame the snapshot throttle skipped, once it is due.
The preview snapshot is only ever written from update_display(), and
at most once per write interval (snapshot_policy). A frame pushed
inside that interval is skipped, and is written by the next
update_display() that comes after it -- but a screen that draws its
card once and then holds it makes no further call. Its frame was on
the panel and never in the preview: soccer's recent/upcoming cards
skip redundant redraws, and the first one after an on-demand start
(pushed a few milliseconds after the controller's clear) left
/api/v3/display/current and the web preview black for the whole
screen while the panel showed the card.
The render loop calls this after each frame. Cheap when nothing is
owed (one attribute read); otherwise the usual policy decides, so
the write still waits out the interval and an unchanged frame is
never re-encoded.
"""
if not self._snapshot_owed:
return
try:
if self._writes_suppressed():
return
with self._update_lock:
self._write_snapshot_if_due()
except Exception as e: # pylint: disable=broad-except
self._log_snapshot_failure(e)
def _log_snapshot_failure(self, error: Exception) -> None:
# Snapshot failures must never break display — but they must not
# be silent either: the snapshot's mtime is the web UI's display
+9 -2
View File
@@ -48,12 +48,19 @@ class PluginOperation:
completed_at: Optional[datetime] = None
def to_dict(self) -> Dict[str, Any]:
"""Convert operation to dictionary for serialization."""
"""Convert operation to dictionary for serialization.
Parameters whose name starts with ``_`` are internal and left out:
PluginOperationQueue keeps the operation's callback there as
``_callback`` until its worker runs it, and a pending operation's
status answered 500 because that function cannot be serialized.
"""
return {
'operation_id': self.operation_id,
'operation_type': self.operation_type.value,
'plugin_id': self.plugin_id,
'parameters': self.parameters,
'parameters': {key: value for key, value in self.parameters.items()
if not str(key).startswith('_')},
'status': self.status.value,
'progress': self.progress,
'message': self.message,
+7 -2
View File
@@ -724,11 +724,14 @@ class RunLoopHarness:
self.clock.at(t, post)
def restore_on_demand(self, plugin_id: str, mode: Optional[str] = None,
duration: Optional[float] = None, pinned: bool = False):
duration: Optional[float] = None, pinned: bool = False,
named_mode: Optional[str] = None):
"""Start with an on-demand session resumed from the cache, as after
a restart: the state _select_startup_plugins restores, then
_populate_on_demand_modes_from_plugin, as __init__ calls it."""
_populate_on_demand_modes_from_plugin, as __init__ calls it. A
session that cannot resume is logged as ``on-demand-error``."""
dc = self.controller
dc._on_demand_named_mode = named_mode
dc.on_demand_active = True
dc.on_demand_plugin_id = plugin_id
dc.on_demand_mode = mode
@@ -739,6 +742,8 @@ class RunLoopHarness:
dc.on_demand_status = 'active'
dc.on_demand_schedule_override = True
dc._populate_on_demand_modes_from_plugin()
if dc.on_demand_status == 'error':
self.log("on-demand-error", dc.on_demand_last_error)
def wifi_message(self, t: float, message: str, duration: float = 5):
def write():
+29
View File
@@ -0,0 +1,29 @@
{
"screens": [
[0.0, "clock", 5.0, "on-demand-start", 6, false],
[5.0, "sports_live", 15.0, "duration", 15, true],
[20.0, "sports_recent", 15.0, "duration", 15, true],
[35.0, "sports_upcoming", 5.0, "on-demand-requested-stop", 6, true],
[40.0, "clock", 20.0, "duration", 20, true],
[60.0, "sports_live", 15.0, "display-false", 11, true],
[75.0, "sports_recent", 15.0, "duration", 15, true],
[90.0, "sports_upcoming", 10.0, "on-demand-start", 11, true],
[100.0, "sports_live", 0.0, "empty", 1, true],
[100.0, "sports_recent", 15.0, "duration", 15, true],
[115.0, "sports_upcoming", 15.0, "duration", 15, true],
[130.0, "sports_live", 0.0, "empty", 1, true],
[130.0, "sports_recent", 10.0, "on-demand-requested-stop", 11, true],
[140.0, "sports_upcoming", 15.0, "duration", 15, true],
[155.0, "clock", 5.0, "horizon", 5, true]
],
"events": [
[5.0, "request", "start:n1"],
[5.0, "on-demand-start", "sports"],
[40.0, "request", "stop:n2"],
[40.0, "on-demand-requested-stop"],
[100.0, "request", "start:n3"],
[100.0, "on-demand-start", "sports"],
[140.0, "request", "stop:n4"],
[140.0, "on-demand-requested-stop"]
]
}
@@ -0,0 +1,10 @@
{
"screens": [
[0.0, "clock", 20.0, "duration", 20, false],
[20.0, "weather", 20.0, "duration", 20, true],
[40.0, "clock", 20.0, "horizon", 20, true]
],
"events": [
[0.0, "on-demand-error", "restore-failed"]
]
}
+4
View File
@@ -50,6 +50,7 @@ server has none.
| `unit/test_store_categories.js` | no | The store's category filter (sandbox): the template ships only All Categories, the rest come from the store's plugins (one per category whatever its case), choosing one filters to it, and a swapped-in select is refilled from the cache keeping the choice |
| `unit/test_github_url_install.js` | no | Install Single Plugin (sandbox, the button as `plugins.html` ships it): no inline `onclick`, so a click or Enter sends exactly one `install-from-url` request and raises no error |
| `unit/test_render_cards.js` | no | `renderInstalledCards` markup, both empty states, and HTML-escaping of hostile plugin metadata |
| `unit/test_plugin_order_list.js` | no | `widgets/plugin-order-list.js` (the Vegas and rotation order lists): a disabled plugin, which gets no row, keeps its slot in the saved order and its Vegas exclusion when the list rewrites its hidden inputs, around reordering and include/exclude; an uninstalled plugin's id is dropped, a failed plugin list leaves the inputs as saved, and only string ids are carried over, once each |
| `unit/test_style_editor_element_keys.js` | no | `elementKeys()`/`styleRows()`/`positionRows()` from `widgets/style-editor.js`: every `customization.layout` entry gets exactly one row -- paired with its style element through core's `x-layout-key` (so `score` belongs to `score_text`, not a second row), or a position row of its own, leaves included -- since the widget claims the whole `layout` block from the generic fallback renderer |
| `unit/test_style_editor_layout_leaf_columns.js` | no | `columnsFor()` from `widgets/style-editor.js`: a layout-only key whose own value is a leaf (no x/y sub-object, e.g. a `show_logo` toggle) gets a self-keyed column instead of a blank, uneditable row |
| `unit/test_style_editor_layout_leaf_collision.js` | no | `columnsFor()` from `widgets/style-editor.js`: a layout-only leaf key still gets its own column even when its name collides with an unrelated element's style sub-field or another layout axis's sub-field |
@@ -57,6 +58,9 @@ server has none.
| `unit/test_store_registry_fields.js` | no | The store card's registry fields from `plugins_manager.js`: the commit that introduced the listed version (a hex SHA only, linked to that tree), the "Needs LEDMatrix X+" warning, a card from an older registry without either, and `isStorePluginInstalled` answering to `aliases` |
| `unit/test_page_registry.js` | no | The page lifecycle in `js/core/registry.js` (a minimal DOM shim): one `init` per `data-page` root, `destroy` and an aborted `ctx.signal` when htmx swaps it away, a vetoed swap keeps it, lazy page modules, a root removed without htmx swept on the next swap |
| `unit/test_core_modules.js` | no | `js/core/api.js` (JSON envelope, HTTP/`status: error`/network errors, abort passthrough, the #683 login redirect, same-server paths only) and `js/core/facade.js` (`window.LEDMatrix`, deprecated aliases) |
| `unit/test_overview_reconciliation_poll.js` | no | The Overview's reconciliation-banner poll from `partials/overview.html`, run in a vm: it gives up after a bounded number of requests when the status never says done, runs only while the Overview is on screen (`LEDVisibility`, its own key), and stops once the banner is shown |
| `unit/test_display_partial_ids.js` | no | `partials/display.html`: every literal `getElementById()` in its inline scripts names an id the partial renders, and moving the brightness slider (the shipped script, in a vm with a fake DOM) updates its label without throwing |
| `unit/test_general_web_login_token.js` | no | `window.webLogin.createToken` from `partials/general.html`, run in a vm: a created API token clears the form's `data-dirty` mark (so a reload does not ask "Leave site?"), a refused one keeps it |
| `unit/test_plugin_action_delegation.js` | no | The document-level card-action delegation and `handlePluginAction` from `plugins_manager.js`, run with the handler inside an IIFE as in the real file: each action is handled once, a Starlark app uninstall goes to `DELETE /starlark/apps/<id>`, and an uninstall is confirmed once |
| `dom/test_installed_dom.js` | yes | The toolbar in a real DOM: pill/search/sort interaction, the HTMX partial re-swap, and a `getComputedStyle` check that `.filter-pill[data-active]` really matches the emitted markup |
| `dom/test_store_dom.js` | yes | Store pagination, per-page, category, tri-state Installed button, and persistence across a re-boot, against the live registry |
+5 -1
View File
@@ -98,7 +98,11 @@ const ok = (l, c, x) => c ? (pass++, console.log(' ok ' + l))
const lists = () => requests.filter(r => r.url === '/api/v3/plugins/installed').length;
const $ = id => doc.getElementById(id);
const order = () => JSON.parse($('rotation_plugin_order_value').value || '[]');
// The rows' ids, in order. The input also keeps saved ids that have no row
// (a disabled plugin's place, see test/js/unit/test_plugin_order_list.js),
// and the saved order comes from whatever config the server has.
const SHOWN = plugins.filter(p => p.enabled).map(p => p.id);
const order = () => JSON.parse($('rotation_plugin_order_value').value || '[]').filter(id => SHOWN.includes(id));
async function swap() {
panel.dispatchEvent(new window.CustomEvent('htmx:beforeSwap', { bubbles: true, detail: { target: panel, shouldSwap: true } }));
panel.innerHTML = partial;
+42
View File
@@ -98,8 +98,50 @@ const get = p => new Promise((res, rej) =>
window.saveMqttBridge();
await tick(150);
ok('save includes password once typed', sent && sent.mqtt_password === 'typed-secret');
// A password with TLS off is refused unless allow_insecure_mqtt is set
// (CWE-319, api_v3/misc.py). The form has to be able to send it, or a
// plain-LAN broker with a password can never be saved from here.
const allowRow = () => $('mqtt-allow-insecure-row');
const shown = el => !!el && !el.classList.contains('hidden');
ok('allow-without-TLS control rendered', !!$('mqtt-allow-insecure'));
ok('allow-without-TLS starts as saved',
!!$('mqtt-allow-insecure') && $('mqtt-allow-insecure').checked === !!bridge.data.config.allow_insecure_mqtt);
ok('allow-without-TLS shown only while TLS is off',
shown(allowRow()) === !$('mqtt-tls').checked);
$('mqtt-tls').checked = true;
$('mqtt-tls').dispatchEvent(new window.Event('change', { bubbles: true }));
ok('ticking TLS hides it', !shown(allowRow()));
$('mqtt-tls').checked = false;
$('mqtt-tls').dispatchEvent(new window.Event('change', { bubbles: true }));
ok('unticking TLS shows it again', shown(allowRow()));
const setAllow = v => { if ($('mqtt-allow-insecure')) $('mqtt-allow-insecure').checked = v; };
setAllow(false);
window.saveMqttBridge();
await tick(150);
ok('save sends allow_insecure_mqtt false when unticked', !!sent && sent.allow_insecure_mqtt === false, sent);
setAllow(true);
window.saveMqttBridge();
await tick(150);
ok('save sends allow_insecure_mqtt true when ticked', !!sent && sent.allow_insecure_mqtt === true, sent);
onPut = null;
// Prefilled from the saved settings, and hidden while TLS is saved on.
bridgePayload = JSON.parse(JSON.stringify(bridge));
bridgePayload.data.config.allow_insecure_mqtt = true;
bridgePayload.data.config.mqtt_tls = false;
window.loadMqttBridge();
await tick(150);
ok('a saved opt-in is prefilled', !!$('mqtt-allow-insecure') && $('mqtt-allow-insecure').checked === true);
bridgePayload.data.config.mqtt_tls = true;
window.loadMqttBridge();
await tick(150);
ok('hidden on load when TLS is saved on', !shown(allowRow()));
bridgePayload = bridge;
window.loadMqttBridge();
await tick(150);
// ── Pixlet editor, idle ────────────────────────────────────────────────
const appIds = (apps.data.apps || []).map(a => a.id);
ok('editor lists the apps on disk',
+5 -1
View File
@@ -17,6 +17,7 @@ const fs = require('fs');
const BASE = process.env.BASE || 'http://localhost:5000';
const UNIT = ['unit/test_list_filter.js', 'unit/test_render_cards.js',
'unit/test_plugin_order_list.js',
'unit/test_html_escaping.js', 'unit/test_style_editor_element_keys.js',
'unit/test_style_editor_layout_leaf_columns.js',
'unit/test_style_editor_layout_leaf_collision.js',
@@ -28,7 +29,10 @@ const UNIT = ['unit/test_list_filter.js', 'unit/test_render_cards.js',
'unit/test_inline_handler_escaping.js',
'unit/test_plugin_action_delegation.js', 'unit/test_file_upload_widget.js',
'unit/test_store_registry_fields.js', 'unit/test_restart_banner.js',
'unit/test_page_registry.js', 'unit/test_core_modules.js'];
'unit/test_page_registry.js', 'unit/test_core_modules.js',
'unit/test_overview_reconciliation_poll.js',
'unit/test_display_partial_ids.js',
'unit/test_general_web_login_token.js'];
const DOM = ['dom/test_installed_dom.js', 'dom/test_store_dom.js', 'dom/test_no_double_fetch.js',
'dom/test_tools_sections.js', 'dom/test_cache_page.js',
'dom/test_durations_page.js', 'dom/test_operation_history_page.js',
+108
View File
@@ -0,0 +1,108 @@
// The Display tab's inline script must only look up elements the partial
// renders.
//
// Its brightness slider handler also wrote to #brightness-display, a "LED
// brightness: N%" line that #387 removed from partials/display.html. The
// lookup returned null, so every movement of the slider threw a TypeError.
// This checks every literal getElementById() in the partial's inline scripts
// against the ids its markup renders, and runs the shipped script in a vm
// with a fake DOM (null for an id the markup lacks, as in a browser) to move
// the slider.
//
// No jsdom and no server needed.
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const PARTIAL = path.resolve(__dirname, '../../../web_interface/templates/v3/partials/display.html');
let pass = 0, fail = 0;
const ok = (label, cond, extra) => cond
? (pass++, console.log(' ok ' + label))
: (fail++, console.log(' FAIL ' + label + (extra !== undefined ? ' ' + JSON.stringify(extra) : '')));
const html = fs.readFileSync(PARTIAL, 'utf8');
const blocks = [...html.matchAll(/<script\b[^>]*>([\s\S]*?)<\/script[^>]*>/gi)];
const scripts = blocks.map(m => m[1]);
// The markup is what lies between the script blocks (sliced around them, not
// a replace(), which CodeQL reads as an incomplete HTML sanitizer).
let markup = '';
let from = 0;
for (const m of blocks) {
markup += html.slice(from, m.index);
from = m.index + m[0].length;
}
markup += html.slice(from);
const rendered = new Set([...markup.matchAll(/\bid="([^"{}]+)"/g)].map(m => m[1]));
console.log('\n── Display partial: element lookups ──');
// 1. Static: every literal lookup names an id the partial renders.
const lookups = scripts.flatMap(s => [...s.matchAll(/getElementById\('([^']+)'\)/g)].map(m => m[1]));
const missing = [...new Set(lookups.filter(id => !rendered.has(id)))];
ok('the inline scripts look elements up', lookups.length > 0, lookups.length);
ok('every looked-up id is rendered by the partial', missing.length === 0, missing);
// 2. Behaviour: moving the brightness slider updates its label and throws nothing.
function fakeElement(id) {
const listeners = {};
const classes = new Set();
return {
id, value: '', textContent: '', min: '', max: '', checked: false,
style: {}, dataset: {}, className: '',
classList: {
add: c => classes.add(c), remove: c => classes.delete(c),
toggle: (c, on) => (on === undefined ? (classes.has(c) ? classes.delete(c) : classes.add(c)) : (on ? classes.add(c) : classes.delete(c))),
contains: c => classes.has(c),
},
addEventListener: (type, fn) => { (listeners[type] ||= []).push(fn); },
dispatchEvent() { return true; },
appendChild() {},
listeners,
};
}
const main = scripts.find(s => s.includes("getElementById('brightness')"));
ok('found the script that wires the brightness slider', !!main);
if (main) {
const elements = new Map();
const document = {
readyState: 'complete',
hidden: false,
getElementById: id => {
if (!rendered.has(id)) return null;
if (!elements.has(id)) elements.set(id, fakeElement(id));
return elements.get(id);
},
createElement: () => fakeElement(''),
createTextNode: () => ({}),
addEventListener() {},
};
const window = {
LEDEscape: { html: v => String(v), attr: v => String(v) },
LEDVisibility: { onActive() {} },
};
const context = {
window, document, console, URLSearchParams,
fetch: () => new Promise(() => {}),
setTimeout: () => 0, clearTimeout() {}, setInterval: () => 0, clearInterval() {},
};
vm.createContext(context);
let loadError = null;
try { vm.runInContext(main, context); } catch (e) { loadError = e; }
ok('the script loads', !loadError, loadError && String(loadError));
const slider = elements.get('brightness');
const handlers = (slider && slider.listeners.input) || [];
ok('the slider has an input handler', handlers.length > 0);
let thrown = null;
slider.value = '42';
try { handlers.forEach(fn => fn.call(slider, { target: slider })); } catch (e) { thrown = e; }
ok('moving the slider throws nothing', !thrown, thrown && String(thrown));
ok('...and shows the new value', elements.get('brightness-value').textContent === '42',
elements.get('brightness-value').textContent);
}
console.log(`\n${pass} passed, ${fail} failed\n`);
process.exit(fail ? 1 : 0);
@@ -0,0 +1,107 @@
// Creating an API token on the General tab must leave its form clean.
//
// app.js marks a form data-dirty on any input in it and clears the mark only
// after a successful htmx request; its beforeunload handler then asks "Leave
// site?" while any visible form is still dirty. The token form posts with
// fetch (window.webLogin.createToken in partials/general.html), so after a
// token was created the form stayed dirty and reloading the page while the
// General tab was open prompted about changes that had been saved.
//
// Runs the shipped inline script in a vm with a fake fetch and DOM -- no jsdom
// and no server needed.
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const PARTIAL = path.resolve(__dirname, '../../../web_interface/templates/v3/partials/general.html');
let pass = 0, fail = 0;
const ok = (label, cond, extra) => cond
? (pass++, console.log(' ok ' + label))
: (fail++, console.log(' FAIL ' + label + (extra !== undefined ? ' ' + JSON.stringify(extra) : '')));
function webLoginScript() {
const html = fs.readFileSync(PARTIAL, 'utf8');
const scripts = [...html.matchAll(/<script\b[^>]*>([\s\S]*?)<\/script[^>]*>/gi)].map(m => m[1]);
const found = scripts.find(s => s.includes('window.webLogin = {'));
if (!found) throw new Error('webLogin script not found in general.html');
return found;
}
function el() {
const classes = new Set(['hidden']);
return {
textContent: '', dataset: {}, style: {}, className: '',
classList: { add: c => classes.add(c), remove: c => classes.delete(c), contains: c => classes.has(c) },
appendChild() {}, addEventListener() {}, querySelector: () => null,
};
}
function load(answer) {
const elements = {
'web-login-tokens': el(),
'web-login-new-token-value': el(),
'web-login-new-token': el(),
};
const notes = [];
const window = { showNotification: (m, t) => notes.push([m, t]), alert() {}, confirm: () => true };
const context = {
window, console,
document: {
getElementById: id => elements[id] || null,
createElement: () => el(),
querySelectorAll: () => [],
},
fetch: () => Promise.resolve({
ok: answer.ok, status: answer.ok ? 200 : 400,
json: () => Promise.resolve(answer.body),
}),
};
vm.createContext(context);
vm.runInContext(webLoginScript(), context);
return { webLogin: context.window.webLogin, elements, notes };
}
function dirtyForm() {
const attrs = new Map([['data-dirty', '']]);
return {
querySelector: sel => (sel === '[name="name"]' ? { value: 'Home Assistant' } : null),
reset() {},
hasAttribute: name => attrs.has(name),
setAttribute: (name, value) => attrs.set(name, String(value)),
removeAttribute: name => attrs.delete(name),
};
}
const flush = async () => { for (let i = 0; i < 10; i++) await new Promise(r => setImmediate(r)); };
(async () => {
console.log('\n── General tab: API token form ──');
{
const t = load({ ok: true, body: {
status: 'success', message: 'Token created',
data: { token: 'lmx_secret', record: { id: 't1', name: 'Home Assistant', prefix: 'lmx_sec' } },
} });
const form = dirtyForm();
t.webLogin.createToken(form);
await flush();
ok('the new token is shown', t.elements['web-login-new-token-value'].textContent === 'lmx_secret');
ok('a created token leaves the form clean (no "Leave site?" on reload)',
!form.hasAttribute('data-dirty'));
}
{
const t = load({ ok: false, body: { status: 'error', message: 'Name is required' } });
const form = dirtyForm();
t.webLogin.createToken(form);
await flush();
ok('a refused request reports the error', t.notes.some(([m, type]) => type === 'error' && /Name is required/.test(m)),
t.notes);
ok('...and keeps the form dirty: nothing was saved', form.hasAttribute('data-dirty'));
}
console.log(`\n${pass} passed, ${fail} failed\n`);
process.exit(fail ? 1 : 0);
})().catch(e => { console.log('HARNESS ERROR: ' + e.stack); process.exit(1); });
@@ -0,0 +1,137 @@
// The Overview's "Plugin Config Warning" poll must end.
//
// The banner script in partials/overview.html asks
// /api/v3/plugins/reconciliation-status every 2 s until startup reconciliation
// says it is done. The route answers done: false whenever its status file is
// missing -- reconciliation raised before writing it, or /tmp was cleaned
// under a long-running web service -- so the poll used to run every 2 s for
// as long as the page stayed open, on every tab. It now gives up after a
// bounded number of tries and runs only while the Overview is on screen
// (LEDVisibility, like the other partials' pollers).
//
// Runs the shipped inline script in a vm with fake timers, fetch and DOM --
// no jsdom and no server needed.
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const PARTIAL = path.resolve(__dirname, '../../../web_interface/templates/v3/partials/overview.html');
let pass = 0, fail = 0;
const ok = (label, cond, extra) => cond
? (pass++, console.log(' ok ' + label))
: (fail++, console.log(' FAIL ' + label + (extra !== undefined ? ' ' + JSON.stringify(extra) : '')));
function bannerScript() {
const html = fs.readFileSync(PARTIAL, 'utf8');
const scripts = [...html.matchAll(/<script\b[^>]*>([\s\S]*?)<\/script[^>]*>/gi)].map(m => m[1]);
const found = scripts.find(s => s.includes('ledmatrix-recon-dismissed'));
if (!found) throw new Error('reconciliation banner script not found in overview.html');
return found;
}
const flush = async () => { for (let i = 0; i < 10; i++) await new Promise(r => setImmediate(r)); };
function load({ payload, visibility = true }) {
const timers = new Map();
let nextId = 1;
const calls = [];
const banner = { style: { setProperty() {} }, dataset: {} };
const text = { textContent: '' };
const registrations = [];
const window = {};
if (visibility) {
window.LEDVisibility = {
onActive(tab, start, stop, key) { registrations.push({ tab, start, stop, key }); start(); },
};
}
const context = {
window,
document: {
getElementById: id => ({ 'reconciliation-banner': banner, 'reconciliation-banner-text': text })[id] || null,
},
sessionStorage: { getItem: () => null, setItem() {} },
fetch: (url) => {
calls.push(url);
return Promise.resolve({ json: () => Promise.resolve(payload()) });
},
setTimeout: (fn) => { const id = nextId++; timers.set(id, fn); return id; },
clearTimeout: (id) => { timers.delete(id); },
};
vm.createContext(context);
vm.runInContext(bannerScript(), context);
const fireTimers = async () => {
const due = [...timers.entries()];
timers.clear();
due.forEach(([, fn]) => fn());
await flush();
};
return { calls, timers, registrations, banner, text, window, fireTimers };
}
(async () => {
console.log('\n── Overview reconciliation poll ──');
// 1. A status file that never says done: the poll stops on its own.
{
const t = load({ payload: () => ({ status: 'success', data: { done: false, unresolved: [] } }) });
await flush();
for (let i = 0; i < 200; i++) await t.fireTimers();
ok('a status that never turns done stops being polled', t.timers.size === 0,
{ pending: t.timers.size, requests: t.calls.length });
ok('...after a bounded number of requests (at most 30, a minute at 2 s)',
t.calls.length > 1 && t.calls.length <= 30, t.calls.length);
}
// 2. Runs only while the Overview is on screen.
{
const t = load({ payload: () => ({ status: 'success', data: { done: false, unresolved: [] } }) });
await flush();
const reg = t.registrations[0];
ok('registers with LEDVisibility for the overview tab', !!reg && reg.tab === 'overview', reg && reg.tab);
ok('under its own key, so it does not replace another overview poller',
!!reg && !!reg.key && reg.key !== 'overview', reg && reg.key);
ok('first request goes out at once', t.calls.length === 1, t.calls.length);
if (reg) {
reg.stop();
ok('leaving the tab cancels the pending retry', t.timers.size === 0, t.timers.size);
for (let i = 0; i < 5; i++) await t.fireTimers();
ok('no requests while another tab is active', t.calls.length === 1, t.calls.length);
reg.start();
await flush();
ok('coming back asks again at once', t.calls.length === 2, t.calls.length);
ok('...and keeps polling', t.timers.size === 1, t.timers.size);
}
}
// 3. A finished reconciliation with findings shows the banner and stops.
{
let done = false;
const t = load({ payload: () => (done
? { status: 'success', data: { done: true, unresolved: [{ plugin_id: 'clock', type: 'plugin_missing_on_disk' }] } }
: { status: 'success', data: { done: false, unresolved: [] } }) });
await flush();
await t.fireTimers();
done = true;
await t.fireTimers();
ok('the banner names the finding once reconciliation is done',
t.text.textContent.includes('clock'), t.text.textContent);
const before = t.calls.length;
for (let i = 0; i < 5; i++) await t.fireTimers();
ok('no more requests once it is done', t.calls.length === before && t.timers.size === 0,
{ before, after: t.calls.length, pending: t.timers.size });
}
// 4. Without LEDVisibility (base.html always has it) it still runs, bounded.
{
const t = load({ visibility: false, payload: () => ({ status: 'success', data: { done: false } }) });
await flush();
ok('runs without LEDVisibility', t.calls.length === 1, t.calls.length);
for (let i = 0; i < 200; i++) await t.fireTimers();
ok('...and is still bounded', t.timers.size === 0 && t.calls.length <= 30, t.calls.length);
}
console.log(`\n${pass} passed, ${fail} failed\n`);
process.exit(fail ? 1 : 0);
})().catch(e => { console.log('HARNESS ERROR: ' + e.stack); process.exit(1); });
+189
View File
@@ -0,0 +1,189 @@
// The shared plugin order list (widgets/plugin-order-list.js) keeps what it
// does not show.
//
// It lists enabled plugins only, and rewrites its hidden inputs from those
// rows as soon as it has drawn them. A disabled plugin's place in the order
// and its Vegas exclusion used to vanish from the inputs on that rewrite, so
// any later save of the Display or Rotation & Durations tab stored them
// without it: re-enabled, the plugin came back at the end of the rotation and
// scrolling in Vegas again. An uninstalled plugin's id is still dropped, as
// before, so the lists don't collect ids nothing can show. Runs the shipped
// widget in a vm with a minimal fake DOM -- no jsdom and no server needed, so
// it runs under test/test_js_unit_suites.py too.
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const WIDGET = path.resolve(__dirname, '../../../web_interface/static/v3/js/widgets/plugin-order-list.js');
let pass = 0, fail = 0;
const ok = (label, cond, extra) => cond
? (pass++, console.log(' ok ' + label))
: (fail++, console.log(' FAIL ' + label + (extra !== undefined ? ' ' + JSON.stringify(extra) : '')));
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
class FakeElement {
constructor(tag) {
this.tagName = tag.toUpperCase();
this.children = [];
this.parent = null;
this.dataset = {};
this.style = {};
this.className = '';
this.value = '';
this.checked = false;
this.listeners = {};
this._text = '';
}
appendChild(child) {
if (child.parent) child.parent.children = child.parent.children.filter(c => c !== child);
child.parent = this;
this.children.push(child);
return child;
}
insertBefore(child, ref) {
if (!ref) return this.appendChild(child);
if (child.parent) child.parent.children = child.parent.children.filter(c => c !== child);
child.parent = this;
this.children.splice(this.children.indexOf(ref), 0, child);
return child;
}
get previousElementSibling() {
const siblings = this.parent ? this.parent.children : [];
return siblings[siblings.indexOf(this) - 1] || null;
}
get nextElementSibling() {
const siblings = this.parent ? this.parent.children : [];
const i = siblings.indexOf(this);
return i < 0 ? null : siblings[i + 1] || null;
}
set textContent(value) { this._text = value; this.children = []; }
get textContent() { return this._text; }
setAttribute() {}
focus() {}
addEventListener(type, fn) { (this.listeners[type] ||= []).push(fn); }
fire(type, event) { (this.listeners[type] || []).forEach(fn => fn.call(this, event || {})); }
descendants() { return this.children.flatMap(c => [c, ...c.descendants()]); }
querySelectorAll(selector) {
const cls = selector.replace(/^\./, '');
return this.descendants().filter(e => e.className.split(/\s+/).includes(cls));
}
querySelector(selector) { return this.querySelectorAll(selector)[0] || null; }
}
/** Run the widget over `plugins` with the given saved inputs; resolves once it has drawn. */
async function mount({ plugins, order, excluded, fetchFails }) {
const els = {
list: new FakeElement('div'),
order: Object.assign(new FakeElement('input'), { value: JSON.stringify(order) }),
};
if (excluded !== undefined) {
els.excluded = Object.assign(new FakeElement('input'), { value: JSON.stringify(excluded) });
}
const context = {
// The widget logs a failed list; expected there, so kept off the output.
console: fetchFails ? Object.assign({}, console, { error: () => {} }) : console,
window: {},
document: {
getElementById: (id) => els[id] || null,
createElement: (tag) => new FakeElement(tag),
createTextNode: (text) => new FakeElement('#text'),
},
fetch: () => (fetchFails ? Promise.reject(new Error('service restarting')) : Promise.resolve({
json: () => Promise.resolve({ status: 'success', data: { plugins } }),
})),
};
vm.createContext(context);
vm.runInContext(fs.readFileSync(WIDGET, 'utf8'), context);
context.window.PluginOrderList.init({
containerId: 'list', orderInputId: 'order',
excludedInputId: excluded !== undefined ? 'excluded' : undefined,
});
await new Promise(resolve => setTimeout(resolve, 0));
const rows = () => els.list.querySelectorAll('.plugin-order-item');
return {
rows,
rowIds: () => rows().map(r => r.dataset.pluginId),
order: () => JSON.parse(els.order.value),
excluded: () => JSON.parse(els.excluded.value),
row: (id) => rows().find(r => r.dataset.pluginId === id),
};
}
const PLUGINS = [
{ id: 'weather', name: 'Weather', enabled: true },
{ id: 'clock', name: 'Clock', enabled: false },
{ id: 'stocks', name: 'Stocks', enabled: true },
];
(async () => {
console.log('\nVegas: a disabled plugin keeps its place and its exclusion');
{
const t = await mount({ plugins: PLUGINS, order: ['weather', 'clock', 'stocks'], excluded: ['clock'] });
ok('only enabled plugins get a row', same(t.rowIds(), ['weather', 'stocks']), t.rowIds());
ok('drawing the list keeps the disabled plugin in the order, in its place',
same(t.order(), ['weather', 'clock', 'stocks']), t.order());
ok('drawing the list keeps its exclusion', same(t.excluded(), ['clock']), t.excluded());
// Move Stocks up: the rows swap, and Clock stays in its saved slot.
const up = t.row('stocks').querySelectorAll('.plugin-order-move')[0];
up.fire('click');
ok('reordering the rows fills the other slots in the new order',
same(t.order(), ['stocks', 'clock', 'weather']), t.order());
const include = t.row('weather').querySelector('.plugin-order-include');
include.checked = false;
include.fire('change');
ok('unchecking a row adds it, and the disabled exclusion stays',
same([...t.excluded()].sort(), ['clock', 'weather']), t.excluded());
include.checked = true;
include.fire('change');
ok('checking it again removes only that one', same(t.excluded(), ['clock']), t.excluded());
}
console.log('\nRotation order: the same, without exclusions');
{
const plugins = [
{ id: 'clock', enabled: true },
{ id: 'off', enabled: false },
{ id: 'weather', enabled: true },
{ id: 'new', enabled: true },
];
const t = await mount({ plugins, order: ['clock', 'off', 'weather'] });
ok('the disabled plugin keeps its slot; a plugin not in the saved order goes last',
same(t.order(), ['clock', 'off', 'weather', 'new']), t.order());
}
console.log('\nAn uninstalled plugin is dropped; a failed list keeps everything');
{
const t = await mount({ plugins: PLUGINS, order: ['weather', 'gone', 'clock', 'stocks'],
excluded: ['gone', 'clock'] });
ok('the disabled plugin is kept and the uninstalled one dropped from the order',
same(t.order(), ['weather', 'clock', 'stocks']), t.order());
ok('and from the exclusions', same(t.excluded(), ['clock']), t.excluded());
}
{
const t = await mount({ plugins: PLUGINS, order: ['weather', 'gone', 'clock', 'stocks'],
excluded: ['gone', 'clock'], fetchFails: true });
// No installed list, so nothing can be told apart: no rows, and the
// inputs keep what was saved, uninstalled ids included.
ok('a failed plugin list draws no rows', t.rowIds().length === 0, t.rowIds());
ok('and leaves the saved order as it was',
same(t.order(), ['weather', 'gone', 'clock', 'stocks']), t.order());
ok('and the saved exclusions', same(t.excluded(), ['gone', 'clock']), t.excluded());
}
console.log('\nOnly what the server would accept is carried over');
{
const t = await mount({ plugins: PLUGINS, order: ['weather', 7, 'clock', null, 'clock', 'stocks'],
excluded: ['clock', 3, 'clock'] });
// /config/main refuses a list holding anything but strings, which would
// block every later Display save; a repeated id is kept once.
ok('non-string and repeated saved ids are dropped from the order',
same(t.order(), ['weather', 'clock', 'stocks']), t.order());
ok('and from the exclusions', same(t.excluded(), ['clock']), t.excluded());
}
console.log(`\n${pass} passed, ${fail} failed`);
process.exit(fail ? 1 : 0);
})().catch(e => { console.error(e); process.exit(1); });
+97
View File
@@ -37,6 +37,7 @@ from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401
START_URL = "/api/v3/display/on-demand/start"
STOP_URL = "/api/v3/display/on-demand/stop"
MAILBOX = "display_on_demand_request"
DISPLAY = "web_interface.blueprints.api_v3.display"
@pytest.fixture
@@ -150,6 +151,102 @@ class TestStartWhileTheServiceIsStopped:
assert response.get_json()["status"] == "error"
class _Mailbox:
"""The CacheManager calls the routes make, over a dict."""
def __init__(self):
self.entries = {}
def set(self, key, value, ttl=None):
self.entries[key] = value
def get(self, key, max_age=300, memory_ttl=None):
return self.entries.get(key)
def delete(self, key):
self.entries.pop(key, None)
class TestARefusedStartLeavesNoRequestBehind:
"""A start the route answers with an error must not run later.
The request was posted (to the mailbox, with the display stopped) before
the route refused it, and the display reads the mailbox for an hour
without looking at a request's age. So "Display service is not running"
(start_service off) or "Failed to start display service" left the
request waiting, and the next time the display started -- minutes later,
by hand -- it ran that plugin, pinned if the request said so.
A socket acknowledgement is the other side of it: the display answered,
so it is running and has the request, whatever systemd says (a display
run by hand or in the emulator has no active unit). That is a success,
not "not running", and no unit is started beside it.
"""
@pytest.fixture
def mailbox(self, api_v3_module, service):
box = _Mailbox()
api_v3_module.api_v3.cache_manager = box
service["state"]["active"] = False
return box
@pytest.mark.parametrize("body", [
{"plugin_id": "weather", "start_service": False},
{"plugin_id": "weather"}, # start_service defaults on
])
def test_a_socket_ack_is_a_success_whatever_systemd_says(
self, api_v3_client, service, mailbox, body):
with patch(f"{DISPLAY}.control_client.on_demand_start",
side_effect=lambda request_id, *a: {"accepted": True}):
response = api_v3_client.post(START_URL, json=body)
assert response.status_code == 200, response.get_json()
assert response.get_json()["data"]["transport"] == "socket"
assert MAILBOX not in mailbox.entries
assert _systemctl_verbs(service["systemctl"]) == [], (
"a unit was started beside a display that answered the socket")
def test_without_start_service_the_request_is_taken_back(
self, api_v3_client, service, mailbox):
response = api_v3_client.post(START_URL, json={
"plugin_id": "weather", "pinned": True, "start_service": False})
assert response.status_code == 400
assert response.get_json()["status"] == "error"
assert MAILBOX not in mailbox.entries
def test_a_start_that_fails_takes_its_request_back(self, api_v3_client, service, mailbox):
service["systemctl"].side_effect = lambda args: {
"returncode": 1, "stdout": "", "stderr": "denied"}
response = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert response.status_code == 500
assert MAILBOX not in mailbox.entries
def test_a_newer_request_is_left_alone_on_the_400(self, api_v3_client, service, mailbox):
newer = {"request_id": "someone-else", "action": "start", "plugin_id": "clock"}
def stopped_and_another_post_lands(*args):
mailbox.entries[MAILBOX] = newer
return {"active": False}
with patch(f"{DISPLAY}._get_display_service_status",
side_effect=stopped_and_another_post_lands):
response = api_v3_client.post(START_URL, json={
"plugin_id": "weather", "start_service": False})
assert response.status_code == 400
assert mailbox.entries[MAILBOX] is newer
def test_a_newer_request_is_left_alone_on_the_500(self, api_v3_client, service, mailbox):
newer = {"request_id": "someone-else", "action": "start", "plugin_id": "clock"}
def start_fails_after_another_post(args):
mailbox.entries[MAILBOX] = newer
return {"returncode": 1, "stdout": "", "stderr": "denied"}
service["systemctl"].side_effect = start_fails_after_another_post
response = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert response.status_code == 500
assert mailbox.entries[MAILBOX] is newer
class TestStop:
def test_stop_posts_a_stop_request_and_leaves_the_service_running(
self, api_v3_client, service):
@@ -0,0 +1,83 @@
"""GET /api/v3/plugins/operation/<id> answers for an operation still waiting.
PluginOperationQueue keeps an operation's callback in its parameters, under
``_callback``, until the worker takes it to run. PluginOperation.to_dict()
returned the parameters as they were, so for a pending operation the route
handed jsonify a function and answered 500 "A system error occurred". That
is every poll of an install queued behind another plugin's: the second of
two installs read as broken until the first one finished.
"""
import json
import sys
import threading
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent))
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
from src.plugin_system.operation_queue import PluginOperationQueue # noqa: E402
from src.plugin_system.operation_types import ( # noqa: E402
OperationType, PluginOperation,
)
def _callback(op):
return {"success": True, "message": "done"}
class TestToDict:
def test_private_parameters_are_left_out(self):
op = PluginOperation(OperationType.INSTALL, "demo",
parameters={"_callback": _callback, "branch": "main"})
assert op.to_dict()["parameters"] == {"branch": "main"}
json.dumps(op.to_dict()) # serializable
def test_the_operation_keeps_its_callback_for_the_worker(self):
op = PluginOperation(OperationType.INSTALL, "demo",
parameters={"_callback": _callback})
op.to_dict()
assert op.parameters["_callback"] is _callback
def test_the_other_fields_are_unchanged(self):
op = PluginOperation(OperationType.UNINSTALL, "demo", operation_id="op-1")
assert op.to_dict() == {
"operation_id": "op-1", "operation_type": "uninstall", "plugin_id": "demo",
"parameters": {}, "status": "pending", "progress": 0.0, "message": "",
"error": None, "result": None,
"created_at": op.created_at.isoformat(), "started_at": None,
"completed_at": None,
}
class TestTheRoute:
@pytest.fixture
def busy_queue(self, api_v3_module):
"""A real queue whose worker is held by another plugin's operation."""
queue = PluginOperationQueue(max_history=10)
api_v3_module.api_v3.operation_queue = queue
started, release = threading.Event(), threading.Event()
def blocker(op):
started.set()
release.wait(10)
return {"success": True, "message": "done"}
queue.enqueue_operation(OperationType.INSTALL, "busy", operation_callback=blocker)
assert started.wait(5)
yield queue
release.set()
queue.shutdown()
def test_a_pending_operation_reports_pending(self, api_v3_client, busy_queue):
op_id = busy_queue.enqueue_operation(
OperationType.INSTALL, "demo", operation_callback=_callback)
response = api_v3_client.get(f"/api/v3/plugins/operation/{op_id}")
assert response.status_code == 200, response.get_json()
data = response.get_json()["data"]
assert data["status"] == "pending"
assert data["plugin_id"] == "demo"
assert "_callback" not in data["parameters"]
+26
View File
@@ -253,6 +253,32 @@ class TestVegasCycleDurations:
assert saved['config']['display']['display_durations'] == {'clock': 45}
class TestMalformedBody:
"""A JSON body that does not parse is the caller's mistake: a 400.
get_json() raised Werkzeug's BadRequest inside the handler's try, whose
catch-all answered 500 CONFIG_SAVE_FAILED with "check file permissions"
advice and logged a traceback at ERROR.
"""
def test_is_a_400_in_the_raw_routes_shape(self, api_v3_client, saved, api_v3_module):
api_v3_module.api_v3.config_manager.get_raw_file_content.return_value = {}
resp = api_v3_client.post('/api/v3/config/main', data='{not json',
content_type='application/json')
assert resp.status_code == 400
assert resp.get_json() == {'status': 'error', 'message': 'Invalid JSON in request body'}
assert 'config' not in saved
raw = api_v3_client.post('/api/v3/config/raw/main', data='{not json',
content_type='application/json')
assert (raw.status_code, raw.get_json()) == (400, resp.get_json())
def test_an_empty_json_post_is_still_no_data(self, api_v3_client, saved):
resp = api_v3_client.post('/api/v3/config/main', data='',
content_type='application/json')
assert resp.status_code == 400
assert resp.get_json()['message'] == 'No data provided'
class TestRawSaveStartsAutoUpdateSetup:
@pytest.fixture
def raw_env(self, api_v3_module, monkeypatch):
+93
View File
@@ -0,0 +1,93 @@
"""POST /api/v3/plugins/action hands ``params`` to the plugin's script intact.
The route runs the script through a generated wrapper, and the params went
into that wrapper as Python source: ``params = {json.dumps(params)}``. JSON is
not Python. ``true``, ``false`` and ``null`` are undefined names there, so any
params holding a boolean or a null died with a NameError before the script
ran. The plugin file manager's category toggle sends ``{"category_name": ...,
"enabled": true}``, so of-the-day's category toggle failed every time with
"Action failed".
The script's side of the contract is unchanged and pinned here too: the
params arrive on stdin as one JSON document, LEDMATRIX_ROOT is set, and what
the script prints to stdout is what the route parses.
"""
import json
import subprocess
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent))
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
ACTION_URL = "/api/v3/plugins/action"
# The action script: report what it was handed, as JSON on stdout.
ECHO_SCRIPT = (
"import json, os, sys\n"
"raw = sys.stdin.read()\n"
"print(json.dumps({'status': 'success', 'got': json.loads(raw),\n"
" 'root': os.environ.get('LEDMATRIX_ROOT')}))\n"
)
@pytest.fixture
def echo_plugin(tmp_path, api_v3_module, monkeypatch):
plugin_dir = tmp_path / "demo"
plugin_dir.mkdir()
(plugin_dir / "manifest.json").write_text(json.dumps({
"id": "demo",
"web_ui_actions": [{"id": "toggle", "type": "script", "script": "echo.py"}],
}), encoding="utf-8")
(plugin_dir / "echo.py").write_text(ECHO_SCRIPT, encoding="utf-8")
api_v3_module.api_v3.plugin_catalog.get_plugin_directory.return_value = str(plugin_dir)
# The route runs `python3`; use this interpreter, so the test does not
# depend on what that name resolves to here.
real_run = subprocess.run
def run(cmd, *args, **kwargs):
if isinstance(cmd, list) and cmd and cmd[0] == "python3":
cmd = [sys.executable] + cmd[1:]
return real_run(cmd, *args, **kwargs)
monkeypatch.setattr(subprocess, "run", run)
return plugin_dir
@pytest.mark.parametrize("params", [
{"category_name": "jokes", "enabled": True}, # the file manager's toggle
{"category_name": "jokes", "enabled": False},
{"filename": None},
{"nested": {"list": [1, None, True, 2.5], "empty": {}}},
{"text": "café ✓ \U0001F600"},
{"text": "he said \"hi\" and 'bye' \\ ''' \"\"\" \n\t end"},
], ids=["true", "false", "null", "nested", "unicode", "quotes"])
def test_the_script_receives_the_params_it_was_sent(api_v3_client, echo_plugin, params):
response = api_v3_client.post(ACTION_URL, json={
"plugin_id": "demo", "action_id": "toggle", "params": params})
body = response.get_json()
assert response.status_code == 200, body
assert body["got"] == params
def test_a_param_cannot_run_code_in_the_wrapper(api_v3_client, echo_plugin, tmp_path):
marker = tmp_path / "PWNED"
hostile = "\"}\nopen(%r, 'w').write('ran')\n#" % str(marker)
params = {"name": hostile, "flag": True}
response = api_v3_client.post(ACTION_URL, json={
"plugin_id": "demo", "action_id": "toggle", "params": params})
assert response.status_code == 200, response.get_json()
assert response.get_json()["got"] == params
assert not marker.exists(), "a param value ran as code"
def test_the_script_still_gets_ledmatrix_root(api_v3_client, echo_plugin, api_v3_module):
response = api_v3_client.post(ACTION_URL, json={
"plugin_id": "demo", "action_id": "toggle", "params": {"enabled": True}})
assert response.status_code == 200, response.get_json()
assert response.get_json()["root"] == str(api_v3_module.PROJECT_ROOT)
@@ -0,0 +1,89 @@
"""A second install or uninstall while one is in progress is a 409, not a 500.
PluginOperationQueue refuses a second operation for a plugin that already
has one waiting or running (test_operation_queue_pending_and_trim.py), and
says so by raising ValueError. /plugins/install let that escape to the
blueprint's catch-all, so a double-clicked Install answered 500 "An error
occurred; see logs for details" while the first install carried on.
/plugins/uninstall caught it in its own catch-all: a 500 "Failed to
uninstall plugin", and an "uninstall failed" entry in the operation
history for an uninstall that never started.
"""
import sys
import threading
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent))
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
from src.plugin_system.operation_queue import PluginOperationQueue # noqa: E402
INSTALL = "/api/v3/plugins/install"
UNINSTALL = "/api/v3/plugins/uninstall"
@pytest.fixture
def installing(api_v3_module, tmp_path):
"""A real queue with an install of "clock" running and held there."""
queue = PluginOperationQueue(max_history=10)
api_v3_module.api_v3.operation_queue = queue
started, release = threading.Event(), threading.Event()
def slow_install(plugin_id, branch=None):
started.set()
release.wait(10)
return True
store = api_v3_module.api_v3.plugin_store_manager
store.install_plugin.side_effect = slow_install
store.get_registry_info.return_value = None
store.plugins_dir = str(tmp_path)
api_v3_module.api_v3.plugin_catalog.get_plugin_directory.return_value = None
yield {"queue": queue, "started": started, "store": store}
release.set()
queue.shutdown()
def _start_first_install(client, installing):
response = client.post(INSTALL, json={"plugin_id": "clock"})
assert response.status_code == 200, response.get_json()
assert installing["started"].wait(5)
def _failed_history(api_v3_module):
return [c for c in api_v3_module.api_v3.operation_history.record_operation.call_args_list
if c.kwargs.get("status") == "failed"]
def test_a_second_install_click_is_a_conflict(api_v3_client, api_v3_module, installing):
_start_first_install(api_v3_client, installing)
response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"})
assert response.status_code == 409, response.get_json()
body = response.get_json()
assert body["status"] == "error"
assert body["error_code"] == "PLUGIN_OPERATION_CONFLICT"
assert "clock" in body["message"]
assert installing["store"].install_plugin.call_count == 1
assert _failed_history(api_v3_module) == []
def test_an_uninstall_during_the_install_is_a_conflict(api_v3_client, api_v3_module,
installing):
_start_first_install(api_v3_client, installing)
response = api_v3_client.post(UNINSTALL, json={"plugin_id": "clock"})
assert response.status_code == 409, response.get_json()
assert response.get_json()["error_code"] == "PLUGIN_OPERATION_CONFLICT"
assert _failed_history(api_v3_module) == [], (
"an uninstall that never started was recorded as failed")
api_v3_module.api_v3.plugin_store_manager.uninstall_plugin.assert_not_called()
def test_another_plugin_is_still_queued(api_v3_client, installing):
_start_first_install(api_v3_client, installing)
response = api_v3_client.post(INSTALL, json={"plugin_id": "weather"})
assert response.status_code == 200, response.get_json()
assert response.get_json()["data"]["operation_id"]
+73
View File
@@ -0,0 +1,73 @@
"""GET /api/v3/plugins/<plugin_id>/static/<path> serves binary files too.
The route opened every file as UTF-8 text, so an image -- what the API
reference says it is for, plugin previews and icons -- failed to decode and
answered 500 "UnicodeDecodeError". Files are now sent as bytes. The text
types the route always set are unchanged, and the path checks are pinned in
test_path_traversal_guards.py::TestServePluginStatic.
"""
import json
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent))
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
PNG = (b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89")
@pytest.fixture
def plugin_dir(tmp_path, api_v3_module):
d = tmp_path / "demo"
(d / "web_ui").mkdir(parents=True)
(d / "manifest.json").write_text(json.dumps({"id": "demo"}), encoding="utf-8")
api_v3_module.api_v3.plugin_catalog.get_plugin_directory.side_effect = (
lambda pid: str(d) if pid == "demo" else None)
return d
def _get(client, path):
return client.get(f"/api/v3/plugins/demo/static/{path}")
def test_an_image_is_served_as_its_bytes(api_v3_client, plugin_dir):
(plugin_dir / "web_ui" / "icon.png").write_bytes(PNG)
response = _get(api_v3_client, "web_ui/icon.png")
assert response.status_code == 200, response.get_json(silent=True)
assert response.mimetype == "image/png"
assert response.data == PNG
def test_an_unknown_binary_file_is_served_too(api_v3_client, plugin_dir):
blob = bytes(range(256))
(plugin_dir / "data.bin").write_bytes(blob)
response = _get(api_v3_client, "data.bin")
assert response.status_code == 200, response.get_json(silent=True)
assert response.data == blob
@pytest.mark.parametrize("name,mimetype", [
("page.html", "text/html"),
("app.js", "application/javascript"),
("style.css", "text/css"),
("data.json", "application/json"),
("notes.txt", "text/plain"),
("README.md", "text/plain"),
("helper.py", "text/plain"),
])
def test_text_files_keep_their_types(api_v3_client, plugin_dir, name, mimetype):
content = "caf\u00e9 \u2713 <p>hi</p>\n"
(plugin_dir / name).write_bytes(content.encode("utf-8"))
response = _get(api_v3_client, name)
assert response.status_code == 200
assert response.mimetype == mimetype
assert response.data == content.encode("utf-8")
def test_a_missing_file_is_still_a_404(api_v3_client, plugin_dir):
assert _get(api_v3_client, "nope.png").status_code == 404
+326
View File
@@ -0,0 +1,326 @@
"""Four web answers that disagreed with the rig they describe (found on ledpi).
1. POST /config/schedule refused the schedule GET returns on a fresh install
(config.template.json: per-day, every day off, schedule disabled) with
"At least one day must be enabled", as did /config/dim-schedule. A
disabled schedule needs no enabled day.
2. A brightness-only POST /config/main answered ``restart_required: true``,
though the display applies brightness live (brightness.set over the
socket, and the config watcher). The flag now says whether anything
changed that the running display does not pick up by itself.
3. /health stayed "healthy" with the display service stopped: only the
sub-checks changed. Service inactive, no socket and no live heartbeat
is now ``display_loop: stopped`` and "degraded".
4. /display/current-status kept answering ``is_display_active: true`` from
the cache for up to 120 s after the display stopped. With no socket and
no live heartbeat it is now unknown.
"""
import copy
import json
import os
import sys
import time
from pathlib import Path
from unittest.mock import patch
import pytest
sys.path.insert(0, str(Path(__file__).parent.parent))
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
from src import display_watchdog # noqa: E402
from src.ipc import client as control_client # noqa: E402
from web_interface import display_state # noqa: E402
REPO = Path(__file__).resolve().parent.parent
TEMPLATE = json.loads((REPO / 'config' / 'config.template.json').read_text(encoding='utf-8'))
@pytest.fixture
def store(api_v3_module, monkeypatch):
state = {'config': {}, 'saves': 0}
api_v3_module.api_v3.config_manager.load_config.side_effect = \
lambda *a, **k: copy.deepcopy(state['config'])
def fake_save(_manager, config, **_kwargs):
state['config'] = copy.deepcopy(config)
state['saves'] += 1
return True, ''
monkeypatch.setattr(api_v3_module, '_save_config_atomic', fake_save)
return state
# --- 1. schedules ---------------------------------------------------------------
SCHEDULE_ROUTES = [('/api/v3/config/schedule', 'schedule'),
('/api/v3/config/dim-schedule', 'dim_schedule')]
@pytest.mark.parametrize('route,section', SCHEDULE_ROUTES)
def test_the_templates_disabled_per_day_schedule_saves_back(api_v3_client, store,
route, section):
stored = copy.deepcopy(TEMPLATE[section])
stored['mode'] = 'per-day'
assert stored['enabled'] is False
assert not any(day['enabled'] for day in stored['days'].values())
store['config'] = {section: copy.deepcopy(stored)}
read = api_v3_client.get(route).get_json()['data']
resp = api_v3_client.post(route, json=read)
assert resp.status_code == 200, resp.get_json()
saved = store['config'][section]
assert saved['enabled'] is False and saved['mode'] == 'per-day'
# The disabled days keep their times: switching one on finds them.
assert saved['days'] == stored['days']
@pytest.mark.parametrize('route,section', SCHEDULE_ROUTES)
def test_an_enabled_per_day_schedule_still_needs_a_day(api_v3_client, store, route, section):
body = copy.deepcopy(TEMPLATE[section])
body.update(enabled=True, mode='per-day')
resp = api_v3_client.post(route, json=body)
assert resp.status_code == 400
assert 'At least one day must be enabled' in resp.get_json()['message']
assert store['saves'] == 0
@pytest.mark.parametrize('route', [r for r, _ in SCHEDULE_ROUTES])
def test_the_pickers_form_post_with_every_day_off_saves(api_v3_client, store, route):
"""What schedule-picker.js posts: flat hidden inputs, booleans as strings,
times for every day."""
body = {'enabled': 'false', 'mode': 'per_day', 'start_time': '07:00', 'end_time': '23:00'}
for day in ('monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday'):
body.update({f'{day}_enabled': 'false', f'{day}_start': '06:30', f'{day}_end': '22:15'})
resp = api_v3_client.post(route, json=body)
assert resp.status_code == 200, resp.get_json()
def test_an_invalid_time_on_a_disabled_day_is_dropped_not_refused(api_v3_client, store):
body = {'enabled': False, 'mode': 'per-day',
'days': {'monday': {'enabled': False, 'start_time': 'soon', 'end_time': '22:00'}}}
resp = api_v3_client.post('/api/v3/config/schedule', json=body)
assert resp.status_code == 200, resp.get_json()
assert store['config']['schedule']['days']['monday'] == {'enabled': False,
'end_time': '22:00'}
# --- 2. restart_required on /config/main ------------------------------------------
STORED_MAIN = {
'timezone': 'America/Chicago',
'display': {
'hardware': {'rows': 32, 'cols': 64, 'chain_length': 2, 'brightness': 90,
'disable_hardware_pulsing': False, 'inverse_colors': False,
'show_refresh_rate': False},
'runtime': {'gpio_slowdown': 4},
'display_durations': {'clock': 15},
'use_short_date_format': False,
},
}
@pytest.fixture
def main_store(store):
store['config'] = copy.deepcopy(STORED_MAIN)
return store
def _save_main(client, body):
with patch('web_interface.blueprints.api_v3.control_client.brightness_set',
side_effect=control_client.ControlError('no_socket', 'x')):
resp = client.post('/api/v3/config/main', data=json.dumps(body),
content_type='application/json')
assert resp.status_code == 200, resp.get_json()
return resp.get_json()
def test_a_brightness_only_save_needs_no_restart(api_v3_client, main_store):
body = _save_main(api_v3_client, {'brightness': 40})
assert main_store['config']['display']['hardware']['brightness'] == 40
assert body['restart_required'] is False
def test_a_brightness_save_on_a_config_without_a_display_section(api_v3_client, store):
"""The route creates display.hardware and display.runtime on the way;
empty sections are not a change."""
store['config'] = {}
assert _save_main(api_v3_client, {'brightness': 40})['restart_required'] is False
def test_the_display_form_with_only_brightness_changed_needs_no_restart(api_v3_client,
main_store):
hw = STORED_MAIN['display']['hardware']
body = {'__form_section': 'display', 'rows': 32, 'cols': 64, 'chain_length': 2,
'brightness': 55, 'gpio_slowdown': 4}
body.update({k: 'on' for k in ('disable_hardware_pulsing', 'inverse_colors',
'show_refresh_rate') if hw[k]})
assert _save_main(api_v3_client, body)['restart_required'] is False
def test_a_mode_duration_needs_no_restart(api_v3_client, main_store):
body = _save_main(api_v3_client, {'duration__clock': 40})
assert main_store['config']['display']['display_durations']['clock'] == 40
assert body['restart_required'] is False
@pytest.mark.parametrize('change', [{'rows': 64}, {'brightness': 40, 'chain_length': 3},
{'gpio_slowdown': 2}, {'timezone': 'UTC'}])
def test_a_setting_the_display_reads_at_startup_still_needs_one(api_v3_client, main_store,
change):
assert _save_main(api_v3_client, change)['restart_required'] is True
def test_restart_needed_compares_leaves():
from web_interface.blueprints.api_v3.config import restart_needed
before = {'display': {'hardware': {'brightness': 90, 'rows': 32}}}
assert not restart_needed(before, copy.deepcopy(before))
assert not restart_needed(before, {'display': {'hardware': {'brightness': 10, 'rows': 32},
'runtime': {}}})
assert restart_needed(before, {'display': {'hardware': {'brightness': 90}}}) # removed
assert not restart_needed({}, {'clock': {'enabled': True}}, live_paths=[('clock',)])
assert restart_needed({}, {'clockwork': {'enabled': True}}, live_paths=[('clock',)])
# --- 3 and 4. a stopped display -----------------------------------------------------
@pytest.fixture
def no_display(monkeypatch, tmp_path):
"""A Pi whose display service has stopped: the socket is expected here
but does not answer, and systemd took the heartbeat's directory away."""
monkeypatch.setattr(display_state, 'socket_supported', lambda: True)
monkeypatch.setattr(display_state, 'client_socket_paths', lambda: [str(tmp_path / 'gone')])
monkeypatch.setattr(display_state, 'read_state', lambda: None)
path = tmp_path / 'display-heartbeat.json'
monkeypatch.setattr(display_watchdog, 'HEARTBEAT_PATH', str(path))
def beat(age, pid=None):
path.write_text(json.dumps({'pid': os.getpid() if pid is None else pid,
'mono': time.monotonic() - age,
'wall': time.time() - age}))
return beat
@pytest.fixture
def service(monkeypatch):
status = {'active': False, 'returncode': 3, 'stdout': 'inactive', 'stderr': ''}
monkeypatch.setattr('web_interface.blueprints.api_v3.misc._get_display_service_status',
lambda: dict(status))
return status
@pytest.fixture
def fresh_preview(tmp_path, monkeypatch):
"""The preview frame the display left behind, under 60 s old: on its own
it kept the hardware check "connected"."""
from web_interface import display_preview
snapshot = tmp_path / 'preview.png'
snapshot.write_bytes(b'png')
monkeypatch.setattr(display_preview, 'SNAPSHOT_PATH', str(snapshot))
def _health(client):
resp = client.get('/api/v3/health')
assert resp.status_code == 200, resp.get_json()
return resp.get_json()['data']
class TestHealth:
def test_a_stopped_display_service_is_degraded(self, api_v3_client, no_display, service,
fresh_preview):
data = _health(api_v3_client)
assert data['services']['display_service']['status'] == 'inactive'
assert data['checks']['display_loop']['status'] == 'stopped'
assert data['status'] == 'degraded'
def test_a_service_still_starting_is_not(self, api_v3_client, no_display, service,
fresh_preview):
"""Active, before its socket and first heartbeat: not stopped."""
service.update(active=True, stdout='active', returncode=0)
data = _health(api_v3_client)
assert data['checks']['display_loop']['status'] == 'not_reported'
assert data['status'] == 'healthy'
def test_a_display_run_by_hand_is_not_stopped(self, api_v3_client, no_display, service,
fresh_preview):
"""The service is off but a display process beats (sudo python3 run.py)."""
no_display(age=2)
data = _health(api_v3_client)
assert data['checks']['display_loop']['status'] == 'running'
assert data['status'] == 'healthy'
@pytest.mark.parametrize('platform', ['no_unix_sockets', 'socket_off'])
def test_without_a_socket_to_expect_nothing_changes(self, api_v3_client, no_display,
service, fresh_preview, monkeypatch,
platform):
"""Windows and the dev server (no systemd unit), or the socket
deliberately off: no heartbeat is no signal, as before."""
if platform == 'no_unix_sockets':
monkeypatch.setattr(display_state, 'socket_supported', lambda: False)
else:
monkeypatch.setattr(display_state, 'client_socket_paths', lambda: [])
service.update(returncode=-1, stdout='', stderr='systemctl not found')
data = _health(api_v3_client)
assert data['checks']['display_loop']['status'] == 'not_reported'
assert data['status'] == 'healthy'
def test_the_status_only_answer_says_degraded(self, api_v3_client, no_display, service,
fresh_preview, monkeypatch):
monkeypatch.setattr('web_interface.blueprints.api_v3.misc.request_is_authenticated',
lambda: False)
resp = api_v3_client.get('/api/v3/health')
assert resp.get_json()['data'] == {'status': 'degraded'}
class TestCurrentStatus:
CACHED = {'mode': 'clock', 'plugin_id': 'clock', 'is_display_active': True,
'on_demand_active': False, 'last_updated': None}
@pytest.fixture
def cached(self, api_v3_module):
entry = dict(self.CACHED, last_updated=time.time() - 30)
cache = api_v3_module.api_v3.cache_manager
cache.get.side_effect = lambda key, *a, **kw: (
dict(entry) if key == 'display_current_state' else None)
return entry
def _status(self, client):
resp = client.get('/api/v3/display/current-status')
assert resp.status_code == 200
return resp.get_json()['data']
def test_a_stopped_display_is_not_reported_active(self, api_v3_client, no_display, cached):
data = self._status(api_v3_client)
assert not data.get('is_display_active')
assert data['mode'] is None and data['last_updated'] is None
assert data['source'] == 'cache'
def test_a_stale_heartbeat_is_not_active_either(self, api_v3_client, no_display, cached):
no_display(age=display_watchdog.HEARTBEAT_STALE_SECONDS + 5)
assert self._status(api_v3_client)['mode'] is None
@pytest.mark.skipif(os.name != 'posix', reason='process_exists answers only on POSIX')
def test_a_heartbeat_from_a_dead_process_is_not_active(self, api_v3_client, no_display,
cached):
no_display(age=1, pid=2 ** 22 + 12345)
assert self._status(api_v3_client)['mode'] is None
def test_a_live_heartbeat_without_a_socket_reads_the_cache(self, api_v3_client,
no_display, cached):
"""An older display with no socket, still running."""
no_display(age=2)
data = self._status(api_v3_client)
assert data['mode'] == 'clock' and data['is_display_active'] is True
@pytest.mark.parametrize('platform', ['no_unix_sockets', 'socket_off'])
def test_without_a_socket_to_expect_the_cache_answers(self, api_v3_client, no_display,
cached, monkeypatch, platform):
if platform == 'no_unix_sockets':
monkeypatch.setattr(display_state, 'socket_supported', lambda: False)
else:
monkeypatch.setattr(display_state, 'client_socket_paths', lambda: [])
data = self._status(api_v3_client)
assert data['mode'] == 'clock' and data['is_display_active'] is True
+38 -1
View File
@@ -58,7 +58,8 @@ class TestFastPath:
for _ in range(10):
again = m.load_config()
assert counts["n"] == 0, "fast path must not re-open any config file"
assert again is first # same aliasing semantics as the full path
assert again == first
assert again is not first # each caller gets its own copy, see below
def test_config_change_triggers_reload(self, mgr):
m, config, secrets, template = mgr
@@ -98,6 +99,33 @@ class TestFastPath:
assert m.load_config()["timezone"] == "America/New_York"
class TestCallersGetACopy:
"""A web handler edits what load_config returned, then validates. When
validation failed, the edit stayed in the cache the fast path serves, and
the next unrelated save wrote it -- a nested secret included, in plain
text, because it had never reached config_secrets.json to be stripped."""
def test_editing_a_loaded_config_does_not_change_the_next_load(self, mgr):
m, config, secrets, template = mgr
loaded = m.load_config()
loaded["display"]["brightness"] = 1
loaded["weather"]["api_key"] = "typed-but-never-saved"
again = m.load_config()
assert again["display"]["brightness"] == 90
assert again["weather"]["api_key"] == "sek"
def test_the_full_path_also_returns_a_copy(self, mgr):
m, config, secrets, template = mgr
m.load_config()["display"]["brightness"] = 1 # first load: full path
assert m.load_config()["display"]["brightness"] == 90
def test_an_edit_never_reaches_a_later_save(self, mgr):
m, config, secrets, template = mgr
m.load_config()["display"]["new_secret"] = "hunter2" # then bailed out
m.save_config(m.load_config()) # some other handler saves
assert "hunter2" not in config.read_text()
class TestSaveCoherence:
def test_save_config_then_load_returns_saved_data(self, mgr, monkeypatch):
m, config, secrets, template = mgr
@@ -111,6 +139,15 @@ class TestSaveCoherence:
assert loaded["weather"]["api_key"] == "sek" # secrets survive in memory
assert counts["n"] == 0 # signature refreshed by save; no re-read
def test_the_saved_dict_does_not_become_the_cache(self, mgr):
m, config, secrets, template = mgr
m.load_config()
new = {"display": {"brightness": 42}, "timezone": "UTC",
"weather": {"api_key": "sek"}}
m.save_config(new)
new["display"]["brightness"] = 7 # the caller keeps using its dict
assert m.load_config()["display"]["brightness"] == 42
def test_cross_process_save_is_picked_up(self, mgr):
"""Another process writing config.json (different mtime) must bust
this process's fast path — the core cross-process guarantee."""
+4 -1
View File
@@ -143,7 +143,10 @@ class TestLoadFastPath:
manager = make_manager(tmp_path, config={"timezone": "UTC"})
first = manager.load_config()
second = manager.load_config()
assert second is first # same aliased dict, no re-read
# A copy of the cached dict, never the dict itself; that it is not
# re-read is test_config_load_cache's test_unchanged_files_are_not_reread
assert second == first
assert second is not first
def test_touching_secrets_file_invalidates_cache(self, tmp_path):
manager = make_manager(
+168
View File
@@ -489,3 +489,171 @@ class TestConcurrency:
assert live["peak"] <= espn_dates.ESPN_CHUNK_WORKERS
assert live["peak"] > 1, "chunks should actually overlap"
class TestEdgeMonths:
"""A window's partial edge months are asked whole and trimmed.
The default scoreboard window -- a fortnight either side of today -- spans
two partial months, so it used to cost 29 day requests per league. ESPN's
``dates=YYYYMMDD`` means a US Eastern day (verified against the live API
on 2026-10-03, 417 of 417 soccer events), so a month answer trimmed to
the window's Eastern days is what the day requests returned.
"""
def test_a_fortnight_either_side_is_two_requests(self):
planned = espn_dates.espn_request_chunks(date(2026, 9, 20), date(2026, 10, 18))
assert planned == [
("202609", (date(2026, 9, 20), date(2026, 9, 30))),
("202610", (date(2026, 10, 1), date(2026, 10, 18))),
]
def test_a_live_polls_two_days_stay_two_days(self):
planned = espn_dates.espn_request_chunks(date(2026, 10, 2), date(2026, 10, 3))
assert planned == [("20261002", None), ("20261003", None)]
def test_the_threshold_is_inclusive(self):
n = espn_dates.ESPN_MONTH_COVER_MIN_DAYS
short = espn_dates.espn_request_chunks(date(2026, 10, 1), date(2026, 10, n - 1))
assert [chunk for chunk, _ in short] == [
"202610%02d" % day for day in range(1, n)]
enough = espn_dates.espn_request_chunks(date(2026, 10, 1), date(2026, 10, n))
assert enough == [("202610", (date(2026, 10, 1), date(2026, 10, n)))]
def test_whole_months_and_short_edges_are_unchanged(self):
planned = espn_dates.espn_request_chunks(date(2026, 8, 30), date(2026, 10, 2))
assert planned == [
("20260830", None), ("20260831", None), ("202609", None),
("20261001", None), ("20261002", None),
]
def test_without_time_zone_data_edges_stay_days(self, monkeypatch):
monkeypatch.setattr(espn_dates, "_EASTERN", None)
planned = espn_dates.espn_request_chunks(date(2026, 9, 20), date(2026, 10, 18))
assert len(planned) == 29
assert all(trim is None for _, trim in planned)
@pytest.mark.parametrize("start,end", [
(date(2026, 9, 20), date(2026, 10, 18)),
(date(2026, 1, 25), date(2026, 3, 3)),
(date(2026, 12, 20), date(2027, 1, 9)),
(date(2026, 10, 5), date(2026, 10, 9)),
])
def test_the_planned_requests_still_cover_every_day_exactly_once(self, start, end):
covered = []
for chunk, trim in espn_dates.espn_request_chunks(start, end):
if trim is None:
covered.extend(days_covered_by([chunk]))
else:
assert chunk == trim[0].strftime("%Y%m") == trim[1].strftime("%Y%m")
covered.extend(trim[0] + timedelta(days=offset)
for offset in range((trim[1] - trim[0]).days + 1))
expected = [start + timedelta(days=offset) for offset in range((end - start).days + 1)]
assert covered == expected
def test_a_trimmed_month_keeps_only_the_windows_eastern_days(self):
september = [
# 03:30Z on the 20th is still the 19th in New York: outside.
{"id": "before", "date": "2026-09-20T03:30Z"},
{"id": "first", "date": "2026-09-20T14:00Z"},
{"id": "late", "date": "2026-09-30T23:30Z"},
]
october = [
{"id": "oct1", "date": "2026-10-01T19:00Z"},
# 03:30Z on the 19th is the evening of the 18th in New York: inside.
{"id": "last", "date": "2026-10-19T03:30Z"},
{"id": "after", "date": "2026-10-19T14:00Z"},
{"id": "undated"},
]
session = FakeSession({"202609": september, "202610": october})
data = fetch_espn_date_chunks(session, URL, params={"dates": "20260920-20261018"})
assert sorted(call["dates"] for call in session.calls) == ["202609", "202610"]
# An event with no readable date is kept, never dropped on a guess.
assert [e["id"] for e in data["events"]] == ["first", "late", "oct1", "last", "undated"]
def test_eastern_standard_time_is_honoured_after_the_clocks_change(self):
# 2026-11-01 ends daylight saving: Eastern is UTC-5 from then on.
november = [
{"id": "out", "date": "2026-11-15T04:30Z"}, # Nov 14, 23:30 EST
{"id": "in", "date": "2026-11-15T05:30Z"}, # Nov 15, 00:30 EST
]
session = FakeSession({"202611": november})
data = fetch_espn_date_chunks(session, URL, params={"dates": "20261115-20261121"})
assert [e["id"] for e in data["events"]] == ["in"]
def test_a_capped_edge_month_re_asks_only_the_windows_days(self):
full = [{"id": "cap%d" % i, "date": "2026-10-05T18:00Z"} for i in range(ESPN_MAX_LIMIT)]
by_chunk = {"202610": full}
by_chunk.update({"202610%02d" % day: [{"id": "o%02d" % day}] for day in range(1, 32)})
session = FakeSession(by_chunk)
data = fetch_espn_date_chunks(session, URL, params={"dates": "20261001-20261010"})
sent = [call["dates"] for call in session.calls]
assert sent[0] == "202610"
assert sorted(sent[1:]) == ["202610%02d" % day for day in range(1, 11)]
assert [e["id"] for e in data["events"]] == ["o%02d" % day for day in range(1, 11)]
class TestProcessWideChunkCap:
"""The chunk cap holds across windows, not per window.
A soccer board starting eight leagues fetches sixteen windows at once.
With a pool of ``ESPN_CHUNK_WORKERS`` each, ~40 requests were in flight
and every one past a session's pool opened a connection -- and a DNS
lookup. On ledpi that was ~90 NameResolutionErrors per start.
"""
def test_concurrent_windows_share_one_budget(self):
live = {"now": 0, "peak": 0}
guard = threading.Lock()
class CountingSession(FakeSession):
def get(self, url, params=None, headers=None, timeout=None):
with guard:
live["now"] += 1
live["peak"] = max(live["peak"], live["now"])
try:
time.sleep(0.01)
return super().get(url, params=params, headers=headers, timeout=timeout)
finally:
with guard:
live["now"] -= 1
sessions = [CountingSession() for _ in range(6)]
# Six leagues, so the fetch service cannot merge them into one, on a
# host with no token bucket: earlier tests may have spent ESPN's
# burst, and a bucket paced at 20/s would serialise these by itself.
threads = [
threading.Thread(target=fetch_espn_date_chunks,
args=(session, "https://scores.example.test/league%d" % index),
kwargs={"params": {"dates": "20260101-20261231"}})
for index, session in enumerate(sessions)
]
for thread in threads:
thread.start()
for thread in threads:
thread.join(timeout=30)
assert all(len(session.calls) == 12 for session in sessions)
assert live["peak"] <= espn_dates.ESPN_CHUNK_WORKERS
assert live["peak"] > 1, "chunks should still overlap"
def test_a_fresh_process_skips_the_doomed_range_request():
"""Every start used to spend one 400 per window learning that ranges are
still rejected -- eleven at once from a soccer board. A new process now
starts inside the retry period instead."""
import subprocess
import sys
from pathlib import Path
out = subprocess.run(
[sys.executable, "-c",
"import src.common.espn_dates as e; print(e._ranges_known_rejected())"],
cwd=str(Path(__file__).resolve().parents[1]),
capture_output=True, text=True, timeout=60,
)
assert out.stdout.strip() == "True", out.stderr
+38 -2
View File
@@ -576,6 +576,42 @@ class TestCounters:
assert snap["hosts"]["site.api.espn.com"]["requests"] == 1
assert snap["totals"]["bytes"] == 3 * len(b'{"ok": 1}')
def test_wire_bytes_are_the_compressed_size(self, service):
# Built the way requests builds a real response: a urllib3
# HTTPResponse carrying a gzip body, decoded when .content is read.
import gzip
import io
from requests.adapters import HTTPAdapter
from urllib3.response import HTTPResponse
decoded = json.dumps({"events": [{"id": str(i), "name": "x" * 200}
for i in range(50)]}).encode()
wire = gzip.compress(decoded)
def handler(url, kwargs):
raw = HTTPResponse(body=io.BytesIO(wire), status=200,
headers={"Content-Encoding": "gzip",
"Content-Type": "application/json"},
preload_content=False, decode_content=True)
request = requests.Request("GET", url).prepare()
response = HTTPAdapter().build_response(request, raw)
response.content # what Session.get does for a non-streamed call
return response
response = service.get(FakeSession(handler), "https://site.api.espn.com/x")
assert response.content == decoded
totals = _counters(service)
assert totals["bytes"] == len(decoded)
assert totals["wire_bytes"] == len(wire) < len(decoded)
def test_wire_bytes_fall_back_to_the_decoded_size(self, service):
# No urllib3 response behind it (a test double, another adapter):
# count what is known rather than nothing.
service.get(FakeSession(), "https://api.test/x")
totals = _counters(service)
assert totals["wire_bytes"] == totals["bytes"] == len(b'{"ok": 1}')
def test_errors_and_http_errors(self, service):
def handler(url, kwargs):
if url.endswith("/down"):
@@ -670,14 +706,14 @@ class TestCallerIdentity:
assert _counters(global_service, plugin="football-scoreboard")["requests"] == 1
def test_espn_chunks_on_worker_threads_count_against_the_caller(self, global_service):
from src.common.espn_dates import espn_date_chunks, fetch_espn_date_chunks, parse_espn_date_range
from src.common.espn_dates import espn_request_chunks, fetch_espn_date_chunks, parse_espn_date_range
session = FakeSession(lambda url, kw: make_response(body=b'{"events": []}', url=url))
dates = "20260801-20261015"
with plugin_scope("baseball-scoreboard"):
fetch_espn_date_chunks(session, "https://site.api.espn.com/s/scoreboard",
params={"dates": dates})
chunks = len(espn_date_chunks(*parse_espn_date_range(dates)))
chunks = len(espn_request_chunks(*parse_espn_date_range(dates)))
assert chunks > 1
assert len(session.calls) == chunks
assert _counters(global_service, plugin="baseball-scoreboard")["requests"] == chunks
+19
View File
@@ -136,3 +136,22 @@ class TestCleartextCredentialsNeedAnExplicitOptIn:
"allow_insecure_mqtt": "false"})
assert r.status_code == 400
def test_the_settings_read_reports_the_opt_in(self, client, monkeypatch):
"""The Tools form prefills its "Allow without TLS" box from the GET.
Off until someone saves it on, so an untouched form sends false and
the guard above still refuses a cleartext password.
"""
c, _ = client
monkeypatch.setattr(misc, "_mqtt_bridge_service_state",
lambda: {"installed": False, "active": False, "enabled": False})
def read():
return c.get("/api/v3/integrations/mqtt-bridge").get_json()["data"]["config"]
assert read()["allow_insecure_mqtt"] is False
r = c.put(URL, json={"mqtt_password": "hunter2", "mqtt_tls": False,
"allow_insecure_mqtt": True})
assert r.status_code == 200, r.get_json()
assert read()["allow_insecure_mqtt"] is True
+162
View File
@@ -0,0 +1,162 @@
"""Two on-demand edges seen on a rig.
* A request naming a ``*_live`` mode got HTTP 200 and a different mode on
the panel. The session's mode list kept live modes only when the plugin's
has_live_content() said so, and that is the live-priority question,
which the sports plugins answer for favourite teams only: fifteen college
games on, no favourite playing, and ``ncaa_fb_live`` became
``nfl_recent``.
* A restart during a session whose plugin then failed to load (its config
no longer validated) logged "No valid display modes found ... after
restoration" and left the session active with no modes: published as
active for a plugin that was not running, with its cached request kept
for the next restart.
"""
from unittest.mock import MagicMock
import pytest
SPORTS_MODES = ['nfl_live', 'nfl_recent', 'nfl_upcoming',
'ncaa_fb_live', 'ncaa_fb_recent', 'ncaa_fb_upcoming']
def _sports_plugin(has_live_content=False):
plugin = MagicMock(spec=['display', 'has_live_content', 'has_live_priority',
'get_live_modes'])
plugin.has_live_content.return_value = has_live_content
plugin.has_live_priority.return_value = True
plugin.get_live_modes.return_value = []
return plugin
def _register(controller, plugin_id, modes, plugin):
controller.plugin_display_modes[plugin_id] = list(modes)
for mode in modes:
controller.plugin_modes[mode] = plugin
controller.mode_to_plugin_id[mode] = plugin_id
if mode not in controller.available_modes:
controller.available_modes.append(mode)
@pytest.fixture
def football(test_display_controller):
c = test_display_controller
_register(c, 'football-scoreboard', SPORTS_MODES, _sports_plugin())
return c
class TestANamedLiveModeIsShown:
def test_it_is_the_first_screen(self, football):
football._activate_on_demand({'plugin_id': 'football-scoreboard',
'mode': 'ncaa_fb_live'})
assert football.on_demand_active
assert football.current_display_mode == 'ncaa_fb_live'
assert football.on_demand_mode == 'ncaa_fb_live'
def test_the_plugins_other_modes_follow_it(self, football):
football._activate_on_demand({'plugin_id': 'football-scoreboard',
'mode': 'ncaa_fb_live'})
assert football.on_demand_modes[0] == 'ncaa_fb_live'
assert set(football.on_demand_modes[1:]) == {
'nfl_recent', 'nfl_upcoming', 'ncaa_fb_recent', 'ncaa_fb_upcoming'}
def test_pinned_holds_it(self, football):
football._activate_on_demand({'plugin_id': 'football-scoreboard',
'mode': 'ncaa_fb_live', 'pinned': True})
assert football.on_demand_modes == ['ncaa_fb_live']
def test_a_bare_plugin_request_still_skips_quiet_live_modes(self, football):
"""Only a mode asked for by name is kept: a plugin-only request
resolves to the plugin's first mode (nfl_live), and opening on an
empty live screen there is what the ordering exists to avoid."""
football._activate_on_demand({'plugin_id': 'football-scoreboard'})
assert not any(m.endswith('_live') for m in football.on_demand_modes)
def test_a_named_second_live_mode_with_content_leads(self, test_display_controller):
"""With live content both live modes are kept, nfl_live first; a
request naming ncaa_fb_live must still open on it, not rotate away."""
c = test_display_controller
_register(c, 'football-scoreboard', SPORTS_MODES, _sports_plugin(has_live_content=True))
c._activate_on_demand({'plugin_id': 'football-scoreboard', 'mode': 'ncaa_fb_live'})
assert c.on_demand_modes[0] == 'ncaa_fb_live'
assert c.on_demand_modes.count('ncaa_fb_live') == 1
assert 'nfl_live' in c.on_demand_modes[1:]
def test_the_named_mode_survives_a_restart(self, football):
football._activate_on_demand({'plugin_id': 'football-scoreboard',
'mode': 'ncaa_fb_live'})
saved = football.cache_manager.set.call_args_list[-1]
assert saved.args[0] == 'display_on_demand_config'
config = saved.args[1]
assert config['named_mode'] == 'ncaa_fb_live'
football._reset_on_demand_fields()
football._select_startup_plugins(['football-scoreboard'], config)
football._populate_on_demand_modes_from_plugin()
assert football.on_demand_modes[football.on_demand_mode_index] == 'ncaa_fb_live'
class TestARestoreWithNothingToResume:
@pytest.fixture
def restored(self, test_display_controller):
c = test_display_controller
c.config['clock-simple'] = {'enabled': True}
c._select_startup_plugins(['clock-simple'],
{'plugin_id': 'clock-simple', 'mode': 'clock-simple'})
assert c.on_demand_active
# The plugin's load then fails: nothing is registered for it.
c.cache_manager.clear_cache.reset_mock()
c._populate_on_demand_modes_from_plugin()
return c
def test_the_session_ends(self, restored):
assert not restored.on_demand_active
assert restored.on_demand_plugin_id is None
assert not restored.on_demand_schedule_override
def test_it_is_reported_as_an_error(self, restored):
assert restored.on_demand_status == 'error'
assert restored.on_demand_last_error == 'restore-failed'
# The last on-demand state write, not the last write of any key: the
# font-usage publisher thread writes its own key at its own pace.
published = [c for c in restored.cache_manager.set.call_args_list
if c.args and c.args[0] == 'display_on_demand_state'][-1]
assert published.args[1]['status'] == 'error'
assert published.args[1]['error'] == 'restore-failed'
def test_the_cached_request_is_dropped(self, restored):
restored.cache_manager.clear_cache.assert_any_call('display_on_demand_config')
def test_a_plugin_system_failure_ends_a_cached_session_not_yet_restored(
mock_config_manager, mock_display_manager, mock_cache_manager,
test_config_with_plugins, emulator_mode):
"""Initialization can fail before the cached session is read, with
on_demand_active still False: the session must still end, visibly."""
from unittest.mock import patch
from src.display_controller import DisplayController
mock_config_manager.get_config.return_value = test_config_with_plugins
mock_config_manager.load_config.return_value = test_config_with_plugins
mock_cache_manager._memory_cache['display_on_demand_config'] = {
'plugin_id': 'clock-simple', 'mode': 'clock-simple'}
with patch('src.display_controller.ConfigManager', return_value=mock_config_manager), \
patch('src.display_controller.DisplayManager', return_value=mock_display_manager), \
patch('src.display_controller.CacheManager', return_value=mock_cache_manager), \
patch('src.display_controller.FontManager'), \
patch('src.plugin_system.PluginManager', side_effect=RuntimeError("boom")):
controller = DisplayController()
try:
assert controller.plugin_manager is None
assert not controller.on_demand_active
assert controller.on_demand_status == 'error'
assert controller.on_demand_last_error == 'restore-failed'
mock_cache_manager.clear_cache.assert_any_call('display_on_demand_config')
finally:
try:
controller.cleanup()
except Exception:
pass
+28
View File
@@ -133,6 +133,32 @@ def scenario_on_demand_restored(h: RunLoopHarness):
h.restore_on_demand("sports", mode="sports_upcoming", duration=40)
def scenario_on_demand_named_live(h: RunLoopHarness):
# Games are on until t=70, but none involves a favourite, so
# has_live_content() (the live-priority answer) stays False throughout.
# A request naming sports_live still opens on it (it opened on
# sports_recent); asked for again after the games end, it has nothing to
# draw and the session moves on to the plugin's next mode.
h.add_plugin(FakePlugin("clock", ["clock"], duration=20))
h.add_plugin(FakePlugin(
"sports", ["sports_live", "sports_recent", "sports_upcoming"], duration=15,
live_priority=True,
content=lambda t, mode: mode != "sports_live" or t < 70))
h.on_demand_request(5, "n1", plugin_id="sports", mode="sports_live")
h.on_demand_request(40, "n2", action="stop")
h.on_demand_request(100, "n3", plugin_id="sports", mode="sports_live")
h.on_demand_request(140, "n4", action="stop")
def scenario_on_demand_restore_failed(h: RunLoopHarness):
# A restart during a session whose plugin then fails to load: the
# session ends as an error before the first screen, and the rotation
# runs normally from the top.
h.add_plugin(FakePlugin("clock", ["clock"], duration=20))
h.add_plugin(FakePlugin("weather", ["weather"], duration=20))
h.restore_on_demand("gone", mode="gone", duration=40)
def scenario_schedule(h: RunLoopHarness):
# The clock starts at 22:59:30. Off from 23:01 until 23:05 (the window
# spans midnight); dimmed from 23:00 until 23:01.
@@ -194,6 +220,8 @@ SCENARIOS = {
"on_demand": (scenario_on_demand, 240),
"on_demand_pinned": (scenario_on_demand_pinned, 160),
"on_demand_restored": (scenario_on_demand_restored, 100),
"on_demand_named_live": (scenario_on_demand_named_live, 160),
"on_demand_restore_failed": (scenario_on_demand_restore_failed, 60),
"schedule": (scenario_schedule, 400),
"wifi_notice": (scenario_wifi_notice, 150),
"follower": (scenario_follower, 80),
+183
View File
@@ -0,0 +1,183 @@
"""A frame the preview throttle skipped still reaches the snapshot.
The preview snapshot (/api/v3/display/current, the web UI's live preview) is
only written from update_display(), at most once per write interval. A screen
that draws its card once and then holds it -- soccer's recent/upcoming cards
skip redundant redraws -- pushes exactly one frame. When that push lands inside
the interval, e.g. a few milliseconds after the on-demand start's clear wrote a
black frame, the throttle skips it and nothing ever writes it: on ledpi the
preview stayed black for soccer's whole 15 s screen while the panel showed the
card, and the next screen "rendered immediately".
Runs the real DisplayManager on the emulator, like test_display_dirty_tracking.
"""
import os
import sys
import types
os.environ["EMULATOR"] = "true"
import pytest
from PIL import Image
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
@pytest.fixture(scope="module")
def dm(tmp_path_factory):
from src.display_manager import DisplayManager
DisplayManager._instance = None
manager = DisplayManager({
"display": {
"hardware": {"rows": 32, "cols": 64, "chain_length": 2,
"parallel": 1, "brightness": 90},
"runtime": {"gpio_slowdown": 0},
},
}, suppress_test_pattern=True)
manager._snapshot_path = str(
tmp_path_factory.mktemp("owed_snapshot") / "led_matrix_preview.png")
yield manager
DisplayManager._instance = None
@pytest.fixture
def viewer(dm, monkeypatch, tmp_path):
"""A preview is open (1 s write interval); fresh snapshot bookkeeping."""
monkeypatch.setattr(dm, "_viewer_is_fresh", lambda now: True)
dm._viewer_was_fresh = True
dm._snapshot_path = str(tmp_path / "snap.png")
dm._last_snapshot_ts = 0.0
dm._last_snapshot_touch_ts = 0.0
dm._last_snapshot_digest = None
dm._saved_snapshot_digest = None
dm._snapshot_owed = False
dm.set_scrolling_state(False)
return dm
def _lit(path):
with Image.open(path) as img:
return sum(1 for p in img.convert("RGB").getdata() if max(p) > 20)
def _age_last_write(dm, seconds=2.0):
"""As if `seconds` had passed since the last snapshot write."""
dm._last_snapshot_ts -= seconds
dm._last_snapshot_touch_ts -= seconds
def _clear_then_draw_card(dm):
"""The on-demand start's clear, then the card a few ms later."""
dm.clear()
dm.update_display() # black frame: written
assert _lit(dm._snapshot_path) == 0
dm.draw.rectangle([4, 4, 40, 20], fill=(255, 255, 0))
dm.update_display() # the card: inside the interval
def _controller(dm):
from src import display_controller as dc_module
controller = dc_module.DisplayController.__new__(dc_module.DisplayController)
controller.plugin_manager = None
controller.display_manager = dm
return controller
class _HoldingPlugin:
"""Already showing its card: display() returns True and draws nothing."""
plugin_id = "holding"
def __init__(self):
self.calls = 0
def display(self, display_mode=None, force_clear=False):
self.calls += 1
return True
def test_a_held_card_reaches_the_preview_on_the_next_frame(viewer):
dm = viewer
_clear_then_draw_card(dm)
assert _lit(dm._snapshot_path) == 0 # the throttle skipped the card
controller = _controller(dm)
plugin = _HoldingPlugin()
_age_last_write(dm)
# The render loop's next frame: the plugin draws nothing and makes no
# update_display() call, as soccer's switch cards do.
assert controller._display_once(plugin, "soccer_eng.1_recent", True) is True
assert plugin.calls == 1
assert _lit(dm._snapshot_path) > 0
def test_the_owed_write_still_waits_out_the_interval(viewer, monkeypatch):
dm = viewer
_clear_then_draw_card(dm)
saves = []
monkeypatch.setattr(dm, "_save_snapshot", lambda image: saves.append(image))
dm.write_owed_snapshot() # still inside the interval
assert saves == []
_age_last_write(dm)
dm.write_owed_snapshot()
assert len(saves) == 1
# Written: nothing is owed, so later frames do no work and the unchanged
# frame is not encoded again.
assert dm._snapshot_owed is False
_age_last_write(dm)
dm.write_owed_snapshot()
assert len(saves) == 1
def test_a_failed_owed_write_stays_owed_and_is_retried(viewer, monkeypatch):
dm = viewer
_clear_then_draw_card(dm)
_age_last_write(dm)
attempts = []
def failing_save(image):
attempts.append(image)
raise OSError("disk full")
monkeypatch.setattr(dm, "_save_snapshot", failing_save)
dm.write_owed_snapshot() # the write fails
assert len(attempts) == 1
assert dm._snapshot_owed is True # still owed: a held screen
saves = [] # makes no update_display()
monkeypatch.setattr(dm, "_save_snapshot", lambda image: saves.append(image))
dm.write_owed_snapshot() # retried on the next frame
assert len(saves) == 1
assert dm._snapshot_owed is False
def test_nothing_owed_after_a_frame_that_was_written(viewer, monkeypatch):
dm = viewer
dm.draw.rectangle([0, 0, 8, 8], fill=(0, 255, 0))
dm.update_display() # due: written at once
assert dm._snapshot_owed is False
calls = []
monkeypatch.setattr(dm, "_write_snapshot_if_due",
lambda *a, **k: calls.append(a))
dm.write_owed_snapshot()
assert calls == []
def test_an_unchanged_frame_inside_the_interval_is_not_owed(viewer):
dm = viewer
dm.draw.rectangle([0, 0, 8, 8], fill=(0, 0, 255))
dm.update_display()
dm.update_display() # same frame, inside the interval
assert dm._snapshot_owed is False
def test_a_controller_without_the_hook_still_draws():
"""Controllers built without a display manager (tests) are unaffected."""
from src import display_controller as dc_module
controller = dc_module.DisplayController.__new__(dc_module.DisplayController)
controller.plugin_manager = None
plugin = _HoldingPlugin()
assert controller._display_once(plugin, "x", True) is True
controller.display_manager = types.SimpleNamespace()
assert controller._display_once(plugin, "x", True) is True
assert plugin.calls == 2
+11
View File
@@ -612,6 +612,12 @@ class TestEndToEnd:
cached['display_current_state'] = {'mode': 'from-cache', 'last_updated': 1}
path = str(tmp_path / 'control.sock')
monkeypatch.setenv(c.SOCKET_PATH_ENV, path)
# The display is this process, and its render loop is beating: the
# cache is then still its answer once the socket goes.
heartbeat = tmp_path / 'display-heartbeat.json'
heartbeat.write_text(json.dumps({'pid': os.getpid(), 'mono': time.monotonic(),
'wall': time.time()}))
monkeypatch.setattr(display_watchdog, 'HEARTBEAT_PATH', str(heartbeat))
hub = _hub_with_everything()
server = ControlServer(path, state_hub=hub, keepalive=0.2)
assert server.start()
@@ -638,3 +644,8 @@ class TestEndToEnd:
break
time.sleep(0.05)
assert (data['mode'], data['source']) == ('from-cache', 'cache')
# Stopped: systemd takes the heartbeat's directory with it, and the
# cache's last answer is no longer anyone's.
heartbeat.unlink()
data = _data(client, '/api/v3/display/current-status')
assert (data['mode'], data['source']) == (None, 'cache')
@@ -50,11 +50,13 @@ class FakeResult:
self.plugins_to_install = plugins_to_install or []
self.plugins_installed = []
self.plugins_failed = []
self.skipped = []
def to_dict(self):
return {
"success": self.success,
"restored": self.restored,
"skipped": self.skipped,
"errors": self.errors,
"plugins_installed": self.plugins_installed,
"plugins_failed": self.plugins_failed,
@@ -286,6 +288,109 @@ class TestPluginReinstall:
assert body["data"]["plugins_failed"][0]["error"] == "Store manager unavailable"
class TestInstalledPluginsAreNotReinstalled:
""""Reinstall missing plugins" installs only what is missing.
Every plugin the backup listed went to install_plugin, which replaces an
installed copy with a fresh download: restoring onto the same device
re-downloaded all of them inside the request. One installed from its own
URL is not in the registry, so its "reinstall" returned False and the
whole restore answered 500 "Restore failed" with the plugin still there.
"""
@staticmethod
def _installed(tmp_path, *names):
found = {}
for name in names:
(tmp_path / name).mkdir()
found[name] = tmp_path / name
return lambda plugin_id: found.get(plugin_id)
def test_an_installed_plugin_is_skipped_and_a_missing_one_installed(
self, client, restore, tmp_path):
restore.return_value = FakeResult(
plugins_to_install=[{"plugin_id": "clock"}, {"plugin_id": "weather"}])
store = api_v3.plugin_store_manager
store._existing_install.side_effect = self._installed(tmp_path, "clock")
store.install_plugin.return_value = True
response = post(client)
assert response.status_code == 200
store.install_plugin.assert_called_once_with("weather")
data = response.get_json()["data"]
assert data["plugins_installed"] == ["weather"]
assert data["plugins_failed"] == []
assert "plugin:clock (installed)" in data["skipped"]
def test_an_installed_plugin_the_store_cannot_install_is_not_a_failure(
self, client, restore, tmp_path):
restore.return_value = FakeResult(plugins_to_install=[{"plugin_id": "my-3p"}])
store = api_v3.plugin_store_manager
store._existing_install.side_effect = self._installed(tmp_path, "my-3p")
store.install_plugin.return_value = False
response = post(client)
assert response.status_code == 200
assert response.get_json()["data"]["plugins_failed"] == []
store.install_plugin.assert_not_called()
@pytest.fixture
def real_store(self, tmp_path):
from src.plugin_system.store_manager import PluginStoreManager
plugins_dir = tmp_path / "plugin-repos"
for folder, manifest_id in (("ledmatrix-weather", "ledmatrix-weather"),
("my-3p", "my-3p")):
(plugins_dir / folder).mkdir(parents=True)
(plugins_dir / folder / "manifest.json").write_text(
json.dumps({"id": manifest_id, "version": "1.0.0"}))
store = PluginStoreManager(plugins_dir=str(plugins_dir),
uninstalled_registry_path=str(tmp_path / "uninstalled.json"))
# The official weather plugin's registry id differs from the id it
# installs under; my-3p was installed from its own URL.
registry = {"plugins": [{
"id": "weather", "repo": "https://github.com/ChuckBuilds/ledmatrix-plugins",
"plugin_path": "plugins/ledmatrix-weather"}]}
store.registry_cache = registry
store.fetch_registry = lambda *a, **k: registry
store.install_plugin = MagicMock(return_value=True)
api_v3.plugin_store_manager = store
return store
def test_with_the_real_store_aliases_and_third_party_installs_count(
self, client, restore, real_store):
restore.return_value = FakeResult(plugins_to_install=[
{"plugin_id": "weather"}, {"plugin_id": "my-3p"}, {"plugin_id": "clock"}])
response = post(client)
assert response.status_code == 200
real_store.install_plugin.assert_called_once_with("clock")
skipped = response.get_json()["data"]["skipped"]
assert "plugin:weather (installed)" in skipped
assert "plugin:my-3p (installed)" in skipped
class TestFontsCatalogCache:
"""The Fonts tab's catalog is cached for 5 minutes (fonts.py).
Upload and delete clear it; a restore did not, so restored fonts were
missing from the Fonts tab and every font picker until it expired.
"""
@pytest.fixture
def cached_catalog(self):
from web_interface.cache import delete_cached, get_cached, set_cached
set_cached('fonts_catalog', {'fonts': ['5x7.bdf']}, ttl_seconds=300)
yield lambda: get_cached('fonts_catalog', ttl_seconds=300)
delete_cached('fonts_catalog')
def test_a_restore_that_restored_fonts_clears_it(self, client, restore, cached_catalog):
restore.return_value = FakeResult(restored=["config", "fonts (2)"])
assert post(client).status_code == 200
assert cached_catalog() is None
def test_a_restore_without_fonts_keeps_it(self, client, restore, cached_catalog):
restore.return_value = FakeResult(restored=["config"])
assert post(client).status_code == 200
assert cached_catalog() == {'fonts': ['5x7.bdf']}
class TestFailureReporting:
def test_restore_errors_produce_a_500(self, client, restore):
restore.return_value = FakeResult(
@@ -13,7 +13,9 @@ tmp_path so the assertions are against files on disk rather than mock
calls.
"""
import html
import json
import re
import sys
from pathlib import Path
from unittest.mock import MagicMock
@@ -221,3 +223,66 @@ class TestRawEndpointsBypassSecretSeparation:
env.client.post(MAIN, json={"weather": {"api_key": "PLAINTEXT-KEY"}})
# Nothing was moved aside into the secrets file.
assert not env.secrets_file.exists() or "PLAINTEXT-KEY" not in env.secrets_file.read_text()
class TestConfigEditorRoundTrip:
"""The Config Editor tab (/partials/raw-json) and the save it posts to.
The secrets editor is shown masked, like GET /config/secrets: the page is
served to anyone who can reach the port while the optional web login is
off. Its save strips the masks and merges onto the stored file, so a
masked editor saved back as it is changes nothing.
"""
STORED = {
"github": {"api_token": "ghp_REAL_TOKEN_1234"},
"ledmatrix-weather": {"api_key": "WEATHER_KEY_abcdef", "units_id": 42},
"calendar": {"accounts": [{"name": "home", "token": "CAL_TOKEN_9"}]},
"youtube": {"api_key": "YOUR_YOUTUBE_API_KEY", "channel_secret": ""},
}
REAL_VALUES = ("ghp_REAL_TOKEN_1234", "WEATHER_KEY_abcdef", "CAL_TOKEN_9")
@pytest.fixture
def editor(self, env, monkeypatch):
from web_interface.blueprints import pages_v3 as pages_module
env.secrets_file.write_text(json.dumps(self.STORED))
monkeypatch.setattr(pages_module.pages_v3, "config_manager",
env.config_manager, raising=False)
app = Flask(__name__, template_folder=str(project_root / "web_interface" / "templates"))
app.config["TESTING"] = True
app.register_blueprint(pages_module.pages_v3)
app.register_blueprint(api_v3, url_prefix="/api/v3")
return app.test_client()
@staticmethod
def _secrets_textarea(client):
page = client.get("/partials/raw-json")
assert page.status_code == 200
match = re.search(r'<textarea id="secrets-config-editor"[^>]*>(.*?)</textarea>',
page.get_data(as_text=True), re.S)
assert match, "the secrets editor is missing from the partial"
return html.unescape(match.group(1))
def test_the_editor_shows_no_secret_value(self, editor):
text = self._secrets_textarea(editor)
for value in self.REAL_VALUES:
assert value not in text
shown = json.loads(text)
assert shown["github"]["api_token"] == "\u2022" * 8
# Same shape as the file, and "not set" still reads as not set.
assert shown["calendar"]["accounts"][0]["name"] == "\u2022" * 8
assert shown["youtube"] == {"api_key": "YOUR_YOUTUBE_API_KEY", "channel_secret": ""}
def test_saving_it_back_unchanged_keeps_every_secret(self, editor, env):
shown = json.loads(self._secrets_textarea(editor))
response = editor.post(SECRETS, json=shown)
assert response.status_code == 200
assert json.loads(env.secrets_file.read_text()) == self.STORED
def test_editing_one_secret_changes_only_that_one(self, editor, env):
shown = json.loads(self._secrets_textarea(editor))
shown["ledmatrix-weather"]["api_key"] = "NEW_WEATHER_KEY"
assert editor.post(SECRETS, json=shown).status_code == 200
expected = json.loads(json.dumps(self.STORED))
expected["ledmatrix-weather"]["api_key"] = "NEW_WEATHER_KEY"
assert json.loads(env.secrets_file.read_text()) == expected
+13
View File
@@ -169,6 +169,10 @@ class TestGetSchemaProperty:
},
"fifa.world": {"type": "object",
"properties": {"enabled": {"type": "boolean"}}},
"cities": {"type": "array",
"items": {"type": "object",
"properties": {"timezone": {"type": "string"}}}},
"color": {"type": ["array", "null"], "items": {"type": "integer"}},
}
}
@@ -185,6 +189,15 @@ class TestGetSchemaProperty:
prop = _get_schema_property(self.SCHEMA, "fifa.world.enabled")
assert prop == {"type": "boolean"}
def test_an_index_steps_into_the_array_items(self):
# How a table row posts its cells
assert _get_schema_property(self.SCHEMA, "cities.0.timezone") == {"type": "string"}
assert _get_schema_property(self.SCHEMA, "color.2") == {"type": "integer"}
def test_a_non_index_under_an_array_is_not_found(self):
assert _get_schema_property(self.SCHEMA, "cities.timezone") is None
assert _get_schema_property(self.SCHEMA, "cities.0.nope") is None
def test_missing_path_returns_none(self):
assert _get_schema_property(self.SCHEMA, "nope.nope") is None
@@ -0,0 +1,393 @@
"""GET and POST /plugins/config against a real ConfigManager and SchemaManager.
Each class is one bug, reproduced through the endpoint the settings form and
API clients use, with assertions on config.json and config_secrets.json.
"""
import json
from unittest.mock import MagicMock
import pytest
from flask import Flask
from src.config_manager import ConfigManager
from src.plugin_system.schema_manager import SchemaManager
from web_interface.blueprints.api_v3 import api_v3
PLUGIN_ID = "demo"
OTHER_ID = "other"
SCHEMA = {
"type": "object",
"properties": {
"enabled": {"type": "boolean", "default": True},
"api_key": {"type": "string", "x-secret": True, "default": ""},
"city": {"type": "string", "default": "Austin"},
"mqtt": {
"type": "object",
"properties": {
"host": {"type": "string", "default": ""},
"port": {"type": "integer", "default": 1883,
"minimum": 1, "maximum": 65535},
"password": {"type": "string", "x-secret": True, "default": ""},
},
},
"accounts": {
"type": "array",
"default": [],
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"token": {"type": "string", "x-secret": True},
},
},
},
},
}
OTHER_SCHEMA = {
"type": "object",
"properties": {
"enabled": {"type": "boolean", "default": True},
"label": {"type": "string", "default": "x"},
},
}
STORED = {
PLUGIN_ID: {"enabled": True, "city": "Paris",
"mqtt": {"host": "broker", "port": 1883},
"accounts": [{"name": "a"}, {"name": "b"}]},
OTHER_ID: {"enabled": True, "label": "hello"},
}
STORED_SECRETS = {
PLUGIN_ID: {"api_key": "TOPSECRET",
"accounts": [{"token": "TOK-A"}, {"token": "TOK-B"}]},
}
_ATTRS = ('config_manager', 'plugin_catalog', 'plugin_store_manager',
'saved_repositories_manager', 'schema_manager',
'operation_queue', 'operation_history', 'cache_manager')
@pytest.fixture
def env(tmp_path):
config_file = tmp_path / "config.json"
secrets_file = tmp_path / "config_secrets.json"
plugins_dir = tmp_path / "plugins"
for plugin_id, schema in ((PLUGIN_ID, SCHEMA), (OTHER_ID, OTHER_SCHEMA)):
plugin_dir = plugins_dir / plugin_id
plugin_dir.mkdir(parents=True)
(plugin_dir / "config_schema.json").write_text(json.dumps(schema))
(plugin_dir / "manifest.json").write_text(json.dumps({"id": plugin_id}))
config_file.write_text(json.dumps(STORED))
secrets_file.write_text(json.dumps(STORED_SECRETS))
sentinel = object()
originals = {name: getattr(api_v3, name, sentinel) for name in _ATTRS}
config_manager = ConfigManager(config_path=str(config_file),
secrets_path=str(secrets_file))
config_manager.template_path = str(tmp_path / "no-template.json")
plugin_manager = MagicMock()
plugin_manager.plugin_manifests = {PLUGIN_ID: {"id": PLUGIN_ID},
OTHER_ID: {"id": OTHER_ID}}
plugin_manager.plugins_dir = plugins_dir
for name in _ATTRS:
setattr(api_v3, name, MagicMock())
api_v3.config_manager = config_manager
api_v3.schema_manager = SchemaManager(plugins_dir=plugins_dir, project_root=tmp_path)
api_v3.plugin_catalog = plugin_manager
api_v3.operation_queue = None
app = Flask(__name__)
app.config["TESTING"] = True
app.register_blueprint(api_v3, url_prefix="/api/v3")
class Env:
client = app.test_client()
@staticmethod
def use_schema(schema, plugin_id=PLUGIN_ID):
(plugins_dir / plugin_id / "config_schema.json").write_text(json.dumps(schema))
@staticmethod
def store(section, plugin_id=PLUGIN_ID):
main = json.loads(config_file.read_text())
main[plugin_id] = section
config_file.write_text(json.dumps(main))
@staticmethod
def main():
return json.loads(config_file.read_text())
@staticmethod
def secrets():
return json.loads(secrets_file.read_text())
@staticmethod
def post_form(data, plugin_id=PLUGIN_ID):
return Env.client.post(f"/api/v3/plugins/config?plugin_id={plugin_id}",
data=data)
@staticmethod
def post_json(config, plugin_id=PLUGIN_ID):
return Env.client.post("/api/v3/plugins/config",
json={"plugin_id": plugin_id, "config": config})
yield Env
for name, original in originals.items():
if original is sentinel:
if hasattr(api_v3, name):
delattr(api_v3, name)
else:
setattr(api_v3, name, original)
class TestARejectedSaveLeavesNothingBehind:
"""The form save edited the cached config load_config hands out, then
failed validation. The cache kept the edit, and the next save of any
other setting wrote it to config.json -- the rejected value, and a
nested secret typed into the same form in plain text."""
REJECTED = {"mqtt.host": "broker", "mqtt.port": "99999",
"mqtt.password": "hunter2", "__rendered_section": ["mqtt"]}
def test_the_rejected_values_never_reach_config_json(self, env):
assert env.post_form(self.REJECTED).status_code == 400
resp = env.post_json({"label": "bye"}, plugin_id=OTHER_ID)
assert resp.status_code == 200, resp.get_json()
main = env.main()
assert main[OTHER_ID]["label"] == "bye"
assert main[PLUGIN_ID]["mqtt"] == {"host": "broker", "port": 1883}
assert "hunter2" not in json.dumps(main)
def test_the_form_reloads_with_the_stored_values(self, env):
assert env.post_form(self.REJECTED).status_code == 400
assert api_v3.config_manager.load_config()[PLUGIN_ID]["mqtt"]["port"] == 1883
class TestGetMasksSecrets:
"""GET /plugins/config returned the section with config_secrets.json
merged in, secrets and all: the masking #276 added was lost when the
route was rewritten. The settings page and GET /config/secrets mask."""
def test_secrets_come_back_blank(self, env):
data = env.client.get(f"/api/v3/plugins/config?plugin_id={PLUGIN_ID}").get_json()["data"]
assert data["api_key"] == ""
assert data["accounts"] == [{"name": "a", "token": ""}, {"name": "b", "token": ""}]
assert data["city"] == "Paris"
def test_posting_the_response_back_keeps_every_secret(self, env):
data = env.client.get(f"/api/v3/plugins/config?plugin_id={PLUGIN_ID}").get_json()["data"]
resp = env.post_json(data)
assert resp.status_code == 200, resp.get_json()
assert env.secrets()[PLUGIN_ID] == STORED_SECRETS[PLUGIN_ID]
assert "TOPSECRET" not in json.dumps(env.main())
def test_the_settings_form_posting_masked_fields_keeps_every_secret(self, env):
# The page renders secrets blank (pages_v3 masks the same way)
resp = env.post_form({
"api_key": "", "city": "Lyon", "mqtt.host": "broker", "mqtt.port": "1883",
"mqtt.password": "", "__rendered_section": ["api_key", "city", "mqtt"]})
assert resp.status_code == 200, resp.get_json()
assert env.secrets()[PLUGIN_ID] == STORED_SECRETS[PLUGIN_ID]
assert env.main()[PLUGIN_ID]["city"] == "Lyon"
def test_a_plugin_without_a_schema_has_credential_named_fields_blanked(self, env, tmp_path):
(tmp_path / "plugins" / "bare").mkdir()
env.store({"enabled": True, "station": "KAUS"}, plugin_id="bare")
secrets = env.secrets()
secrets["bare"] = {"api_token": "BARE-TOKEN"}
(tmp_path / "config_secrets.json").write_text(json.dumps(secrets))
data = env.client.get("/api/v3/plugins/config?plugin_id=bare").get_json()["data"]
assert data["api_token"] == ""
assert data["station"] == "KAUS"
@pytest.mark.parametrize("section", ["web_auth", "github", "display"])
def test_a_core_section_is_refused(self, env, tmp_path, section):
secrets = env.secrets()
secrets["web_auth"] = {"cookie_secret": "COOKIE-KEY", "password_hash": "HASH"}
secrets["github"] = {"api_token": "ghp_TOKEN"}
(tmp_path / "config_secrets.json").write_text(json.dumps(secrets))
env.store({"hardware": {"rows": 32}}, plugin_id="display")
resp = env.client.get(f"/api/v3/plugins/config?plugin_id={section}")
assert resp.status_code == 400
body = resp.get_data(as_text=True)
assert "COOKIE-KEY" not in body and "ghp_TOKEN" not in body
ROWS_SCHEMA = {
"type": "object",
"properties": {
"enabled": {"type": "boolean", "default": True},
"cities": {
"type": "array",
"x-widget": "array-table",
"default": [],
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"timezone": {"type": "string"},
"lat": {"type": "number"},
"show": {"type": "boolean", "default": True},
},
"required": ["name", "lat"],
},
},
},
}
class TestArrayRowCellsFollowTheItemSchema:
"""A table row posts its cells as ``cities.0.timezone``. The schema
lookup stopped at the array, so each cell was parsed blind: a blank
optional text cell became null and a text cell holding digits became a
number, and either failed validation -- every save of the page, for as
long as the row existed (geochron's city without a timezone, a countdown
named "2027")."""
ROW = {"cities.0.name": "Tokyo", "cities.0.timezone": "Asia/Tokyo",
"cities.0.lat": "35.68", "cities.0.show": "true",
"__rendered_section": ["cities"]}
@pytest.fixture(autouse=True)
def _rows(self, env):
env.use_schema(ROWS_SCHEMA)
env.store({"enabled": True, "cities": [
{"name": "Tokyo", "timezone": "Asia/Tokyo", "lat": 35.68, "show": True}]})
def test_a_blank_optional_text_cell_saves(self, env):
resp = env.post_form({**self.ROW, "cities.0.timezone": ""})
assert resp.status_code == 200, resp.get_json()
assert env.main()[PLUGIN_ID]["cities"][0]["timezone"] == ""
def test_a_text_cell_of_digits_stays_text(self, env):
resp = env.post_form({**self.ROW, "cities.0.name": "2027"})
assert resp.status_code == 200, resp.get_json()
assert env.main()[PLUGIN_ID]["cities"][0]["name"] == "2027"
def test_number_and_boolean_cells_still_convert(self, env):
resp = env.post_form({**self.ROW, "cities.0.show": "false"})
assert resp.status_code == 200, resp.get_json()
assert env.main()[PLUGIN_ID]["cities"] == [
{"name": "Tokyo", "timezone": "Asia/Tokyo", "lat": 35.68, "show": False}]
class TestMaskedSecretCellsInARow:
"""The same lookup: a row's secret cell, rendered blank, came back as
null and failed validation, so a plugin with secrets in a list could not
be saved from its settings page at all."""
def test_the_stored_tokens_survive_a_save_of_the_form(self, env):
resp = env.post_form({
"city": "Lyon", "accounts.0.name": "a", "accounts.0.token": "",
"accounts.1.name": "b", "accounts.1.token": "",
"__rendered_section": ["city", "accounts"]})
assert resp.status_code == 200, resp.get_json()
assert env.secrets()[PLUGIN_ID] == STORED_SECRETS[PLUGIN_ID]
assert env.main()[PLUGIN_ID]["accounts"] == [{"name": "a"}, {"name": "b"}]
class TestABlankSecretIsLeftAsStored:
"""The form renders a secret blank and posts the blank back. For a
required secret with no default (youtube-stats' api_key) the blank was
read as null, failed validation, and blocked every save of the page
until the key was typed in again."""
@pytest.fixture(autouse=True)
def _required_secret(self, env):
schema = json.loads(json.dumps(SCHEMA))
del schema["properties"]["api_key"]["default"]
schema["required"] = ["api_key"]
env.use_schema(schema)
def test_saving_other_settings_keeps_the_stored_secret(self, env):
resp = env.post_form({"api_key": "", "city": "Lyon",
"__rendered_section": ["api_key", "city"]})
assert resp.status_code == 200, resp.get_json()
assert env.main()[PLUGIN_ID]["city"] == "Lyon"
assert env.secrets()[PLUGIN_ID]["api_key"] == "TOPSECRET"
def test_a_new_secret_is_still_saved(self, env):
resp = env.post_form({"api_key": "NEW-KEY", "city": "Lyon",
"__rendered_section": ["api_key", "city"]})
assert resp.status_code == 200, resp.get_json()
assert env.secrets()[PLUGIN_ID]["api_key"] == "NEW-KEY"
def test_a_changed_secret_then_left_blank_stays_changed(self, env):
# The second save must not write back what the first one's load
# had merged in (the old key)
env.post_form({"api_key": "NEW-KEY", "__rendered_section": ["api_key"]})
resp = env.post_form({"api_key": "", "city": "Nice",
"__rendered_section": ["api_key", "city"]})
assert resp.status_code == 200, resp.get_json()
assert env.secrets()[PLUGIN_ID]["api_key"] == "NEW-KEY"
def test_a_blank_list_secret_is_left_as_stored_too(self, env, tmp_path):
schema = json.loads(json.dumps(SCHEMA))
schema["properties"]["tokens"] = {"type": "array", "x-secret": True,
"items": {"type": "string"}, "default": []}
env.use_schema(schema)
secrets = env.secrets()
secrets[PLUGIN_ID]["tokens"] = ["t1", "t2"]
(tmp_path / "config_secrets.json").write_text(json.dumps(secrets))
resp = env.post_form({"tokens": "", "city": "Lyon",
"__rendered_section": ["tokens", "city"]})
assert resp.status_code == 200, resp.get_json()
assert env.secrets()[PLUGIN_ID]["tokens"] == ["t1", "t2"]
class TestSaveRefusesWhatIsNotAPluginId:
"""GET and reset refuse a core section or a malformed id; the save took
any of them. ``{"plugin_id": "display"}`` merged unvalidated values into
the core display section, and an id that was not a string raised a
TypeError, answered as a 500."""
def test_a_core_section_is_refused_and_left_alone(self, env):
env.store({"hardware": {"rows": 32}}, plugin_id="display")
resp = env.post_json({"hardware": {"rows": "banana"}}, plugin_id="display")
assert resp.status_code == 400
assert env.main()["display"] == {"hardware": {"rows": 32}}
def test_the_form_save_refuses_one_too(self, env):
resp = env.post_form({"password_hash": "x"}, plugin_id="web_auth")
assert resp.status_code == 400
assert "web_auth" not in env.main()
@pytest.mark.parametrize("plugin_id", [["demo"], {"id": "demo"}, 7, "", "../demo"])
def test_a_malformed_id_is_a_400(self, env, plugin_id):
resp = env.post_json({"city": "Lyon"}, plugin_id=plugin_id)
assert resp.status_code == 400
class TestTextFieldsKeepWhatWasTyped:
"""A text field holding "true", "False", "[1, 2]" or "{}" was converted
to a boolean, list or object before the schema's type was consulted, and
the save then failed validation for a perfectly good string."""
@pytest.mark.parametrize("typed", ["true", "False", "[1, 2]", "{}", "42"])
def test_a_text_field(self, env, typed):
resp = env.post_form({"city": typed, "__rendered_section": ["city"]})
assert resp.status_code == 200, resp.get_json()
assert env.main()[PLUGIN_ID]["city"] == typed
def test_a_nullable_text_field(self, env):
schema = json.loads(json.dumps(SCHEMA))
schema["properties"]["nickname"] = {"type": ["string", "null"], "default": None}
env.use_schema(schema)
resp = env.post_form({"nickname": "false", "__rendered_section": ["nickname"]})
assert resp.status_code == 200, resp.get_json()
assert env.main()[PLUGIN_ID]["nickname"] == "false"
def test_other_types_still_convert(self, env):
resp = env.post_form({"mqtt.host": "true", "mqtt.port": "8883",
"__rendered_section": ["mqtt"]})
assert resp.status_code == 200, resp.get_json()
assert env.main()[PLUGIN_ID]["mqtt"] == {"host": "true", "port": 8883}
@@ -221,7 +221,9 @@ class TestTheWebProcessNeverRunsAPlugin:
def test_saving_its_section_through_the_main_config(self, web):
body = web.post("/api/v3/config/main", {PLUGIN_ID: {"message": "via main"}})
assert web.stored()["message"] == "via main"
assert body["restart_required"] is True
# The display's config watcher hands the section to the running
# plugin (on_config_change), as for /plugins/config: no restart.
assert body["restart_required"] is False
assert web.ran() == []
def test_resetting_its_config(self, web):
@@ -957,6 +957,19 @@ def _get_schema_property(schema, key_path):
i = j
matched = True
break
# Through an array to its items: a table row posts its cells
# as "cities.0.timezone", where the index names no property.
# Stopping here left each cell parsed with no schema at all,
# so a blank text cell became null and "2027" a number.
items = prop.get('items') if _schema_type_is(prop, 'array') else None
if isinstance(items, dict) and parts[j].isdigit():
if j + 1 == len(parts):
return items
if 'properties' in items:
current = items['properties']
i = j + 1
matched = True
break
# Matched a non-object before consuming the path — can't go deeper.
return None
if not matched:
@@ -1040,6 +1053,16 @@ def _parse_form_value_with_schema(value, key_path, schema):
# Handle None/empty values
if value is None or (isinstance(value, str) and value.strip() == ''):
# The form draws a stored secret blank, so a blank secret means
# "unchanged", and "" is what the save drops as unchanged
# (remove_empty_secrets). A required one with no default fell
# through to None below, failed validation, and blocked every save
# of the page until the secret was typed in again. Not _SKIP_FIELD:
# that keeps the merged value from load_config(), which the save
# would then write back to config_secrets.json. Text secrets only:
# a list or object one gets its empty value below, dropped the same.
if prop and prop.get('x-secret') and prop.get('type', 'string') == 'string':
return ""
# A nullable field left blank means null, not an empty container.
# This is the inherit sentinel for per-mode style overrides: an
# empty list there would read as "the user chose no colour" rather
@@ -1074,6 +1097,14 @@ def _parse_form_value_with_schema(value, key_path, schema):
if isinstance(value, str):
stripped = value.strip()
# A text field keeps what was typed. The guesses below ran first, so
# "true", "False", "[1, 2]" or "{}" in a text field became a boolean,
# list or object, and the save failed validation for a good string.
declared = prop.get('type') if isinstance(prop, dict) else None
if declared == 'string' or (isinstance(declared, list) and
[t for t in declared if t != 'null'] == ['string']):
return value
# Check for boolean strings
if stripped.lower() == 'true':
return True
+25
View File
@@ -16,6 +16,7 @@ import web_interface.blueprints.api_v3 as _pkg
# as module attributes, and a value binding would not see the patch.
# Several are also called from helpers that live in __init__, so the
# package is the only patch point that covers every caller.
from web_interface.cache import delete_cached
@api_v3.route('/backup/preview', methods=['GET'])
@@ -85,6 +86,17 @@ _RESTORE_OPTION_KEYS = frozenset((
'restore_config', 'restore_secrets', 'restore_wifi', 'restore_fonts',
'restore_plugin_uploads', 'reinstall_plugins',
))
def _installed_path(psm, plugin_id):
"""Where the store finds ``plugin_id`` installed, or None.
The same lookup install_plugin makes to decide that a copy exists: the
id, or an id the registry proves is the same plugin (``aliases``, the
``plugin_path`` name), never a bare ``ledmatrix-<id>`` folder.
"""
found = psm._existing_install(plugin_id)
return found if isinstance(found, Path) and found.exists() else None
@api_v3.route('/backup/restore', methods=['POST'])
def backup_restore():
"""Restore a backup ZIP with optional RestoreOptions."""
@@ -134,6 +146,10 @@ def backup_restore():
os.unlink(tmp_path)
except OSError:
pass
# Restored fonts reach the Fonts tab through a catalog cached for five
# minutes (fonts.py); upload and delete clear it, and so must this.
if any(str(item).startswith('fonts') for item in result.restored):
delete_cached('fonts_catalog')
# Reinstall plugins if requested and store manager available
if options.reinstall_plugins and result.plugins_to_install:
@@ -143,6 +159,15 @@ def backup_restore():
if not pid:
continue
try:
# Only what is missing. install_plugin replaces an installed
# copy with a fresh download, so restoring onto the same
# device re-downloaded every plugin, and one installed from
# its own URL (not in the registry) "failed" and failed the
# whole restore while it sat there installed. The store's
# own lookup, so registry aliases count as installed too.
if psm and _installed_path(psm, pid) is not None:
result.skipped.append(f'plugin:{pid} (installed)')
continue
if psm and hasattr(psm, 'install_plugin'):
ok = psm.install_plugin(pid)
if ok:
+89 -12
View File
@@ -17,6 +17,8 @@ from src.pi5_matrix_support import is_raspberry_pi_5
from web_interface.cache import invalidate_cache
from web_interface.auth import SECTION as _WEB_AUTH_SECTION, strip_auth_section
import web_interface.blueprints.api_v3 as _pkg
import copy
from typing import Any, Dict, Iterable, Tuple
# Read through the module rather than bound by value: tests patch these
# as module attributes, and a value binding would not see the patch.
@@ -33,6 +35,50 @@ FORM_SECTION_FIELD = '__form_section'
GENERAL_FIELDS = ('timezone', 'city', 'state', 'country', 'web_display_autostart',
'plugins_directory', 'auto_update_enabled', 'auto_update_channel')
#: Settings in config.json the running display applies without a restart,
#: as key paths (a path covers everything under it). Brightness goes over the
#: control socket (brightness.set) and the config watcher's refresh
#: (DisplayController._refresh_config_cache); the per-mode durations are read
#: from the live config each time a mode starts (_get_display_duration).
#: Plugin sections are live as well (each plugin's on_config_change), and
#: save_main_config adds the ones a request saves.
LIVE_CONFIG_PATHS: Tuple[Tuple[str, ...], ...] = (
('display', 'hardware', 'brightness'),
('display', 'display_durations'),
)
_MISSING = object()
def _config_leaves(config: Any, prefix: Tuple[str, ...] = ()) -> Dict[Tuple[str, ...], Any]:
"""Every non-dict value in ``config``, by key path. An empty dict has none,
so a section created empty on the way to a field is not a change."""
if not isinstance(config, dict):
return {prefix: config}
leaves: Dict[Tuple[str, ...], Any] = {}
for key, value in config.items():
leaves.update(_config_leaves(value, prefix + (str(key),)))
return leaves
def restart_needed(before: Dict[str, Any], after: Dict[str, Any],
live_paths: Iterable[Tuple[str, ...]] = LIVE_CONFIG_PATHS) -> bool:
"""Does going from config ``before`` to ``after`` need a display restart?
True when anything changed outside ``live_paths``. A save that changes
only live settings, or nothing at all, does not.
"""
live = tuple(live_paths)
old, new = _config_leaves(before), _config_leaves(after)
for path in set(old) | set(new):
if old.get(path, _MISSING) == new.get(path, _MISSING):
continue
if not any(path[:len(prefix)] == prefix for prefix in live):
return True
return False
#: Top-level fields save_main_config stores somewhere of its own (location,
#: plugin_system, ...), never as a config key of the same name.
_MAPPED_TOP_LEVEL_FIELDS = GENERAL_FIELDS + (
@@ -72,6 +118,22 @@ def _day_setting(data, day, flat_key, nested_key):
return False, None
def _disabled_day_times(data, day, start_key, end_key):
"""The times posted for a day that is off, the valid ones.
Nothing reads them while the day is off, but the schedule picker posts
them and GET returns them, so keeping them means turning the day back on
finds what was there. An invalid one is dropped rather than refused, for
the same reason.
"""
times = {}
for field, key in (('start_time', start_key), ('end_time', end_key)):
value = _day_setting(data, day, key, field)[1]
if value and _validate_time_format(value)[0]:
times[field] = value
return times
@api_v3.route('/config/main', methods=['GET'])
def get_main_config():
"""Get main configuration, with credentials redacted."""
@@ -257,11 +319,16 @@ def save_schedule_config():
day_config['start_time'] = start_time
day_config['end_time'] = end_time
else:
day_config.update(_disabled_day_times(data, day, start_key, end_key))
schedule_config['days'][day] = day_config
# Validate that at least one day is enabled in per-day mode
if enabled_days_count == 0:
# An enabled per-day schedule needs a day to be on. A disabled
# one does not: every day off with the schedule off is what
# config.template.json ships, so refusing it meant a fresh
# install could not post back the schedule GET returned.
if enabled_days_count == 0 and enabled_value:
return error_response(
ErrorCode.VALIDATION_ERROR,
"At least one day must be enabled in per-day schedule mode",
@@ -465,11 +532,13 @@ def save_dim_schedule_config():
day_config['start_time'] = start_time
day_config['end_time'] = end_time
else:
day_config.update(_disabled_day_times(data, day, start_key, end_key))
dim_schedule_config['days'][day] = day_config
# Validate that at least one day is enabled in per-day mode
if enabled_days_count == 0:
# As for the on/off schedule: only an enabled one needs a day on.
if enabled_days_count == 0 and enabled_value:
return error_response(
ErrorCode.VALIDATION_ERROR,
"At least one day must be enabled in per-day dim schedule mode",
@@ -507,7 +576,11 @@ def save_main_config():
# Try to get JSON data first, fallback to form data
data = None
if request.is_json:
data = request.get_json()
# silent=True, as in save_raw_main_config: get_json() raised
# Werkzeug's BadRequest into the catch-all below, a 500.
data = request.get_json(silent=True)
if data is None and request.get_data():
return jsonify({'status': 'error', 'message': 'Invalid JSON in request body'}), 400
if data is not None and not isinstance(data, dict):
return jsonify({'status': 'error', 'message': 'Request body must be a JSON object'}), 400
else:
@@ -553,6 +626,8 @@ def save_main_config():
# Merge with existing config (similar to original implementation)
current_config = api_v3.config_manager.load_config()
# What was stored, to tell which settings this save changed.
stored_config = copy.deepcopy(current_config)
was_auto_update_enabled = bool((current_config.get('auto_update') or {}).get('enabled'))
is_general_update = any(k in data for k in GENERAL_FIELDS)
@@ -1190,13 +1265,15 @@ def save_main_config():
message = f'{message}. {note}'
except Exception:
logger.warning("Automatic update setup could not be started", exc_info=True)
# Display hardware, rotation/durations and general settings take
# effect after a display restart; the UI shows its restart banner on
# this flag.
extra = {'restart_required': True}
# Brightness is the exception: the display applies a saved one
# without a restart. Over the control socket it lands at once,
# instead of when the config watcher next looks (up to ~2 s).
# Display hardware, rotation order and general settings take effect
# after a display restart; the UI shows its restart banner on this
# flag. Brightness, mode durations and plugin settings are applied by
# the running display (LIVE_CONFIG_PATHS), so a save that changed
# only those -- or nothing -- does not ask for one.
live_paths = LIVE_CONFIG_PATHS + tuple((plugin_id,) for plugin_id in plugin_keys_to_remove)
extra = {'restart_required': restart_needed(stored_config, current_config, live_paths)}
# Over the control socket a saved brightness lands at once, instead
# of when the config watcher next looks (up to ~2 s).
if 'brightness' in data:
saved = (current_config.get('display', {}).get('hardware', {}) or {}).get('brightness')
if isinstance(saved, int) and not isinstance(saved, bool):
+60 -7
View File
@@ -69,6 +69,26 @@ def _deliver_on_demand(payload):
return 'mailbox', reason
def _withdraw_on_demand(request_id):
"""Take a start request the route has refused back out of the mailbox.
The display reads the mailbox for an hour without looking at a
request's age, so one left there after an error answer ran whenever the
display next started. Only this request is removed: the mailbox is
re-read and cleared only while it still holds this request_id, as the
display's _consume_on_demand_request does, so a newer request posted in
the meantime stays for the display to take.
"""
cache = _cache_manager()
try:
current = cache.get('display_on_demand_request', max_age=3600, memory_ttl=0)
if isinstance(current, dict) and current.get('request_id') == request_id:
cache.delete('display_on_demand_request')
except Exception: # the route is answering an error already
logger.warning("Could not withdraw on-demand request %s from the mailbox",
request_id, exc_info=True)
@api_v3.route('/display/current', methods=['GET'])
def get_display_current():
"""The latest display preview, as the /stream/display SSE stream sends it.
@@ -259,9 +279,26 @@ def start_on_demand_display():
}
transport, socket_error = _deliver_on_demand(request_payload)
# A socket acknowledgement is the display itself answering: it is
# running and has the request queued, whatever systemd says (a display
# run by hand or in the emulator has no active unit). So nothing is
# checked or started for it -- that answered "not running" for a request
# that had already taken effect. The service is still reported the way
# _ensure_display_service_running reports a running one.
if transport == 'socket':
service_result = (dict(_get_display_service_status(), started=False)
if start_service else None)
return _on_demand_started(request_id, resolved_plugin, resolved_mode,
duration, pinned, service_result, transport,
socket_error)
service_status = _get_display_service_status()
if not service_status.get('active') and not start_service:
# The request is in the mailbox, and the display reads it whenever
# it next starts: taken back out, or a request answered with this
# error ran later anyway.
_withdraw_on_demand(request_id)
return jsonify({
'status': 'error',
'message': 'Display service is not running. Please start the display service or enable "Start Service" option.',
@@ -285,16 +322,25 @@ def start_on_demand_display():
service_result = _ensure_display_service_running()
# Check if service actually started
if service_result and not service_result.get('active'):
_withdraw_on_demand(request_id)
return jsonify({
'status': 'error',
'message': 'Failed to start display service. Please check service logs or start it manually.',
'service_result': service_result
}), 500
return _on_demand_started(request_id, resolved_plugin, resolved_mode,
duration, pinned, service_result, transport,
socket_error)
def _on_demand_started(request_id, plugin_id, mode, duration, pinned,
service_result, transport, socket_error):
"""The success answer of /display/on-demand/start."""
response_data = {
'request_id': request_id,
'plugin_id': resolved_plugin,
'mode': resolved_mode,
'plugin_id': plugin_id,
'mode': mode,
'duration': duration,
'pinned': pinned,
'service': service_result,
@@ -340,15 +386,22 @@ def get_current_display_status():
Read from the display's state stream over the control socket when it is
available (``source: "socket"``). Otherwise from what the display
publishes to the shared cache (display_controller._publish_current_mode_state)
when the active mode changes (``source: "cache"``).
when the active mode changes (``source: "cache"``). Unknown (every field
None) when the socket and the heartbeat both say the display is gone
(display_state.display_gone).
"""
state = display_state.current_status(display_state.read_state())
snapshot = display_state.read_state()
state = display_state.current_status(snapshot)
source = 'socket'
if state is None:
source = 'cache'
cache = _cache_manager()
# memory_ttl=0: written by the display service; see get_on_demand_status.
state = cache.get('display_current_state', max_age=120, memory_ttl=0)
# A stopped display leaves its last answer in the cache, where it
# read as on (is_display_active: true) for the 120 s max_age. With
# no socket and no live heartbeat there is no display behind it.
if not display_state.display_gone(snapshot):
cache = _cache_manager()
# memory_ttl=0: written by the display service; see get_on_demand_status.
state = cache.get('display_current_state', max_age=120, memory_ttl=0)
if state is None:
state = {
'mode': None,
+21
View File
@@ -102,6 +102,7 @@ def get_health():
# the only signal, as it always was.
# The display reports the same beat's age over the control socket's
# state stream, measured in memory; the file is the fallback.
snapshot = None
try:
snapshot = display_state.read_state()
if snapshot is not None:
@@ -132,6 +133,26 @@ def get_health():
'error': 'see logs for details'
}
# A stopped display service. The heartbeat's absence alone says
# nothing (the dev server, the emulator and Windows write none), so
# the overall status stayed "healthy" with the display down until the
# last preview frame it left aged past 60 s (hardware: stale). Together
# the three signals are definite: systemd says the service is not
# active, the control socket does not answer, and there is no live
# heartbeat (display_state.display_gone, which is never true where
# the platform has no socket or it is switched off).
try:
if (not display_service_status.get('active')
and display_state.display_gone(snapshot)):
health_status['checks']['display_loop'] = {
'status': 'stopped',
'note': 'The display service is not running',
'source': 'service',
}
except Exception:
logger.warning("Health check could not tell whether the display is stopped",
exc_info=True)
# Check hardware connectivity (if display manager available)
try:
snapshot_path = display_preview.SNAPSHOT_PATH
@@ -3,8 +3,12 @@
Routes decorate the shared `api_v3` Blueprint from the package `__init__`,
so their endpoint names do not depend on which module they live in.
"""
import mimetypes
from flask import send_file
from web_interface.blueprints.api_v3 import (
PROJECT_ROOT, Response, _plugin_directory, api_v3, datetime, hashlib,
PROJECT_ROOT, _plugin_directory, api_v3, datetime, hashlib,
json, jsonify, logger, os, request, uuid,
)
from src.common.path_safety import (
@@ -231,8 +235,8 @@ def serve_plugin_static(plugin_id, file_path):
if not requested_file.exists() or not requested_file.is_file():
return jsonify({'status': 'error', 'message': 'File not found'}), 404
# Determine content type
content_type = 'text/plain'
# Determine content type. Text keeps the types this route always set;
# anything else (an icon, a preview image) gets its own.
name = requested_file.name
if name.endswith('.html'):
content_type = 'text/html'
@@ -242,12 +246,14 @@ def serve_plugin_static(plugin_id, file_path):
content_type = 'text/css'
elif name.endswith('.json'):
content_type = 'application/json'
else:
guessed = mimetypes.guess_type(name)[0]
content_type = ('text/plain' if not guessed or guessed.startswith('text/')
else guessed)
# Read and return file
with open(requested_file, 'r', encoding='utf-8') as f:
content = f.read()
return Response(content, mimetype=content_type)
# Sent as bytes. Opening it as UTF-8 text failed to decode any binary
# file, so an image answered 500 UnicodeDecodeError.
return send_file(requested_file, mimetype=content_type)
@api_v3.route('/plugins/assets/delete', methods=['POST'])
@@ -9,13 +9,15 @@ from web_interface.blueprints.api_v3 import (
_enhance_schema_with_core_properties, _non_plugin_id_error,
_filter_config_by_schema, _get_schema_property,
_hidden_array_item_property, _plugin_directory,
_parse_form_value_with_schema, _schema_allows_null, _schema_type_is,
_set_missing_booleans_to_false, _set_nested_value, api_v3, datetime,
deep_merge, error_response, exception_error_response, find_secret_fields,
json, jsonify, logger, merge_secrets, os, remove_empty_secrets, request,
separate_secrets, success_response, validate_request_json,
_parse_form_value_with_schema, _redact_credentials, _schema_allows_null,
_schema_type_is, _set_missing_booleans_to_false, _set_nested_value, api_v3,
datetime, deep_merge, error_response, exception_error_response,
find_secret_fields, json, jsonify, logger, merge_secrets, os,
remove_empty_secrets, request, separate_secrets, success_response,
validate_request_json,
)
from src.web_interface.config_arrays import coerce_array_shapes
from src.web_interface.secret_helpers import mask_secret_fields
from src.web_interface.validators import dedup_unique_arrays
import web_interface.blueprints.api_v3 as _pkg
# Read through the module rather than bound by value: tests patch these
@@ -43,6 +45,12 @@ def get_plugin_config():
context={'missing_params': ['plugin_id']},
status_code=400
)
# load_config() merges config_secrets.json in, core sections
# included: ?plugin_id=web_auth returned the login's cookie key and
# password hash, and ?plugin_id=github the Plugin Store token.
id_error = _non_plugin_id_error(plugin_id)
if id_error:
return id_error
# Get plugin configuration from config manager
main_config = api_v3.config_manager.load_config()
@@ -52,12 +60,13 @@ def get_plugin_config():
# missing fields, reading legacy booleans as objects first: what the
# plugin runs with, and what posts back through the JSON save
schema_mgr = api_v3.schema_manager
schema = None
if schema_mgr:
try:
from src.plugin_system.schema_manager import prepare_plugin_config
schema = schema_mgr.load_schema(plugin_id, use_cache=True)
defaults = schema_mgr.generate_default_config(plugin_id, use_cache=True)
plugin_config = prepare_plugin_config(
plugin_config, schema_mgr.load_schema(plugin_id, use_cache=True), defaults)
plugin_config = prepare_plugin_config(plugin_config, schema, defaults)
except Exception as e:
# Log but don't fail - defaults merge is best effort
logger.warning("Could not merge defaults for %s: %s", plugin_id, e)
@@ -158,6 +167,17 @@ def get_plugin_config():
'display_duration': 30
}
# Secrets go out blank, as the settings page renders them (#276 added
# this; #330 dropped it). Blank, not the bullets GET /config/secrets
# uses: the save reads a blank secret as "unchanged", so this
# response posts back without erasing one.
properties = schema.get('properties') if isinstance(schema, dict) else None
if isinstance(properties, dict):
plugin_config = mask_secret_fields(plugin_config, properties)
else:
# No schema to mark them: blank whatever is named like one
plugin_config = _redact_credentials(plugin_config)
return success_response(data=plugin_config)
except Exception as e:
return exception_error_response(e, ErrorCode.CONFIG_LOAD_FAILED)
@@ -183,6 +203,12 @@ def save_plugin_config():
if error:
return error
plugin_id = data['plugin_id']
# As reset and uninstall do: {"plugin_id": "display"} merged
# unvalidated values into the core display section, and an id
# that was not a string raised a TypeError, answered as a 500.
id_error = _non_plugin_id_error(plugin_id)
if id_error:
return id_error
submitted_config = data.get('config', {})
if not isinstance(submitted_config, dict):
return error_response(
@@ -201,6 +227,9 @@ def save_plugin_config():
'plugin_id required in query string',
status_code=400
)
id_error = _non_plugin_id_error(plugin_id)
if id_error:
return id_error
# Load existing config as base (partial form updates should merge, not replace)
existing_config = {}
+29 -10
View File
@@ -81,6 +81,27 @@ def _listed_plugin_dir(base: Path, name: str) -> Optional[Path]:
return None
def _enqueue_or_conflict(operation_type, plugin_id, callback):
"""``(operation_id, None)``, or ``(None, a 409 response)``.
The queue raises ValueError when the plugin already has an operation
waiting or running -- a double-clicked Install, an uninstall during an
install. That is the caller's timing, not a server fault: it reached
the client as a 500, and the uninstall route recorded a failed
uninstall that had never started.
"""
try:
return api_v3.operation_queue.enqueue_operation(
operation_type, plugin_id, operation_callback=callback), None
except ValueError:
return None, error_response(
ErrorCode.PLUGIN_OPERATION_CONFLICT,
f'Plugin {plugin_id} already has an install, update or uninstall '
'in progress; wait for it to finish, then try again',
status_code=409
)
@api_v3.route('/plugins/update', methods=['POST'])
def update_plugin():
"""Update plugin"""
@@ -402,11 +423,10 @@ def uninstall_plugin():
preserve_config=preserve_config)}
# Enqueue operation
operation_id = api_v3.operation_queue.enqueue_operation(
OperationType.UNINSTALL,
plugin_id,
operation_callback=uninstall_callback
)
operation_id, conflict = _enqueue_or_conflict(
OperationType.UNINSTALL, plugin_id, uninstall_callback)
if conflict:
return conflict
return success_response(
data={'operation_id': operation_id},
@@ -538,11 +558,10 @@ def install_plugin():
raise Exception(error_msg)
# Enqueue operation
operation_id = api_v3.operation_queue.enqueue_operation(
OperationType.INSTALL,
plugin_id,
operation_callback=install_callback
)
operation_id, conflict = _enqueue_or_conflict(
OperationType.INSTALL, plugin_id, install_callback)
if conflict:
return conflict
branch_msg = f" (branch: {branch})" if branch else ""
return success_response(
+8 -1
View File
@@ -439,6 +439,10 @@ sys.exit(proc.returncode)
import tempfile
import json as json_lib
# The params reach the wrapper on its stdin, never in
# its source: written there as `params = <JSON>`, a
# true, false or null was an undefined name and the
# wrapper died with a NameError before the script ran.
params_json = json_lib.dumps(action_params)
with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as wrapper:
wrapper.write(f'''import sys
@@ -449,6 +453,9 @@ import json
# Set LEDMATRIX_ROOT
os.environ['LEDMATRIX_ROOT'] = r"{PROJECT_ROOT}"
# The params, as JSON on this wrapper's own stdin
params = json.loads(sys.stdin.read())
# Run the script and provide params as JSON via stdin
proc = subprocess.Popen(
[sys.executable, r"{script_file}"],
@@ -460,7 +467,6 @@ proc = subprocess.Popen(
)
# Send params as JSON to stdin
params = {params_json}
stdout, _ = proc.communicate(input=json.dumps(params), timeout=120)
print(stdout)
sys.exit(proc.returncode)
@@ -470,6 +476,7 @@ sys.exit(proc.returncode)
try:
result = subprocess.run(
['python3', wrapper_path],
input=params_json,
capture_output=True,
text=True,
timeout=120,
+8 -3
View File
@@ -11,7 +11,7 @@ _SAFE_PLUGIN_ID_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}$')
_SAFE_WEB_UI_FILE_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}\.html$')
_SAFE_WIDGET_NAME_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}$')
_SAFE_WIDGET_SCRIPT_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}\.js$')
from src.web_interface.secret_helpers import mask_secret_fields
from src.web_interface.secret_helpers import mask_all_secret_values, mask_secret_fields
from src.plugin_system.schema_manager import plugin_config_defaults, prepare_plugin_config
from src.common.path_safety import resolve_under, safe_path_component
from src.pi5_matrix_support import is_raspberry_pi_5
@@ -623,9 +623,14 @@ def _load_raw_json_partial():
main_config_data = pages_v3.config_manager.get_raw_file_content('main')
# The web login section (password and token hashes) is managed in
# General > Security, never in this editor; its save keeps it.
# The rest is masked, as GET /api/v3/config/secrets masks it: this
# page is served to anyone who can reach the port while the web
# login is off, and it was handing them every credential in the
# file. The save strips the masks and merges onto the stored file
# (save_raw_secrets_config), so a value left masked stays as it is.
from web_interface.auth import strip_auth_section
secrets_config_data = strip_auth_section(
pages_v3.config_manager.get_raw_file_content('secrets'))
secrets_config_data = mask_all_secret_values(strip_auth_section(
pages_v3.config_manager.get_raw_file_content('secrets')))
main_config_json = json.dumps(main_config_data, indent=4)
secrets_config_json = json.dumps(secrets_config_data, indent=4)
+35 -2
View File
@@ -133,6 +133,39 @@ def on_demand_state(snapshot: Optional[Dict[str, Any]],
return state
def display_gone(snapshot: Optional[Dict[str, Any]]) -> bool:
"""Is there positively no display behind a fallback to the cache?
True only when the socket should be there (this platform has one and it
is not switched off) but gave no ``snapshot``, and the render loop's
heartbeat file says nothing is running either: it is absent (systemd
removes its directory when the service stops), stale, or written by a
process that no longer exists -- #726's rules for the runtime snapshot.
Then what the display last left in the cache is a dead process's answer.
False whenever the answer is in doubt: a snapshot came in, the socket is
off or unsupported (Windows, the test suite, a deliberate ``off``), or a
live heartbeat says the display is running without a socket (an older
display). Those read the cache exactly as before.
"""
if snapshot is not None:
return False
if not socket_supported() or not client_socket_paths():
return False
from src import display_watchdog
from src.plugin_system.plugin_runtime import process_exists
heartbeat = display_watchdog.read_heartbeat(display_watchdog.HEARTBEAT_PATH)
if heartbeat is None:
return True
age = display_watchdog.heartbeat_age(heartbeat)
if age is None or age >= display_watchdog.HEARTBEAT_STALE_SECONDS:
return True
pid = heartbeat.get('pid')
if isinstance(pid, int) and not isinstance(pid, bool) and process_exists(pid) is False:
return True
return False
def loop_heartbeat_age(snapshot: Optional[Dict[str, Any]]) -> Optional[float]:
"""The render loop's heartbeat age now; None when the display has no
beat to report yet (or there is no snapshot)."""
@@ -141,5 +174,5 @@ def loop_heartbeat_age(snapshot: Optional[Dict[str, Any]]) -> Optional[float]:
return control_client.snapshot_loop_age(snapshot)
__all__ = ['current_status', 'loop_heartbeat_age', 'on_demand_state', 'read_state',
'stop_subscription']
__all__ = ['current_status', 'display_gone', 'loop_heartbeat_age', 'on_demand_state',
'read_state', 'stop_subscription']
@@ -18,7 +18,9 @@
* });
*
* The container re-renders from /api/v3/plugins/installed each init; the
* hidden input(s) must already hold the saved order/exclusions (JSON).
* hidden input(s) must already hold the saved order/exclusions (JSON). Saved
* ids of disabled plugins (installed, but without a row) stay in them, in
* their saved places; ids of plugins no longer installed are dropped.
*/
(function() {
'use strict';
@@ -39,17 +41,60 @@
const excludedInput = options.excludedInputId ? document.getElementById(options.excludedInputId) : null;
if (!container || !orderInput) return;
// The saved lists as the inputs held them when the rows were drawn.
// Only enabled plugins get a row, and the inputs are rewritten from
// the rows, so a disabled plugin's place and exclusion have to be
// carried over from these: dropped, the next Display or Durations
// save stored the lists without it, and once re-enabled it came back
// at the end of the rotation and scrolling in Vegas again.
let savedOrder = [];
let savedExcluded = [];
// Every installed plugin's id, enabled or not, from the same
// response. A saved id outside it belongs to an uninstalled plugin
// and is dropped, as every save used to; without the list, nothing
// is dropped.
let installedIds = null;
// Saved ids of installed plugins with no row, once each. Only
// strings: /config/main refuses a list holding anything else, which
// would block every save.
function unlisted(saved, rowIds) {
const seen = new Set(rowIds);
return saved.filter(id => {
if (typeof id !== 'string' || seen.has(id)) return false;
if (installedIds && !installedIds.has(id)) return false;
seen.add(id);
return true;
});
}
function syncInputs() {
const order = [];
const rowIds = [];
const excluded = [];
container.querySelectorAll('.plugin-order-item').forEach(item => {
const pluginId = item.dataset.pluginId;
order.push(pluginId);
rowIds.push(pluginId);
const checkbox = item.querySelector('.plugin-order-include');
if (checkbox && !checkbox.checked) excluded.push(pluginId);
});
orderInput.value = JSON.stringify(order);
if (excludedInput) excludedInput.value = JSON.stringify(excluded);
// An id without a row keeps its saved slot; the rows fill the
// other slots in their current order, and any rows left over
// (plugins not in the saved order) go last.
const kept = new Set(unlisted(savedOrder, rowIds));
const order = [];
let next = 0;
savedOrder.forEach(id => {
if (kept.has(id)) {
order.push(id);
kept.delete(id);
} else if (rowIds.includes(id) && next < rowIds.length) {
order.push(rowIds[next++]);
}
});
orderInput.value = JSON.stringify(order.concat(rowIds.slice(next)));
if (excludedInput) {
excludedInput.value = JSON.stringify(excluded.concat(unlisted(savedExcluded, rowIds)));
}
}
function setupDragAndDrop() {
@@ -104,6 +149,7 @@
.then(data => {
const allPlugins = (data.data && data.data.plugins) || data.plugins || [];
const plugins = allPlugins.filter(p => p.enabled);
installedIds = new Set(allPlugins.map(p => p && p.id));
if (plugins.length === 0) {
const empty = document.createElement('p');
empty.className = 'text-sm text-gray-500 italic';
@@ -125,6 +171,8 @@
// (e.g. a saved value of "null"); normalize to arrays.
if (!Array.isArray(currentOrder)) currentOrder = [];
if (!Array.isArray(excluded)) excluded = [];
savedOrder = currentOrder;
savedExcluded = excluded;
// Saved order first, then any newly enabled plugins.
const orderedPlugins = [];
@@ -884,7 +884,6 @@ With this off a live game takes over the whole display with the full-screen scor
// Update brightness display
document.getElementById('brightness').addEventListener('input', function() {
document.getElementById('brightness-value').textContent = this.value;
document.getElementById('brightness-display').textContent = this.value;
});
@@ -404,6 +404,10 @@
document.getElementById('web-login-new-token-value').textContent = res.d.data.token;
document.getElementById('web-login-new-token').classList.remove('hidden');
form.reset();
// app.js marks a form dirty on input and clears the mark only
// after an htmx save; this one posts with fetch, so clear it
// here or a reload asks "Leave site?" about a saved token.
form.removeAttribute('data-dirty');
notify(res.d.message || 'Token created', 'success');
}).catch(function(err) { notify('Request failed: ' + err.message, 'error'); });
},
@@ -55,19 +55,40 @@
<script>
(function () {
var DISMISS_KEY = 'ledmatrix-recon-dismissed';
// Startup reconciliation is done within seconds of the web service
// starting. The route also answers done: false when its status file is
// missing (reconciliation raised before writing it, or /tmp was cleaned
// under a long-running service), which used to keep this polling every
// 2 s for as long as the page was open, on every tab. So: give up after
// a minute, and poll only while the Overview is on screen.
var POLL_MS = 2000;
var MAX_POLLS = 30;
var _recon_timer = null;
var _polls = 0;
var _finished = false; // done, given up, or dismissed
var _active = false;
var _inFlight = false;
function checkReconciliation() {
_recon_timer = null;
_inFlight = true;
fetch('/api/v3/plugins/reconciliation-status')
.then(function (r) { return r.json(); })
.then(function (resp) {
_inFlight = false;
if (_finished) return;
var d = resp.data || {};
if (!d.done) {
// Reconciliation still running — poll again shortly
_recon_timer = setTimeout(checkReconciliation, 2000);
// Reconciliation still running (or it never wrote its
// status): ask again shortly, a bounded number of times.
if (++_polls >= MAX_POLLS) {
_finished = true;
return;
}
if (_active) _recon_timer = setTimeout(checkReconciliation, POLL_MS);
return;
}
_recon_timer = null;
_finished = true;
if (!d.unresolved || d.unresolved.length === 0) return;
var key = d.unresolved.map(function (i) { return i.plugin_id; }).sort().join(',');
if (sessionStorage.getItem(DISMISS_KEY) === key) return;
@@ -102,13 +123,33 @@
banner.dataset.dismissKey = key;
banner.style.setProperty('display', 'flex', 'important');
})
.catch(function () {});
.catch(function () { _inFlight = false; });
}
function startReconciliationPoll() {
_active = true;
if (!_finished && _recon_timer === null && !_inFlight) checkReconciliation();
}
function stopReconciliationPoll() {
_active = false;
if (_recon_timer !== null) {
clearTimeout(_recon_timer);
_recon_timer = null;
}
}
// Keyed apart from any other Overview registration, which a shared key
// would replace.
if (window.LEDVisibility) {
window.LEDVisibility.onActive('overview', startReconciliationPoll,
stopReconciliationPoll, 'overview-reconciliation');
} else {
startReconciliationPoll();
}
checkReconciliation();
window.dismissReconciliationBanner = function () {
var banner = document.getElementById('reconciliation-banner');
banner.style.setProperty('display', 'none', 'important');
_finished = true;
if (_recon_timer !== null) {
clearTimeout(_recon_timer);
_recon_timer = null;
@@ -1093,6 +1093,12 @@
Use TLS
<span class="text-xs text-gray-500">(a password without TLS crosses the network in the clear)</span>
</label>
<label id="mqtt-allow-insecure-row" class="${c.mqtt_tls ? 'hidden' : 'flex'} items-center gap-2 text-sm text-gray-700">
<input id="mqtt-allow-insecure" type="checkbox" ${c.allow_insecure_mqtt ? 'checked' : ''}
class="rounded border-gray-300">
Allow without TLS (trusted network)
<span class="text-xs text-gray-500">(needed to save a password while TLS is off)</span>
</label>
<div class="flex items-center justify-between gap-4 pt-2">
<p class="text-xs text-gray-500">
@@ -1132,6 +1138,15 @@
if (clearBtn) clearBtn.addEventListener('click', () => clearMqttPassword());
const clearTokenBtn = document.getElementById('mqtt-clear-api-token');
if (clearTokenBtn) clearTokenBtn.addEventListener('click', () => clearMqttApiToken());
// The cleartext opt-in only means something while TLS is off.
const tlsBox = document.getElementById('mqtt-tls');
const allowRow = document.getElementById('mqtt-allow-insecure-row');
if (tlsBox && allowRow) {
tlsBox.addEventListener('change', () => {
allowRow.classList.toggle('hidden', tlsBox.checked);
allowRow.classList.toggle('flex', !tlsBox.checked);
});
}
}
window.loadMqttBridge = function() {
@@ -1160,6 +1175,9 @@
on_demand_duration: val('mqtt-duration') === '' ? null : val('mqtt-duration'),
log_level: val('mqtt-log-level'),
mqtt_tls: !!(document.getElementById('mqtt-tls') || {}).checked,
// The server refuses a password with TLS off unless this is set
// (CWE-319); it is off until the user ticks it.
allow_insecure_mqtt: !!(document.getElementById('mqtt-allow-insecure') || {}).checked,
};
// Only send a password when one was typed; blank means "leave it alone".
const pw = val('mqtt-password');