From 0b039c875f9a97c2848282136682f20875aa9683 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Sat, 3 Oct 2026 22:30:16 -0400 Subject: [PATCH 01/37] fix(web): plugin settings endpoints - a refused save no longer leaks into config.json; GET masks secrets (#742) * fix(config): load_config hands each caller a private copy ConfigManager.load_config() returned its cached self.config itself (the mtime fast path from #410 kept the full path's aliasing). Web handlers edit what they load and then validate: the plugin form save applies the posted fields to the loaded section (a shallow .copy(), so nested dicts were the cache's own), and save_main_config sets its checkboxes before it checks auto_update_channel. When the save was refused, the edit stayed in the cache the fast path serves, and the next save of any other setting wrote it to config.json: the refused value, and a nested secret typed into the same form (mqtt.password, league.espn_s2, flightaware.api_key) in plain text, since it never reached config_secrets.json to be stripped. The form also reloaded showing the refused values. load_config() now returns a private copy on both paths, and save_config/save_config_atomic keep a copy of what they were given, so nothing a caller edits reaches the cache unless it is saved. Fixing it here rather than in each handler covers every route that edits before it validates. No caller relies on editing the cache without saving: every src/ and web_interface/ caller either reads, or saves the dict it edited. get_config() still returns the live dict for the display process's readers. The copy is a pickle round trip: on a Pi 4 with its real 64 KiB config, 2.0 ms against 6.9 ms for copy.deepcopy (json round trip 3.4 ms). Two tests asserted the aliasing itself and now assert a copy. Co-Authored-By: Claude Opus 5.5 * fix(web): GET /plugins/config masks secrets and refuses core sections The route returned the plugin's section as load_config() has it, with config_secrets.json merged in: API keys and tokens went out in plain text. #276 masked them here; #330's rewrite of the route dropped it, while the settings page and GET /config/secrets kept masking. It also took any plugin_id, so ?plugin_id=web_auth returned the login's cookie-signing key and password hash, and ?plugin_id=github the Plugin Store token, which GET /config/main strips and redacts. The route now refuses what _non_plugin_id_error refuses for reset and uninstall (core sections, malformed ids) with a 400, and blanks x-secret fields with mask_secret_fields after the defaults merge, as the page does. A plugin with no schema has its credential-named fields blanked by _redact_credentials, as GET /config/main does. Blank rather than the bullets of GET /config/secrets: the save drops a blank secret as "unchanged" (remove_empty_secrets) but would store the bullets, so the response must post back as it came. Tested: GET, then POST the response unchanged, keeps every stored secret. Co-Authored-By: Claude Opus 5.5 * fix(web): parse a table row's cells against the list's item schema An array of objects drawn as a table posts each cell as "cities.0.timezone". _get_schema_property stopped at "cities" (an array, not an object with properties), so _parse_form_value_with_schema got no schema for the cell and guessed: a blank optional text cell became None and a text cell holding digits became an int. Validation refused both, so every save of the page failed for as long as such a row existed -- geochron's city without a timezone, a countdown named "2027". A secret cell is always drawn blank, so a plugin with secrets in its rows could not be saved from the form at all. The lookup now steps from an index segment into the array's items: to the item schema itself for "color.2", into its properties for a row cell. Number, boolean and required cells convert as before. Co-Authored-By: Claude Opus 5.5 * fix(web): a blank secret field saves as "unchanged", required or not The settings page draws a stored secret blank (mask_secret_fields) and posts the blank back. _parse_form_value_with_schema turned a blank optional string into "" -- which the save drops as unchanged (remove_empty_secrets) -- but a blank required one into None. For a secret that is required with no default (youtube-stats' api_key) that None failed validation, so every save of the page was refused until the key was typed in again. A blank text secret (x-secret, type string) now parses to "", whatever its required list says; a list or object secret keeps getting [] or {}, which the save drops the same way. Not _SKIP_FIELD: skipping keeps the value load_config() merged in, and the save would then write it back to config_secrets.json -- after a secret change the cached section can still hold the old one, so that write reverted it. A test covers that sequence. Co-Authored-By: Claude Opus 5.5 * fix(web): POST /plugins/config refuses core sections and malformed ids Reset and uninstall check the plugin id with _non_plugin_id_error; the save did not. {"plugin_id": "display", "config": {...}} found no schema, so nothing was validated or filtered, and the body was merged into the core display section along with "enabled": true -- rows: "banana" included. A plugin_id that was not a string (a list, an object, a number) reached config.get() or the schema lookup, raised TypeError, and came back as a 500. Both the JSON and the form path now call _non_plugin_id_error first and answer its 400. Co-Authored-By: Claude Opus 5.5 * fix(web): a text field keeps "true", "[1, 2]" and "{}" as typed _parse_form_value_with_schema guessed before it consulted the schema: "true"/"false" became booleans, and a value starting with "[" or "{" that parsed as JSON became a list or object, whatever the field's type. A text setting holding "true", "False", "[1, 2]" or "{}" was then refused by validation ("Expected type string, got bool"), and the save with it. A field whose schema type is string, or string-or-null, now returns the posted text as it came. Every other type goes through the conversions as before; numbers in text fields were already left alone. Co-Authored-By: Claude Opus 5.5 * fix(config): copy the cached config without pickle _private_copy was a pickle round trip. It only ever unpickled bytes it had just made from our own dict, so nothing untrusted reached it, but it put pickle in the config path and Codacy failed the PR for it (B301/B403). The config is JSON data, so copying its dicts and lists is a full copy; every other value is immutable. Measured on ledpi (Pi 4) with its real 60 KiB config: 2.11 ms, against 1.92 ms for pickle and 6.75 ms for copy.deepcopy. Co-Authored-By: Claude Opus 5.5 * docs(changelog): describe the config copy without pickle Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 51 +++ src/config_manager.py | 53 ++- test/test_config_load_cache.py | 39 +- test/test_config_manager_secrets.py | 5 +- test/web_interface/test_api_v3_helpers.py | 13 + .../test_plugin_config_endpoints.py | 393 ++++++++++++++++++ web_interface/blueprints/api_v3/__init__.py | 31 ++ .../blueprints/api_v3/plugin_config.py | 43 +- 8 files changed, 609 insertions(+), 19 deletions(-) create mode 100644 test/web_interface/test_plugin_config_endpoints.py diff --git a/CHANGELOG.md b/CHANGELOG.md index cc9a0c8d..52e08dcb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -618,6 +618,57 @@ policies are unchanged. the plugin leaves rotation until the cooldown ends, the same as a raising `update()`. The display still moves straight on to the next mode. A hung `display()` is still recorded once, as a hang. +- A plugin settings save that failed validation no longer leaks into the next + save. `ConfigManager.load_config()` returned its cached config itself (the + fast path from #410), so the form save's edits went into the cache before + validation ran, and a refused save left them there. The next save of any + other setting (another plugin's, a plugin toggle, the schedule) wrote them + to config.json: the refused value, and a nested secret typed into the same + form (`mqtt.password`, `league.espn_s2`, `flightaware.api_key`) in plain + text, because it had never reached config_secrets.json to be stripped. + The form also reloaded showing the refused values. `load_config()` now + returns a private copy, and the saves keep one, so nothing a caller edits + reaches the cache unless it is saved. The copy duplicates only the dicts + and lists (every other JSON value is immutable): 2.1 ms for a real 60 KiB + config on a Pi 4, against 6.8 ms for `copy.deepcopy`. +- `GET /api/v3/plugins/config` no longer returns secrets. It sent back the + plugin's section with config_secrets.json merged in, API keys and tokens + in plain text: the masking #276 added was dropped in #330. It also took + any id, so `?plugin_id=web_auth` returned the login's cookie-signing key + and password hash and `?plugin_id=github` the Plugin Store token. Secret + fields now come back blank, as the settings page renders them, and a + plugin with no schema has its credential-named fields blanked, as + `GET /config/main` does. Blank rather than the `••••••••` of + `GET /config/secrets`, because the save reads a blank secret as + "unchanged", so a client can post the response back without erasing + one. Core sections and malformed ids get a 400, as they already did from + reset and uninstall. +- Plugin settings with a table (a list of rows, such as geochron's cities + or the countdowns) save again when a text cell is blank or holds only + digits. A row posts its cells as `cities.0.timezone`, and the schema + lookup stopped at the list, so each cell was parsed with no schema: a + blank optional text cell became null, and a name like "2027" became a + number. Either failed validation, and every save of the page failed for + as long as the row existed. A plugin with a secret in its rows could not + be saved from the page at all, since the secret cell is drawn blank. The + lookup now steps from the index into the list's item schema. +- A plugin whose API key is required and has no default (youtube-stats) + can be saved from its settings page without typing the key in again. The + page draws a stored secret blank and posts the blank back; for a required + secret the save read that blank as null, failed validation, and refused + every save of the page. A blank secret field now means "unchanged", as it + already did for an optional one. +- `POST /api/v3/plugins/config` refuses a core section or a malformed + plugin id with a 400, as reset and uninstall already did. + `{"plugin_id": "display", ...}` merged unvalidated values into the core + display section (and added `"enabled": true` to it), and an id that was + not a string answered with a 500. +- A plugin text setting saves what was typed when that looks like a + boolean or JSON. The form save tried `true`/`false` and `[...]`/`{...}` + before it looked at the schema, so a text field holding "true", "False", + "[1, 2]" or "{}" was stored as a boolean, list or object, and the save + failed validation. Text fields, nullable ones included, are now taken as + typed; other types convert as before. - A WiFi notice (such as "Connected to HomeNet" or "AP mode on") now shows within about a second of being posted. It was only checked between screens, so a 5 s notice posted during a 20 s screen expired before that diff --git a/src/config_manager.py b/src/config_manager.py index dd7982a3..25911d2a 100644 --- a/src/config_manager.py +++ b/src/config_manager.py @@ -46,6 +46,35 @@ from src.common.permission_utils import ( get_config_dir_mode ) + +def _private_copy(config: Dict[str, Any]) -> Dict[str, Any]: + """A deep copy of ``config`` that shares nothing with it. + + load_config() hands one out per call, and the saves keep one, so the + cached config is never an object a caller holds. A web handler edits what + it loaded, validates, and may refuse the save; when the cache was that + same object, the refused edit stayed in it, and the next save of any + other setting wrote it to config.json -- a nested secret included, in + plain text, since it had never reached config_secrets.json to be + stripped. + + The config is JSON data, so only its dicts and lists need copying; every + other value in it is immutable. On a Pi 4 with a real 60 KiB config this + takes 2.1 ms against copy.deepcopy's 6.8 ms, on a path ~30 handlers call + (a pickle round trip is no faster, 1.9 ms, and brings pickle into the + config path for nothing). + """ + return _copy_containers(config) + + +def _copy_containers(value: Any) -> Any: + if isinstance(value, dict): + return {key: _copy_containers(item) for key, item in value.items()} + if isinstance(value, list): + return [_copy_containers(item) for item in value] + return value + + class ConfigManager: """ Reads and writes the main application configuration files. @@ -126,9 +155,10 @@ class ConfigManager: validate_after_write=validate_after_write ) - # Update in-memory config if save was successful + # Update in-memory config if save was successful. A copy: the caller + # still holds new_config_data (see _private_copy). if result.status == SaveResultStatus.SUCCESS: - self.config = new_config_data + self.config = _private_copy(new_config_data) # In-memory config now matches what was just written, so the # load_config fast path may return it. It still carries the # merged secrets that were stripped on disk; that matches a full @@ -208,14 +238,16 @@ class ConfigManager: Fast path: when config.json, config_secrets.json and the template are all unchanged since the last successful load (mtime_ns + size), - the already-parsed self.config is returned without touching the - files — same aliasing semantics as the full path, which also - returns self.config. + a copy of the already-parsed self.config is returned without + touching the files. + + Either way the caller gets its own copy (see _private_copy): editing + it changes nothing here until it is saved. """ try: current_sig = self._files_signature() if self.config and self._loaded_sig == current_sig: - return self.config + return _private_copy(self.config) # Check if config file exists, if not create from template if not os.path.exists(self.config_path): @@ -249,8 +281,8 @@ class ConfigManager: # Signature taken AFTER load + migration (migration may write the # config back), so it reflects exactly what was read/written. self._loaded_sig = self._files_signature() - return self.config - + return _private_copy(self.config) + except FileNotFoundError as e: # Only config.json can get here: a missing or unreadable secrets # file is handled where it is read. @@ -355,8 +387,9 @@ class ConfigManager: try: atomic_write_json(self.config_path, config_to_write) - # Update the in-memory config to the new state (which includes secrets for runtime) - self.config = new_config_data + # Update the in-memory config to the new state (which includes + # secrets for runtime), as a copy -- see _private_copy + self.config = _private_copy(new_config_data) self._loaded_sig = self._files_signature() self.logger.info(f"Configuration successfully saved to {os.path.abspath(self.config_path)}") if secrets_content: diff --git a/test/test_config_load_cache.py b/test/test_config_load_cache.py index e0d634fb..3b44e364 100644 --- a/test/test_config_load_cache.py +++ b/test/test_config_load_cache.py @@ -58,7 +58,8 @@ class TestFastPath: for _ in range(10): again = m.load_config() assert counts["n"] == 0, "fast path must not re-open any config file" - assert again is first # same aliasing semantics as the full path + assert again == first + assert again is not first # each caller gets its own copy, see below def test_config_change_triggers_reload(self, mgr): m, config, secrets, template = mgr @@ -98,6 +99,33 @@ class TestFastPath: assert m.load_config()["timezone"] == "America/New_York" +class TestCallersGetACopy: + """A web handler edits what load_config returned, then validates. When + validation failed, the edit stayed in the cache the fast path serves, and + the next unrelated save wrote it -- a nested secret included, in plain + text, because it had never reached config_secrets.json to be stripped.""" + + def test_editing_a_loaded_config_does_not_change_the_next_load(self, mgr): + m, config, secrets, template = mgr + loaded = m.load_config() + loaded["display"]["brightness"] = 1 + loaded["weather"]["api_key"] = "typed-but-never-saved" + again = m.load_config() + assert again["display"]["brightness"] == 90 + assert again["weather"]["api_key"] == "sek" + + def test_the_full_path_also_returns_a_copy(self, mgr): + m, config, secrets, template = mgr + m.load_config()["display"]["brightness"] = 1 # first load: full path + assert m.load_config()["display"]["brightness"] == 90 + + def test_an_edit_never_reaches_a_later_save(self, mgr): + m, config, secrets, template = mgr + m.load_config()["display"]["new_secret"] = "hunter2" # then bailed out + m.save_config(m.load_config()) # some other handler saves + assert "hunter2" not in config.read_text() + + class TestSaveCoherence: def test_save_config_then_load_returns_saved_data(self, mgr, monkeypatch): m, config, secrets, template = mgr @@ -111,6 +139,15 @@ class TestSaveCoherence: assert loaded["weather"]["api_key"] == "sek" # secrets survive in memory assert counts["n"] == 0 # signature refreshed by save; no re-read + def test_the_saved_dict_does_not_become_the_cache(self, mgr): + m, config, secrets, template = mgr + m.load_config() + new = {"display": {"brightness": 42}, "timezone": "UTC", + "weather": {"api_key": "sek"}} + m.save_config(new) + new["display"]["brightness"] = 7 # the caller keeps using its dict + assert m.load_config()["display"]["brightness"] == 42 + def test_cross_process_save_is_picked_up(self, mgr): """Another process writing config.json (different mtime) must bust this process's fast path — the core cross-process guarantee.""" diff --git a/test/test_config_manager_secrets.py b/test/test_config_manager_secrets.py index da24a734..dd6138b3 100644 --- a/test/test_config_manager_secrets.py +++ b/test/test_config_manager_secrets.py @@ -143,7 +143,10 @@ class TestLoadFastPath: manager = make_manager(tmp_path, config={"timezone": "UTC"}) first = manager.load_config() second = manager.load_config() - assert second is first # same aliased dict, no re-read + # A copy of the cached dict, never the dict itself; that it is not + # re-read is test_config_load_cache's test_unchanged_files_are_not_reread + assert second == first + assert second is not first def test_touching_secrets_file_invalidates_cache(self, tmp_path): manager = make_manager( diff --git a/test/web_interface/test_api_v3_helpers.py b/test/web_interface/test_api_v3_helpers.py index fbd142bf..068e97e3 100644 --- a/test/web_interface/test_api_v3_helpers.py +++ b/test/web_interface/test_api_v3_helpers.py @@ -169,6 +169,10 @@ class TestGetSchemaProperty: }, "fifa.world": {"type": "object", "properties": {"enabled": {"type": "boolean"}}}, + "cities": {"type": "array", + "items": {"type": "object", + "properties": {"timezone": {"type": "string"}}}}, + "color": {"type": ["array", "null"], "items": {"type": "integer"}}, } } @@ -185,6 +189,15 @@ class TestGetSchemaProperty: prop = _get_schema_property(self.SCHEMA, "fifa.world.enabled") assert prop == {"type": "boolean"} + def test_an_index_steps_into_the_array_items(self): + # How a table row posts its cells + assert _get_schema_property(self.SCHEMA, "cities.0.timezone") == {"type": "string"} + assert _get_schema_property(self.SCHEMA, "color.2") == {"type": "integer"} + + def test_a_non_index_under_an_array_is_not_found(self): + assert _get_schema_property(self.SCHEMA, "cities.timezone") is None + assert _get_schema_property(self.SCHEMA, "cities.0.nope") is None + def test_missing_path_returns_none(self): assert _get_schema_property(self.SCHEMA, "nope.nope") is None diff --git a/test/web_interface/test_plugin_config_endpoints.py b/test/web_interface/test_plugin_config_endpoints.py new file mode 100644 index 00000000..d1a52314 --- /dev/null +++ b/test/web_interface/test_plugin_config_endpoints.py @@ -0,0 +1,393 @@ +"""GET and POST /plugins/config against a real ConfigManager and SchemaManager. + +Each class is one bug, reproduced through the endpoint the settings form and +API clients use, with assertions on config.json and config_secrets.json. +""" + +import json +from unittest.mock import MagicMock + +import pytest +from flask import Flask + +from src.config_manager import ConfigManager +from src.plugin_system.schema_manager import SchemaManager +from web_interface.blueprints.api_v3 import api_v3 + +PLUGIN_ID = "demo" +OTHER_ID = "other" + +SCHEMA = { + "type": "object", + "properties": { + "enabled": {"type": "boolean", "default": True}, + "api_key": {"type": "string", "x-secret": True, "default": ""}, + "city": {"type": "string", "default": "Austin"}, + "mqtt": { + "type": "object", + "properties": { + "host": {"type": "string", "default": ""}, + "port": {"type": "integer", "default": 1883, + "minimum": 1, "maximum": 65535}, + "password": {"type": "string", "x-secret": True, "default": ""}, + }, + }, + "accounts": { + "type": "array", + "default": [], + "items": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "token": {"type": "string", "x-secret": True}, + }, + }, + }, + }, +} + +OTHER_SCHEMA = { + "type": "object", + "properties": { + "enabled": {"type": "boolean", "default": True}, + "label": {"type": "string", "default": "x"}, + }, +} + +STORED = { + PLUGIN_ID: {"enabled": True, "city": "Paris", + "mqtt": {"host": "broker", "port": 1883}, + "accounts": [{"name": "a"}, {"name": "b"}]}, + OTHER_ID: {"enabled": True, "label": "hello"}, +} + +STORED_SECRETS = { + PLUGIN_ID: {"api_key": "TOPSECRET", + "accounts": [{"token": "TOK-A"}, {"token": "TOK-B"}]}, +} + +_ATTRS = ('config_manager', 'plugin_catalog', 'plugin_store_manager', + 'saved_repositories_manager', 'schema_manager', + 'operation_queue', 'operation_history', 'cache_manager') + + +@pytest.fixture +def env(tmp_path): + config_file = tmp_path / "config.json" + secrets_file = tmp_path / "config_secrets.json" + plugins_dir = tmp_path / "plugins" + for plugin_id, schema in ((PLUGIN_ID, SCHEMA), (OTHER_ID, OTHER_SCHEMA)): + plugin_dir = plugins_dir / plugin_id + plugin_dir.mkdir(parents=True) + (plugin_dir / "config_schema.json").write_text(json.dumps(schema)) + (plugin_dir / "manifest.json").write_text(json.dumps({"id": plugin_id})) + config_file.write_text(json.dumps(STORED)) + secrets_file.write_text(json.dumps(STORED_SECRETS)) + + sentinel = object() + originals = {name: getattr(api_v3, name, sentinel) for name in _ATTRS} + + config_manager = ConfigManager(config_path=str(config_file), + secrets_path=str(secrets_file)) + config_manager.template_path = str(tmp_path / "no-template.json") + plugin_manager = MagicMock() + plugin_manager.plugin_manifests = {PLUGIN_ID: {"id": PLUGIN_ID}, + OTHER_ID: {"id": OTHER_ID}} + plugin_manager.plugins_dir = plugins_dir + + for name in _ATTRS: + setattr(api_v3, name, MagicMock()) + api_v3.config_manager = config_manager + api_v3.schema_manager = SchemaManager(plugins_dir=plugins_dir, project_root=tmp_path) + api_v3.plugin_catalog = plugin_manager + api_v3.operation_queue = None + + app = Flask(__name__) + app.config["TESTING"] = True + app.register_blueprint(api_v3, url_prefix="/api/v3") + + class Env: + client = app.test_client() + + @staticmethod + def use_schema(schema, plugin_id=PLUGIN_ID): + (plugins_dir / plugin_id / "config_schema.json").write_text(json.dumps(schema)) + + @staticmethod + def store(section, plugin_id=PLUGIN_ID): + main = json.loads(config_file.read_text()) + main[plugin_id] = section + config_file.write_text(json.dumps(main)) + + @staticmethod + def main(): + return json.loads(config_file.read_text()) + + @staticmethod + def secrets(): + return json.loads(secrets_file.read_text()) + + @staticmethod + def post_form(data, plugin_id=PLUGIN_ID): + return Env.client.post(f"/api/v3/plugins/config?plugin_id={plugin_id}", + data=data) + + @staticmethod + def post_json(config, plugin_id=PLUGIN_ID): + return Env.client.post("/api/v3/plugins/config", + json={"plugin_id": plugin_id, "config": config}) + + yield Env + + for name, original in originals.items(): + if original is sentinel: + if hasattr(api_v3, name): + delattr(api_v3, name) + else: + setattr(api_v3, name, original) + + +class TestARejectedSaveLeavesNothingBehind: + """The form save edited the cached config load_config hands out, then + failed validation. The cache kept the edit, and the next save of any + other setting wrote it to config.json -- the rejected value, and a + nested secret typed into the same form in plain text.""" + + REJECTED = {"mqtt.host": "broker", "mqtt.port": "99999", + "mqtt.password": "hunter2", "__rendered_section": ["mqtt"]} + + def test_the_rejected_values_never_reach_config_json(self, env): + assert env.post_form(self.REJECTED).status_code == 400 + + resp = env.post_json({"label": "bye"}, plugin_id=OTHER_ID) + assert resp.status_code == 200, resp.get_json() + + main = env.main() + assert main[OTHER_ID]["label"] == "bye" + assert main[PLUGIN_ID]["mqtt"] == {"host": "broker", "port": 1883} + assert "hunter2" not in json.dumps(main) + + def test_the_form_reloads_with_the_stored_values(self, env): + assert env.post_form(self.REJECTED).status_code == 400 + assert api_v3.config_manager.load_config()[PLUGIN_ID]["mqtt"]["port"] == 1883 + + +class TestGetMasksSecrets: + """GET /plugins/config returned the section with config_secrets.json + merged in, secrets and all: the masking #276 added was lost when the + route was rewritten. The settings page and GET /config/secrets mask.""" + + def test_secrets_come_back_blank(self, env): + data = env.client.get(f"/api/v3/plugins/config?plugin_id={PLUGIN_ID}").get_json()["data"] + assert data["api_key"] == "" + assert data["accounts"] == [{"name": "a", "token": ""}, {"name": "b", "token": ""}] + assert data["city"] == "Paris" + + def test_posting_the_response_back_keeps_every_secret(self, env): + data = env.client.get(f"/api/v3/plugins/config?plugin_id={PLUGIN_ID}").get_json()["data"] + resp = env.post_json(data) + assert resp.status_code == 200, resp.get_json() + assert env.secrets()[PLUGIN_ID] == STORED_SECRETS[PLUGIN_ID] + assert "TOPSECRET" not in json.dumps(env.main()) + + def test_the_settings_form_posting_masked_fields_keeps_every_secret(self, env): + # The page renders secrets blank (pages_v3 masks the same way) + resp = env.post_form({ + "api_key": "", "city": "Lyon", "mqtt.host": "broker", "mqtt.port": "1883", + "mqtt.password": "", "__rendered_section": ["api_key", "city", "mqtt"]}) + assert resp.status_code == 200, resp.get_json() + assert env.secrets()[PLUGIN_ID] == STORED_SECRETS[PLUGIN_ID] + assert env.main()[PLUGIN_ID]["city"] == "Lyon" + + def test_a_plugin_without_a_schema_has_credential_named_fields_blanked(self, env, tmp_path): + (tmp_path / "plugins" / "bare").mkdir() + env.store({"enabled": True, "station": "KAUS"}, plugin_id="bare") + secrets = env.secrets() + secrets["bare"] = {"api_token": "BARE-TOKEN"} + (tmp_path / "config_secrets.json").write_text(json.dumps(secrets)) + data = env.client.get("/api/v3/plugins/config?plugin_id=bare").get_json()["data"] + assert data["api_token"] == "" + assert data["station"] == "KAUS" + + @pytest.mark.parametrize("section", ["web_auth", "github", "display"]) + def test_a_core_section_is_refused(self, env, tmp_path, section): + secrets = env.secrets() + secrets["web_auth"] = {"cookie_secret": "COOKIE-KEY", "password_hash": "HASH"} + secrets["github"] = {"api_token": "ghp_TOKEN"} + (tmp_path / "config_secrets.json").write_text(json.dumps(secrets)) + env.store({"hardware": {"rows": 32}}, plugin_id="display") + resp = env.client.get(f"/api/v3/plugins/config?plugin_id={section}") + assert resp.status_code == 400 + body = resp.get_data(as_text=True) + assert "COOKIE-KEY" not in body and "ghp_TOKEN" not in body + + +ROWS_SCHEMA = { + "type": "object", + "properties": { + "enabled": {"type": "boolean", "default": True}, + "cities": { + "type": "array", + "x-widget": "array-table", + "default": [], + "items": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "timezone": {"type": "string"}, + "lat": {"type": "number"}, + "show": {"type": "boolean", "default": True}, + }, + "required": ["name", "lat"], + }, + }, + }, +} + + +class TestArrayRowCellsFollowTheItemSchema: + """A table row posts its cells as ``cities.0.timezone``. The schema + lookup stopped at the array, so each cell was parsed blind: a blank + optional text cell became null and a text cell holding digits became a + number, and either failed validation -- every save of the page, for as + long as the row existed (geochron's city without a timezone, a countdown + named "2027").""" + + ROW = {"cities.0.name": "Tokyo", "cities.0.timezone": "Asia/Tokyo", + "cities.0.lat": "35.68", "cities.0.show": "true", + "__rendered_section": ["cities"]} + + @pytest.fixture(autouse=True) + def _rows(self, env): + env.use_schema(ROWS_SCHEMA) + env.store({"enabled": True, "cities": [ + {"name": "Tokyo", "timezone": "Asia/Tokyo", "lat": 35.68, "show": True}]}) + + def test_a_blank_optional_text_cell_saves(self, env): + resp = env.post_form({**self.ROW, "cities.0.timezone": ""}) + assert resp.status_code == 200, resp.get_json() + assert env.main()[PLUGIN_ID]["cities"][0]["timezone"] == "" + + def test_a_text_cell_of_digits_stays_text(self, env): + resp = env.post_form({**self.ROW, "cities.0.name": "2027"}) + assert resp.status_code == 200, resp.get_json() + assert env.main()[PLUGIN_ID]["cities"][0]["name"] == "2027" + + def test_number_and_boolean_cells_still_convert(self, env): + resp = env.post_form({**self.ROW, "cities.0.show": "false"}) + assert resp.status_code == 200, resp.get_json() + assert env.main()[PLUGIN_ID]["cities"] == [ + {"name": "Tokyo", "timezone": "Asia/Tokyo", "lat": 35.68, "show": False}] + + +class TestMaskedSecretCellsInARow: + """The same lookup: a row's secret cell, rendered blank, came back as + null and failed validation, so a plugin with secrets in a list could not + be saved from its settings page at all.""" + + def test_the_stored_tokens_survive_a_save_of_the_form(self, env): + resp = env.post_form({ + "city": "Lyon", "accounts.0.name": "a", "accounts.0.token": "", + "accounts.1.name": "b", "accounts.1.token": "", + "__rendered_section": ["city", "accounts"]}) + assert resp.status_code == 200, resp.get_json() + assert env.secrets()[PLUGIN_ID] == STORED_SECRETS[PLUGIN_ID] + assert env.main()[PLUGIN_ID]["accounts"] == [{"name": "a"}, {"name": "b"}] + + +class TestABlankSecretIsLeftAsStored: + """The form renders a secret blank and posts the blank back. For a + required secret with no default (youtube-stats' api_key) the blank was + read as null, failed validation, and blocked every save of the page + until the key was typed in again.""" + + @pytest.fixture(autouse=True) + def _required_secret(self, env): + schema = json.loads(json.dumps(SCHEMA)) + del schema["properties"]["api_key"]["default"] + schema["required"] = ["api_key"] + env.use_schema(schema) + + def test_saving_other_settings_keeps_the_stored_secret(self, env): + resp = env.post_form({"api_key": "", "city": "Lyon", + "__rendered_section": ["api_key", "city"]}) + assert resp.status_code == 200, resp.get_json() + assert env.main()[PLUGIN_ID]["city"] == "Lyon" + assert env.secrets()[PLUGIN_ID]["api_key"] == "TOPSECRET" + + def test_a_new_secret_is_still_saved(self, env): + resp = env.post_form({"api_key": "NEW-KEY", "city": "Lyon", + "__rendered_section": ["api_key", "city"]}) + assert resp.status_code == 200, resp.get_json() + assert env.secrets()[PLUGIN_ID]["api_key"] == "NEW-KEY" + + def test_a_changed_secret_then_left_blank_stays_changed(self, env): + # The second save must not write back what the first one's load + # had merged in (the old key) + env.post_form({"api_key": "NEW-KEY", "__rendered_section": ["api_key"]}) + resp = env.post_form({"api_key": "", "city": "Nice", + "__rendered_section": ["api_key", "city"]}) + assert resp.status_code == 200, resp.get_json() + assert env.secrets()[PLUGIN_ID]["api_key"] == "NEW-KEY" + + def test_a_blank_list_secret_is_left_as_stored_too(self, env, tmp_path): + schema = json.loads(json.dumps(SCHEMA)) + schema["properties"]["tokens"] = {"type": "array", "x-secret": True, + "items": {"type": "string"}, "default": []} + env.use_schema(schema) + secrets = env.secrets() + secrets[PLUGIN_ID]["tokens"] = ["t1", "t2"] + (tmp_path / "config_secrets.json").write_text(json.dumps(secrets)) + resp = env.post_form({"tokens": "", "city": "Lyon", + "__rendered_section": ["tokens", "city"]}) + assert resp.status_code == 200, resp.get_json() + assert env.secrets()[PLUGIN_ID]["tokens"] == ["t1", "t2"] + + +class TestSaveRefusesWhatIsNotAPluginId: + """GET and reset refuse a core section or a malformed id; the save took + any of them. ``{"plugin_id": "display"}`` merged unvalidated values into + the core display section, and an id that was not a string raised a + TypeError, answered as a 500.""" + + def test_a_core_section_is_refused_and_left_alone(self, env): + env.store({"hardware": {"rows": 32}}, plugin_id="display") + resp = env.post_json({"hardware": {"rows": "banana"}}, plugin_id="display") + assert resp.status_code == 400 + assert env.main()["display"] == {"hardware": {"rows": 32}} + + def test_the_form_save_refuses_one_too(self, env): + resp = env.post_form({"password_hash": "x"}, plugin_id="web_auth") + assert resp.status_code == 400 + assert "web_auth" not in env.main() + + @pytest.mark.parametrize("plugin_id", [["demo"], {"id": "demo"}, 7, "", "../demo"]) + def test_a_malformed_id_is_a_400(self, env, plugin_id): + resp = env.post_json({"city": "Lyon"}, plugin_id=plugin_id) + assert resp.status_code == 400 + + +class TestTextFieldsKeepWhatWasTyped: + """A text field holding "true", "False", "[1, 2]" or "{}" was converted + to a boolean, list or object before the schema's type was consulted, and + the save then failed validation for a perfectly good string.""" + + @pytest.mark.parametrize("typed", ["true", "False", "[1, 2]", "{}", "42"]) + def test_a_text_field(self, env, typed): + resp = env.post_form({"city": typed, "__rendered_section": ["city"]}) + assert resp.status_code == 200, resp.get_json() + assert env.main()[PLUGIN_ID]["city"] == typed + + def test_a_nullable_text_field(self, env): + schema = json.loads(json.dumps(SCHEMA)) + schema["properties"]["nickname"] = {"type": ["string", "null"], "default": None} + env.use_schema(schema) + resp = env.post_form({"nickname": "false", "__rendered_section": ["nickname"]}) + assert resp.status_code == 200, resp.get_json() + assert env.main()[PLUGIN_ID]["nickname"] == "false" + + def test_other_types_still_convert(self, env): + resp = env.post_form({"mqtt.host": "true", "mqtt.port": "8883", + "__rendered_section": ["mqtt"]}) + assert resp.status_code == 200, resp.get_json() + assert env.main()[PLUGIN_ID]["mqtt"] == {"host": "true", "port": 8883} diff --git a/web_interface/blueprints/api_v3/__init__.py b/web_interface/blueprints/api_v3/__init__.py index f8a903ab..b40105be 100644 --- a/web_interface/blueprints/api_v3/__init__.py +++ b/web_interface/blueprints/api_v3/__init__.py @@ -957,6 +957,19 @@ def _get_schema_property(schema, key_path): i = j matched = True break + # Through an array to its items: a table row posts its cells + # as "cities.0.timezone", where the index names no property. + # Stopping here left each cell parsed with no schema at all, + # so a blank text cell became null and "2027" a number. + items = prop.get('items') if _schema_type_is(prop, 'array') else None + if isinstance(items, dict) and parts[j].isdigit(): + if j + 1 == len(parts): + return items + if 'properties' in items: + current = items['properties'] + i = j + 1 + matched = True + break # Matched a non-object before consuming the path — can't go deeper. return None if not matched: @@ -1040,6 +1053,16 @@ def _parse_form_value_with_schema(value, key_path, schema): # Handle None/empty values if value is None or (isinstance(value, str) and value.strip() == ''): + # The form draws a stored secret blank, so a blank secret means + # "unchanged", and "" is what the save drops as unchanged + # (remove_empty_secrets). A required one with no default fell + # through to None below, failed validation, and blocked every save + # of the page until the secret was typed in again. Not _SKIP_FIELD: + # that keeps the merged value from load_config(), which the save + # would then write back to config_secrets.json. Text secrets only: + # a list or object one gets its empty value below, dropped the same. + if prop and prop.get('x-secret') and prop.get('type', 'string') == 'string': + return "" # A nullable field left blank means null, not an empty container. # This is the inherit sentinel for per-mode style overrides: an # empty list there would read as "the user chose no colour" rather @@ -1074,6 +1097,14 @@ def _parse_form_value_with_schema(value, key_path, schema): if isinstance(value, str): stripped = value.strip() + # A text field keeps what was typed. The guesses below ran first, so + # "true", "False", "[1, 2]" or "{}" in a text field became a boolean, + # list or object, and the save failed validation for a good string. + declared = prop.get('type') if isinstance(prop, dict) else None + if declared == 'string' or (isinstance(declared, list) and + [t for t in declared if t != 'null'] == ['string']): + return value + # Check for boolean strings if stripped.lower() == 'true': return True diff --git a/web_interface/blueprints/api_v3/plugin_config.py b/web_interface/blueprints/api_v3/plugin_config.py index 8db8b168..fddb2d17 100644 --- a/web_interface/blueprints/api_v3/plugin_config.py +++ b/web_interface/blueprints/api_v3/plugin_config.py @@ -9,13 +9,15 @@ from web_interface.blueprints.api_v3 import ( _enhance_schema_with_core_properties, _non_plugin_id_error, _filter_config_by_schema, _get_schema_property, _hidden_array_item_property, _plugin_directory, - _parse_form_value_with_schema, _schema_allows_null, _schema_type_is, - _set_missing_booleans_to_false, _set_nested_value, api_v3, datetime, - deep_merge, error_response, exception_error_response, find_secret_fields, - json, jsonify, logger, merge_secrets, os, remove_empty_secrets, request, - separate_secrets, success_response, validate_request_json, + _parse_form_value_with_schema, _redact_credentials, _schema_allows_null, + _schema_type_is, _set_missing_booleans_to_false, _set_nested_value, api_v3, + datetime, deep_merge, error_response, exception_error_response, + find_secret_fields, json, jsonify, logger, merge_secrets, os, + remove_empty_secrets, request, separate_secrets, success_response, + validate_request_json, ) from src.web_interface.config_arrays import coerce_array_shapes +from src.web_interface.secret_helpers import mask_secret_fields from src.web_interface.validators import dedup_unique_arrays import web_interface.blueprints.api_v3 as _pkg # Read through the module rather than bound by value: tests patch these @@ -43,6 +45,12 @@ def get_plugin_config(): context={'missing_params': ['plugin_id']}, status_code=400 ) + # load_config() merges config_secrets.json in, core sections + # included: ?plugin_id=web_auth returned the login's cookie key and + # password hash, and ?plugin_id=github the Plugin Store token. + id_error = _non_plugin_id_error(plugin_id) + if id_error: + return id_error # Get plugin configuration from config manager main_config = api_v3.config_manager.load_config() @@ -52,12 +60,13 @@ def get_plugin_config(): # missing fields, reading legacy booleans as objects first: what the # plugin runs with, and what posts back through the JSON save schema_mgr = api_v3.schema_manager + schema = None if schema_mgr: try: from src.plugin_system.schema_manager import prepare_plugin_config + schema = schema_mgr.load_schema(plugin_id, use_cache=True) defaults = schema_mgr.generate_default_config(plugin_id, use_cache=True) - plugin_config = prepare_plugin_config( - plugin_config, schema_mgr.load_schema(plugin_id, use_cache=True), defaults) + plugin_config = prepare_plugin_config(plugin_config, schema, defaults) except Exception as e: # Log but don't fail - defaults merge is best effort logger.warning("Could not merge defaults for %s: %s", plugin_id, e) @@ -158,6 +167,17 @@ def get_plugin_config(): 'display_duration': 30 } + # Secrets go out blank, as the settings page renders them (#276 added + # this; #330 dropped it). Blank, not the bullets GET /config/secrets + # uses: the save reads a blank secret as "unchanged", so this + # response posts back without erasing one. + properties = schema.get('properties') if isinstance(schema, dict) else None + if isinstance(properties, dict): + plugin_config = mask_secret_fields(plugin_config, properties) + else: + # No schema to mark them: blank whatever is named like one + plugin_config = _redact_credentials(plugin_config) + return success_response(data=plugin_config) except Exception as e: return exception_error_response(e, ErrorCode.CONFIG_LOAD_FAILED) @@ -183,6 +203,12 @@ def save_plugin_config(): if error: return error plugin_id = data['plugin_id'] + # As reset and uninstall do: {"plugin_id": "display"} merged + # unvalidated values into the core display section, and an id + # that was not a string raised a TypeError, answered as a 500. + id_error = _non_plugin_id_error(plugin_id) + if id_error: + return id_error submitted_config = data.get('config', {}) if not isinstance(submitted_config, dict): return error_response( @@ -201,6 +227,9 @@ def save_plugin_config(): 'plugin_id required in query string', status_code=400 ) + id_error = _non_plugin_id_error(plugin_id) + if id_error: + return id_error # Load existing config as base (partial form updates should merge, not replace) existing_config = {} From 8a0cce1aaf0bb7ef7146c639d98e81193047b4ca Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Sat, 3 Oct 2026 22:30:28 -0400 Subject: [PATCH 02/37] fix(web): mask the Config Editor's secrets; keep disabled plugins' rotation slot and Vegas exclusion; restore only missing plugins (#743) * fix(web): mask the Config Editor's secrets like GET /config/secrets The Config Editor tab (/partials/raw-json) filled its config_secrets.json editor with the file as it is on disk. GET /api/v3/config/secrets masks every value because the interface is reachable without a login by default, but this page handed the same credentials (GitHub token, Home Assistant token, plugin API keys) to anyone who loaded it. The masked-save path in save_raw_secrets_config was written for a masked editor and never got one. _load_raw_json_partial now masks the section with mask_all_secret_values after strip_auth_section, exactly as the GET does. Saving it back is safe: save_raw_secrets_config drops the masks (strip_masked_values) and merges the rest onto the stored file (deep_merge), so an untouched secret stays as it is and a replaced mask is the only value that changes. The config.json editor is left as it is. Its save (save_raw_main_config) writes the posted object verbatim, with no mask stripping or merge, so a masked main editor would write the bullets over any credential it holds. Masking it needs a merge-on-save of its own first. Tests: TestConfigEditorRoundTrip renders the partial over a real ConfigManager, checks no real value is in the editor, and posts the editor back unchanged (the file is identical) and with one mask replaced (only that value changes). Co-Authored-By: Claude Opus 5.5 * fix(web): keep disabled plugins in the saved rotation order and Vegas exclusions PluginOrderList draws one row per enabled plugin and, once drawn, rewrites its hidden inputs (plugin_rotation_order, vegas_plugin_order, vegas_excluded_plugins) from those rows. A disabled plugin has no row, so merely opening the Display or Rotation & Durations tab took it out of the inputs, and the next save of that form stored the lists without it. Exclude Clock from Vegas, disable it, change the brightness, re-enable it: Clock was scrolling in Vegas again and had moved to the end of the rotation. syncInputs now keeps the saved ids that have no row. In the order, each one keeps its saved slot and the rows fill the other slots in their current order, with rows not in the saved order last, as before. In the exclusions they follow the unchecked rows. Only string ids are carried over, once each: /config/main refuses a list holding anything else, which would block every later save of the tab. Tests: test/js/unit/test_plugin_order_list.js runs the shipped widget in a vm with a fake DOM (draw, reorder, include/exclude, the rotation list, junk ids) and is in run_all.js and the README. The durations DOM suite now reads only its own rows' ids from the input, since a rig's saved order can hold others. Co-Authored-By: Claude Opus 5.5 * fix(web): a restore reinstalls only the plugins that are missing POST /backup/restore with reinstall_plugins (the "Reinstall missing plugins" box) passed every plugin in the backup's plugins.json to install_plugin(). That replaces an installed copy with a fresh download, so a restore onto the same device re-downloaded every plugin inside the request. A plugin installed from its own URL is not in the registry, so its install returned False, plugins_failed set success to False, and the restore answered 500 "Restore incomplete ... plugins not reinstalled: " (shown as "Restore failed") with the plugin still installed and the config restored. Each plugin is now looked up first with the store's _existing_install, the same lookup install_plugin makes to decide a copy exists: the id, or an id the registry proves is the same plugin (aliases, the plugin_path name), and never a bare ledmatrix- folder (#686). One that is installed is recorded in result.skipped as "plugin: (installed)", which the page lists under Skipped; a missing one is installed as before. The list_installed_plugins docstring said every listed plugin is reinstalled and now says otherwise. Tests: TestInstalledPluginsAreNotReinstalled, with a mocked store (installed skipped, missing installed; an installed plugin the store can't install is not a failure) and with a real PluginStoreManager (a registry alias and a third-party install are skipped, a missing plugin installed). Co-Authored-By: Claude Opus 5.5 * fix(web): /config/main answers malformed JSON with a 400 save_main_config read a JSON body with request.get_json(), which raises Werkzeug's BadRequest for a body that does not parse (or an empty one sent as application/json). That happened inside the handler's try, so the catch-all answered 500 CONFIG_SAVE_FAILED with "Check file permissions on config directory" among its suggested fixes and logged a traceback at ERROR, for what was the caller's mistake. It now reads with get_json(silent=True), as save_raw_main_config does, and answers a sent-but-unparseable body with the same 400 {"status": "error", "message": "Invalid JSON in request body"}. An empty JSON body falls through to the existing 400 "No data provided". The change is limited to the lines that read the body. Tests: TestMalformedBody in test_api_v3_partial_main_save.py (the 400 and its shape, identical to /config/raw/main's, and nothing saved; the empty body). Co-Authored-By: Claude Opus 5.5 * fix(web): a restore that brings back fonts clears the font catalog cache GET /api/v3/fonts/catalog caches its answer as fonts_catalog for five minutes. Font upload and delete clear that entry (fonts.py), but POST /backup/restore copies user fonts into assets/fonts without touching it, so restored fonts were missing from the Fonts tab and every font picker until the cache expired. backup_restore now clears fonts_catalog when the result lists restored fonts (restore_backup records them as "fonts ()"). A restore that restored no fonts leaves the cache alone. Tests: TestFontsCatalogCache in test_api_v3_backup_restore.py. Co-Authored-By: Claude Opus 5.5 * fix(web): drop uninstalled plugins from the carried-over order and exclusions 2b34f254 made the plugin order list keep every saved id that has no row, so a disabled plugin keeps its rotation slot and Vegas exclusion. That also kept the ids of plugins that have since been uninstalled: they stayed in plugin_rotation_order and vegas_excluded_plugins for good, where before the next save of the tab dropped them. The widget already fetches /api/v3/plugins/installed, every installed plugin with its enabled flag, and draws only the enabled ones. It now keeps that response's full id set and carries over only saved ids that are installed but have no row (disabled). An id outside the set is dropped, as before. With no list, nothing is dropped: a failed request draws no rows and leaves the inputs as saved, and the carry-over keeps everything if the set was never filled. Tests: test/js/unit/test_plugin_order_list.js adds a disabled plugin kept while an uninstalled one is dropped (order and exclusions; fails on 2b34f254), and a failed plugin list leaving both inputs as saved. The CHANGELOG bullet and the README row say so. Co-Authored-By: Claude Opus 5.5 * test(js): register the order-list suite apart from other branches' suites Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 36 ++++ src/backup_manager.py | 5 +- test/js/README.md | 1 + test/js/dom/test_durations_page.js | 6 +- test/js/run_all.js | 1 + test/js/unit/test_plugin_order_list.js | 189 ++++++++++++++++++ test/test_api_v3_partial_main_save.py | 26 +++ .../test_api_v3_backup_restore.py | 105 ++++++++++ test/web_interface/test_api_v3_config_raw.py | 65 ++++++ web_interface/blueprints/api_v3/backup.py | 25 +++ web_interface/blueprints/api_v3/config.py | 6 +- web_interface/blueprints/pages_v3.py | 11 +- .../static/v3/js/widgets/plugin-order-list.js | 58 +++++- 13 files changed, 522 insertions(+), 12 deletions(-) create mode 100644 test/js/unit/test_plugin_order_list.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 52e08dcb..a9e7a23b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -677,6 +677,42 @@ policies are unchanged. notice is what shows next, and Vegas resumes after it; before, a rotation screen showed instead and the notice expired behind it. An active on-demand session still holds the panel until it ends. +- The Config Editor tab no longer shows API keys and tokens in plain + text. Its `config_secrets.json` editor (`/partials/raw-json`) was filled + with the file as it is on disk, so while the web login is off (the + default) anyone who could reach the port could read every credential, + although `GET /api/v3/config/secrets` masks them. The editor now shows the + same masked values. Saving it unchanged changes nothing, because the save + drops the masks and merges onto the stored file; to change a secret, + replace its mask. A list of secrets still needs every entry's real value + to be changed. The `config.json` editor is unchanged: its save writes the + file as given, so a mask there would be stored. +- A disabled plugin keeps its place in the rotation order and its Vegas + exclusion when the Display or Rotation & Durations tab is saved. The order + lists show enabled plugins only and rewrite their hidden inputs from those + rows as soon as they are drawn, so any save of either tab stored the lists + without the disabled plugin. Once re-enabled, it came back at the end of + the rotation and scrolling in Vegas again. A disabled plugin's saved id + now stays in its saved place (`widgets/plugin-order-list.js`); the id of + a plugin that is no longer installed is still dropped. +- Restoring a backup with "Reinstall missing plugins" installs only the + plugins that are missing. Every plugin the backup listed was sent to the + store's install, which replaces an installed copy with a fresh download, + so a restore onto the same device re-downloaded all of them in one + request. A plugin installed from its own URL is not in the registry, so + its "reinstall" failed and the restore answered "Restore failed" while + the plugin sat there installed. An installed plugin, found by the store's + own lookup (registry aliases included), is now listed under Skipped as + `plugin: (installed)`. +- `POST /api/v3/config/main` answers a JSON body that does not parse with + 400 `Invalid JSON in request body`, as `/config/raw/main` does, and an + empty JSON body with 400 `No data provided`. Both were a 500 + `CONFIG_SAVE_FAILED` suggesting file permissions and disk space, with a + traceback logged at ERROR: `get_json()` raised inside the handler's + catch-all. +- Fonts restored from a backup show up in the Fonts tab and the font + pickers straight away. The font catalog is cached for five minutes, and + upload and delete cleared it but a restore did not. - A game that goes live now takes over the panel within about a second. Live priority was only checked between screens, so a game that went live during a 30 s screen waited for that screen to end. The frame loops and the diff --git a/src/backup_manager.py b/src/backup_manager.py index bcd7b342..3d3d64c0 100644 --- a/src/backup_manager.py +++ b/src/backup_manager.py @@ -213,8 +213,9 @@ def list_installed_plugins(project_root: Path) -> List[Dict[str, Any]]: The plugins are the ``manifest.json`` files in the configured plugin directory (see :func:`_plugins_directory`), with the manifest's version; ``enabled`` is config.json's flag by the display's rule (a missing flag - is disabled). A restore reinstalls every listed plugin and takes enabled - state from the restored config.json, so ``enabled`` is informational. + is disabled). A restore installs each listed plugin that is missing and + takes enabled state from the restored config.json, so ``enabled`` is + informational. ``data/plugin_state.json`` is not read: it only ever repeated config's enabled flags and the manifests' versions, and is retired (nothing diff --git a/test/js/README.md b/test/js/README.md index c28d2de8..b7921ef1 100644 --- a/test/js/README.md +++ b/test/js/README.md @@ -50,6 +50,7 @@ server has none. | `unit/test_store_categories.js` | no | The store's category filter (sandbox): the template ships only All Categories, the rest come from the store's plugins (one per category whatever its case), choosing one filters to it, and a swapped-in select is refilled from the cache keeping the choice | | `unit/test_github_url_install.js` | no | Install Single Plugin (sandbox, the button as `plugins.html` ships it): no inline `onclick`, so a click or Enter sends exactly one `install-from-url` request and raises no error | | `unit/test_render_cards.js` | no | `renderInstalledCards` markup, both empty states, and HTML-escaping of hostile plugin metadata | +| `unit/test_plugin_order_list.js` | no | `widgets/plugin-order-list.js` (the Vegas and rotation order lists): a disabled plugin, which gets no row, keeps its slot in the saved order and its Vegas exclusion when the list rewrites its hidden inputs, around reordering and include/exclude; an uninstalled plugin's id is dropped, a failed plugin list leaves the inputs as saved, and only string ids are carried over, once each | | `unit/test_style_editor_element_keys.js` | no | `elementKeys()`/`styleRows()`/`positionRows()` from `widgets/style-editor.js`: every `customization.layout` entry gets exactly one row -- paired with its style element through core's `x-layout-key` (so `score` belongs to `score_text`, not a second row), or a position row of its own, leaves included -- since the widget claims the whole `layout` block from the generic fallback renderer | | `unit/test_style_editor_layout_leaf_columns.js` | no | `columnsFor()` from `widgets/style-editor.js`: a layout-only key whose own value is a leaf (no x/y sub-object, e.g. a `show_logo` toggle) gets a self-keyed column instead of a blank, uneditable row | | `unit/test_style_editor_layout_leaf_collision.js` | no | `columnsFor()` from `widgets/style-editor.js`: a layout-only leaf key still gets its own column even when its name collides with an unrelated element's style sub-field or another layout axis's sub-field | diff --git a/test/js/dom/test_durations_page.js b/test/js/dom/test_durations_page.js index 0dfc59e4..c6701768 100644 --- a/test/js/dom/test_durations_page.js +++ b/test/js/dom/test_durations_page.js @@ -98,7 +98,11 @@ const ok = (l, c, x) => c ? (pass++, console.log(' ok ' + l)) const lists = () => requests.filter(r => r.url === '/api/v3/plugins/installed').length; const $ = id => doc.getElementById(id); - const order = () => JSON.parse($('rotation_plugin_order_value').value || '[]'); + // The rows' ids, in order. The input also keeps saved ids that have no row + // (a disabled plugin's place, see test/js/unit/test_plugin_order_list.js), + // and the saved order comes from whatever config the server has. + const SHOWN = plugins.filter(p => p.enabled).map(p => p.id); + const order = () => JSON.parse($('rotation_plugin_order_value').value || '[]').filter(id => SHOWN.includes(id)); async function swap() { panel.dispatchEvent(new window.CustomEvent('htmx:beforeSwap', { bubbles: true, detail: { target: panel, shouldSwap: true } })); panel.innerHTML = partial; diff --git a/test/js/run_all.js b/test/js/run_all.js index 1e6a5437..81017a45 100755 --- a/test/js/run_all.js +++ b/test/js/run_all.js @@ -17,6 +17,7 @@ const fs = require('fs'); const BASE = process.env.BASE || 'http://localhost:5000'; const UNIT = ['unit/test_list_filter.js', 'unit/test_render_cards.js', + 'unit/test_plugin_order_list.js', 'unit/test_html_escaping.js', 'unit/test_style_editor_element_keys.js', 'unit/test_style_editor_layout_leaf_columns.js', 'unit/test_style_editor_layout_leaf_collision.js', diff --git a/test/js/unit/test_plugin_order_list.js b/test/js/unit/test_plugin_order_list.js new file mode 100644 index 00000000..f690a246 --- /dev/null +++ b/test/js/unit/test_plugin_order_list.js @@ -0,0 +1,189 @@ +// The shared plugin order list (widgets/plugin-order-list.js) keeps what it +// does not show. +// +// It lists enabled plugins only, and rewrites its hidden inputs from those +// rows as soon as it has drawn them. A disabled plugin's place in the order +// and its Vegas exclusion used to vanish from the inputs on that rewrite, so +// any later save of the Display or Rotation & Durations tab stored them +// without it: re-enabled, the plugin came back at the end of the rotation and +// scrolling in Vegas again. An uninstalled plugin's id is still dropped, as +// before, so the lists don't collect ids nothing can show. Runs the shipped +// widget in a vm with a minimal fake DOM -- no jsdom and no server needed, so +// it runs under test/test_js_unit_suites.py too. + +const fs = require('fs'); +const path = require('path'); +const vm = require('vm'); +const WIDGET = path.resolve(__dirname, '../../../web_interface/static/v3/js/widgets/plugin-order-list.js'); + +let pass = 0, fail = 0; +const ok = (label, cond, extra) => cond + ? (pass++, console.log(' ok ' + label)) + : (fail++, console.log(' FAIL ' + label + (extra !== undefined ? ' ' + JSON.stringify(extra) : ''))); +const same = (a, b) => JSON.stringify(a) === JSON.stringify(b); + +class FakeElement { + constructor(tag) { + this.tagName = tag.toUpperCase(); + this.children = []; + this.parent = null; + this.dataset = {}; + this.style = {}; + this.className = ''; + this.value = ''; + this.checked = false; + this.listeners = {}; + this._text = ''; + } + appendChild(child) { + if (child.parent) child.parent.children = child.parent.children.filter(c => c !== child); + child.parent = this; + this.children.push(child); + return child; + } + insertBefore(child, ref) { + if (!ref) return this.appendChild(child); + if (child.parent) child.parent.children = child.parent.children.filter(c => c !== child); + child.parent = this; + this.children.splice(this.children.indexOf(ref), 0, child); + return child; + } + get previousElementSibling() { + const siblings = this.parent ? this.parent.children : []; + return siblings[siblings.indexOf(this) - 1] || null; + } + get nextElementSibling() { + const siblings = this.parent ? this.parent.children : []; + const i = siblings.indexOf(this); + return i < 0 ? null : siblings[i + 1] || null; + } + set textContent(value) { this._text = value; this.children = []; } + get textContent() { return this._text; } + setAttribute() {} + focus() {} + addEventListener(type, fn) { (this.listeners[type] ||= []).push(fn); } + fire(type, event) { (this.listeners[type] || []).forEach(fn => fn.call(this, event || {})); } + descendants() { return this.children.flatMap(c => [c, ...c.descendants()]); } + querySelectorAll(selector) { + const cls = selector.replace(/^\./, ''); + return this.descendants().filter(e => e.className.split(/\s+/).includes(cls)); + } + querySelector(selector) { return this.querySelectorAll(selector)[0] || null; } +} + +/** Run the widget over `plugins` with the given saved inputs; resolves once it has drawn. */ +async function mount({ plugins, order, excluded, fetchFails }) { + const els = { + list: new FakeElement('div'), + order: Object.assign(new FakeElement('input'), { value: JSON.stringify(order) }), + }; + if (excluded !== undefined) { + els.excluded = Object.assign(new FakeElement('input'), { value: JSON.stringify(excluded) }); + } + const context = { + // The widget logs a failed list; expected there, so kept off the output. + console: fetchFails ? Object.assign({}, console, { error: () => {} }) : console, + window: {}, + document: { + getElementById: (id) => els[id] || null, + createElement: (tag) => new FakeElement(tag), + createTextNode: (text) => new FakeElement('#text'), + }, + fetch: () => (fetchFails ? Promise.reject(new Error('service restarting')) : Promise.resolve({ + json: () => Promise.resolve({ status: 'success', data: { plugins } }), + })), + }; + vm.createContext(context); + vm.runInContext(fs.readFileSync(WIDGET, 'utf8'), context); + context.window.PluginOrderList.init({ + containerId: 'list', orderInputId: 'order', + excludedInputId: excluded !== undefined ? 'excluded' : undefined, + }); + await new Promise(resolve => setTimeout(resolve, 0)); + const rows = () => els.list.querySelectorAll('.plugin-order-item'); + return { + rows, + rowIds: () => rows().map(r => r.dataset.pluginId), + order: () => JSON.parse(els.order.value), + excluded: () => JSON.parse(els.excluded.value), + row: (id) => rows().find(r => r.dataset.pluginId === id), + }; +} + +const PLUGINS = [ + { id: 'weather', name: 'Weather', enabled: true }, + { id: 'clock', name: 'Clock', enabled: false }, + { id: 'stocks', name: 'Stocks', enabled: true }, +]; + +(async () => { + console.log('\nVegas: a disabled plugin keeps its place and its exclusion'); + { + const t = await mount({ plugins: PLUGINS, order: ['weather', 'clock', 'stocks'], excluded: ['clock'] }); + ok('only enabled plugins get a row', same(t.rowIds(), ['weather', 'stocks']), t.rowIds()); + ok('drawing the list keeps the disabled plugin in the order, in its place', + same(t.order(), ['weather', 'clock', 'stocks']), t.order()); + ok('drawing the list keeps its exclusion', same(t.excluded(), ['clock']), t.excluded()); + + // Move Stocks up: the rows swap, and Clock stays in its saved slot. + const up = t.row('stocks').querySelectorAll('.plugin-order-move')[0]; + up.fire('click'); + ok('reordering the rows fills the other slots in the new order', + same(t.order(), ['stocks', 'clock', 'weather']), t.order()); + + const include = t.row('weather').querySelector('.plugin-order-include'); + include.checked = false; + include.fire('change'); + ok('unchecking a row adds it, and the disabled exclusion stays', + same([...t.excluded()].sort(), ['clock', 'weather']), t.excluded()); + include.checked = true; + include.fire('change'); + ok('checking it again removes only that one', same(t.excluded(), ['clock']), t.excluded()); + } + + console.log('\nRotation order: the same, without exclusions'); + { + const plugins = [ + { id: 'clock', enabled: true }, + { id: 'off', enabled: false }, + { id: 'weather', enabled: true }, + { id: 'new', enabled: true }, + ]; + const t = await mount({ plugins, order: ['clock', 'off', 'weather'] }); + ok('the disabled plugin keeps its slot; a plugin not in the saved order goes last', + same(t.order(), ['clock', 'off', 'weather', 'new']), t.order()); + } + + console.log('\nAn uninstalled plugin is dropped; a failed list keeps everything'); + { + const t = await mount({ plugins: PLUGINS, order: ['weather', 'gone', 'clock', 'stocks'], + excluded: ['gone', 'clock'] }); + ok('the disabled plugin is kept and the uninstalled one dropped from the order', + same(t.order(), ['weather', 'clock', 'stocks']), t.order()); + ok('and from the exclusions', same(t.excluded(), ['clock']), t.excluded()); + } + { + const t = await mount({ plugins: PLUGINS, order: ['weather', 'gone', 'clock', 'stocks'], + excluded: ['gone', 'clock'], fetchFails: true }); + // No installed list, so nothing can be told apart: no rows, and the + // inputs keep what was saved, uninstalled ids included. + ok('a failed plugin list draws no rows', t.rowIds().length === 0, t.rowIds()); + ok('and leaves the saved order as it was', + same(t.order(), ['weather', 'gone', 'clock', 'stocks']), t.order()); + ok('and the saved exclusions', same(t.excluded(), ['gone', 'clock']), t.excluded()); + } + + console.log('\nOnly what the server would accept is carried over'); + { + const t = await mount({ plugins: PLUGINS, order: ['weather', 7, 'clock', null, 'clock', 'stocks'], + excluded: ['clock', 3, 'clock'] }); + // /config/main refuses a list holding anything but strings, which would + // block every later Display save; a repeated id is kept once. + ok('non-string and repeated saved ids are dropped from the order', + same(t.order(), ['weather', 'clock', 'stocks']), t.order()); + ok('and from the exclusions', same(t.excluded(), ['clock']), t.excluded()); + } + + console.log(`\n${pass} passed, ${fail} failed`); + process.exit(fail ? 1 : 0); +})().catch(e => { console.error(e); process.exit(1); }); diff --git a/test/test_api_v3_partial_main_save.py b/test/test_api_v3_partial_main_save.py index 87c830cc..897d17d6 100644 --- a/test/test_api_v3_partial_main_save.py +++ b/test/test_api_v3_partial_main_save.py @@ -253,6 +253,32 @@ class TestVegasCycleDurations: assert saved['config']['display']['display_durations'] == {'clock': 45} +class TestMalformedBody: + """A JSON body that does not parse is the caller's mistake: a 400. + + get_json() raised Werkzeug's BadRequest inside the handler's try, whose + catch-all answered 500 CONFIG_SAVE_FAILED with "check file permissions" + advice and logged a traceback at ERROR. + """ + + def test_is_a_400_in_the_raw_routes_shape(self, api_v3_client, saved, api_v3_module): + api_v3_module.api_v3.config_manager.get_raw_file_content.return_value = {} + resp = api_v3_client.post('/api/v3/config/main', data='{not json', + content_type='application/json') + assert resp.status_code == 400 + assert resp.get_json() == {'status': 'error', 'message': 'Invalid JSON in request body'} + assert 'config' not in saved + raw = api_v3_client.post('/api/v3/config/raw/main', data='{not json', + content_type='application/json') + assert (raw.status_code, raw.get_json()) == (400, resp.get_json()) + + def test_an_empty_json_post_is_still_no_data(self, api_v3_client, saved): + resp = api_v3_client.post('/api/v3/config/main', data='', + content_type='application/json') + assert resp.status_code == 400 + assert resp.get_json()['message'] == 'No data provided' + + class TestRawSaveStartsAutoUpdateSetup: @pytest.fixture def raw_env(self, api_v3_module, monkeypatch): diff --git a/test/web_interface/test_api_v3_backup_restore.py b/test/web_interface/test_api_v3_backup_restore.py index a7f0f123..ca576e11 100644 --- a/test/web_interface/test_api_v3_backup_restore.py +++ b/test/web_interface/test_api_v3_backup_restore.py @@ -50,11 +50,13 @@ class FakeResult: self.plugins_to_install = plugins_to_install or [] self.plugins_installed = [] self.plugins_failed = [] + self.skipped = [] def to_dict(self): return { "success": self.success, "restored": self.restored, + "skipped": self.skipped, "errors": self.errors, "plugins_installed": self.plugins_installed, "plugins_failed": self.plugins_failed, @@ -286,6 +288,109 @@ class TestPluginReinstall: assert body["data"]["plugins_failed"][0]["error"] == "Store manager unavailable" +class TestInstalledPluginsAreNotReinstalled: + """"Reinstall missing plugins" installs only what is missing. + + Every plugin the backup listed went to install_plugin, which replaces an + installed copy with a fresh download: restoring onto the same device + re-downloaded all of them inside the request. One installed from its own + URL is not in the registry, so its "reinstall" returned False and the + whole restore answered 500 "Restore failed" with the plugin still there. + """ + + @staticmethod + def _installed(tmp_path, *names): + found = {} + for name in names: + (tmp_path / name).mkdir() + found[name] = tmp_path / name + return lambda plugin_id: found.get(plugin_id) + + def test_an_installed_plugin_is_skipped_and_a_missing_one_installed( + self, client, restore, tmp_path): + restore.return_value = FakeResult( + plugins_to_install=[{"plugin_id": "clock"}, {"plugin_id": "weather"}]) + store = api_v3.plugin_store_manager + store._existing_install.side_effect = self._installed(tmp_path, "clock") + store.install_plugin.return_value = True + response = post(client) + assert response.status_code == 200 + store.install_plugin.assert_called_once_with("weather") + data = response.get_json()["data"] + assert data["plugins_installed"] == ["weather"] + assert data["plugins_failed"] == [] + assert "plugin:clock (installed)" in data["skipped"] + + def test_an_installed_plugin_the_store_cannot_install_is_not_a_failure( + self, client, restore, tmp_path): + restore.return_value = FakeResult(plugins_to_install=[{"plugin_id": "my-3p"}]) + store = api_v3.plugin_store_manager + store._existing_install.side_effect = self._installed(tmp_path, "my-3p") + store.install_plugin.return_value = False + response = post(client) + assert response.status_code == 200 + assert response.get_json()["data"]["plugins_failed"] == [] + store.install_plugin.assert_not_called() + + @pytest.fixture + def real_store(self, tmp_path): + from src.plugin_system.store_manager import PluginStoreManager + plugins_dir = tmp_path / "plugin-repos" + for folder, manifest_id in (("ledmatrix-weather", "ledmatrix-weather"), + ("my-3p", "my-3p")): + (plugins_dir / folder).mkdir(parents=True) + (plugins_dir / folder / "manifest.json").write_text( + json.dumps({"id": manifest_id, "version": "1.0.0"})) + store = PluginStoreManager(plugins_dir=str(plugins_dir), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + # The official weather plugin's registry id differs from the id it + # installs under; my-3p was installed from its own URL. + registry = {"plugins": [{ + "id": "weather", "repo": "https://github.com/ChuckBuilds/ledmatrix-plugins", + "plugin_path": "plugins/ledmatrix-weather"}]} + store.registry_cache = registry + store.fetch_registry = lambda *a, **k: registry + store.install_plugin = MagicMock(return_value=True) + api_v3.plugin_store_manager = store + return store + + def test_with_the_real_store_aliases_and_third_party_installs_count( + self, client, restore, real_store): + restore.return_value = FakeResult(plugins_to_install=[ + {"plugin_id": "weather"}, {"plugin_id": "my-3p"}, {"plugin_id": "clock"}]) + response = post(client) + assert response.status_code == 200 + real_store.install_plugin.assert_called_once_with("clock") + skipped = response.get_json()["data"]["skipped"] + assert "plugin:weather (installed)" in skipped + assert "plugin:my-3p (installed)" in skipped + + +class TestFontsCatalogCache: + """The Fonts tab's catalog is cached for 5 minutes (fonts.py). + + Upload and delete clear it; a restore did not, so restored fonts were + missing from the Fonts tab and every font picker until it expired. + """ + + @pytest.fixture + def cached_catalog(self): + from web_interface.cache import delete_cached, get_cached, set_cached + set_cached('fonts_catalog', {'fonts': ['5x7.bdf']}, ttl_seconds=300) + yield lambda: get_cached('fonts_catalog', ttl_seconds=300) + delete_cached('fonts_catalog') + + def test_a_restore_that_restored_fonts_clears_it(self, client, restore, cached_catalog): + restore.return_value = FakeResult(restored=["config", "fonts (2)"]) + assert post(client).status_code == 200 + assert cached_catalog() is None + + def test_a_restore_without_fonts_keeps_it(self, client, restore, cached_catalog): + restore.return_value = FakeResult(restored=["config"]) + assert post(client).status_code == 200 + assert cached_catalog() == {'fonts': ['5x7.bdf']} + + class TestFailureReporting: def test_restore_errors_produce_a_500(self, client, restore): restore.return_value = FakeResult( diff --git a/test/web_interface/test_api_v3_config_raw.py b/test/web_interface/test_api_v3_config_raw.py index fe30749b..31529ab9 100644 --- a/test/web_interface/test_api_v3_config_raw.py +++ b/test/web_interface/test_api_v3_config_raw.py @@ -13,7 +13,9 @@ tmp_path so the assertions are against files on disk rather than mock calls. """ +import html import json +import re import sys from pathlib import Path from unittest.mock import MagicMock @@ -221,3 +223,66 @@ class TestRawEndpointsBypassSecretSeparation: env.client.post(MAIN, json={"weather": {"api_key": "PLAINTEXT-KEY"}}) # Nothing was moved aside into the secrets file. assert not env.secrets_file.exists() or "PLAINTEXT-KEY" not in env.secrets_file.read_text() + + +class TestConfigEditorRoundTrip: + """The Config Editor tab (/partials/raw-json) and the save it posts to. + + The secrets editor is shown masked, like GET /config/secrets: the page is + served to anyone who can reach the port while the optional web login is + off. Its save strips the masks and merges onto the stored file, so a + masked editor saved back as it is changes nothing. + """ + + STORED = { + "github": {"api_token": "ghp_REAL_TOKEN_1234"}, + "ledmatrix-weather": {"api_key": "WEATHER_KEY_abcdef", "units_id": 42}, + "calendar": {"accounts": [{"name": "home", "token": "CAL_TOKEN_9"}]}, + "youtube": {"api_key": "YOUR_YOUTUBE_API_KEY", "channel_secret": ""}, + } + REAL_VALUES = ("ghp_REAL_TOKEN_1234", "WEATHER_KEY_abcdef", "CAL_TOKEN_9") + + @pytest.fixture + def editor(self, env, monkeypatch): + from web_interface.blueprints import pages_v3 as pages_module + env.secrets_file.write_text(json.dumps(self.STORED)) + monkeypatch.setattr(pages_module.pages_v3, "config_manager", + env.config_manager, raising=False) + app = Flask(__name__, template_folder=str(project_root / "web_interface" / "templates")) + app.config["TESTING"] = True + app.register_blueprint(pages_module.pages_v3) + app.register_blueprint(api_v3, url_prefix="/api/v3") + return app.test_client() + + @staticmethod + def _secrets_textarea(client): + page = client.get("/partials/raw-json") + assert page.status_code == 200 + match = re.search(r'', + page.get_data(as_text=True), re.S) + assert match, "the secrets editor is missing from the partial" + return html.unescape(match.group(1)) + + def test_the_editor_shows_no_secret_value(self, editor): + text = self._secrets_textarea(editor) + for value in self.REAL_VALUES: + assert value not in text + shown = json.loads(text) + assert shown["github"]["api_token"] == "\u2022" * 8 + # Same shape as the file, and "not set" still reads as not set. + assert shown["calendar"]["accounts"][0]["name"] == "\u2022" * 8 + assert shown["youtube"] == {"api_key": "YOUR_YOUTUBE_API_KEY", "channel_secret": ""} + + def test_saving_it_back_unchanged_keeps_every_secret(self, editor, env): + shown = json.loads(self._secrets_textarea(editor)) + response = editor.post(SECRETS, json=shown) + assert response.status_code == 200 + assert json.loads(env.secrets_file.read_text()) == self.STORED + + def test_editing_one_secret_changes_only_that_one(self, editor, env): + shown = json.loads(self._secrets_textarea(editor)) + shown["ledmatrix-weather"]["api_key"] = "NEW_WEATHER_KEY" + assert editor.post(SECRETS, json=shown).status_code == 200 + expected = json.loads(json.dumps(self.STORED)) + expected["ledmatrix-weather"]["api_key"] = "NEW_WEATHER_KEY" + assert json.loads(env.secrets_file.read_text()) == expected diff --git a/web_interface/blueprints/api_v3/backup.py b/web_interface/blueprints/api_v3/backup.py index 2184861b..ba5191b0 100644 --- a/web_interface/blueprints/api_v3/backup.py +++ b/web_interface/blueprints/api_v3/backup.py @@ -16,6 +16,7 @@ import web_interface.blueprints.api_v3 as _pkg # as module attributes, and a value binding would not see the patch. # Several are also called from helpers that live in __init__, so the # package is the only patch point that covers every caller. +from web_interface.cache import delete_cached @api_v3.route('/backup/preview', methods=['GET']) @@ -85,6 +86,17 @@ _RESTORE_OPTION_KEYS = frozenset(( 'restore_config', 'restore_secrets', 'restore_wifi', 'restore_fonts', 'restore_plugin_uploads', 'reinstall_plugins', )) +def _installed_path(psm, plugin_id): + """Where the store finds ``plugin_id`` installed, or None. + + The same lookup install_plugin makes to decide that a copy exists: the + id, or an id the registry proves is the same plugin (``aliases``, the + ``plugin_path`` name), never a bare ``ledmatrix-`` folder. + """ + found = psm._existing_install(plugin_id) + return found if isinstance(found, Path) and found.exists() else None + + @api_v3.route('/backup/restore', methods=['POST']) def backup_restore(): """Restore a backup ZIP with optional RestoreOptions.""" @@ -134,6 +146,10 @@ def backup_restore(): os.unlink(tmp_path) except OSError: pass + # Restored fonts reach the Fonts tab through a catalog cached for five + # minutes (fonts.py); upload and delete clear it, and so must this. + if any(str(item).startswith('fonts') for item in result.restored): + delete_cached('fonts_catalog') # Reinstall plugins if requested and store manager available if options.reinstall_plugins and result.plugins_to_install: @@ -143,6 +159,15 @@ def backup_restore(): if not pid: continue try: + # Only what is missing. install_plugin replaces an installed + # copy with a fresh download, so restoring onto the same + # device re-downloaded every plugin, and one installed from + # its own URL (not in the registry) "failed" and failed the + # whole restore while it sat there installed. The store's + # own lookup, so registry aliases count as installed too. + if psm and _installed_path(psm, pid) is not None: + result.skipped.append(f'plugin:{pid} (installed)') + continue if psm and hasattr(psm, 'install_plugin'): ok = psm.install_plugin(pid) if ok: diff --git a/web_interface/blueprints/api_v3/config.py b/web_interface/blueprints/api_v3/config.py index 41720c96..f2265370 100644 --- a/web_interface/blueprints/api_v3/config.py +++ b/web_interface/blueprints/api_v3/config.py @@ -507,7 +507,11 @@ def save_main_config(): # Try to get JSON data first, fallback to form data data = None if request.is_json: - data = request.get_json() + # silent=True, as in save_raw_main_config: get_json() raised + # Werkzeug's BadRequest into the catch-all below, a 500. + data = request.get_json(silent=True) + if data is None and request.get_data(): + return jsonify({'status': 'error', 'message': 'Invalid JSON in request body'}), 400 if data is not None and not isinstance(data, dict): return jsonify({'status': 'error', 'message': 'Request body must be a JSON object'}), 400 else: diff --git a/web_interface/blueprints/pages_v3.py b/web_interface/blueprints/pages_v3.py index 0c32779e..260ad7be 100644 --- a/web_interface/blueprints/pages_v3.py +++ b/web_interface/blueprints/pages_v3.py @@ -11,7 +11,7 @@ _SAFE_PLUGIN_ID_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}$') _SAFE_WEB_UI_FILE_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}\.html$') _SAFE_WIDGET_NAME_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}$') _SAFE_WIDGET_SCRIPT_RE = re.compile(r'^[a-zA-Z0-9_-]{1,64}\.js$') -from src.web_interface.secret_helpers import mask_secret_fields +from src.web_interface.secret_helpers import mask_all_secret_values, mask_secret_fields from src.plugin_system.schema_manager import plugin_config_defaults, prepare_plugin_config from src.common.path_safety import resolve_under, safe_path_component from src.pi5_matrix_support import is_raspberry_pi_5 @@ -623,9 +623,14 @@ def _load_raw_json_partial(): main_config_data = pages_v3.config_manager.get_raw_file_content('main') # The web login section (password and token hashes) is managed in # General > Security, never in this editor; its save keeps it. + # The rest is masked, as GET /api/v3/config/secrets masks it: this + # page is served to anyone who can reach the port while the web + # login is off, and it was handing them every credential in the + # file. The save strips the masks and merges onto the stored file + # (save_raw_secrets_config), so a value left masked stays as it is. from web_interface.auth import strip_auth_section - secrets_config_data = strip_auth_section( - pages_v3.config_manager.get_raw_file_content('secrets')) + secrets_config_data = mask_all_secret_values(strip_auth_section( + pages_v3.config_manager.get_raw_file_content('secrets'))) main_config_json = json.dumps(main_config_data, indent=4) secrets_config_json = json.dumps(secrets_config_data, indent=4) diff --git a/web_interface/static/v3/js/widgets/plugin-order-list.js b/web_interface/static/v3/js/widgets/plugin-order-list.js index 184c99b7..34b7f68b 100644 --- a/web_interface/static/v3/js/widgets/plugin-order-list.js +++ b/web_interface/static/v3/js/widgets/plugin-order-list.js @@ -18,7 +18,9 @@ * }); * * The container re-renders from /api/v3/plugins/installed each init; the - * hidden input(s) must already hold the saved order/exclusions (JSON). + * hidden input(s) must already hold the saved order/exclusions (JSON). Saved + * ids of disabled plugins (installed, but without a row) stay in them, in + * their saved places; ids of plugins no longer installed are dropped. */ (function() { 'use strict'; @@ -39,17 +41,60 @@ const excludedInput = options.excludedInputId ? document.getElementById(options.excludedInputId) : null; if (!container || !orderInput) return; + // The saved lists as the inputs held them when the rows were drawn. + // Only enabled plugins get a row, and the inputs are rewritten from + // the rows, so a disabled plugin's place and exclusion have to be + // carried over from these: dropped, the next Display or Durations + // save stored the lists without it, and once re-enabled it came back + // at the end of the rotation and scrolling in Vegas again. + let savedOrder = []; + let savedExcluded = []; + // Every installed plugin's id, enabled or not, from the same + // response. A saved id outside it belongs to an uninstalled plugin + // and is dropped, as every save used to; without the list, nothing + // is dropped. + let installedIds = null; + + // Saved ids of installed plugins with no row, once each. Only + // strings: /config/main refuses a list holding anything else, which + // would block every save. + function unlisted(saved, rowIds) { + const seen = new Set(rowIds); + return saved.filter(id => { + if (typeof id !== 'string' || seen.has(id)) return false; + if (installedIds && !installedIds.has(id)) return false; + seen.add(id); + return true; + }); + } + function syncInputs() { - const order = []; + const rowIds = []; const excluded = []; container.querySelectorAll('.plugin-order-item').forEach(item => { const pluginId = item.dataset.pluginId; - order.push(pluginId); + rowIds.push(pluginId); const checkbox = item.querySelector('.plugin-order-include'); if (checkbox && !checkbox.checked) excluded.push(pluginId); }); - orderInput.value = JSON.stringify(order); - if (excludedInput) excludedInput.value = JSON.stringify(excluded); + // An id without a row keeps its saved slot; the rows fill the + // other slots in their current order, and any rows left over + // (plugins not in the saved order) go last. + const kept = new Set(unlisted(savedOrder, rowIds)); + const order = []; + let next = 0; + savedOrder.forEach(id => { + if (kept.has(id)) { + order.push(id); + kept.delete(id); + } else if (rowIds.includes(id) && next < rowIds.length) { + order.push(rowIds[next++]); + } + }); + orderInput.value = JSON.stringify(order.concat(rowIds.slice(next))); + if (excludedInput) { + excludedInput.value = JSON.stringify(excluded.concat(unlisted(savedExcluded, rowIds))); + } } function setupDragAndDrop() { @@ -104,6 +149,7 @@ .then(data => { const allPlugins = (data.data && data.data.plugins) || data.plugins || []; const plugins = allPlugins.filter(p => p.enabled); + installedIds = new Set(allPlugins.map(p => p && p.id)); if (plugins.length === 0) { const empty = document.createElement('p'); empty.className = 'text-sm text-gray-500 italic'; @@ -125,6 +171,8 @@ // (e.g. a saved value of "null"); normalize to arrays. if (!Array.isArray(currentOrder)) currentOrder = []; if (!Array.isArray(excluded)) excluded = []; + savedOrder = currentOrder; + savedExcluded = excluded; // Saved order first, then any newly enabled plugins. const orderedPlugins = []; From 5ad5e9aa595ef0097583045fec1433c74ea487ed Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Sat, 3 Oct 2026 22:30:39 -0400 Subject: [PATCH 03/37] fix(web): plugin action params, refused on-demand starts, pending-operation 500, double-click 409, binary static files (#744) * fix(web): pass plugin action params to the wrapper on stdin POST /api/v3/plugins/action runs a plugin's script through a generated Python wrapper, and the params went into that wrapper's source as `params = `. JSON true, false and null are undefined names in Python, so any params holding one made the wrapper die with a NameError before the script ran, and the route answered "Action failed". The plugin file manager's category toggle sends {"category_name": ..., "enabled": true}, so of-the-day's category toggle failed every time. The wrapper now reads the params from its own stdin (json.loads) and the route passes them there; nothing taken from the request is written into the generated source any more. The script's side is unchanged: the same json.dumps(params) on its stdin, LEDMATRIX_ROOT set, stdout parsed. Co-Authored-By: Claude Opus 5.5 * fix(web): a refused on-demand start leaves no request in the mailbox POST /api/v3/display/on-demand/start delivered the request (control socket, else the file mailbox) before it checked the display service. With the service stopped the socket is absent, so the request went to the mailbox; the route then answered 400 "Display service is not running" when start_service was off, or 500 "Failed to start display service" when the start failed. The display reads that mailbox with max_age=3600 and never checks a request's timestamp, so the next time it was started it ran the refused request, pinned if asked. The service is now checked before anything is delivered, and nothing is posted when start_service is off and the service is down. When the start itself fails, the request is withdrawn from the mailbox, but only while the mailbox still holds this request_id (the compare-before-delete the display's _consume_on_demand_request uses), so a newer request posted in the meantime is left for the display. Co-Authored-By: Claude Opus 5.5 * fix(web): a pending plugin operation's status no longer answers 500 PluginOperationQueue.enqueue_operation stores the operation's callback in operation.parameters['_callback'], and the worker pops it only when it runs the operation. PluginOperation.to_dict() returned parameters as they were, so GET /api/v3/plugins/operation/ for an operation still waiting in the queue (an install queued behind another plugin's) handed jsonify a function and answered 500 "A system error occurred" on every poll until the worker reached it. to_dict() now leaves out parameters whose name starts with "_". The operation itself keeps its callback for the worker; every other field of the answer, and the operation-history records (a different class), are unchanged. Co-Authored-By: Claude Opus 5.5 * fix(web): a second install or uninstall of a busy plugin is a 409 PluginOperationQueue.enqueue_operation raises ValueError when the plugin already has an operation waiting or running. /plugins/install did not catch it, so a double-clicked Install (the button is never disabled) answered 500 "An error occurred; see logs for details" from the blueprint's catch-all while the first install carried on. /plugins/uninstall caught it in its own catch-all: a 500 "Failed to uninstall plugin", plus an "uninstall failed" operation-history record for an uninstall that never started. Both routes now enqueue through _enqueue_or_conflict, which turns the queue's refusal into a 409 PLUGIN_OPERATION_CONFLICT naming the plugin, and records nothing. Co-Authored-By: Claude Opus 5.5 * fix(web): serve binary plugin static files instead of a 500 GET /api/v3/plugins//static/ read every file with open(..., 'r', encoding='utf-8') and returned the decoded text, so any binary file -- a plugin icon or preview image, which is what the REST API reference says the route is for -- raised UnicodeDecodeError and answered 500. The file is now sent with send_file, as bytes. HTML, JavaScript, CSS and JSON keep the content types the route always set, and other text keeps text/plain; anything else gets the type mimetypes knows it by (image/png for a .png). The plugin id and path validation and the resolve_under containment check are untouched. Co-Authored-By: Claude Opus 5.5 * fix(web): a socket-acknowledged on-demand start is a success cdaeb385 checked the systemd unit before delivering the on-demand request, so a display run by hand or in the emulator (no active unit) with start_service off now got nothing, where before the request went over the control socket and took effect behind a 400. A socket acknowledgement is the display itself saying it is running and has the request queued, so it is the better witness than systemd. The request is delivered first again. When the display acknowledged it over the socket, the route answers success without consulting systemd for the "not running" 400 and without starting the unit (with start_service on it tried to start a second display beside the one that answered); the service is still reported the way _ensure_display_service_running reports a running one. When it went to the mailbox, the 400 (service down, start_service off) and the failed-start 500 both withdraw this request_id from the mailbox, leaving a newer request alone, so neither refusal runs later. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 39 ++++++++ src/plugin_system/operation_types.py | 11 ++- test/test_api_v3_on_demand_restart.py | 97 +++++++++++++++++++ test/test_api_v3_operation_status_pending.py | 83 ++++++++++++++++ test/test_api_v3_plugin_action_params.py | 93 ++++++++++++++++++ test/test_api_v3_plugin_operation_conflict.py | 89 +++++++++++++++++ test/test_api_v3_plugin_static_files.py | 73 ++++++++++++++ web_interface/blueprints/api_v3/display.py | 50 +++++++++- .../blueprints/api_v3/plugin_assets.py | 22 +++-- .../blueprints/api_v3/plugin_store.py | 39 ++++++-- web_interface/blueprints/api_v3/plugins.py | 9 +- 11 files changed, 582 insertions(+), 23 deletions(-) create mode 100644 test/test_api_v3_operation_status_pending.py create mode 100644 test/test_api_v3_plugin_action_params.py create mode 100644 test/test_api_v3_plugin_operation_conflict.py create mode 100644 test/test_api_v3_plugin_static_files.py diff --git a/CHANGELOG.md b/CHANGELOG.md index a9e7a23b..68ee3d9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -722,6 +722,45 @@ policies are unchanged. screen showed first and the game came after it. Each check also asks each plugin `has_live_content()` once, where a plugin registered under several modes used to be asked once per mode. +- A plugin action whose params hold `true`, `false` or `null` runs again. + `/api/v3/plugins/action` wrote the params into the source of the wrapper + that runs the plugin's script, and those JSON words are not Python, so the + wrapper stopped with a NameError and the action answered "Action failed". + The plugin file manager's category toggle sends `"enabled": true`, so + turning a category on or off in of-the-day always failed. The params now + reach the wrapper on its stdin; the script still receives them as JSON on + its own stdin, as before. +- An on-demand request that `/api/v3/display/on-demand/start` refuses no + longer runs later. With the display stopped the request goes to the + display's mailbox, and the display reads that mailbox for an hour without + looking at a request's age. So with "Start display service" unticked, the + answer was "Display service is not running", yet the next time the + display was started it ran that plugin, pinned if the request said so. + The same happened after "Failed to start display service". On either + refusal the route now takes its request back out of the mailbox, unless a + newer one has replaced it. A request the display acknowledges over the + control socket is now a success whatever systemd reports: a display run + by hand or in the emulator was told "not running" for a request it had + already taken, and with "Start display service" ticked the route tried to + start the service beside it. +- `/api/v3/plugins/operation/` reports a queued operation as `pending` + instead of answering 500. The queue keeps an operation's callback among + its parameters until it runs, and the status route tried to send that + function as JSON. An install queued behind another plugin's install + failed every status poll until the first one finished. Parameters whose + name starts with `_` are internal and are no longer in the answer. +- A second click on Install while that plugin is still installing, or an + Uninstall during its install, now answers 409 "already has an install, + update or uninstall in progress" instead of 500 "An error occurred". The + first operation carried on either way. The uninstall route also stopped + recording a failed uninstall in the operation history for an uninstall + that never started. +- `/api/v3/plugins//static/` serves images and other + binary files. It opened every file as UTF-8 text, so a plugin's icon or + preview image answered 500 `UnicodeDecodeError`. Files are now sent as + they are on disk, an image with its own content type; HTML, JavaScript, + CSS, JSON and other text keep the types they had. The path checks are + unchanged. - The display schedule turns the panel off at exactly the end time. A window now runs from its start time up to, but not including, its end time: with 07:00-23:00 the panel is on at 07:00 and off at 23:00. Before, the end diff --git a/src/plugin_system/operation_types.py b/src/plugin_system/operation_types.py index 7181263b..950b0fd3 100644 --- a/src/plugin_system/operation_types.py +++ b/src/plugin_system/operation_types.py @@ -48,12 +48,19 @@ class PluginOperation: completed_at: Optional[datetime] = None def to_dict(self) -> Dict[str, Any]: - """Convert operation to dictionary for serialization.""" + """Convert operation to dictionary for serialization. + + Parameters whose name starts with ``_`` are internal and left out: + PluginOperationQueue keeps the operation's callback there as + ``_callback`` until its worker runs it, and a pending operation's + status answered 500 because that function cannot be serialized. + """ return { 'operation_id': self.operation_id, 'operation_type': self.operation_type.value, 'plugin_id': self.plugin_id, - 'parameters': self.parameters, + 'parameters': {key: value for key, value in self.parameters.items() + if not str(key).startswith('_')}, 'status': self.status.value, 'progress': self.progress, 'message': self.message, diff --git a/test/test_api_v3_on_demand_restart.py b/test/test_api_v3_on_demand_restart.py index 17d31edd..9b889f83 100644 --- a/test/test_api_v3_on_demand_restart.py +++ b/test/test_api_v3_on_demand_restart.py @@ -37,6 +37,7 @@ from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401 START_URL = "/api/v3/display/on-demand/start" STOP_URL = "/api/v3/display/on-demand/stop" MAILBOX = "display_on_demand_request" +DISPLAY = "web_interface.blueprints.api_v3.display" @pytest.fixture @@ -150,6 +151,102 @@ class TestStartWhileTheServiceIsStopped: assert response.get_json()["status"] == "error" +class _Mailbox: + """The CacheManager calls the routes make, over a dict.""" + + def __init__(self): + self.entries = {} + + def set(self, key, value, ttl=None): + self.entries[key] = value + + def get(self, key, max_age=300, memory_ttl=None): + return self.entries.get(key) + + def delete(self, key): + self.entries.pop(key, None) + + +class TestARefusedStartLeavesNoRequestBehind: + """A start the route answers with an error must not run later. + + The request was posted (to the mailbox, with the display stopped) before + the route refused it, and the display reads the mailbox for an hour + without looking at a request's age. So "Display service is not running" + (start_service off) or "Failed to start display service" left the + request waiting, and the next time the display started -- minutes later, + by hand -- it ran that plugin, pinned if the request said so. + + A socket acknowledgement is the other side of it: the display answered, + so it is running and has the request, whatever systemd says (a display + run by hand or in the emulator has no active unit). That is a success, + not "not running", and no unit is started beside it. + """ + + @pytest.fixture + def mailbox(self, api_v3_module, service): + box = _Mailbox() + api_v3_module.api_v3.cache_manager = box + service["state"]["active"] = False + return box + + @pytest.mark.parametrize("body", [ + {"plugin_id": "weather", "start_service": False}, + {"plugin_id": "weather"}, # start_service defaults on + ]) + def test_a_socket_ack_is_a_success_whatever_systemd_says( + self, api_v3_client, service, mailbox, body): + with patch(f"{DISPLAY}.control_client.on_demand_start", + side_effect=lambda request_id, *a: {"accepted": True}): + response = api_v3_client.post(START_URL, json=body) + assert response.status_code == 200, response.get_json() + assert response.get_json()["data"]["transport"] == "socket" + assert MAILBOX not in mailbox.entries + assert _systemctl_verbs(service["systemctl"]) == [], ( + "a unit was started beside a display that answered the socket") + + def test_without_start_service_the_request_is_taken_back( + self, api_v3_client, service, mailbox): + response = api_v3_client.post(START_URL, json={ + "plugin_id": "weather", "pinned": True, "start_service": False}) + assert response.status_code == 400 + assert response.get_json()["status"] == "error" + assert MAILBOX not in mailbox.entries + + def test_a_start_that_fails_takes_its_request_back(self, api_v3_client, service, mailbox): + service["systemctl"].side_effect = lambda args: { + "returncode": 1, "stdout": "", "stderr": "denied"} + response = api_v3_client.post(START_URL, json={"plugin_id": "weather"}) + assert response.status_code == 500 + assert MAILBOX not in mailbox.entries + + def test_a_newer_request_is_left_alone_on_the_400(self, api_v3_client, service, mailbox): + newer = {"request_id": "someone-else", "action": "start", "plugin_id": "clock"} + + def stopped_and_another_post_lands(*args): + mailbox.entries[MAILBOX] = newer + return {"active": False} + + with patch(f"{DISPLAY}._get_display_service_status", + side_effect=stopped_and_another_post_lands): + response = api_v3_client.post(START_URL, json={ + "plugin_id": "weather", "start_service": False}) + assert response.status_code == 400 + assert mailbox.entries[MAILBOX] is newer + + def test_a_newer_request_is_left_alone_on_the_500(self, api_v3_client, service, mailbox): + newer = {"request_id": "someone-else", "action": "start", "plugin_id": "clock"} + + def start_fails_after_another_post(args): + mailbox.entries[MAILBOX] = newer + return {"returncode": 1, "stdout": "", "stderr": "denied"} + + service["systemctl"].side_effect = start_fails_after_another_post + response = api_v3_client.post(START_URL, json={"plugin_id": "weather"}) + assert response.status_code == 500 + assert mailbox.entries[MAILBOX] is newer + + class TestStop: def test_stop_posts_a_stop_request_and_leaves_the_service_running( self, api_v3_client, service): diff --git a/test/test_api_v3_operation_status_pending.py b/test/test_api_v3_operation_status_pending.py new file mode 100644 index 00000000..d107b566 --- /dev/null +++ b/test/test_api_v3_operation_status_pending.py @@ -0,0 +1,83 @@ +"""GET /api/v3/plugins/operation/ answers for an operation still waiting. + +PluginOperationQueue keeps an operation's callback in its parameters, under +``_callback``, until the worker takes it to run. PluginOperation.to_dict() +returned the parameters as they were, so for a pending operation the route +handed jsonify a function and answered 500 "A system error occurred". That +is every poll of an install queued behind another plugin's: the second of +two installs read as broken until the first one finished. +""" + +import json +import sys +import threading +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +from src.plugin_system.operation_queue import PluginOperationQueue # noqa: E402 +from src.plugin_system.operation_types import ( # noqa: E402 + OperationType, PluginOperation, +) + + +def _callback(op): + return {"success": True, "message": "done"} + + +class TestToDict: + def test_private_parameters_are_left_out(self): + op = PluginOperation(OperationType.INSTALL, "demo", + parameters={"_callback": _callback, "branch": "main"}) + assert op.to_dict()["parameters"] == {"branch": "main"} + json.dumps(op.to_dict()) # serializable + + def test_the_operation_keeps_its_callback_for_the_worker(self): + op = PluginOperation(OperationType.INSTALL, "demo", + parameters={"_callback": _callback}) + op.to_dict() + assert op.parameters["_callback"] is _callback + + def test_the_other_fields_are_unchanged(self): + op = PluginOperation(OperationType.UNINSTALL, "demo", operation_id="op-1") + assert op.to_dict() == { + "operation_id": "op-1", "operation_type": "uninstall", "plugin_id": "demo", + "parameters": {}, "status": "pending", "progress": 0.0, "message": "", + "error": None, "result": None, + "created_at": op.created_at.isoformat(), "started_at": None, + "completed_at": None, + } + + +class TestTheRoute: + @pytest.fixture + def busy_queue(self, api_v3_module): + """A real queue whose worker is held by another plugin's operation.""" + queue = PluginOperationQueue(max_history=10) + api_v3_module.api_v3.operation_queue = queue + started, release = threading.Event(), threading.Event() + + def blocker(op): + started.set() + release.wait(10) + return {"success": True, "message": "done"} + + queue.enqueue_operation(OperationType.INSTALL, "busy", operation_callback=blocker) + assert started.wait(5) + yield queue + release.set() + queue.shutdown() + + def test_a_pending_operation_reports_pending(self, api_v3_client, busy_queue): + op_id = busy_queue.enqueue_operation( + OperationType.INSTALL, "demo", operation_callback=_callback) + response = api_v3_client.get(f"/api/v3/plugins/operation/{op_id}") + assert response.status_code == 200, response.get_json() + data = response.get_json()["data"] + assert data["status"] == "pending" + assert data["plugin_id"] == "demo" + assert "_callback" not in data["parameters"] diff --git a/test/test_api_v3_plugin_action_params.py b/test/test_api_v3_plugin_action_params.py new file mode 100644 index 00000000..a890290a --- /dev/null +++ b/test/test_api_v3_plugin_action_params.py @@ -0,0 +1,93 @@ +"""POST /api/v3/plugins/action hands ``params`` to the plugin's script intact. + +The route runs the script through a generated wrapper, and the params went +into that wrapper as Python source: ``params = {json.dumps(params)}``. JSON is +not Python. ``true``, ``false`` and ``null`` are undefined names there, so any +params holding a boolean or a null died with a NameError before the script +ran. The plugin file manager's category toggle sends ``{"category_name": ..., +"enabled": true}``, so of-the-day's category toggle failed every time with +"Action failed". + +The script's side of the contract is unchanged and pinned here too: the +params arrive on stdin as one JSON document, LEDMATRIX_ROOT is set, and what +the script prints to stdout is what the route parses. +""" + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +ACTION_URL = "/api/v3/plugins/action" + +# The action script: report what it was handed, as JSON on stdout. +ECHO_SCRIPT = ( + "import json, os, sys\n" + "raw = sys.stdin.read()\n" + "print(json.dumps({'status': 'success', 'got': json.loads(raw),\n" + " 'root': os.environ.get('LEDMATRIX_ROOT')}))\n" +) + + +@pytest.fixture +def echo_plugin(tmp_path, api_v3_module, monkeypatch): + plugin_dir = tmp_path / "demo" + plugin_dir.mkdir() + (plugin_dir / "manifest.json").write_text(json.dumps({ + "id": "demo", + "web_ui_actions": [{"id": "toggle", "type": "script", "script": "echo.py"}], + }), encoding="utf-8") + (plugin_dir / "echo.py").write_text(ECHO_SCRIPT, encoding="utf-8") + api_v3_module.api_v3.plugin_catalog.get_plugin_directory.return_value = str(plugin_dir) + + # The route runs `python3`; use this interpreter, so the test does not + # depend on what that name resolves to here. + real_run = subprocess.run + + def run(cmd, *args, **kwargs): + if isinstance(cmd, list) and cmd and cmd[0] == "python3": + cmd = [sys.executable] + cmd[1:] + return real_run(cmd, *args, **kwargs) + + monkeypatch.setattr(subprocess, "run", run) + return plugin_dir + + +@pytest.mark.parametrize("params", [ + {"category_name": "jokes", "enabled": True}, # the file manager's toggle + {"category_name": "jokes", "enabled": False}, + {"filename": None}, + {"nested": {"list": [1, None, True, 2.5], "empty": {}}}, + {"text": "café ✓ \U0001F600"}, + {"text": "he said \"hi\" and 'bye' \\ ''' \"\"\" \n\t end"}, +], ids=["true", "false", "null", "nested", "unicode", "quotes"]) +def test_the_script_receives_the_params_it_was_sent(api_v3_client, echo_plugin, params): + response = api_v3_client.post(ACTION_URL, json={ + "plugin_id": "demo", "action_id": "toggle", "params": params}) + body = response.get_json() + assert response.status_code == 200, body + assert body["got"] == params + + +def test_a_param_cannot_run_code_in_the_wrapper(api_v3_client, echo_plugin, tmp_path): + marker = tmp_path / "PWNED" + hostile = "\"}\nopen(%r, 'w').write('ran')\n#" % str(marker) + params = {"name": hostile, "flag": True} + response = api_v3_client.post(ACTION_URL, json={ + "plugin_id": "demo", "action_id": "toggle", "params": params}) + assert response.status_code == 200, response.get_json() + assert response.get_json()["got"] == params + assert not marker.exists(), "a param value ran as code" + + +def test_the_script_still_gets_ledmatrix_root(api_v3_client, echo_plugin, api_v3_module): + response = api_v3_client.post(ACTION_URL, json={ + "plugin_id": "demo", "action_id": "toggle", "params": {"enabled": True}}) + assert response.status_code == 200, response.get_json() + assert response.get_json()["root"] == str(api_v3_module.PROJECT_ROOT) diff --git a/test/test_api_v3_plugin_operation_conflict.py b/test/test_api_v3_plugin_operation_conflict.py new file mode 100644 index 00000000..eacd622c --- /dev/null +++ b/test/test_api_v3_plugin_operation_conflict.py @@ -0,0 +1,89 @@ +"""A second install or uninstall while one is in progress is a 409, not a 500. + +PluginOperationQueue refuses a second operation for a plugin that already +has one waiting or running (test_operation_queue_pending_and_trim.py), and +says so by raising ValueError. /plugins/install let that escape to the +blueprint's catch-all, so a double-clicked Install answered 500 "An error +occurred; see logs for details" while the first install carried on. +/plugins/uninstall caught it in its own catch-all: a 500 "Failed to +uninstall plugin", and an "uninstall failed" entry in the operation +history for an uninstall that never started. +""" + +import sys +import threading +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +from src.plugin_system.operation_queue import PluginOperationQueue # noqa: E402 + +INSTALL = "/api/v3/plugins/install" +UNINSTALL = "/api/v3/plugins/uninstall" + + +@pytest.fixture +def installing(api_v3_module, tmp_path): + """A real queue with an install of "clock" running and held there.""" + queue = PluginOperationQueue(max_history=10) + api_v3_module.api_v3.operation_queue = queue + started, release = threading.Event(), threading.Event() + + def slow_install(plugin_id, branch=None): + started.set() + release.wait(10) + return True + + store = api_v3_module.api_v3.plugin_store_manager + store.install_plugin.side_effect = slow_install + store.get_registry_info.return_value = None + store.plugins_dir = str(tmp_path) + api_v3_module.api_v3.plugin_catalog.get_plugin_directory.return_value = None + yield {"queue": queue, "started": started, "store": store} + release.set() + queue.shutdown() + + +def _start_first_install(client, installing): + response = client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 200, response.get_json() + assert installing["started"].wait(5) + + +def _failed_history(api_v3_module): + return [c for c in api_v3_module.api_v3.operation_history.record_operation.call_args_list + if c.kwargs.get("status") == "failed"] + + +def test_a_second_install_click_is_a_conflict(api_v3_client, api_v3_module, installing): + _start_first_install(api_v3_client, installing) + response = api_v3_client.post(INSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 409, response.get_json() + body = response.get_json() + assert body["status"] == "error" + assert body["error_code"] == "PLUGIN_OPERATION_CONFLICT" + assert "clock" in body["message"] + assert installing["store"].install_plugin.call_count == 1 + assert _failed_history(api_v3_module) == [] + + +def test_an_uninstall_during_the_install_is_a_conflict(api_v3_client, api_v3_module, + installing): + _start_first_install(api_v3_client, installing) + response = api_v3_client.post(UNINSTALL, json={"plugin_id": "clock"}) + assert response.status_code == 409, response.get_json() + assert response.get_json()["error_code"] == "PLUGIN_OPERATION_CONFLICT" + assert _failed_history(api_v3_module) == [], ( + "an uninstall that never started was recorded as failed") + api_v3_module.api_v3.plugin_store_manager.uninstall_plugin.assert_not_called() + + +def test_another_plugin_is_still_queued(api_v3_client, installing): + _start_first_install(api_v3_client, installing) + response = api_v3_client.post(INSTALL, json={"plugin_id": "weather"}) + assert response.status_code == 200, response.get_json() + assert response.get_json()["data"]["operation_id"] diff --git a/test/test_api_v3_plugin_static_files.py b/test/test_api_v3_plugin_static_files.py new file mode 100644 index 00000000..c2bfcc45 --- /dev/null +++ b/test/test_api_v3_plugin_static_files.py @@ -0,0 +1,73 @@ +"""GET /api/v3/plugins//static/ serves binary files too. + +The route opened every file as UTF-8 text, so an image -- what the API +reference says it is for, plugin previews and icons -- failed to decode and +answered 500 "UnicodeDecodeError". Files are now sent as bytes. The text +types the route always set are unchanged, and the path checks are pinned in +test_path_traversal_guards.py::TestServePluginStatic. +""" + +import json +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +PNG = (b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01" + b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89") + + +@pytest.fixture +def plugin_dir(tmp_path, api_v3_module): + d = tmp_path / "demo" + (d / "web_ui").mkdir(parents=True) + (d / "manifest.json").write_text(json.dumps({"id": "demo"}), encoding="utf-8") + api_v3_module.api_v3.plugin_catalog.get_plugin_directory.side_effect = ( + lambda pid: str(d) if pid == "demo" else None) + return d + + +def _get(client, path): + return client.get(f"/api/v3/plugins/demo/static/{path}") + + +def test_an_image_is_served_as_its_bytes(api_v3_client, plugin_dir): + (plugin_dir / "web_ui" / "icon.png").write_bytes(PNG) + response = _get(api_v3_client, "web_ui/icon.png") + assert response.status_code == 200, response.get_json(silent=True) + assert response.mimetype == "image/png" + assert response.data == PNG + + +def test_an_unknown_binary_file_is_served_too(api_v3_client, plugin_dir): + blob = bytes(range(256)) + (plugin_dir / "data.bin").write_bytes(blob) + response = _get(api_v3_client, "data.bin") + assert response.status_code == 200, response.get_json(silent=True) + assert response.data == blob + + +@pytest.mark.parametrize("name,mimetype", [ + ("page.html", "text/html"), + ("app.js", "application/javascript"), + ("style.css", "text/css"), + ("data.json", "application/json"), + ("notes.txt", "text/plain"), + ("README.md", "text/plain"), + ("helper.py", "text/plain"), +]) +def test_text_files_keep_their_types(api_v3_client, plugin_dir, name, mimetype): + content = "caf\u00e9 \u2713

hi

\n" + (plugin_dir / name).write_bytes(content.encode("utf-8")) + response = _get(api_v3_client, name) + assert response.status_code == 200 + assert response.mimetype == mimetype + assert response.data == content.encode("utf-8") + + +def test_a_missing_file_is_still_a_404(api_v3_client, plugin_dir): + assert _get(api_v3_client, "nope.png").status_code == 404 diff --git a/web_interface/blueprints/api_v3/display.py b/web_interface/blueprints/api_v3/display.py index cbc83946..163fbc21 100644 --- a/web_interface/blueprints/api_v3/display.py +++ b/web_interface/blueprints/api_v3/display.py @@ -69,6 +69,26 @@ def _deliver_on_demand(payload): return 'mailbox', reason +def _withdraw_on_demand(request_id): + """Take a start request the route has refused back out of the mailbox. + + The display reads the mailbox for an hour without looking at a + request's age, so one left there after an error answer ran whenever the + display next started. Only this request is removed: the mailbox is + re-read and cleared only while it still holds this request_id, as the + display's _consume_on_demand_request does, so a newer request posted in + the meantime stays for the display to take. + """ + cache = _cache_manager() + try: + current = cache.get('display_on_demand_request', max_age=3600, memory_ttl=0) + if isinstance(current, dict) and current.get('request_id') == request_id: + cache.delete('display_on_demand_request') + except Exception: # the route is answering an error already + logger.warning("Could not withdraw on-demand request %s from the mailbox", + request_id, exc_info=True) + + @api_v3.route('/display/current', methods=['GET']) def get_display_current(): """The latest display preview, as the /stream/display SSE stream sends it. @@ -259,9 +279,26 @@ def start_on_demand_display(): } transport, socket_error = _deliver_on_demand(request_payload) + # A socket acknowledgement is the display itself answering: it is + # running and has the request queued, whatever systemd says (a display + # run by hand or in the emulator has no active unit). So nothing is + # checked or started for it -- that answered "not running" for a request + # that had already taken effect. The service is still reported the way + # _ensure_display_service_running reports a running one. + if transport == 'socket': + service_result = (dict(_get_display_service_status(), started=False) + if start_service else None) + return _on_demand_started(request_id, resolved_plugin, resolved_mode, + duration, pinned, service_result, transport, + socket_error) + service_status = _get_display_service_status() if not service_status.get('active') and not start_service: + # The request is in the mailbox, and the display reads it whenever + # it next starts: taken back out, or a request answered with this + # error ran later anyway. + _withdraw_on_demand(request_id) return jsonify({ 'status': 'error', 'message': 'Display service is not running. Please start the display service or enable "Start Service" option.', @@ -285,16 +322,25 @@ def start_on_demand_display(): service_result = _ensure_display_service_running() # Check if service actually started if service_result and not service_result.get('active'): + _withdraw_on_demand(request_id) return jsonify({ 'status': 'error', 'message': 'Failed to start display service. Please check service logs or start it manually.', 'service_result': service_result }), 500 + return _on_demand_started(request_id, resolved_plugin, resolved_mode, + duration, pinned, service_result, transport, + socket_error) + + +def _on_demand_started(request_id, plugin_id, mode, duration, pinned, + service_result, transport, socket_error): + """The success answer of /display/on-demand/start.""" response_data = { 'request_id': request_id, - 'plugin_id': resolved_plugin, - 'mode': resolved_mode, + 'plugin_id': plugin_id, + 'mode': mode, 'duration': duration, 'pinned': pinned, 'service': service_result, diff --git a/web_interface/blueprints/api_v3/plugin_assets.py b/web_interface/blueprints/api_v3/plugin_assets.py index 750f76fc..9fea89fc 100644 --- a/web_interface/blueprints/api_v3/plugin_assets.py +++ b/web_interface/blueprints/api_v3/plugin_assets.py @@ -3,8 +3,12 @@ Routes decorate the shared `api_v3` Blueprint from the package `__init__`, so their endpoint names do not depend on which module they live in. """ +import mimetypes + +from flask import send_file + from web_interface.blueprints.api_v3 import ( - PROJECT_ROOT, Response, _plugin_directory, api_v3, datetime, hashlib, + PROJECT_ROOT, _plugin_directory, api_v3, datetime, hashlib, json, jsonify, logger, os, request, uuid, ) from src.common.path_safety import ( @@ -231,8 +235,8 @@ def serve_plugin_static(plugin_id, file_path): if not requested_file.exists() or not requested_file.is_file(): return jsonify({'status': 'error', 'message': 'File not found'}), 404 - # Determine content type - content_type = 'text/plain' + # Determine content type. Text keeps the types this route always set; + # anything else (an icon, a preview image) gets its own. name = requested_file.name if name.endswith('.html'): content_type = 'text/html' @@ -242,12 +246,14 @@ def serve_plugin_static(plugin_id, file_path): content_type = 'text/css' elif name.endswith('.json'): content_type = 'application/json' + else: + guessed = mimetypes.guess_type(name)[0] + content_type = ('text/plain' if not guessed or guessed.startswith('text/') + else guessed) - # Read and return file - with open(requested_file, 'r', encoding='utf-8') as f: - content = f.read() - - return Response(content, mimetype=content_type) + # Sent as bytes. Opening it as UTF-8 text failed to decode any binary + # file, so an image answered 500 UnicodeDecodeError. + return send_file(requested_file, mimetype=content_type) @api_v3.route('/plugins/assets/delete', methods=['POST']) diff --git a/web_interface/blueprints/api_v3/plugin_store.py b/web_interface/blueprints/api_v3/plugin_store.py index e776918d..d0f31feb 100644 --- a/web_interface/blueprints/api_v3/plugin_store.py +++ b/web_interface/blueprints/api_v3/plugin_store.py @@ -81,6 +81,27 @@ def _listed_plugin_dir(base: Path, name: str) -> Optional[Path]: return None +def _enqueue_or_conflict(operation_type, plugin_id, callback): + """``(operation_id, None)``, or ``(None, a 409 response)``. + + The queue raises ValueError when the plugin already has an operation + waiting or running -- a double-clicked Install, an uninstall during an + install. That is the caller's timing, not a server fault: it reached + the client as a 500, and the uninstall route recorded a failed + uninstall that had never started. + """ + try: + return api_v3.operation_queue.enqueue_operation( + operation_type, plugin_id, operation_callback=callback), None + except ValueError: + return None, error_response( + ErrorCode.PLUGIN_OPERATION_CONFLICT, + f'Plugin {plugin_id} already has an install, update or uninstall ' + 'in progress; wait for it to finish, then try again', + status_code=409 + ) + + @api_v3.route('/plugins/update', methods=['POST']) def update_plugin(): """Update plugin""" @@ -402,11 +423,10 @@ def uninstall_plugin(): preserve_config=preserve_config)} # Enqueue operation - operation_id = api_v3.operation_queue.enqueue_operation( - OperationType.UNINSTALL, - plugin_id, - operation_callback=uninstall_callback - ) + operation_id, conflict = _enqueue_or_conflict( + OperationType.UNINSTALL, plugin_id, uninstall_callback) + if conflict: + return conflict return success_response( data={'operation_id': operation_id}, @@ -538,11 +558,10 @@ def install_plugin(): raise Exception(error_msg) # Enqueue operation - operation_id = api_v3.operation_queue.enqueue_operation( - OperationType.INSTALL, - plugin_id, - operation_callback=install_callback - ) + operation_id, conflict = _enqueue_or_conflict( + OperationType.INSTALL, plugin_id, install_callback) + if conflict: + return conflict branch_msg = f" (branch: {branch})" if branch else "" return success_response( diff --git a/web_interface/blueprints/api_v3/plugins.py b/web_interface/blueprints/api_v3/plugins.py index ec319e3e..15c3df9d 100644 --- a/web_interface/blueprints/api_v3/plugins.py +++ b/web_interface/blueprints/api_v3/plugins.py @@ -439,6 +439,10 @@ sys.exit(proc.returncode) import tempfile import json as json_lib + # The params reach the wrapper on its stdin, never in + # its source: written there as `params = `, a + # true, false or null was an undefined name and the + # wrapper died with a NameError before the script ran. params_json = json_lib.dumps(action_params) with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as wrapper: wrapper.write(f'''import sys @@ -449,6 +453,9 @@ import json # Set LEDMATRIX_ROOT os.environ['LEDMATRIX_ROOT'] = r"{PROJECT_ROOT}" +# The params, as JSON on this wrapper's own stdin +params = json.loads(sys.stdin.read()) + # Run the script and provide params as JSON via stdin proc = subprocess.Popen( [sys.executable, r"{script_file}"], @@ -460,7 +467,6 @@ proc = subprocess.Popen( ) # Send params as JSON to stdin -params = {params_json} stdout, _ = proc.communicate(input=json.dumps(params), timeout=120) print(stdout) sys.exit(proc.returncode) @@ -470,6 +476,7 @@ sys.exit(proc.returncode) try: result = subprocess.run( ['python3', wrapper_path], + input=params_json, capture_output=True, text=True, timeout=120, From 2236ff308183f03e158939cf5da5423a28efe31c Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Sat, 3 Oct 2026 22:30:51 -0400 Subject: [PATCH 04/37] fix(web-ui): MQTT password without TLS, Overview poll that never stopped, brightness slider error, token form left dirty (#745) * fix(web-ui): let the MQTT bridge form save a password without TLS PUT /api/v3/integrations/mqtt-bridge/config refuses a stored password while mqtt_tls is off unless allow_insecure_mqtt is set (the CWE-319 guard in api_v3/misc.py). The Tools tab form neither rendered a control for that flag nor sent it, so a password-protected broker on a LAN without TLS could never be saved from the UI, and once such a password was in bridge_config.json every later save from the form was refused. The form now shows "Allow without TLS (trusted network)" while "Use TLS" is unchecked, prefilled from the GET's config.allow_insecure_mqtt, and mqttBody() sends its state as allow_insecure_mqtt. The box is off until the user ticks it, so the server's guard still refuses a cleartext password by default. Tests: the Tools DOM suite checks the control, its show/hide with the TLS box, the prefill and the value saved; a Flask test pins that the GET reports the opt-in (false until saved on). Co-Authored-By: Claude Opus 5.5 * fix(web-ui): stop the Overview reconciliation poll from running forever The reconciliation banner script in partials/overview.html re-asked /api/v3/plugins/reconciliation-status every 2 s until the answer said done, with no limit. The route answers done: false whenever ledmatrix_reconciliation.json is missing or unreadable, which happens when _run_startup_reconciliation raises before writing it or when /tmp is cleaned under a long-running web service (reconciliation runs once per process). The browser then sent that request every 2 s for as long as the page stayed open, on every tab, since the poll was never tied to the Overview being visible. The poll now gives up after 30 tries (a minute) and runs only while the Overview is the active, visible tab, registered with LEDVisibility under its own key like the other partials' pollers. Dismissing the banner ends it too. Test: test/js/unit/test_overview_reconciliation_poll.js runs the shipped script in a vm with fake timers and fetch. Co-Authored-By: Claude Opus 5.5 * fix(web-ui): drop the Display tab's lookup of a removed brightness label The brightness slider's input handler in partials/display.html set the text of both #brightness-value and #brightness-display. #387 (978a03b42) removed the "LED brightness: N%" line that carried #brightness-display, so getElementById returned null and every step of the slider threw "Cannot set properties of null" into the console. The visible label still updated, because it is written first. The dead lookup is removed. Test: test/js/unit/test_display_partial_ids.js checks every literal getElementById() in the partial's inline scripts against the ids its markup renders, and runs the shipped script in a vm to move the slider. Co-Authored-By: Claude Opus 5.5 * fix(web-ui): a created API token leaves the General tab's form clean app.js marks a form data-dirty on any input inside it and removes the mark only after a successful htmx request; its beforeunload handler asks "Leave site?" while a visible form is still dirty. The API token form in partials/general.html posts through window.webLogin.createToken with fetch, so the mark survived the token being created and a reload of the page with the General tab open prompted about a change that had already been saved. createToken now removes data-dirty after a successful create, next to the form.reset() it already did. A refused request keeps the mark. Test: test/js/unit/test_general_web_login_token.js runs the shipped script in a vm with a fake fetch and DOM. Co-Authored-By: Claude Opus 5.5 * test(js): match
@@ -216,7 +195,7 @@ the password inputs keep their own labels. #} -
+ {% if web_login.enabled %}
@@ -251,7 +230,7 @@ {% if web_login.enabled %}

Turn login off

- +
created {{ (token.created_at or '')[:10] }}
@@ -289,7 +267,7 @@

No tokens yet.

{% endfor %}
- +
@@ -312,131 +290,5 @@
- {% endif %} + diff --git a/web_interface/templates/v3/partials/schedule.html b/web_interface/templates/v3/partials/schedule.html index 8b376983..5ed53ea6 100644 --- a/web_interface/templates/v3/partials/schedule.html +++ b/web_interface/templates/v3/partials/schedule.html @@ -1,4 +1,13 @@ {% import 'v3/partials/_macros.html' as ui %} +{# No inline script: static/v3/js/pages/schedule.js draws both schedule + pickers from the data-*-config attributes, reports each form's save and + keeps the dim brightness label current. The page registry + (static/v3/js/core/registry.js) starts it when this root appears and stops + it when the partial is swapped away. The forms carry data-reports-result so + app.js leaves the save notification to the page. #} +

Schedule Settings

@@ -12,7 +21,7 @@ hx-ext="json-enc" hx-headers='{"Content-Type": "application/json"}' hx-swap="none" - hx-on:htmx:after-request="handleScheduleResponse(event)" + data-reports-result class="space-y-6"> @@ -41,7 +50,7 @@ hx-ext="json-enc" hx-headers='{"Content-Type": "application/json"}' hx-swap="none" - hx-on:htmx:after-request="handleDimScheduleResponse(event)" + data-reports-result class="space-y-6"> @@ -56,8 +65,7 @@ min="0" max="100" value="{{ dim_schedule_config.dim_brightness | default(30) }}" - class="flex-1 h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer accent-blue-600" - oninput="document.getElementById('dim_brightness_display').textContent = this.value + '%'"> + class="flex-1 h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer accent-blue-600"> {{ dim_schedule_config.dim_brightness | default(30) }}% @@ -77,198 +85,4 @@
- - +
From 0577c807ebd9840e994e52223fdd2f256c44b5aa Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 01:39:02 -0400 Subject: [PATCH 18/37] feat(plugins): request_on_demand() / end_on_demand() -- plugins ask for the screen in-process (#768) * feat(plugins): request_on_demand() / end_on_demand() -- plugins ask for the screen in-process Four plugins (birdnet-go, mqtt-notifications, on-air, pomodoro-timer) take the screen by writing the display_on_demand_request mailbox, which the display reads once a second while the control socket is up and which stage 5 removes. This is the in-process way in that stage needed. - BasePlugin.request_on_demand(mode=None, duration=None, pinned=False) and end_on_demand(), safe from any thread, go through PluginManager to DisplayController.submit_plugin_on_demand, which only queues (at most 32) and wakes the render thread through ControlServer.wake(). The render thread applies them in _drain_control_commands, after socket commands, through _handle_on_demand_request, so they land within a frame; without a socket, on the next pending-changes pass. - A plugin's stop ends only its own session; a mailbox stop still ends any. - Both answer the request id, or None with no display in the process (web interface, check_plugin.py), a full queue, or a mock manager -- a plugin's cue to write the mailbox, which the display still reads. - docs/PLUGIN_API_REFERENCE.md documents the hasattr pattern for plugins that must keep working on older cores; IPC_CONTROL_SOCKET.md and the CHANGELOG are updated. Co-Authored-By: Claude Opus 5.5 * fix(display): wire the on-demand handler only on a manager that has it Tests and the golden traces stand in simpler plugin managers. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 24 ++ docs/IPC_CONTROL_SOCKET.md | 31 ++- docs/PLUGIN_API_REFERENCE.md | 80 ++++++ src/display_controller.py | 118 +++++++- src/ipc/server.py | 16 +- src/plugin_system/base_plugin.py | 59 ++++ src/plugin_system/plugin_manager.py | 74 +++++ test/test_plugin_on_demand_api.py | 418 ++++++++++++++++++++++++++++ 8 files changed, 799 insertions(+), 21 deletions(-) create mode 100644 test/test_plugin_on_demand_api.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c94c537..4607b5e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,30 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +### Plugins ask for the screen in-process: `request_on_demand()` / `end_on_demand()` + +The in-process way in that stage 5 of the control socket needed +(`docs/IPC_CONTROL_SOCKET.md`, "Plugins in the display process"). + +- **`BasePlugin.request_on_demand(mode=None, duration=None, pinned=False)`** + shows the plugin now, and **`BasePlugin.end_on_demand()`** gives the + screen back. Both are safe from any thread (an MQTT callback, a timer + thread): `PluginManager.request_on_demand()` / `end_on_demand()` hand the + request to `DisplayController.submit_plugin_on_demand()`, which only + queues it (at most 32) and wakes the render thread through the control + socket's flag (`ControlServer.wake()`). The render thread applies it with + the socket's commands, through the same handler as a web on-demand + request, so it lands within a frame rather than on the mailbox's + once-a-second look. Both return the request id, or `None` when no display + runs in the process (the web interface, `scripts/check_plugin.py`) or the + queue is full. +- **A plugin's stop ends only its own session.** A mailbox stop still ends + any session, whoever started it. +- **Older cores.** Plugins detect the methods with `hasattr` and write the + `display_on_demand_request` mailbox when they are missing or answer + `None`; the pattern is in `docs/PLUGIN_API_REFERENCE.md` ("On-demand + display"). The display still reads the mailbox for plugins that write it. + ### Web UI: Schedule and General are ES-module pages (stage 3) - The Schedule and General tabs follow stage 2 (#727): their inline diff --git a/docs/IPC_CONTROL_SOCKET.md b/docs/IPC_CONTROL_SOCKET.md index 38505772..b377a374 100644 --- a/docs/IPC_CONTROL_SOCKET.md +++ b/docs/IPC_CONTROL_SOCKET.md @@ -489,7 +489,7 @@ restart banner, as before. | Mailbox | Written by | Read by the display | While the socket is up | |---|---|---|---| -| `display_on_demand_request` | the web interface, only on fallback; four plugins directly (birdnet-go, mqtt-notifications, on-air, pomodoro-timer) | the render thread, `_poll_on_demand_requests()` | looked at every 1 s (`MAILBOX_POLL_INTERVAL_WITH_SOCKET`), 0.25 s without a socket | +| `display_on_demand_request` | the web interface, only on fallback; plugins that predate `BasePlugin.request_on_demand()`, or run on a core without it | the render thread, `_poll_on_demand_requests()` | looked at every 1 s (`MAILBOX_POLL_INTERVAL_WITH_SOCKET`), 0.25 s without a socket | | `plugin_error_clear_request` | the web interface, only on fallback | the error publisher's thread, every 5 s tick | unchanged rate | A look is one `stat()` of the mailbox file (`CacheManager.file_signature`): @@ -502,8 +502,25 @@ the mailbox instead of being re-read until it expires. A request that comes through the on-demand mailbox while the socket is up is logged once per writer (`came through the file mailbox although the -control socket is up`), which names the plugins that still need an -in-process way in before the mailbox is removed. +control socket is up`), which names the plugins that still write it. + +### Plugins in the display process + +A plugin asks for the screen with `BasePlugin.request_on_demand()` and gives +it back with `end_on_demand()` (see "On-demand display" in +[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md)). Neither goes through +the socket or a file: `PluginManager` hands the mailbox-shaped request, +marked `source: 'plugin'`, to `DisplayController.submit_plugin_on_demand`, +which queues it in memory (at most `PLUGIN_ON_DEMAND_QUEUE_SIZE`, 32) from +whatever thread the plugin called on, and wakes the render thread through +the socket's queue flag (`ControlServer.wake()`). The render thread applies +it in `_drain_control_commands`, after the socket's commands, through the +same `_handle_on_demand_request`, so it lands within a frame like a socket +command. Without a socket it lands on the next pending-changes pass (typically +within 0.25 s). A plugin's stop ends only a session that plugin owns. The four +plugins that wrote the mailbox (birdnet-go, mqtt-notifications, on-air, +pomodoro-timer) use it where the core has it and write the mailbox +otherwise. ## Robustness @@ -659,9 +676,11 @@ device never touches the live display. running display (their routes say so); they are not mailboxes. 5. **Remove the mailboxes (next release).** Once every device has run a display with stage 4, the web interface stops writing both mailboxes and - the display stops reading them. The four plugins that write - `display_on_demand_request` need an in-process way to ask for the screen - first. The display also stops writing `display_current_state`, + the display stops reading them. The four plugins that wrote + `display_on_demand_request` now have an in-process way to ask for the + screen (`BasePlugin.request_on_demand()` / `end_on_demand()`, see + "Plugins in the display process"); they keep the mailbox write only as + their fallback on older cores. The display also stops writing `display_current_state`, `display_on_demand_state` and `plugin_runtime_snapshot` once the web interface no longer falls back to them. diff --git a/docs/PLUGIN_API_REFERENCE.md b/docs/PLUGIN_API_REFERENCE.md index 200dba87..4341a62a 100644 --- a/docs/PLUGIN_API_REFERENCE.md +++ b/docs/PLUGIN_API_REFERENCE.md @@ -488,6 +488,78 @@ working for the plugin itself. `get_vegas_segment_width()` read the `vegas_panel_count` config value, which has never affected Vegas — a card's width comes from `get_vegas_content()` and `vegas_width_pct`. +### On-demand display + +A plugin that reacts to something outside the rotation (an MQTT message, a +timer, a detection) can take the screen for it, and give it back. Both +methods are safe from any thread, including an MQTT callback: they only +queue the request, and the display applies it on its render thread within a +frame or so, exactly like an on-demand start or stop from the web interface. + +#### `request_on_demand(mode=None, duration=None, pinned=False) -> Optional[str]` + +Show this plugin now. + +- `mode`: one of the plugin's display modes; `None` for its first. +- `duration`: seconds before the rotation resumes; `None` (or `0`) for no + limit, until `end_on_demand()` or the user stops it. +- `pinned`: stay on `mode` instead of cycling through the plugin's other + modes. + +Returns the request id once the display has queued it, or `None` when +there is no display in this process to ask (the web interface's plugin +manager, `scripts/check_plugin.py`) or its queue is full. A bad argument +(a `mode` that is not a string, a `duration` that is not a number) raises +`ValueError`. + +#### `end_on_demand() -> Optional[str]` + +Give the screen back. Ends only a session this plugin owns: a session the +user started for another plugin, or one that already ended, is left alone. +Returns the request id once queued, or `None` as above. + +#### Older cores: feature detection + +These methods are new after core 3.8.0 (see `CHANGELOG.md`). Before them, +plugins wrote the `display_on_demand_request` cache key (the "mailbox") +themselves. The display reads it only once a second while the control +socket is up, and it will be removed in a future release (see +[IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md), stage 5). A plugin that +must keep working on older cores checks for the method, and writes the +mailbox only when the method is missing or answers `None`: + +```python +import time, uuid + +def _show_alert(self): + if hasattr(self, "request_on_demand") and self.request_on_demand( + mode="my_alert", duration=15): + return + # Older core, or no display in this process: the mailbox, as before. + self.cache_manager.set("display_on_demand_request", { + "request_id": str(uuid.uuid4()), "action": "start", + "plugin_id": self.plugin_id, "mode": "my_alert", + "duration": 15, "pinned": False, "timestamp": time.time(), + }) + +def _release(self): + if hasattr(self, "end_on_demand") and self.end_on_demand(): + return + self.cache_manager.set("display_on_demand_request", { + "request_id": str(uuid.uuid4()), "action": "stop", + "plugin_id": self.plugin_id, "timestamp": time.time(), + }) +``` + +Keep `ledmatrix_min_version` where it is: the fallback is what keeps the +plugin working on older cores. A mailbox stop ends any on-demand session, +whoever started it; `end_on_demand()` ends only the plugin's own. + +Both methods answer a request id only when the plugin manager returned a +string, so a test that gives the plugin a `MagicMock()` plugin manager gets +`None` and exercises the mailbox path. To test the new path, set +`plugin_manager.request_on_demand.return_value = "some-id"`. + > The full source for `BasePlugin` lives in > `src/plugin_system/base_plugin.py`. If a method here disagrees with the > source, the source wins — please open an issue or PR to fix the doc. @@ -966,6 +1038,14 @@ if info: self.logger.info(f"Plugin: {info['name']}, Version: {info.get('version')}") ``` +#### `request_on_demand(plugin_id, mode=None, duration=None, pinned=False)` / `end_on_demand(plugin_id)` + +What `BasePlugin.request_on_demand()` and `end_on_demand()` call, with the +plugin's own id. Call those instead; see +[On-demand display](#on-demand-display). The display controller routes them +to itself with `set_on_demand_handler()`; a plugin manager without a +display behind it answers `None`. + #### `get_all_plugin_info() -> List[Dict[str, Any]]` Get information for all plugins. diff --git a/src/display_controller.py b/src/display_controller.py index dd64c6dd..84bfbe8c 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -394,6 +394,12 @@ class DisplayController: plugin_time = time.time() self.plugin_manager = None self._plugin_runtime_publisher = None + # On-demand requests plugins make in this process (BasePlugin. + # request_on_demand / end_on_demand), from any thread; the render + # thread drains them with the socket's commands. Created before the + # plugins load, because a plugin may ask from its first thread. + self._plugin_on_demand: deque = deque() + self._plugin_on_demand_lock = threading.Lock() self.plugin_modes = {} # mode -> plugin_instance mapping for plugin-first dispatch self.mode_to_plugin_id: Dict[str, str] = {} self.plugin_display_modes: Dict[str, List[str]] = {} @@ -508,6 +514,12 @@ class DisplayController: cache_manager=self.cache_manager, font_manager=self.font_manager ) + # BasePlugin.request_on_demand() / end_on_demand() land here. + # Before any plugin loads: a plugin may ask from its first thread. + # getattr: tests and golden traces stand in simpler managers. + set_handler = getattr(self.plugin_manager, 'set_on_demand_handler', None) + if callable(set_handler): + set_handler(self.submit_plugin_on_demand) # The web UI's loaded / state / error_info for each plugin read # what this publishes. Started before loading, so the loads that @@ -1882,6 +1894,10 @@ class DisplayController: _on_demand_mailbox: Optional[MailboxWatch] = None #: Writers whose mailbox requests have been logged (_note_mailbox_request). _mailbox_writers_logged: FrozenSet[str] = frozenset() + #: Most plugin on-demand requests waiting for the render thread at once. + #: A plugin that asks faster than the display drains (four times a + #: second at worst) is refused, not queued without end. + PLUGIN_ON_DEMAND_QUEUE_SIZE = 32 def _service_pending_changes(self) -> None: """Apply changes made elsewhere while the display thread is busy. @@ -1906,7 +1922,7 @@ class DisplayController: # A command queued on the control socket skips the floor: it is in # memory, so applying it now costs no disk read. if (last is not None and now - last < self.PENDING_CHANGES_INTERVAL - and not (self._control_server and self._control_server.has_pending)): + and not self._control_command_pending()): return self._last_pending_service = now @@ -2105,11 +2121,15 @@ class DisplayController: here. A plugin reload waits for the top of the next loop pass, where no plugin is on the stack (_apply_pending_plugin_reloads); until then the current screen ends early (_plugin_reload_pending). + + Plugins' own on-demand requests (submit_plugin_on_demand) are + applied here too, after the socket's, with or without a socket. """ server = self._control_server - if server is None or not server.has_pending: - return - for command in server.drain(): + # drain() clears the wake flag before the plugin queue is read below, + # so a plugin request queued from here on wakes the next wait. + commands = server.drain() if server is not None and server.has_pending else [] + for command in commands: try: if command.cmd == ControlCommand.BRIGHTNESS_SET: self._apply_control_brightness(command) @@ -2121,24 +2141,78 @@ class DisplayController: logger.exception("Failed to apply control socket command %s", command.request_id) command.fail(ControlErrorCode.INTERNAL, 'the display failed to apply it') + self._drain_plugin_on_demand() + + # -- plugins' in-process on-demand requests --------------------------------- + + def submit_plugin_on_demand(self, request: Dict[str, Any]) -> bool: + """Queue a plugin's on-demand request for the render thread. Any thread. + + ``PluginManager.request_on_demand`` / ``end_on_demand`` (which + BasePlugin's methods of the same names call) build ``request``: the + mailbox's shape, with ``source: 'plugin'`` and the asking plugin's + id. Nothing here touches the panel or the on-demand state; the render + thread applies the request where it applies a socket command + (_drain_control_commands), through _handle_on_demand_request, and is + woken for it when the control socket is up. True when it was queued; + False (logged) when the queue is full. + """ + pending = self.__dict__.get('_plugin_on_demand') + lock = self.__dict__.get('_plugin_on_demand_lock') + if pending is None or lock is None: + return False # a controller built without __init__ (tests) + with lock: + if len(pending) >= self.PLUGIN_ON_DEMAND_QUEUE_SIZE: + logger.warning("Plugin on-demand queue full; refusing %s %s from %s", + request.get('action'), request.get('request_id'), + request.get('plugin_id')) + return False + pending.append(dict(request)) + server = self._control_server + if server is not None: + server.wake() + return True + + def _plugin_on_demand_pending(self) -> bool: + """A plugin's on-demand request is waiting. A length check: no lock.""" + return bool(self.__dict__.get('_plugin_on_demand')) + + def _drain_plugin_on_demand(self) -> None: + """Apply the plugins' queued on-demand requests, oldest first. Render thread.""" + pending = self.__dict__.get('_plugin_on_demand') + while pending: + try: + request = pending.popleft() + except IndexError: + break + try: + self._handle_on_demand_request(request) + except Exception: # pylint: disable=broad-except + logger.exception("Failed to apply on-demand request %s from plugin %s", + request.get('request_id'), request.get('plugin_id')) def _wait_for_control(self, timeout: float) -> bool: """Sleep up to ``timeout``, waking early for a control socket command. True when a command is waiting. Without a socket (Windows, switched - off, tests) this is the plain sleep it replaces. + off, tests) this is the plain sleep it replaces, unless a plugin's + on-demand request is already waiting. """ server = self._control_server wait = getattr(server, 'wait_for_command', None) if server is not None else None if wait is None: + if self._plugin_on_demand_pending(): + return True time.sleep(timeout) return False return bool(wait(timeout)) def _control_command_pending(self) -> bool: - """A socket command is queued: Vegas checks this every frame.""" + """A socket command or a plugin's on-demand request is queued: Vegas + checks this every frame.""" server = self._control_server - return bool(server is not None and server.has_pending) + return bool((server is not None and server.has_pending) + or self._plugin_on_demand_pending()) def _wait_frame_interval(self, interval: float, screen: Screen) -> Optional[ScreenPlan]: """The static screen's sleep between frames, woken by socket commands. @@ -2450,21 +2524,36 @@ class DisplayController: def _handle_on_demand_request(self, request: Dict[str, Any]) -> None: """Process one on-demand request, from the mailbox or the control socket. - A socket command carries ``source: 'socket'``. Only a mailbox request - is removed from the mailbox afterwards: a socket command never put - anything there, so that would be a disk read and maybe a delete for - nothing. + A socket command carries ``source: 'socket'``, and a plugin's own + request (submit_plugin_on_demand) ``source: 'plugin'``. Only a + mailbox request is removed from the mailbox afterwards: the others + never put anything there, so that would be a disk read and maybe a + delete for nothing. + + A plugin's stop ends only that plugin's own session: a plugin + releasing the screen must not end one the user started for + another plugin. (A stop through the mailbox ends any session, as it + always has.) """ request_id = request.get('request_id') if not request_id: return - from_mailbox = request.get('source') != 'socket' + source = request.get('source') + from_mailbox = source not in ('socket', 'plugin') action = request.get('action') # For stop requests, always process them (don't check processed_id) # This allows stopping even if the same stop request was sent before if action == 'stop': + if source == 'plugin' and not ( + self.on_demand_active + and self.on_demand_plugin_id == request.get('plugin_id')): + logger.debug("On-demand stop %s from plugin %s ignored: it does not own " + "the screen (on-demand %s, plugin %s)", request_id, + request.get('plugin_id'), self.on_demand_status, + self.on_demand_plugin_id) + return logger.info("Received on-demand stop request %s", request_id) # Always process stop requests, even if same request_id (user might click multiple times) if self.on_demand_active: @@ -2507,8 +2596,9 @@ class DisplayController: self._consume_on_demand_request(request_id) return - logger.info("Received on-demand request %s: %s (plugin_id=%s, mode=%s)", - request_id, action, request.get('plugin_id'), request.get('mode')) + logger.info("Received on-demand request %s: %s (plugin_id=%s, mode=%s, via %s)", + request_id, action, request.get('plugin_id'), request.get('mode'), + 'mailbox' if from_mailbox else source) # Mark as processed BEFORE processing (to prevent duplicate processing) self.cache_manager.set('display_on_demand_processed_id', request_id, ttl=3600) diff --git a/src/ipc/server.py b/src/ipc/server.py index 1b81ca04..60b3d060 100644 --- a/src/ipc/server.py +++ b/src/ipc/server.py @@ -772,8 +772,22 @@ class ControlServer: """ return self._pending.wait(timeout) + def wake(self) -> None: + """Wake the render thread as a queued command would, with nothing queued. + + For work that reaches the display another way in the same process (a + plugin's on-demand request, ``DisplayController.submit_plugin_on_demand``): + the render thread returns from :meth:`wait_for_command` and drains, + and reads the caller's own queue there. Safe from any thread. + """ + self._pending.set() + def drain(self) -> List[QueuedCommand]: - """Every queued command, oldest first. Called from the render thread.""" + """Every queued command, oldest first. Called from the render thread. + + Clears the wake flag first, so anything queued (or woken for) while + this runs wakes the next wait again. + """ commands: List[QueuedCommand] = [] self._pending.clear() while True: diff --git a/src/plugin_system/base_plugin.py b/src/plugin_system/base_plugin.py index 54314cd9..2b676eb1 100644 --- a/src/plugin_system/base_plugin.py +++ b/src/plugin_system/base_plugin.py @@ -1070,6 +1070,65 @@ class BasePlugin(ABC): if callable(notify): notify(self.plugin_id) + def request_on_demand(self, mode: Optional[str] = None, + duration: Optional[float] = None, + pinned: bool = False) -> Optional[str]: + """ + Take the screen now: show this plugin on demand. Safe from any thread. + + For a plugin that reacts to something outside the rotation -- an MQTT + message, a timer, a detection -- and wants the panel for it. The + request goes straight to the display in this process and is applied + on its render thread within a frame or so, exactly like an on-demand + start from the web interface. + + Args: + mode: One of this plugin's display modes; None for its first. + duration: Seconds to show it before the rotation resumes; None + (or zero) for no limit, until end_on_demand() or the user + stops it. + pinned: Stay on ``mode`` instead of cycling through the + plugin's other modes. + + Returns: + The request id once the display has queued it, or None when + there is no display in this process to ask (the web interface, + scripts/check_plugin.py) or its queue is full. A plugin that + also runs on cores without this method writes the + ``display_on_demand_request`` mailbox on None, as before; see + "On-demand display" in docs/PLUGIN_API_REFERENCE.md. + + Example:: + + if not (hasattr(self, 'request_on_demand') + and self.request_on_demand(mode='my_alert', duration=15)): + self._write_on_demand_mailbox(...) # older cores + """ + request = getattr(getattr(self, 'plugin_manager', None), 'request_on_demand', None) + if not callable(request): + return None + request_id = request(self.plugin_id, mode=mode, duration=duration, pinned=pinned) + # Only a real id counts: a test's MagicMock manager answers a mock, + # which must read as "not taken" so the plugin's fallback runs. + return request_id if isinstance(request_id, str) else None + + def end_on_demand(self) -> Optional[str]: + """ + Give the screen back: end this plugin's on-demand session. Any thread. + + Ends only a session this plugin owns. One the user started for + another plugin, or a session that already ended, is left alone. The + rotation resumes where it left off. + + Returns: + The request id once queued, or None as request_on_demand() does. + """ + end = getattr(getattr(self, 'plugin_manager', None), 'end_on_demand', None) + if not callable(end): + return None + request_id = end(self.plugin_id) + return request_id if isinstance(request_id, str) else None + def get_vegas_participation(self) -> str: """ How this plugin takes part in Vegas mode: ``'scroll'``, ``'pause'`` or diff --git a/src/plugin_system/plugin_manager.py b/src/plugin_system/plugin_manager.py index 69c9548b..1d2961fb 100644 --- a/src/plugin_system/plugin_manager.py +++ b/src/plugin_system/plugin_manager.py @@ -15,6 +15,7 @@ import sys import time import threading import types +import uuid from pathlib import Path from typing import Callable, Dict, List, NamedTuple, Optional, Any, Tuple, Union import logging @@ -214,6 +215,9 @@ class PluginManager: # add_update_listener(). A tuple, replaced rather than mutated, so the # worker can iterate it without a lock. self._update_listeners: Tuple[Callable[[str], None], ...] = () + # Where plugins' on-demand requests go: the display controller's + # submit_plugin_on_demand. See set_on_demand_handler(). + self._on_demand_handler: Optional[Callable[[Dict[str, Any]], bool]] = None # Config changes that found the plugin's lock busy, latest per plugin, # with the instance they were meant for. See apply_config_change(). self._deferred_config_changes: Dict[str, Tuple[Any, Dict[str, Any]]] = {} @@ -1844,3 +1848,73 @@ class PluginManager: done = sorted(self._completed_updates) self._completed_updates.clear() return done + + # -- on-demand requests from plugins ------------------------------------- + + def set_on_demand_handler( + self, handler: Optional[Callable[[Dict[str, Any]], bool]]) -> None: + """Route plugins' on-demand requests to ``handler`` (None: nowhere). + + The display controller sets its ``submit_plugin_on_demand`` here + before any plugin loads. The handler takes a mailbox-shaped request + from any thread, queues it for the render thread and returns True, + or False when it could not. A plugin manager with no handler (the + web interface's, a test's, scripts/check_plugin.py's) has no screen + to give, so request_on_demand() there answers None. + """ + self._on_demand_handler = handler + + def request_on_demand(self, plugin_id: str, mode: Optional[str] = None, + duration: Optional[float] = None, + pinned: bool = False) -> Optional[str]: + """Ask the display to show ``plugin_id`` now. Safe from any thread. + + BasePlugin.request_on_demand() lands here; see it for the arguments. + Returns the request id once the display has queued the request (it + is applied on the render thread within a frame or so), or None when + this process has no display to ask or its queue is full. + """ + if not isinstance(plugin_id, str) or not plugin_id: + raise ValueError('plugin_id is required') + if mode is not None and (not isinstance(mode, str) or not mode): + raise ValueError('mode must be a non-empty string or None') + if duration is not None: + if isinstance(duration, bool) or not isinstance(duration, (int, float)): + raise ValueError('duration must be a number of seconds or None') + if not math.isfinite(duration) or duration <= 0: + duration = None # the display reads these as "no limit" too + else: + duration = float(duration) + return self._submit_on_demand({ + 'action': 'start', 'plugin_id': plugin_id, 'mode': mode, + 'duration': duration, 'pinned': bool(pinned)}) + + def end_on_demand(self, plugin_id: str) -> Optional[str]: + """Give the screen back, if ``plugin_id``'s on-demand session has it. + + BasePlugin.end_on_demand() lands here. A session the plugin does not + own (the user started another plugin from the web interface, say) is + left alone. Returns the request id once queued, or None as + request_on_demand() does. + """ + if not isinstance(plugin_id, str) or not plugin_id: + raise ValueError('plugin_id is required') + return self._submit_on_demand({'action': 'stop', 'plugin_id': plugin_id}) + + def _submit_on_demand(self, request: Dict[str, Any]) -> Optional[str]: + # __dict__.get: tests build bare managers with PluginManager.__new__. + handler = self.__dict__.get('_on_demand_handler') + if handler is None: + return None + request_id = str(uuid.uuid4()) + request.update({'request_id': request_id, 'timestamp': time.time(), + 'source': 'plugin'}) + try: + accepted = handler(request) + except Exception as exc: # pylint: disable=broad-except + self._warn_rate_limited( + "on-demand-handler", + "The on-demand request from plugin %s failed: %r", + request.get('plugin_id'), exc) + return None + return request_id if accepted else None diff --git a/test/test_plugin_on_demand_api.py b/test/test_plugin_on_demand_api.py new file mode 100644 index 00000000..667724a6 --- /dev/null +++ b/test/test_plugin_on_demand_api.py @@ -0,0 +1,418 @@ +"""Plugins asking for the screen in-process: BasePlugin.request_on_demand() +and end_on_demand(). + +A plugin running in the display process used to write the +``display_on_demand_request`` mailbox, which the display reads once a second +while the control socket is up. These tests pin the way in that replaces it: + +* BasePlugin -> PluginManager -> DisplayController.submit_plugin_on_demand, + which only queues, from any thread; +* the render thread applies the queue where it applies socket commands, + through the mailbox's own handler, without the mailbox's read floor, and + woken by the control socket when it is up; +* a plugin's stop ends only its own session; +* no display to ask (the web interface's plugin manager, an old core's + plugin manager) answers None, which is a plugin's cue to fall back to the + mailbox; +* the mailbox still works for plugins that write it. +""" + +import logging +import threading +import time +from unittest.mock import MagicMock + +import pytest + +from src.ipc.server import ControlServer +from src.plugin_system.base_plugin import BasePlugin +from src.plugin_system.plugin_manager import PluginManager + + +class _Plugin(BasePlugin): + def update(self): + pass + + def display(self, force_clear=False): + pass + + +def _plugin(plugin_id, manager): + plugin = _Plugin.__new__(_Plugin) + plugin.plugin_id = plugin_id + plugin.plugin_manager = manager + return plugin + + +def _manager(handler=None): + manager = PluginManager.__new__(PluginManager) + manager.logger = logging.getLogger('test.plugin_on_demand') + if handler is not None: + manager.set_on_demand_handler(handler) + return manager + + +class _WakeServer: + """The parts of ControlServer the controller uses, with no socket.""" + + def __init__(self): + self.woken = 0 + self.has_pending = False + + def wake(self): + self.woken += 1 + self.has_pending = True + + def drain(self): + self.has_pending = False + return [] + + +@pytest.fixture +def controller(test_display_controller): + c_ = test_display_controller + c_.on_demand_active = False + c_.on_demand_request_id = None + c_._last_on_demand_poll = None + mailbox = {'value': None} + + def fake_get(key, *a, **kw): + if key == 'display_on_demand_request': + return mailbox['value'] + return None + + c_.cache_manager.get = MagicMock(side_effect=fake_get) + c_.cache_manager.set = MagicMock() + c_.cache_manager.delete = MagicMock() + c_._activate_on_demand = MagicMock() + c_.mailbox = mailbox + return c_ + + +@pytest.fixture +def wired(controller): + """A real PluginManager wired to the controller, as __init__ wires it.""" + manager = _manager(controller.submit_plugin_on_demand) + return controller, manager + + +class TestWiring: + def test_the_controller_wires_its_plugin_manager(self, controller): + controller.plugin_manager.set_on_demand_handler.assert_called_once_with( + controller.submit_plugin_on_demand) + + def test_a_start_reaches_the_mailbox_handler(self, wired): + controller, manager = wired + rid = _plugin('pomodoro-timer', manager).request_on_demand( + mode='pomodoro', duration=30, pinned=True) + assert isinstance(rid, str) and rid + controller._activate_on_demand.assert_not_called() # only queued + controller._poll_on_demand_requests() + controller._activate_on_demand.assert_called_once() + request = controller._activate_on_demand.call_args.args[0] + assert request['request_id'] == rid + assert request['action'] == 'start' + assert request['plugin_id'] == 'pomodoro-timer' + assert request['mode'] == 'pomodoro' + assert request['duration'] == 30.0 and request['pinned'] is True + assert request['source'] == 'plugin' + assert controller.on_demand_request_id == rid + + def test_a_plugin_request_never_touches_the_mailbox(self, wired): + controller, manager = wired + _plugin('on-air', manager).request_on_demand(mode='on_air') + controller._drain_control_commands() + controller._activate_on_demand.assert_called_once() + mailbox_reads = [call for call in controller.cache_manager.get.call_args_list + if call.args[0] == 'display_on_demand_request'] + assert mailbox_reads == [] + controller.cache_manager.delete.assert_not_called() + + def test_requests_apply_in_order(self, wired): + controller, manager = wired + seen = [] + controller._activate_on_demand = MagicMock( + side_effect=lambda r: seen.append(r['mode'])) + plugin = _plugin('p', manager) + for mode in ('a', 'b', 'c'): + plugin.request_on_demand(mode=mode) + controller._poll_on_demand_requests() + assert seen == ['a', 'b', 'c'] + + def test_the_mailbox_still_works_for_older_plugins(self, wired): + controller, manager = wired + controller.mailbox['value'] = {'request_id': 'mb', 'action': 'start', + 'plugin_id': 'birdnet-go'} + _plugin('on-air', manager).request_on_demand() + controller._poll_on_demand_requests() + ids = [call.args[0]['request_id'] for call in controller._activate_on_demand.call_args_list] + assert 'mb' in ids and len(ids) == 2 + + def test_a_failing_request_is_contained(self, wired): + controller, manager = wired + calls = [] + + def activate(request): + calls.append(request['mode']) + if request['mode'] == 'bad': + raise RuntimeError('plugin exploded') + + controller._activate_on_demand = MagicMock(side_effect=activate) + plugin = _plugin('p', manager) + plugin.request_on_demand(mode='bad') + plugin.request_on_demand(mode='good') + controller._poll_on_demand_requests() + assert calls == ['bad', 'good'] + + +class TestPromptness: + def test_a_plugin_request_skips_the_pending_changes_floor(self, wired): + controller, manager = wired + controller._control_server = None + controller._service_pending_changes() + _plugin('p', manager).request_on_demand() + controller._service_pending_changes() # well inside the 0.25 s floor + controller._activate_on_demand.assert_called_once() + + def test_a_plugin_request_skips_the_mailbox_floor(self, wired): + controller, manager = wired + controller._control_server = _WakeServer() + controller._poll_on_demand_requests() # sets the 1 s mailbox floor + _plugin('p', manager).request_on_demand() + controller._poll_on_demand_requests() + controller._activate_on_demand.assert_called_once() + + def test_it_wakes_the_control_socket_wait(self, wired): + controller, manager = wired + server = ControlServer('/nonexistent/control.sock') # never started + controller._control_server = server + assert not server.wait_for_command(0) + _plugin('p', manager).request_on_demand() + assert server.has_pending + assert controller._wait_for_control(5.0) is True # returns at once + assert controller._control_command_pending() + controller._poll_on_demand_requests() + controller._activate_on_demand.assert_called_once() + assert not server.has_pending + assert not controller._control_command_pending() + + def test_without_a_socket_a_waiting_request_cuts_the_sleep(self, wired): + controller, manager = wired + controller._control_server = None + _plugin('p', manager).request_on_demand() + started = time.monotonic() + assert controller._wait_for_control(5.0) is True + assert time.monotonic() - started < 1.0 + assert controller._control_command_pending() + + def test_nothing_waiting_keeps_the_floor(self, controller): + controller._control_server = None + controller._poll_on_demand_requests = MagicMock() + controller._service_pending_changes() + controller._service_pending_changes() + assert controller._poll_on_demand_requests.call_count == 1 + + +class TestThreads: + def test_requests_from_many_threads_all_land_in_order_per_thread(self, wired): + controller, manager = wired + seen = [] + controller._activate_on_demand = MagicMock( + side_effect=lambda r: seen.append(r['mode'])) + controller.PLUGIN_ON_DEMAND_QUEUE_SIZE = 10_000 + threads_n, each = 8, 50 + barrier = threading.Barrier(threads_n) + + def ask(n): + plugin = _plugin(f'p{n}', manager) + barrier.wait() + for i in range(each): + assert plugin.request_on_demand(mode=f'{n}:{i}') + + threads = [threading.Thread(target=ask, args=(n,)) for n in range(threads_n)] + for t in threads: + t.start() + # Drain while they ask, as the render thread would. + while any(t.is_alive() for t in threads): + controller._drain_control_commands() + for t in threads: + t.join() + controller._drain_control_commands() + assert len(seen) == threads_n * each + for n in range(threads_n): + mine = [int(m.split(':')[1]) for m in seen if m.startswith(f'{n}:')] + assert mine == list(range(each)) + + def test_a_full_queue_refuses(self, wired, caplog): + controller, manager = wired + controller.PLUGIN_ON_DEMAND_QUEUE_SIZE = 2 + plugin = _plugin('p', manager) + assert plugin.request_on_demand() + assert plugin.request_on_demand() + assert plugin.request_on_demand() is None + assert 'queue full' in caplog.text + controller._poll_on_demand_requests() + assert controller._activate_on_demand.call_count == 2 + assert plugin.request_on_demand() # room again + + +class TestStop: + def test_a_plugin_ends_its_own_session(self, wired): + controller, manager = wired + controller.on_demand_active = True + controller.on_demand_plugin_id = 'on-air' + controller._clear_on_demand = MagicMock() + assert _plugin('on-air', manager).end_on_demand() + controller._poll_on_demand_requests() + controller._clear_on_demand.assert_called_once_with(reason='requested-stop') + controller.cache_manager.delete.assert_not_called() + + def test_a_plugin_cannot_end_another_plugins_session(self, wired): + controller, manager = wired + controller.on_demand_active = True + controller.on_demand_plugin_id = 'clock' # the user started it + controller.on_demand_request_id = 'user' + controller._clear_on_demand = MagicMock() + _plugin('pomodoro-timer', manager).end_on_demand() + controller._poll_on_demand_requests() + controller._clear_on_demand.assert_not_called() + assert controller.on_demand_request_id == 'user' + + def test_a_stop_with_no_session_does_nothing(self, wired): + controller, manager = wired + controller.on_demand_status = 'error' + controller._clear_on_demand = MagicMock() + _plugin('on-air', manager).end_on_demand() + controller._poll_on_demand_requests() + controller._clear_on_demand.assert_not_called() + + def test_a_mailbox_stop_still_ends_any_session(self, wired): + controller, _ = wired + controller.on_demand_active = True + controller.on_demand_plugin_id = 'clock' + controller._clear_on_demand = MagicMock() + controller.mailbox['value'] = {'request_id': 's', 'action': 'stop', + 'plugin_id': 'on-air'} + controller._poll_on_demand_requests() + controller._clear_on_demand.assert_called_once_with(reason='requested-stop') + + def test_start_then_stop_from_one_thread_ends_the_session(self, wired): + controller, manager = wired + + def activate(request): + controller.on_demand_active = True + controller.on_demand_plugin_id = request['plugin_id'] + + controller._activate_on_demand = MagicMock(side_effect=activate) + controller._clear_on_demand = MagicMock() + plugin = _plugin('pomodoro-timer', manager) + plugin.request_on_demand(mode='pomodoro', pinned=True) + plugin.end_on_demand() + controller._poll_on_demand_requests() + controller._activate_on_demand.assert_called_once() + controller._clear_on_demand.assert_called_once_with(reason='requested-stop') + + +class TestNoDisplay: + """None is a plugin's cue to write the mailbox instead.""" + + def test_a_manager_with_no_handler_answers_none(self): + plugin = _plugin('p', _manager()) + assert plugin.request_on_demand() is None + assert plugin.end_on_demand() is None + + def test_no_plugin_manager_answers_none(self): + plugin = _plugin('p', None) + assert plugin.request_on_demand() is None + assert plugin.end_on_demand() is None + + def test_an_old_cores_plugin_manager_answers_none(self): + class OldManager: + plugin_manifests = {} + + plugin = _plugin('p', OldManager()) + assert plugin.request_on_demand() is None + assert plugin.end_on_demand() is None + + def test_a_handler_that_raises_answers_none(self): + def broken(request): + raise RuntimeError('boom') + + plugin = _plugin('p', _manager(broken)) + assert plugin.request_on_demand() is None + assert plugin.end_on_demand() is None + + def test_a_handler_that_refuses_answers_none(self): + plugin = _plugin('p', _manager(lambda request: False)) + assert plugin.request_on_demand() is None + + def test_a_controller_built_without_init_refuses(self): + from src.display_controller import DisplayController + bare = DisplayController.__new__(DisplayController) + assert bare.submit_plugin_on_demand({'action': 'start'}) is False + assert bare._plugin_on_demand_pending() is False + bare._drain_plugin_on_demand() # nothing to do, no error + + def test_the_feature_detection_pattern(self): + """The hasattr pattern from docs/PLUGIN_API_REFERENCE.md.""" + writes = [] + + class OldCorePlugin: # an older core's BasePlugin has no such method + pass + + for plugin, expect_mailbox in ((OldCorePlugin(), True), + (_plugin('p', _manager()), True), + (_plugin('p', _manager(lambda r: True)), False)): + writes.clear() + if not (hasattr(plugin, 'request_on_demand') + and plugin.request_on_demand(mode='m')): + writes.append('mailbox') + assert (writes == ['mailbox']) is expect_mailbox + + +class TestArguments: + def test_the_manager_shapes_the_request(self): + got = [] + plugin = _plugin('p', _manager(lambda r: got.append(r) or True)) + plugin.request_on_demand() + plugin.end_on_demand() + start, stop = got + assert start['plugin_id'] == 'p' and start['mode'] is None + assert start['duration'] is None and start['pinned'] is False + assert start['source'] == 'plugin' and start['timestamp'] > 0 + assert stop == {'action': 'stop', 'plugin_id': 'p', 'request_id': stop['request_id'], + 'timestamp': stop['timestamp'], 'source': 'plugin'} + assert start['request_id'] != stop['request_id'] + + @pytest.mark.parametrize('duration', [0, -5, float('inf'), float('nan')]) + def test_no_positive_duration_means_no_limit(self, duration): + got = [] + _plugin('p', _manager(lambda r: got.append(r) or True)).request_on_demand( + duration=duration) + assert got[0]['duration'] is None + + @pytest.mark.parametrize('kwargs', [{'mode': 5}, {'mode': ''}, {'duration': '30'}, + {'duration': True}]) + def test_bad_arguments_raise(self, kwargs): + plugin = _plugin('p', _manager(lambda r: True)) + with pytest.raises(ValueError): + plugin.request_on_demand(**kwargs) + + +class TestMockManagers: + def test_a_magicmock_manager_reads_as_not_taken(self): + """A plugin's test with a MagicMock manager keeps its mailbox path.""" + plugin = _plugin('p', MagicMock()) + assert plugin.request_on_demand(mode='m') is None + assert plugin.end_on_demand() is None + plugin.plugin_manager.request_on_demand.assert_called_once_with( + 'p', mode='m', duration=None, pinned=False) + plugin.plugin_manager.end_on_demand.assert_called_once_with('p') + + def test_a_mocked_id_is_passed_through(self): + manager = MagicMock() + manager.request_on_demand.return_value = 'rid' + manager.end_on_demand.return_value = 'rid2' + plugin = _plugin('p', manager) + assert plugin.request_on_demand() == 'rid' + assert plugin.end_on_demand() == 'rid2' From 3f920f28991705da65229aa0b2f72b22474611a9 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 08:26:26 -0400 Subject: [PATCH 19/37] fix(fonts): unloading a plugin forgets its manifest fonts (#757) * fix(fonts): unloading a plugin forgets its manifest fonts PluginManager.unload_plugin() and the failed-load cleanup only called FontManager.forget_manager_fonts(), which drops usage data. The plugin's manifest registrations stayed: plugin_fonts / plugin_font_catalogs, its plugin_id::family entries in font_catalog, and cached font objects for them -- so its fonts kept resolving after unload and a family a reinstalled manifest dropped stayed registered. The deprecated unregister_plugin_fonts did this cleanup but nothing called it; it was removed in #708. Add FontManager.forget_plugin_fonts(plugin_id) and call it from both paths alongside forget_manager_fonts (each guarded on its own, so a font manager stub with only one still works). FontManager takes no locks, so like forget_manager_fonts it uses atomic pops over snapshots. A reload (unload + load) registers the manifest fonts again and they resolve. Raised by CodeRabbit on #709. Co-Authored-By: Claude Opus 5.5 * fix(fonts): call the font manager's forget methods without a None-able local Pylint E1102 (Codacy) read getattr(..., None) as possibly not callable. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 8 + docs/FONT_MANAGER.md | 4 +- src/font_manager.py | 33 ++++ src/plugin_system/plugin_manager.py | 29 ++-- test/test_font_manager.py | 143 ++++++++++++++++++ ...test_plugin_manager_failed_load_cleanup.py | 1 + 6 files changed, 206 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4607b5e0..71c3fbe6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -162,6 +162,14 @@ Internal; no behaviour change. Stage 3 of `docs/RUN_LOOP_REDESIGN.md`. ### Fixed +- Unloading a plugin now forgets the fonts its manifest registered, not only + the fonts it reported using. Its `plugin_id::family` entries kept resolving + and their cached font objects stayed alive until a restart, and a family a + reinstalled plugin's manifest dropped stayed registered. The new + `FontManager.forget_plugin_fonts(plugin_id)` does the cleanup; + `PluginManager.unload_plugin()` and a failed load call it alongside + `forget_manager_fonts()`, and a reload registers the manifest's fonts again. + - The web preview and `/api/v3/display/current` no longer stay black for a whole screen that draws its card once and then holds it. The snapshot is written from `update_display()` at most once per write interval, so a frame diff --git a/docs/FONT_MANAGER.md b/docs/FONT_MANAGER.md index 6d2dc61f..9ad0cb17 100644 --- a/docs/FONT_MANAGER.md +++ b/docs/FONT_MANAGER.md @@ -203,6 +203,7 @@ Current methods: | `measure_text(text, font)` | `(width, height, baseline)` | | `get_font_height(font)` | Line height | | `register_plugin_fonts(plugin_id, font_manifest)` | Register a plugin's fonts (core calls it at load) | +| `forget_plugin_fonts(plugin_id)` | Drop a plugin's manifest fonts and their cached objects (core calls it when a plugin unloads) | | `clear_cache()` | Drop cached fonts and metrics | | `font_catalog` (attribute) | Family name → file path | @@ -218,5 +219,6 @@ Removed in 3.8.0, after logging a deprecation warning on first call since | `get_performance_stats()` | — | | `set_override()`, `remove_override()`, `get_overrides()` | a font field in your plugin's config schema | | `get_manager_fonts()`, `get_detected_fonts()` | — | -| `get_plugin_fonts()`, `unregister_plugin_fonts()` | — | +| `get_plugin_fonts()` | — | +| `unregister_plugin_fonts()` | `forget_plugin_fonts()` (core calls it on unload) | | `add_font()`, `remove_font()`, `validate_font()` | the web UI's Fonts tab | diff --git a/src/font_manager.py b/src/font_manager.py index c722c712..17990983 100644 --- a/src/font_manager.py +++ b/src/font_manager.py @@ -222,6 +222,39 @@ class FontManager: logger.error(f"Error registering fonts for plugin {plugin_id}: {e}", exc_info=True) return False + def forget_plugin_fonts(self, plugin_id: str) -> bool: + """Drop the fonts ``plugin_id``'s manifest registered: its manifest + and catalog, its ``plugin_id::family`` entries in font_catalog, and + cached font objects for those families. Called by core when a plugin + is unloaded, so a reload registers from its current manifest and a + removed plugin's fonts stop resolving. + + FontManager takes no locks; like forget_manager_fonts this relies on + single dict operations being atomic and iterates snapshots, so a + render thread calling get_font() meanwhile cannot break it. Returns + True if the plugin had registered fonts. + """ + prefix = f"{plugin_id}::" + manifest = self.plugin_fonts.pop(plugin_id, None) + catalog = self.plugin_font_catalogs.pop(plugin_id, None) + # Every namespaced entry, not just the families in the catalog: one + # whose file failed to load never made it into the catalog, and a + # caller may have added one directly. + for family in list(self.font_catalog): + if family.startswith(prefix): + self.font_catalog.pop(family, None) + # get_font() keys the cache f"{family}_{size_px}". + dropped = [key for key in list(self.font_cache) if key.startswith(prefix)] + for key in dropped: + self.font_cache.pop(key, None) + if dropped: + # Font objects someone may hold were dropped; see cache_generation. + self.cache_generation += 1 + if manifest is None and catalog is None: + return False + logger.info("Forgot fonts of plugin %s", plugin_id) + return True + def _validate_font_manifest(self, font_manifest: Dict[str, Any]) -> bool: """Validate the structure of a plugin's font manifest.""" required_fields = ["fonts"] diff --git a/src/plugin_system/plugin_manager.py b/src/plugin_system/plugin_manager.py index 1d2961fb..2ff6fd1f 100644 --- a/src/plugin_system/plugin_manager.py +++ b/src/plugin_system/plugin_manager.py @@ -601,11 +601,21 @@ class PluginManager: self.plugin_loader.unregister_plugin_modules(plugin_id) except Exception as e: # pragma: no cover - defensive self.logger.debug("Could not drop modules of %s: %s", plugin_id, e) - try: - if self.font_manager is not None and hasattr(self.font_manager, 'forget_manager_fonts'): - self.font_manager.forget_manager_fonts(plugin_id) - except Exception as e: - self.logger.debug("Could not forget fonts of %s: %s", plugin_id, e) + self._forget_plugin_fonts(plugin_id) + + def _forget_plugin_fonts(self, plugin_id: str) -> None: + """Drop what the FontManager holds for a plugin: the fonts its + instance reported using (the Fonts tab's "Used by") and the fonts its + manifest registered. Never raises.""" + if self.font_manager is None: + return + for name in ('forget_manager_fonts', 'forget_plugin_fonts'): + if not hasattr(self.font_manager, name): + continue + try: + getattr(self.font_manager, name)(plugin_id) + except Exception as e: + self.logger.debug("Could not forget fonts of %s (%s): %s", plugin_id, name, e) #: Config keys the **core** reads out of a plugin's own config block. The #: plugin never declares them, so a schema with @@ -850,12 +860,9 @@ class PluginManager: # Delegate sub-module and cached-module cleanup to the loader self.plugin_loader.unregister_plugin_modules(plugin_id) - # Its font registrations go with it (the Fonts tab's "Used by"). - try: - if self.font_manager is not None and hasattr(self.font_manager, 'forget_manager_fonts'): - self.font_manager.forget_manager_fonts(plugin_id) - except Exception as e: - self.logger.debug("Could not forget fonts of %s: %s", plugin_id, e) + # Its font registrations go with it: the fonts it reported using + # and the ones its manifest registered. + self._forget_plugin_fonts(plugin_id) # Update state self.state_manager.set_state(plugin_id, PluginState.UNLOADED) diff --git a/test/test_font_manager.py b/test/test_font_manager.py index b377c616..90a3847a 100644 --- a/test/test_font_manager.py +++ b/test/test_font_manager.py @@ -185,6 +185,149 @@ class TestPluginFonts: assert fm.font_catalog["my-plugin::bundled"] == str(plugin_dir / "fonts" / "Bundled.ttf") +class TestForgetPluginFonts: + """forget_plugin_fonts drops what a plugin's manifest registered. Before + it, unloading a plugin left its fonts resolvable and its cached font + objects alive until a restart.""" + + @staticmethod + def _register(fm, root, plugin_id, family="bundled"): + plugin_dir = root / plugin_id + (plugin_dir / "fonts").mkdir(parents=True, exist_ok=True) + font_file = plugin_dir / "fonts" / f"{family}.ttf" + if not font_file.exists(): # a loaded font may hold it open (Windows) + shutil.copy(resolve_asset_path("assets/fonts/PressStart2P-Regular.ttf"), font_file) + manifest = {"fonts": [{"family": family, "source": f"plugin://fonts/{family}.ttf"}]} + assert fm.register_plugin_fonts(plugin_id, manifest, plugin_dir=plugin_dir) + return plugin_dir + + @staticmethod + def _entries_of(fm, plugin_id): + prefix = f"{plugin_id}::" + return { + "plugin_fonts": plugin_id in fm.plugin_fonts, + "plugin_font_catalogs": plugin_id in fm.plugin_font_catalogs, + "font_catalog": [k for k in fm.font_catalog if k.startswith(prefix)], + "font_cache": [k for k in fm.font_cache if k.startswith(prefix)], + } + + NONE = {"plugin_fonts": False, "plugin_font_catalogs": False, + "font_catalog": [], "font_cache": []} + + def test_unload_leaves_no_plugin_entries(self, fm, tmp_path): + self._register(fm, tmp_path, "alpha") + fm.resolve_font("alpha.title", "bundled", 8, plugin_id="alpha") + fm.get_font("alpha::bundled", 10) + assert self._entries_of(fm, "alpha")["font_cache"] # cached before + gen = fm.cache_generation + + assert fm.forget_plugin_fonts("alpha") is True + + assert self._entries_of(fm, "alpha") == self.NONE + assert fm.cache_generation == gen + 1 + # The family no longer resolves to the plugin's file. + assert fm.font_catalog.get("alpha::bundled") is None + + def test_other_plugins_and_core_fonts_are_untouched(self, fm, tmp_path): + self._register(fm, tmp_path, "alpha") + self._register(fm, tmp_path, "beta") + # A plugin whose id is a prefix of another's must not take it along. + self._register(fm, tmp_path, "alpha-two") + for pid in ("alpha", "beta", "alpha-two"): + fm.get_font(f"{pid}::bundled", 8) + core_font = fm.get_font("press_start", 8) + beta_before = self._entries_of(fm, "beta") + alpha_two_before = self._entries_of(fm, "alpha-two") + + fm.forget_plugin_fonts("alpha") + + assert self._entries_of(fm, "beta") == beta_before + assert self._entries_of(fm, "alpha-two") == alpha_two_before + assert fm.get_font("press_start", 8) is core_font + + def test_reload_re_registers_cleanly(self, fm, tmp_path): + plugin_dir = self._register(fm, tmp_path, "alpha") + old = fm.get_font("alpha::bundled", 8) + fm.forget_plugin_fonts("alpha") + + self._register(fm, tmp_path, "alpha") + + assert fm.font_catalog["alpha::bundled"] == str(plugin_dir / "fonts" / "bundled.ttf") + font = fm.resolve_font("alpha.title", "bundled", 8, plugin_id="alpha") + assert isinstance(font, ImageFont.FreeTypeFont) + assert font is not old # loaded fresh, not the dropped cache entry + + def test_a_family_the_new_manifest_drops_stops_resolving(self, fm, tmp_path): + self._register(fm, tmp_path, "alpha", family="old_face") + fm.forget_plugin_fonts("alpha") + self._register(fm, tmp_path, "alpha", family="new_face") + + assert "alpha::old_face" not in fm.font_catalog + assert "alpha::new_face" in fm.font_catalog + + def test_unknown_plugin_is_a_no_op(self, fm): + catalog = dict(fm.font_catalog) + gen = fm.cache_generation + + assert fm.forget_plugin_fonts("never-registered") is False + + assert fm.font_catalog == catalog + assert fm.cache_generation == gen + + +class TestPluginManagerReloadFonts: + """Through PluginManager: unloading a plugin forgets its manifest fonts, + and reload_plugin (unload + load) registers them again so they resolve.""" + + PLUGIN_ID = "font-reload-demo" + MODULE = "plugin_font_reload_demo" + + def test_unload_forgets_and_reload_resolves(self, tmp_path): + import sys + from src.plugin_system.plugin_manager import PluginManager + + plugins_dir = tmp_path / "plugins" + plugin_dir = plugins_dir / self.PLUGIN_ID + (plugin_dir / "fonts").mkdir(parents=True) + shutil.copy(resolve_asset_path("assets/fonts/PressStart2P-Regular.ttf"), + plugin_dir / "fonts" / "Bundled.ttf") + manifest = {"id": self.PLUGIN_ID, "name": "Demo", "class_name": "Demo", + "entry_point": "manager.py", + "fonts": {"fonts": [{"family": "bundled", + "source": "plugin://fonts/Bundled.ttf"}]}} + (plugin_dir / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8") + (plugin_dir / "manager.py").write_text( + "class Demo:\n" + " def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager):\n" + " self.enabled = True\n", encoding="utf-8") + + pm = PluginManager(plugins_dir=str(plugins_dir)) + fm = FontManager({}) + pm.font_manager = fm + pm.plugin_manifests[self.PLUGIN_ID] = manifest + key = f"{self.PLUGIN_ID}::bundled" + try: + assert pm.load_plugin(self.PLUGIN_ID) is True + assert key in fm.font_catalog + fm.register_manager_font(self.PLUGIN_ID, "demo.title", "bundled", 8) + old = fm.resolve_font("demo.title", "bundled", 8, plugin_id=self.PLUGIN_ID) + + assert pm.unload_plugin(self.PLUGIN_ID) is True + assert self.PLUGIN_ID not in fm.plugin_fonts + assert self.PLUGIN_ID not in fm.plugin_font_catalogs + assert key not in fm.font_catalog + assert not [k for k in fm.font_cache if k.startswith(f"{self.PLUGIN_ID}::")] + assert self.PLUGIN_ID not in fm.manager_fonts + + assert pm.reload_plugin(self.PLUGIN_ID) is True + assert fm.font_catalog[key] == str(plugin_dir / "fonts" / "Bundled.ttf") + font = fm.resolve_font("demo.title", "bundled", 8, plugin_id=self.PLUGIN_ID) + assert isinstance(font, ImageFont.FreeTypeFont) + assert font is not old + finally: + sys.modules.pop(self.MODULE, None) + + class TestDownloadFont: """_download_font: plugin fonts declared by URL, cached in temp_font_dir.""" diff --git a/test/test_plugin_manager_failed_load_cleanup.py b/test/test_plugin_manager_failed_load_cleanup.py index a5e10824..f6e8fc00 100644 --- a/test/test_plugin_manager_failed_load_cleanup.py +++ b/test/test_plugin_manager_failed_load_cleanup.py @@ -69,6 +69,7 @@ def test_fixed_plugin_loads_new_code_after_failed_load(plugin_env, first_source) assert MODULE_NAME not in sys.modules assert PLUGIN_ID not in pm.plugin_loader._loaded_modules pm.font_manager.forget_manager_fonts.assert_called_with(PLUGIN_ID) + pm.font_manager.forget_plugin_fonts.assert_called_with(PLUGIN_ID) (plugin_dir / "manager.py").write_text(_FIXED, encoding="utf-8") assert pm.load_plugin(PLUGIN_ID) is True From bb475a79ea8beeba72c0aa4c51974fa65fd9e5be Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 09:06:56 -0400 Subject: [PATCH 20/37] fix(plugins): keep a plugin's tokens and local files across store updates (#755) * fix(plugins): keep a plugin's tokens and local files across store updates A monorepo plugin update replaces the plugin directory with the fresh download and deletes the old copy, taking with it everything the plugin wrote beside itself. On 2026-10-04 updating calendar 1.2.9 -> 1.2.12 deleted token.pickle and credentials.json, and the calendar stopped until they were restored from a backup. Before the set-aside copy is discarded (store update, reinstall over an existing copy, install_from_url replace), carry over files the plugin's .gitignore excludes plus known secret/state files (*.pickle, token.json, credentials.json, config_secrets.json, .pkce_code_verifier). Files the new release ships win; byte code and .git are not carried; if a copy fails the old copy is kept. The git-pull path no longer sweeps untracked tokens into its auto-stash, which is never popped. Co-Authored-By: Claude Opus 5.5 * fix(plugins): find the new copy via _existing_install, as install_plugin does Co-Authored-By: Claude Opus 5.5 * test(on-demand): find the write under test by key, not by position TestARestoreWithNothingToResume took the last cache_manager.set call to be the on-demand state, but the controller's font-usage publisher thread writes font_usage_snapshot to the same mock, and on a slow runner it lands last. Failing on main since #748 (Python 3.11 job). Same fix for the named-mode restart test, which had the same race. Co-Authored-By: Claude Opus 5.5 * test(starlark): fake only the editor launch, not every Popen in the request TestPixletEditorHostDefaultsButDoesNotOverride patched subprocess.Popen for the whole request. When the captive-portal before_request hook's 30s AP-mode cache had expired, its `systemctl is-active hostapd` check went through subprocess.run, got the fake process, and raised TypeError (run() uses the process as a context manager): a 500 instead of 200. Seen on the Python 3.13 job; reproduced locally by forcing the cache to expire. Other calls now reach the real Popen. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 11 + src/plugin_system/plugin_local_files.py | 204 ++++++++++++++++ src/plugin_system/store_install.py | 37 ++- src/plugin_system/store_update.py | 29 ++- test/test_store_update_keeps_local_files.py | 244 ++++++++++++++++++++ 5 files changed, 516 insertions(+), 9 deletions(-) create mode 100644 src/plugin_system/plugin_local_files.py create mode 100644 test/test_store_update_keeps_local_files.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 71c3fbe6..3ba95dce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1590,6 +1590,17 @@ read any of them: ### Fixes +- Updating a plugin from the store no longer deletes the files it wrote + beside itself. A monorepo update replaces the plugin directory with the + fresh download and deletes the old copy, so calendar's Google OAuth files + (`token.pickle`, `credentials.json`) were lost on every update and the + calendar stopped until they were restored by hand. Before the old copy is + removed, the update now copies over anything the plugin's `.gitignore` + excludes plus known secret/state files (`*.pickle`, `token.json`, + `credentials.json`, `config_secrets.json`, `.pkce_code_verifier`); files the + new release ships are never overwritten, and byte code is not carried. A + plugin updated with `git pull` no longer sweeps an untracked token into the + auto-stash, which is never popped (`src/plugin_system/plugin_local_files.py`). - Quieter routine logging. Every rotation logged each mode twice ("Switching to mode", then "Processing mode"), and a mode with nothing to show added "display() returned False" and "No content to display". Those diff --git a/src/plugin_system/plugin_local_files.py b/src/plugin_system/plugin_local_files.py new file mode 100644 index 00000000..1a3ac2f4 --- /dev/null +++ b/src/plugin_system/plugin_local_files.py @@ -0,0 +1,204 @@ +""" +Files a plugin writes beside itself at runtime, which an update must keep. + +A store update replaces a plugin's directory with a fresh download and then +deletes the old copy. Anything the plugin created there -- OAuth tokens, a +client-secrets file, a PKCE verifier, cached state -- is in no release, so the +fresh download does not contain it and deleting the old copy destroys it. On +2026-10-04 updating calendar 1.2.9 -> 1.2.12 that way deleted its +``token.pickle`` and ``credentials.json``, and the calendar stopped until they +were restored from a backup. + +What counts as "the plugin's own local file" is the union of: + +* :data:`KNOWN_STATE_PATTERNS` -- secret and state files plugins are known to + write, kept even when a plugin forgot to gitignore them; and +* whatever the plugin's own ``.gitignore`` (old copy or new) excludes. A file + the author ignores is by definition not part of a release. + +A file the new release ships is never overwritten: tracked content wins. Byte +code (``__pycache__``, ``*.pyc``) and ``.git`` are never carried, since they +belong to the old code rather than to the user. +""" + +from __future__ import annotations + +import fnmatch +import os +import re +import shutil +from pathlib import Path +from typing import Iterable, List, Optional, Pattern, Tuple + +__all__ = [ + 'KNOWN_STATE_PATTERNS', + 'carry_over_local_files', + 'is_known_state_file', + 'local_files_to_keep', +] + +# Basename globs. Kept even when the plugin's .gitignore does not list them. +KNOWN_STATE_PATTERNS: Tuple[str, ...] = ( + 'token.pickle', + '*.pickle', + 'token.json', + 'credentials.json', + 'config_secrets.json', + '.pkce_code_verifier', +) + +_NEVER_CARRY_DIRS = frozenset({'.git', '__pycache__'}) +_NEVER_CARRY_SUFFIXES = ('.pyc', '.pyo') + + +def is_known_state_file(rel_path: str) -> bool: + """True when ``rel_path``'s basename is a known secret/state file.""" + name = rel_path.replace('\\', '/').rsplit('/', 1)[-1] + return any(fnmatch.fnmatchcase(name, p) for p in KNOWN_STATE_PATTERNS) + + +class _GitIgnore: + """The subset of gitignore semantics plugin .gitignore files use. + + Supports comments, ``!`` negation (last match wins), a trailing ``/`` for + directory-only patterns, anchoring by a leading or embedded ``/``, ``*``, + ``?``, ``[...]`` and ``**``. As in git, a file under an ignored directory + is ignored regardless of later negations. + """ + + def __init__(self, lines: Iterable[str]): + self._rules: List[Tuple[Pattern[str], bool, bool]] = [] + for raw in lines: + line = raw.rstrip('\n').rstrip() + if not line or line.startswith('#'): + continue + negate = line.startswith('!') + if negate: + line = line[1:] + elif line.startswith('\\'): + line = line[1:] + dir_only = line.endswith('/') + line = line.rstrip('/') + if not line: + continue + anchored = '/' in line + line = line.lstrip('/') + body = self._translate(line) + regex = body if anchored else r'(?:.*/)?' + body + self._rules.append((re.compile(r'\A' + regex + r'\Z'), negate, dir_only)) + + @staticmethod + def _translate(pattern: str) -> str: + out, i, n = [], 0, len(pattern) + while i < n: + if pattern.startswith('**/', i): + out.append(r'(?:.*/)?') + i += 3 + elif pattern.startswith('/**', i) and i + 3 == n: + out.append(r'/.*') + i += 3 + elif pattern.startswith('**', i): + out.append(r'.*') + i += 2 + elif pattern[i] == '*': + out.append(r'[^/]*') + i += 1 + elif pattern[i] == '?': + out.append(r'[^/]') + i += 1 + elif pattern[i] == '[': + end = pattern.find(']', i + 1) + if end == -1: + out.append(re.escape('[')) + i += 1 + else: + cls = pattern[i + 1:end] + if cls.startswith('!'): + cls = '^' + cls[1:] + out.append('[' + cls.replace('\\', '\\\\') + ']') + i = end + 1 + else: + out.append(re.escape(pattern[i])) + i += 1 + return ''.join(out) + + def _decide(self, rel: str, is_dir: bool) -> Optional[bool]: + verdict = None + for regex, negate, dir_only in self._rules: + if dir_only and not is_dir: + continue + if regex.match(rel): + verdict = not negate + return verdict + + def ignores(self, rel_path: str) -> bool: + if not self._rules: + return False + parts = rel_path.replace('\\', '/').split('/') + for depth in range(1, len(parts)): + if self._decide('/'.join(parts[:depth]), True): + return True + return bool(self._decide('/'.join(parts), False)) + + +def _read_gitignore(plugin_dir: Path) -> List[str]: + try: + return (plugin_dir / '.gitignore').read_text( + encoding='utf-8', errors='replace').splitlines() + except OSError: + return [] + + +def local_files_to_keep(old_dir: Path, new_dir: Path) -> List[str]: + """Relative paths (``/``-separated) in ``old_dir`` to copy into ``new_dir``. + + Regular files only; symlinks and anything the new release already ships + are skipped. + """ + old_dir, new_dir = Path(old_dir), Path(new_dir) + ignore = _GitIgnore(_read_gitignore(old_dir) + _read_gitignore(new_dir)) + keep: List[str] = [] + for root, dirs, files in os.walk(old_dir): + dirs[:] = sorted(d for d in dirs if d not in _NEVER_CARRY_DIRS + and not os.path.islink(os.path.join(root, d))) + rel_root = os.path.relpath(root, old_dir) + for name in sorted(files): + if name.endswith(_NEVER_CARRY_SUFFIXES): + continue + full = os.path.join(root, name) + if os.path.islink(full) or not os.path.isfile(full): + continue + rel = name if rel_root == '.' else f"{rel_root}/{name}".replace('\\', '/') + if not (is_known_state_file(rel) or ignore.ignores(rel)): + continue + if os.path.lexists(new_dir / rel): + continue + keep.append(rel) + return keep + + +def carry_over_local_files( + old_dir: Path, new_dir: Path +) -> Tuple[List[str], List[Tuple[str, str]]]: + """Copy the plugin's local files from ``old_dir`` into ``new_dir``. + + Copies rather than moves, so ``old_dir`` stays a complete copy until the + caller deletes it. Returns ``(copied, failed)`` where ``failed`` pairs a + relative path with the error; the caller should keep ``old_dir`` when + anything failed. + """ + copied: List[str] = [] + failed: List[Tuple[str, str]] = [] + try: + candidates = local_files_to_keep(old_dir, new_dir) + except OSError as e: + return copied, [('.', str(e))] + for rel in candidates: + dest = Path(new_dir) / rel + try: + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(Path(old_dir) / rel, dest) + copied.append(rel) + except OSError as e: + failed.append((rel, str(e))) + return copied, failed diff --git a/src/plugin_system/store_install.py b/src/plugin_system/store_install.py index 1665f7bc..8e8a90b4 100644 --- a/src/plugin_system/store_install.py +++ b/src/plugin_system/store_install.py @@ -22,6 +22,7 @@ from src.plugin_system.plugin_loader import ( contained_plugin_dir, requirements_to_install, ) from src.plugin_system.plugin_dirs import BACKUP_MARKER +from src.plugin_system.plugin_local_files import carry_over_local_files from src.plugin_system.repo_urls import ( USER_AGENT, github_api_headers, github_owner_repo, normalize_repo_url, ) @@ -92,7 +93,9 @@ class _InstallMixin: raise if installed: - self._discard_backup(plugin_id, backup_path, "install") + self._discard_backup( + plugin_id, backup_path, "install", + new_path=self._existing_install(plugin_id) or plugin_path) return True self._restore_backup(plugin_id, plugin_path, backup_path, "Install") @@ -133,8 +136,33 @@ class _InstallMixin: return f"could not set aside {plugin_path}: {e}" return None - def _discard_backup(self, plugin_id: str, backup_path: Path, action: str) -> None: - """Remove the set-aside copy after a successful (re)install.""" + def _discard_backup( + self, plugin_id: str, backup_path: Path, action: str, + new_path: Optional[Path] = None, + ) -> None: + """Remove the set-aside copy after a successful (re)install. + + With ``new_path`` (where the new copy landed), first carries the + plugin's own runtime files -- OAuth tokens, client secrets, anything + its .gitignore excludes -- from the old copy into the new one: no + release contains them, so deleting the old copy would destroy them. + See src/plugin_system/plugin_local_files.py. If any could not be + copied the old copy is kept, so nothing is lost. + """ + if new_path is not None and new_path.is_dir(): + copied, failed = carry_over_local_files(backup_path, new_path) + if copied: + self.logger.info( + "Kept %d local file(s) of %s across the %s: %s", + len(copied), plugin_id, action, ", ".join(copied)) + if failed: + self.logger.error( + "Could not carry %s's local files into the new copy (%s); " + "the previous copy is kept at %s -- copy them back by hand", + plugin_id, + "; ".join(f"{rel}: {err}" for rel, err in failed), + backup_path) + return if not self._safe_remove_directory(backup_path): self.logger.warning( "%s of %s succeeded but the previous copy at %s could not be " @@ -542,7 +570,8 @@ class _InstallMixin: raise temp_dir = None # Prevent cleanup since we moved it if backup_path is not None: - self._discard_backup(plugin_id, backup_path, "install") + self._discard_backup( + plugin_id, backup_path, "install", new_path=final_path) # Install dependencies self._install_dependencies(final_path) diff --git a/src/plugin_system/store_update.py b/src/plugin_system/store_update.py index 89f63a4b..4c1ec480 100644 --- a/src/plugin_system/store_update.py +++ b/src/plugin_system/store_update.py @@ -10,6 +10,9 @@ import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep from pathlib import Path from typing import Dict, Optional, Tuple from src.plugin_system.plugin_dirs import BACKUP_MARKER +from src.plugin_system.plugin_local_files import ( + KNOWN_STATE_PATTERNS, is_known_state_file, +) from src.plugin_system.repo_urls import same_repo @@ -302,7 +305,11 @@ class _UpdateMixin: installed = False if installed: - self._discard_backup(plugin_id, backup_path, "update") + # install_plugin may land the new copy under the manifest id + # rather than the old directory name. + self._discard_backup( + plugin_id, backup_path, "update", + new_path=self._existing_install(plugin_id) or plugin_path) return True # Bad network, registry error...: the user keeps a working plugin. @@ -509,8 +516,12 @@ class _UpdateMixin: for line in untracked_result.stdout.strip().split('\n'): if line.startswith('??'): # Untracked file - file_path = line[3:].strip() - untracked_files.append(file_path) + file_path = line[3:].strip().strip('"') + # Tokens and secrets stay out of the + # stash (see below), so they alone are + # not a reason to stash. + if not is_known_state_file(file_path): + untracked_files.append(file_path) # Check for tracked file changes status_result = subprocess.run( @@ -537,9 +548,17 @@ class _UpdateMixin: if has_changes: self.logger.info(f"Stashing local changes in {plugin_id} before update") try: - # Use -u to include untracked files in stash + # Use -u to include untracked files in stash -- + # except the plugin's tokens and secrets, which a + # repo may have forgotten to gitignore. The stash + # is never popped, so a stashed token.pickle would + # vanish from the plugin and break it. + stash_cmd = ( + ['git', '-C', str(plugin_path), 'stash', 'push', '-u', + '-m', f'LEDMatrix auto-stash before update {plugin_id}', '--', '.'] + + [f':(exclude,glob)**/{p}' for p in KNOWN_STATE_PATTERNS]) stash_result = subprocess.run( - ['git', '-C', str(plugin_path), 'stash', 'push', '-u', '-m', f'LEDMatrix auto-stash before update {plugin_id}'], + stash_cmd, capture_output=True, text=True, timeout=30, diff --git a/test/test_store_update_keeps_local_files.py b/test/test_store_update_keeps_local_files.py new file mode 100644 index 00000000..28618236 --- /dev/null +++ b/test/test_store_update_keeps_local_files.py @@ -0,0 +1,244 @@ +"""A plugin update must keep the files the plugin wrote beside itself. + +Field incident, 2026-10-04: updating calendar 1.2.9 -> 1.2.12 from the web UI +replaced plugin-repos/calendar/ with the fresh download and deleted the old +copy -- and with it token.pickle and credentials.json, the plugin's Google +OAuth files. No release contains them (the repo gitignores them), so the hot +reload logged "Credentials file not found" and the calendar stayed broken +until the files were restored by hand. + +Both update routes are covered: a monorepo plugin (registry ``plugin_path``), +which is reinstalled into a fresh directory, and a plugin installed from its +own git repository, which is updated with ``git pull`` after an auto-stash. +""" + +import json +import shutil +import subprocess + +import pytest + +from src.plugin_system.plugin_local_files import ( + is_known_state_file, local_files_to_keep, +) +from src.plugin_system.store_manager import PluginStoreManager + +PLUGIN_ID = "calendar" + + +def _manifest(version): + return {"id": PLUGIN_ID, "name": "Calendar", "class_name": "CalendarPlugin", + "display_modes": ["calendar"], "version": version} + + +def _write_release(target, version): + """What a download of ``version`` puts on disk.""" + target.mkdir(parents=True, exist_ok=True) + (target / "manifest.json").write_text(json.dumps(_manifest(version))) + (target / "manager.py").write_text(f"VERSION = {version!r}\n") + (target / ".gitignore").write_text("credentials.json\ntoken.pickle\ncache/\n") + + +def _drop_local_files(plugin_dir): + """What the plugin writes at runtime: OAuth files plus cached state.""" + (plugin_dir / "token.pickle").write_bytes(b"\x80\x04oauth-token") + (plugin_dir / "credentials.json").write_text('{"installed": {}}') + (plugin_dir / "cache").mkdir() + (plugin_dir / "cache" / "events.json").write_text("[]") + + +def _assert_local_files_kept(plugin_dir): + assert (plugin_dir / "token.pickle").read_bytes() == b"\x80\x04oauth-token" + assert (plugin_dir / "credentials.json").read_text() == '{"installed": {}}' + assert (plugin_dir / "cache" / "events.json").read_text() == "[]" + + +def _leftover_backups(plugins_dir): + return [p.name for p in plugins_dir.iterdir() if "standalone-backup" in p.name] + + +@pytest.fixture +def store(tmp_path, monkeypatch): + mgr = PluginStoreManager( + plugins_dir=str(tmp_path / "plugin-repos"), + uninstalled_registry_path=str(tmp_path / "uninstalled.json")) + mgr.plugins_dir.mkdir(parents=True, exist_ok=True) + monkeypatch.setattr(mgr, "_install_dependencies", lambda *a, **k: True) + monkeypatch.setattr(mgr, "fetch_registry", lambda *a, **k: {"plugins": []}) + return mgr + + +class TestMonorepoUpdate: + @pytest.fixture + def installed(self, store, monkeypatch): + registry_entry = { + "id": PLUGIN_ID, "repo": "https://github.com/ChuckBuilds/ledmatrix-plugins", + "plugin_path": "plugins/calendar", "branch": "main", + "latest_version": "1.2.9", + } + monkeypatch.setattr(store, "get_plugin_info", lambda *a, **k: registry_entry) + release = {"version": "1.2.9"} + + def fake_monorepo_download(download_url, plugin_subpath, target): + assert plugin_subpath == "plugins/calendar" + _write_release(target, release["version"]) + return True + + monkeypatch.setattr(store, "_install_from_monorepo", fake_monorepo_download) + assert store.install_plugin(PLUGIN_ID) is True + + def publish(version): + registry_entry["latest_version"] = release["version"] = version + return store, store.plugins_dir / PLUGIN_ID, publish + + def test_update_keeps_token_and_gitignored_files(self, installed): + store, plugin_dir, publish = installed + _drop_local_files(plugin_dir) + + publish("1.2.12") + assert store.update_plugin(PLUGIN_ID) is True + + assert json.loads((plugin_dir / "manifest.json").read_text())["version"] == "1.2.12" + _assert_local_files_kept(plugin_dir) + assert _leftover_backups(store.plugins_dir) == [] + + def test_token_is_kept_even_when_the_release_does_not_gitignore_it(self, installed): + store, plugin_dir, publish = installed + (plugin_dir / ".gitignore").unlink() + (plugin_dir / "token.pickle").write_bytes(b"tok") + (plugin_dir / "config_secrets.json").write_text("{}") + + publish("1.2.12") + assert store.update_plugin(PLUGIN_ID) is True + + assert (plugin_dir / "token.pickle").read_bytes() == b"tok" + assert (plugin_dir / "config_secrets.json").read_text() == "{}" + + def test_release_content_wins_and_old_code_is_not_carried(self, installed): + store, plugin_dir, publish = installed + # A file the old copy had that the new release dropped, byte code, and + # an old copy of a file the new release also ships. + (plugin_dir / "removed_module.py").write_text("OLD = True\n") + (plugin_dir / "__pycache__").mkdir() + (plugin_dir / "__pycache__" / "manager.cpython-313.pyc").write_bytes(b"pyc") + + publish("1.2.12") + assert store.update_plugin(PLUGIN_ID) is True + + assert not (plugin_dir / "removed_module.py").exists() + assert not (plugin_dir / "__pycache__").exists() + assert "1.2.12" in (plugin_dir / "manager.py").read_text() + + def test_reinstall_over_an_existing_copy_keeps_them_too(self, installed): + store, plugin_dir, publish = installed + _drop_local_files(plugin_dir) + + assert store.install_plugin(PLUGIN_ID) is True + + _assert_local_files_kept(plugin_dir) + assert _leftover_backups(store.plugins_dir) == [] + + +class TestInstallFromUrlReplace: + def test_replacing_an_installed_copy_keeps_the_token(self, store, monkeypatch): + plugin_dir = store.plugins_dir / PLUGIN_ID + _write_release(plugin_dir, "1.0.0") + _drop_local_files(plugin_dir) + + def fake_clone(repo_url, target, branches): + _write_release(target, "2.0.0") + return "main" + + monkeypatch.setattr(store, "_install_via_git", fake_clone) + result = store.install_from_url( + "https://github.com/example/ledmatrix-calendar", plugin_id=PLUGIN_ID) + + assert result["success"] is True + assert json.loads((plugin_dir / "manifest.json").read_text())["version"] == "2.0.0" + _assert_local_files_kept(plugin_dir) + + +def _git(*args, cwd): + subprocess.run(["git", "-c", "user.email=t@example.com", "-c", "user.name=t", + "-c", "core.autocrlf=false", *args], + cwd=cwd, check=True, capture_output=True) + + +@pytest.mark.skipif(shutil.which("git") is None, reason="git not installed") +class TestGitRepoUpdate: + @pytest.fixture + def cloned(self, store, tmp_path, monkeypatch): + monkeypatch.setattr(store, "get_plugin_info", lambda *a, **k: None) + upstream = tmp_path / "upstream" + _write_release(upstream, "1.0.0") + # This repo does NOT gitignore the token: an untracked, non-ignored + # file is exactly what `git stash push -u` used to sweep away. + (upstream / ".gitignore").write_text("cache/\n") + _git("init", "-q", "-b", "main", cwd=upstream) + _git("add", ".", cwd=upstream) + _git("commit", "-qm", "1.0.0", cwd=upstream) + + plugin_dir = store.plugins_dir / PLUGIN_ID + _git("clone", "-q", str(upstream), str(plugin_dir), cwd=tmp_path) + + def publish(version): + (upstream / "manifest.json").write_text(json.dumps(_manifest(version))) + _git("commit", "-qam", version, cwd=upstream) + return store, plugin_dir, publish + + def test_pull_update_keeps_untracked_token(self, cloned): + store, plugin_dir, publish = cloned + _drop_local_files(plugin_dir) + # An unrelated untracked file, so the update really does stash. + (plugin_dir / "notes.txt").write_text("scratch") + + publish("1.1.0") + assert store.update_plugin(PLUGIN_ID) is True + + assert json.loads((plugin_dir / "manifest.json").read_text())["version"] == "1.1.0" + _assert_local_files_kept(plugin_dir) + + def test_token_alone_does_not_trigger_a_stash(self, cloned): + store, plugin_dir, publish = cloned + (plugin_dir / "token.pickle").write_bytes(b"tok") + + publish("1.1.0") + assert store.update_plugin(PLUGIN_ID) is True + + assert (plugin_dir / "token.pickle").read_bytes() == b"tok" + stashes = subprocess.run(["git", "-C", str(plugin_dir), "stash", "list"], + capture_output=True, text=True, check=True) + assert stashes.stdout.strip() == "" + + +class TestWhatIsKept: + @pytest.mark.parametrize("path,expected", [ + ("token.pickle", True), + ("data/session.pickle", True), + ("credentials.json", True), + ("token.json", True), + ("config_secrets.json", True), + (".pkce_code_verifier", True), + ("manager.py", False), + ("config.json", False), + ]) + def test_known_state_files(self, path, expected): + assert is_known_state_file(path) is expected + + def test_gitignore_rules(self, tmp_path): + old, new = tmp_path / "old", tmp_path / "new" + new.mkdir() + for rel in ["a.log", "logs/x.txt", "sub/deep/b.log", "keep.log", + "anchored.txt", "sub/anchored.txt", "assets/x/y_backup/z.png", + "manager.py", "shipped.log"]: + (old / rel).parent.mkdir(parents=True, exist_ok=True) + (old / rel).write_text("x") + (new / "shipped.log").write_text("new") + (old / ".gitignore").write_text( + "# comment\n*.log\n!keep.log\nlogs/\n/anchored.txt\n" + "assets/**/*_backup/\n") + + assert local_files_to_keep(old, new) == [ + "a.log", "anchored.txt", "assets/x/y_backup/z.png", + "logs/x.txt", "sub/deep/b.log", + ] From b638b91169ad4be515c488cf3062c6d09e9214d3 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 09:36:12 -0400 Subject: [PATCH 21/37] feat(common): sports_game_over -- the reconciled game-over check (sports family 5) (#770) * feat(common): sports_game_over -- the reconciled game-over check (sports family 5) New hardware-free module src/common/sports_game_over.py with SportsGameOverMixin._is_game_really_over, the scoreboards' SportsLive check that drops a game ESPN still lists as live, copied from ledmatrix-plugins claude/family5-reconcile once the nine copies (five bodies) became one. Over on a final period text; from period FINAL_PERIOD on, also on a 0:00 clock string unless the score is level (a tie at the end of regulation goes to overtime; a game that ends tied ends on its final status). FINAL_PERIOD is the one per-sport seam, a class attribute defaulting to None (the clock never ends a game); the scoreboards declare 3 (hockey), 4 (basketball, football, lacrosse) or None (afl, nrl, soccer, baseball, ufc). - test/test_sports_game_over.py: the plugins' pinned matrix folded to the three FINAL_PERIOD values, edge shapes, the tie guard, ufc's recorded ESPN MMA states, an override deferring through super() (baseball), the base order with SportsLiveSharedMixin._detect_stale_games, host contract. - test/test_sports_game_over_parity.py: with LEDMATRIX_PLUGINS, compares the body with every plugin copy (drift-report normalisation plus decorators) and each plugin's FINAL_PERIOD with the owner's decision. - mypy ratchet, src/common/README.md, CHANGELOG (Unreleased, New modules). - docs/SPORTS_UNIFICATION.md: family 5 status and decisions; the seam tables now match the code (FINAL_PERIOD defaults to None; the CLOCK_COUNTS_DOWN seam never existed and is gone from the doc). Co-Authored-By: Claude Opus 5.5 * docs: cite ledmatrix-plugins #625 for the family 5 reconcile Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 10 + docs/SPORTS_UNIFICATION.md | 55 ++++-- mypy-clean.txt | 1 + src/common/README.md | 11 ++ src/common/sports_game_over.py | 125 ++++++++++++ src/common/sports_shared.py | 4 +- test/test_sports_game_over.py | 275 +++++++++++++++++++++++++++ test/test_sports_game_over_parity.py | 123 ++++++++++++ 8 files changed, 584 insertions(+), 20 deletions(-) create mode 100644 src/common/sports_game_over.py create mode 100644 test/test_sports_game_over.py create mode 100644 test/test_sports_game_over_parity.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ba95dce..0a260dad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -668,6 +668,16 @@ policies are unchanged. - `src/display_arbiter.py` -- the display loop's Arbiter (see Tooling). Core-internal: plugins have no reason to import it, so it sets no `ledmatrix_min_version` floor. +- `src/common/sports_game_over.py` -- `SportsGameOverMixin`, sports + consolidation family 5: `_is_game_really_over`, the scoreboards' + `SportsLive` check that drops a game ESPN still lists as live, once the + plugins made their five bodies one. Over on a final period text, or on a + 0:00 clock from period `FINAL_PERIOD` on unless the score is level (a tie + at the end of regulation goes to overtime). `FINAL_PERIOD` is the per-sport + class attribute, `None` by default (the clock never ends a game); the + scoreboards declare 3 (hockey), 4 (basketball, football, lacrosse) or + `None`. List the mixin before `SportsLiveSharedMixin`. A plugin may import + it once it floors on the release that ships it, and deletes its copy then. ### Tooling diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index fe72fe06..3c7072f2 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -91,6 +91,7 @@ more. Shared sports code lives in `src/common`: | `sports_live_scroll.py` | next release | `SportsLiveScrollMixin` — rebuild a live scroll strip mid-cycle, keeping the marquee's place | | `sports_display_rules.py` | next release | `SportsCardOptionsMixin`, `SportsGameRulesMixin` — scorebug date options, the no-favourites filter, non-favourite live dwell | | `sports_font_path.py` | next release | `resolve_font_path` — what the plugins' `_resolve_font_path` copies return | +| `sports_game_over.py` | next release | `SportsGameOverMixin` — `_is_game_really_over`, with the `FINAL_PERIOD` seam (family 5) | Each is described in [src/common/README.md](../src/common/README.md). @@ -134,8 +135,7 @@ constants rather than behavior: | Attribute | Meaning | Default | |---|---|---| -| `FINAL_PERIOD` | Period at/after which a zero clock can mean "over" | `4` (hockey overrides to `3`) | -| `CLOCK_COUNTS_DOWN` | Whether `0:00` means "expired" | `True` (soccer/afl/nrl override to `False` — their clocks count up, so `0:00` is kickoff) | +| `FINAL_PERIOD` | Period from which a 0:00 clock ends a game (`sports_game_over`) | `None`: the clock never ends a game (afl, nrl, soccer, baseball, ufc). Hockey sets `3`; basketball, football and lacrosse `4` | | `COALESCE_SCORING_SEQUENCE` | Fold score increments arriving during an active celebration into that one celebration | `False` (football overrides to `True` — a touchdown lands as +6, then +1 for the extra point) | ### Why these are seams and not branches @@ -146,11 +146,14 @@ so NRL matches favorites on team ID. Flattening every plugin to abbreviations would silently select the wrong club for NRL users. The base declares the seam, NRL fills it, and core never learns the string `"nrl"`. -`CLOCK_COUNTS_DOWN` exists for the same reason in the opposite direction: a +`FINAL_PERIOD` exists for the same reason in the opposite direction: a soccer clock reading `0:00` means the match has not kicked off, so running the -clock-expiry branch there would evict live games. +clock-expiry rule there would evict live games. Those sports declare `None`, +and so do baseball (innings, not a clock) and ufc (a bout ends only on ESPN's +final status). One attribute covers both questions, whether the clock can end +a game and from which period, so no separate count-down flag was added. -`COALESCE_SCORING_SEQUENCE` is the third of the same kind. In football one +`COALESCE_SCORING_SEQUENCE` is another of the same kind. In football one scoring play arrives as two score updates, so the follow-up must be folded into the first celebration; in soccer two increments a few seconds apart are two real goals, and folding them would swallow one. Neither default is "right" — which is @@ -298,6 +301,20 @@ Left in the plugins, though identical: renderers) is already core's, in `SportsHelpersMixin`; a renderer that wants it can inherit that. +### Family 5: the game-over check (core done; adoption waits for a release) + +The pilot of the method below. ledmatrix-plugins `scripts/test_game_over_check.py` +(#621) pinned 3,115 answers across the nine plugins first; the reconcile +(ledmatrix-plugins #625) made the five bodies one and +changed only the cells the owner's decisions under +[Product decisions](#product-decisions-each-family-needs) explain: ufc's +clock rule (65 cells), baseball's dormant one (53, every one a game with a +`period` baseball's games never carry), and a level score at 0:00 (five +cells in hockey, basketball, football and lacrosse). The harness renders +were pixel-identical. `src/common/sports_game_over.py` holds the body; +`test/test_sports_game_over_parity.py` compares it, and each plugin's +`FINAL_PERIOD`, with the plugin copies. + ### Why the method changes Byte-identical promotion has nearly run dry. Measured on ledmatrix-plugins @@ -333,8 +350,8 @@ game-over check); the report measures each method in it. The procedure: line in each plugin. - *A per-sport fact* (hockey ends in period 3; a soccer clock counts up). Make it a declared class constant or override point with a default, as - `FINAL_PERIOD`, `CLOCK_COUNTS_DOWN`, `COALESCE_SCORING_SEQUENCE` and - `_favorite_key` are, and add it to the tables above. Never a sport-name + `FINAL_PERIOD`, `COALESCE_SCORING_SEQUENCE` and `_favorite_key` are, + and add it to the tables above. Never a sport-name branch: core must not learn sport names. - *A product difference*: anything a user can see (which games show, a colour, a date, a badge, how long a screen stays). The owner picks the @@ -382,7 +399,7 @@ release. | # | Family | Methods (variants) | Why here | |---|---|---|---| | 4 | Identical sweep | `manager.py`: `_dispatch_switch_refresh`, `_favorite_team_is_live`, `get_vegas_priority_weight`, `_game_involves`, `_favorite_scan_targets`, `_favorite_scan_games`, `_get_total_games_for_manager` (all nine, 1); the live-scroll helpers `_preserving_scroll_position`, `_refresh_live_scroll_managers`, `_live_scroll_managers`, `_note_live_scroll_built`, `_live_scroll_needs_rebuild`, `_live_scroll_fields` (eight, 1). `sports.py`: `_card_option`, `_filtered_or_all`, `_effective_live_duration`, `_recent_date_text` (eight, 1). 58 identical families in all | Nothing to decide; brings `manager.py` into core as a `SportsPluginHostMixin`. `_resolve_font_path` (identical in nine `sports.py` and eight renderers) becomes `sports_font_path.resolve_font_path`, not `font_layout.resolve_asset_path`, which skips the cwd. Core side done; see [Stage 4](#stage-4-the-identical-sweep-core-done-adoption-waits-for-a-release) | -| 5 | Game-over check | `SportsLive._is_game_really_over` (5) | Pure logic, no pixels; its seams (`FINAL_PERIOD`, `CLOCK_COUNTS_DOWN`) were designed in B1. The pilot for the procedure | +| 5 | Game-over check | `SportsLive._is_game_really_over` (5) | Pure logic, no pixels; one seam, `FINAL_PERIOD`. The pilot for the procedure. Reconciled to one body and promoted as `sports_game_over`; adoption waits for the release that ships it. See [Family 5](#family-5-the-game-over-check-core-done-adoption-waits-for-a-release) | | 6 | Favourite matching | `_is_favorite_game` (7 across three classes), `_select_games_for_display` (2: nrl), `_select_recent_games_for_display` (3) | Everything that asks "is this a favourite" goes through the 3.5.0 `_favorite_key` seam | | 7 | Other-games rotation | `_by_importance`, `_other_games_window`, `_advance_other_games_if_due` (2 each: football), `_rotate_other_games_on_display` (2: ufc) | One outlier each; football carries two fixes the other eight lack | | 8 | Rankings | `_fetch_team_rankings` (3), `_choose_poll` (3), `_load_division_team_ids`, `_passes_other_filters`, `_best_rank`, `_is_ranked_game` (2 each: football) | Needs 7; the rank badge and the "ranked only" filter read it | @@ -416,17 +433,17 @@ family 9 prepares. Owner calls to make before (or while) reconciling. Items marked *verify* are suspected behaviour that needs a payload or a rig to confirm first. -- **5, game-over check.** Which rule each sport gets: the clock never ends a - game in afl, nrl and soccer (`CLOCK_COUNTS_DOWN = False`); hockey ends at - 0:00 from period 3, basketball, football and lacrosse from period 4. - baseball and ufc share a copy that reads a missing clock as "0:00": dormant - in baseball (its games carry no `period`), and not triggered by ufc's round - breaks either. ESPN sends a break as `STATUS_END_OF_ROUND` with displayClock - `-`, not `0:00` (verified against recorded payloads; ledmatrix-plugins#580 - pins it). Whatever rule ufc gets must not read `-` as `0:00`. Decide ufc's - rule: no clock rule (ESPN's `STATUS_FINAL` is the only end signal it needs; - this also closes a ~1 s window at the horn when the ticking clock reads - `0:00`), or its own final period. +- **5, game-over check. Decided 2026-10-05, done:** one seam, + `FINAL_PERIOD`: hockey 3; basketball, football and lacrosse 4; `None` (the + clock never ends a game) for afl, nrl and soccer (clocks that count up), + baseball (its games carry no `period`, so the old rule was dormant) and + ufc (a bout ends only on ESPN's final status, which also closes the ~1 s + window at the horn when the ticking clock reads `0:00`; ESPN's round-break + displayClock `-` was never a zero clock, ledmatrix-plugins#580). Only a + non-empty clock string counts (the baseball/ufc copy read a missing clock + as `0:00`). A score level at 0:00 is not over: the game stays live through + the break before overtime, and one that really ends tied ends on its final + status. Baseball keeps its postponed/suspended override in `BaseballLive`. - **6, favourite matching.** NRL keeps matching favourites by team id (abbreviations collide: NEW, CAN), through `_favorite_key` rather than its own copies of the selection methods. Six plugins log the recent-games diff --git a/mypy-clean.txt b/mypy-clean.txt index 63f6f5bc..ffc7a6c5 100644 --- a/mypy-clean.txt +++ b/mypy-clean.txt @@ -38,6 +38,7 @@ src/common/sports_celebration.py src/common/sports_display_rules.py src/common/sports_fetch.py src/common/sports_font_path.py +src/common/sports_game_over.py src/common/sports_live_scroll.py src/common/sports_plugin_host.py src/common/sports_scroll.py diff --git a/src/common/README.md b/src/common/README.md index a0f46287..b84198c7 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -46,6 +46,7 @@ Rules for the package: | [`sports_display_rules`](#sports_display_rules) | Which games a scoreboard shows, for how long, and its scorebug date line | Yes (scoreboards) | 3.8.0 | | [`sports_fetch`](#sports_fetch) | Scoreboard season fetch, lookback and live-odds decisions | Yes (scoreboards) | 3.7.0 | | [`sports_font_path`](#sports_font_path) | Find a scoreboard's bundled font whatever the cwd | Yes (scoreboards) | 3.8.0 | +| [`sports_game_over`](#sports_game_over) | Whether a game ESPN still lists as live has ended | Yes (scoreboards) | next release | | [`sports_game_renderer`](#sports_game_renderer) | Scoreboard scroll/Vegas card geometry | Yes (scoreboards) | 3.3.0 | | [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 | | [`sports_live_scroll`](#sports_live_scroll) | Rebuild a live scroll strip mid-cycle without moving it | Yes (scoreboards) | 3.8.0 | @@ -294,6 +295,16 @@ path as given when it exists (relative to the cwd), else `font_layout.resolve_asset_path(path)`. What the scoreboards' `_resolve_font_path` copies return on a core that ships it. +### sports_game_over + +[`sports_game_over.py`](sports_game_over.py). `SportsGameOverMixin`: +`_is_game_really_over(game)`, the `SportsLive` check that drops a game ESPN +still lists as live (`SportsLiveSharedMixin._detect_stale_games` calls it). +Over on a final period text, or on a 0:00 clock from period `FINAL_PERIOD` +on unless the score is level. `FINAL_PERIOD` is a class attribute the host +sets per sport; the default `None` means the clock never ends a game. List +it before `SportsLiveSharedMixin`. + ### sports_game_renderer [`sports_game_renderer.py`](sports_game_renderer.py). diff --git a/src/common/sports_game_over.py b/src/common/sports_game_over.py new file mode 100644 index 00000000..3288af7e --- /dev/null +++ b/src/common/sports_game_over.py @@ -0,0 +1,125 @@ +"""Whether a game ESPN still lists as live has in fact ended (sports family 5). + +``SportsGameOverMixin._is_game_really_over`` is the scoreboards' +``SportsLive._is_game_really_over``, reconciled in ledmatrix-plugins +#625 from five bodies into one and copied here under +its existing name. ``SportsLiveSharedMixin._detect_stale_games`` +(``src.common.sports_shared``) calls it on every live game, and the plugins' +live-priority filters call it too, to drop a game ESPN still reports as +in progress. + +A game is over when its period text says final. From period ``FINAL_PERIOD`` +on, a clock reading 0:00 ends it too, unless the score is level: a tie at the +end of regulation goes to overtime (or a shootout), and a game that does end +tied says final. Only a clock *string* is read ("0:00" and ":00" are zero; +":40", "0.0" and ESPN's "-" between MMA rounds are not), and a missing or +unreadable score leaves the decision to the clock. + +``FINAL_PERIOD`` is the one per-sport fact, a class attribute rather than a +sport-name branch. The scoreboards declare it on their ``SportsLive``: + +- 3: hockey; +- 4: basketball, football, lacrosse; +- ``None`` (this default; the clock never ends a game): afl, nrl and soccer, + whose clocks count up; baseball, which has innings; ufc, whose bouts end + only on ESPN's final status. + +A sport can still override the method and defer to it, as baseball's +``BaseballLive`` does to end postponed and suspended games first. + +A new module rather than another method on ``sports_shared``, for the reason +``sports_helpers`` gives: a missing module fails at load, where the version +checks see it; a missing method fails mid-update. + +WHAT A HOST MUST PROVIDE +------------------------ +Derived by walking every ``self.`` the mixin reads; the host-contract +test in ``test/test_sports_game_over.py`` fails if a read is added without +being listed here. + +- ``logger`` -- a ``logging.Logger``; the method logs its verdict at DEBUG. +- ``FINAL_PERIOD`` -- defaulted here to ``None``; set it on the host class. + +The method reads the game dict's ``away_abbr``, ``home_abbr``, +``period_text``, ``period``, ``clock``, ``away_score`` and ``home_score`` +(``_extract_game_details_common``'s keys); any of them may be missing or +null. + +BASE ORDER +---------- +List the mixin before ``SportsLiveSharedMixin`` -- +``class SportsLive(SportsGameOverMixin, SportsLiveSharedMixin, SportsCore)`` -- +so the shared mixin's ``_detect_stale_games`` finds this method through the +MRO. Neither shared mixin defines it, so the order does not change which body +runs today; it keeps the method next to its caller should one ever be added +there. A method on the plugin's own class still wins, and its ``super()`` +reaches this one. The mixin has no ``__init__`` and no state. +""" + +import logging +from typing import Dict, Optional + + +class SportsGameOverMixin: + """The live manager's "is this game really over?" check. See module docstring.""" + + # The host contract, declared for type checking only. + logger: logging.Logger + + #: Period from which a 0:00 clock ends a game; None: the clock never does. + FINAL_PERIOD: Optional[int] = None + + def _is_game_really_over(self, game: Dict) -> bool: + """Whether a game ESPN still lists as live has in fact ended. + + It has when its period text says final. From period ``FINAL_PERIOD`` + on, a clock at 0:00 ends it too, unless the score is level: a tie at + the end of regulation goes to overtime, and a game that does end tied + says final. With ``FINAL_PERIOD = None`` the clock never ends a game. + """ + game_str = f"{game.get('away_abbr')}@{game.get('home_abbr')}" + + # ESPN can send the key as null, and .get()'s default only covers a + # missing key, so a None here crashed the whole live update. + raw_period_text = game.get("period_text") + period_text = raw_period_text.lower() if isinstance(raw_period_text, str) else "" + if "final" in period_text: + self.logger.debug( + f"_is_game_really_over({game_str}): " + f"returning True - 'final' in period_text='{period_text}'" + ) + return True + + # Same for a null or non-numeric period: treat it as period 0. + try: + period = int(game.get("period") or 0) + except (TypeError, ValueError, OverflowError): + period = 0 + # Only a clock string is read: "0:00" and ":00" are zero; ":40" is not. + clock = game.get("clock") + clock_at_zero = isinstance(clock, str) and clock.replace(":", "").strip() in ("000", "00") + + if self.FINAL_PERIOD is not None and period >= self.FINAL_PERIOD and clock_at_zero: + try: + tied = int(game["away_score"]) == int(game["home_score"]) + except (KeyError, TypeError, ValueError, OverflowError): + tied = False # a missing or unreadable score leaves it to the clock + if not tied: + self.logger.debug( + f"_is_game_really_over({game_str}): " + f"returning True - clock at 0:00 (clock='{clock}', period={period})" + ) + return True + self.logger.debug( + f"_is_game_really_over({game_str}): " + f"returning False - tied at 0:00 (period={period}), overtime next" + ) + return False + + self.logger.debug( + f"_is_game_really_over({game_str}): returning False" + ) + return False + + +__all__ = ["SportsGameOverMixin"] diff --git a/src/common/sports_shared.py b/src/common/sports_shared.py index 954a9b1f..f1efbbd7 100644 --- a/src/common/sports_shared.py +++ b/src/common/sports_shared.py @@ -57,7 +57,9 @@ Methods that stay per-plugin, because they are not identical across the eight ``_get_layout_offset``, ``_by_importance``, ``_other_games_window``, ``_upcoming_date_and_time_text``, ``_extract_game_details_common``, ``_load_division_team_ids``, ``_get_timezone``, ``_is_favorite_game``, -``_is_game_really_over``, ``_is_ranked_game``, ``_passes_other_filters``. +``_is_ranked_game``, ``_passes_other_filters``. (``_is_game_really_over``, +which ``_detect_stale_games`` below calls, was here too until the plugins +reconciled it; it is now ``src.common.sports_game_over``.) Of the fourteen shared class constants, thirteen are identical everywhere and live here. Only ``_SCORE_PROBE_TEXT`` varies -- afl and basketball reach three digits diff --git a/test/test_sports_game_over.py b/test/test_sports_game_over.py new file mode 100644 index 00000000..4e83e75f --- /dev/null +++ b/test/test_sports_game_over.py @@ -0,0 +1,275 @@ +"""src.common.sports_game_over: behaviour, host contract and base order. + +The matrix is ledmatrix-plugins' ``scripts/test_game_over_check.py`` (the +table the family 5 reconcile was checked against) folded to the three +``FINAL_PERIOD`` values the nine scoreboards declare: None (afl, baseball, +nrl, soccer, ufc), 4 (basketball, football, lacrosse) and 3 (hockey). +Baseball's postponed/suspended override stays in its plugin and is not here. +""" + +import ast +import logging +import time +from pathlib import Path + +import pytest + +from src.common import sports_game_over +from src.common.sports_game_over import SportsGameOverMixin +from src.common.sports_shared import SportsLiveSharedMixin + +LOG = logging.getLogger("test_sports_game_over") + + +def host(final_period): + """A live manager stand-in declaring ``FINAL_PERIOD`` as a plugin does.""" + cls = type("Live", (SportsGameOverMixin,), {"FINAL_PERIOD": final_period}) + h = cls() + h.logger = LOG + return h + + +MISSING = object() # the key is absent from the game dict + + +def game(period_text="", period=MISSING, clock=MISSING, away="1", home="2"): + g = {"away_abbr": "AWY", "home_abbr": "HOM", "away_score": away, + "home_score": home, "period_text": period_text} + if period is not MISSING: + g["period"] = period + if clock is not MISSING: + g["clock"] = clock + return g + + +# --------------------------------------------------------------------------- +# The matrix: clock x period, for each FINAL_PERIOD. Scores 1-2. +# --------------------------------------------------------------------------- + +FINAL_PERIODS = (None, 4, 3) +PERIODS = (MISSING, 1, 2, 3, 4, 5, 6) +CLOCKS = {"12:00": "12:00", "0:00": "0:00", ":00": ":00", "0.0": "0.0", + "-": "-", "''": "", "None": None, "missing": MISSING} + +#: The period text each ESPN status carries. Only "Final" contains "final"; +#: the method reads no status, so every other text answers the same row. +LIVE_TEXTS = { + "in progress": lambda p: "" if p is MISSING else f"P{p}", + "end of period": lambda p: "" if p is MISSING else f"End P{p}", + "halftime": lambda p: "Halftime", + "end of round": lambda p: "" if p is MISSING else f"End R{p}", + "postponed": lambda p: "Postponed", +} + +#: clock -> one cell per period (missing, 1..6) for FINAL_PERIOD None, 4, 3. +EXPECTED_LIVE = { + "12:00": "....... ....... .......", + "0:00": "....... ....YYY ...YYYY", + ":00": "....... ....YYY ...YYYY", + "0.0": "....... ....... .......", + "-": "....... ....... .......", + "''": "....... ....... .......", + "None": "....... ....... .......", + "missing": "....... ....... .......", +} + + +def row(text_for, clock): + return " ".join( + "".join("Y" if host(fp)._is_game_really_over(game(text_for(p), p, clock)) else "." + for p in PERIODS) + for fp in FINAL_PERIODS) + + +@pytest.mark.parametrize("status", sorted(LIVE_TEXTS)) +@pytest.mark.parametrize("clock_label", sorted(CLOCKS)) +def test_a_live_period_text(status, clock_label): + assert row(LIVE_TEXTS[status], CLOCKS[clock_label]) == EXPECTED_LIVE[clock_label] + + +@pytest.mark.parametrize("clock_label", sorted(CLOCKS)) +def test_a_final_period_text_is_always_over(clock_label): + assert row(lambda p: "Final", CLOCKS[clock_label]) == "YYYYYYY YYYYYYY YYYYYYY" + + +#: label -> (game, one cell per FINAL_PERIOD None, 4, 3) +EDGES = { + "period_text None, P4 0:00": (game(None, 4, "0:00"), ".YY"), + "period None, 0:00": (game("", None, "0:00"), "..."), + "period 'OT', 0:00": (game("OT", "OT", "0:00"), "..."), + "period '4' (str), 0:00": (game("P4", "4", "0:00"), ".YY"), + "clock int 0, P4": (game("P4", 4, 0), "..."), + "clock float 0.0, P4": (game("P4", 4, 0.0), "..."), + "clock ' 0:00 ', P4": (game("P4", 4, " 0:00 "), ".YY"), + "clock '00:00', P4": (game("P4", 4, "00:00"), "..."), + "period_text 'Final/OT', P5 0:00": (game("Final/OT", 5, "0:00"), "YYY"), + "period_text 'FINAL', P1 12:00": (game("FINAL", 1, "12:00"), "YYY"), +} + + +@pytest.mark.parametrize("label", sorted(EDGES)) +def test_edge_shapes(label): + g, want = EDGES[label] + got = "".join("Y" if host(fp)._is_game_really_over(dict(g)) else "." for fp in FINAL_PERIODS) + assert got == want + + +# --------------------------------------------------------------------------- +# The tie guard: level at 0:00 is overtime, not the end. +# --------------------------------------------------------------------------- + +class TestTieGuard: + @pytest.mark.parametrize("fp,period", [(4, 4), (4, 5), (3, 3), (3, 4), (3, 5)]) + def test_level_at_zero_is_not_over(self, fp, period): + assert host(fp)._is_game_really_over(game("", period, "0:00", "2", "2")) is False + + def test_level_scores_compare_as_numbers(self): + assert host(4)._is_game_really_over(game("", 4, "0:00", 2, "2")) is False + assert host(4)._is_game_really_over(game("", 4, "0:00", " 2 ", "2")) is False + + def test_a_game_that_ends_level_ends_on_final(self): + assert host(4)._is_game_really_over(game("Final/OT", 5, "0:00", "2", "2")) is True + + def test_level_before_the_final_period_was_never_over(self): + assert host(4)._is_game_really_over(game("", 3, "0:00", "2", "2")) is False + + @pytest.mark.parametrize("away,home", [ + (MISSING, MISSING), (None, None), ("", ""), ("2", None), + ("2.0", "2.0"), ({"value": 2}, {"value": 2}), ("inf", "inf"), + ]) + def test_an_unreadable_score_leaves_it_to_the_clock(self, away, home): + g = game("", 4, "0:00") + for key, value in (("away_score", away), ("home_score", home)): + if value is MISSING: + del g[key] + else: + g[key] = value + assert host(4)._is_game_really_over(g) is True + + def test_float_infinity_does_not_raise(self): + assert host(4)._is_game_really_over( + game("", 4, "0:00", float("inf"), float("inf"))) is True + + +# --------------------------------------------------------------------------- +# ufc: ESPN MMA payloads, as ufc's _extract_game_details stores them +# (ledmatrix-plugins plugins/ufc-scoreboard/test/fixtures/espn_mma_round_states.json). +# --------------------------------------------------------------------------- + +UFC_RECORDED = { + "in_round_3_of_3": ("R3", 3, "1:21"), + "break_after_round_1": ("R1", 1, "-"), + "end_of_round_after_stoppage": ("R2", 2, "0:51"), + "walkouts_five_rounder": ("", 0, "-"), + "final_five_round_decision": ("R5", 5, "5:00"), + "final_five_round_stoppage": ("R5", 5, "1:38"), + "final_three_round_decision": ("R3", 3, "5:00"), + "final_three_round_stoppage": ("R2", 2, "4:07"), + "break_after_round_4_of_5": ("R4", 4, "-"), + "end_of_round_5_awaiting_decision": ("R5", 5, "-"), +} + + +class TestUfc: + @pytest.mark.parametrize("name", sorted(UFC_RECORDED)) + def test_no_recorded_state_is_over_here(self, name): + """A finished bout leaves the live list on is_final, before this is asked.""" + text, period, clock = UFC_RECORDED[name] + assert host(None)._is_game_really_over(game(text, period, clock, "0", "0")) is False + + @pytest.mark.parametrize("fp", FINAL_PERIODS) + def test_a_round_break_dash_is_never_a_zero_clock(self, fp): + assert host(fp)._is_game_really_over(game("R4", 4, "-", "1", "2")) is False + + @pytest.mark.parametrize("clock", ["0:00", None, MISSING]) + def test_the_horn_does_not_end_a_bout(self, clock): + assert host(None)._is_game_really_over(game("R5", 5, clock, "1", "2")) is False + + +# --------------------------------------------------------------------------- +# Wiring: the default, overrides, and the live mixin's caller. +# --------------------------------------------------------------------------- + +def test_the_default_is_no_clock_rule(): + assert SportsGameOverMixin.FINAL_PERIOD is None + + +def test_an_override_defers_through_super(): + """baseball's BaseballLive: its own check first, then the shared one.""" + + class Baseballish(SportsGameOverMixin): + logger = LOG + + def _is_game_really_over(self, game): + if game.get("status") == "status_postponed": + return True + return super()._is_game_really_over(game) + + b = Baseballish() + assert b._is_game_really_over(dict(game("", 6, "0:00"), status="status_postponed")) is True + assert b._is_game_really_over(game("", 6, "0:00")) is False + assert b._is_game_really_over(game("Final", 9, None)) is True + + +class _Live(SportsGameOverMixin, SportsLiveSharedMixin): + """A SportsLive stand-in in the documented base order.""" + + FINAL_PERIOD = 4 + + def __init__(self): + self.logger = LOG + self.stale_game_timeout = 600 + self.game_update_timestamps = {} + + +class TestBaseOrder: + def test_the_documented_order_resolves_this_method(self): + assert _Live._is_game_really_over is SportsGameOverMixin._is_game_really_over + mro = _Live.__mro__ + assert mro.index(SportsGameOverMixin) < mro.index(SportsLiveSharedMixin) + + def test_neither_shared_mixin_defines_it(self): + """So the base order cannot change which body runs.""" + from src.common.sports_shared import SportsCoreSharedMixin + for mixin in (SportsLiveSharedMixin, SportsCoreSharedMixin): + assert "_is_game_really_over" not in vars(mixin) + + def test_detect_stale_games_drops_an_over_game_through_it(self): + live = _Live() + live.game_update_timestamps = {"over": {"last_seen": time.time()}, + "on": {"last_seen": time.time()}} + games = [dict(game("", 4, "0:00"), id="over"), + dict(game("", 4, "0:00", "2", "2"), id="on")] + live._detect_stale_games(games) + assert [g["id"] for g in games] == ["on"] + assert "over" not in live.game_update_timestamps + + def test_the_class_value_wins_over_the_default(self): + assert _Live().FINAL_PERIOD == 4 + assert _Live()._is_game_really_over(game("", 4, "0:00")) is True + + +# --------------------------------------------------------------------------- +# Host contract +# --------------------------------------------------------------------------- + +def _self_reads(): + tree = ast.parse(Path(sports_game_over.__file__).read_text(encoding="utf-8")) + cls = next(n for n in tree.body + if isinstance(n, ast.ClassDef) and n.name == "SportsGameOverMixin") + return {node.attr for node in ast.walk(cls) + if isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load) + and isinstance(node.value, ast.Name) and node.value.id == "self"} + + +class TestHostContract: + def test_every_host_read_is_documented(self): + undocumented = sorted(n for n in _self_reads() + if f"``{n}``" not in sports_game_over.__doc__) + assert undocumented == [], f"read but not in the host contract: {undocumented}" + + def test_the_mixin_creates_no_state(self): + assert "__init__" not in vars(SportsGameOverMixin) + assert not hasattr(SportsGameOverMixin, "logger") + assert sorted(n for n in vars(SportsGameOverMixin) if not n.startswith("__")) == [ + "FINAL_PERIOD", "_is_game_really_over"] diff --git a/test/test_sports_game_over_parity.py b/test/test_sports_game_over_parity.py new file mode 100644 index 00000000..8f09aaee --- /dev/null +++ b/test/test_sports_game_over_parity.py @@ -0,0 +1,123 @@ +"""sports_game_over still matches every plugin copy, and each plugin's FINAL_PERIOD. + +``SportsGameOverMixin._is_game_really_over`` was copied from the scoreboards' +``SportsLive._is_game_really_over`` once family 5 had made the nine copies one +body. The plugins delete their copies once they floor on the release that +ships this module. Until each has, a copy that changes on its own is a fix one +side has and the other lacks. + +Point LEDMATRIX_PLUGINS at a ledmatrix-plugins checkout and the method is +compared with every plugin copy using ``scripts/sports_drift_report.py``'s own +normalisation (the AST with docstrings and annotations dropped), plus the +decorators. A copy that is gone counts as adopted when the plugin's +``sports.py`` names the module. Each plugin's ``SportsLive.FINAL_PERIOD`` is +compared with the value the owner decided for its sport, which stays in the +plugin after adoption. Without the variable this skips: core CI has no plugins +checkout. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +from src.common import sports_game_over + +REPO = Path(__file__).resolve().parents[1] + +#: The owner's decision (docs/SPORTS_UNIFICATION.md, family 5): the period +#: from which a 0:00 clock ends a game, None where the clock never does. +FINAL_PERIOD = { + "afl": None, "baseball": None, "basketball": 4, "football": 4, + "hockey": 3, "lacrosse": 4, "nrl": None, "soccer": None, "ufc": None, +} +NAME = "_is_game_really_over" + + +def _drift_report(): + """scripts/sports_drift_report.py, loaded by path (scripts/ is no package).""" + spec = importlib.util.spec_from_file_location( + "sports_drift_report", REPO / "scripts" / "sports_drift_report.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +DRIFT = _drift_report() + + +def _plugins_root(): + root = DRIFT.resolve_plugins_dir(os.environ.get("LEDMATRIX_PLUGINS")) + if root is None: + pytest.skip("set LEDMATRIX_PLUGINS to a ledmatrix-plugins checkout to " + "compare this module against the plugin copies") + return root + + +def _class(tree, name): + return next(n for n in tree.body if isinstance(n, ast.ClassDef) and n.name == name) + + +def _method(cls): + return next((n for n in cls.body + if isinstance(n, ast.FunctionDef) and n.name == NAME), None) + + +def _fingerprint(node): + return (DRIFT._digest(node, DRIFT._Canonical()), + tuple(ast.unparse(d) for d in node.decorator_list)) + + +def _final_period(cls): + for node in cls.body: + if (isinstance(node, (ast.Assign, ast.AnnAssign)) and node.value is not None): + target = node.targets[0] if isinstance(node, ast.Assign) else node.target + if isinstance(target, ast.Name) and target.id == "FINAL_PERIOD": + return ast.literal_eval(node.value) + raise AssertionError("SportsLive declares no FINAL_PERIOD") + + +def _ours(): + tree = ast.parse(Path(sports_game_over.__file__).read_text(encoding="utf-8")) + return _class(tree, "SportsGameOverMixin") + + +def test_the_mixin_holds_one_method_and_the_default(): + names = sorted(n.name if isinstance(n, ast.FunctionDef) else n.target.id + for n in _ours().body if isinstance(n, (ast.FunctionDef, ast.AnnAssign)) + and (isinstance(n, ast.FunctionDef) or n.value is not None)) + assert names == ["FINAL_PERIOD", NAME] + assert _final_period(_ours()) is None + + +@pytest.mark.parametrize("sport", sorted(FINAL_PERIOD)) +def test_every_remaining_plugin_copy_matches(sport): + root = _plugins_root() + source = (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8") + live = _class(ast.parse(source), "SportsLive") + copy = _method(live) + if copy is None: + assert sports_game_over.__name__ in source, ( + f"{sport}: no {NAME} on SportsLive and no {sports_game_over.__name__} import") + else: + assert _fingerprint(copy) == _fingerprint(_method(_ours())), ( + f"{NAME} in {sport} differs from sports_game_over. " + f"Port the change to both, or stop treating it as shared.") + + +@pytest.mark.parametrize("sport", sorted(FINAL_PERIOD)) +def test_every_plugin_declares_its_final_period(sport): + root = _plugins_root() + source = (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8") + assert _final_period(_class(ast.parse(source), "SportsLive")) == FINAL_PERIOD[sport] + + +def test_the_drift_report_still_calls_it_identical(): + root = _plugins_root() + families = DRIFT.build(root, ("sports.py",)) + rows = {(r["file"], r["family"]): r + for r in (DRIFT.summarise(k, v) for k, v in families.items())} + row = rows.get(("sports.py", NAME)) + assert row is None or row["worst_class_variants"] == 1 From 6fb2dc35951de94740c64c08a11cfeb2f2a08600 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 09:52:22 -0400 Subject: [PATCH 22/37] fix(sports): honour every pending kickoff in the idle back-off, not just the first (#772) _note_scheduled_start_candidate kept one kickoff. While it was inside its 15-minute grace every later kickoff was refused, and by the time the grace ended the later one had passed and was refused again as already past. So of two favourites kicking off within 15 minutes of each other, the second lost its own grace: if the first game was not live by then (a rain delay, a postponement, ESPN slow to flip it) and ESPN had not flipped the second either, the back-off went straight back to its ceiling and the second game was noticed up to that late. Later kickoffs now wait in a short queue (_later_scheduled_starts, the earliest 8). When the current kickoff's grace ends, the earliest queued one still inside its own grace takes over -- including one that has already passed. A kickoff still holds the live cadence for at most its own grace, so a postponed game costs the same quarter of an hour as before, and _next_scheduled_start_ts keeps its meaning for anything that reads or sets it. The promotion is a module function, so the mixin's method set is unchanged. Table tests replay the idle loop on a fake clock over kickoff schedules; mutation-checked (the old code fails 10 of the new tests). Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 14 ++++ src/common/sports_shared.py | 58 ++++++++++++++-- test/test_sports_shared.py | 130 ++++++++++++++++++++++++++++++++++++ 3 files changed, 196 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a260dad..2534fd02 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -194,6 +194,20 @@ Internal; no behaviour change. Stage 3 of `docs/RUN_LOOP_REDESIGN.md`. (`refresh_registry_in_background()`, backing off for a minute after an offline failure), so a later load has them. The store, install and update paths still fetch as before. +- A sports live manager's idle back-off now honours every pending kickoff, + not just the first. `_note_scheduled_start_candidate()` kept one kickoff + and, while it was inside its 15-minute grace, refused every later one; by + the time the grace ended the later one had passed and was refused again. + So of two favourites kicking off within 15 minutes of each other, the + second lost its own grace: if the first game was not live by then (a rain + delay, a postponement, ESPN slow to flip it) and ESPN had not flipped the + second either, the back-off went back to its ceiling and the second game + was noticed up to the ceiling (15 minutes by default) late. Later + kickoffs now wait in a short queue (`_later_scheduled_starts`, the + earliest 8) and each takes over with a grace of its own when the one + before it expires. A kickoff still + holds the live cadence for at most its own grace, so a postponed game + costs the same quarter of an hour as before. ### ESPN date-range fetches: fewer requests, fewer at once diff --git a/src/common/sports_shared.py b/src/common/sports_shared.py index f1efbbd7..e6d6403d 100644 --- a/src/common/sports_shared.py +++ b/src/common/sports_shared.py @@ -122,6 +122,32 @@ _DEFAULT_LIVE_IDLE_MAX_SECONDS = 900 _KICKOFF_GRACE_SECONDS = 900 #: Fallback cadence around a kickoff when the manager has no update_interval. _KICKOFF_POLL_FLOOR = 30 +#: How many kickoffs after the current one a live manager remembers. Only the +#: earliest few can matter before the next look refreshes the list, so this +#: bounds the memory without dropping a kickoff the board would wait for. +_KICKOFF_QUEUE_MAX = 8 + + +def _current_scheduled_start(host: Any, now: float) -> Optional[float]: + """The kickoff a live manager is honouring now, promoting the next queued one. + + ``_next_scheduled_start_ts`` is the kickoff being honoured: the earliest + one ahead of us, or one that has just passed and is inside its grace. + Kickoffs behind it wait in ``_later_scheduled_starts``. When the current + one's grace runs out, the earliest queued kickoff that is not itself past + its grace takes over -- including one that has already passed, so a + second kickoff inside the first one's grace still gets a grace of its own. + """ + current: Optional[float] = getattr(host, "_next_scheduled_start_ts", None) + if current and current > now - _KICKOFF_GRACE_SECONDS: + return current + queued: Optional[List[float]] = getattr(host, "_later_scheduled_starts", None) + if queued: + alive = sorted(s for s in queued if s > now - _KICKOFF_GRACE_SECONDS) + current = alive.pop(0) if alive else None + host._later_scheduled_starts = alive + host._next_scheduled_start_ts = current + return current if current and current > now - _KICKOFF_GRACE_SECONDS else None def _resolve_font_path(path: str) -> str: @@ -1296,11 +1322,11 @@ class SportsLiveSharedMixin: otherwise look like another empty check and escalate the back-off again, right when the game is actually starting. """ - start = getattr(self, "_next_scheduled_start_ts", None) + now = time.time() + start = _current_scheduled_start(self, now) if not start: return interval live = getattr(self, "update_interval", None) or _KICKOFF_POLL_FLOOR - now = time.time() if now < start: return max(live, min(interval, int(start - now))) if now - start <= _KICKOFF_GRACE_SECONDS: @@ -1315,6 +1341,16 @@ class SportsLiveSharedMixin: already has. Self-correcting: a stored start that has passed is replaced by the next one offered, so a postponed game cannot pin the cadence to a kickoff that never happens. + + Every pending kickoff is honoured, not just the first. A kickoff that + arrives while an earlier one is inside its grace is queued in + ``_later_scheduled_starts`` (the earliest _KICKOFF_QUEUE_MAX of them) + and takes over when that grace ends, with a grace of its own. Keeping + only the one kickoff dropped the second of two favourites starting + within the grace of each other: it was refused while the first held + the slot, and refused again once it had passed, so if ESPN had not + flipped it live by the end of the first grace the back-off went + straight back to its ceiling and the game was noticed up to that late. """ if not isinstance(details, dict): return @@ -1331,7 +1367,7 @@ class SportsLiveSharedMixin: now = time.time() if candidate <= now: return - current = getattr(self, "_next_scheduled_start_ts", None) + current = _current_scheduled_start(self, now) # A kickoff that has only just passed is *kept*, not replaced by the # next one on the card. Replacing it immediately is what made the grace # window in _clamp_to_scheduled_start dead code: the moment 13:00 came @@ -1341,10 +1377,20 @@ class SportsLiveSharedMixin: # polled at 13:00:45, found nothing live because ESPN had not flipped # the status yet, and then went quiet for the next quarter of an hour, # which is the behaviour this whole clamp exists to prevent. - if (current is None - or current <= now - _KICKOFF_GRACE_SECONDS - or candidate < current): + # + # Nor is it forgotten: whichever kickoff loses is queued behind the + # one honoured now, so it gets its own grace when that one's ends. + if current is None: self._next_scheduled_start_ts = candidate + return + if candidate == current: + return + if candidate < current: + self._next_scheduled_start_ts, candidate = candidate, current + queued = getattr(self, "_later_scheduled_starts", None) or [] + if candidate not in queued: + self._later_scheduled_starts = sorted( + [*queued, candidate])[:_KICKOFF_QUEUE_MAX] #: How long a game that finished live is still reported by #: finished_games_snapshot(): long enough for the recent-games list, which diff --git a/test/test_sports_shared.py b/test/test_sports_shared.py index 1cdd6fc2..b9ee0a28 100644 --- a/test/test_sports_shared.py +++ b/test/test_sports_shared.py @@ -465,6 +465,136 @@ class TestLiveMixin: # The safety property that makes it correct: 30s beats 600s. assert h._idle_live_interval() == h.update_interval + # ---- every pending kickoff is honoured, not just the first ------------ + # + # One stored kickoff held the slot through its grace and refused every + # later one; a later one that had passed by the time the grace ended was + # refused again as "already past". So of two favourites kicking off ten + # minutes apart, the second lost its grace: if ESPN had not flipped it live + # by the end of the FIRST game's grace, the back-off returned to its + # ceiling and the game was noticed up to that late. That bites whenever the + # first game is not live by then -- a rain delay, a postponement, ESPN slow + # to flip it -- since a live first game keeps the live cadence anyway. + + @staticmethod + def _replay(monkeypatch, kickoffs, flips, until, poll=30, ceiling=900): + """Drive a live manager's idle loop over a schedule on a fake clock. + + ``kickoffs`` are start offsets in seconds from t=0 (the first look), + ``flips`` how long after its start ESPN reports each game live (None: + postponed, never live). Every + look offers each not-yet-live game, as the live loop does, then sleeps + for whatever the back-off returns. Returns, per game, how long after it + went live it was first seen live -- None if never. + """ + clock = [1_800_000_000.0] + monkeypatch.setattr(sports_shared.time, "time", lambda: clock[0]) + h = _LiveHost(no_data_interval=300) + h.live_idle_max_interval = ceiling + h.update_interval = poll + h._empty_live_streak = 30 # idle all morning: at the ceiling + t0 = clock[0] + seen = [None] * len(kickoffs) + while clock[0] - t0 < until: + now = clock[0] - t0 + live = [f is not None and k + f <= now for k, f in zip(kickoffs, flips)] + for i, is_live in enumerate(live): + if is_live and seen[i] is None: + seen[i] = now - (kickoffs[i] + flips[i]) + start = datetime.fromtimestamp(t0 + kickoffs[i], tz=timezone.utc) + h._note_scheduled_start_candidate( + {"is_live": is_live, "is_halftime": False, + "start_time_utc": start}) + # A real board keeps polling at the live cadence while anything is + # live; a game here stays live for an hour after it flips. + on = any(f is not None and k + f <= now < k + f + 3600 + for k, f in zip(kickoffs, flips)) + h._note_live_fetch(on) + clock[0] += poll if on else h._idle_live_interval() + assert len(getattr(h, "_later_scheduled_starts", None) or ()) <= sports_shared._KICKOFF_QUEUE_MAX + return seen + + @pytest.mark.parametrize("kickoffs,flips", [ + # (start offsets, ESPN's flip delay per game), both in seconds. + pytest.param([1800], [120], id="one kickoff, flipped 2 min late"), + pytest.param([1800, 2400], [0, 840], + id="first live on time, the second flipped 14 min late"), + pytest.param([1800, 2400], [None, 840], + id="first postponed, the second 10 min later flipped 14 min late"), + pytest.param([1800, 2400], [1200, 840], + id="first in a 20 min delay, the second flipped 14 min late"), + pytest.param([1800, 2100, 2520], [None, None, 600], + id="three inside one grace, the last flipped 10 min late"), + pytest.param([1800, 1800, 2400], [None, None, 700], + id="two at the same time, then one 10 min later"), + pytest.param([1800, 2700], [None, 840], + id="second kickoff 15 min later, flipped 14 min late"), + pytest.param([1800 + 60 * i for i in range(20)], + [None] * 19 + [840], + id="twenty kickoffs a minute apart overflow the queue"), + ]) + def test_every_pending_kickoff_gets_its_grace(self, monkeypatch, kickoffs, flips): + seen = self._replay(monkeypatch, kickoffs, flips, + until=max(kickoffs) + 3 * 3600) + late = [s for s, f in zip(seen, flips) + if f is not None and (s is None or s > 30)] + assert not late, "games noticed late (s after going live): %r" % (seen,) + + def test_a_kickoff_that_never_flips_costs_one_grace_then_backs_off(self, monkeypatch): + # A postponed game keeps the live cadence for its grace and no longer: + # remembering more kickoffs must not pin the poll to dead ones. + clock = [1_800_000_000.0] + monkeypatch.setattr(sports_shared.time, "time", lambda: clock[0]) + h = self._idle_host() + t0 = clock[0] + for offset in (600, 900): + h._note_scheduled_start_candidate( + {"start_time_utc": datetime.fromtimestamp(t0 + offset, tz=timezone.utc)}) + clock[0] = t0 + 900 + sports_shared._KICKOFF_GRACE_SECONDS - 1 + assert h._idle_live_interval() == h.update_interval + clock[0] = t0 + 900 + sports_shared._KICKOFF_GRACE_SECONDS + 1 + assert h._idle_live_interval() == 900 + assert not getattr(h, "_later_scheduled_starts", None) + + def test_a_queued_kickoff_past_its_own_grace_is_skipped(self, monkeypatch): + # After a long sleep (or a run of looks that never woke the manager) + # several queued kickoffs may have gone stale at once. The one still + # inside its grace must win, not the first stale one in the queue. + clock = [1_800_000_000.0] + monkeypatch.setattr(sports_shared.time, "time", lambda: clock[0]) + h = self._idle_host() + t0 = clock[0] + for offset in (600, 700, 1500): + h._note_scheduled_start_candidate( + {"start_time_utc": datetime.fromtimestamp(t0 + offset, tz=timezone.utc)}) + clock[0] = t0 + 1500 + 150 # 600 and 700 are past their grace + assert h._idle_live_interval() == h.update_interval + assert h._next_scheduled_start_ts == t0 + 1500 + + def test_the_queue_keeps_the_earliest_kickoffs(self, monkeypatch): + clock = [1_800_000_000.0] + monkeypatch.setattr(sports_shared.time, "time", lambda: clock[0]) + h = self._idle_host() + t0 = clock[0] + cap = sports_shared._KICKOFF_QUEUE_MAX + for offset in reversed(range(1, cap + 6)): # latest first + h._note_scheduled_start_candidate( + {"start_time_utc": datetime.fromtimestamp(t0 + 600 * offset, tz=timezone.utc)}) + assert h._next_scheduled_start_ts == t0 + 600 + assert h._later_scheduled_starts == [t0 + 600 * i for i in range(2, cap + 2)] + + def test_a_kickoff_offered_twice_is_kept_once(self, monkeypatch): + clock = [1_800_000_000.0] + monkeypatch.setattr(sports_shared.time, "time", lambda: clock[0]) + h = self._idle_host() + t0 = clock[0] + for _ in range(3): + for offset in (600, 1200): + h._note_scheduled_start_candidate( + {"start_time_utc": datetime.fromtimestamp(t0 + offset, tz=timezone.utc)}) + assert h._next_scheduled_start_ts == t0 + 600 + assert h._later_scheduled_starts == [t0 + 1200] + def test_finding_a_live_game_resets_the_streak(self): h = _LiveHost() h._note_live_fetch(False) From c20c0beac283fd3ebc7bc98edba348c8971a752d Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 09:53:43 -0400 Subject: [PATCH 23/37] feat(web): the Display tab is an ES-module page, with a page-visibility service (stage 4) (#771) * feat(web): the Display tab is an ES-module page, with a page-visibility service (stage 4) - core/visibility.js: each page gets ctx.visibility (whileVisible, every, isVisible). Work registered there runs only while the page's tab is active and the browser tab visible, and ends when the page is swapped out. It reads the active tab from window.LEDVisibility, so it agrees with the classic partials. The registry gained a mountContext option for per-mount services. - pages/display.js replaces display.html's two inline scripts. The 5 s sync status poll runs through ctx.visibility.every; the status and scroll-speed hint requests go through ctx.api with ctx.signal, as does the Vegas order widget's plugin-list request. The Advanced toggle is a delegated data-action; window.updateSyncUI is a deprecated alias. - New DOM suites test_visibility_service.js and test_display_page.js; test_display_partial_ids.js imports the module. Co-Authored-By: Claude Opus 5.5 * refactor(web): no computed keys in the Display page's readout and destroy Codacy's object-injection rule flagged v[id] and ctx.state[name]. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 28 ++ docs/WEB_FRONTEND_ARCHITECTURE.md | 42 ++- test/js/README.md | 4 +- test/js/dom/test_display_page.js | 325 ++++++++++++++++++ test/js/dom/test_visibility_service.js | 211 ++++++++++++ test/js/run_all.js | 3 +- test/js/unit/test_display_partial_ids.js | 144 ++++---- test/js/unit/test_html_escaping.js | 9 +- test/js/unit/test_page_registry.js | 40 +++ test/web_interface/test_es_modules.py | 5 +- web_interface/static/v3/js/core/boot.js | 6 + web_interface/static/v3/js/core/registry.js | 15 +- web_interface/static/v3/js/core/visibility.js | 144 ++++++++ web_interface/static/v3/js/pages/display.js | 324 +++++++++++++++++ .../templates/v3/partials/display.html | 301 +--------------- 15 files changed, 1223 insertions(+), 378 deletions(-) create mode 100644 test/js/dom/test_display_page.js create mode 100644 test/js/dom/test_visibility_service.js create mode 100644 web_interface/static/v3/js/core/visibility.js create mode 100644 web_interface/static/v3/js/pages/display.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 2534fd02..f0ba87e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,34 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +### Web UI: the Display tab is an ES-module page, with a page-visibility service (stage 4) + +- New `static/v3/js/core/visibility.js`: each page module gets + `ctx.visibility` with `whileVisible(start, stop)`, `every(ms, fn)` and + `isVisible()`. Work registered there runs only while the page's tab is the + active tab and the browser tab is visible, and ends when the page is + swapped out, with no teardown code in the page. It reads the active tab + from `window.LEDVisibility`, so it agrees with the classic partials that + still use that directly. The page registry gained a `mountContext` option + for services bound to one mounted page. +- The Display tab's inline scripts are now `static/v3/js/pages/display.js`. + The partial has no inline script, `onclick` or `onchange` any more. The + multi-display sync status poll (every 5 s) runs through + `ctx.visibility.every`; the status and scroll-speed hint requests go + through `core/api.js` with the page's abort signal, and so does the Vegas + order widget's plugin-list request. +- Behaviour differences: with sync on, opening the tab asks for the status + once instead of twice, and a Display tab loaded while not on screen waits + until it is. A login redirect during a poll no longer flashes "Sync status + unavailable". The `window.syncStatusInterval` timer id is gone (nothing + read it). A pending scroll-hint request or widget retry is dropped when + the partial is swapped out. +- `window.updateSyncUI` keeps working as a deprecated alias through + `window.LEDMatrix` (one console warning). +- New suites `test/js/dom/test_visibility_service.js` and + `dom/test_display_page.js`; `unit/test_display_partial_ids.js` imports the + module instead of slicing the template. + ### Plugins ask for the screen in-process: `request_on_demand()` / `end_on_demand()` The in-process way in that stage 5 of the control socket needed diff --git a/docs/WEB_FRONTEND_ARCHITECTURE.md b/docs/WEB_FRONTEND_ARCHITECTURE.md index b7ae12ba..962107da 100644 --- a/docs/WEB_FRONTEND_ARCHITECTURE.md +++ b/docs/WEB_FRONTEND_ARCHITECTURE.md @@ -42,7 +42,8 @@ static/v3/js/ registry.js page lifecycle: init/destroy on htmx swaps api.js fetch wrapper for /api/v3 (JSON envelope, login redirect) facade.js window.LEDMatrix and deprecated aliases - (later) escape.js, notify.js, dialog.js, streams.js, visibility.js, + visibility.js ctx.visibility: a page's timers run only while it is on screen + (later) escape.js, notify.js, dialog.js, streams.js, store.js (the one installed-plugin store), form/renderer.js pages/ one module per tab partial cache.js export init(root, ctx), destroy(root, ctx) @@ -81,6 +82,12 @@ The conventions the converted pages share: to the module's export of the same name and warns once. - **Timers are cleared in `destroy()`**, the one thing `ctx.signal` cannot undo by itself. +- **Polling goes through `ctx.visibility`.** A refresh that repeats + (`ctx.visibility.every(ms, fn)`) or work that should run only while the + page is on screen (`ctx.visibility.whileVisible(start, stop)`) is + registered there, never with a bare `setInterval`. It runs only while the + page's tab is the active tab and the browser tab is visible, and it ends + when the page is destroyed, with no code in `destroy()`. - **A page reports its own htmx saves.** A form whose result a page module shows (an `htmx:afterRequest` listener on the page root, in place of an `hx-on` attribute naming a global) carries `data-reports-result`. `app.js` @@ -111,6 +118,7 @@ Each mount gets a `ctx` object: | `ctx.state` | A per-mount object for the page's own state | | `ctx.api` | Shared service from `boot.js` | | `ctx.notify` | Shared service from `boot.js` | +| `ctx.visibility` | This page's handle on `core/visibility.js` (below), made per mount by `boot.js` through the registry's `mountContext` option | A page that passes `{ signal: ctx.signal }` to `addEventListener` and `fetch` needs no teardown code. Its listeners and in-flight requests go @@ -119,6 +127,30 @@ example: its delete buttons use one delegated listener, rows are built with `textContent` rather than markup strings, and a newer load supersedes an older one. +### Page visibility + +`core/visibility.js` gives each mounted page `ctx.visibility`: + +| Member | What it does | +|---|---| +| `whileVisible(start, stop)` | Runs `start()` when the page comes on screen (at once, if it mounts on screen) and `stop()` when it leaves. Returns a function that ends the registration, running `stop()` first if needed | +| `every(ms, fn)` | `fn()` at once, then every `ms` while on screen. The interval is cleared while hidden and restarted, with an immediate `fn()`, when the page is back. Returns the same kind of end function | +| `isVisible()` | True while the page is on screen | +| `tab` | The tab the page belongs to: its name, or `forPage(ctx, { tab })` | + +"On screen" means the page's tab is the active tab and the browser tab is +visible. Everything a page registered ends when its `ctx.signal` aborts, +after `destroy()`, so a swapped-out partial leaves no interval behind. + +The answer comes from `window.LEDVisibility` (`app-shell.js`), read at call +time, so the page modules and the classic partials that still call it +(Overview, Logs, Tools) agree on the active tab, and the SSE streams keep +pausing with them. Each registration takes its own `LEDVisibility` key, so +registrations never replace each other or a classic partial's. Without +`LEDVisibility` (a page outside `base.html`), the browser tab's visibility +alone decides. Moving the tracker itself into the module (the shell table +below) changes only `core/visibility.js`. + ### One facade `window.LEDMatrix` is the only global the module code adds: @@ -249,7 +281,7 @@ are the inline script in each partial today. | 5 | Backup & Restore | 232 lines, now 0 | **Done in stage 2.** Its 5 globals (`exportBackup`, `loadBackupList`, `validateRestoreFile`, `clearRestore`, `runRestore`) are deprecated aliases; the buttons are delegated `data-action`s. Uploads go through `ctx.api.request(..., { body: formData })` (`api.js` gained a raw `body` option) | | 6 | Schedule | 193 lines, now 0 | **Done in stage 3.** Its 2 `hx-on` response handlers (`handleScheduleResponse`, `handleDimScheduleResponse`) are one `htmx:afterRequest` listener on the page root, and deprecated aliases. The forms are marked `data-reports-result` so `app.js` does not repeat the server's message. The saved schedules reach the module as JSON in `data-schedule-config` / `data-dim-schedule-config` instead of being templated into the script | | 7 | General | 153 lines, now 0 | **Done in stage 3.** The Security section's three forms and two buttons are delegated `data-action`s (one submit and one click listener); `window.webLogin` is a deprecated alias of an object with its five methods. Login requests go through `ctx.api`, so the login redirect is quiet. The settings form keeps its `hx-on` call to the shared `showSaveResult`, as Rotation's does | -| 8 | Display | 231 | First page with `LEDVisibility` timers: those move to a `ctx.visibility` service that stops on destroy | +| 8 | Display | 292 lines (2 scripts), now 0 | **Done in stage 4.** The first page with a timer: the 5 s multi-display sync poll is `ctx.visibility.every(5000, ...)` (above), so it runs only while the tab is on screen and stops when the partial is swapped out. Its one global, `updateSyncUI` (the Role menu's `onchange`), is a deprecated alias; the Advanced section's `onclick` is a delegated `data-action="toggle-section"` that calls the shared `toggleSection`. The status poll and the scroll-speed hint go through `ctx.api` with `ctx.signal`, as does the Vegas order widget's plugin-list request. The settings form keeps its `hx-on` call to `showSaveResult` and its `onsubmit` call to `fixInvalidNumberInputs`, as Rotation's does | | 9 | Overview | 410 (4 scripts) | First-run surface: Getting Started, update banner, live preview. Five globals | | 10 | WiFi | 364 | `x-data="wifiSetup()"` is defined by its own script. Moves to `Alpine.data()` registered from the module. AP-mode first screen, so it needs the AP-mode test on a real device | | 11 | Fonts | 681 | Large, but self-contained (6 globals) | @@ -266,7 +298,7 @@ the order: | `showNotification` | 4 versions | `core/notify.js` | | The modal helper | `utils/dialog.js` | `core/dialog.js` | | SSE streams | `app-shell.js` | `core/streams.js` | -| `LEDVisibility` | `app-shell.js` | `core/visibility.js` | +| `LEDVisibility` | `app-shell.js` | `core/visibility.js` (the page-facing `ctx.visibility` is there since step 8; it reads the tracker from `app-shell.js`) | Each move leaves the old global as an alias. When the last inline script is gone, the script re-execution in `htmx-config.js` and the "HTMX never @@ -286,13 +318,15 @@ Unit suites need only node. They import the shipped modules directly: | Suite | Kind | What it covers | |---|---|---| -| `unit/test_page_registry.js` | Unit, minimal DOM shim | The lifecycle: one init per root, destroy on swap, a veto keeps the page, swaps elsewhere leave it alone, the sweep, lazy loading, a destroy while loading, error containment | +| `unit/test_page_registry.js` | Unit, minimal DOM shim | The lifecycle: one init per root, destroy on swap, a veto keeps the page, swaps elsewhere leave it alone, the sweep, lazy loading, a destroy while loading, error containment, `mountContext` fields per mount | +| `dom/test_visibility_service.js` | DOM: real `LEDVisibility` from `app-shell.js`, real registry, no server | `whileVisible` and `every` start and stop with the active tab and the browser tab's visibility; no interval runs while hidden or after a swap-out; one interval after five swaps; registrations never replace each other or a classic partial's; a destroyed page registers nothing; a throwing `start()` is contained; the no-`LEDVisibility` fallback | | `unit/test_core_modules.js` | Unit | `api.js` (envelope, errors, abort, login redirect, path check) and `facade.js` (facade, aliases) | | `dom/test_cache_page.js` | DOM: real partial, real API shape | No inline script; one request per swap and per Refresh after five swaps; a cancelled request draws nothing; hostile keys stay text; delete, empty, error, network and login states | | `dom/test_durations_page.js` | DOM: real partial, real widget, real API shape | One plugin-list request per swap; Move down moves one place after five swaps; the swap cancels a request in flight; a late-loading widget is waited for, and a page swapped away while waiting starts nothing; hostile names stay text | | `dom/test_operation_history_page.js` | DOM: real partial, real API shape | One history request per swap and per Refresh; the plugin filter filled once (from `PluginAPI`'s cache when loaded); paging, filters, debounced search, Clear (one DELETE), error/network/login states, cancel on swap; hostile ids, users and errors stay text | | `dom/test_raw_json_page.js` | DOM: real partial, real config | One POST per Save after five swaps, to the right file; Format and Validate act once; invalid JSON never sent and its message stays text; a save survives a swap and is still reported; the old globals' entry points | | `dom/test_schedule_page.js` | DOM: real partial, real widget | Both pickers drawn once per swap from the saved config; after five swaps each form's answer is one notification (message, fallback, refused, non-JSON, `null`), a request from outside the forms none; the brightness label; a late widget waited for, a page swapped away while waiting draws nothing; the old globals' entry points | +| `dom/test_display_page.js` | DOM: real partial, real widget, real `LEDVisibility`, real API shape | After five swaps one page, one sync interval, the Vegas order drawn once and each control acting once (brightness, resolution, the two show/hide toggles, the Advanced toggle, one debounced hint request); the sync poll only while on screen and never after a swap-out; sync states and hostile peer names as text, failure and login answers; a late widget waited for; `updateSyncUI`'s entry point | | `dom/test_general_page.js` | DOM: real partial, real widget, real API shape | The timezone picker drawn once per swap with the saved zone; the settings form left to htmx; after five swaps each Security action makes one request (create, copy, revoke and its cancel, password and its mismatch); hostile token names stay text; refused, network and login answers; a create made before a swap is still reported and draws nothing; `webLogin`'s entry points | | `dom/test_backup_restore_page.js` | DOM: real partial, real API shape | One request per Refresh, Delete, Export (busy button ignores a second click), Inspect and Restore after five swaps; the upload's fields and the six restore options; reads cancelled by a swap, writes not; hostile file and host names stay text; the old globals' entry points | | `test/web_interface/test_es_modules.py` | pytest | MIME type; `no-cache` without `?v` and immutable with it; `boot.js` loads last; every import resolves inside `core/` and `pages/`; the converted pages are exactly the registered ones, each with its module, `init`, and one root in the rendered partial; a converted partial has no ` @@ -830,7 +795,7 @@ With this off a live game takes over the whole display with the full-screen scor
- @@ -879,261 +844,3 @@ With this off a live game takes over the whole display with the full-screen scor
- - From a669d781f5a7d402f82c623a2ac42dd600eeab20 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 10:11:17 -0400 Subject: [PATCH 24/37] chore: prepare the 3.8.1 release (#756) * chore: prepare the 3.8.1 release Bumps src.__version__ to 3.8.1 and moves the Unreleased CHANGELOG entries under a 3.8.1 heading, with an empty Unreleased above it. The reason for the release is smooth scrolling at held-frame speeds. On 3.8.0 the default 50 px/s snapped to a stepped 48 px/s on a 120 Hz panel, and any scroll slower than one pixel per refresh showed a half-pixel step across the middle of the panel. Both are fixed on main (#710, #711) but were in no release, so every stable-channel device still had them. - #710's CHANGELOG entry had been filed under 3.8.0 although it merged after the v3.8.0 tag; it moves to 3.8.1. - #711 had no CHANGELOG entry; it gets one. scripts/check_release_version.py v3.8.1 passes. Co-Authored-By: Claude Opus 5.5 * docs: sports_game_over and draw_text_outlined ship in 3.8.1 Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 50 ++++++++++++++++++++++++++++---------- docs/SPORTS_UNIFICATION.md | 2 +- src/__init__.py | 2 +- src/common/README.md | 4 +-- 4 files changed, 41 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0ba87e2..abeee525 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,42 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +## 3.8.1 + +Smooth scrolling at the slower speeds, and the fixes and performance work +since 3.8.0. Highlights: the default 50 px/s and every other held-frame speed +now scroll cleanly (below), Raspberry Pi OS Bookworm is supported alongside +Trixie, updates refresh the systemd units, the display control socket gains +stages 2 and 3, the shared fetch service lands (stages 1 and 2), and a run of +web UI and Plugin Manager fixes. One new module is for plugins: +`src.common.sports_game_over` (sports family 5), which the scoreboards adopt +by flooring on 3.8.1; the other new modules are core-internal and set no +`ledmatrix_min_version` floor. + +### Scroll speed + +These two entries were the reason for this release: on 3.8.0 a slow scroll +either stepped or showed a half-pixel tear across the middle of the panel, +so only speeds of one pixel per refresh looked right. + +- The Vegas Scroll Speed slider now says what the panel will do with the speed + it is on, and offers the nearest smooth ones to click. Only speeds that advance + a whole number of pixels per refresh look smooth, and which those are depends + on the panel (`GET /api/v3/config/scroll-speed-advice`, built on + `scroll_config.speed_advice()`; it uses the refresh the display measured, not + the `limit_refresh_rate_hz` cap). The slider steps by 1 px/s instead of 5. +- The default 50 px/s no longer snaps to a stepped 48 px/s (2 px every 5 + refreshes, 24 fps) on a 120 Hz panel: `solve_crisp()` now prefers 60 or 40 px/s, + which move one pixel at a time. 100 Hz panels are unaffected. (#710) +- A held-frame scroll (one pixel every two or more refreshes, such as 50 or + 60 px/s on a 100-120 Hz panel) no longer shows a half-pixel step across the + middle of the panel. Scan-order compensation ran only at one frame per + refresh; a held frame is now presented as a sequence of swaps + (`scan_order.refresh_plan()`), so the half of the panel that scans later + steps one refresh after the rest. It is skipped when a blit takes more than + half a refresh, since the second blit has to land before the next vsync. + (#711) + ### Web UI: the Display tab is an ES-module page, with a page-visibility service (stage 4) - New `static/v3/js/core/visibility.js`: each page module gets @@ -719,7 +755,7 @@ policies are unchanged. class attribute, `None` by default (the clock never ends a game); the scoreboards declare 3 (hockey), 4 (basketball, football, lacrosse) or `None`. List the mixin before `SportsLiveSharedMixin`. A plugin may import - it once it floors on the release that ships it, and deletes its copy then. + it once it floors on 3.8.1, and deletes its copy then. (#770) ### Tooling @@ -1373,18 +1409,6 @@ guard the import, since the loader's version check is advisory). processes, or turns the socket off with `off`. A non-root dev run uses a private per-user path under the temp directory. -### Scroll speed - -- The Vegas Scroll Speed slider now says what the panel will do with the speed - it is on, and offers the nearest smooth ones to click. Only speeds that advance - a whole number of pixels per refresh look smooth, and which those are depends - on the panel (`GET /api/v3/config/scroll-speed-advice`, built on - `scroll_config.speed_advice()`; it uses the refresh the display measured, not - the `limit_refresh_rate_hz` cap). The slider steps by 1 px/s instead of 5. -- The default 50 px/s no longer snaps to a stepped 48 px/s (2 px every 5 - refreshes, 24 fps) on a 120 Hz panel: `solve_crisp()` now prefers 60 or 40 px/s, - which move one pixel at a time. 100 Hz panels are unaffected. - ### Update channels - Devices no longer pick up every merge to `main`. A new setting, diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index 3c7072f2..2585829c 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -91,7 +91,7 @@ more. Shared sports code lives in `src/common`: | `sports_live_scroll.py` | next release | `SportsLiveScrollMixin` — rebuild a live scroll strip mid-cycle, keeping the marquee's place | | `sports_display_rules.py` | next release | `SportsCardOptionsMixin`, `SportsGameRulesMixin` — scorebug date options, the no-favourites filter, non-favourite live dwell | | `sports_font_path.py` | next release | `resolve_font_path` — what the plugins' `_resolve_font_path` copies return | -| `sports_game_over.py` | next release | `SportsGameOverMixin` — `_is_game_really_over`, with the `FINAL_PERIOD` seam (family 5) | +| `sports_game_over.py` | 3.8.1 | `SportsGameOverMixin` — `_is_game_really_over`, with the `FINAL_PERIOD` seam (family 5) | Each is described in [src/common/README.md](../src/common/README.md). diff --git a/src/__init__.py b/src/__init__.py index b181d528..9bf9335e 100644 --- a/src/__init__.py +++ b/src/__init__.py @@ -4,5 +4,5 @@ LEDMatrix Display System Core source package for the LED Matrix Display project. """ -__version__ = "3.8.0" +__version__ = "3.8.1" diff --git a/src/common/README.md b/src/common/README.md index b84198c7..be1d264d 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -46,7 +46,7 @@ Rules for the package: | [`sports_display_rules`](#sports_display_rules) | Which games a scoreboard shows, for how long, and its scorebug date line | Yes (scoreboards) | 3.8.0 | | [`sports_fetch`](#sports_fetch) | Scoreboard season fetch, lookback and live-odds decisions | Yes (scoreboards) | 3.7.0 | | [`sports_font_path`](#sports_font_path) | Find a scoreboard's bundled font whatever the cwd | Yes (scoreboards) | 3.8.0 | -| [`sports_game_over`](#sports_game_over) | Whether a game ESPN still lists as live has ended | Yes (scoreboards) | next release | +| [`sports_game_over`](#sports_game_over) | Whether a game ESPN still lists as live has ended | Yes (scoreboards) | 3.8.1 | | [`sports_game_renderer`](#sports_game_renderer) | Scoreboard scroll/Vegas card geometry | Yes (scoreboards) | 3.3.0 | | [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 | | [`sports_live_scroll`](#sports_live_scroll) | Rebuild a live scroll strip mid-cycle without moving it | Yes (scoreboards) | 3.8.0 | @@ -402,7 +402,7 @@ Created by `DisplayController`; works with any plugin. `draw_multiline_text()`, `create_text_image()`. `draw_text_outlined(draw, xy, text, font, fill, outline_color=(0, 0, 0), -offsets=OUTLINE_SQUARE)` (Unreleased) draws the text in `outline_color` at +offsets=OUTLINE_SQUARE)` (3.8.1) draws the text in `outline_color` at each offset, then in `fill` on top: the same pixels as one `draw.text` per offset, but the string is rasterized once. `OUTLINE_SQUARE` is the eight-sided one-pixel outline the scoreboards draw, `OUTLINE_CROSS` the From e745ae8060b6381939860ca874c57cf7822d1bf3 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 10:47:29 -0400 Subject: [PATCH 25/37] feat(display): cap malloc arenas in-process and malloc_trim between screens (#774) * feat(display): cap malloc arenas in-process and malloc_trim between screens Co-Authored-By: Claude Opus 5.5 * test: malloc_tuning with ctypes mocked; add to the mypy ratchet Co-Authored-By: Claude Opus 5.5 * docs(changelog): malloc arena cap and malloc_trim between screens Co-Authored-By: Claude Opus 5.5 * docs(changelog): spacing Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 23 +++++ mypy-clean.txt | 1 + run.py | 6 ++ src/display_controller.py | 7 ++ src/malloc_tuning.py | 123 ++++++++++++++++++++++ test/test_malloc_tuning.py | 206 +++++++++++++++++++++++++++++++++++++ 6 files changed, 366 insertions(+) create mode 100644 src/malloc_tuning.py create mode 100644 test/test_malloc_tuning.py diff --git a/CHANGELOG.md b/CHANGELOG.md index abeee525..44724d01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,29 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +### The display hands freed memory back to the OS + +The display process's resident memory climbed in steps for hours while the +data it held stayed flat: glibc keeps what Python frees in per-thread malloc +arenas and returns little of it. `src/malloc_tuning.py` (new, standard library +only, a no-op off Linux/glibc) does two things in-process, so it reaches +devices without re-running the installer: + +- **Arena cap at start-up.** `run.py` calls `mallopt(M_ARENA_MAX, 2)` before any + thread exists, the same cap as the unit's `Environment=MALLOC_ARENA_MAX=2`. + Units installed before that line never got it (systemd runs the copy in + `/etc/systemd/system`); a `MALLOC_ARENA_MAX` in the environment still wins. +- **`malloc_trim(0)` between screens**, at most every 5 minutes, from the top of + the render loop where no frame is being drawn. Measured on a Pi 4: 2-11 ms + per call. + +On ledpi (Pi 4, 192x48, Vegas on, nine plugins, a unit without +`MALLOC_ARENA_MAX`), alternated main / branch / branch / main arms of 2.5 h: +two hours in, resident memory was 551 MB on main (the second main arm was +already at 651 MB after 1 h 44 min) against 412 and 386 MB with this change, +and the 20-minute frame soaks came out at 0.147-0.165% late against main's +0.151-0.188%. + ## 3.8.1 Smooth scrolling at the slower speeds, and the fixes and performance work diff --git a/mypy-clean.txt b/mypy-clean.txt index ffc7a6c5..aa3ba212 100644 --- a/mypy-clean.txt +++ b/mypy-clean.txt @@ -59,6 +59,7 @@ src/ipc/contract.py src/ipc/server.py src/logging_config.py src/logo_downloader.py +src/malloc_tuning.py src/matrix_support.py src/pi5_matrix_support.py src/plugin_system/__init__.py diff --git a/run.py b/run.py index c327d549..af7d2c6c 100755 --- a/run.py +++ b/run.py @@ -14,6 +14,12 @@ project_dir = os.path.dirname(os.path.abspath(__file__)) if project_dir not in sys.path: sys.path.insert(0, project_dir) +# Cap glibc's malloc arenas before any thread exists (arenas already made +# stay): the in-process twin of the unit's MALLOC_ARENA_MAX=2, for units +# installed before that line. A no-op off glibc. See src/malloc_tuning.py. +from src import malloc_tuning +malloc_tuning.cap_arenas() + # Under systemd the watchdog clock is already running, and start-up (plugin # loads, initial updates) takes far longer than the render loop's limit. Widen # it before anything slow is imported; the render loop narrows it again once diff --git a/src/display_controller.py b/src/display_controller.py index 84bfbe8c..44a70f84 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -37,6 +37,7 @@ from concurrent.futures import ThreadPoolExecutor, as_completed # pylint: disab import pytz from src import display_watchdog +from src.malloc_tuning import MallocTrimmer from src.display_arbiter import ( Arbiter, ArbiterInputs, ArbiterState, FramePolicy, ScreenPlan, Source, WifiNotice, live_pick, live_takeover, on_demand_bound, rotation_plan, @@ -4217,6 +4218,7 @@ class DisplayController: logger.info(f"Initial mode set to: {self.current_display_mode} (index: {self.current_mode_index}, total modes: {len(self.available_modes)})") self._publish_current_mode_state() runner = ScreenRunner(_MODULE_CLOCK, _ScreenHost(self), logger) + trimmer = MallocTrimmer() while True: # Arms the watchdog after the first frame -- or after the @@ -4224,6 +4226,11 @@ class DisplayController: # it from then on. display_watchdog.watchdog.loop_pass() + # Between screens, nothing being drawn: every few minutes hand + # the memory glibc is holding for freed images back to the OS + # (src/malloc_tuning.py). A clock read when none is due. + trimmer.maybe_trim() + # Apply plugin enable/disable edits saved via the web UI. The # config-watcher thread only sets the flag; loading/unloading and # rebuilding available_modes happens here on the render thread so diff --git a/src/malloc_tuning.py b/src/malloc_tuning.py new file mode 100644 index 00000000..a2fcf632 --- /dev/null +++ b/src/malloc_tuning.py @@ -0,0 +1,123 @@ +"""Keep glibc's malloc from holding on to memory the display has freed. + +The display process allocates and frees PIL images and numpy buffers all day +from a dozen threads. glibc gives each allocating thread its own malloc arena +(up to 8 x CPU count) and returns little of what is freed inside them to the +OS, so resident memory climbs for hours while the live data stays flat. Two +in-process remedies, both standard library only (ctypes) and both no-ops off +Linux/glibc: + +* :func:`cap_arenas` -- ``mallopt(M_ARENA_MAX, 2)``, the in-process twin of the + unit's ``Environment=MALLOC_ARENA_MAX=2``. Units installed before that line + existed never got it (systemd runs the copy in /etc/systemd/system), so the + process applies it itself. Call it before any other thread starts: arenas + already created stay. A ``MALLOC_ARENA_MAX`` set in the environment wins. +* :class:`MallocTrimmer` -- ``malloc_trim(0)`` at most every few minutes, + called from the render loop between screens, where no frame is being drawn. + glibc 2.8+ releases free pages from the middle of every arena, not only the + top of the main heap. + +Without glibc (macOS, Windows, musl, the dev server on any of them) nothing is +loaded and every call returns False. +""" +import ctypes +import logging +import os +import sys +import time +from typing import Any, Callable, Optional + +logger = logging.getLogger(__name__) + +#: glibc's mallopt() parameter number for the arena cap (malloc.h). +M_ARENA_MAX = -8 + +#: The arena cap applied when the environment does not set one; the same value +#: as the unit's ``MALLOC_ARENA_MAX``. +DEFAULT_ARENA_MAX = 2 + +#: Seconds between malloc_trim() calls. A trim takes about 1-20 ms on a Pi 4, +#: so this keeps it far from frame timing while still returning memory long +#: before it piles up. +TRIM_INTERVAL_SECONDS = 300.0 + +_UNLOADED = object() +_libc: Any = _UNLOADED + + +def _load_libc() -> Optional[Any]: + """The process's C library if it is glibc with malloc_trim, else None.""" + global _libc + if _libc is _UNLOADED: + _libc = None + if sys.platform.startswith('linux'): + try: + libc = ctypes.CDLL(None) + # gnu_get_libc_version is glibc-only, so musl (which has + # mallopt but no malloc_trim) is left alone as a whole. + if all(hasattr(libc, name) for name in + ('gnu_get_libc_version', 'malloc_trim', 'mallopt')): + libc.malloc_trim.argtypes = [ctypes.c_size_t] + libc.malloc_trim.restype = ctypes.c_int + libc.mallopt.argtypes = [ctypes.c_int, ctypes.c_int] + libc.mallopt.restype = ctypes.c_int + _libc = libc + except (OSError, AttributeError, TypeError): + logger.debug("glibc malloc controls unavailable", exc_info=True) + return _libc + + +def cap_arenas(max_arenas: int = DEFAULT_ARENA_MAX) -> bool: + """Cap glibc's malloc arenas at ``max_arenas``. True when the cap was set. + + Skipped when ``MALLOC_ARENA_MAX`` is in the environment: glibc has read it + already, and an operator who set it chose that value. + """ + if os.environ.get('MALLOC_ARENA_MAX'): + return False + libc = _load_libc() + if libc is None: + return False + try: + return bool(libc.mallopt(M_ARENA_MAX, int(max_arenas))) + except Exception: # pylint: disable=broad-except + logger.debug("mallopt(M_ARENA_MAX) failed", exc_info=True) + return False + + +class MallocTrimmer: + """Calls ``malloc_trim(0)`` at most once per ``interval`` seconds. + + :meth:`maybe_trim` is meant for an idle point of the render loop; it costs + one clock read when no trim is due. The first trim comes one interval + after construction, so start-up's allocations have settled. + """ + + def __init__(self, interval: float = TRIM_INTERVAL_SECONDS, + clock: Callable[[], float] = time.monotonic) -> None: + self._interval = interval + self._clock = clock + self._libc = _load_libc() + self._next = clock() + interval + + @property + def available(self) -> bool: + return self._libc is not None + + def maybe_trim(self) -> bool: + """Trim if one is due. True when malloc_trim ran and released memory.""" + if self._libc is None: + return False + now = self._clock() + if now < self._next: + return False + self._next = now + self._interval + try: + released = bool(self._libc.malloc_trim(0)) + except Exception: # pylint: disable=broad-except + logger.debug("malloc_trim failed; not trying again", exc_info=True) + self._libc = None + return False + logger.debug("malloc_trim(0) took %.1f ms, released=%s", + (self._clock() - now) * 1000.0, released) + return released diff --git a/test/test_malloc_tuning.py b/test/test_malloc_tuning.py new file mode 100644 index 00000000..438cce48 --- /dev/null +++ b/test/test_malloc_tuning.py @@ -0,0 +1,206 @@ +"""src/malloc_tuning.py: glibc arena cap and periodic malloc_trim, ctypes mocked.""" +import ctypes +from pathlib import Path +from unittest import mock + +import pytest + +from src import malloc_tuning as mt + + +class FakeLibc: + """Stands in for ctypes.CDLL(None) on glibc: records calls.""" + + def __init__(self, trim_result=1, glibc=True): + self.trims = [] + self.mallopts = [] + self._trim_result = trim_result + if glibc: + self.gnu_get_libc_version = lambda: b'2.41' + self.malloc_trim = mock.Mock(side_effect=self._trim) + self.mallopt = mock.Mock(side_effect=self._mallopt) + + def _trim(self, pad): + self.trims.append(pad) + if isinstance(self._trim_result, Exception): + raise self._trim_result + return self._trim_result + + def _mallopt(self, param, value): + self.mallopts.append((param, value)) + return 1 + + +@pytest.fixture(autouse=True) +def fresh_libc(monkeypatch): + """Each test loads the C library itself; nothing real is called.""" + monkeypatch.setattr(mt, '_libc', mt._UNLOADED) + monkeypatch.delenv('MALLOC_ARENA_MAX', raising=False) + yield + + +def _on_glibc(monkeypatch, libc): + monkeypatch.setattr(mt.sys, 'platform', 'linux') + cdll = mock.Mock(return_value=libc) + monkeypatch.setattr(mt.ctypes, 'CDLL', cdll) + return cdll + + +class Clock: + def __init__(self, t=1000.0): + self.t = t + + def __call__(self): + return self.t + + +# -- loading ---------------------------------------------------------------- + +@pytest.mark.parametrize('platform', ['win32', 'darwin', 'freebsd14']) +def test_not_linux_loads_nothing(monkeypatch, platform): + monkeypatch.setattr(mt.sys, 'platform', platform) + cdll = mock.Mock(side_effect=AssertionError('must not load')) + monkeypatch.setattr(mt.ctypes, 'CDLL', cdll) + assert mt._load_libc() is None + assert mt.cap_arenas() is False + trimmer = mt.MallocTrimmer(interval=0) + assert not trimmer.available + assert trimmer.maybe_trim() is False + cdll.assert_not_called() + + +def test_linux_without_glibc_is_a_noop(monkeypatch): + """musl: no gnu_get_libc_version (and no malloc_trim) -- nothing is called.""" + libc = FakeLibc(glibc=False) + del libc.malloc_trim + _on_glibc(monkeypatch, libc) + assert mt._load_libc() is None + assert mt.cap_arenas() is False + assert mt.MallocTrimmer(interval=0).maybe_trim() is False + assert libc.mallopts == [] + + +def test_cdll_failure_is_a_noop(monkeypatch): + monkeypatch.setattr(mt.sys, 'platform', 'linux') + monkeypatch.setattr(mt.ctypes, 'CDLL', mock.Mock(side_effect=OSError('no libc'))) + assert mt._load_libc() is None + assert mt.cap_arenas() is False + + +def test_loads_once(monkeypatch): + cdll = _on_glibc(monkeypatch, FakeLibc()) + mt._load_libc() + mt._load_libc() + mt.MallocTrimmer() + assert cdll.call_count == 1 + + +def test_declares_c_signatures(monkeypatch): + libc = FakeLibc() + _on_glibc(monkeypatch, libc) + mt._load_libc() + assert libc.malloc_trim.argtypes == [ctypes.c_size_t] + assert libc.mallopt.argtypes == [ctypes.c_int, ctypes.c_int] + + +# -- cap_arenas --------------------------------------------------------------- + +def test_cap_arenas_calls_mallopt(monkeypatch): + libc = FakeLibc() + _on_glibc(monkeypatch, libc) + assert mt.cap_arenas() is True + assert libc.mallopts == [(mt.M_ARENA_MAX, 2)] + assert mt.M_ARENA_MAX == -8 # glibc's malloc.h + + +def test_cap_arenas_defers_to_the_environment(monkeypatch): + libc = FakeLibc() + _on_glibc(monkeypatch, libc) + monkeypatch.setenv('MALLOC_ARENA_MAX', '4') + assert mt.cap_arenas() is False + assert libc.mallopts == [] + + +def test_cap_arenas_swallows_errors(monkeypatch): + libc = FakeLibc() + libc.mallopt = mock.Mock(side_effect=RuntimeError('boom')) + _on_glibc(monkeypatch, libc) + assert mt.cap_arenas() is False + + +def test_cap_arenas_matches_the_unit(): + """The in-process default is the value the unit's MALLOC_ARENA_MAX carries.""" + unit = (Path(__file__).resolve().parent.parent / 'systemd' / 'ledmatrix.service').read_text() + assert f'Environment=MALLOC_ARENA_MAX={mt.DEFAULT_ARENA_MAX}\n' in unit + + +# -- MallocTrimmer ------------------------------------------------------------ + +def test_trim_waits_one_interval_then_rate_limits(monkeypatch): + libc = FakeLibc() + _on_glibc(monkeypatch, libc) + clock = Clock() + trimmer = mt.MallocTrimmer(interval=300, clock=clock) + assert trimmer.available + assert trimmer.maybe_trim() is False # start-up: not yet + clock.t += 299.9 + assert trimmer.maybe_trim() is False + clock.t += 0.1 + assert trimmer.maybe_trim() is True + assert libc.trims == [0] + clock.t += 100 + assert trimmer.maybe_trim() is False # rate-limited + clock.t += 200 + assert trimmer.maybe_trim() is True + assert libc.trims == [0, 0] + + +def test_trim_reports_nothing_released(monkeypatch): + libc = FakeLibc(trim_result=0) + _on_glibc(monkeypatch, libc) + clock = Clock() + trimmer = mt.MallocTrimmer(interval=10, clock=clock) + clock.t += 10 + assert trimmer.maybe_trim() is False + assert libc.trims == [0] + + +def test_trim_failure_disables_trimming(monkeypatch): + libc = FakeLibc(trim_result=RuntimeError('boom')) + _on_glibc(monkeypatch, libc) + clock = Clock() + trimmer = mt.MallocTrimmer(interval=10, clock=clock) + clock.t += 10 + assert trimmer.maybe_trim() is False + clock.t += 10 + assert trimmer.maybe_trim() is False + assert libc.trims == [0] # not retried + assert not trimmer.available + + +# -- wiring ------------------------------------------------------------------- + +def test_run_py_caps_arenas_before_threads(): + """run.py applies the cap before the watchdog or the controller import.""" + src = (Path(__file__).resolve().parent.parent / 'run.py').read_text() + cap = src.index('malloc_tuning.cap_arenas()') + assert cap < src.index('display_watchdog.watchdog.begin_startup()') + assert cap < src.index('from src.display_controller import main') + + +def test_render_loop_trims_between_screens(): + src = (Path(__file__).resolve().parent.parent / 'src' / 'display_controller.py').read_text() + loop = src.index('display_watchdog.watchdog.loop_pass()') + trim = src.index('trimmer.maybe_trim()') + assert loop < trim < src.index('outcome = runner.run(plan, manager_to_display)') + + +@pytest.mark.skipif(not mt.sys.platform.startswith('linux'), reason='glibc only') +def test_real_libc_on_linux(): + """On a real Linux C library the calls go through without raising.""" + if mt._load_libc() is None: + pytest.skip('not glibc') + trimmer = mt.MallocTrimmer(interval=0) + assert trimmer.available + assert trimmer.maybe_trim() in (True, False) + assert trimmer.available # did not fail and disable itself From 3bdb5bff3b7dba88a9f4334f658b46bb087863f5 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 12:33:05 -0400 Subject: [PATCH 26/37] feat(common): sports_favorites -- the reconciled favourite matching (sports family 6) (#775) * feat(common): sports_favorites -- the reconciled favourite matching (sports family 6) New hardware-free module src/common/sports_favorites.py, copied from ledmatrix-plugins claude/family6-reconcile once the nine scoreboards made _is_favorite_game (seven bodies), _select_games_for_display (two) and _select_recent_games_for_display (three) one body each. One mixin per class that carries the methods, so adopting one gives no manager a method it did not have: - SportsFavoritesMixin (SportsCore): _is_favorite_game and _favorite_code. - SportsUpcomingFavoritesMixin: _select_games_for_display. - SportsRecentFavoritesMixin: _select_recent_games_for_display. Each side of a game is named by the 3.5.0 _favorite_key seam (SportsHelpersMixin; the abbreviation by default, nrl overrides it with the ESPN team id and None for a missing id) and compared with favorite_teams stripped and upper-cased. The selection methods give each favourite up to the per-team limit, count a game between two favourites for both, treat only games with an id as possible duplicates and log their summary at INFO. - test/test_sports_favorites.py: the plugins' pinned cases for an abbreviation host and an id-keyed (nrl-style) host -- case, spaces, ids, the NEW collision, the "None" favourite, missing keys; selection order, limits, duplicates and the id-less fix, the INFO summary; host contract, one carrier per method, and SportsGameRulesMixin reaching the shared body. - test/test_sports_favorites_parity.py: with LEDMATRIX_PLUGINS, compares each body with every plugin copy (drift-report normalisation plus decorators), checks no other plugin class carries a copy, and that only nrl overrides _favorite_key. - mypy ratchet, src/common/README.md, CHANGELOG (Unreleased, New modules). - sports_helpers docstrings: _favorite_key now has a caller and an override. - docs/SPORTS_UNIFICATION.md: family 6 status and decisions, and the seam table. SportsCoreSharedMixin._round_robin_favorites still groups by raw abbreviation or _team_in: it is not one of the plugin bodies, so it waits for a later family. Co-Authored-By: Claude Opus 5.5 * docs(sports): family 6 also routes the Upcoming favourites-only filter and three live boosts ledmatrix-plugins claude/family6-reconcile now sends the Upcoming update()'s favourites-only pre-filter and the basketball, hockey and lacrosse live favourite boost through _is_favorite_game, so a lower-case favourite works on a favourites-only Upcoming board. The module is unchanged (update() is not promoted); the parity test still passes against the branch. Updates the pinned row and cell counts and what is left for later families. Co-Authored-By: Claude Opus 5.5 * docs: cite ledmatrix-plugins #635 for the family 6 reconcile Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 13 ++ docs/SPORTS_UNIFICATION.md | 52 ++++- mypy-clean.txt | 1 + src/common/README.md | 14 ++ src/common/sports_favorites.py | 263 +++++++++++++++++++++++++ src/common/sports_helpers.py | 22 +-- test/test_sports_favorites.py | 280 +++++++++++++++++++++++++++ test/test_sports_favorites_parity.py | 135 +++++++++++++ 8 files changed, 761 insertions(+), 19 deletions(-) create mode 100644 src/common/sports_favorites.py create mode 100644 test/test_sports_favorites.py create mode 100644 test/test_sports_favorites_parity.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 44724d01..90e0e87e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,19 @@ already at 651 MB after 1 h 44 min) against 412 and 386 MB with this change, and the 20-minute frame soaks came out at 0.147-0.165% late against main's 0.151-0.188%. +### New modules + +- `src/common/sports_favorites.py` -- sports consolidation family 6, once the + plugins made `_is_favorite_game` (seven bodies), `_select_games_for_display` + (two) and `_select_recent_games_for_display` (three) one each: + `SportsFavoritesMixin` (`SportsCore`: `_is_favorite_game`, `_favorite_code`), + `SportsUpcomingFavoritesMixin` and `SportsRecentFavoritesMixin` (the + favourites-only picks). Each side of a game is named by the 3.5.0 + `_favorite_key` seam and compared with `favorite_teams` stripped and + upper-cased; nrl overrides the key with the ESPN team id. Only a game with an + id can be a duplicate. A plugin may inherit the mixins once it floors on the + release that ships this module, and deletes its copies then. + ## 3.8.1 Smooth scrolling at the slower speeds, and the fixes and performance work diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index 2585829c..f30e8a4c 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -92,6 +92,7 @@ more. Shared sports code lives in `src/common`: | `sports_display_rules.py` | next release | `SportsCardOptionsMixin`, `SportsGameRulesMixin` — scorebug date options, the no-favourites filter, non-favourite live dwell | | `sports_font_path.py` | next release | `resolve_font_path` — what the plugins' `_resolve_font_path` copies return | | `sports_game_over.py` | 3.8.1 | `SportsGameOverMixin` — `_is_game_really_over`, with the `FINAL_PERIOD` seam (family 5) | +| `sports_favorites.py` | next release | `SportsFavoritesMixin`, `SportsUpcomingFavoritesMixin`, `SportsRecentFavoritesMixin` — `_is_favorite_game` and the favourites-only picks, on the `_favorite_key` seam (family 6) | Each is described in [src/common/README.md](../src/common/README.md). @@ -103,7 +104,8 @@ modules taken from the plugin copies, each a **new module** rather than growth on an existing one: a plugin that deletes a method copy and relies on an older module having gained it fails at runtime with an `AttributeError`, while a missing module fails at load, where the version checks can see it. -`sports_helpers.py` holds `_favorite_key`, the override point listed below. +`sports_helpers.py` holds `_favorite_key`, the override point listed below; +`sports_favorites.py` is what calls it. Each promoted module has a parity test that compares its bodies against the plugin copies when `LEDMATRIX_PLUGINS` points at a checkout (`test_sports_helpers.py`, `test_sports_stage3_parity.py`), and @@ -126,7 +128,7 @@ deprecation cycle. | `_custom_scorebug_layout(game, draw)` | Per-sport overlay on the base layout | no-op | | `score_phrase(points, team_abbr)` | Celebration wording (`"GOOOOAAALLL!"` vs `"TOUCHDOWN!"`). `points` is the score delta, which sports with variable-value scores use to name the play | `" SCORES!"` — only consulted when `CelebrationMixin` is present | | `win_phrase(team_abbr)` | Win-celebration wording | `" WINS!"` — mixin only | -| `_favorite_key(game, side)` | Which view-model field identifies a team for favorites matching | `game["_abbr"]` | +| `_favorite_key(game, side)` | Which view-model field identifies a team for favorites matching. `sports_favorites` compares it, and each `favorite_teams` entry, stripped and upper-cased; a `None` matches nothing | `game["_abbr"]`. nrl returns the ESPN team id, `None` when it is missing | | `_config_schema_path()` | Plugin's `config_schema.json` — returning it routes `_get_layout_offset` through the `src.element_style` resolver (and gives it the defaults to compare against) | `None`, i.e. the classic inline `customization.layout` read | | `_font_root()` | Directory to resolve `assets/fonts` against | core install root | @@ -315,6 +317,35 @@ were pixel-identical. `src/common/sports_game_over.py` holds the body; `test/test_sports_game_over_parity.py` compares it, and each plugin's `FINAL_PERIOD`, with the plugin copies. +### Family 6: favourite matching (core done; adoption waits for a release) + +ledmatrix-plugins `scripts/test_favourite_matching.py` (#634) pinned 204 rows +across the nine plugins first: `_is_favorite_game` on each manager role, the +two selection methods, the real `update()` with favourites-only on and off, +and the INFO summary; the reconcile extends it to 217 (a lower-case and a +padded favourite through `update()`, and the live favourite boost). The reconcile (ledmatrix-plugins +#635) made `_is_favorite_game` one body on `SportsCore` +(afl and soccer's `SportsUpcoming` copies and five `SportsLive` copies, all +redundant, are gone), added `_favorite_code` beside it, and gave nrl a +`_favorite_key` override instead of its own copies. So that a lower-case +favourite works on a favourites-only Upcoming board, the Upcoming `update()`'s +favourites-only pre-filter and the basketball, hockey and lacrosse live boost +now ask `_is_favorite_game` too (a one-line change each; `update()` itself is +family 13). Of 3,897 cells only those the decisions above explain changed: +case and spaces in eight plugins (30-40 each), the id-less duplicate fix (6-8 +each), nrl's key (6) and its "None" match (6), and the INFO line in baseball, +football and ufc. The harness renders were byte-identical. `src/common/sports_favorites.py` holds the +bodies, one mixin per carrying class; `test/test_sports_favorites_parity.py` +compares them with the plugin copies and checks that only nrl overrides +`_favorite_key`. + +Left for later families: the live screens' favourites-only filter +(`_classify_live_game` and its inline copies) and favourites-first sort still +compare abbreviations exactly, and +`SportsCoreSharedMixin._round_robin_favorites` groups favourites by raw +abbreviation (or by `_team_in` where a plugin has one) instead of through +`_favorite_key`. The result-colour helpers also wait (decision above). + ### Why the method changes Byte-identical promotion has nearly run dry. Measured on ledmatrix-plugins @@ -400,7 +431,7 @@ release. |---|---|---|---| | 4 | Identical sweep | `manager.py`: `_dispatch_switch_refresh`, `_favorite_team_is_live`, `get_vegas_priority_weight`, `_game_involves`, `_favorite_scan_targets`, `_favorite_scan_games`, `_get_total_games_for_manager` (all nine, 1); the live-scroll helpers `_preserving_scroll_position`, `_refresh_live_scroll_managers`, `_live_scroll_managers`, `_note_live_scroll_built`, `_live_scroll_needs_rebuild`, `_live_scroll_fields` (eight, 1). `sports.py`: `_card_option`, `_filtered_or_all`, `_effective_live_duration`, `_recent_date_text` (eight, 1). 58 identical families in all | Nothing to decide; brings `manager.py` into core as a `SportsPluginHostMixin`. `_resolve_font_path` (identical in nine `sports.py` and eight renderers) becomes `sports_font_path.resolve_font_path`, not `font_layout.resolve_asset_path`, which skips the cwd. Core side done; see [Stage 4](#stage-4-the-identical-sweep-core-done-adoption-waits-for-a-release) | | 5 | Game-over check | `SportsLive._is_game_really_over` (5) | Pure logic, no pixels; one seam, `FINAL_PERIOD`. The pilot for the procedure. Reconciled to one body and promoted as `sports_game_over`; adoption waits for the release that ships it. See [Family 5](#family-5-the-game-over-check-core-done-adoption-waits-for-a-release) | -| 6 | Favourite matching | `_is_favorite_game` (7 across three classes), `_select_games_for_display` (2: nrl), `_select_recent_games_for_display` (3) | Everything that asks "is this a favourite" goes through the 3.5.0 `_favorite_key` seam | +| 6 | Favourite matching | `_is_favorite_game` (7 across three classes), `_select_games_for_display` (2: nrl), `_select_recent_games_for_display` (3) | Everything that asks "is this a favourite" goes through the 3.5.0 `_favorite_key` seam. Reconciled to one body each and promoted as `sports_favorites`; adoption waits for the release that ships it. See [Family 6](#family-6-favourite-matching-core-done-adoption-waits-for-a-release) | | 7 | Other-games rotation | `_by_importance`, `_other_games_window`, `_advance_other_games_if_due` (2 each: football), `_rotate_other_games_on_display` (2: ufc) | One outlier each; football carries two fixes the other eight lack | | 8 | Rankings | `_fetch_team_rankings` (3), `_choose_poll` (3), `_load_division_team_ids`, `_passes_other_filters`, `_best_rank`, `_is_ranked_game` (2 each: football) | Needs 7; the rank badge and the "ranked only" filter read it | | 9 | Live fetch and odds | `_fetch_todays_games` (5), `_fetch_odds` (3), `_attach_odds_to_rotated_games` (3) | The prerequisite for one shared ESPN poller across plugins | @@ -444,10 +475,17 @@ suspected behaviour that needs a payload or a rig to confirm first. as `0:00`). A score level at 0:00 is not over: the game stays live through the break before overtime, and one that really ends tied ends on its final status. Baseball keeps its postponed/suspended override in `BaseballLive`. -- **6, favourite matching.** NRL keeps matching favourites by team id - (abbreviations collide: NEW, CAN), through `_favorite_key` rather than its - own copies of the selection methods. Six plugins log the recent-games - selection at INFO; baseball, football and ufc do not. +- **6, favourite matching. Decided 2026-10-05, done:** each side of a game is + named by `_favorite_key` (the abbreviation; NRL overrides it with the ESPN + team id, and `None` for a missing id, which fixes a favourite typed "None" + matching every game without one) and compared with `favorite_teams` + stripped and upper-cased, so " bos" matches BOS. NRL's ambiguous "NEW" + still matches nothing and is logged; routing the result-colour helpers + (`side_is_favorite`, which tint both NEW clubs) through `_favorite_key` is + left for a later family. The recent-games selection logs at INFO in all + nine. ufc stays on the shared body, dormant: its favourites are fighters, + which its MMA managers match themselves (a follow-up). Fix ported: only a + game with an id can be a duplicate in the selection methods. - **7, other-games rotation.** football advances the rotation window under `_games_lock` (update() and display() both advance it; interleaved, a window of games is skipped) and fixes a favourites-only pool that recomposed diff --git a/mypy-clean.txt b/mypy-clean.txt index aa3ba212..fa11e729 100644 --- a/mypy-clean.txt +++ b/mypy-clean.txt @@ -36,6 +36,7 @@ src/common/sports_card.py src/common/sports_card_wrappers.py src/common/sports_celebration.py src/common/sports_display_rules.py +src/common/sports_favorites.py src/common/sports_fetch.py src/common/sports_font_path.py src/common/sports_game_over.py diff --git a/src/common/README.md b/src/common/README.md index be1d264d..432b0fad 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -44,6 +44,7 @@ Rules for the package: | [`sports_card_wrappers`](#sports_card_wrappers) | The game renderer's `sports_card` delegations | Yes (scoreboards) | 3.7.0 | | [`sports_celebration`](#sports_celebration) | Draw a scoreboard's score/win celebration | Yes (scoreboards) | 3.7.0 | | [`sports_display_rules`](#sports_display_rules) | Which games a scoreboard shows, for how long, and its scorebug date line | Yes (scoreboards) | 3.8.0 | +| [`sports_favorites`](#sports_favorites) | Which games involve a favourite team, and the favourites-only picks | Yes (scoreboards) | next release | | [`sports_fetch`](#sports_fetch) | Scoreboard season fetch, lookback and live-odds decisions | Yes (scoreboards) | 3.7.0 | | [`sports_font_path`](#sports_font_path) | Find a scoreboard's bundled font whatever the cwd | Yes (scoreboards) | 3.8.0 | | [`sports_game_over`](#sports_game_over) | Whether a game ESPN still lists as live has ended | Yes (scoreboards) | 3.8.1 | @@ -280,6 +281,19 @@ list it before `SportsCoreSharedMixin`) and `SportsGameRulesMixin` `_effective_live_duration()`, the shorter dwell for a non-favourite live game). +### sports_favorites + +[`sports_favorites.py`](sports_favorites.py). Sports family 6, one mixin per +class that carried the methods: `SportsFavoritesMixin` (`SportsCore`: +`_is_favorite_game(game)` and `_favorite_code(value)`), +`SportsUpcomingFavoritesMixin` (`_select_games_for_display`) and +`SportsRecentFavoritesMixin` (`_select_recent_games_for_display`). Each side +of a game is named by `_favorite_key` (from `SportsHelpersMixin`; NRL +overrides it with the team id) and compared with `favorite_teams` stripped and +upper-cased. The selection methods give each favourite up to the per-team +limit, count a game between two favourites for both, and treat only games +with an id as possible duplicates. + ### sports_fetch [`sports_fetch.py`](sports_fetch.py). `SportsFetchMixin`: the `SportsCore` diff --git a/src/common/sports_favorites.py b/src/common/sports_favorites.py new file mode 100644 index 00000000..88e57f4c --- /dev/null +++ b/src/common/sports_favorites.py @@ -0,0 +1,263 @@ +"""Which games involve a favourite team, and which of them to show (sports family 6). + +The scoreboards' favourite matching, reconciled in ledmatrix-plugins +(family 6) from seven ``_is_favorite_game`` bodies, two +``_select_games_for_display`` and three ``_select_recent_games_for_display`` +into one each, and copied here under their existing names: + +- ``SportsFavoritesMixin`` (``SportsCore``): ``_is_favorite_game(game)``, + asked by ``SportsCoreSharedMixin._favorites_first``, the switch-mode + favourite boost (``SportsHelpersMixin._next_switch_index``), the + non-favourite live dwell (``SportsGameRulesMixin._effective_live_duration``) + and the plugins' live rotation; and ``_favorite_code(value)``, the + normalisation both sides of every comparison go through. +- ``SportsUpcomingFavoritesMixin`` (``SportsUpcoming``): + ``_select_games_for_display``, the favourites-only pick of upcoming games. +- ``SportsRecentFavoritesMixin`` (``SportsRecent``): + ``_select_recent_games_for_display``, the same for finished games, most + recent first. + +Each mixin carries only what its class already had, so no manager gains a +method it did not have. + +THE RULE +-------- +Each side of a game is named by ``_favorite_key(game, side)``, the override +point ``SportsHelpersMixin`` (``src.common.sports_helpers``) has carried since +3.5.0: the team abbreviation by default. A sport whose abbreviations are not +unique overrides it -- NRL returns the ESPN team id (and None when the id is +missing), because "NEW" is both Newcastle and New Zealand. That value and every +entry of ``favorite_teams`` are compared as ``_favorite_code`` leaves them: +stripped and upper-cased, a blank or missing value matching nothing. So +" bos" in the config matches BOS. + +The selection methods give each favourite team up to the per-team limit +(``upcoming_games_to_show`` / ``recent_games_to_show``); a game between two +favourites counts for both. Only a game with an id can be a duplicate: two +games without one are two games. + +A new module rather than more methods on ``sports_shared`` or +``sports_helpers``, for the reason ``sports_helpers`` gives: a missing module +fails at load, where the version checks see it; a missing method fails +mid-update. + +WHAT A HOST MUST PROVIDE +------------------------ +Derived by walking every ``self.`` the mixins read; the host-contract +test in ``test/test_sports_favorites.py`` fails if a read is added without +being listed here. + +- ``favorite_teams`` -- the resolved favourites list (``_is_favorite_game``). + The selection methods are handed the list instead. +- ``_favorite_key`` -- ``SportsHelpersMixin`` supplies the default. +- ``_favorite_code`` -- from ``SportsFavoritesMixin``, which the Upcoming and + Recent classes inherit through their ``SportsCore``. +- ``logger`` -- the selection methods log each pick at DEBUG and a summary at + INFO. +- ``upcoming_games_to_show`` (Upcoming) and ``recent_games_to_show`` (Recent) + -- the per-team limits. + +The methods read the game dict's ``id`` and ``start_time_utc`` (selection), +whatever ``_favorite_key`` reads (``home_abbr`` / ``away_abbr`` by default), +and ``home_abbr`` / ``away_abbr`` again for the DEBUG line; any may be missing. + +BASE ORDER +---------- +No other mixin defines these methods, so the position in the bases does not +change which body runs; a method on the plugin's own class still wins. The +mixins have no ``__init__`` and no state. +""" + +import logging +from datetime import datetime, timezone +from typing import Callable, Dict, List, Optional + + +class SportsFavoritesMixin: + """``SportsCore``'s favourite check. See module docstring.""" + + # The host contract, declared for type checking only. + favorite_teams: List[str] + _favorite_key: Callable[[Dict, str], Optional[str]] + + @staticmethod + def _favorite_code(value) -> Optional[str]: + """``value`` as favourites are compared: stripped and upper-cased. + + None for a missing or blank value, which matches nothing. + """ + if value is None: + return None + return str(value).strip().upper() or None + + def _is_favorite_game(self, game: Dict) -> bool: + """Does either side of this game belong to a favourite team? + + ``_favorite_key`` names each side (the abbreviation; nrl overrides it + with the ESPN team id), and both it and ``favorite_teams`` are compared + as ``_favorite_code`` normalises them, so " bos" matches BOS. + """ + favorites = {self._favorite_code(team) for team in self.favorite_teams or ()} + favorites.discard(None) + return any( + self._favorite_code(self._favorite_key(game, side)) in favorites + for side in ("home", "away") + ) + + +class SportsUpcomingFavoritesMixin: + """``SportsUpcoming``'s favourites-only pick. See module docstring.""" + + # The host contract, declared for type checking only. + logger: logging.Logger + upcoming_games_to_show: int + _favorite_key: Callable[[Dict, str], Optional[str]] + _favorite_code: Callable[[object], Optional[str]] + + def _select_games_for_display( + self, processed_games: List[Dict], favorite_teams: List[str] + ) -> List[Dict]: + """ + Single-pass game selection with proper deduplication and counting. + + When a game involves two favorite teams, it counts toward BOTH teams' limits. + This prevents unexpected game counts from the multi-pass algorithm. + Teams are matched as _is_favorite_game matches them. Only a game with + an id can be a duplicate: two games without one are two games. + """ + sorted_games = sorted( + processed_games, + key=lambda g: g.get("start_time_utc") + or datetime.max.replace(tzinfo=timezone.utc), + ) + + if not favorite_teams: + return sorted_games + + selected_games = [] + selected_ids = set() + team_counts: Dict[Optional[str], int] = { + code: 0 for code in map(self._favorite_code, favorite_teams) if code + } + + for game in sorted_games: + game_id = game.get("id") + if game_id is not None and game_id in selected_ids: + continue + + home = self._favorite_code(self._favorite_key(game, "home")) + away = self._favorite_code(self._favorite_key(game, "away")) + + home_fav = home in team_counts + away_fav = away in team_counts + + if not home_fav and not away_fav: + continue + + home_needs = home_fav and team_counts[home] < self.upcoming_games_to_show + away_needs = away_fav and team_counts[away] < self.upcoming_games_to_show + + if home_needs or away_needs: + selected_games.append(game) + if game_id is not None: + selected_ids.add(game_id) + if home_fav: + team_counts[home] += 1 + if away_fav: + team_counts[away] += 1 + + self.logger.debug( + f"Selected game {game.get('away_abbr')}@{game.get('home_abbr')}: " + f"team_counts={team_counts}" + ) + + if all(c >= self.upcoming_games_to_show for c in team_counts.values()): + self.logger.debug("All favorite teams satisfied, stopping selection") + break + + self.logger.info( + f"Selected {len(selected_games)} games for {len(favorite_teams)} " + f"favorite teams: {team_counts}" + ) + return selected_games + + +class SportsRecentFavoritesMixin: + """``SportsRecent``'s favourites-only pick. See module docstring.""" + + # The host contract, declared for type checking only. + logger: logging.Logger + recent_games_to_show: int + _favorite_key: Callable[[Dict, str], Optional[str]] + _favorite_code: Callable[[object], Optional[str]] + + def _select_recent_games_for_display( + self, processed_games: List[Dict], favorite_teams: List[str] + ) -> List[Dict]: + """ + Single-pass game selection for recent games with proper deduplication. + + When a game involves two favorite teams, it counts toward BOTH teams' limits. + Games are sorted by most recent first. + Teams are matched as _is_favorite_game matches them. Only a game with + an id can be a duplicate: two games without one are two games. + """ + sorted_games = sorted( + processed_games, + key=lambda g: g.get("start_time_utc") + or datetime.min.replace(tzinfo=timezone.utc), + reverse=True, + ) + + if not favorite_teams: + return sorted_games + + selected_games = [] + selected_ids = set() + team_counts: Dict[Optional[str], int] = { + code: 0 for code in map(self._favorite_code, favorite_teams) if code + } + + for game in sorted_games: + game_id = game.get("id") + if game_id is not None and game_id in selected_ids: + continue + + home = self._favorite_code(self._favorite_key(game, "home")) + away = self._favorite_code(self._favorite_key(game, "away")) + + home_fav = home in team_counts + away_fav = away in team_counts + + if not home_fav and not away_fav: + continue + + home_needs = home_fav and team_counts[home] < self.recent_games_to_show + away_needs = away_fav and team_counts[away] < self.recent_games_to_show + + if home_needs or away_needs: + selected_games.append(game) + if game_id is not None: + selected_ids.add(game_id) + if home_fav: + team_counts[home] += 1 + if away_fav: + team_counts[away] += 1 + + self.logger.debug( + f"Selected recent game {game.get('away_abbr')}@{game.get('home_abbr')}: " + f"team_counts={team_counts}" + ) + + if all(c >= self.recent_games_to_show for c in team_counts.values()): + self.logger.debug("All favorite teams satisfied, stopping selection") + break + + self.logger.info( + f"Selected {len(selected_games)} recent games for {len(favorite_teams)} " + f"favorite teams: {team_counts}" + ) + return selected_games + + +__all__ = ["SportsFavoritesMixin", "SportsUpcomingFavoritesMixin", "SportsRecentFavoritesMixin"] diff --git a/src/common/sports_helpers.py b/src/common/sports_helpers.py index b78f0ca5..640769a2 100644 --- a/src/common/sports_helpers.py +++ b/src/common/sports_helpers.py @@ -34,8 +34,9 @@ first core release that ships it (see ``CHANGELOG.md``). ``_favorite_key`` is the one method not taken from the plugins: it is the override point from the since-removed ``src/base_classes`` sports core, -carried here so later phases (shared celebrations and game selection) have a -hardware-free home for the seam. No plugin defines it today and nothing in this module calls it. +carried here as the hardware-free home for the seam. ``sports_favorites`` +calls it; nrl overrides it with the ESPN team id. Nothing in this module +calls it. WHAT A HOST MUST PROVIDE ------------------------ @@ -47,8 +48,8 @@ listed here. - ``mode_config`` (dict) and ``logger`` -- ``_setting_int``. ``league`` is read with ``getattr`` for the warning text only. - ``games_list`` and ``current_game_index`` -- ``_next_switch_index``; plus - ``_is_favorite_game`` (called with a game), which stays per-plugin and is - only called when + ``_is_favorite_game`` (called with a game; ``sports_favorites`` has the + shared body), only called when ``favorite_rotation_boost`` is above 1. ``favorite_rotation_boost`` itself defaults to 1 on the mixin. - ``last_game_switch`` -- ``_reset_dwell_on_reentry``, read with ``getattr`` @@ -216,15 +217,12 @@ class SportsHelpersMixin: rather than a branch so core never has to learn the string "nrl":: def _favorite_key(self, game, side): - return str(game.get(f"{side}_id")) + team_id = game.get(f"{side}_id") + return None if team_id is None else str(team_id) - An override that stringifies should note that a missing id becomes the - literal ``"None"``, which would spuriously match a favorites list - containing that string. The default returns ``None`` for a missing - abbreviation, which never matches. - - Carried from the since-removed ``src/base_classes`` sports core for - later phases; nothing in this module calls it yet. + It returns ``None`` for a missing id rather than ``str(None)``, which + would match a favourite typed "None". A ``None`` never matches. + ``sports_favorites`` compares the value stripped and upper-cased. """ return game.get(f"{side}_abbr") diff --git a/test/test_sports_favorites.py b/test/test_sports_favorites.py new file mode 100644 index 00000000..b7e3db70 --- /dev/null +++ b/test/test_sports_favorites.py @@ -0,0 +1,280 @@ +"""src.common.sports_favorites: behaviour, the _favorite_key seam, host contract. + +The cases follow ledmatrix-plugins' ``scripts/test_favourite_matching.py`` +(the tables the family 6 reconcile was checked against), with the favourites +given as each plugin's resolver hands them over: as typed for the +abbreviation sports, as ESPN team ids for an NRL-style host that overrides +``_favorite_key``. +""" + +import ast +import logging +from datetime import datetime, timedelta, timezone +from pathlib import Path + +import pytest + +from src.common import sports_favorites +from src.common.sports_favorites import ( + SportsFavoritesMixin, + SportsRecentFavoritesMixin, + SportsUpcomingFavoritesMixin, +) +from src.common.sports_helpers import SportsHelpersMixin + +LOG = logging.getLogger("test_sports_favorites") + + +def _id_key(self, game, side): + """NRL's override: the ESPN team id, None when it is missing.""" + team_id = game.get(f"{side}_id") + return None if team_id is None else str(team_id) + + +def host(favorites, by_id=False, limit=3): + """A manager stand-in: the three mixins over SportsHelpersMixin's default key.""" + attrs = {"_favorite_key": _id_key} if by_id else {} + cls = type("Host", (SportsUpcomingFavoritesMixin, SportsRecentFavoritesMixin, + SportsFavoritesMixin, SportsHelpersMixin), attrs) + h = cls() + h.logger = LOG + h.favorite_teams = favorites + h.upcoming_games_to_show = h.recent_games_to_show = limit + return h + + +TEAM = {"1": "AAA", "2": "BBB", "3": "CCC", "4": "DDD", "41": "NEW", "42": "NEW"} + + +def match(home, away, **extra): + g = {"home_id": home, "home_abbr": TEAM[home], "away_id": away, "away_abbr": TEAM[away]} + g.update(extra) + return g + + +GAMES = { + "AAA home v BBB": match("1", "2"), + "BBB home v AAA": match("2", "1"), + "CCC v DDD": match("3", "4"), + "Knights (NEW 41) v CCC": match("41", "3"), + "Warriors (NEW 42) v CCC": match("42", "3"), + "AAA v BBB, no ids": {"home_abbr": "AAA", "away_abbr": "BBB"}, + "ids 1 v 2, no abbrs": {"home_id": "1", "away_id": "2"}, + "AAA v BBB, int ids": match("1", "2", home_id=1, away_id=2), + "lower-case abbrs": {"home_abbr": "aaa ", "away_abbr": "bbb"}, + "empty game": {}, +} + +#: label -> favorite_teams as the resolver hands it over. +FAVORITES = { + "none": [], + "AAA": ["AAA"], + "aaa": ["aaa"], + "' AAA '": [" AAA "], + "1": ["1"], + "int 1": [1], + "AAA,CCC": ["AAA", "CCC"], + "NEW": ["NEW"], + "41": ["41"], + "'None'": ["None"], + "blank": ["", " "], +} + +#: (favourites, game) -> answer with the abbreviation key, then the id key. +EXPECTED_IS_FAVORITE = { + "AAA home v BBB": {"AAA": "Y.", "aaa": "Y.", "' AAA '": "Y.", "1": ".Y", "int 1": ".Y", + "AAA,CCC": "Y."}, + "BBB home v AAA": {"AAA": "Y.", "aaa": "Y.", "' AAA '": "Y.", "1": ".Y", "int 1": ".Y", + "AAA,CCC": "Y."}, + "CCC v DDD": {"AAA,CCC": "Y."}, + "Knights (NEW 41) v CCC": {"AAA,CCC": "Y.", "NEW": "Y.", "41": ".Y"}, + "Warriors (NEW 42) v CCC": {"AAA,CCC": "Y.", "NEW": "Y."}, + "AAA v BBB, no ids": {"AAA": "Y.", "aaa": "Y.", "' AAA '": "Y.", "AAA,CCC": "Y."}, + "ids 1 v 2, no abbrs": {"1": ".Y", "int 1": ".Y"}, + "AAA v BBB, int ids": {"AAA": "Y.", "aaa": "Y.", "' AAA '": "Y.", "1": ".Y", + "int 1": ".Y", "AAA,CCC": "Y."}, + "lower-case abbrs": {"AAA": "Y.", "aaa": "Y.", "' AAA '": "Y.", "AAA,CCC": "Y."}, + "empty game": {}, +} + + +@pytest.mark.parametrize("game_label", sorted(GAMES)) +@pytest.mark.parametrize("fav_label", sorted(FAVORITES)) +def test_is_favorite_game(fav_label, game_label): + want = EXPECTED_IS_FAVORITE[game_label].get(fav_label, "..") + got = "".join("Y" if host(FAVORITES[fav_label], by_id)._is_favorite_game(dict(GAMES[game_label])) + else "." for by_id in (False, True)) + assert got == want + + +class TestFavoriteCode: + @pytest.mark.parametrize("value, code", [ + ("bos", "BOS"), (" BOS ", "BOS"), ("BOS", "BOS"), (41, "41"), + ("", None), (" ", None), (None, None), + ]) + def test_normalises(self, value, code): + assert SportsFavoritesMixin._favorite_code(value) == code + + def test_a_missing_id_is_not_the_string_none(self): + """str(None) would match a favourite typed "None"; None matches nothing.""" + h = host(["None"], by_id=True) + assert h._is_favorite_game({"home_abbr": "AAA", "away_abbr": "BBB"}) is False + assert h._is_favorite_game({}) is False + + def test_a_none_favorites_list_matches_nothing(self): + assert host(None)._is_favorite_game(dict(GAMES["AAA home v BBB"])) is False + + +# --------------------------------------------------------------------------- +# Selection. A shuffled slate: two games share id s2, two have no id, s9 has +# no start time. Hours from now; Recent gets the same slate in the past. +# --------------------------------------------------------------------------- + +NOW = datetime(2026, 10, 5, 15, tzinfo=timezone.utc) +SLATE = (("s5", "41", "2", 5), ("s1", "1", "2", 1), ("s3", "4", "3", 3), + ("s2", "3", "1", 2), ("s7", "2", "3", 7), ("s4", "1", "4", 4), + ("s6", "42", "4", 6), ("s2", "1", "4", 8), ("s9", "1", "3", None), + (None, "3", "1", 9), (None, "2", "1", 10)) + + +def slate(recent): + sign = -1 if recent else 1 + games = [] + for gid, home, away, hours in SLATE: + g = match(home, away, id=gid) + if hours is not None: + g["start_time_utc"] = NOW + timedelta(hours=sign * hours) + games.append(g) + return games + + +def pick(favorites, limit, recent, by_id=False): + h = host(favorites, by_id, limit) + method = h._select_recent_games_for_display if recent else h._select_games_for_display + return ",".join(g["id"] or "~" for g in method(slate(recent), favorites)) or "none" + + +ALL = "s1,s2,s3,s4,s5,s6,s7,s2,~,~,s9" + +#: (favourites, per-team limit) -> picked ids, the same for Upcoming and Recent. +EXPECTED_SELECT = { + (("AAA",), 1): "s1", (("AAA",), 2): "s1,s2", (("AAA",), 5): "s1,s2,s4,~,~", + (("aaa",), 5): "s1,s2,s4,~,~", ((" AAA ",), 2): "s1,s2", + (("AAA", "CCC"), 1): "s1,s2", (("AAA", "CCC"), 2): "s1,s2,s3", + (("AAA", "CCC"), 5): "s1,s2,s3,s4,s7,~,~,s9", + (("AAA", "ZZZ"), 2): "s1,s2", (("NEW",), 2): "s5,s6", + (("ZZZ",), 2): "none", ((), 1): ALL, +} + + +@pytest.mark.parametrize("recent", [False, True], ids=["upcoming", "recent"]) +@pytest.mark.parametrize("favorites, limit", sorted(EXPECTED_SELECT)) +def test_select(favorites, limit, recent): + assert pick(list(favorites), limit, recent) == EXPECTED_SELECT[(favorites, limit)] + + +class TestSelectByTeamId: + """The NRL-style host: the key is the team id, so NEW is two teams.""" + + @pytest.mark.parametrize("recent", [False, True]) + def test_one_club_of_a_shared_abbreviation(self, recent): + assert pick(["41"], 2, recent, by_id=True) == "s5" + + @pytest.mark.parametrize("recent", [False, True]) + def test_an_unresolved_abbreviation_matches_nothing(self, recent): + assert pick(["NEW"], 2, recent, by_id=True) == "none" + + def test_ids_select_like_abbreviations(self): + assert pick(["1"], 5, False, by_id=True) == "s1,s2,s4,~,~" + + +class TestSelectionRules: + def test_a_game_between_two_favourites_counts_for_both(self): + h = host(["AAA", "BBB"], limit=1) + picked = h._select_games_for_display(slate(False), ["AAA", "BBB"]) + assert [g["id"] for g in picked] == ["s1"] + + def test_games_without_an_id_are_never_duplicates(self): + games = [match("1", "2"), match("1", "3")] + assert len(host(["AAA"])._select_games_for_display(games, ["AAA"])) == 2 + + def test_a_reused_id_is_a_duplicate(self): + games = [match("1", "2", id="x"), match("1", "3", id="x")] + assert len(host(["AAA"])._select_games_for_display(games, ["AAA"])) == 1 + + def test_upcoming_is_soonest_first_and_recent_newest_first(self): + assert pick(["CCC"], 5, False) == "s2,s3,s7,~,s9" + assert pick(["CCC"], 5, True) == "s2,s3,s7,~,s9" + + def test_the_handed_list_is_used_not_favorite_teams(self): + h = host(["CCC"], limit=1) + assert [g["id"] for g in h._select_games_for_display(slate(False), ["AAA"])] == ["s1"] + + @pytest.mark.parametrize("recent", [False, True]) + def test_the_summary_is_logged_at_info(self, recent, caplog): + with caplog.at_level(logging.INFO, logger=LOG.name): + pick(["AAA"], 1, recent) + name = "_select_recent_games_for_display" if recent else "_select_games_for_display" + assert [r.levelno for r in caplog.records if r.funcName == name + and r.levelno >= logging.INFO] == [logging.INFO] + + +# --------------------------------------------------------------------------- +# Carriers and host contract +# --------------------------------------------------------------------------- + +MIXINS = { + "SportsFavoritesMixin": ["_favorite_code", "_is_favorite_game"], + "SportsUpcomingFavoritesMixin": ["_select_games_for_display"], + "SportsRecentFavoritesMixin": ["_select_recent_games_for_display"], +} + + +def _classes(): + tree = ast.parse(Path(sports_favorites.__file__).read_text(encoding="utf-8")) + return {n.name: n for n in tree.body if isinstance(n, ast.ClassDef)} + + +def _self_reads(cls): + return {node.attr for node in ast.walk(cls) + if isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load) + and isinstance(node.value, ast.Name) and node.value.id == "self"} + + +class TestHostContract: + def test_each_mixin_carries_only_its_class_methods(self): + """So adopting one gives no manager a method it did not have.""" + for name, methods in MIXINS.items(): + mixin = getattr(sports_favorites, name) + assert sorted(n for n in vars(mixin) if not n.startswith("__")) == methods + + def test_every_host_read_is_documented(self): + reads = set().union(*(_self_reads(c) for c in _classes().values())) + undocumented = sorted(n for n in reads if f"``{n}``" not in sports_favorites.__doc__) + assert undocumented == [], f"read but not in the host contract: {undocumented}" + + def test_the_key_comes_from_sports_helpers(self): + """The seam stays where 3.5.0 put it; this module only calls it.""" + assert "_favorite_key" in vars(SportsHelpersMixin) + assert all("_favorite_key" not in vars(getattr(sports_favorites, n)) for n in MIXINS) + + def test_no_other_shared_mixin_defines_these(self): + from src.common import sports_display_rules, sports_shared + others = [sports_shared.SportsCoreSharedMixin, sports_shared.SportsRecentSharedMixin, + sports_shared.SportsLiveSharedMixin, SportsHelpersMixin, + sports_display_rules.SportsGameRulesMixin] + for methods in MIXINS.values(): + for name in methods: + assert not any(name in vars(o) for o in others), name + + def test_the_shared_callers_reach_it(self): + """_favorites_first and the live dwell ask _is_favorite_game; one body answers.""" + from src.common.sports_display_rules import SportsGameRulesMixin + + class Host(SportsGameRulesMixin, SportsFavoritesMixin, SportsHelpersMixin): + favorite_teams = ["aaa"] + game_display_duration = 15 + non_favorite_live_game_duration = 5 + + assert Host()._effective_live_duration(dict(GAMES["AAA home v BBB"])) == 15 + assert Host()._effective_live_duration(dict(GAMES["CCC v DDD"])) == 5 diff --git a/test/test_sports_favorites_parity.py b/test/test_sports_favorites_parity.py new file mode 100644 index 00000000..407b7172 --- /dev/null +++ b/test/test_sports_favorites_parity.py @@ -0,0 +1,135 @@ +"""sports_favorites still matches every plugin copy, and only nrl overrides the key. + +``src.common.sports_favorites`` was copied from the scoreboards once family 6 +had made each method one body in all nine: ``SportsCore._favorite_code`` and +``_is_favorite_game``, ``SportsUpcoming._select_games_for_display`` and +``SportsRecent._select_recent_games_for_display``. The plugins delete their +copies once they floor on the release that ships this module. Until each has, a +copy that changes on its own is a fix one side has and the other lacks. + +Point LEDMATRIX_PLUGINS at a ledmatrix-plugins checkout and each method is +compared with every plugin copy using ``scripts/sports_drift_report.py``'s own +normalisation (the AST with docstrings and annotations dropped), plus the +decorators. A copy that is gone counts as adopted when the plugin's +``sports.py`` names the module. The owner's decision that only nrl overrides +``_favorite_key`` (with the team id) is checked too; that override stays in +the plugin after adoption. Without the variable this skips: core CI has no +plugins checkout. +""" + +import ast +import importlib.util +import os +from pathlib import Path + +import pytest + +from src.common import sports_favorites + +REPO = Path(__file__).resolve().parents[1] +SPORTS = ("afl", "baseball", "basketball", "football", "hockey", "lacrosse", + "nrl", "soccer", "ufc") + +#: plugin class -> (our mixin, the methods it carries) +CARRIERS = { + "SportsCore": ("SportsFavoritesMixin", ("_favorite_code", "_is_favorite_game")), + "SportsUpcoming": ("SportsUpcomingFavoritesMixin", ("_select_games_for_display",)), + "SportsRecent": ("SportsRecentFavoritesMixin", ("_select_recent_games_for_display",)), +} + +#: The owner's decision (docs/SPORTS_UNIFICATION.md, family 6): the sports +#: that name a team by something other than its abbreviation. +OVERRIDES_FAVORITE_KEY = {"nrl"} + + +def _drift_report(): + """scripts/sports_drift_report.py, loaded by path (scripts/ is no package).""" + spec = importlib.util.spec_from_file_location( + "sports_drift_report", REPO / "scripts" / "sports_drift_report.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +DRIFT = _drift_report() + + +def _plugins_root(): + root = DRIFT.resolve_plugins_dir(os.environ.get("LEDMATRIX_PLUGINS")) + if root is None: + pytest.skip("set LEDMATRIX_PLUGINS to a ledmatrix-plugins checkout to " + "compare this module against the plugin copies") + return root + + +def _class(tree, name): + return next(n for n in tree.body if isinstance(n, ast.ClassDef) and n.name == name) + + +def _method(cls, name): + return next((n for n in cls.body + if isinstance(n, ast.FunctionDef) and n.name == name), None) + + +def _fingerprint(node): + return (DRIFT._digest(node, DRIFT._Canonical()), + tuple(ast.unparse(d) for d in node.decorator_list)) + + +def _ours(mixin): + tree = ast.parse(Path(sports_favorites.__file__).read_text(encoding="utf-8")) + return _class(tree, mixin) + + +def _plugin_tree(root, sport): + source = (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8") + return source, ast.parse(source) + + +CASES = [(sport, cls, name) for sport in SPORTS + for cls, (_, names) in CARRIERS.items() for name in names] + + +@pytest.mark.parametrize("sport, cls, name", CASES) +def test_every_remaining_plugin_copy_matches(sport, cls, name): + source, tree = _plugin_tree(_plugins_root(), sport) + copy = _method(_class(tree, cls), name) + if copy is None: + assert sports_favorites.__name__ in source, ( + f"{sport}: no {name} on {cls} and no {sports_favorites.__name__} import") + else: + ours = _method(_ours(CARRIERS[cls][0]), name) + assert _fingerprint(copy) == _fingerprint(ours), ( + f"{cls}.{name} in {sport} differs from sports_favorites. " + f"Port the change to both, or stop treating it as shared.") + + +@pytest.mark.parametrize("sport", SPORTS) +def test_no_other_plugin_class_carries_a_copy(sport): + """A copy on another class (afl's old SportsUpcoming._is_favorite_game) would shadow the shared one.""" + _, tree = _plugin_tree(_plugins_root(), sport) + shared = {name: cls for cls, (_, names) in CARRIERS.items() for name in names} + strays = [f"{node.name}.{name}" for node in tree.body if isinstance(node, ast.ClassDef) + for name, home in shared.items() + if node.name != home and _method(node, name) is not None] + assert strays == [] + + +def test_only_the_decided_sports_override_the_key(): + root = _plugins_root() + overriding = {sport for sport in SPORTS + if any(_method(node, "_favorite_key") is not None + for node in _plugin_tree(root, sport)[1].body + if isinstance(node, ast.ClassDef))} + assert overriding == OVERRIDES_FAVORITE_KEY + + +def test_the_drift_report_still_calls_them_identical(): + root = _plugins_root() + families = DRIFT.build(root, ("sports.py",)) + rows = {(r["file"], r["family"]): r + for r in (DRIFT.summarise(k, v) for k, v in families.items())} + for _, names in CARRIERS.values(): + for name in names: + row = rows.get(("sports.py", name)) + assert row is None or row["worst_class_variants"] == 1, name From e40bc47d28d484e89d7ba49f023c166aeb945ed3 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 12:50:05 -0400 Subject: [PATCH 27/37] chore: prepare the 3.8.2 release (#776) Bumps src.__version__ to 3.8.2 and turns Unreleased into ## 3.8.2: the display's malloc arena cap and between-screen malloc_trim (#774), and src.common.sports_favorites (#775, sports family 6), which the scoreboards adopt by flooring on 3.8.2. src/common/README.md and docs/SPORTS_UNIFICATION.md say 3.8.2 for it; the SPORTS_UNIFICATION module table also said "next release" for the four stage 4 modules, which shipped in 3.8.0. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 10 ++++++++-- docs/SPORTS_UNIFICATION.md | 10 +++++----- src/__init__.py | 2 +- src/common/README.md | 2 +- 4 files changed, 15 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 90e0e87e..7f729e9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,12 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +## 3.8.2 + +The display hands freed memory back to the OS (#774), and sports consolidation +family 6: `src.common.sports_favorites`, which the scoreboards adopt by +flooring on 3.8.2 (#775). + ### The display hands freed memory back to the OS The display process's resident memory climbed in steps for hours while the @@ -52,8 +58,8 @@ and the 20-minute frame soaks came out at 0.147-0.165% late against main's favourites-only picks). Each side of a game is named by the 3.5.0 `_favorite_key` seam and compared with `favorite_teams` stripped and upper-cased; nrl overrides the key with the ESPN team id. Only a game with an - id can be a duplicate. A plugin may inherit the mixins once it floors on the - release that ships this module, and deletes its copies then. + id can be a duplicate. A plugin may inherit the mixins once it floors on + 3.8.2, and deletes its copies then. (#775) ## 3.8.1 diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index f30e8a4c..4363e43d 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -87,12 +87,12 @@ more. Shared sports code lives in `src/common`: | `sports_celebration.py` | 3.7.0 | `SportsCelebrationMixin` — draws the score/win takeover; colour helpers | | `sports_fetch.py` | 3.7.0 | `SportsFetchMixin` — season fetch, live lookback and live-odds decisions | | `sports_card_wrappers.py` | 3.7.0 | `SportsCardWrappersMixin` — the game renderer's `sports_card` delegations | -| `sports_plugin_host.py` | next release | `SportsPluginHostMixin` — the plugin class's (`manager.py`) identical helpers: Vegas weight, off-thread switch refresh | -| `sports_live_scroll.py` | next release | `SportsLiveScrollMixin` — rebuild a live scroll strip mid-cycle, keeping the marquee's place | -| `sports_display_rules.py` | next release | `SportsCardOptionsMixin`, `SportsGameRulesMixin` — scorebug date options, the no-favourites filter, non-favourite live dwell | -| `sports_font_path.py` | next release | `resolve_font_path` — what the plugins' `_resolve_font_path` copies return | +| `sports_plugin_host.py` | 3.8.0 | `SportsPluginHostMixin` — the plugin class's (`manager.py`) identical helpers: Vegas weight, off-thread switch refresh | +| `sports_live_scroll.py` | 3.8.0 | `SportsLiveScrollMixin` — rebuild a live scroll strip mid-cycle, keeping the marquee's place | +| `sports_display_rules.py` | 3.8.0 | `SportsCardOptionsMixin`, `SportsGameRulesMixin` — scorebug date options, the no-favourites filter, non-favourite live dwell | +| `sports_font_path.py` | 3.8.0 | `resolve_font_path` — what the plugins' `_resolve_font_path` copies return | | `sports_game_over.py` | 3.8.1 | `SportsGameOverMixin` — `_is_game_really_over`, with the `FINAL_PERIOD` seam (family 5) | -| `sports_favorites.py` | next release | `SportsFavoritesMixin`, `SportsUpcomingFavoritesMixin`, `SportsRecentFavoritesMixin` — `_is_favorite_game` and the favourites-only picks, on the `_favorite_key` seam (family 6) | +| `sports_favorites.py` | 3.8.2 | `SportsFavoritesMixin`, `SportsUpcomingFavoritesMixin`, `SportsRecentFavoritesMixin` — `_is_favorite_game` and the favourites-only picks, on the `_favorite_key` seam (family 6) | Each is described in [src/common/README.md](../src/common/README.md). diff --git a/src/__init__.py b/src/__init__.py index 9bf9335e..add0e258 100644 --- a/src/__init__.py +++ b/src/__init__.py @@ -4,5 +4,5 @@ LEDMatrix Display System Core source package for the LED Matrix Display project. """ -__version__ = "3.8.1" +__version__ = "3.8.2" diff --git a/src/common/README.md b/src/common/README.md index 432b0fad..66140ede 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -44,7 +44,7 @@ Rules for the package: | [`sports_card_wrappers`](#sports_card_wrappers) | The game renderer's `sports_card` delegations | Yes (scoreboards) | 3.7.0 | | [`sports_celebration`](#sports_celebration) | Draw a scoreboard's score/win celebration | Yes (scoreboards) | 3.7.0 | | [`sports_display_rules`](#sports_display_rules) | Which games a scoreboard shows, for how long, and its scorebug date line | Yes (scoreboards) | 3.8.0 | -| [`sports_favorites`](#sports_favorites) | Which games involve a favourite team, and the favourites-only picks | Yes (scoreboards) | next release | +| [`sports_favorites`](#sports_favorites) | Which games involve a favourite team, and the favourites-only picks | Yes (scoreboards) | 3.8.2 | | [`sports_fetch`](#sports_fetch) | Scoreboard season fetch, lookback and live-odds decisions | Yes (scoreboards) | 3.7.0 | | [`sports_font_path`](#sports_font_path) | Find a scoreboard's bundled font whatever the cwd | Yes (scoreboards) | 3.8.0 | | [`sports_game_over`](#sports_game_over) | Whether a game ESPN still lists as live has ended | Yes (scoreboards) | 3.8.1 | From e4f5e49ff76e8e0b600d23c3ffdfe33c2b415773 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Mon, 5 Oct 2026 17:42:27 -0400 Subject: [PATCH 28/37] test(sports): treat an adopted sports_helpers copy as parity, not missing (#777) * test(sports): treat an adopted sports_helpers copy as parity, not missing The scoreboards deleted their copies of the sports_helpers bodies and constants when they adopted SportsHelpersMixin (ledmatrix-plugins #563/#564), so the 19 parity tests in test/test_sports_helpers.py failed whenever LEDMATRIX_PLUGINS pointed at a plugins checkout. A copy that is gone now counts as adopted when the plugin imports src.common.sports_helpers, as the stage 3/4 and game-over parity tests already do; a copy that remains must still match. Co-Authored-By: Claude Opus 5.5 * test(sports): _adopted checks for a real import via the AST, not a text match Co-Authored-By: Claude Sonnet 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 10 +++++++ test/test_sports_helpers.py | 54 ++++++++++++++++++++++++++++--------- 2 files changed, 52 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f729e9e..dc41ddef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,16 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +### Tooling + +- `test/test_sports_helpers.py`'s parity tests pass again with + `LEDMATRIX_PLUGINS` set. The scoreboards deleted their copies of the + `sports_helpers` bodies and constants when they adopted `SportsHelpersMixin` + (ledmatrix-plugins #563/#564), and the 19 tests still expected them. A copy + that is gone now counts as adopted when the plugin imports + `src.common.sports_helpers`, as the stage 3/4 and game-over parity tests + already do; a copy that remains must still match. + ## 3.8.2 The display hands freed memory back to the OS (#774), and sports consolidation diff --git a/test/test_sports_helpers.py b/test/test_sports_helpers.py index f2ef99a6..1d6fb951 100644 --- a/test/test_sports_helpers.py +++ b/test/test_sports_helpers.py @@ -8,7 +8,9 @@ loses those tests with it. The parity class is what keeps "byte-identical" true after this lands. Point LEDMATRIX_PLUGINS at a ledmatrix-plugins checkout and every promoted body is compared, as a docstring-stripped AST, against every plugin copy that carries -it. Without the variable it skips rather than fails, since core CI has no +it. A copy that is gone counts as adopted when the plugin imports +src.common.sports_helpers (plugins#563/#564 did that for every scoreboard). +Without the variable it skips rather than fails, since core CI has no plugins checkout; ledmatrix-plugins CI runs the same comparison against core (scripts/check_sports_helpers_parity.py, ledmatrix-plugins#495). """ @@ -572,6 +574,24 @@ def _core_definitions(): return out +def _sports_source(root, sport): + return (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8") + + +def _adopted(source): + """Gone is fine once the plugin uses the module; otherwise the finder is + not seeing its copy.""" + name = sports_helpers.__name__ + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.ImportFrom): + if node.module == name or any( + f"{node.module}.{a.name}" == name for a in node.names): + return True + elif isinstance(node, ast.Import) and any(a.name == name for a in node.names): + return True + return False + + class TestParityWithPlugins: @pytest.mark.parametrize("name", sorted(PROMOTED)) def test_body_matches_every_plugin_copy(self, name): @@ -580,11 +600,11 @@ class TestParityWithPlugins: ours = _dump(_core_definitions()[name]) drifted, missing = [], [] for sport in carriers: - defs = _definitions(ast.parse( - (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8"))) - theirs = defs[where].get(plugin_name) + source = _sports_source(root, sport) + theirs = _definitions(ast.parse(source))[where].get(plugin_name) if theirs is None: - missing.append(sport) + if not _adopted(source): + missing.append(sport) elif _dump(theirs) != ours: drifted.append(sport) assert missing == [], f"{plugin_name} no longer in: {missing}" @@ -594,10 +614,20 @@ class TestParityWithPlugins: @pytest.mark.parametrize("sport", SCOREBOARDS) def test_constants_match(self, sport): - root = _plugins_root() - defs = _definitions(ast.parse( - (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8"))) - assert ast.literal_eval(defs["module"]["_MIN_WINDOW_DAYS"].value) == MIN_WINDOW_DAYS - assert ast.literal_eval(defs["module"]["_MAX_WINDOW_DAYS"].value) == MAX_WINDOW_DAYS - gap = defs["SportsCore"]["_DWELL_REENTRY_GAP_SECONDS"].value - assert math.isclose(ast.literal_eval(gap), SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS) + source = _sports_source(_plugins_root(), sport) + defs = _definitions(ast.parse(source)) + expected = { + ("module", "_MIN_WINDOW_DAYS"): MIN_WINDOW_DAYS, + ("module", "_MAX_WINDOW_DAYS"): MAX_WINDOW_DAYS, + ("SportsCore", "_DWELL_REENTRY_GAP_SECONDS"): + SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS, + } + missing = [] + for (where, name), value in expected.items(): + node = defs[where].get(name) + if node is None: + if not _adopted(source): + missing.append(name) + else: + assert math.isclose(ast.literal_eval(node.value), value), name + assert missing == [], f"not found in {sport}: {missing}" From ec117a35a115391bff1b4ea5383765ab59c6df67 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Tue, 6 Oct 2026 08:50:23 -0400 Subject: [PATCH 29/37] fix(install): stop libblockdev matching the desktop-environment check (#780) * fix(install): stop libblockdev matching the desktop-environment check dpkg -l | grep -E '^ii.*kde' searched the description column too, so unrelated packages (libblockdev-*) set DESKTOP_DETECTED on Raspberry Pi OS Lite. Match on the installed package name only, anchored. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VZNWFWprcFYGfuf1JAyrBJ * docs(install): correct why the desktop check matched libblockdev The unanchored .*kde matched the package name mid-word ("bloc-kde-v"), not only description text. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VZNWFWprcFYGfuf1JAyrBJ --------- Co-authored-by: Claude --- first_time_install.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/first_time_install.sh b/first_time_install.sh index 65cd66fc..ea494c12 100755 --- a/first_time_install.sh +++ b/first_time_install.sh @@ -91,7 +91,11 @@ if [ -r "$LM_OS_RELEASE_FILE" ]; then DESKTOP_DETECTED=0 # grep without -q: -q exits at the first match, dpkg then dies of SIGPIPE, # and pipefail turns a found desktop into "not found". - if dpkg -l | grep -E "^ii.*raspberrypi-ui-mods|^ii.*lxde|^ii.*xfce|^ii.*gnome|^ii.*kde" >/dev/null; then + # Match installed package names from their start: the unanchored ".*kde" + # matched mid-word (libblockdev-* = "bloc-kde-v") on Lite, and `dpkg -l` + # lines also carry descriptions that could match. + if dpkg-query -W -f='${db:Status-Abbrev} ${binary:Package}\n' 2>/dev/null \ + | grep -E "^ii +(raspberrypi-ui-mods|lxde|xfce|gnome|kde)" >/dev/null; then DESKTOP_DETECTED=1 fi if systemctl list-units --type=service --state=running 2>/dev/null | grep -qE "lightdm|gdm3|sddm|lxdm"; then From 7c5fa9cfdb57197fba26498755e66cf31c11abea Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Tue, 6 Oct 2026 08:50:37 -0400 Subject: [PATCH 30/37] perf(fetch): cache ESPN scoreboard windows without the parts nothing reads (#749) The sports scoreboards cache their Recent/Upcoming window as the raw ESPN response, and it stays parsed in the memory tier while fresh. Measured on hdpi, most of it is never drawn: per-team stat leaders, athlete cards (featuredAthletes, probables), team and event links, headlines, video highlights and geo broadcasts. None of those keys is read by core or by any plugin in ledmatrix-plugins. BackgroundDataService now drops them from an ESPN /scoreboard response before caching and delivering it (src/common/espn_payload.py), keeping everything else. On the five hdpi windows that is 10.6MB -> 3.0MB of JSON and ~40MB -> ~12MB of parsed objects, and parsing an expired window gets 3-4x cheaper. submit_fetch_request(slim_payload=False) caches a response whole. Claude-Session: https://claude.ai/code/session_01BkfgXMqqwn2w4NN7LRzhxy Co-authored-by: Claude Opus 5.5 --- src/background_data_service.py | 21 ++++- src/common/espn_payload.py | 97 ++++++++++++++++++++ test/test_espn_payload.py | 162 +++++++++++++++++++++++++++++++++ 3 files changed, 279 insertions(+), 1 deletion(-) create mode 100644 src/common/espn_payload.py create mode 100644 test/test_espn_payload.py diff --git a/src/background_data_service.py b/src/background_data_service.py index be860bb9..9a8f0250 100644 --- a/src/background_data_service.py +++ b/src/background_data_service.py @@ -34,6 +34,7 @@ from src.common.fetch_service import ( plugin_scope, share_connection_pool, ) +from src.common.espn_payload import is_espn_scoreboard_url, slim_scoreboard_payload from src.common.espn_dates import ( RANGE_RETRY_SECONDS, _note_range_rejected, @@ -83,6 +84,10 @@ class FetchRequest: # the cache with the callbacks suppressed -- joiners waiting forever for a # fetch that did, in fact, succeed. commit_claimed: bool = False + # Trim an ESPN scoreboard response before it is cached and delivered + # (src/common/espn_payload.py). Set by whoever created the request; a + # submitter that joins the fetch gets the same payload. + slim_payload: bool = True result: Optional[Any] = None error: Optional[str] = None # The plugin that submitted the request, so the fetch service counts the @@ -249,7 +254,8 @@ class BackgroundDataService: timeout: Optional[int] = None, max_retries: int = 3, priority: int = 1, - callback: Optional[Callable] = None) -> str: + callback: Optional[Callable] = None, + slim_payload: bool = True) -> str: """ Submit a background fetch request. @@ -265,6 +271,11 @@ class BackgroundDataService: priority: Accepted for compatibility and ignored; requests run in submission order. callback: Optional callback function when request completes + slim_payload: Drop the parts of an ESPN scoreboard response no + scoreboard reads (stat leaders, athlete cards, links, + headlines, highlights) before caching it; see + src/common/espn_payload.py. Only ESPN /scoreboard URLs are + touched. Pass False to cache the response whole. Returns: Request ID for tracking the fetch operation @@ -336,6 +347,7 @@ class BackgroundDataService: priority=priority, callback=callback, owner=owner, + slim_payload=slim_payload, ) with self._lock: @@ -497,6 +509,13 @@ class BackgroundDataService: ) return result + # Most of an ESPN scoreboard response is never drawn, and the + # cached copy stays parsed in the memory tier while it is fresh. + # Trimmed before the write so the cache, request.result and the + # callbacks all see the same payload. See src/common/espn_payload.py. + if request.slim_payload and is_espn_scoreboard_url(request.url): + slim_scoreboard_payload(data) + # Cache the data self.cache_manager.set(request.cache_key, data) diff --git a/src/common/espn_payload.py b/src/common/espn_payload.py new file mode 100644 index 00000000..00ff118c --- /dev/null +++ b/src/common/espn_payload.py @@ -0,0 +1,97 @@ +"""Drop the parts of an ESPN scoreboard payload no scoreboard reads. + +The sports scoreboards cache their Recent/Upcoming window (14 days back, 7 +ahead) as the raw ESPN response, and that record stays parsed in the memory +cache for as long as it is fresh. Most of it is never drawn. Measured on hdpi +(2026-10-02) the MLB window was 3.35MB of JSON and 13.5MB of Python objects, +and the five windows together ~40MB, mostly in: + +* ``competitors[].leaders`` / ``competitions[].leaders`` -- per-team and + per-game stat leaders (28% of the MLB window) +* ``competitors[].team.links`` / ``event.links`` -- web and app URLs +* ``status.featuredAthletes`` and ``competitors[].probables`` -- athlete + cards with headshots and season stats +* ``competitions[].headlines`` / ``highlights`` -- article and video blurbs + (28% of the college-football window) +* ``competitions[].geoBroadcasts`` + +None of those keys is read by core or by any plugin in ledmatrix-plugins +(checked 2026-10-02 across every scoreboard, the odds ticker and the +leaderboard), while everything that is read -- odds, records, linescores, +situation, statistics, notes, broadcasts, venue -- is kept. Dropping them +takes the five windows from ~40MB to ~12MB of parsed objects and the files from +10.6MB to 3.0MB, so the reads that parse an expired window on the render +thread get 3-4x cheaper too. + +:func:`slim_scoreboard_payload` changes the payload in place, and only ever +removes the keys listed here: anything it does not know about is left alone. +""" + +from typing import Any, Dict +from urllib.parse import urlsplit + +# Per level of the payload, the keys removed. Kept deliberately explicit: +# adding a key here means checking that nothing reads it first. +_EVENT_DROP = ("links",) +_COMPETITION_DROP = ("leaders", "headlines", "highlights", "geoBroadcasts") +_STATUS_DROP = ("featuredAthletes",) +_COMPETITOR_DROP = ("leaders", "probables") +_TEAM_DROP = ("links",) + + +def is_espn_scoreboard_url(url: Any) -> bool: + """Whether ``url`` is an ESPN site-API scoreboard endpoint.""" + if not isinstance(url, str): + return False + try: + parts = urlsplit(url) + except ValueError: + return False + host = (parts.hostname or "").lower() + if host != "espn.com" and not host.endswith(".espn.com"): + return False + return parts.path.rstrip("/").endswith("/scoreboard") + + +def _drop(obj: Any, keys) -> None: + if isinstance(obj, dict): + for key in keys: + obj.pop(key, None) + + +def slim_scoreboard_payload(payload: Any) -> Any: + """Remove the unread parts of an ESPN scoreboard payload, in place. + + Returns ``payload`` for convenience. Anything that is not shaped like a + scoreboard (not a dict, no ``events`` list, odd entries) is passed over + untouched rather than raising. + """ + if not isinstance(payload, dict): + return payload + events = payload.get("events") + if not isinstance(events, list): + return payload + for event in events: + if not isinstance(event, dict): + continue + _drop(event, _EVENT_DROP) + competitions = event.get("competitions") + if not isinstance(competitions, list): + continue + for competition in competitions: + if not isinstance(competition, dict): + continue + _drop(competition, _COMPETITION_DROP) + _drop(competition.get("status"), _STATUS_DROP) + competitors = competition.get("competitors") + if not isinstance(competitors, list): + continue + for competitor in competitors: + if not isinstance(competitor, dict): + continue + _drop(competitor, _COMPETITOR_DROP) + _drop(competitor.get("team"), _TEAM_DROP) + return payload + + +__all__ = ["is_espn_scoreboard_url", "slim_scoreboard_payload"] diff --git a/test/test_espn_payload.py b/test/test_espn_payload.py new file mode 100644 index 00000000..1605c586 --- /dev/null +++ b/test/test_espn_payload.py @@ -0,0 +1,162 @@ +"""Tests for src/common/espn_payload.py and its use by BackgroundDataService.""" + +import copy +import time +from unittest.mock import MagicMock, Mock, patch + +import pytest + +from src.background_data_service import BackgroundDataService, shutdown_background_service +from src.common.espn_payload import is_espn_scoreboard_url, slim_scoreboard_payload + +SCOREBOARD = "https://site.api.espn.com/apis/site/v2/sports/baseball/mlb/scoreboard" + + +def _event(): + """One event carrying every key the slimming drops and a sample of the + keys scoreboards read, at the depth ESPN puts them.""" + competitor = { + "id": "10", + "homeAway": "home", + "score": "5", + "team": {"abbreviation": "NYY", "logo": "https://a/l.png", + "links": [{"href": "https://espn.com/team"}]}, + "records": [{"summary": "90-60"}], + "linescores": [{"value": 1}], + "statistics": [{"name": "hits", "displayValue": "9"}], + "leaders": [{"name": "avg", "leaders": [{"athlete": {"id": "1"}}]}], + "probables": [{"athlete": {"id": "2"}, "statistics": []}], + } + return { + "id": "401", + "date": "2026-10-01T23:05Z", + "links": [{"href": "https://espn.com/game"}], + "status": {"type": {"state": "post"}}, + "competitions": [{ + "status": {"type": {"state": "post", "shortDetail": "Final"}, + "featuredAthletes": [{"athlete": {"id": "3"}}]}, + "competitors": [competitor, dict(copy.deepcopy(competitor), homeAway="away")], + "odds": [{"details": "NYY -150", "overUnder": 8.5}], + "situation": {"outs": 2}, + "notes": [{"headline": "Game 1"}], + "broadcasts": [{"names": ["FOX"]}], + "venue": {"fullName": "Yankee Stadium"}, + "leaders": [{"name": "hits"}], + "headlines": [{"description": "recap"}], + "highlights": [{"links": {"source": {}}}], + "geoBroadcasts": [{"media": {"shortName": "FOX"}}], + }], + } + + +class TestSlimScoreboardPayload: + def test_drops_exactly_the_listed_keys(self): + payload = {"leagues": [{"id": "10"}], "events": [_event()]} + slim_scoreboard_payload(payload) + event = payload["events"][0] + competition = event["competitions"][0] + assert "links" not in event + for key in ("leaders", "headlines", "highlights", "geoBroadcasts"): + assert key not in competition + assert "featuredAthletes" not in competition["status"] + for competitor in competition["competitors"]: + assert "leaders" not in competitor + assert "probables" not in competitor + assert "links" not in competitor["team"] + + def test_keeps_everything_else_unchanged(self): + """Removing the dropped keys from the original by hand gives exactly + the slimmed payload: nothing else moved, changed or went missing.""" + original = {"leagues": [{"id": "10"}], "events": [_event(), _event()]} + expected = copy.deepcopy(original) + for event in expected["events"]: + del event["links"] + competition = event["competitions"][0] + for key in ("leaders", "headlines", "highlights", "geoBroadcasts"): + del competition[key] + del competition["status"]["featuredAthletes"] + for competitor in competition["competitors"]: + del competitor["leaders"], competitor["probables"] + del competitor["team"]["links"] + assert slim_scoreboard_payload(original) == expected + + def test_in_place_and_returns_payload(self): + payload = {"events": [_event()]} + assert slim_scoreboard_payload(payload) is payload + + @pytest.mark.parametrize("payload", [ + None, [], "x", {}, {"events": None}, {"events": "x"}, + {"events": [None, 1, "x", {"competitions": None}]}, + {"events": [{"competitions": [None, {"status": None, "competitors": None}]}]}, + {"events": [{"competitions": [{"competitors": [None, {"team": None}]}]}]}, + ]) + def test_odd_shapes_pass_through(self, payload): + before = copy.deepcopy(payload) + assert slim_scoreboard_payload(payload) == before + + +class TestIsEspnScoreboardUrl: + @pytest.mark.parametrize("url", [ + SCOREBOARD, + SCOREBOARD + "/", + "http://site.api.espn.com/apis/site/v2/sports/football/college-football/scoreboard", + ]) + def test_scoreboards(self, url): + assert is_espn_scoreboard_url(url) + + @pytest.mark.parametrize("url", [ + None, "", 12, + "https://site.api.espn.com/apis/site/v2/sports/baseball/mlb/teams", + "https://site.api.espn.com/apis/site/v2/sports/football/nfl/summary", + "https://example.com/scoreboard", + "https://espn.com.evil.example/apis/x/scoreboard", + "https://notespn.com/apis/x/scoreboard", + ]) + def test_not_scoreboards(self, url): + assert not is_espn_scoreboard_url(url) + + +@pytest.fixture +def service(): + shutdown_background_service() + cache = MagicMock() + cache.get.return_value = None + svc = BackgroundDataService(cache, max_workers=1, request_timeout=5) + yield svc + svc.shutdown(wait=False) + shutdown_background_service() + + +def _run(service, url, **kwargs): + response = Mock(status_code=200) + response.json.return_value = {"events": [_event()]} + response.raise_for_status.return_value = None + delivered = [] + with patch.object(service.session, "get", return_value=response): + req_id = service.submit_fetch_request( + sport="mlb", year=2026, url=url, cache_key="mlb_schedule_window_14_7", + callback=lambda result: delivered.append(result.data), **kwargs) + deadline = time.time() + 5 + while not service.is_request_complete(req_id) and time.time() < deadline: + time.sleep(0.02) + cached = service.cache_manager.set.call_args[0][1] + return cached, delivered + + +class TestBackgroundServiceSlims: + def test_espn_scoreboard_is_cached_and_delivered_slimmed(self, service): + cached, delivered = _run(service, SCOREBOARD) + competition = cached["events"][0]["competitions"][0] + assert "leaders" not in competition + assert "probables" not in competition["competitors"][0] + assert competition["odds"] and competition["situation"] + # The callback sees the very payload that was cached. + assert delivered and delivered[0] is cached + + def test_opt_out_caches_whole_response(self, service): + cached, _ = _run(service, SCOREBOARD, slim_payload=False) + assert cached == {"events": [_event()]} + + def test_other_urls_untouched(self, service): + cached, _ = _run(service, "https://example.com/feed") + assert cached == {"events": [_event()]} From cb06124b42c21acc339994273e0b708f2d2a755a Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Tue, 6 Oct 2026 08:50:48 -0400 Subject: [PATCH 31/37] fix: a Vegas static pause survives a non-numeric display duration; aliased store installs ask for a restart (#753) * fix(vegas): a display duration that is not a number no longer cancels a static pause The Vegas static pause compared plugin.get_display_duration() with the clock. clock-simple, calendar and countdown return their display_duration setting straight from config.json, so a value saved as "20" or null reached that comparison as a string or None. The TypeError went to the pause's broad except, which ended the pause: the plugin flashed up and the scroll went straight on, at every one of its turns. inf held the pause until something interrupted it, and NaN, False, 0 or a negative number ended it at once. The pause now reads the duration the way the rotation has since #739, with the same helper, then the rotation's fallbacks: 30 s for anything that is not a number or a get_display_duration() that raises, 15 s for a number at or below zero. Logged once per plugin. test_vegas_static_mode.py's pauses used 0 to mean "no wait"; they now use 0.01. The helper moves from display_controller (_finite_seconds) to base_plugin (finite_seconds), unchanged: the coordinator cannot import from display_controller, which imports src.vegas_mode at module level, and a new src module would turn ledmatrix-plugins' min-core table check red until it was listed. base_plugin is already loaded whenever either one is. Tests: test/test_vegas_static_pause_duration.py, on a fake clock, including TestSameAsTheRotation, which runs every value through both the pause and the rotation's _get_display_duration/_resolve_durations. Co-Authored-By: Claude Opus 5.5 * fix(web): a store install asks for a restart by the id it installed as POST /plugins/install decides restart_required from whether config.json already enables the plugin: the display loads a plugin when its enabled flag changes, so one already enabled (a reinstall, or a config carried over) keeps running the copy it loaded until a restart. The route read that flag under the registry id. Weather, Music, Stocks and Leaderboard install under the id their manifests declare (weather -> ledmatrix-weather), which is the config section's id, so reinstalling an enabled one never reported that a restart was needed. Both the queued and the direct path now look up the installed id once (#746's _installed_plugin_id) and use it for the plugin_id they answer with and for the enabled check. Tests: test/test_api_v3_install_restart_installed_id.py, through the Flask test client, both paths. Co-Authored-By: Claude Opus 5.5 * fix(vegas): type the static pause fallback on its own (mypy ratchet) _static_pause_duration assigned the fallback to `seconds`, which the except branch typed as float before finite_seconds() reassigned it as float | None. A separate `fallback` keeps both types exact; behaviour is unchanged. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 19 ++ src/display_controller.py | 17 +- src/plugin_system/base_plugin.py | 21 ++ src/vegas_mode/coordinator.py | 52 ++++- ...est_api_v3_install_restart_installed_id.py | 123 +++++++++++ test/test_vegas_static_mode.py | 3 +- test/test_vegas_static_pause_duration.py | 197 ++++++++++++++++++ .../blueprints/api_v3/plugin_store.py | 13 +- 8 files changed, 422 insertions(+), 23 deletions(-) create mode 100644 test/test_api_v3_install_restart_installed_id.py create mode 100644 test/test_vegas_static_pause_duration.py diff --git a/CHANGELOG.md b/CHANGELOG.md index dc41ddef..eed96cd3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1284,6 +1284,25 @@ policies are unchanged. a runtime publisher that stops still goes `stale`, and a subscription that goes quiet still falls back to the cache. The cache path's 120 s rule is unchanged. +- A plugin that pauses the Vegas scroll gets its pause when its display + duration is not a plain number. Several plugins (clock-simple, calendar, + countdown) return `display_duration` as it is in config.json, so a value + saved as `"20"` or `null` (the raw config editor, a hand edit) reached the + pause as a string or None; comparing it with the clock raised, and the + plugin flashed up and the scroll went straight on, at every one of its + turns. `inf` held the pause until something interrupted it, and 0, a + negative number or NaN ended it at once. The pause now reads the duration + as the rotation does (`finite_seconds()` in `base_plugin`): a numeric + string counts, anything else that is not a finite number (or a + `get_display_duration()` that raises) pauses for 30 s, and a number at or + below zero for 15 s, with one warning per plugin. +- Reinstalling Weather, Music, Stocks or Leaderboard from the Plugin Store + while it is enabled asks for a display restart, as reinstalling any other + enabled plugin does. `POST /api/v3/plugins/install` looked for the + plugin's `enabled` flag under the store id (`weather`), but its config + section is under the id its manifest declares (`ledmatrix-weather`), so + `restart_required` was always false and the display kept running the + copy it had loaded. The check now uses the installed id. ### Scrolling diff --git a/src/display_controller.py b/src/display_controller.py index 44a70f84..de61f2aa 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -25,7 +25,6 @@ import os import inspect import signal import json -import math import threading import types from collections import deque @@ -64,6 +63,7 @@ from src.ipc.contract import ( PluginReloadResult, ) from src.ipc.server import ControlServer, QueuedCommand, StateHub, start_control_server +from src.plugin_system.base_plugin import finite_seconds from src.vegas_mode.render_pipeline import SYNC_SEND_INTERVAL # Get logger with consistent configuration @@ -101,19 +101,6 @@ _MIN_INITIAL_UPDATE_TIMEOUT_SECONDS = 2.0 DEFAULT_DYNAMIC_DURATION_CAP = 180.0 -def _finite_seconds(value: Any) -> Optional[float]: - """``value`` as seconds when it is a finite number or a numeric string, - else None. A bool is not a number here, though it is an int: True would - read as a one-second screen.""" - if isinstance(value, bool): - return None - try: - seconds = float(value) - except (TypeError, ValueError, OverflowError): - return None - return seconds if math.isfinite(seconds) else None - - class _PluginReloadJob: """A ``plugin.reload`` whose slow half runs off the render thread. @@ -1569,7 +1556,7 @@ class DisplayController: except Exception as err: # pylint: disable=broad-except problem = f"get_display_duration() raised {type(err).__name__}: {err}" else: - seconds = _finite_seconds(value) + seconds = finite_seconds(value) if seconds is not None: return seconds problem = f"display duration {value!r} is not a number" diff --git a/src/plugin_system/base_plugin.py b/src/plugin_system/base_plugin.py index 2b676eb1..9acdaecf 100644 --- a/src/plugin_system/base_plugin.py +++ b/src/plugin_system/base_plugin.py @@ -11,6 +11,7 @@ Stability: Stable - maintains backward compatibility from abc import ABC, abstractmethod from enum import Enum from typing import Dict, Any, Optional, List +import math import os import sys from src.deprecation import deprecated, warn_deprecated @@ -240,6 +241,26 @@ def resolve_vegas_participation(plugin: Any, plugin_id: Optional[str] = None) -> return legacy_vegas_participation(plugin) +def finite_seconds(value: Any) -> Optional[float]: + """``value`` as seconds when it is a finite number or a numeric string, + else None. A bool is not a number here, though it is an int: True would + read as a one-second screen. + + How the core reads a plugin's get_display_duration() -- the rotation + (DisplayController._get_display_duration) and the Vegas static pause -- + which several plugins answer straight from config.json, so a value saved + as "20" or null arrives as a string or None. A number at or below zero is + returned as it is; each caller has its own rule for that. + """ + if isinstance(value, bool): + return None + try: + seconds = float(value) + except (TypeError, ValueError, OverflowError): + return None + return seconds if math.isfinite(seconds) else None + + class BasePlugin(ABC): """ Base class that all plugins must inherit from. diff --git a/src/vegas_mode/coordinator.py b/src/vegas_mode/coordinator.py index 8eb0f0ab..8c296c60 100644 --- a/src/vegas_mode/coordinator.py +++ b/src/vegas_mode/coordinator.py @@ -18,10 +18,11 @@ import math import sys import time import threading -from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING +from typing import Optional, Dict, Any, FrozenSet, List, Callable, TYPE_CHECKING from src import display_watchdog from src.common import render_gate +from src.plugin_system.base_plugin import finite_seconds from src.vegas_mode.config import VegasModeConfig from src.vegas_mode.elements import LiveEpochs from src.vegas_mode.plugin_adapter import PluginAdapter @@ -53,6 +54,14 @@ _FPS_HEARTBEAT_INTERVAL = 300.0 #: every plugin. Game state doesn't change within a quarter second. _LIVE_PRIORITY_CHECK_INTERVAL = 0.25 +#: Seconds a static pause shows a plugin whose display duration can't be +#: used, as long as the rotation shows it: 30 when get_display_duration() +#: raises or answers something that is not a number +#: (DisplayController._get_display_duration), 15 when it answers a number at +#: or below zero (DisplayController._resolve_durations). +_UNREADABLE_DURATION = 30.0 +_NOT_POSITIVE_DURATION = 15.0 + def _percentile(ordered: List[float], fraction: float) -> float: """Nearest-rank percentile of an already-sorted list. @@ -92,6 +101,9 @@ class VegasModeCoordinator: _live_reason: Optional[str] = None # Set only while Vegas has changed the GIL switch interval; read with getattr. _saved_switch_interval: Optional[float] + #: Plugins already warned about a display duration the pause can't use, + #: so a bad setting logs once, not at every turn. Replaced, not mutated. + _duration_warned: FrozenSet[str] = frozenset() def __init__( self, @@ -1010,7 +1022,7 @@ class VegasModeCoordinator: # Wait for the plugin's display duration. Monotonic, like the # iteration clock: an NTP step on an RTC-less Pi would otherwise # end the pause at once or stretch it by the correction. - duration = plugin.get_display_duration() + duration = self._static_pause_duration(plugin) start = time.monotonic() while time.monotonic() - start < duration: @@ -1046,6 +1058,42 @@ class VegasModeCoordinator: return True + def _static_pause_duration(self, plugin: 'BasePlugin') -> float: + """Seconds a static pause shows ``plugin``: its display duration, + read the way the rotation reads it. + + Several plugins return their display_duration setting straight from + config.json, so one saved as "20" or null came back as a string or + None; comparing it with the clock raised, and the pause's broad + except ended the pause at every one of the plugin's turns. inf + paused until something interrupted it, and NaN, False, 0 or a + negative number ended the pause at once. A numeric string counts + (finite_seconds); anything else, or a raise, gets + _UNREADABLE_DURATION, and a number at or below zero + _NOT_POSITIVE_DURATION, logged once per plugin. + """ + try: + value = plugin.get_display_duration() + except Exception as err: # pylint: disable=broad-except + problem = f"get_display_duration() raised {type(err).__name__}: {err}" + fallback = _UNREADABLE_DURATION + else: + seconds = finite_seconds(value) + if seconds is not None and seconds > 0: + return seconds + if seconds is None: + problem = f"display duration {value!r} is not a number" + fallback = _UNREADABLE_DURATION + else: + problem = f"display duration {value!r} is not above zero" + fallback = _NOT_POSITIVE_DURATION + plugin_id = plugin.plugin_id + if plugin_id not in self._duration_warned: + self._duration_warned = self._duration_warned | {plugin_id} + logger.warning("[%s] %s; its static pause lasts %.0fs (logged once)", + plugin_id, problem, fallback) + return fallback + def _end_static_pause(self) -> None: """End static pause and restore scroll state.""" should_resume_scrolling = False diff --git a/test/test_api_v3_install_restart_installed_id.py b/test/test_api_v3_install_restart_installed_id.py new file mode 100644 index 00000000..d3508063 --- /dev/null +++ b/test/test_api_v3_install_restart_installed_id.py @@ -0,0 +1,123 @@ +"""POST /plugins/install asks for a restart by the id the plugin installed as. + +A store install needs a display restart when config.json already enables the +plugin (a reinstall, or a config carried over): the display loads a plugin +when its ``enabled`` flag changes, and this flag did not. The route read the +flag under the registry id it was given. An aliased entry installs under +another id -- ``weather`` installs a directory whose manifest declares +``ledmatrix-weather``, and its config section is ``ledmatrix-weather`` -- so +reinstalling an enabled Weather never reported that a restart was needed, +and the display kept running the old copy. +""" + +import json +from unittest.mock import MagicMock + +import pytest + +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401 + +INSTALL = "/api/v3/plugins/install" + + +@pytest.fixture +def store(api_v3_module, tmp_path): + """The store installs registry entry ``weather`` as ``installed_id``.""" + manager = api_v3_module.api_v3.plugin_store_manager + manager.install_plugin.return_value = True + manager.get_registry_info.return_value = None + manager._find_plugin_path.return_value = None + + def installs_as(installed_id): + path = tmp_path / installed_id + path.mkdir() + (path / "manifest.json").write_text(json.dumps({"id": installed_id}), + encoding="utf-8") + manager._find_plugin_path.side_effect = ( + lambda pid: path if pid == "weather" else None) + + manager.installs_as = installs_as + return manager + + +@pytest.fixture +def config(api_v3_module): + """config.json with an ``enabled`` flag for each plugin id given.""" + def sections(enabled): + api_v3_module.api_v3.config_manager.load_config.return_value = { + plugin_id: {"enabled": flag} for plugin_id, flag in enabled.items()} + return sections + + +@pytest.fixture +def queued(api_v3_module): + queue = MagicMock() + + def enqueue(operation_type, plugin_id, operation_callback=None): + queue.callback_result = operation_callback(MagicMock()) + return "op-1" + + queue.enqueue_operation.side_effect = enqueue + api_v3_module.api_v3.operation_queue = queue + return queue + + +def _direct(client): + return client.post(INSTALL, json={"plugin_id": "weather"}).get_json() + + +def _queued(client, queue): + client.post(INSTALL, json={"plugin_id": "weather"}) + return queue.callback_result + + +class TestDirectInstall: + def test_an_aliased_install_enabled_under_its_installed_id_asks_for_a_restart( + self, api_v3_client, store, config): + store.installs_as("ledmatrix-weather") + config({"ledmatrix-weather": True}) + body = _direct(api_v3_client) + assert body["status"] == "success" + assert body["restart_required"] is True + assert body["restart_message"] + + def test_an_enabled_section_under_the_registry_id_alone_does_not( + self, api_v3_client, store, config): + """The display knows the plugin as ledmatrix-weather; nothing runs + under a section called weather.""" + store.installs_as("ledmatrix-weather") + config({"weather": True}) + assert _direct(api_v3_client)["restart_required"] is False + + def test_an_aliased_install_that_is_not_enabled_needs_no_restart( + self, api_v3_client, store, config): + store.installs_as("ledmatrix-weather") + config({"ledmatrix-weather": False}) + assert _direct(api_v3_client)["restart_required"] is False + + def test_an_install_under_its_own_id_is_unchanged(self, api_v3_client, store, config): + store.installs_as("weather") + config({"weather": True}) + assert _direct(api_v3_client)["restart_required"] is True + + def test_an_install_that_cannot_be_found_uses_the_requested_id( + self, api_v3_client, store, config): + config({"weather": True}) + assert _direct(api_v3_client)["restart_required"] is True + + +class TestQueuedInstall: + def test_an_aliased_install_enabled_under_its_installed_id_asks_for_a_restart( + self, api_v3_client, store, config, queued): + store.installs_as("ledmatrix-weather") + config({"ledmatrix-weather": True}) + result = _queued(api_v3_client, queued) + assert result["success"] is True + assert result["restart_required"] is True + assert result["restart_message"] + + def test_an_enabled_section_under_the_registry_id_alone_does_not( + self, api_v3_client, store, config, queued): + store.installs_as("ledmatrix-weather") + config({"weather": True}) + assert _queued(api_v3_client, queued)["restart_required"] is False diff --git a/test/test_vegas_static_mode.py b/test/test_vegas_static_mode.py index 2d3f6679..3bfe1962 100644 --- a/test/test_vegas_static_mode.py +++ b/test/test_vegas_static_mode.py @@ -219,7 +219,8 @@ class TestCoordinatorStaticPause: def _plugin(self): plugin = MagicMock() plugin.plugin_id = 'clock' - plugin.get_display_duration.return_value = 0 + # A moment: zero would pause 15 s, as the rotation shows it. + plugin.get_display_duration.return_value = 0.01 return plugin def test_trigger_comes_from_the_pipeline(self): diff --git a/test/test_vegas_static_pause_duration.py b/test/test_vegas_static_pause_duration.py new file mode 100644 index 00000000..957e45f7 --- /dev/null +++ b/test/test_vegas_static_pause_duration.py @@ -0,0 +1,197 @@ +"""A Vegas static pause lasts as long as the rotation shows the plugin. + +The pause asked the plugin for get_display_duration() and compared the +answer with the clock. Several plugins (clock-simple, calendar, countdown) +return their display_duration setting as it is in config.json, so one saved +as "20" or null -- the raw config editor, a hand edit -- reached that +comparison as a string or None. The TypeError went to the pause's broad +except, which ended the pause: the plugin flashed up and the scroll went on, +at every one of its turns. inf paused until something interrupted it, and +NaN, False, 0 or a negative number ended the pause at once. + +The pause now reads the answer the way the rotation does since #739, with +the same helper (base_plugin.finite_seconds): a numeric string counts; +anything else that is not a finite number, or a raise, gets the rotation's +30 s; a number at or below zero gets its 15 s. +""" + +import logging +import os +import threading +from types import SimpleNamespace +from unittest.mock import MagicMock + +os.environ.setdefault("EMULATOR", "true") + +import pytest + +from src.vegas_mode import coordinator + +NOT_NUMBERS = [None, '', 'twenty', True, False, float('nan'), float('inf'), + 'inf', '1e400', [20], {'seconds': 20}] +NOT_ABOVE_ZERO = [0, -5, '-5', '0'] +NUMBERS = [('20', 20.0), (' 7.5 ', 7.5), (12, 12.0), (12.5, 12.5)] + + +class FakeClock: + """time.monotonic/time.sleep for the pause loop: sleeping moves the clock.""" + + #: A pause still going after this long never ends (inf did that). + LIMIT = 3600.0 + + def __init__(self): + self.now = 0.0 + + def monotonic(self): + return self.now + + def sleep(self, seconds): + self.now += seconds + if self.now > self.LIMIT: + raise RuntimeError("the static pause never ended") + + +@pytest.fixture +def clock(monkeypatch): + fake = FakeClock() + monkeypatch.setattr(coordinator, 'time', fake) + return fake + + +def _plugin(duration, plugin_id='clock-simple'): + plugin = MagicMock() + plugin.plugin_id = plugin_id + plugin.get_display_duration.return_value = duration + return plugin + + +def _coord(*plugins): + coord = coordinator.VegasModeCoordinator.__new__(coordinator.VegasModeCoordinator) + coord.render_pipeline = MagicMock() + coord.render_pipeline.get_scroll_position.return_value = 0 + coord.display_manager = MagicMock() + locks = {plugin.plugin_id: threading.Lock() for plugin in plugins} + coord.plugin_manager = SimpleNamespace(get_plugin_lock=locks.__getitem__) + coord._state_lock = threading.Lock() + coord._static_pause_active = False + coord._saved_scroll_position = None + coord._should_stop = False + coord._live_priority_active = False + coord._live_priority_check = None + coord._interrupt_check = None + coord.stats = {'static_pauses': 0} + return coord + + +def _pause(coord, plugin, clock): + """One static pause: (whether it completed, how long it lasted).""" + start = clock.now + completed = coord._handle_static_pause(plugin) + return completed, clock.now - start + + +class TestPauseLength: + @pytest.mark.parametrize('value, seconds', NUMBERS) + def test_numbers_and_numeric_strings_are_used(self, clock, value, seconds): + plugin = _plugin(value) + completed, lasted = _pause(_coord(plugin), plugin, clock) + assert completed is True + assert lasted == pytest.approx(seconds, abs=0.15) + + @pytest.mark.parametrize('value', NOT_NUMBERS, ids=repr) + def test_anything_but_a_finite_number_pauses_for_30s(self, clock, value): + plugin = _plugin(value) + completed, lasted = _pause(_coord(plugin), plugin, clock) + assert completed is True + assert lasted == pytest.approx(30.0, abs=0.15) + plugin.display.assert_called_once_with(force_clear=True) + + @pytest.mark.parametrize('value', NOT_ABOVE_ZERO, ids=repr) + def test_a_number_not_above_zero_pauses_for_15s(self, clock, value): + plugin = _plugin(value) + completed, lasted = _pause(_coord(plugin), plugin, clock) + assert completed is True + assert lasted == pytest.approx(15.0, abs=0.15) + + def test_a_raising_get_display_duration_pauses_for_30s(self, clock): + plugin = _plugin(None) + plugin.get_display_duration.side_effect = KeyError('display_duration') + completed, lasted = _pause(_coord(plugin), plugin, clock) + assert completed is True + assert lasted == pytest.approx(30.0, abs=0.15) + + def test_a_good_value_after_a_bad_one_is_used(self, clock): + plugin = _plugin(None) + coord = _coord(plugin) + assert _pause(coord, plugin, clock)[1] == pytest.approx(30.0, abs=0.15) + plugin.get_display_duration.return_value = 45 + assert _pause(coord, plugin, clock)[1] == pytest.approx(45.0, abs=0.15) + + def test_the_pause_can_still_be_interrupted(self, clock): + plugin = _plugin('twenty') + coord = _coord(plugin) + coord._interrupt_check = lambda: clock.now >= 5 + completed, lasted = _pause(coord, plugin, clock) + assert completed is False + assert lasted == pytest.approx(5.0, abs=0.15) + + +class TestWarning: + def test_logged_once_per_plugin(self, clock, caplog): + clock_plugin = _plugin('twenty') + calendar = _plugin(None, plugin_id='calendar') + coord = _coord(clock_plugin, calendar) + with caplog.at_level(logging.WARNING, logger='src.vegas_mode.coordinator'): + for _ in range(3): + for plugin in (clock_plugin, calendar): + coord._handle_static_pause(plugin) + warnings = [r.getMessage() for r in caplog.records + if 'display duration' in r.getMessage()] + assert len(warnings) == 2 + assert any('clock-simple' in m and "'twenty'" in m for m in warnings) + assert any('calendar' in m and 'None' in m for m in warnings) + + +class TestFiniteSeconds: + """The shared rule: what counts as a number of seconds.""" + + @pytest.mark.parametrize('value, seconds', NUMBERS + [(0, 0.0), ('-5', -5.0)]) + def test_numbers_and_numeric_strings(self, value, seconds): + from src.plugin_system.base_plugin import finite_seconds + result = finite_seconds(value) + assert result == seconds and isinstance(result, float) + + @pytest.mark.parametrize('value', NOT_NUMBERS + [pytest.param(10 ** 400, id='10**400')], + ids=repr) + def test_anything_else_is_none(self, value): + from src.plugin_system.base_plugin import finite_seconds + assert finite_seconds(value) is None + + +def _rotation_seconds(plugin): + """How long the rotation shows ``plugin`` (no dynamic duration, no + Rotation & Durations override): the two calls run() makes for a screen. + """ + from src.display_controller import DisplayController + dc = object.__new__(DisplayController) + dc.config = {} + dc.plugin_modes = {'mode': plugin} + return dc._resolve_durations(plugin, 'mode', dc._get_display_duration('mode'), False)[1] + + +class TestSameAsTheRotation: + """The pause and the rotation share finite_seconds; this pins their + fallbacks (30 s, 15 s) to each other too.""" + + @pytest.mark.parametrize('value', [value for value, _ in NUMBERS] + + NOT_NUMBERS + NOT_ABOVE_ZERO, ids=repr) + def test_the_pause_lasts_as_long_as_the_rotation_shows_it(self, clock, value): + plugin = _plugin(value) + expected = _rotation_seconds(plugin) + assert _pause(_coord(plugin), plugin, clock)[1] == pytest.approx(expected, abs=0.15) + + def test_a_raise_too(self, clock): + plugin = _plugin(None) + plugin.get_display_duration.side_effect = KeyError('display_duration') + expected = _rotation_seconds(plugin) + assert _pause(_coord(plugin), plugin, clock)[1] == pytest.approx(expected, abs=0.15) diff --git a/web_interface/blueprints/api_v3/plugin_store.py b/web_interface/blueprints/api_v3/plugin_store.py index d0f31feb..c1c70e84 100644 --- a/web_interface/blueprints/api_v3/plugin_store.py +++ b/web_interface/blueprints/api_v3/plugin_store.py @@ -530,11 +530,13 @@ def install_plugin(): ) branch_msg = f" (branch: {branch})" if branch else "" - # plugin_id: the id to enable it by (see _installed_plugin_id). + # plugin_id: the id to enable it by, and the id its config + # section is under (see _installed_plugin_id). + installed_id = _installed_plugin_id(plugin_id) return {'success': True, 'message': f'Plugin {plugin_id} installed successfully{branch_msg}', - 'plugin_id': _installed_plugin_id(plugin_id), - **_store_restart_fields('install', _plugin_enabled_in_config(plugin_id))} + 'plugin_id': installed_id, + **_store_restart_fields('install', _plugin_enabled_in_config(installed_id))} else: error_msg = f'Failed to install plugin {plugin_id}' if branch: @@ -588,10 +590,11 @@ def install_plugin(): ) branch_msg = f" (branch: {branch})" if branch else "" + installed_id = _installed_plugin_id(plugin_id) return success_response( message=f'Plugin installed successfully{branch_msg}', - extra={'plugin_id': _installed_plugin_id(plugin_id), - **_store_restart_fields('install', _plugin_enabled_in_config(plugin_id))}) + extra={'plugin_id': installed_id, + **_store_restart_fields('install', _plugin_enabled_in_config(installed_id))}) else: error_msg = f'Failed to install plugin {plugin_id}' if branch: From 6c533d62afb2c208a875af5e06c0327f81bf15d2 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Tue, 6 Oct 2026 08:51:15 -0400 Subject: [PATCH 32/37] fix(plugins): web mode lookups use the modes the display registered (#668) (#769) * fix(plugins): web mode lookups use the modes the display registered (#668) A plugin may compute its display modes from its config: soccer-scoreboard registers soccer__live/recent/upcoming for every custom_leagues entry, which no manifest can list ahead of time. The display always rotated them (_register_loaded_plugin prefers plugin.modes), but the web process reads plugins as files, so /display/modes, the on-demand dialog and on-demand/start with a mode and no plugin_id saw only manifests -- a custom league's mode was missing from every list and 404'd on lookup. - PluginStateManager.record_modes(): the controller records what it registered, on the loaded record (an unload or reload forgets it) - the runtime snapshot carries it per plugin as "modes" (bounded), and PluginRuntimeView.display_modes() reports it only while live - PluginCatalog takes a runtime_source; get_plugin_display_modes and find_plugin_for_mode prefer the live modes, falling back to the manifest when the display is stopped or has not loaded the plugin. The view is read at most once a second, so a listing is one read, not one per plugin. No manifest or plugin change needed. Co-Authored-By: Claude Opus 5.5 * fix(plugins): call the runtime view's display_modes directly Codacy flagged the getattr/callable indirection as 'lookup is not callable'. The view is a PluginRuntimeView or None; anything else raises inside the existing try and falls back to the manifest. Co-Authored-By: Claude Opus 5.5 * fix(plugins): address review -- no manifest fallback for live plugins, keep mode names whole, send registered spelling Co-Authored-By: Claude Sonnet 5.5 --------- Co-authored-by: Claude Opus 5.5 --- docs/ARCHITECTURE.md | 7 +- docs/REST_API_REFERENCE.md | 8 +- src/display_controller.py | 9 + src/plugin_system/plugin_catalog.py | 73 +++++- src/plugin_system/plugin_runtime.py | 30 ++- src/plugin_system/plugin_state.py | 26 +- test/test_api_v3_display_modes.py | 13 +- test/test_live_display_modes.py | 265 +++++++++++++++++++++ test/test_plugin_runtime_snapshot.py | 2 +- web_interface/app.py | 8 + web_interface/blueprints/api_v3/display.py | 19 +- web_interface/blueprints/api_v3/plugins.py | 5 +- 12 files changed, 442 insertions(+), 23 deletions(-) create mode 100644 test/test_live_display_modes.py diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7a095786..0018abb7 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -143,7 +143,9 @@ loaded and when. Nothing else keeps plugin state: `DisplayController` right after it creates the `PluginManager`, writes the cache key `plugin_runtime_snapshot`: per plugin `loaded`, `state`, `error` (type, a redacted message of at most 200 characters, when, recoverable), -`version` and `loaded_at`, plus `published_at`, `stale_after` and `running`. +`version`, `loaded_at` and `modes` (the display modes `DisplayController` +registered -- `plugin.modes` when the plugin computes them, else the +manifest's), plus `published_at`, `stale_after` and `running`. The cache is on disk, usually the SD card, so it writes when something a reader sees changes -- throttled to once per 10 s -- and otherwise once a minute as a heartbeat. RUNNING, which every `update()` passes through, is @@ -159,6 +161,9 @@ truth cannot leak into a response. `/api/v3/plugins/installed` returns `loaded`, `state`, `error_info`, `loaded_version` and `loaded_at` per plugin and `data.runtime` (`status`, `published_at`, `age_seconds`); `/api/v3/plugins/state` returns the same beside the desired state. +`PluginCatalog.get_plugin_display_modes` and `find_plugin_for_mode` prefer a +live view's `modes` to the manifest's `display_modes`, so `/display/modes` +and on-demand see modes a plugin generates from its config (#668). **Reconciliation** ([`state_reconciliation.py`](../src/plugin_system/state_reconciliation.py)) diff --git a/docs/REST_API_REFERENCE.md b/docs/REST_API_REFERENCE.md index 46693f7f..57d79ceb 100644 --- a/docs/REST_API_REFERENCE.md +++ b/docs/REST_API_REFERENCE.md @@ -363,9 +363,11 @@ it. This is the list the force-display dialog offers. Send the reported `plugin_id` alongside `mode` when starting an on-demand display: `/display/on-demand/start` falls back to `find_plugin_for_mode` when -`plugin_id` is omitted, and that lookup only sees modes declared in a static -manifest — a plugin whose modes are generated (each installed Starlark app is -one) returns 404 there. +`plugin_id` is omitted. While the display is running, this list and that +lookup use the modes the display registered, including ones a plugin generates +from its config (each installed Starlark app, each soccer `custom_leagues` +entry). With the display stopped, or for a plugin it has not loaded, both see +only the modes its manifest declares. Triggers plugin discovery, which is otherwise lazy — so a caller that never opens the dashboard still gets the full list. diff --git a/src/display_controller.py b/src/display_controller.py index de61f2aa..bdefde24 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -4611,6 +4611,15 @@ class DisplayController: display_modes = [plugin_id] with self._plugin_modes_lock: self.plugin_display_modes[plugin_id] = list(display_modes) + # Into the runtime snapshot the web interface reads, so its mode + # lists and on-demand lookups see computed modes too (#668). + state_manager = getattr(self.plugin_manager, 'state_manager', None) + record_modes = getattr(state_manager, 'record_modes', None) + if callable(record_modes): + try: + record_modes(plugin_id, list(display_modes)) + except Exception as e: # reporting must never break registration + logger.debug("Could not record display modes for %s: %s", plugin_id, e) # Subscribe to config changes for per-plugin hot-reload. Bind plugin_id # and instance as defaults so each plugin's callback targets its own diff --git a/src/plugin_system/plugin_catalog.py b/src/plugin_system/plugin_catalog.py index 8e0a25f0..bd80f2ab 100644 --- a/src/plugin_system/plugin_catalog.py +++ b/src/plugin_system/plugin_catalog.py @@ -15,7 +15,8 @@ reads through a catalog unchanged. It has nothing that runs a plugin: no ``load_plugin``, ``get_plugin`` or ``plugins``. Runtime state -- whether the display has a plugin loaded, its health, its -errors -- is not here either. The display process publishes what it knows to +errors -- is not here either, with one exception: given a ``runtime_source``, +the mode lookups prefer the modes the running display registered. The display process publishes what it knows to the shared cache (health and resource metrics, the current mode, the error aggregator snapshot), and the web routes read those publications. What the display does not publish (which plugins it has loaded, its plugin state @@ -26,8 +27,9 @@ See docs/ARCHITECTURE.md ("Web and display processes"). import json import threading +import time from pathlib import Path -from typing import Any, Dict, List, Optional, Union, cast +from typing import Any, Callable, Dict, List, Optional, Union, cast from src.common.permission_utils import ( ensure_directory_permissions, get_plugin_dir_mode, @@ -39,6 +41,10 @@ from src.plugin_system.plugin_dirs import ( PathLike = Union[str, Path] +#: How long one read of the display's runtime view answers mode lookups. A +#: listing asks once per plugin; the cache copy is a file read each time. +_RUNTIME_VIEW_TTL_SECONDS = 1.0 + class PluginCatalog: """Manifests, schemas, config and versions of the installed plugins. @@ -49,10 +55,17 @@ class PluginCatalog: """ def __init__(self, plugins_dir: PathLike, config_manager: Optional[Any] = None, - schema_manager: Optional[Any] = None) -> None: + schema_manager: Optional[Any] = None, + runtime_source: Optional[Callable[[], Any]] = None) -> None: self.plugins_dir: Path = Path(plugins_dir) self.config_manager = config_manager self.schema_manager = schema_manager + # Returns the display's PluginRuntimeView + # (src/plugin_system/plugin_runtime.py). Its live view carries the + # modes the display registered, which the mode lookups below prefer + # to the manifest's. None: manifests only. + self.runtime_source = runtime_source + self._runtime_view_memo: Optional[tuple] = None self.logger = get_logger(__name__) # Guards plugin_manifests/plugin_directories: request threads read @@ -172,23 +185,67 @@ class PluginCatalog: by_manifest=False) return str(plugin_dir) if plugin_dir is not None else None - def get_plugin_display_modes(self, plugin_id: str) -> List[str]: - """The manifest's ``display_modes``, or []. + def _runtime_view(self) -> Any: + """The display's runtime view, read at most once a second; None + without a source or when reading it fails.""" + if self.runtime_source is None: + return None + now = time.monotonic() + memo = self._runtime_view_memo + if memo is not None and now - memo[0] < _RUNTIME_VIEW_TTL_SECONDS: + return memo[1] + try: + view = self.runtime_source() + except Exception as exc: # a lookup must still answer from manifests + self.logger.debug("Could not read the display's runtime view: %s", exc) + view = None + self._runtime_view_memo = (now, view) + return view - What the display actually rotates can differ: a plugin may compute - its modes at run time (``plugin.modes``). This is the declared list. + def _live_display_modes(self, plugin_id: str) -> Optional[List[str]]: + """The modes the running display registered for ``plugin_id``, or None.""" + view = self._runtime_view() + if view is None: + return None + try: + modes = view.display_modes(plugin_id) + except Exception as exc: # includes a source returning something else + self.logger.debug("Could not read display modes for %s: %s", plugin_id, exc) + return None + return list(modes) if isinstance(modes, list) and modes else None + + def get_plugin_display_modes(self, plugin_id: str) -> List[str]: + """The modes the display registered for the plugin, else the + manifest's ``display_modes``, else []. + + A plugin may compute its modes at run time (``plugin.modes``): each + league soccer-scoreboard's ``custom_leagues`` adds is a mode no + manifest can list ahead of time (#668). The running display + publishes what it registered, and that wins while the display is + live and has the plugin loaded. Otherwise -- display stopped, plugin + disabled -- the declared list is the best answer there is. """ + live = self._live_display_modes(plugin_id) + if live is not None: + return live with self._lock: manifest = self.plugin_manifests.get(plugin_id) modes = (manifest or {}).get('display_modes', []) return list(modes) if isinstance(modes, list) else [] def find_plugin_for_mode(self, mode: str) -> Optional[str]: - """The plugin whose manifest declares ``mode`` (case-insensitive).""" + """The plugin that registered ``mode`` on the running display, else + the one whose manifest declares it (case-insensitive both ways).""" wanted = mode.strip().lower() with self._lock: manifests = dict(self.plugin_manifests) + for plugin_id in manifests: + live = self._live_display_modes(plugin_id) + if live and any(m.lower() == wanted for m in live): + return plugin_id for plugin_id, manifest in manifests.items(): + if self._live_display_modes(plugin_id): + continue # the display's list is the truth for this plugin modes = manifest.get('display_modes') if isinstance(modes, list) and any( isinstance(m, str) and m.lower() == wanted for m in modes): diff --git a/src/plugin_system/plugin_runtime.py b/src/plugin_system/plugin_runtime.py index 2a541396..25f79468 100644 --- a/src/plugin_system/plugin_runtime.py +++ b/src/plugin_system/plugin_runtime.py @@ -56,7 +56,7 @@ import os import threading import time from dataclasses import dataclass, field, replace -from typing import Any, Callable, Dict, Optional +from typing import Any, Callable, Dict, List, Optional from src import display_watchdog from src.logging_config import get_logger @@ -100,6 +100,9 @@ _ERROR_MESSAGE_CHARS = 200 _ERROR_TYPE_CHARS = 80 _ID_CHARS = 100 _VERSION_CHARS = 40 +#: Bounds on a plugin's published ``modes``: a plugin computes them, so a +#: runaway list must not bloat a file written to the SD card. +_MAX_MODES = 200 #: Reader statuses. Only LIVE carries runtime facts. LIVE = "live" @@ -154,6 +157,15 @@ def summarize_error(error_info: Optional[Dict[str, Any]]) -> Optional[Dict[str, } +def _published_modes(modes: Any) -> Optional[List[str]]: + """The registered display modes as a snapshot carries them, or None.""" + if not isinstance(modes, list): + return None + # A name is a key the display matches exactly: drop one too long to + # carry whole rather than clip it into a different name. + return [m for m in modes if isinstance(m, str) and len(m) <= _ID_CHARS][:_MAX_MODES] + + def build_runtime_snapshot(state_manager: Any, *, started_at: float, now: Optional[float] = None, running: bool = True, @@ -173,6 +185,7 @@ def build_runtime_snapshot(state_manager: Any, *, started_at: float, "error": summarize_error(record.get("error_info")), "version": _clip(version, _VERSION_CHARS) if version else None, "loaded_at": _epoch(record.get("loaded_at")), + "modes": _published_modes(record.get("modes")), } return { "schema": SNAPSHOT_SCHEMA, @@ -416,6 +429,21 @@ class PluginRuntimeView: "loaded_at": record.get("loaded_at"), } + def display_modes(self, plugin_id: str) -> Optional[List[str]]: + """The display modes the display registered for ``plugin_id``: what + it rotates and accepts on-demand, including modes a plugin computes + from its config. None unless the view is live and the plugin is + loaded with its modes registered -- the caller then falls back to + the manifest's ``display_modes``.""" + if not self.live: + return None + record = self.plugins.get(plugin_id) + modes = record.get("modes") if isinstance(record, dict) else None + if not isinstance(modes, list): + return None + modes = [m for m in modes if isinstance(m, str)] + return modes or None + def describe(self) -> Dict[str, Any]: """The view's own status, for a response to carry beside the facts.""" return { diff --git a/src/plugin_system/plugin_state.py b/src/plugin_system/plugin_state.py index 17755890..f89a9b75 100644 --- a/src/plugin_system/plugin_state.py +++ b/src/plugin_system/plugin_state.py @@ -10,7 +10,7 @@ snapshot ``plugin_runtime.PluginRuntimePublisher`` publishes from it. import threading import time from enum import Enum -from typing import Optional, Dict, Any +from typing import Any, Dict, List, Optional from datetime import datetime import logging @@ -231,6 +231,26 @@ class PluginStateManager: } self._note_change() + def record_modes(self, plugin_id: str, modes: List[str]) -> None: + """Record the display modes the display registered for ``plugin_id``. + + Called by the DisplayController each time it registers the plugin. + These are the modes it actually rotates and accepts on-demand -- + ``plugin.modes`` when the plugin computes them (a soccer league the + user added under ``custom_leagues``), else the manifest's list -- and + the web interface has no other way to learn them (#668). Kept on the + loaded record, so an unload or a reload's fresh record_loaded() + forgets them until the plugin is registered again. + """ + with self._lock: + loaded = self._loaded.get(plugin_id) + if loaded is None: + return + modes = [str(m) for m in modes] + if loaded.get('modes') != modes: + loaded['modes'] = modes + self._note_change() + def record_unloaded(self, plugin_id: str) -> None: """Forget the loaded record alone, keeping state and error info: for an unload that failed after the instance was already dropped.""" @@ -243,7 +263,8 @@ class PluginStateManager: section so a concurrent load or unload is seen whole or not at all. Per plugin: ``state`` (published_state()'s value), ``loaded``, - ``version`` and ``loaded_at`` (None unless loaded) and ``error_info`` + ``version``, ``loaded_at`` and ``modes`` (None unless loaded; ``modes`` + also None until the display registers it) and ``error_info`` (a copy, or None). """ with self._lock: @@ -257,6 +278,7 @@ class PluginStateManager: 'loaded': loaded is not None, 'version': loaded['version'] if loaded else None, 'loaded_at': loaded['loaded_at'] if loaded else None, + 'modes': list(loaded['modes']) if loaded and 'modes' in loaded else None, 'error_info': dict(info) if info is not None else None, } return records diff --git a/test/test_api_v3_display_modes.py b/test/test_api_v3_display_modes.py index a59913f0..5258c043 100644 --- a/test/test_api_v3_display_modes.py +++ b/test/test_api_v3_display_modes.py @@ -7,7 +7,7 @@ manifest.json off disk and reimplemented PluginManager's own fallbacks. """ import json -from unittest.mock import MagicMock +from unittest.mock import MagicMock, patch import pytest @@ -155,3 +155,14 @@ class TestOneBadConfigSectionDoesNotBlankTheList: side_effect=RuntimeError("GET https://x/y?api_key=SEC123 failed")) body = api_v3_client.get('/api/v3/display/modes').get_json() assert 'SEC123' not in json.dumps(body) + + +class TestOnDemandUsesTheRegisteredSpelling: + def test_a_mode_differing_in_case_is_sent_as_registered(self, client): + with patch('web_interface.blueprints.api_v3.display._deliver_on_demand', + return_value=('socket', None)) as deliver: + response = client.post('/api/v3/display/on-demand/start', + json={'plugin_id': 'football-scoreboard', + 'mode': 'NFL_LIVE', 'start_service': False}) + assert response.status_code == 200, response.get_json() + assert deliver.call_args.args[0]['mode'] == 'nfl_live' diff --git a/test/test_live_display_modes.py b/test/test_live_display_modes.py new file mode 100644 index 00000000..05e695fe --- /dev/null +++ b/test/test_live_display_modes.py @@ -0,0 +1,265 @@ +"""The web interface sees the display modes the display actually registered (#668). + +A plugin may compute its modes from its config: soccer-scoreboard registers +``soccer__live/recent/upcoming`` for every league the user adds under +``custom_leagues``, and no manifest can list those ahead of time. The display +always rotated them -- DisplayController._register_loaded_plugin prefers +``plugin.modes`` -- but the web process reads plugins as files, so its mode +listing (/display/modes, the on-demand dialog) and find_plugin_for_mode +(/display/on-demand/start with a mode and no plugin_id) saw only manifests. + +The display now records each plugin's registered modes in its plugin state, +the runtime snapshot carries them, and PluginCatalog prefers them while the +snapshot is live, falling back to the manifest when it is not. +""" +import json +import sys +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from src.cache_manager import CacheManager # noqa: E402 +from src.plugin_system import plugin_runtime as rt # noqa: E402 +from src.plugin_system.plugin_catalog import PluginCatalog # noqa: E402 +from src.plugin_system.plugin_runtime import ( # noqa: E402 + PluginRuntimePublisher, build_runtime_snapshot, read_plugin_runtime, + view_from_snapshot, +) +from src.plugin_system.plugin_state import PluginState, PluginStateManager # noqa: E402 +from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402 + +DECLARED = ["soccer_eng.1_live", "soccer_eng.1_recent", "soccer_eng.1_upcoming"] +CUSTOM = ["soccer_sco.1_live", "soccer_sco.1_recent", "soccer_sco.1_upcoming"] +REGISTERED = DECLARED + CUSTOM + + +def _loaded_states(modes=None): + states = PluginStateManager() + states.set_state("soccer-scoreboard", PluginState.ENABLED) + states.record_loaded("soccer-scoreboard", "2.24.1") + if modes is not None: + states.record_modes("soccer-scoreboard", modes) + return states + + +@pytest.fixture +def shared_cache(tmp_path, monkeypatch): + """Two cache managers over one directory: the display's and the web's.""" + monkeypatch.setattr(CacheManager, "_get_writable_cache_dir", + lambda self: str(tmp_path / "cache")) + (tmp_path / "cache").mkdir() + display_cache, web_cache = CacheManager(), CacheManager() + yield display_cache, web_cache + display_cache.stop_cleanup_thread() + web_cache.stop_cleanup_thread() + + +@pytest.fixture +def plugins_dir(tmp_path): + root = tmp_path / "plugins" + for plugin_id, modes in (("soccer-scoreboard", DECLARED), ("clock-simple", ["clock"])): + (root / plugin_id).mkdir(parents=True) + (root / plugin_id / "manifest.json").write_text(json.dumps({ + "id": plugin_id, "name": plugin_id, "version": "1.0.0", + "class_name": "P", "display_modes": modes}), encoding="utf-8") + return root + + +# --- The display records what it registered --------------------------------- + +class TestStateManagerRecordsModes: + def test_runtime_records_carry_them(self): + assert _loaded_states(REGISTERED).runtime_records()[ + "soccer-scoreboard"]["modes"] == REGISTERED + + def test_none_until_registered(self): + assert _loaded_states().runtime_records()["soccer-scoreboard"]["modes"] is None + + def test_a_new_list_is_a_change_the_same_one_is_not(self): + """change_count drives the publisher: re-registering an unchanged + plugin must not cost an SD-card write.""" + states = _loaded_states(DECLARED) + before = states.change_count + states.record_modes("soccer-scoreboard", list(DECLARED)) + assert states.change_count == before + states.record_modes("soccer-scoreboard", REGISTERED) + assert states.change_count == before + 1 + + def test_ignored_for_a_plugin_that_is_not_loaded(self): + states = PluginStateManager() + states.record_modes("ghost", ["ghost"]) + assert "ghost" not in states.runtime_records() + + def test_unload_forgets_them(self): + states = _loaded_states(REGISTERED) + states.clear_state("soccer-scoreboard") + assert "soccer-scoreboard" not in states.runtime_records() + + def test_a_reload_starts_without_them_until_registered_again(self): + states = _loaded_states(REGISTERED) + states.record_loaded("soccer-scoreboard", "2.25.0") + assert states.runtime_records()["soccer-scoreboard"]["modes"] is None + + +class TestControllerRecordsOnRegistration: + def test_plugin_modes_reach_the_state_manager(self, test_display_controller): + """_register_loaded_plugin is the one path every load, enable and + reload goes through.""" + c = test_display_controller + states = _loaded_states() + plugin = MagicMock() + plugin.modes = list(REGISTERED) + c.plugin_manager.state_manager = states + c.plugin_manager.get_plugin = MagicMock(return_value=plugin) + c.plugin_manager.plugin_manifests = {"soccer-scoreboard": {"display_modes": DECLARED}} + + c._register_loaded_plugin("soccer-scoreboard") + + assert states.runtime_records()["soccer-scoreboard"]["modes"] == REGISTERED + + def test_a_failing_state_manager_does_not_break_registration(self, test_display_controller): + c = test_display_controller + plugin = MagicMock() + plugin.modes = ["clock"] + c.plugin_manager.state_manager.record_modes = MagicMock(side_effect=RuntimeError("x")) + c.plugin_manager.get_plugin = MagicMock(return_value=plugin) + c.plugin_manager.plugin_manifests = {} + + assert c._register_loaded_plugin("clock-simple") == ["clock"] + assert c.mode_to_plugin_id["clock"] == "clock-simple" + + +# --- The snapshot carries them; only a live view reports them --------------- + +class TestSnapshotAndView: + NOW = 1_800_000_000.0 + + def _view(self, states, running=True, published_at=None): + snapshot = build_runtime_snapshot(states, started_at=1.0, now=self.NOW, + running=running) + if published_at is not None: + snapshot["published_at"] = published_at + return view_from_snapshot(snapshot, now=self.NOW) + + def test_live_view_reports_the_registered_modes(self): + assert self._view(_loaded_states(REGISTERED)).display_modes( + "soccer-scoreboard") == REGISTERED + + def test_stale_and_stopped_views_report_nothing(self): + states = _loaded_states(REGISTERED) + assert self._view(states, published_at=self.NOW - 10_000).display_modes( + "soccer-scoreboard") is None + assert self._view(states, running=False).display_modes("soccer-scoreboard") is None + + def test_unregistered_or_unknown_plugins_report_nothing(self): + view = self._view(_loaded_states()) + assert view.display_modes("soccer-scoreboard") is None + assert view.display_modes("not-loaded") is None + + def test_a_runaway_list_is_bounded(self): + modes = [f"m{i}" for i in range(1000)] + ["x" * 500] + snapshot = build_runtime_snapshot(_loaded_states(modes), started_at=1.0, now=self.NOW) + published = snapshot["plugins"]["soccer-scoreboard"]["modes"] + assert len(published) == rt._MAX_MODES + + def test_a_mode_name_is_kept_whole_or_dropped(self): + long_mode = "x" * (rt._ID_CHARS + 1) + snapshot = build_runtime_snapshot(_loaded_states(["ok", long_mode]), + started_at=1.0, now=self.NOW) + assert snapshot["plugins"]["soccer-scoreboard"]["modes"] == ["ok"] + + def test_non_strings_from_a_hand_made_snapshot_are_dropped(self): + snapshot = {"schema": rt.SNAPSHOT_SCHEMA, "running": True, + "published_at": self.NOW, "plugins": { + "p": {"loaded": True, "modes": ["a", 3, None]}}} + assert view_from_snapshot(snapshot, now=self.NOW).display_modes("p") == ["a"] + + +# --- The web's catalog prefers them ------------------------------------------- + +class TestCatalog: + def _catalog(self, plugins_dir, web_cache): + catalog = PluginCatalog(plugins_dir, + runtime_source=lambda: read_plugin_runtime(web_cache)) + catalog.discover_plugins() + return catalog + + def test_live_display_modes_win_over_the_manifest(self, plugins_dir, shared_cache): + display_cache, web_cache = shared_cache + PluginRuntimePublisher(display_cache, _loaded_states(REGISTERED)).tick() + catalog = self._catalog(plugins_dir, web_cache) + assert catalog.get_plugin_display_modes("soccer-scoreboard") == REGISTERED + + def test_a_custom_league_mode_resolves_to_its_plugin(self, plugins_dir, shared_cache): + """What /display/on-demand/start does with a mode and no plugin_id.""" + display_cache, web_cache = shared_cache + PluginRuntimePublisher(display_cache, _loaded_states(REGISTERED)).tick() + catalog = self._catalog(plugins_dir, web_cache) + assert catalog.find_plugin_for_mode("SOCCER_SCO.1_LIVE") == "soccer-scoreboard" + + def test_a_plugin_the_display_has_not_loaded_falls_back_to_its_manifest( + self, plugins_dir, shared_cache): + display_cache, web_cache = shared_cache + PluginRuntimePublisher(display_cache, _loaded_states(REGISTERED)).tick() + catalog = self._catalog(plugins_dir, web_cache) + assert catalog.get_plugin_display_modes("clock-simple") == ["clock"] + assert catalog.find_plugin_for_mode("clock") == "clock-simple" + + def test_a_mode_the_display_dropped_does_not_resolve_by_manifest( + self, plugins_dir, shared_cache): + display_cache, web_cache = shared_cache + PluginRuntimePublisher(display_cache, _loaded_states(CUSTOM)).tick() + catalog = self._catalog(plugins_dir, web_cache) + assert catalog.find_plugin_for_mode("soccer_eng.1_live") is None + + def test_a_stopped_display_falls_back_to_manifests(self, plugins_dir, shared_cache): + display_cache, web_cache = shared_cache + publisher = PluginRuntimePublisher(display_cache, _loaded_states(REGISTERED)) + publisher.tick() + publisher.stop() + catalog = self._catalog(plugins_dir, web_cache) + assert catalog.get_plugin_display_modes("soccer-scoreboard") == DECLARED + assert catalog.find_plugin_for_mode("soccer_sco.1_live") is None + + def test_no_runtime_source_is_manifests_only(self, plugins_dir): + catalog = PluginCatalog(plugins_dir) + catalog.discover_plugins() + assert catalog.get_plugin_display_modes("soccer-scoreboard") == DECLARED + + def test_a_failing_runtime_source_is_manifests_only(self, plugins_dir): + def broken(): + raise OSError("cache gone") + catalog = PluginCatalog(plugins_dir, runtime_source=broken) + catalog.discover_plugins() + assert catalog.get_plugin_display_modes("soccer-scoreboard") == DECLARED + + def test_one_listing_reads_the_view_once(self, plugins_dir): + source = MagicMock(return_value=None) + catalog = PluginCatalog(plugins_dir, runtime_source=source) + catalog.discover_plugins() + for _ in range(10): + catalog.get_plugin_display_modes("soccer-scoreboard") + catalog.find_plugin_for_mode("clock") + assert source.call_count == 1 + + +class TestDisplayModesRoute: + def test_lists_the_custom_league_modes(self, api_v3_module, api_v3_client, # noqa: F811 + plugins_dir, shared_cache): + display_cache, web_cache = shared_cache + PluginRuntimePublisher(display_cache, _loaded_states(REGISTERED)).tick() + api = api_v3_module.api_v3 + api.plugin_catalog = PluginCatalog( + plugins_dir, runtime_source=lambda: read_plugin_runtime(web_cache)) + api.config_manager.load_config = MagicMock(return_value={ + "soccer-scoreboard": {"enabled": True}}) + + response = api_v3_client.get("/api/v3/display/modes") + + assert response.status_code == 200, response.get_data(as_text=True) + modes = {m["mode"]: m for m in response.get_json()["data"]["modes"]} + assert set(modes) == set(REGISTERED) + assert modes["soccer_sco.1_live"]["plugin_id"] == "soccer-scoreboard" diff --git a/test/test_plugin_runtime_snapshot.py b/test/test_plugin_runtime_snapshot.py index 30dda43c..f937b3dd 100644 --- a/test/test_plugin_runtime_snapshot.py +++ b/test/test_plugin_runtime_snapshot.py @@ -162,7 +162,7 @@ class TestPublisher: assert snapshot["stale_after"] == rt.STALE_AFTER assert snapshot["plugins"] == {"clock": { "loaded": True, "state": "enabled", "error": None, - "version": "1.0.0", "loaded_at": 10.0}} + "version": "1.0.0", "loaded_at": 10.0, "modes": None}} def test_changes_are_throttled_and_quiet_displays_refresh(self): cache = MagicMock() diff --git a/web_interface/app.py b/web_interface/app.py index 4f4ca235..73ae0ca2 100644 --- a/web_interface/app.py +++ b/web_interface/app.py @@ -159,10 +159,18 @@ schema_manager = SchemaManager( # saves reach the running plugins through the display's config watcher; what # the display knows at run time (health, metrics, errors, current mode) it # publishes to the shared cache. See docs/ARCHITECTURE.md. +def _catalog_runtime_view(): + """The display's runtime view, for the catalog's mode lookups. Imported + on call, as the startup reconciliation below imports it.""" + from web_interface.blueprints.api_v3 import _plugin_runtime_view + return _plugin_runtime_view() + + plugin_catalog = PluginCatalog( plugins_dir=plugins_dir, config_manager=config_manager, schema_manager=schema_manager, + runtime_source=_catalog_runtime_view, ) # Initialize operation queue for plugin operations diff --git a/web_interface/blueprints/api_v3/display.py b/web_interface/blueprints/api_v3/display.py index 3cbe6c60..74652433 100644 --- a/web_interface/blueprints/api_v3/display.py +++ b/web_interface/blueprints/api_v3/display.py @@ -153,10 +153,12 @@ def get_display_modes(): same list the force-display dialog offers, from the source that owns it. Knowing each mode's plugin_id also matters because /display/on-demand/start - falls back to find_plugin_for_mode when plugin_id is omitted, and that - lookup only sees modes declared in a static manifest -- a plugin whose - modes are generated (each installed Starlark app is one) 404s there. - Sending the plugin_id from this list skips the lookup entirely. + falls back to find_plugin_for_mode when plugin_id is omitted. While the + display is running, both that lookup and this list use the modes it + registered, so modes a plugin generates from its config (each installed + Starlark app, each soccer custom league) are found (#668); with the + display stopped they see only what manifests declare. Sending the + plugin_id from this list skips the lookup entirely. Query params: include_disabled: '1' to list modes of disabled plugins too. They can @@ -277,6 +279,15 @@ def start_on_demand_display(): if not resolved_plugin: return jsonify({'status': 'error', 'message': f'Mode {resolved_mode} not found'}), 404 + # The display matches mode names exactly: pass the registered spelling + # when the caller's differs only in case. + if api_v3.plugin_catalog and resolved_plugin and resolved_mode: + wanted = resolved_mode.strip().lower() + for registered in api_v3.plugin_catalog.get_plugin_display_modes(resolved_plugin): + if isinstance(registered, str) and registered.lower() == wanted: + resolved_mode = registered + break + # On-demand works with disabled plugins: the running display loads one # for the session and unloads it afterwards, leaving config.json alone # (DisplayController._load_plugin_for_on_demand). Logged for debugging. diff --git a/web_interface/blueprints/api_v3/plugins.py b/web_interface/blueprints/api_v3/plugins.py index 1cd2e394..f2b5bb1b 100644 --- a/web_interface/blueprints/api_v3/plugins.py +++ b/web_interface/blueprints/api_v3/plugins.py @@ -150,8 +150,9 @@ def get_installed_plugins(): vegas_participation, vegas_participation_source = _vegas_participation( plugin_id, plugin_config, plugin_info) - # The modes the manifest declares, from the catalog as /display/modes - # and on-demand/start read them. The on-demand modal offers these; + # The plugin's modes, from the catalog as /display/modes and + # on-demand/start read them: what the running display registered, + # else what the manifest declares. The on-demand modal offers these; # without them it offered only the plugin id, which the display # turns into the first mode. Strings only: a manifest is hand-edited. declared_modes = api_v3.plugin_catalog.get_plugin_display_modes(plugin_id) From 87f255b2fa0023b816cf918a1c1d387c80d005ec Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Tue, 6 Oct 2026 13:35:28 -0400 Subject: [PATCH 33/37] docs(common): list espn_payload in the common README (#782) #749 added src/common/espn_payload.py without a summary row or section, so test_common_readme_lists_every_module fails on main. Claude-Session: https://claude.ai/code/session_01VZNWFWprcFYGfuf1JAyrBJ Co-authored-by: Claude --- src/common/README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/common/README.md b/src/common/README.md index 66140ede..6dbb924b 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -28,6 +28,7 @@ Rules for the package: | [`api_helper`](#api_helper) | HTTP GET/POST with caching and rate limiting | Yes | — | | [`bdf_font`](#bdf_font) | Load and draw BDF bitmap fonts | Yes, if drawing BDF text directly | 3.5.0 | | [`espn_dates`](#espn_dates) | Fetch ESPN scoreboards across a date range | Yes (scoreboards) | 3.5.0 | +| [`espn_payload`](#espn_payload) | Drop the parts of an ESPN scoreboard payload no scoreboard reads | No, core-internal (used by `BackgroundDataService`) | n/a | | [`favorite_team_check`](#favorite_team_check) | Log why a favourite team code shows nothing | Yes (scoreboards) | 3.6.0 | | [`fetch_service`](#fetch_service) | Pooled, merged, budgeted and counted HTTP for core fetch paths | No, core-internal (reached through `api_helper` and `espn_dates`) | n/a | | [`font_layout`](#font_layout) | Reproducible TrueType loading, crisp sizes | Yes | 3.4.0 | @@ -120,6 +121,18 @@ Every request goes through [`fetch_service`](#fetch_service), the chunks counted against the plugin that asked. Scoreboard plugins also bundle a copy for older cores. +### espn_payload + +[`espn_payload.py`](espn_payload.py). Core-internal. ESPN scoreboard +responses carry stat leaders, athlete cards, links, headlines and highlights +that no scoreboard draws. `slim_scoreboard_payload(payload)` removes exactly +those keys, in place, and leaves everything it does not know about alone; +`is_espn_scoreboard_url(url)` says whether a URL is an ESPN site-API +scoreboard. `BackgroundDataService` slims each scoreboard window before +caching it, which cuts the five sports windows from ~40MB to ~12MB of parsed +objects. Adding a key to the drop lists means first checking that nothing +reads it. + ### favorite_team_check [`favorite_team_check.py`](favorite_team_check.py). From d98b4797273cb907be93ea3da71e55b229a9a40c Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Tue, 6 Oct 2026 13:35:49 -0400 Subject: [PATCH 34/37] fix(install): only a running desktop stops the install; detect desktops by metapackage (#781) * fix(install): detect desktops by their metapackages, not name prefixes The Lite check matched any installed package starting with gnome/kde/ xfce/lxde, so standalone parts (gnome-keyring, xfce4-terminal, lxde-icon-theme) rejected a Lite system. Match whole names of desktop metapackages and session managers instead. Also catch desktops the prefixes missed: Raspberry Pi OS Trixie replaced raspberrypi-ui-mods with rpd-wayland-core / rpd-x-core, Debian tasksel desktops (task-*-desktop), and multi-arch names (plasma-workspace:arm64). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VZNWFWprcFYGfuf1JAyrBJ * docs(common): list espn_payload in the common README #749 added src/common/espn_payload.py without a summary row or section, so test_common_readme_lists_every_module fails on main. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VZNWFWprcFYGfuf1JAyrBJ (cherry picked from commit 8333f23e5c53e2662ddf7c1b2c35f4f98dd74552) * fix(install): only a running desktop stops the install A desktop costs the panel CPU only while it runs, so a running display manager (checked with systemctl is-active, including the generic display-manager alias, instead of a grep -q pipe under pipefail) still stops the installer. Desktop packages or session files on a Pi that boots to the console now print a warning and the install continues. Adds installer OS-check tests for running, installed-only and Lite systems, including the libblockdev and gnome-keyring false positives. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VZNWFWprcFYGfuf1JAyrBJ --------- Co-authored-by: Claude --- docs/TROUBLESHOOTING.md | 5 ++- first_time_install.sh | 47 ++++++++++++++-------- test/test_install_os_support.py | 69 ++++++++++++++++++++++++++++----- 3 files changed, 94 insertions(+), 27 deletions(-) diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 7721e70e..e1e37510 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -101,7 +101,10 @@ python3 --version Imager, choosing Raspberry Pi OS Lite (64-bit). Trixie is recommended; Bookworm (Legacy) also works. An in-place upgrade from Bullseye is not supported by Raspberry Pi and is not worth the risk. -- "Desktop environment detected": use the Lite image, not the desktop one. +- "A desktop is running": use the Lite image, not the desktop one, or boot + to the console with `sudo systemctl set-default multi-user.target` and + reboot. Desktop packages that are installed but not running only produce a + warning, and the install continues. - "python3 is Python 3.x; LEDMatrix needs Python 3.11 or newer": something has replaced the system `python3`. Point it back at the OS's own Python (`/usr/bin/python3` should be 3.11 on Bookworm, 3.13 on Trixie). diff --git a/first_time_install.sh b/first_time_install.sh index ea494c12..a4af1a6b 100755 --- a/first_time_install.sh +++ b/first_time_install.sh @@ -86,29 +86,44 @@ if [ -r "$LM_OS_RELEASE_FILE" ]; then OS_CHECK_FAILED=1 fi - # Check if it's the Lite version (no desktop environment) - # Check for desktop packages or desktop services - DESKTOP_DETECTED=0 + # Check for a desktop. A desktop only competes with the panel for CPU while + # it runs, so a running display manager stops the install; desktop packages + # or session files on a Pi that boots to the console are only a warning. + DESKTOP_RUNNING=0 + DESKTOP_INSTALLED=0 + # display-manager is the alias every Debian display manager registers. + for dm in display-manager lightdm gdm gdm3 sddm lxdm; do + if systemctl is-active --quiet "$dm" 2>/dev/null; then + DESKTOP_RUNNING=1 + fi + done # grep without -q: -q exits at the first match, dpkg then dies of SIGPIPE, # and pipefail turns a found desktop into "not found". - # Match installed package names from their start: the unanchored ".*kde" - # matched mid-word (libblockdev-* = "bloc-kde-v") on Lite, and `dpkg -l` - # lines also carry descriptions that could match. + # Desktop metapackages and session managers, matched as whole installed + # package names: an unanchored ".*kde" matched libblockdev-* ("bloc-kde-v"), + # and a "gnome" prefix matched standalone parts such as gnome-keyring. + # Trixie replaced raspberrypi-ui-mods with the rpd-*-core metapackages. + DESKTOP_PACKAGES='raspberrypi-ui-mods|rpd-wayland-core|rpd-x-core' + DESKTOP_PACKAGES+='|lxde|lxde-core|lxsession|xfce4|xfce4-session' + DESKTOP_PACKAGES+='|gnome-shell|gnome-session|kde-plasma-desktop|plasma-desktop' + DESKTOP_PACKAGES+='|plasma-workspace|task-desktop|task-[a-z0-9]+-desktop' if dpkg-query -W -f='${db:Status-Abbrev} ${binary:Package}\n' 2>/dev/null \ - | grep -E "^ii +(raspberrypi-ui-mods|lxde|xfce|gnome|kde)" >/dev/null; then - DESKTOP_DETECTED=1 - fi - if systemctl list-units --type=service --state=running 2>/dev/null | grep -qE "lightdm|gdm3|sddm|lxdm"; then - DESKTOP_DETECTED=1 + | grep -E "^ii +(${DESKTOP_PACKAGES})(:[a-z0-9]+)?$" >/dev/null; then + DESKTOP_INSTALLED=1 fi if [ -d /usr/share/raspberrypi-ui-mods ] || [ -d /usr/share/xsessions ]; then - DESKTOP_DETECTED=1 + DESKTOP_INSTALLED=1 fi - - if [ "$DESKTOP_DETECTED" -eq 1 ]; then - echo "✗ ERROR: Desktop environment detected - this script requires Raspberry Pi OS Lite" - echo " Please use Raspberry Pi OS Lite (not the full desktop version)" + + if [ "$DESKTOP_RUNNING" -eq 1 ]; then + echo "✗ ERROR: A desktop is running - this script requires Raspberry Pi OS Lite" + echo " Please use Raspberry Pi OS Lite (not the full desktop version), or boot" + echo " to the console: sudo systemctl set-default multi-user.target && sudo reboot" OS_CHECK_FAILED=1 + elif [ "$DESKTOP_INSTALLED" -eq 1 ]; then + echo "⚠ WARNING: Desktop packages are installed, but no desktop is running." + echo " Continuing. Keep the Pi booting to the console: a running desktop" + echo " competes with the LED panel for CPU and can make it flicker." else echo "✓ Lite version confirmed (no desktop environment)" fi diff --git a/test/test_install_os_support.py b/test/test_install_os_support.py index 95f1fe43..4a53d5da 100644 --- a/test/test_install_os_support.py +++ b/test/test_install_os_support.py @@ -50,9 +50,11 @@ def _stub(bin_dir: Path, name: str, body: str) -> None: path.chmod(0o755) -def _stubs(tmp_path: Path, python_version="3.11", network="NetworkManager") -> Path: +def _stubs(tmp_path: Path, python_version="3.11", network="NetworkManager", + active=(), packages=()) -> Path: """python3 reports ``python_version`` (None: not installed); systemctl - reports ``network`` as the only active unit; dpkg lists no desktop.""" + reports ``network`` and ``active`` as the only active units; dpkg-query + lists ``packages`` as installed (none by default, so no desktop).""" bin_dir = tmp_path / "bin" bin_dir.mkdir(exist_ok=True) if python_version is None: @@ -61,10 +63,14 @@ def _stubs(tmp_path: Path, python_version="3.11", network="NetworkManager") -> P else: _stub(bin_dir, "python3", f'case "$*" in *"%d.%d.%d"*) echo "{python_version}.1" ;; ' f'*) echo "{python_version}" ;; esac\n') - _stub(bin_dir, "systemctl", - f'case "$*" in *"is-active --quiet {network}") exit 0 ;; esac\nexit 3\n') + units = "|".join(f'*"is-active --quiet {unit}"' for unit in (network, *active)) + _stub(bin_dir, "systemctl", f'case "$*" in {units}) exit 0 ;; esac\nexit 3\n') _stub(bin_dir, "dpkg", "exit 0\n") - _stub(bin_dir, "dpkg-query", "exit 1\n") + if packages: + listing = "".join(f"ii {name}\\n" for name in packages) + _stub(bin_dir, "dpkg-query", f'printf "{listing}"\n') + else: + _stub(bin_dir, "dpkg-query", "exit 1\n") _stub(bin_dir, "ping", "exit 0\n") return bin_dir @@ -149,20 +155,22 @@ class TestLibrary: # --- first_time_install.sh's OS check ------------------------------------------ -def _os_check_section() -> str: +def _os_check_section(marker_root: str = "/nonexistent") -> str: """first_time_install.sh from the OS check up to the next section, with - the desktop-marker directories pointed somewhere that cannot exist.""" + the desktop-marker directories moved under ``marker_root`` (by default + somewhere that cannot exist).""" text = FIRST_TIME.read_text(encoding="utf-8").replace("\r\n", "\n") start = text.index("# Check OS version") end = text.index("# The user who ran the installer") section = text[start:end] for marker in ("/usr/share/raspberrypi-ui-mods", "/usr/share/xsessions"): assert marker in section - section = section.replace(marker, "/nonexistent" + marker) + section = section.replace(marker, marker_root + marker) return section -def run_os_check(tmp_path: Path, release: str, **stub_args) -> subprocess.CompletedProcess: +def run_os_check(tmp_path: Path, release: str, marker_root: str = "/nonexistent", + **stub_args) -> subprocess.CompletedProcess: """Run the OS check as the installer would, from a copy of the project layout so ``$(dirname "$0")/scripts/install/lib_os.sh`` resolves.""" project = tmp_path / "project" @@ -171,7 +179,7 @@ def run_os_check(tmp_path: Path, release: str, **stub_args) -> subprocess.Comple script = project / "first_time_install.sh" script.write_text("set -Eeuo pipefail\n" "trap 'echo ERR-TRAP line $LINENO >&2; exit 99' ERR\n" - + _os_check_section() + '\necho "SECTION-DONE"\n', + + _os_check_section(marker_root) + '\necho "SECTION-DONE"\n', encoding="utf-8", newline="\n") env = _env(tmp_path, release, _stubs(tmp_path, **stub_args)) return subprocess.run(["bash", str(script)], capture_output=True, text=True, env=env) @@ -230,6 +238,47 @@ class TestInstallerOsCheck: result = run_os_check(tmp_path, "trixie", python_version="3.13") assert "✓ NetworkManager is managing the network" in result.stdout + # A running desktop stops the install; one that is only installed warns. + + @pytest.mark.parametrize("unit", ["display-manager", "lightdm", "gdm", "sddm"]) + def test_running_desktop_stops(self, tmp_path, unit): + result = run_os_check(tmp_path, "trixie", python_version="3.13", active=(unit,)) + assert result.returncode == 1, result.stdout + result.stderr + assert "A desktop is running" in result.stdout + assert "multi-user.target" in result.stdout + assert "SECTION-DONE" not in result.stdout + + @pytest.mark.parametrize("package", [ + "raspberrypi-ui-mods", "rpd-wayland-core", "rpd-x-core", "xfce4", + "lxde-core", "gnome-shell", "kde-plasma-desktop", "plasma-workspace:arm64", + "task-desktop", "task-mate-desktop", + ]) + def test_installed_desktop_that_is_not_running_warns(self, tmp_path, package): + result = run_os_check(tmp_path, "trixie", python_version="3.13", + packages=("bash", package)) + assert result.returncode == 0, result.stdout + result.stderr + assert "Desktop packages are installed, but no desktop is running" in result.stdout + assert "✓ OS requirements met" in result.stdout + + def test_desktop_session_files_warn(self, tmp_path): + (tmp_path / "markers" / "usr" / "share" / "xsessions").mkdir(parents=True) + result = run_os_check(tmp_path, "trixie", python_version="3.13", + marker_root=str(tmp_path / "markers")) + assert result.returncode == 0, result.stdout + result.stderr + assert "Desktop packages are installed, but no desktop is running" in result.stdout + + @pytest.mark.parametrize("packages", [ + # libblockdev contains "kde" mid-word; the old check stopped on it. + ("libblockdev-crypto3", "libblockdev3:arm64"), + ("gnome-keyring", "xfce4-terminal", "xfconf", "lxde-icon-theme", + "kde-cli-tools", "gnome-session-common", "task-ssh-server", "rpd-plym-splash"), + ]) + def test_lite_with_desktop_named_parts_is_lite(self, tmp_path, packages): + result = run_os_check(tmp_path, "trixie", python_version="3.13", packages=packages) + assert result.returncode == 0, result.stdout + result.stderr + assert "✓ Lite version confirmed" in result.stdout + assert "WARNING: Desktop" not in result.stdout + # --- check_system_compatibility.sh --------------------------------------------- From 358911cb201f07abcbaa443018bc534fe301d147 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 8 Oct 2026 14:31:01 -0400 Subject: [PATCH 35/37] chore: prepare the 3.8.3 release (#784) Bumps src.__version__ to 3.8.3 and turns Unreleased into ## 3.8.3, adding entries for what merged since 3.8.2: the installer's desktop check (#780, #781) -- 3.8.2 stops fresh Raspberry Pi OS Lite installs with "Desktop environment detected", and the one-shot installer installs the newest release, so the fix only reaches users once it is tagged -- the ESPN scoreboard payload slimming (#749), the Vegas static-pause and store restart fixes (#753), and plugin-computed display modes in the web UI (#769). scripts/check_release_version.py v3.8.3 passes. Claude-Session: https://claude.ai/code/session_01VZNWFWprcFYGfuf1JAyrBJ Co-authored-by: Claude --- CHANGELOG.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++++- src/__init__.py | 2 +- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eed96cd3..5d98ee1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,60 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +## 3.8.3 + +Fresh installs on Raspberry Pi OS Lite work again: 3.8.2's installer reported +a desktop on Lite and stopped (#780, #781). Also lighter cached ESPN +scoreboard windows (#749), a Vegas static-pause fix and a store reinstall +restart prompt (#753), and display modes a plugin computes from its config in +the web UI (#769). + +### Install + +- The installer no longer stops Raspberry Pi OS Lite with "Desktop environment + detected". Its package check searched whole `dpkg -l` lines, and `.*kde` + matched inside `libblockdev` ("bloc**kde**v"), which Lite ships; it now + matches installed package names from their start (#780). Desktop + metapackages and session managers are matched as whole names, so + `gnome-keyring` and similar standalone parts no longer count, and the list + now includes Raspberry Pi OS Trixie's `rpd-wayland-core` / `rpd-x-core` + (which replaced `raspberrypi-ui-mods`), Debian's `task-*-desktop` and + multi-arch names (#781). +- Only a **running** desktop stops the install: a display manager that + `systemctl is-active` reports (`display-manager`, lightdm, gdm, sddm, + lxdm), with directions to boot to the console instead. Desktop packages or + session files on a Pi that boots to the console print a warning and the + install continues (#781). + +### Fixed + +- A Vegas static pause no longer ends at once for a plugin whose + `display_duration` is not a number (a string such as `"20"` or `null` from + config.json, as clock-simple, calendar and countdown return it). The pause + reads the duration the way the rotation does, with the same fallbacks: 30 s + for anything that is not a number, 15 s for zero or less. The helper moved + from `display_controller._finite_seconds` to `base_plugin.finite_seconds`, + unchanged (#753). +- Reinstalling an enabled plugin from the store now asks for a restart for + plugins that install under their manifest id (Weather as + `ledmatrix-weather`, Music, Stocks, Leaderboard): the route checked the + enabled flag under the registry id (#753). +- `/display/modes`, the on-demand dialog and `on-demand/start` by mode see the + modes a plugin computes from its config (soccer-scoreboard's custom + leagues), which no manifest can list. The display records the modes it + registered in the runtime snapshot, and the web catalog prefers them while + the plugin is loaded, falling back to the manifest otherwise. No manifest or + plugin change needed (#769, fixes #668). + +### Performance + +- `BackgroundDataService` drops the parts of an ESPN `/scoreboard` response no + scoreboard reads (stat leaders, athlete cards, links, headlines, highlights, + geo broadcasts) before caching it (`src/common/espn_payload.py`, + core-internal). Measured on one Pi (hdpi), the five scoreboard windows went + from 10.6 MB to 3.0 MB of JSON and ~40 MB to ~12 MB of parsed objects. + `submit_fetch_request(slim_payload=False)` caches a response whole (#749). + ### Tooling - `test/test_sports_helpers.py`'s parity tests pass again with @@ -27,7 +81,9 @@ accepts both, but the store flags the old spelling as deprecated (ledmatrix-plugins #563/#564), and the 19 tests still expected them. A copy that is gone now counts as adopted when the plugin imports `src.common.sports_helpers`, as the stage 3/4 and game-over parity tests - already do; a copy that remains must still match. + already do; a copy that remains must still match. (#777) +- `src/common/README.md` lists `espn_payload`, which + `test_common_readme_lists_every_module` requires (#782). ## 3.8.2 diff --git a/src/__init__.py b/src/__init__.py index add0e258..6623f60a 100644 --- a/src/__init__.py +++ b/src/__init__.py @@ -4,5 +4,5 @@ LEDMatrix Display System Core source package for the LED Matrix Display project. """ -__version__ = "3.8.2" +__version__ = "3.8.3" From 370c8fe273e5d4bd4df4c6ef98d92389468c1c47 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 8 Oct 2026 15:55:50 -0400 Subject: [PATCH 36/37] chore: remove dead code, deprecate unused plugin APIs (over-engineering audit) (#783) * chore: remove dead code, deprecate unused plugin APIs (over-engineering audit) Whole-tree audit. Every symbol was checked against core, the plugin monorepo and all eight third-party plugins in plugins.json first. - Deprecate (removal 3.10.0) plugin-facing methods nothing calls: LogoDownloader bulk download, ConfigManager backup/secret wrappers, APIHelper extras, BackgroundDataService poll API, PluginManager / PluginStateManager info readers, and a few CacheManager, FontManager, BaseOddsManager, DynamicTeamResolver methods and PluginTestCase. plugin_api_usage.py learns their receiver names; DEPRECATIONS doc regenerated. - Remove core-internal dead code: CacheMetrics, Vegas status/stats plumbing, sync "new cycle" message (followers ignore unknown types), unused operation types, test-only PluginCatalog readers, IPC to_dict and ping, _parse_form_value, CacheStrategyProtocol, ErrorAggregator callbacks, duplicate web response helpers. - Web UI: drop never-mounted json-file-manager.js, the example widget, utils/error_handler.js, four uncalled PluginAPI methods, and 29 escapeHtml shims (call window.LEDEscape directly). Public globals, BaseWidget and widget names unchanged. - Remove six one-off scripts (owner decision) and the unused markupsafe and pytest-mock pins. Co-Authored-By: Claude Opus 5.5 * fix(web): calendar picker error text goes in a text node, not innerHTML Same output as the escaped innerHTML it replaces; clears Codacy's XSS-pattern alerts on the line. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 64 ++ docs/ARCHITECTURE.md | 10 +- docs/DEPRECATIONS_3.8.md | 374 +++++--- docs/MULTI_ROOT_WORKSPACE_SETUP.md | 6 +- mypy-clean.txt | 1 - requirements-test.txt | 1 - scripts/README.md | 12 +- scripts/add_defaults_to_schemas.py | 231 ----- scripts/analyze_plugin_schemas.py | 279 ------ scripts/dev/README.md | 3 +- scripts/dev/run_emulator.sh | 13 - scripts/plugin_api_usage.py | 11 +- scripts/test_captive_portal.sh | 149 ---- scripts/update_plugin_repos.py | 43 - scripts/verify_wifi_before_testing.sh | 225 ----- src/background_data_service.py | 4 + src/base_odds_manager.py | 3 + src/cache/__init__.py | 1 - src/cache/cache_metrics.py | 134 --- src/cache/cache_strategy.py | 41 +- src/cache/disk_cache.py | 26 +- src/cache_manager.py | 16 +- src/common/api_helper.py | 8 + src/common/sync_manager.py | 47 +- src/config_manager.py | 7 + src/dynamic_team_resolver.py | 3 + src/error_aggregator.py | 42 +- src/font_manager.py | 3 + src/ipc/client.py | 5 - src/ipc/contract.py | 28 - src/logo_downloader.py | 8 + src/plugin_system/operation_history.py | 26 +- src/plugin_system/operation_queue.py | 61 +- src/plugin_system/operation_types.py | 31 - src/plugin_system/plugin_catalog.py | 66 +- src/plugin_system/plugin_manager.py | 6 + src/plugin_system/plugin_state.py | 7 + src/plugin_system/state_reconciliation.py | 8 +- src/plugin_system/store_registry.py | 11 +- src/plugin_system/testing/plugin_test_base.py | 2 + src/vegas_mode/config.py | 43 - src/vegas_mode/coordinator.py | 41 +- src/vegas_mode/render_pipeline.py | 55 +- src/vegas_mode/stream_manager.py | 49 +- src/web_interface/api_helpers.py | 19 +- src/web_interface/config_arrays.py | 4 +- src/web_interface/error_handler.py | 82 +- src/web_interface/errors.py | 25 +- test/_api_v3_test_helpers.py | 3 +- test/js/unit/test_html_escaping.js | 43 +- test/js/unit/test_inline_handler_escaping.js | 6 +- test/js/unit/test_render_cards.js | 4 +- test/js/unit/test_store_registry_fields.js | 2 +- test/js/unit/test_update_all.js | 5 +- test/test_cache_manager.py | 78 +- test/test_cache_strategy_intervals.py | 62 +- test/test_deprecation.py | 39 +- test/test_display_pending_changes.py | 1 - test/test_error_aggregator.py | 49 -- test/test_frame_ops.py | 3 - test/test_ipc_contract.py | 4 +- test/test_ipc_display_stage2.py | 1 - test/test_ipc_server.py | 28 +- test/test_ipc_state_stream.py | 2 +- test/test_operation_queue_pending_and_trim.py | 6 +- test/test_render_gate.py | 1 - test/test_scroll_helper_lazy_image.py | 3 - test/test_sync_manager.py | 63 +- test/test_vegas_config.py | 49 +- test/test_vegas_continuous_refresh.py | 1 - test/test_vegas_coordinator_config.py | 1 - test/test_vegas_coordinator_iteration.py | 2 - test/test_vegas_crisp_pacing.py | 3 - test/test_vegas_density.py | 23 +- test/test_vegas_elements_layout.py | 3 - test/test_vegas_live_apply.py | 3 - test/test_vegas_live_integration.py | 2 +- test/test_vegas_live_lifecycle.py | 3 - test/test_vegas_live_property.py | 3 - test/test_vegas_prepared_blocks.py | 3 - test/test_vegas_static_mode.py | 5 - test/test_web_smoke.py | 2 +- .../integration/test_plugin_operations.py | 27 +- test/web_interface/test_api_v3_helpers.py | 39 - test/web_interface/test_error_handler.py | 64 +- .../test_plugin_operation_queue.py | 19 +- .../test_state_reconciliation.py | 5 - .../test_web_process_runs_no_plugin_code.py | 24 +- web_interface/app.py | 8 +- web_interface/blueprints/api_v3/__init__.py | 57 +- web_interface/requirements.txt | 1 - web_interface/static/v3/app.js | 5 +- web_interface/static/v3/js/app-early.js | 2 +- web_interface/static/v3/js/app-shell.js | 2 +- .../static/v3/js/plugins/api_client.js | 49 -- .../static/v3/js/plugins/state_manager.js | 5 +- .../static/v3/js/utils/error_handler.js | 350 -------- web_interface/static/v3/js/widgets/README.md | 5 +- .../static/v3/js/widgets/color-picker.js | 12 +- .../static/v3/js/widgets/date-picker.js | 14 +- .../static/v3/js/widgets/day-selector.js | 6 +- .../static/v3/js/widgets/email-input.js | 8 +- .../v3/js/widgets/example-color-picker.js | 200 ----- .../v3/js/widgets/file-upload-single.js | 18 +- .../static/v3/js/widgets/font-selector.js | 14 +- .../v3/js/widgets/google-calendar-picker.js | 6 +- .../static/v3/js/widgets/json-file-manager.js | 821 ------------------ .../static/v3/js/widgets/notification.js | 6 +- .../static/v3/js/widgets/number-input.js | 20 +- .../static/v3/js/widgets/password-input.js | 8 +- .../v3/js/widgets/plugin-file-manager.js | 72 +- .../static/v3/js/widgets/radio-group.js | 10 +- .../static/v3/js/widgets/schedule-picker.js | 20 +- .../static/v3/js/widgets/select-dropdown.js | 8 +- web_interface/static/v3/js/widgets/slider.js | 22 +- .../static/v3/js/widgets/text-input.js | 16 +- .../static/v3/js/widgets/textarea.js | 8 +- .../static/v3/js/widgets/time-picker.js | 8 +- .../static/v3/js/widgets/time-range.js | 16 +- .../static/v3/js/widgets/timezone-selector.js | 10 +- .../static/v3/js/widgets/toggle-switch.js | 8 +- .../static/v3/js/widgets/url-input.js | 12 +- web_interface/static/v3/plugins_manager.js | 108 ++- web_interface/tailwind/tailwind.config.js | 7 + web_interface/templates/v3/base.html | 1 - web_interface/templates/v3/partials/logs.html | 30 +- .../templates/v3/partials/tools.html | 81 +- web_interface/widget_bundle.py | 8 +- 128 files changed, 915 insertions(+), 4165 deletions(-) delete mode 100755 scripts/add_defaults_to_schemas.py delete mode 100755 scripts/analyze_plugin_schemas.py delete mode 100755 scripts/dev/run_emulator.sh delete mode 100755 scripts/test_captive_portal.sh delete mode 100755 scripts/update_plugin_repos.py delete mode 100755 scripts/verify_wifi_before_testing.sh delete mode 100644 src/cache/cache_metrics.py delete mode 100644 web_interface/static/v3/js/utils/error_handler.js delete mode 100644 web_interface/static/v3/js/widgets/example-color-picker.js delete mode 100644 web_interface/static/v3/js/widgets/json-file-manager.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d98ee1d..e2012211 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -85,6 +85,70 @@ the web UI (#769). - `src/common/README.md` lists `espn_payload`, which `test_common_readme_lists_every_module` requires (#782). +### Dead code removed, unused plugin APIs deprecated + +An over-engineering audit of the whole tree. Every symbol below was checked +against core, the plugin monorepo and all eight third-party plugins in +`plugins.json` before it went. Nothing a plugin imports was removed; +plugin-facing methods only get `@deprecated` (see below). + +- **Deprecated for removal in 3.10.0** (warn once per process, in + `journalctl -u ledmatrix`). No plugin in core, the monorepo or the registry + calls them. `docs/DEPRECATIONS_3.8.md` is the regenerated scan, which + `scripts/plugin_api_usage.py` now runs for these owners too: + - `LogoDownloader`: the bulk-download and RGBA-conversion methods + (`fetch_teams_data`, `extract_teams_from_data`, + `download_missing_logos_for_league`, `download_all_ncaa_football_logos`, + `download_all_missing_logos`, `convert_image_to_rgba`, + `convert_all_logos_to_rgba`). `download_missing_logo()` stays. + - `ConfigManager`: `rollback_config`, `list_backups`, + `validate_config_file`, `get_secret`, `cleanup_orphaned_plugin_configs`, + `validate_all_plugin_configs`. + - `APIHelper`: `fetch_espn_scoreboard`/`_standings`/`_rankings`, + `set_cache`, `get_cache`, `set_rate_limit`, `get_request_stats`. `get()` + stays. + - `BackgroundDataService`: `get_result`, `is_request_complete`, + `get_request_status` (pass `callback=` to `submit_fetch_request()`). + - `PluginManager`: `get_all_plugins`, `get_plugin_info`, + `get_all_plugin_info`, `get_plugin_display_modes`, `find_plugin_for_mode`. + `PluginStateManager`: `is_loaded`, `is_running`, `is_error`, + `get_last_update`, `get_error_info`, `get_state_info`. + - `CacheManager.load_cache`, `CacheManager.generate_sport_cache_key`, + `FontManager.measure_text`, `FontManager.get_native_bdf_size`, + `BaseOddsManager.get_odds_for_games`, `BaseOddsManager.format_odds_summary`, + `DynamicTeamResolver.get_available_dynamic_teams`, + `DynamicTeamResolver.is_dynamic_team`, `PluginTestCase`. +- **Removed (core-internal, no caller):** + - `src/cache/cache_metrics.py` + - Vegas status/stats plumbing that nothing read (`get_status`, + `get_current_scroll_info`, `get_buffer_status`, `VegasModeConfig.to_dict`) + - the sync "new cycle" message, which no follower ever handled (followers + now ignore any message type they don't know) + - unused `OperationType` members, `PluginOperation.from_dict`, + `cancel_operation` + - the test-only `PluginCatalog` readers + - `IPC *Args.to_dict` and `client.ping()` + - `_parse_form_value` + - `CacheStrategyProtocol` + - `ErrorAggregator.on_pattern_detected` and `clear_old_records` + - the duplicate `create_error_response`/`create_success_response` +- **Web UI:** + - `json-file-manager.js` was never mounted: the schema widget renders the + plugin's own file manager in an iframe. + - `example-color-picker.js` was a docs example; `utils/error_handler.js` had + one fallback caller. + - The 29 one-line `escapeHtml` shims now call `window.LEDEscape` directly. + - Four uncalled `PluginAPI` methods are gone. + - `window.escapeHtml`, `BaseWidget` and every widget name are unchanged. +- **Scripts and dependencies:** + - One-off scripts removed: `add_defaults_to_schemas.py`, + `analyze_plugin_schemas.py`, `test_captive_portal.sh`, + `verify_wifi_before_testing.sh`, `dev/run_emulator.sh` (use + `python3 run.py -e`), `update_plugin_repos.py` (use + `git -C ../ledmatrix-plugins pull`). + - Unused pins dropped: `markupsafe` (Flask still installs it) and + `pytest-mock`. + ## 3.8.2 The display hands freed memory back to the OS (#774), and sports consolidation diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 0018abb7..20882ad6 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -76,10 +76,10 @@ for the protocol, the permission model and the plan to retire the mailboxes. Only the display process imports plugin code, instantiates plugins and calls their lifecycle hooks (`update`, `display`, `on_config_change`, `on_enable`, `on_disable`). The web process is metadata-only: it reads plugins as files -through `PluginCatalog` -([`src/plugin_system/plugin_catalog.py`](../src/plugin_system/plugin_catalog.py)) --- manifests, config schemas (through `SchemaManager`), each plugin's -section of `config.json`, and installed versions. The catalog keeps the +-- manifests and directories through `PluginCatalog` +([`src/plugin_system/plugin_catalog.py`](../src/plugin_system/plugin_catalog.py)), +config schemas through `SchemaManager`, and each plugin's section of +`config.json` through `ConfigManager`. The catalog keeps the read-only method names of `PluginManager` and has nothing that can run a plugin (no `load_plugin`, `get_plugin` or `plugins`). @@ -299,7 +299,7 @@ and must not vouch for it. | Base class plugins implement | [`base_plugin.py`](../src/plugin_system/base_plugin.py) (`BasePlugin`, `VegasDisplayMode`) | | Finding a plugin's directory | [`plugin_dirs.py`](../src/plugin_system/plugin_dirs.py): manifest `id` first, then directory `` or `ledmatrix-` | | Discovery, load, unload, scheduled updates (display process) | [`plugin_manager.py`](../src/plugin_system/plugin_manager.py) (`PluginManager`) | -| Manifest, schema, config and version reads (web process) | [`plugin_catalog.py`](../src/plugin_system/plugin_catalog.py) (`PluginCatalog`; see [who runs plugins](#web-and-display-processes-who-runs-plugins)) | +| Manifest reads (web process) | [`plugin_catalog.py`](../src/plugin_system/plugin_catalog.py) (`PluginCatalog`; see [who runs plugins](#web-and-display-processes-who-runs-plugins)) | | Import and instantiate | [`plugin_loader.py`](../src/plugin_system/plugin_loader.py) (`PluginLoader.load_plugin()`: dependencies, module, class) | | Timeouts | [`plugin_executor.py`](../src/plugin_system/plugin_executor.py) (`PluginExecutor`, 30 s default; a timed-out thread is abandoned, not killed) | | Circuit breaker | [`plugin_health.py`](../src/plugin_system/plugin_health.py) (`PluginHealthTracker`: 3 consecutive failures open the circuit for 300 s) | diff --git a/docs/DEPRECATIONS_3.8.md b/docs/DEPRECATIONS_3.8.md index c07841f6..fdf62fec 100644 --- a/docs/DEPRECATIONS_3.8.md +++ b/docs/DEPRECATIONS_3.8.md @@ -2,61 +2,73 @@ Generated by `scripts/plugin_api_usage.py` — do not edit by hand; re-run it (see [How to re-run](#how-to-re-run)). -- Scanned: 2026-10-01, core 3.7.0 -- Monorepo: [ChuckBuilds/ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) (main @ 4de1d134), 46 plugins +- Scanned: 2026-10-05, core 3.8.2 +- Monorepo: [ChuckBuilds/ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) (main @ 662fb86f), 46 plugins - Third-party plugins: 8 with their own repo in `plugins.json` (f1-live, gif-player, pga-tour-leaderboard, plex-marquee, ledmatrix-dresden-departures, tidbyt-baseball-scoreboard, sleeper-fantasy, ledmatrix-nascar) -**37 deprecated methods: 36 unused, 1 still used, 0 need review.** +**45 deprecated methods: 31 unused, 3 still used, 11 need review.** Counted per plugin: a *call* is `.method` on an object named like the owner (`cache_manager`, `display_manager`, `font_manager`, `plugin_manager`), or on `self`/`super()` in a subclass; an *override* is `def method` in a subclass of the owner. *Review* hits are `.method` on a receiver whose type the scan cannot tell. *Internal* hits sit inside another deprecated core method and go with it. *Unrelated* hits are a different class's own method with the same name (a name collision), and never block removal; neither do hits in test files. | Method | Removal | Core | Plugins (calls / overrides) | Name collisions & tests | Verdict | |---|---|---|---|---|---| -| `CacheManager.has_data_changed` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.update_cache` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.setup_persistent_cache` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.get_sport_live_interval` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.get_sport_key_from_cache_key` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.get_background_cached_data` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.is_background_data_available` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.record_cache_hit` | 3.8.0 | core (1 internal) | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.record_cache_miss` | 3.8.0 | core (1 internal) | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.record_fetch_time` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.get_cache_metrics` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.log_cache_metrics` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `CacheManager.get_memory_cache_stats` | 3.8.0 | core tests (3 test calls) | — | — | unused — safe to remove in 3.8.0 | -| `DisplayManager.draw_sun` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `DisplayManager.draw_cloud` | 3.8.0 | core (2 internals) | — | ledmatrix-weather (1 unrelated) | unused — safe to remove in 3.8.0 | -| `DisplayManager.draw_rain` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `DisplayManager.draw_snow` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `DisplayManager.draw_weather_icon` | 3.8.0 | core (1 internal) | — | ledmatrix-weather (5 unrelateds) | unused — safe to remove in 3.8.0 | -| `DisplayManager.draw_text_with_icons` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `DisplayManager.get_scrolling_stats` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_manager_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_detected_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.unregister_plugin_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_plugin_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.set_override` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.remove_override` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_overrides` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_available_fonts` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_size_tokens` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_performance_stats` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.get_font_catalog` | 3.8.0 | core tests (1 test call) | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.add_font` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.remove_font` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `FontManager.validate_font` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | -| `BasePlugin.get_supported_vegas_modes` | 3.9.0 | core tests (2 test reviews) | blackjack (1 call, 1 override); calendar (1 override); olympics (1 override) | — | still used by blackjack, calendar, olympics — keep or migrate first | +| `BackgroundDataService.get_result` | 3.10.0 | core tests (15 test reviews) | — | — | unused — safe to remove in 3.10.0 | +| `BackgroundDataService.is_request_complete` | 3.10.0 | core tests (11 test reviews) | — | — | unused — safe to remove in 3.10.0 | +| `BackgroundDataService.get_request_status` | 3.10.0 | core tests (2 test reviews) | — | — | unused — safe to remove in 3.10.0 | +| `BaseOddsManager.get_odds_for_games` | 3.10.0 | core tests (3 test reviews) | — | — | unused — safe to remove in 3.10.0 | +| `BaseOddsManager.format_odds_summary` | 3.10.0 | core tests (5 test reviews) | — | — | unused — safe to remove in 3.10.0 | +| `CacheManager.load_cache` | 3.10.0 | core tests (1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `CacheManager.generate_sport_cache_key` | 3.10.0 | core tests (2 test calls, 1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `APIHelper.fetch_espn_scoreboard` | 3.10.0 | core tests (1 test call, 4 test reviews) | — | football-scoreboard (1 test review); hockey-scoreboard (1 test review); ufc-scoreboard (3 test reviews) | unused — safe to remove in 3.10.0 | +| `APIHelper.fetch_espn_standings` | 3.10.0 | core tests (1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `APIHelper.fetch_espn_rankings` | 3.10.0 | core tests (1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `APIHelper.set_cache` | 3.10.0 | core tests (1 test call, 1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `APIHelper.get_cache` | 3.10.0 | core tests (1 test call) | — | — | unused — safe to remove in 3.10.0 | +| `APIHelper.set_rate_limit` | 3.10.0 | core tests (10 test calls, 3 test reviews) | — | — | unused — safe to remove in 3.10.0 | +| `APIHelper.get_request_stats` | 3.10.0 | — | — | — | unused — safe to remove in 3.10.0 | +| `ConfigManager.rollback_config` | 3.10.0 | core (1 internal); core tests (1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `ConfigManager.list_backups` | 3.10.0 | core (1 internal); core tests (1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `ConfigManager.validate_config_file` | 3.10.0 | core (1 internal) | — | — | unused — safe to remove in 3.10.0 | +| `ConfigManager.get_secret` | 3.10.0 | core tests (5 test calls, 1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `ConfigManager.cleanup_orphaned_plugin_configs` | 3.10.0 | core tests (2 test calls, 1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `ConfigManager.validate_all_plugin_configs` | 3.10.0 | core tests (1 test call, 1 test review) | — | — | unused — safe to remove in 3.10.0 | +| `DynamicTeamResolver.get_available_dynamic_teams` | 3.10.0 | — | — | — | unused — safe to remove in 3.10.0 | +| `DynamicTeamResolver.is_dynamic_team` | 3.10.0 | — | — | — | unused — safe to remove in 3.10.0 | +| `FontManager.get_native_bdf_size` | 3.10.0 | core tests (3 test calls) | — | — | unused — safe to remove in 3.10.0 | +| `FontManager.measure_text` | 3.10.0 | core tests (5 test calls) | — | — | unused — safe to remove in 3.10.0 | +| `LogoDownloader.fetch_teams_data` | 3.10.0 | core (2 internals) | — | — | still used by core — keep or migrate first | +| `LogoDownloader.extract_teams_from_data` | 3.10.0 | core (2 internals) | — | — | still used by core — keep or migrate first | +| `LogoDownloader.download_missing_logos_for_league` | 3.10.0 | core (1 call, 1 internal); core tests (2 test calls) | — | — | still used by core — keep or migrate first | +| `LogoDownloader.download_all_ncaa_football_logos` | 3.10.0 | core tests (2 test calls) | — | — | unused — safe to remove in 3.10.0 | +| `LogoDownloader.download_all_missing_logos` | 3.10.0 | — | — | — | unused — safe to remove in 3.10.0 | +| `LogoDownloader.convert_image_to_rgba` | 3.10.0 | core (1 internal) | — | — | unused — safe to remove in 3.10.0 | +| `LogoDownloader.convert_all_logos_to_rgba` | 3.10.0 | — | — | — | unused — safe to remove in 3.10.0 | +| `BasePlugin.get_supported_vegas_modes` | 3.9.0 | core tests (2 test reviews) | — | — | unused — safe to remove in 3.9.0 | | `BasePlugin.get_vegas_segment_width` | 3.9.0 | core tests (1 test review) | — | — | unused — safe to remove in 3.9.0 | -| `PluginManager.get_enabled_plugins` | 3.8.0 | — | — | — | unused — safe to remove in 3.8.0 | +| `PluginManager.get_all_plugins` | 3.10.0 | — | football-scoreboard (1 review) | football-scoreboard (2 test reviews); hockey-scoreboard (1 test review) | needs review: possible use in football-scoreboard | +| `PluginManager.get_plugin_info` | 3.10.0 | core (12 reviews, 1 internal); core tests (6 test calls, 9 test reviews) | — | — | needs review: possible use in core | +| `PluginManager.get_all_plugin_info` | 3.10.0 | core (3 reviews); core tests (2 test calls, 11 test reviews) | — | — | needs review: possible use in core | +| `PluginManager.get_plugin_display_modes` | 3.10.0 | core (4 reviews); core tests (3 test calls, 5 test reviews) | — | — | needs review: possible use in core | +| `PluginManager.find_plugin_for_mode` | 3.10.0 | core (2 reviews) | — | — | needs review: possible use in core | +| `PluginStateManager.is_loaded` | 3.10.0 | core (1 internal) | — | — | needs review: possible use in core | +| `PluginStateManager.is_running` | 3.10.0 | core (1 internal) | — | — | needs review: possible use in core | +| `PluginStateManager.is_error` | 3.10.0 | core (1 internal) | — | — | needs review: possible use in core | +| `PluginStateManager.get_error_info` | 3.10.0 | core (1 internal); core tests (4 test calls) | — | — | needs review: possible use in core | +| `PluginStateManager.get_last_update` | 3.10.0 | core (1 internal) | — | — | needs review: possible use in core | +| `PluginStateManager.get_state_info` | 3.10.0 | core (1 internal); core tests (7 test reviews) | — | — | needs review: possible use in core | +| `PluginTestCase.setUp` | 3.10.0 | — | — | basketball-scoreboard (2 test unrelateds); cricket-scoreboard (1 test unrelated); hockey-scoreboard (1 test unrelated); nrl-scoreboard (1 test unrelated) | unused — safe to remove in 3.10.0 | -## Unused — safe to remove (36) +## Unused — safe to remove (31) -`CacheManager.has_data_changed`, `CacheManager.update_cache`, `CacheManager.setup_persistent_cache`, `CacheManager.get_sport_live_interval`, `CacheManager.get_sport_key_from_cache_key`, `CacheManager.get_background_cached_data`, `CacheManager.is_background_data_available`, `CacheManager.record_cache_hit`, `CacheManager.record_cache_miss`, `CacheManager.record_fetch_time`, `CacheManager.get_cache_metrics`, `CacheManager.log_cache_metrics`, `CacheManager.get_memory_cache_stats`, `DisplayManager.draw_sun`, `DisplayManager.draw_cloud`, `DisplayManager.draw_rain`, `DisplayManager.draw_snow`, `DisplayManager.draw_weather_icon`, `DisplayManager.draw_text_with_icons`, `DisplayManager.get_scrolling_stats`, `FontManager.get_manager_fonts`, `FontManager.get_detected_fonts`, `FontManager.unregister_plugin_fonts`, `FontManager.get_plugin_fonts`, `FontManager.set_override`, `FontManager.remove_override`, `FontManager.get_overrides`, `FontManager.get_available_fonts`, `FontManager.get_size_tokens`, `FontManager.get_performance_stats`, `FontManager.get_font_catalog`, `FontManager.add_font`, `FontManager.remove_font`, `FontManager.validate_font`, `BasePlugin.get_vegas_segment_width`, `PluginManager.get_enabled_plugins` +`BackgroundDataService.get_result`, `BackgroundDataService.is_request_complete`, `BackgroundDataService.get_request_status`, `BaseOddsManager.get_odds_for_games`, `BaseOddsManager.format_odds_summary`, `CacheManager.load_cache`, `CacheManager.generate_sport_cache_key`, `APIHelper.fetch_espn_scoreboard`, `APIHelper.fetch_espn_standings`, `APIHelper.fetch_espn_rankings`, `APIHelper.set_cache`, `APIHelper.get_cache`, `APIHelper.set_rate_limit`, `APIHelper.get_request_stats`, `ConfigManager.rollback_config`, `ConfigManager.list_backups`, `ConfigManager.validate_config_file`, `ConfigManager.get_secret`, `ConfigManager.cleanup_orphaned_plugin_configs`, `ConfigManager.validate_all_plugin_configs`, `DynamicTeamResolver.get_available_dynamic_teams`, `DynamicTeamResolver.is_dynamic_team`, `FontManager.get_native_bdf_size`, `FontManager.measure_text`, `LogoDownloader.download_all_ncaa_football_logos`, `LogoDownloader.download_all_missing_logos`, `LogoDownloader.convert_image_to_rgba`, `LogoDownloader.convert_all_logos_to_rgba`, `BasePlugin.get_supported_vegas_modes`, `BasePlugin.get_vegas_segment_width`, `PluginTestCase.setUp` -## Still used — keep or migrate first (1) +## Still used — keep or migrate first (3) -`BasePlugin.get_supported_vegas_modes` +`LogoDownloader.fetch_teams_data`, `LogoDownloader.extract_teams_from_data`, `LogoDownloader.download_missing_logos_for_league` + +## Needs review (11) + +`PluginManager.get_all_plugins`, `PluginManager.get_plugin_info`, `PluginManager.get_all_plugin_info`, `PluginManager.get_plugin_display_modes`, `PluginManager.find_plugin_for_mode`, `PluginStateManager.is_loaded`, `PluginStateManager.is_running`, `PluginStateManager.is_error`, `PluginStateManager.get_error_info`, `PluginStateManager.get_last_update`, `PluginStateManager.get_state_info` ## Every hit @@ -64,94 +76,254 @@ File paths are relative to the plugin's directory (core: the repo root). | Method | Where | File:line | Kind | Code | |---|---|---|---|---| -| `CacheManager.get_sport_live_interval` | core | src/cache/cache_strategy.py:28 | unrelated | `def get_sport_live_interval(self, sport_key: str) -> int:` | -| `CacheManager.get_sport_live_interval` | core | src/cache/cache_strategy.py:60 | unrelated | `live_interval = self.get_sport_live_interval(sport_key)` | -| `CacheManager.get_sport_live_interval` | core | src/cache_manager.py:785 | unrelated | `return self._strategy_component.get_sport_live_interval(sport_key)` | -| `CacheManager.get_sport_key_from_cache_key` | core | src/cache/cache_strategy.py:214 | unrelated | `def get_sport_key_from_cache_key(self, key: str) -> Optional[str]:` | -| `CacheManager.get_sport_key_from_cache_key` | core | src/cache_manager.py:806 | unrelated | `return self._strategy_component.get_sport_key_from_cache_key(key)` | -| `CacheManager.get_sport_key_from_cache_key` | core | src/cache_manager.py:816 | unrelated | `sport_key = self._strategy_component.get_sport_key_from_cache_key(key)` | -| `CacheManager.record_cache_hit` | core | src/cache_manager.py:869 | internal (in `CacheManager.get_background_cached_data`) | `self.record_cache_hit('background')` | -| `CacheManager.record_cache_miss` | core | src/cache_manager.py:876 | internal (in `CacheManager.get_background_cached_data`) | `self.record_cache_miss('background')` | -| `CacheManager.record_fetch_time` | core | src/cache/cache_metrics.py:67 | unrelated | `def record_fetch_time(self, duration: float) -> None:` | -| `CacheManager.record_fetch_time` | core | src/cache_manager.py:922 | unrelated | `self._metrics_component.record_fetch_time(duration)` | -| `CacheManager.get_memory_cache_stats` | core tests | test/test_cache_manager_memory_tier.py:43 | test call | `stats = cm.get_memory_cache_stats()` | -| `CacheManager.get_memory_cache_stats` | core tests | test/test_cache_manager_memory_tier.py:63 | test call | `assert cm.get_memory_cache_stats()["last_cleanup"] >= before` | -| `CacheManager.get_memory_cache_stats` | core tests | test/test_cache_manager_memory_tier.py:68 | test call | `stats = cm.get_memory_cache_stats()` | -| `DisplayManager.draw_sun` | core | src/plugin_system/testing/visual_display_manager.py:417 | unrelated | `def draw_sun(self, x: int, y: int, size: int = 16):` | -| `DisplayManager.draw_cloud` | core | src/display_manager.py:1359 | internal (in `DisplayManager.draw_rain`) | `self.draw_cloud(x, y, size)` | -| `DisplayManager.draw_cloud` | core | src/display_manager.py:1374 | internal (in `DisplayManager.draw_snow`) | `self.draw_cloud(x, y, size)` | -| `DisplayManager.draw_cloud` | core | src/plugin_system/testing/visual_display_manager.py:421 | unrelated | `def draw_cloud(self, x: int, y: int, size: int = 16, color: Tuple[int, int, int] = (200, 200, 200)):` | -| `DisplayManager.draw_cloud` | ledmatrix-weather | weather_icons.py:184 | unrelated | `def draw_cloud(draw: ImageDraw, x: int, y: int, size: int = 16, color: tuple = (200, 200, 200)):` | -| `DisplayManager.draw_rain` | core | src/plugin_system/testing/visual_display_manager.py:425 | unrelated | `def draw_rain(self, x: int, y: int, size: int = 16):` | -| `DisplayManager.draw_snow` | core | src/plugin_system/testing/visual_display_manager.py:429 | unrelated | `def draw_snow(self, x: int, y: int, size: int = 16):` | -| `DisplayManager.draw_weather_icon` | core | src/display_manager.py:1518 | internal (in `DisplayManager.draw_text_with_icons`) | `self.draw_weather_icon(icon_type, icon_x, icon_y)` | -| `DisplayManager.draw_weather_icon` | core | src/plugin_system/testing/visual_display_manager.py:510 | unrelated | `def draw_weather_icon(self, condition: str, x: int, y: int, size: int = 16) -> None:` | -| `DisplayManager.draw_weather_icon` | core | src/plugin_system/testing/visual_display_manager.py:533 | unrelated | `self.draw_weather_icon(icon_type, icon_x, icon_y)` | -| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:84 | unrelated | `def draw_weather_icon(image, icon_code, x, y, size):` | -| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:1280 | unrelated | `WeatherIcons.draw_weather_icon(img, icon_code, icon_x, icon_y,` | -| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:1559 | unrelated | `WeatherIcons.draw_weather_icon(img, forecast['icon'], icon_x, icon_y, icon_size)` | -| `DisplayManager.draw_weather_icon` | ledmatrix-weather | manager.py:1650 | unrelated | `WeatherIcons.draw_weather_icon(img, forecast['icon'], icon_x, icon_y, icon_size)` | -| `DisplayManager.draw_weather_icon` | ledmatrix-weather | weather_icons.py:168 | unrelated | `def draw_weather_icon(image: Image.Image, icon_code: str, x: int, y: int, size: int = DEFAULT_SIZE):` | -| `DisplayManager.draw_text_with_icons` | core | src/plugin_system/testing/visual_display_manager.py:526 | unrelated | `def draw_text_with_icons(self, text: str, icons: List[tuple] = None,` | -| `FontManager.get_font_catalog` | core tests | test/test_deprecation.py:229 | test call | `assert fm.get_font_catalog() == fm.font_catalog` | +| `BackgroundDataService.get_result` | core tests | test/test_background_data_service.py:85 | test review | `result = service.get_result(req_id)` | +| `BackgroundDataService.get_result` | core tests | test/test_background_data_service.py:107 | test review | `seen["filed"] = service.get_result(result.request_id) is result` | +| `BackgroundDataService.get_result` | core tests | test/test_background_data_service.py:148 | test review | `result = service.get_result(req_id)` | +| `BackgroundDataService.get_result` | core tests | test/test_background_data_service.py:165 | test review | `result = service.get_result(req_id)` | +| `BackgroundDataService.get_result` | core tests | test/test_background_data_service.py:254 | test review | `assert service.get_result("unknown") is None` | +| `BackgroundDataService.get_result` | core tests | test/test_background_data_service.py:391 | test review | `assert service.get_result(rid).cached is True` | +| `BackgroundDataService.get_result` | core tests | test/test_background_fetch_dedupe.py:198 | test review | `assert service.get_result(req).success is True` | +| `BackgroundDataService.get_result` | core tests | test/test_background_payload_release.py:85 | test review | `result = service.get_result(req_id)` | +| `BackgroundDataService.get_result` | core tests | test/test_background_payload_release.py:117 | test review | `stored = service.get_result(req_id)` | +| `BackgroundDataService.get_result` | core tests | test/test_background_payload_release.py:144 | test review | `assert service.get_result(req_id).data == PAYLOAD` | +| `BackgroundDataService.get_result` | core tests | test/test_background_payload_release.py:154 | test review | `stored = service.get_result(req_id)` | +| `BackgroundDataService.get_result` | core tests | test/test_background_payload_release.py:171 | test review | `assert service.get_result(req_id).data is None` | +| `BackgroundDataService.get_result` | core tests | test/test_background_payload_release.py:193 | test review | `assert service.get_result(req_id).data == PAYLOAD` | +| `BackgroundDataService.get_result` | core tests | test/test_background_payload_release.py:278 | test review | `stored = service.get_result(first)` | +| `BackgroundDataService.get_result` | core tests | test/test_fetch_service.py:703 | test review | `assert bds.get_result(request_id).success` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service.py:145 | test review | `while not service.is_request_complete(req_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service.py:162 | test review | `while not service.is_request_complete(req_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service.py:194 | test review | `while not service.is_request_complete(req_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service.py:211 | test review | `while not service.is_request_complete(req_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service.py:246 | test review | `assert service.is_request_complete("r2") is False` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service.py:251 | test review | `assert service.is_request_complete("r3") is True` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service_espn_ranges.py:91 | test review | `while not service.is_request_complete(request_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_data_service_espn_ranges.py:154 | test review | `while not service.is_request_complete(request_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_fetch_dedupe.py:55 | test review | `while not service.is_request_complete(req_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_background_payload_release.py:67 | test review | `while not service.is_request_complete(req_id) and time.time() < deadline:` | +| `BackgroundDataService.is_request_complete` | core tests | test/test_fetch_service.py:701 | test review | `while not bds.is_request_complete(request_id) and time.monotonic() < deadline:` | +| `BackgroundDataService.get_request_status` | core tests | test/test_background_data_service.py:223 | test review | `assert service.get_request_status("nonexistent") is None` | +| `BackgroundDataService.get_request_status` | core tests | test/test_background_fetch_dedupe.py:448 | test review | `assert service.get_request_status(rid) is FetchStatus.CANCELLED, (` | +| `BaseOddsManager.get_odds_for_games` | core tests | test/test_base_odds_manager.py:356 | test review | `result = manager.get_odds_for_games(games)` | +| `BaseOddsManager.get_odds_for_games` | core tests | test/test_base_odds_manager.py:374 | test review | `result = manager.get_odds_for_games(games)` | +| `BaseOddsManager.get_odds_for_games` | core tests | test/test_base_odds_manager.py:385 | test review | `result = manager.get_odds_for_games([game])` | +| `BaseOddsManager.format_odds_summary` | core tests | test/test_base_odds_manager.py:322 | test review | `result = manager.format_odds_summary({` | +| `BaseOddsManager.format_odds_summary` | core tests | test/test_base_odds_manager.py:329 | test review | `result = manager.format_odds_summary(FULL_EXTRACTED)` | +| `BaseOddsManager.format_odds_summary` | core tests | test/test_base_odds_manager.py:333 | test review | `assert manager.format_odds_summary(None) == 'No odds available'` | +| `BaseOddsManager.format_odds_summary` | core tests | test/test_base_odds_manager.py:336 | test review | `assert manager.format_odds_summary({}) == 'No odds available'` | +| `BaseOddsManager.format_odds_summary` | core tests | test/test_base_odds_manager.py:339 | test review | `assert manager.format_odds_summary(` | +| `CacheManager.load_cache` | core tests | test/conftest.py:219 | test review | `mock.load_cache = Mock(side_effect=mock_get)` | +| `CacheManager.generate_sport_cache_key` | core tests | test/test_background_data_service.py:42 | test review | `m.generate_sport_cache_key.return_value = "test_key"` | +| `CacheManager.generate_sport_cache_key` | core tests | test/test_background_data_service.py:356 | test call | `expected = CacheManager.generate_sport_cache_key(None, sport, date_str)` | +| `CacheManager.generate_sport_cache_key` | core tests | test/test_background_data_service.py:367 | test call | `theirs = cm_module.CacheManager.generate_sport_cache_key(None, "nba")` | +| `APIHelper.fetch_espn_scoreboard` | core tests | test/test_api_helper.py:214 | test review | `result = helper.fetch_espn_scoreboard('football', 'nfl')` | +| `APIHelper.fetch_espn_scoreboard` | core tests | test/test_api_helper.py:230 | test review | `helper.fetch_espn_scoreboard('basketball', 'nba', date='20250115')` | +| `APIHelper.fetch_espn_scoreboard` | core tests | test/test_api_helper.py:239 | test review | `helper.fetch_espn_scoreboard('basketball', 'nba', date='20250115', cache_key='mine')` | +| `APIHelper.fetch_espn_scoreboard` | core tests | test/test_api_helper.py:251 | test review | `assert helper.fetch_espn_scoreboard('basketball', 'nba', date='20250115') == {` | +| `APIHelper.fetch_espn_scoreboard` | core tests | test/test_espn_scoreboard_cache.py:199 | test call | `helper.fetch_espn_scoreboard("football", "nfl", date=self.DAY)` | +| `APIHelper.fetch_espn_scoreboard` | football-scoreboard | test_espn_date_ranges.py:119 | test review | `helper = sys.modules[sports.fetch_espn_scoreboard.__module__]` | +| `APIHelper.fetch_espn_scoreboard` | hockey-scoreboard | test_espn_date_ranges.py:106 | test review | `helper = sys.modules[sports.fetch_espn_scoreboard.__module__]` | +| `APIHelper.fetch_espn_scoreboard` | ufc-scoreboard | test_every_bout_is_its_own_fight.py:103 | test review | `_real_fetch = sports.fetch_espn_scoreboard` | +| `APIHelper.fetch_espn_scoreboard` | ufc-scoreboard | test_every_bout_is_its_own_fight.py:108 | test review | `sports.fetch_espn_scoreboard = lambda *a, **k: board(name)` | +| `APIHelper.fetch_espn_scoreboard` | ufc-scoreboard | test_every_bout_is_its_own_fight.py:113 | test review | `sports.fetch_espn_scoreboard = _real_fetch` | +| `APIHelper.fetch_espn_standings` | core tests | test/test_api_helper.py:258 | test review | `helper.fetch_espn_standings('football', 'nfl')` | +| `APIHelper.fetch_espn_rankings` | core tests | test/test_api_helper.py:269 | test review | `helper.fetch_espn_rankings('football', 'college-football')` | +| `APIHelper.set_cache` | core tests | test/test_api_helper.py:128 | test review | `helper.set_cache('k', {'a': 1}, ttl=42)` | +| `APIHelper.set_cache` | core tests | test/test_api_helper.py:350 | test call | `assert helper.set_cache('k', {'a': 1}) is None` | +| `APIHelper.get_cache` | core tests | test/test_api_helper.py:348 | test call | `assert helper.get_cache('k') is None` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:42 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:57 | test review | `helper.set_rate_limit(5)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:72 | test review | `helper.set_rate_limit(5)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:91 | test review | `helper.set_rate_limit(5)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:150 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:305 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:313 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:326 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:334 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_api_helper.py:346 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_espn_scoreboard_cache.py:197 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_fetch_service.py:726 | test call | `helper.set_rate_limit(0)` | +| `APIHelper.set_rate_limit` | core tests | test/test_fetch_service.py:1143 | test call | `helper.set_rate_limit(0)` | +| `ConfigManager.rollback_config` | core | src/config_manager.py:191 | internal (in `ConfigManager.rollback_config`) | `success = atomic_mgr.rollback_config(backup_version)` | +| `ConfigManager.rollback_config` | core | src/config_manager_atomic.py:297 | unrelated | `def rollback_config(self, backup_version: Optional[str] = None) -> bool:` | +| `ConfigManager.rollback_config` | core tests | test/test_config_durable_writes.py:345 | test review | `assert manager.rollback_config()` | +| `ConfigManager.list_backups` | core | src/config_manager.py:212 | internal (in `ConfigManager.list_backups`) | `return atomic_mgr.list_backups()` | +| `ConfigManager.list_backups` | core | src/config_manager_atomic.py:334 | unrelated | `def list_backups(self) -> List[BackupInfo]:` | +| `ConfigManager.list_backups` | core | src/config_manager_atomic.py:309 | unrelated | `backups = self.list_backups()` | +| `ConfigManager.list_backups` | core tests | test/test_config_durable_writes.py:323 | test review | `assert [b.path for b in manager.list_backups()] == [` | +| `ConfigManager.validate_config_file` | core | src/config_manager.py:226 | internal (in `ConfigManager.validate_config_file`) | `return atomic_mgr.validate_config_file(config_path)` | +| `ConfigManager.validate_config_file` | core | src/config_manager_atomic.py:426 | unrelated | `def validate_config_file(self, config_path: Optional[str] = None) -> ValidationResult:` | +| `ConfigManager.get_secret` | core tests | test/conftest.py:246 | test review | `mock.get_secret = Mock(side_effect=mock_get_secret)` | +| `ConfigManager.get_secret` | core tests | test/test_config_manager.py:317 | test call | `assert manager.get_secret("api_key") == "secret123"` | +| `ConfigManager.get_secret` | core tests | test/test_config_manager.py:318 | test call | `assert manager.get_secret("token") == "token456"` | +| `ConfigManager.get_secret` | core tests | test/test_config_manager.py:319 | test call | `assert manager.get_secret("nonexistent") is None` | +| `ConfigManager.get_secret` | core tests | test/test_config_manager.py:325 | test call | `assert manager.get_secret("api_key") is None` | +| `ConfigManager.get_secret` | core tests | test/test_config_manager.py:337 | test call | `assert manager.get_secret("api_key") is None` | +| `ConfigManager.cleanup_orphaned_plugin_configs` | core tests | test/test_config_manager.py:447 | test call | `removed = manager.cleanup_orphaned_plugin_configs(["plugin1", "plugin2"])` | +| `ConfigManager.cleanup_orphaned_plugin_configs` | core tests | test/test_core_config_key_adopters.py:80 | test review | `removed = manager.cleanup_orphaned_plugin_configs(['installed'])` | +| `ConfigManager.cleanup_orphaned_plugin_configs` | core tests | test/test_web_auth.py:616 | test call | `config_manager.cleanup_orphaned_plugin_configs([])` | +| `ConfigManager.validate_all_plugin_configs` | core tests | test/test_core_config_key_adopters.py:91 | test review | `results = manager.validate_all_plugin_configs(schema_manager)` | +| `ConfigManager.validate_all_plugin_configs` | core tests | test/test_retired_plugin_config_keys.py:112 | test call | `results = config_manager.validate_all_plugin_configs(schema_manager)` | +| `FontManager.get_native_bdf_size` | core tests | test/test_font_manager.py:104 | test call | `assert fm.get_native_bdf_size("five_by_seven") == 7` | +| `FontManager.get_native_bdf_size` | core tests | test/test_font_manager.py:107 | test call | `assert fm.get_native_bdf_size("press_start") is None` | +| `FontManager.get_native_bdf_size` | core tests | test/test_font_manager.py:110 | test call | `assert fm.get_native_bdf_size("no-such-family") is None` | +| `FontManager.measure_text` | core tests | test/test_font_manager.py:116 | test call | `width, height, baseline = fm.measure_text("SCORE", font)` | +| `FontManager.measure_text` | core tests | test/test_font_manager.py:119 | test call | `assert fm.measure_text("SCORE", font) == (width, height, baseline)` | +| `FontManager.measure_text` | core tests | test/test_font_manager.py:124 | test call | `short, _, _ = fm.measure_text("AB", font)` | +| `FontManager.measure_text` | core tests | test/test_font_manager.py:125 | test call | `long, _, _ = fm.measure_text("ABCD", font)` | +| `FontManager.measure_text` | core tests | test/test_font_manager.py:132 | test call | `fm.measure_text("X", font)` | +| `LogoDownloader.fetch_teams_data` | core | src/logo_downloader.py:639 | internal (in `LogoDownloader.download_missing_logos_for_league`) | `data = self.fetch_teams_data(league)` | +| `LogoDownloader.fetch_teams_data` | core | src/logo_downloader.py:695 | internal (in `LogoDownloader.download_all_ncaa_football_logos`) | `data = self.fetch_teams_data(league)` | +| `LogoDownloader.extract_teams_from_data` | core | src/logo_downloader.py:645 | internal (in `LogoDownloader.download_missing_logos_for_league`) | `teams = self.extract_teams_from_data(data, league)` | +| `LogoDownloader.extract_teams_from_data` | core | src/logo_downloader.py:701 | internal (in `LogoDownloader.download_all_ncaa_football_logos`) | `teams = self.extract_teams_from_data(data, league)` | +| `LogoDownloader.download_missing_logos_for_league` | core | src/logo_downloader.py:786 | internal (in `LogoDownloader.download_all_missing_logos`) | `downloaded, failed = self.download_missing_logos_for_league(league, force_download)` | +| `LogoDownloader.download_missing_logos_for_league` | core | src/logo_downloader.py:1034 | call | `return downloader.download_missing_logos_for_league(league, force_download)` | +| `LogoDownloader.download_missing_logos_for_league` | core tests | test/test_logo_downloader.py:292 | test call | `downloader.download_missing_logos_for_league("nfl")` | +| `LogoDownloader.download_missing_logos_for_league` | core tests | test/test_logo_downloader.py:304 | test call | `downloader.download_missing_logos_for_league("nfl")` | +| `LogoDownloader.download_all_ncaa_football_logos` | core tests | test/test_logo_downloader.py:319 | test call | `downloader.download_all_ncaa_football_logos()` | +| `LogoDownloader.download_all_ncaa_football_logos` | core tests | test/test_logo_downloader.py:332 | test call | `downloader.download_all_ncaa_football_logos()` | +| `LogoDownloader.convert_image_to_rgba` | core | src/logo_downloader.py:897 | internal (in `LogoDownloader.convert_all_logos_to_rgba`) | `if self.convert_image_to_rgba(logo_file):` | | `BasePlugin.get_supported_vegas_modes` | core tests | test/test_vegas_participation.py:356 | test review | `assert plugin.get_supported_vegas_modes() == [` | | `BasePlugin.get_supported_vegas_modes` | core tests | test/test_vegas_participation.py:358 | test review | `assert plugin.get_supported_vegas_modes()` | -| `BasePlugin.get_supported_vegas_modes` | blackjack | manager.py:732 | override | `def get_supported_vegas_modes(self):` | -| `BasePlugin.get_supported_vegas_modes` | blackjack | manager.py:695 | call | `if mode in self.get_supported_vegas_modes():` | -| `BasePlugin.get_supported_vegas_modes` | calendar | manager.py:875 | override | `def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:` | -| `BasePlugin.get_supported_vegas_modes` | olympics | manager.py:624 | override | `def get_supported_vegas_modes(self) -> List[VegasDisplayMode]:` | | `BasePlugin.get_vegas_segment_width` | core tests | test/test_vegas_participation.py:359 | test review | `assert plugin.get_vegas_segment_width() == 2` | +| `PluginManager.get_all_plugins` | core | src/plugin_system/testing/mocks.py:218 | unrelated | `def get_all_plugins(self) -> Dict[str, Any]:` | +| `PluginManager.get_all_plugins` | football-scoreboard | emulator_demo.py:68 | review | `mock_plugin_manager.get_all_plugins = Mock(return_value=[])` | +| `PluginManager.get_all_plugins` | football-scoreboard | test_dynamic_duration.py:64 | test review | `mock_plugin_manager.get_all_plugins = Mock(return_value=[])` | +| `PluginManager.get_all_plugins` | football-scoreboard | test_football_plugin.py:74 | test review | `mock_plugin_manager.get_all_plugins = Mock(return_value=[])` | +| `PluginManager.get_all_plugins` | hockey-scoreboard | test_hockey_emulator.py:99 | test review | `mock_plugin_manager.get_all_plugins = Mock(return_value=[])` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/plugin_catalog.py:120 | unrelated | `def get_plugin_info(self, plugin_id: str) -> Optional[Dict[str, Any]]:` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/plugin_catalog.py:133 | unrelated | `return [info for info in (self.get_plugin_info(pid) for pid in ids) if info]` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/plugin_manager.py:1001 | internal (in `PluginManager.get_all_plugin_info`) | `return [info for info in [self.get_plugin_info(pid) for pid in pids] if info]` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/store_install.py:208 | review | `plugin_info = self.get_plugin_info(plugin_id, fetch_latest_from_github=True, force_refresh=True)` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/store_registry.py:801 | unrelated | `def get_plugin_info(self, plugin_id: str, fetch_latest_from_github: bool = True, force_refresh: bool = False) -> Optional[Dict]:` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/store_update.py:357 | review | `plugin_info_remote = self.get_plugin_info(plugin_id, fetch_latest_from_github=True, force_refresh=True)` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/store_update.py:362 | review | `plugin_info_remote = self.get_plugin_info(alt_id, fetch_latest_from_github=True, force_refresh=True)` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/store_update.py:685 | review | `plugin_info_remote = self.get_plugin_info(plugin_id, fetch_latest_from_github=True, force_refresh=True)` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/store_update.py:691 | review | `plugin_info_remote = self.get_plugin_info(alt_id, fetch_latest_from_github=True, force_refresh=True)` | +| `PluginManager.get_plugin_info` | core | src/plugin_system/testing/mocks.py:223 | unrelated | `def get_plugin_info(self, plugin_id: str) -> Optional[Dict[str, Any]]:` | +| `PluginManager.get_plugin_info` | core | web_interface/blueprints/api_v3/plugin_store.py:216 | review | `remote_info = api_v3.plugin_store_manager.get_plugin_info(plugin_id, fetch_latest_from_github=True)` | +| `PluginManager.get_plugin_info` | core | web_interface/blueprints/api_v3/plugin_store.py:334 | review | `plugin_info = api_v3.plugin_store_manager.get_plugin_info(plugin_id)` | +| `PluginManager.get_plugin_info` | core | web_interface/blueprints/api_v3/plugin_store.py:545 | review | `elif not api_v3.plugin_store_manager.get_plugin_info(plugin_id):` | +| `PluginManager.get_plugin_info` | core | web_interface/blueprints/api_v3/plugin_store.py:602 | review | `elif not api_v3.plugin_store_manager.get_plugin_info(plugin_id):` | +| `PluginManager.get_plugin_info` | core | web_interface/blueprints/pages_v3.py:224 | review | `info = pages_v3.plugin_catalog.get_plugin_info(pid) or {}` | +| `PluginManager.get_plugin_info` | core | web_interface/blueprints/pages_v3.py:722 | review | `plugin_info = pages_v3.plugin_catalog.get_plugin_info(plugin_id)` | +| `PluginManager.get_plugin_info` | core | web_interface/blueprints/pages_v3.py:727 | review | `plugin_info = pages_v3.plugin_catalog.get_plugin_info(plugin_id)` | +| `PluginManager.get_plugin_info` | core tests | test/test_api_v3_plugin_install_endpoints.py:111 | test review | `manager.get_plugin_info.return_value = None` | +| `PluginManager.get_plugin_info` | core tests | test/test_api_v3_plugin_install_endpoints.py:119 | test review | `manager.get_plugin_info.return_value = {"id": "clock"}` | +| `PluginManager.get_plugin_info` | core tests | test/test_pages_v3_path_guards.py:47 | test call | `plugin_manager.get_plugin_info.return_value = {"name": "Weather", "version": "1.0.0"}` | +| `PluginManager.get_plugin_info` | core tests | test/test_registry_id_resolution.py:81 | test review | `_ids(store.get_plugin_info("ledmatrix-weather", fetch_latest_from_github=False))` | +| `PluginManager.get_plugin_info` | core tests | test/test_store_manager_caches.py:612 | test review | `info = self.sm.get_plugin_info("foo", fetch_latest_from_github=True, force_refresh=True)` | +| `PluginManager.get_plugin_info` | core tests | test/test_store_non_plugin_entries.py:37 | test review | `store.get_plugin_info = MagicMock(return_value=dict(SKIN))` | +| `PluginManager.get_plugin_info` | core tests | test/test_web_plugin_dir_resolution.py:86 | test review | `api.plugin_store_manager.get_plugin_info = MagicMock(return_value=None)` | +| `PluginManager.get_plugin_info` | core tests | test/test_web_plugin_dir_resolution.py:118 | test review | `api.plugin_store_manager.get_plugin_info = MagicMock(return_value=None)` | +| `PluginManager.get_plugin_info` | core tests | test/test_web_plugin_dir_resolution.py:178 | test call | `plugin_manager.get_plugin_info.return_value = {"id": "weather", "name": "Weather"}` | +| `PluginManager.get_plugin_info` | core tests | test/web_interface/test_plugin_config_form_defaults.py:119 | test call | `pm.get_plugin_info.return_value = {"name": "Demo", "version": "1.0.0"}` | +| `PluginManager.get_plugin_info` | core tests | test/web_interface/test_plugin_config_schema_expansion.py:88 | test call | `pm.get_plugin_info.return_value = {"id": plugin_id, "name": plugin_id}` | +| `PluginManager.get_plugin_info` | core tests | test/web_interface/test_plugin_widget_route.py:226 | test call | `pm.get_plugin_info.return_value = {"id": plugin_id, "name": plugin_id}` | +| `PluginManager.get_plugin_info` | core tests | test/web_interface/test_plugin_widget_route.py:228 | test call | `pm.get_plugin_info.return_value["version"] = version` | +| `PluginManager.get_plugin_info` | core tests | test/web_interface/test_update_all_plugins.py:41 | test review | `sm.get_plugin_info.return_value = None` | +| `PluginManager.get_plugin_info` | core tests | test/web_interface/test_web_process_runs_no_plugin_code.py:132 | test review | `store.get_plugin_info.return_value = None` | +| `PluginManager.get_all_plugin_info` | core | src/plugin_system/plugin_catalog.py:129 | unrelated | `def get_all_plugin_info(self) -> List[Dict[str, Any]]:` | +| `PluginManager.get_all_plugin_info` | core | web_interface/blueprints/api_v3/plugins.py:68 | review | `all_plugin_info = api_v3.plugin_catalog.get_all_plugin_info()` | +| `PluginManager.get_all_plugin_info` | core | web_interface/blueprints/pages_v3.py:209 | review | `pi.get('id') for pi in pages_v3.plugin_catalog.get_all_plugin_info()` | +| `PluginManager.get_all_plugin_info` | core | web_interface/blueprints/pages_v3.py:559 | review | `infos = sorted(pages_v3.plugin_catalog.get_all_plugin_info(),` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_api_v3_installed_display_modes.py:30 | test review | `api.plugin_catalog.get_all_plugin_info = MagicMock(return_value=[info])` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_api_v3_installed_plugin_icon.py:24 | test review | `api.plugin_catalog.get_all_plugin_info = MagicMock(return_value=[info])` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_installed_list_registry_offline.py:71 | test review | `api.plugin_catalog.get_all_plugin_info = MagicMock(return_value=[` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_onboarding_checklist.py:68 | test review | `mock_pm.get_all_plugin_info.return_value = []` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_plugin_manager_load_failures.py:78 | test call | `infos = {i["id"]: i for i in pm.get_all_plugin_info()}` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_plugin_runtime_snapshot.py:422 | test review | `api.plugin_catalog.get_all_plugin_info = MagicMock(return_value=[` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_vegas_participation.py:447 | test review | `api.plugin_catalog.get_all_plugin_info = MagicMock(return_value=[info])` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_web_api.py:551 | test review | `mock_plugin_catalog.get_all_plugin_info.return_value = [` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_web_api.py:570 | test review | `mock_plugin_catalog.get_all_plugin_info.return_value = [` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_web_api.py:592 | test review | `mock_plugin_catalog.get_all_plugin_info.return_value = [` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_web_plugin_dir_resolution.py:61 | test review | `api.plugin_catalog.get_all_plugin_info = MagicMock(return_value=[info])` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_web_plugin_dir_resolution.py:195 | test call | `plugin_manager.get_all_plugin_info.assert_not_called()` | +| `PluginManager.get_all_plugin_info` | core tests | test/test_web_smoke.py:95 | test review | `mock_pm.get_all_plugin_info.return_value = [` | +| `PluginManager.get_plugin_display_modes` | core | src/plugin_system/plugin_catalog.py:175 | unrelated | `def get_plugin_display_modes(self, plugin_id: str) -> List[str]:` | +| `PluginManager.get_plugin_display_modes` | core | web_interface/blueprints/api_v3/display.py:197 | review | `plugin_modes = api_v3.plugin_catalog.get_plugin_display_modes(plugin_id) or [plugin_id]` | +| `PluginManager.get_plugin_display_modes` | core | web_interface/blueprints/api_v3/display.py:267 | review | `modes = api_v3.plugin_catalog.get_plugin_display_modes(resolved_plugin)` | +| `PluginManager.get_plugin_display_modes` | core | web_interface/blueprints/api_v3/plugins.py:157 | review | `declared_modes = api_v3.plugin_catalog.get_plugin_display_modes(plugin_id)` | +| `PluginManager.get_plugin_display_modes` | core | web_interface/blueprints/pages_v3.py:565 | review | `modes = pages_v3.plugin_catalog.get_plugin_display_modes(pid) or [pid]` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_api_v3_display_modes.py:40 | test call | `pm.get_plugin_display_modes = MagicMock(` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_api_v3_display_modes.py:100 | test call | `pm.get_plugin_display_modes = MagicMock(return_value=[])` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_api_v3_display_modes.py:122 | test call | `pm.get_plugin_display_modes = MagicMock(` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_api_v3_installed_display_modes.py:31 | test review | `api.plugin_catalog.get_plugin_display_modes = MagicMock(return_value=declared_modes)` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_api_v3_installed_display_modes.py:39 | test review | `api.plugin_catalog.get_plugin_display_modes.assert_any_call('football-scoreboard')` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_installed_list_registry_offline.py:75 | test review | `api.plugin_catalog.get_plugin_display_modes = MagicMock(return_value=[])` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_onboarding_checklist.py:69 | test review | `mock_pm.get_plugin_display_modes.side_effect = lambda pid: []` | +| `PluginManager.get_plugin_display_modes` | core tests | test/test_web_smoke.py:99 | test review | `mock_pm.get_plugin_display_modes.side_effect = (` | +| `PluginManager.find_plugin_for_mode` | core | src/plugin_system/plugin_catalog.py:186 | unrelated | `def find_plugin_for_mode(self, mode: str) -> Optional[str]:` | +| `PluginManager.find_plugin_for_mode` | core | web_interface/blueprints/api_v3/display.py:271 | review | `resolved_plugin = api_v3.plugin_catalog.find_plugin_for_mode(resolved_mode)` | +| `PluginManager.find_plugin_for_mode` | core | web_interface/blueprints/api_v3/display.py:276 | review | `resolved_plugin = api_v3.plugin_catalog.find_plugin_for_mode(resolved_mode)` | +| `PluginStateManager.is_loaded` | core | src/plugin_system/plugin_state.py:299 | internal (in `PluginStateManager.get_state_info`) | `'is_loaded': self.is_loaded(plugin_id),` | +| `PluginStateManager.is_running` | core | src/plugin_system/plugin_state.py:301 | internal (in `PluginStateManager.get_state_info`) | `'is_running': self.is_running(plugin_id),` | +| `PluginStateManager.is_error` | core | src/plugin_system/plugin_state.py:302 | internal (in `PluginStateManager.get_state_info`) | `'is_error': self.is_error(plugin_id),` | +| `PluginStateManager.get_error_info` | core | src/plugin_system/plugin_state.py:305 | internal (in `PluginStateManager.get_state_info`) | `'error_info': self.get_error_info(plugin_id),` | +| `PluginStateManager.get_error_info` | core tests | test/test_async_plugin_updates.py:236 | test call | `error = pm.state_manager.get_error_info(plugin_id)` | +| `PluginStateManager.get_error_info` | core tests | test/test_plugin_hang_containment.py:191 | test call | `error_info = pm.state_manager.get_error_info('hung')` | +| `PluginStateManager.get_error_info` | core tests | test/test_sports_sunset_matrix.py:212 | test call | `f"{manager.state_manager.get_error_info(plugin_id)}"` | +| `PluginStateManager.get_error_info` | core tests | test/test_sports_sunset_matrix.py:285 | test call | `info = manager.state_manager.get_error_info(plugin_id)` | +| `PluginStateManager.get_last_update` | core | src/plugin_system/plugin_state.py:304 | internal (in `PluginStateManager.get_state_info`) | `'last_update': self.get_last_update(plugin_id),` | +| `PluginStateManager.get_state_info` | core | src/plugin_system/plugin_manager.py:987 | internal (in `PluginManager.get_plugin_info`) | `info['state'] = self.state_manager.get_state_info(plugin_id)` | +| `PluginStateManager.get_state_info` | core tests | test/test_plugin_state_transition_count.py:39 | test review | `info = manager.get_state_info("clock")` | +| `PluginStateManager.get_state_info` | core tests | test/test_plugin_state_transition_count.py:51 | test review | `info = manager.get_state_info("clock")` | +| `PluginStateManager.get_state_info` | core tests | test/test_plugin_state_transition_count.py:62 | test review | `assert manager.get_state_info("clock")["state_history_count"] == 101` | +| `PluginStateManager.get_state_info` | core tests | test/test_plugin_state_transition_count.py:63 | test review | `assert manager.get_state_info("weather")["state_history_count"] == 1` | +| `PluginStateManager.get_state_info` | core tests | test/test_plugin_state_transition_count.py:74 | test review | `info = manager.get_state_info("clock")` | +| `PluginStateManager.get_state_info` | core tests | test/test_plugin_state_transition_count.py:95 | test review | `info = m.get_state_info("clock")` | +| `PluginStateManager.get_state_info` | core tests | test/test_plugin_state_transition_count.py:130 | test review | `info = manager.get_state_info("clock")` | ## Sources scanned | Source | Group | Python files | Hits | |---|---|---|---| -| core | core | 172 | 20 | -| core tests | core-tests | 347 | 17 | +| core | core | 188 | 50 | +| core tests | core-tests | 416 | 176 | | 7-segment-clock | monorepo | 3 | 0 | -| afl-scoreboard | monorepo | 35 | 0 | -| baseball-scoreboard | monorepo | 61 | 0 | -| basketball-scoreboard | monorepo | 49 | 0 | -| birdnet-go | monorepo | 2 | 0 | -| blackjack | monorepo | 7 | 2 | -| calendar | monorepo | 5 | 1 | +| afl-scoreboard | monorepo | 36 | 0 | +| baseball-scoreboard | monorepo | 71 | 0 | +| basketball-scoreboard | monorepo | 51 | 2 | +| birdnet-go | monorepo | 3 | 0 | +| blackjack | monorepo | 7 | 0 | +| calendar | monorepo | 5 | 0 | | christmas-countdown | monorepo | 3 | 0 | | clock-simple | monorepo | 2 | 0 | | countdown | monorepo | 5 | 0 | -| cricket-scoreboard | monorepo | 8 | 0 | +| cricket-scoreboard | monorepo | 8 | 1 | | f1-scoreboard | monorepo | 15 | 0 | | fantasy-blitz | monorepo | 13 | 0 | -| football-scoreboard | monorepo | 74 | 0 | +| football-scoreboard | monorepo | 78 | 4 | | geochron | monorepo | 10 | 0 | | hello-world | monorepo | 2 | 0 | -| hockey-scoreboard | monorepo | 52 | 0 | +| hockey-scoreboard | monorepo | 57 | 3 | | incoming-packages | monorepo | 8 | 0 | | jellyfin-now-playing | monorepo | 4 | 0 | -| lacrosse-scoreboard | monorepo | 40 | 0 | +| lacrosse-scoreboard | monorepo | 41 | 0 | | ledmatrix-elections | monorepo | 12 | 0 | | ledmatrix-flights | monorepo | 48 | 0 | -| ledmatrix-leaderboard | monorepo | 9 | 0 | -| ledmatrix-music | monorepo | 11 | 0 | +| ledmatrix-leaderboard | monorepo | 10 | 0 | +| ledmatrix-music | monorepo | 12 | 0 | | ledmatrix-stocks | monorepo | 7 | 0 | -| ledmatrix-weather | monorepo | 15 | 6 | +| ledmatrix-weather | monorepo | 15 | 0 | | march-madness | monorepo | 4 | 0 | | masters-tournament | monorepo | 10 | 0 | | mqtt-notifications | monorepo | 4 | 0 | -| news | monorepo | 6 | 0 | +| news | monorepo | 7 | 0 | | nfl-draft | monorepo | 3 | 0 | | nfl-stat-leaders | monorepo | 8 | 0 | -| nrl-scoreboard | monorepo | 30 | 0 | -| odds-ticker | monorepo | 9 | 0 | +| nrl-scoreboard | monorepo | 31 | 1 | +| odds-ticker | monorepo | 10 | 0 | | of-the-day | monorepo | 14 | 0 | -| olympics | monorepo | 16 | 1 | -| on-air | monorepo | 2 | 0 | +| olympics | monorepo | 16 | 0 | +| on-air | monorepo | 3 | 0 | | pomodoro-timer | monorepo | 3 | 0 | -| soccer-scoreboard | monorepo | 47 | 0 | -| static-image | monorepo | 3 | 0 | -| stock-news | monorepo | 3 | 0 | +| soccer-scoreboard | monorepo | 50 | 0 | +| static-image | monorepo | 4 | 0 | +| stock-news | monorepo | 4 | 0 | | text-display | monorepo | 4 | 0 | | tide-display | monorepo | 3 | 0 | -| ufc-scoreboard | monorepo | 38 | 0 | +| ufc-scoreboard | monorepo | 40 | 3 | | web-ui-info | monorepo | 2 | 0 | | youtube-stats | monorepo | 5 | 0 | | f1-live | third-party | 10 | 0 | diff --git a/docs/MULTI_ROOT_WORKSPACE_SETUP.md b/docs/MULTI_ROOT_WORKSPACE_SETUP.md index 17e0fe09..34a37295 100644 --- a/docs/MULTI_ROOT_WORKSPACE_SETUP.md +++ b/docs/MULTI_ROOT_WORKSPACE_SETUP.md @@ -44,8 +44,8 @@ and symlink the plugin directories you are working on into LEDMatrix's ### 1. The plugin monorepo Clone ledmatrix-plugins into the same parent directory as LEDMatrix (the -workspace file and `scripts/update_plugin_repos.py` look for -`../ledmatrix-plugins` relative to the LEDMatrix root): +workspace file looks for `../ledmatrix-plugins` relative to the LEDMatrix +root): ```bash cd ~/Github @@ -86,7 +86,7 @@ the plugin from there. See the ```bash cd ~/Github/LEDMatrix -python3 scripts/update_plugin_repos.py # git pull in ../ledmatrix-plugins +git -C ../ledmatrix-plugins pull # the sibling monorepo checkout # or ./scripts/dev/dev_plugin_setup.sh update # git pull in every linked checkout ``` diff --git a/mypy-clean.txt b/mypy-clean.txt index fa11e729..a5b8fae0 100644 --- a/mypy-clean.txt +++ b/mypy-clean.txt @@ -14,7 +14,6 @@ src/auto_update_setup.py src/backup_manager.py src/base_odds_manager.py src/cache/__init__.py -src/cache/cache_metrics.py src/cache/cache_strategy.py src/cache/memory_cache.py src/common/__init__.py diff --git a/requirements-test.txt b/requirements-test.txt index e1f7dfc6..cf19ded6 100644 --- a/requirements-test.txt +++ b/requirements-test.txt @@ -2,7 +2,6 @@ # Install alongside requirements.txt: pip install -r requirements.txt -r requirements-test.txt pytest>=9.0.3,<10.0.0 pytest-cov>=4.1.0,<8.0.0 -pytest-mock>=3.11.0,<4.0.0 freezegun>=1.2,<2 # deterministic time for golden-image tests psutil>=6.0.0,<7.0.0 # optional at runtime; installed for tests so the # /system/status endpoint's real path is exercised diff --git a/scripts/README.md b/scripts/README.md index 3f1ad3e2..1dba9b67 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -15,7 +15,7 @@ display; **diagnostic** — run by hand on a Pi when something is wrong. | [`install/`](install/README.md) | keep | The installers: one-shot, services, sudoers/WiFi permissions, cache setup, and the shared `lib_*.sh` helpers `first_time_install.sh` sources | | [`fix_perms/`](fix_perms/README.md) | keep | Permission repair scripts, plus the two root helpers the web interface runs through sudo (`safe_plugin_rm.sh`, `safe_pip_install.sh`) | | [`utils/`](utils/README.md) | keep | Scripts run by systemd units or the web interface (conditional web start, WiFi monitor, update verify, DNS fix, Pixlet config editor, cache clearing) | -| [`dev/`](dev/README.md) | dev-only | Plugin linking, emulator runner, Vegas density audit, Pillow smoke test | +| [`dev/`](dev/README.md) | dev-only | Plugin linking, Vegas density audit, Pillow smoke test | | `templates/` | dev-only | `dev_preview.html`, the page `dev_server.py` serves | ## Top-level scripts @@ -43,22 +43,18 @@ display; **diagnostic** — run by hand on a Pi when something is wrong. | `scroll_speeds.py` | keep | Shows and tries the scroll speeds your panel can display cleanly | | `sports_drift_report.py` | keep | Counts the different bodies of each method across the nine scoreboards in a `ledmatrix-plugins` checkout (report-only CI job; docs/SPORTS_UNIFICATION.md) | | `troubleshoot_captive_portal.sh` | diagnostic | Troubleshoots captive-portal WiFi setup after you can SSH back in | -| `update_plugin_repos.py` | dev-only | Pulls the latest `ledmatrix-plugins` monorepo | | `verify_installation.sh` | diagnostic | Checks that an installation completed correctly | | `verify_wifi_setup.sh` | diagnostic | Health check of the WiFi management setup | -## Candidates for removal +## Hand-run tools nothing else references Nothing in the repo (docs, CI, tests, other scripts or code) refers to these. -They are kept for now; each one needs an owner decision before it goes. +They are run by hand and were kept by owner decision (October 2026); the +old one-off schema fixers and WiFi test scripts listed here were removed. | Script | What it does | |---|---| -| `add_defaults_to_schemas.py` | One-off: adds missing `default` values to plugin config schemas | -| `analyze_plugin_schemas.py` | One-off: reports duplicate/inconsistent fields across plugin schemas | | `audit_plugins.py` | AST security audit of plugin code; says it is "designed to run in CI" but no workflow runs it | | `audit_render_path.py` | Finds blocking calls reachable from a plugin's `display()` | | `sports_scroll_check.py` | Drives a sports scoreboard scroll on the panel and reports its pacing | -| `test_captive_portal.sh` | Tests the captive portal from a device connected to the AP | -| `verify_wifi_before_testing.sh` | Pre-flight check before unplugging Ethernet to test WiFi | | `dev/test_pillow_compat.py` | Pillow API smoke test to run after upgrading Pillow | diff --git a/scripts/add_defaults_to_schemas.py b/scripts/add_defaults_to_schemas.py deleted file mode 100755 index 6433269a..00000000 --- a/scripts/add_defaults_to_schemas.py +++ /dev/null @@ -1,231 +0,0 @@ -#!/usr/bin/env python3 -""" -Script to add default values to plugin config schemas where missing. - -This ensures that configs never start with None values, improving user experience -and preventing validation errors. -""" - -import json -import sys -from pathlib import Path -from typing import Any, Dict, List - - -def get_default_for_field(prop: Dict[str, Any]) -> Any: - """ - Determine a sensible default value for a field based on its type and constraints. - - Args: - prop: Field property schema - - Returns: - Default value or None if no default should be added - """ - prop_type = prop.get('type') - - # Handle union types (array with multiple types) - if isinstance(prop_type, list): - # Use the first non-null type - prop_type = next((t for t in prop_type if t != 'null'), prop_type[0] if prop_type else 'string') - - if prop_type == 'boolean': - return False - - elif prop_type == 'number': - # For numbers, use minimum if available, or a sensible default - minimum = prop.get('minimum') - maximum = prop.get('maximum') - - if minimum is not None: - return minimum - elif maximum is not None: - # Use a reasonable fraction of max (like 30% or minimum 1) - return max(1, int(maximum * 0.3)) - else: - # No constraints, use 0 - return 0 - - elif prop_type == 'integer': - # Similar to number - minimum = prop.get('minimum') - maximum = prop.get('maximum') - - if minimum is not None: - return minimum - elif maximum is not None: - return max(1, int(maximum * 0.3)) - else: - return 0 - - elif prop_type == 'string': - # Only add default for strings if it makes sense - # Check if there's an enum - use first value - enum_values = prop.get('enum') - if enum_values: - return enum_values[0] - - # For optional string fields, empty string might be okay, but be cautious - # We'll skip adding defaults for strings unless explicitly needed - return None - - elif prop_type == 'array': - # Empty array as default - return [] - - elif prop_type == 'object': - # Empty object - but we'll handle nested objects separately - return {} - - return None - - -def should_add_default(prop: Dict[str, Any], field_path: str) -> bool: - """ - Determine if we should add a default value to this field. - - Args: - prop: Field property schema - field_path: Dot-separated path to the field - - Returns: - True if default should be added - """ - # Skip if already has a default - if 'default' in prop: - return False - - # Skip secret fields (they should be user-provided) - if prop.get('x-secret', False): - return False - - # Skip API keys and similar sensitive fields - field_name = field_path.split('.')[-1].lower() - sensitive_keywords = ['key', 'password', 'secret', 'token', 'auth', 'credential'] - if any(keyword in field_name for keyword in sensitive_keywords): - return False - - prop_type = prop.get('type') - if isinstance(prop_type, list): - prop_type = next((t for t in prop_type if t != 'null'), prop_type[0] if prop_type else None) - - # Only add defaults for certain types - if prop_type in ('boolean', 'number', 'integer', 'array'): - return True - - # For strings, only if there's an enum - if prop_type == 'string' and 'enum' in prop: - return True - - return False - - -def add_defaults_recursive(schema: Dict[str, Any], path: str = "", modified: List[str] = None) -> bool: - """ - Recursively add default values to schema fields. - - Args: - schema: Schema dictionary to modify - path: Current path in the schema (for logging) - modified: List to track which fields were modified - - Returns: - True if any modifications were made - """ - if modified is None: - modified = [] - - if not isinstance(schema, dict) or 'properties' not in schema: - return False - - changes_made = False - - for key, prop in schema['properties'].items(): - if not isinstance(prop, dict): - continue - - current_path = f"{path}.{key}" if path else key - - # Check nested objects - if prop.get('type') == 'object' and 'properties' in prop: - if add_defaults_recursive(prop, current_path, modified): - changes_made = True - - # Add default if appropriate - if should_add_default(prop, current_path): - default_value = get_default_for_field(prop) - if default_value is not None: - prop['default'] = default_value - modified.append(current_path) - changes_made = True - print(f" Added default to {current_path}: {default_value} (type: {prop.get('type')})") - - return changes_made - - -def process_schema_file(schema_path: Path) -> bool: - """ - Process a single schema file to add defaults. - - Args: - schema_path: Path to the schema file - - Returns: - True if file was modified - """ - print(f"\nProcessing: {schema_path}") - - try: - with open(schema_path, 'r', encoding='utf-8') as f: - schema = json.load(f) - except Exception as e: - print(f" Error reading schema: {e}") - return False - - modified_fields = [] - changes_made = add_defaults_recursive(schema, modified=modified_fields) - - if changes_made: - # Write back with pretty formatting - with open(schema_path, 'w', encoding='utf-8') as f: - json.dump(schema, f, indent=2, ensure_ascii=False) - f.write('\n') # Add trailing newline - - print(f" ✓ Modified {len(modified_fields)} fields") - return True - else: - print(" ✓ No changes needed") - return False - - -def main(): - """Main entry point.""" - project_root = Path(__file__).parent.parent - plugins_dir = project_root / 'plugin-repos' - - if not plugins_dir.exists(): - print(f"Error: Plugins directory not found: {plugins_dir}") - sys.exit(1) - - # Find all config_schema.json files - schema_files = list(plugins_dir.rglob('config_schema.json')) - - if not schema_files: - print("No config_schema.json files found") - sys.exit(0) - - print(f"Found {len(schema_files)} schema files") - - modified_count = 0 - for schema_file in sorted(schema_files): - if process_schema_file(schema_file): - modified_count += 1 - - print(f"\n{'='*60}") - print(f"Summary: Modified {modified_count} out of {len(schema_files)} schema files") - print(f"{'='*60}") - - -if __name__ == '__main__': - main() - diff --git a/scripts/analyze_plugin_schemas.py b/scripts/analyze_plugin_schemas.py deleted file mode 100755 index 6720dd8c..00000000 --- a/scripts/analyze_plugin_schemas.py +++ /dev/null @@ -1,279 +0,0 @@ -#!/usr/bin/env python3 -""" -Analyze all plugin config schemas to identify issues: -- Duplicate fields -- Inconsistencies -- Missing common fields -- Naming variations -- Formatting issues -""" - -import json -from pathlib import Path -from typing import Dict, List, Any -import jsonschema -from jsonschema import Draft7Validator - -# Standard common fields that should be in all plugins -STANDARD_COMMON_FIELDS = { - "enabled": { - "type": "boolean", - "default": False, - "description": "Enable or disable this plugin", - "required": True, - "order": 1 - }, - "display_duration": { - "type": "number", - "default": 15, - "minimum": 1, - "maximum": 300, - "description": "How long to display this plugin in seconds", - "order": 2 - }, - "live_priority": { - "type": "boolean", - "default": False, - "description": "Enable live priority takeover when plugin has live content", - "order": 3 - }, - "high_performance_transitions": { - "type": "boolean", - "default": False, - "description": "Use high-performance transitions (120 FPS) instead of standard (30 FPS)", - "order": 4 - }, - "update_interval": { - "type": "integer", - "default": 60, - "minimum": 1, - "description": "How often to refresh data in seconds", - "order": 5 - }, - "transition": { - "type": "object", - "order": 6 - } -} - -def find_duplicate_fields(schema: Dict[str, Any], path: str = "") -> List[str]: - """Find duplicate field definitions within a schema.""" - duplicates = [] - seen_fields = {} - - def check_properties(props: Dict[str, Any], current_path: str): - if not isinstance(props, dict): - return - - for key, value in props.items(): - full_path = f"{current_path}.{key}" if current_path else key - if key in seen_fields: - duplicates.append(f"Duplicate field '{key}' at {full_path} (also at {seen_fields[key]})") - else: - seen_fields[key] = full_path - - # Recursively check nested objects - if isinstance(value, dict): - if "properties" in value: - check_properties(value["properties"], full_path) - elif "items" in value and isinstance(value["items"], dict): - if "properties" in value["items"]: - check_properties(value["items"]["properties"], f"{full_path}[items]") - - if "properties" in schema: - check_properties(schema["properties"], "") - - return duplicates - -def validate_schema_syntax(schema_path: Path) -> tuple[bool, List[str]]: - """Validate JSON Schema syntax.""" - try: - with open(schema_path, 'r', encoding='utf-8') as f: - schema = json.load(f) - - # Validate schema structure - Draft7Validator.check_schema(schema) - return True, [] - except json.JSONDecodeError as e: - return False, [f"JSON syntax error: {str(e)}"] - except jsonschema.SchemaError as e: - return False, [f"Schema validation error: {str(e)}"] - except Exception as e: - return False, [f"Error: {str(e)}"] - -def analyze_schema(schema_path: Path) -> Dict[str, Any]: - """Analyze a single schema file.""" - plugin_id = schema_path.parent.name - analysis = { - "plugin_id": plugin_id, - "path": str(schema_path), - "valid": False, - "errors": [], - "warnings": [], - "has_title": False, - "has_description": False, - "common_fields": {}, - "missing_common_fields": [], - "naming_issues": [], - "duplicates": [], - "property_order": [], - "update_interval_variant": None - } - - try: - with open(schema_path, 'r', encoding='utf-8') as f: - schema = json.load(f) - - # Check for title and description - analysis["has_title"] = "title" in schema - analysis["has_description"] = "description" in schema - - if not analysis["has_title"]: - analysis["warnings"].append("Missing 'title' field at root level") - if not analysis["has_description"]: - analysis["warnings"].append("Missing 'description' field at root level") - - # Validate schema syntax - is_valid, errors = validate_schema_syntax(schema_path) - analysis["valid"] = is_valid - analysis["errors"].extend(errors) - - if not is_valid: - return analysis - - # Check for duplicate fields - duplicates = find_duplicate_fields(schema) - analysis["duplicates"] = duplicates - - # Check properties - if "properties" not in schema: - analysis["errors"].append("Missing 'properties' field") - return analysis - - properties = schema["properties"] - - # Check common fields - for field_name, field_spec in STANDARD_COMMON_FIELDS.items(): - if field_name in properties: - analysis["common_fields"][field_name] = properties[field_name] - else: - # Check for variants - if field_name == "update_interval": - # Check for update_interval_seconds variant - if "update_interval_seconds" in properties: - analysis["update_interval_variant"] = "update_interval_seconds" - analysis["naming_issues"].append( - "Uses 'update_interval_seconds' instead of 'update_interval'" - ) - else: - analysis["missing_common_fields"].append(field_name) - else: - analysis["missing_common_fields"].append(field_name) - - # Check property order (enabled should be first) - prop_keys = list(properties.keys()) - analysis["property_order"] = prop_keys - - if prop_keys and prop_keys[0] != "enabled": - analysis["warnings"].append( - f"'enabled' is not first property. First property is '{prop_keys[0]}'" - ) - - # Check for required fields - required = schema.get("required", []) - if "enabled" not in required: - analysis["warnings"].append("'enabled' is not in required fields") - - except Exception as e: - analysis["errors"].append(f"Failed to analyze schema: {str(e)}") - - return analysis - -def main(): - """Main analysis function.""" - project_root = Path(__file__).parent.parent - plugins_dir = project_root / "plugin-repos" - - if not plugins_dir.exists(): - print(f"Plugins directory not found: {plugins_dir}") - return - - results = [] - - # Find all config_schema.json files - schema_files = list(plugins_dir.glob("*/config_schema.json")) - - print(f"Found {len(schema_files)} plugin schemas to analyze\n") - - for schema_path in sorted(schema_files): - print(f"Analyzing {schema_path.parent.name}...") - analysis = analyze_schema(schema_path) - results.append(analysis) - - # Print summary - print("\n" + "="*80) - print("ANALYSIS SUMMARY") - print("="*80) - - for result in results: - print(f"\n{result['plugin_id']}:") - print(f" Valid: {result['valid']}") - - if result['errors']: - print(f" Errors ({len(result['errors'])}):") - for error in result['errors']: - print(f" - {error}") - - if result['warnings']: - print(f" Warnings ({len(result['warnings'])}):") - for warning in result['warnings']: - print(f" - {warning}") - - if result['duplicates']: - print(f" Duplicates ({len(result['duplicates'])}):") - for dup in result['duplicates']: - print(f" - {dup}") - - if result['missing_common_fields']: - print(f" Missing common fields: {', '.join(result['missing_common_fields'])}") - - if result['naming_issues']: - print(" Naming issues:") - for issue in result['naming_issues']: - print(f" - {issue}") - - if result['property_order'] and result['property_order'][0] != 'enabled': - print(f" Property order: First is '{result['property_order'][0]}' (should be 'enabled')") - - # Overall statistics - print("\n" + "="*80) - print("OVERALL STATISTICS") - print("="*80) - - valid_count = sum(1 for r in results if r['valid']) - has_title_count = sum(1 for r in results if r['has_title']) - has_description_count = sum(1 for r in results if r['has_description']) - enabled_first_count = sum(1 for r in results if r['property_order'] and r['property_order'][0] == 'enabled') - total_errors = sum(len(r['errors']) for r in results) - total_warnings = sum(len(r['warnings']) for r in results) - total_duplicates = sum(len(r['duplicates']) for r in results) - - print(f"Total plugins: {len(results)}") - print(f"Valid schemas: {valid_count}/{len(results)}") - print(f"Has title: {has_title_count}/{len(results)}") - print(f"Has description: {has_description_count}/{len(results)}") - print(f"'enabled' first: {enabled_first_count}/{len(results)}") - print(f"Total errors: {total_errors}") - print(f"Total warnings: {total_warnings}") - print(f"Total duplicates: {total_duplicates}") - - # Save detailed report - report_path = project_root / "plugin_schema_analysis.json" - with open(report_path, 'w', encoding='utf-8') as f: - json.dump(results, f, indent=2) - - print(f"\nDetailed report saved to: {report_path}") - -if __name__ == "__main__": - main() - diff --git a/scripts/dev/README.md b/scripts/dev/README.md index 6dfdc676..c07e1b85 100644 --- a/scripts/dev/README.md +++ b/scripts/dev/README.md @@ -5,7 +5,6 @@ This directory contains scripts and utilities for development and testing. ## Scripts - **`dev_plugin_setup.sh`** - Sets up plugin development environment by linking plugin repositories -- **`run_emulator.sh`** - Runs the LED Matrix display in emulator mode (for development without hardware) - **`vegas_audit.py`** - Measures how much of the Vegas ticker strip actually shows content (dead-frame ratio) - **`test_pillow_compat.py`** - Pillow API smoke test to run after upgrading Pillow (`python3 scripts/dev/test_pillow_compat.py`) @@ -28,6 +27,6 @@ links. To use a fork or another clone location, copy ### Running Emulator ```bash -./scripts/dev/run_emulator.sh +python3 run.py -e ``` diff --git a/scripts/dev/run_emulator.sh b/scripts/dev/run_emulator.sh deleted file mode 100755 index 66d0fda8..00000000 --- a/scripts/dev/run_emulator.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash -# LEDMatrix Emulator Runner -# This script runs the LEDMatrix system in emulator mode for development and testing - -echo "Starting LEDMatrix Emulator..." -echo "Press Ctrl+C to stop" -echo "" - -# Set emulator mode -export EMULATOR=true - -# Run the main application -python3 run.py diff --git a/scripts/plugin_api_usage.py b/scripts/plugin_api_usage.py index 0d7fd1eb..69455249 100644 --- a/scripts/plugin_api_usage.py +++ b/scripts/plugin_api_usage.py @@ -75,6 +75,14 @@ OWNER_RECEIVERS: Dict[str, Tuple[str, ...]] = { "DisplayManager": ("display_manager", "display_mgr", "displaymanager", "display", "dm"), "FontManager": ("font_manager", "font_mgr", "fontmanager", "fonts", "fm"), "PluginManager": ("plugin_manager", "plugin_mgr", "pluginmanager", "pm"), + "PluginStateManager": ("state_manager", "plugin_state", "state_mgr"), + "ConfigManager": ("config_manager", "config_mgr", "configmanager"), + "LogoDownloader": ("logo_downloader", "downloader", "logodownloader"), + "APIHelper": ("api_helper", "apihelper", "api"), + "BackgroundDataService": ("background_service", "background_data_service", "bg_service", + "data_service"), + "BaseOddsManager": ("odds_manager", "oddsmanager", "odds"), + "DynamicTeamResolver": ("dynamic_resolver", "team_resolver", "resolver"), } #: Directories never scanned (vendored environments, VCS metadata, caches). @@ -330,8 +338,7 @@ class _Scanner(ast.NodeVisitor): return "call" definers = self.local_definers.get(m.method, ()) if name in definers or self.built.get(name or "") in definers: - # e.g. the weather plugin's WeatherIcons.draw_sun, or - # self._strategy_component = CacheStrategy(); ...get_sport_live_interval() + # e.g. the weather plugin's WeatherIcons.draw_sun return "unrelated" return "review" diff --git a/scripts/test_captive_portal.sh b/scripts/test_captive_portal.sh deleted file mode 100755 index d30a1bb6..00000000 --- a/scripts/test_captive_portal.sh +++ /dev/null @@ -1,149 +0,0 @@ -#!/bin/bash - -# Test script for captive portal functionality -# This script tests the captive portal from a device connected to the AP network - -set -e - -PI_IP="192.168.4.1" -PI_PORT="5000" -BASE_URL="http://${PI_IP}:${PI_PORT}" - -echo "==========================================" -echo "Captive Portal Functionality Test" -echo "==========================================" -echo "" -echo "Make sure you're connected to 'LEDMatrix-Setup' network" -echo "Pi IP: ${PI_IP}" -echo "Web Interface Port: ${PI_PORT}" -echo "" - -# Colors for output -GREEN='\033[0;32m' -RED='\033[0;31m' -YELLOW='\033[1;33m' -NC='\033[0m' # No Color - -# Test counter -PASSED=0 -FAILED=0 - -test_result() { - if [ $1 -eq 0 ]; then - echo -e "${GREEN}✓${NC} $2" - ((PASSED++)) - else - echo -e "${RED}✗${NC} $2" - ((FAILED++)) - fi -} - -# Test 1: Check if Pi is reachable -echo "1. Testing Pi connectivity..." -if ping -c 1 -W 2 ${PI_IP} > /dev/null 2>&1; then - test_result 0 "Pi is reachable at ${PI_IP}" -else - test_result 1 "Pi is NOT reachable at ${PI_IP}" - echo " Make sure you're connected to LEDMatrix-Setup network" - exit 1 -fi - -# Test 2: DNS Redirection -echo "" -echo "2. Testing DNS redirection..." -DNS_RESULT=$(nslookup google.com 2>/dev/null | grep -i "address" | tail -1 | awk '{print $2}') -if [ "$DNS_RESULT" = "${PI_IP}" ]; then - test_result 0 "DNS redirection works (google.com resolves to ${PI_IP})" -else - test_result 1 "DNS redirection failed (got ${DNS_RESULT}, expected ${PI_IP})" -fi - -# Test 3: HTTP Redirect -echo "" -echo "3. Testing HTTP redirect..." -HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -L --max-time 5 "${BASE_URL}/google.com" 2>/dev/null || echo "000") -if [ "$HTTP_CODE" = "200" ]; then - test_result 0 "HTTP redirect works (got 200, redirected to setup page)" -else - test_result 1 "HTTP redirect failed (got ${HTTP_CODE})" -fi - -# Test 4: Captive Portal Detection Endpoints -echo "" -echo "4. Testing captive portal detection endpoints..." - -# iOS/macOS -IOS_RESPONSE=$(curl -s --max-time 5 "${BASE_URL}/hotspot-detect.html" 2>/dev/null || echo "") -if echo "$IOS_RESPONSE" | grep -qi "success"; then - test_result 0 "iOS/macOS endpoint works" -else - test_result 1 "iOS/macOS endpoint failed" -fi - -# Android -ANDROID_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "${BASE_URL}/generate_204" 2>/dev/null || echo "000") -if [ "$ANDROID_CODE" = "204" ]; then - test_result 0 "Android endpoint works" -else - test_result 1 "Android endpoint failed (got ${ANDROID_CODE})" -fi - -# Windows -WIN_RESPONSE=$(curl -s --max-time 5 "${BASE_URL}/connecttest.txt" 2>/dev/null || echo "") -if echo "$WIN_RESPONSE" | grep -qi "microsoft"; then - test_result 0 "Windows endpoint works" -else - test_result 1 "Windows endpoint failed" -fi - -# Firefox -FF_RESPONSE=$(curl -s --max-time 5 "${BASE_URL}/success.txt" 2>/dev/null || echo "") -if echo "$FF_RESPONSE" | grep -qi "success"; then - test_result 0 "Firefox endpoint works" -else - test_result 1 "Firefox endpoint failed" -fi - -# Test 5: API Endpoints (should NOT redirect) -echo "" -echo "5. Testing API endpoints (should work normally)..." -API_RESPONSE=$(curl -s --max-time 5 "${BASE_URL}/api/v3/wifi/status" 2>/dev/null || echo "") -if echo "$API_RESPONSE" | grep -qi "status"; then - test_result 0 "API endpoints work (not redirected)" -else - test_result 1 "API endpoints failed or were redirected" -fi - -# Test 6: Main Interface (should be accessible) -echo "" -echo "6. Testing main interface accessibility..." -MAIN_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "${BASE_URL}/v3" 2>/dev/null || echo "000") -if [ "$MAIN_CODE" = "200" ]; then - test_result 0 "Main interface is accessible" -else - test_result 1 "Main interface failed (got ${MAIN_CODE})" -fi - -# Summary -echo "" -echo "==========================================" -echo "Test Summary" -echo "==========================================" -echo -e "${GREEN}Passed: ${PASSED}${NC}" -echo -e "${RED}Failed: ${FAILED}${NC}" -echo "" - -if [ $FAILED -eq 0 ]; then - echo -e "${GREEN}All tests passed! Captive portal is working correctly.${NC}" - exit 0 -else - echo -e "${YELLOW}Some tests failed. Check the output above for details.${NC}" - echo "" - echo "Troubleshooting tips:" - echo "1. Verify AP mode is active: sudo systemctl status hostapd" - echo "2. Check dnsmasq config: sudo cat /etc/dnsmasq.conf" - echo "3. Check web interface logs: sudo journalctl -u ledmatrix-web -n 50" - echo "4. Verify you're connected to LEDMatrix-Setup network" - exit 1 -fi - diff --git a/scripts/update_plugin_repos.py b/scripts/update_plugin_repos.py deleted file mode 100755 index 653e4e2e..00000000 --- a/scripts/update_plugin_repos.py +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env python3 -""" -Update the ledmatrix-plugins monorepo by pulling latest changes. -""" - -import subprocess -import sys -from pathlib import Path - -MONOREPO_DIR = Path(__file__).parent.parent.parent / "ledmatrix-plugins" - - -def main(): - if not MONOREPO_DIR.exists(): - print(f"Error: Monorepo not found: {MONOREPO_DIR}") - return 1 - - if not (MONOREPO_DIR / ".git").exists(): - print(f"Error: {MONOREPO_DIR} is not a git repository") - return 1 - - print(f"Updating {MONOREPO_DIR}...") - try: - result = subprocess.run( - ["git", "-C", str(MONOREPO_DIR), "pull"], - capture_output=True, - text=True, - timeout=120, - ) - except subprocess.TimeoutExpired: - print(f"Error: git pull timed out after 120 seconds for {MONOREPO_DIR}") - return 1 - - if result.returncode == 0: - print(result.stdout.strip()) - return 0 - else: - print(f"Error: {result.stderr.strip()}") - return 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/verify_wifi_before_testing.sh b/scripts/verify_wifi_before_testing.sh deleted file mode 100755 index e29b734d..00000000 --- a/scripts/verify_wifi_before_testing.sh +++ /dev/null @@ -1,225 +0,0 @@ -#!/bin/bash -# Pre-Testing WiFi Verification Script -# Run this BEFORE disconnecting Ethernet to ensure WiFi is ready - -# Don't use set -e as it can cause premature exits with arithmetic operations -# Instead, we'll check return codes explicitly where needed -set -u # Fail on undefined variables - -echo "==========================================" -echo "WiFi Pre-Testing Verification" -echo "==========================================" -echo "" -echo "This script verifies WiFi is enabled and working" -echo "before you disconnect Ethernet for captive portal testing." -echo "" - -# Colors -GREEN='\033[0;32m' -RED='\033[0;31m' -YELLOW='\033[1;33m' -NC='\033[0m' - -# Check counter -PASSED=0 -FAILED=0 -WARNINGS=0 - -check_result() { - local result=$1 - local message=$2 - if [ $result -eq 0 ]; then - echo -e "${GREEN}✓${NC} $message" - PASSED=$((PASSED + 1)) - else - echo -e "${RED}✗${NC} $message" - FAILED=$((FAILED + 1)) - fi -} - -warn_result() { - local message=$2 - echo -e "${YELLOW}⚠${NC} $message" - WARNINGS=$((WARNINGS + 1)) -} - -# Check 1: WiFi interface exists -echo "1. Checking WiFi interface..." -if ip link show wlan0 > /dev/null 2>&1; then - check_result 0 "WiFi interface wlan0 exists" -else - check_result 1 "WiFi interface wlan0 NOT found" - echo " → Check if WiFi adapter is connected" - echo " → Run: lsusb (for USB WiFi) or check built-in WiFi" - exit 1 -fi - -# Check 2: WiFi radio is enabled -echo "" -echo "2. Checking WiFi radio status..." -WIFI_STATUS=$(nmcli radio wifi 2>/dev/null || echo "unknown") -if echo "$WIFI_STATUS" | grep -qi "enabled"; then - check_result 0 "WiFi radio is enabled" -elif echo "$WIFI_STATUS" | grep -qi "disabled"; then - check_result 1 "WiFi radio is DISABLED" - echo " → Enabling WiFi..." - sudo nmcli radio wifi on - sleep 2 - if nmcli radio wifi | grep -qi "enabled"; then - check_result 0 "WiFi radio enabled successfully" - else - check_result 1 "Failed to enable WiFi radio" - exit 1 - fi -else - warn_result 1 "Could not determine WiFi radio status" -fi - -# Check 3: WiFi can scan for networks -echo "" -echo "3. Testing WiFi scanning capability..." -SCAN_RESULT=$(timeout 10 nmcli device wifi list 2>&1 | head -5) -if [ $? -eq 0 ] && [ -n "$SCAN_RESULT" ]; then - NETWORK_COUNT=$(echo "$SCAN_RESULT" | wc -l) - if [ "$NETWORK_COUNT" -gt 1 ]; then - check_result 0 "WiFi scanning works (found networks)" - echo " Sample networks found:" - echo "$SCAN_RESULT" | head -3 | sed 's/^/ /' - else - warn_result 1 "WiFi scanning works but no networks found" - echo " → This might be okay if you're in a remote location" - echo " → Make sure you can see networks when you need to connect" - fi -else - check_result 1 "WiFi scanning FAILED" - echo " → WiFi adapter may not be working properly" - echo " → Check: dmesg | grep -i wifi" - exit 1 -fi - -# Check 4: Current network connections -echo "" -echo "4. Checking current network status..." -ETH_STATUS=$(nmcli device status | grep "ethernet" | grep -v "unavailable" | head -1 || echo "") -WIFI_STATUS=$(nmcli device status | grep "wifi" | head -1 || echo "") - -if echo "$ETH_STATUS" | grep -q "connected"; then - ETH_NAME=$(echo "$ETH_STATUS" | awk '{print $1}') - ETH_IP=$(ip addr show $ETH_NAME 2>/dev/null | grep "inet " | awk '{print $2}' | cut -d/ -f1 | head -1) - check_result 0 "Ethernet is connected ($ETH_NAME)" - if [ -n "$ETH_IP" ]; then - echo " Ethernet IP: $ETH_IP" - fi -else - warn_result 1 "Ethernet is NOT connected" - echo " → You may already be on WiFi only" -fi - -if echo "$WIFI_STATUS" | grep -q "connected"; then - WIFI_NAME=$(echo "$WIFI_STATUS" | awk '{print $1}') - WIFI_IP=$(ip addr show $WIFI_NAME 2>/dev/null | grep "inet " | awk '{print $2}' | cut -d/ -f1 | head -1) - WIFI_SSID=$(nmcli -t -f active,ssid dev wifi | grep "^yes:" | cut -d: -f2 | head -1) - check_result 0 "WiFi is connected ($WIFI_NAME)" - if [ -n "$WIFI_SSID" ]; then - echo " Connected to: $WIFI_SSID" - fi - if [ -n "$WIFI_IP" ]; then - echo " WiFi IP: $WIFI_IP" - fi - echo "" - echo " ⚠ You are already connected via WiFi!" - echo " → You may want to disconnect WiFi first to test captive portal" - echo " → Or test from a different device" -else - if echo "$WIFI_STATUS" | grep -q "disconnected"; then - check_result 0 "WiFi is disconnected (ready for AP mode)" - else - warn_result 1 "WiFi status unclear" - fi -fi - -# Check 5: Internet connectivity test -echo "" -echo "5. Testing internet connectivity..." -if ping -c 2 -W 3 8.8.8.8 > /dev/null 2>&1; then - check_result 0 "Internet connectivity working" - echo " → You have internet access via current connection" -else - warn_result 1 "No internet connectivity detected" - echo " → This might be okay if you're testing in isolation" - echo " → But you won't be able to download packages if needed" -fi - -# Check 6: Saved WiFi connections -echo "" -echo "6. Checking saved WiFi connections..." -SAVED_CONNECTIONS=$(nmcli connection show | grep -i wifi | wc -l) -if [ "$SAVED_CONNECTIONS" -gt 0 ]; then - check_result 0 "Found $SAVED_CONNECTIONS saved WiFi connection(s)" - echo " Saved connections:" - nmcli connection show | grep -i wifi | awk '{print " - " $1}' | head -5 - echo "" - echo " → You can reconnect using: sudo nmcli connection up " -else - warn_result 1 "No saved WiFi connections found" - echo " → Make sure you know your WiFi SSID and password" - echo " → You'll need them to reconnect after testing" -fi - -# Check 7: Required services -echo "" -echo "7. Checking required services..." -if systemctl is-active --quiet hostapd 2>/dev/null; then - warn_result 1 "hostapd is already running (AP mode may be active)" -else - check_result 0 "hostapd service is stopped (normal)" -fi - -if systemctl is-active --quiet dnsmasq 2>/dev/null; then - warn_result 1 "dnsmasq is already running (AP mode may be active)" -else - check_result 0 "dnsmasq service is stopped (normal)" -fi - -# Check 8: WiFi monitor service -echo "" -echo "8. Checking WiFi monitor service..." -if systemctl is-active --quiet ledmatrix-wifi-monitor 2>/dev/null; then - check_result 0 "WiFi monitor service is running" -else - warn_result 1 "WiFi monitor service is NOT running" - echo " → Start with: sudo systemctl start ledmatrix-wifi-monitor" -fi - -# Summary -echo "" -echo "==========================================" -echo "Verification Summary" -echo "==========================================" -echo -e "${GREEN}Passed: ${PASSED}${NC}" -echo -e "${YELLOW}Warnings: ${WARNINGS}${NC}" -echo -e "${RED}Failed: ${FAILED}${NC}" -echo "" - -if [ $FAILED -eq 0 ]; then - if [ $WARNINGS -eq 0 ]; then - echo -e "${GREEN}✓ All checks passed! WiFi is ready for testing.${NC}" - echo "" - echo "Next steps:" - echo "1. You can safely disconnect Ethernet" - echo "2. Enable AP mode to test captive portal" - echo "3. Use emergency_reconnect.sh if you need to reconnect" - else - echo -e "${YELLOW}⚠ Checks passed with warnings.${NC}" - echo "" - echo "WiFi appears ready, but review warnings above." - echo "You can proceed with testing, but be aware of the warnings." - fi - exit 0 -else - echo -e "${RED}✗ Some checks failed. Please fix issues before testing.${NC}" - echo "" - echo "Do NOT disconnect Ethernet until all issues are resolved!" - exit 1 -fi - diff --git a/src/background_data_service.py b/src/background_data_service.py index 9a8f0250..2c670ad9 100644 --- a/src/background_data_service.py +++ b/src/background_data_service.py @@ -43,6 +43,7 @@ from src.common.espn_dates import ( fetch_espn_date_chunks, parse_espn_date_range, ) +from src.deprecation import deprecated # Configure logging logger = logging.getLogger(__name__) @@ -698,6 +699,7 @@ class BackgroundDataService: raise last_exception + @deprecated("3.10.0", "pass callback= to submit_fetch_request()") def get_result(self, request_id: str) -> Optional[FetchResult]: """ Get the result of a fetch request. @@ -714,6 +716,7 @@ class BackgroundDataService: with self._lock: return self.completed_requests.get(request_id) + @deprecated("3.10.0", "pass callback= to submit_fetch_request()") def is_request_complete(self, request_id: str) -> bool: """ Check if a request has completed. @@ -730,6 +733,7 @@ class BackgroundDataService: with self._lock: return request_id in self.completed_requests + @deprecated("3.10.0", "pass callback= to submit_fetch_request()") def get_request_status(self, request_id: str) -> Optional[FetchStatus]: """ Get the status of a fetch request. diff --git a/src/base_odds_manager.py b/src/base_odds_manager.py index 69185f06..845f5c5e 100644 --- a/src/base_odds_manager.py +++ b/src/base_odds_manager.py @@ -21,6 +21,7 @@ from typing import Dict, Any, Optional, List, cast from src.common.api_helper import DEFAULT_HTTP_HEADERS from src.common.fetch_service import fetch_get, share_connection_pool from src.common.json_body import response_json +from src.deprecation import deprecated @@ -277,6 +278,7 @@ class BaseOddsManager: self.logger.warning(f"Unexpected response structure: {json.dumps(data, indent=2)}") return None + @deprecated("3.10.0", "call get_odds() for each game") def get_odds_for_games(self, games: List[Dict[str, Any]]) -> List[Dict[str, Any]]: """ Fetch odds for multiple games efficiently. @@ -335,6 +337,7 @@ class BaseOddsManager: return False + @deprecated("3.10.0") def format_odds_summary(self, odds_data: Optional[Dict[str, Any]]) -> str: """ Format odds data into a human-readable summary. diff --git a/src/cache/__init__.py b/src/cache/__init__.py index a4dc26a9..a86b1986 100644 --- a/src/cache/__init__.py +++ b/src/cache/__init__.py @@ -5,6 +5,5 @@ Provides specialized cache components: - MemoryCache: In-memory caching - DiskCache: Persistent disk caching - CacheStrategy: Cache strategy management -- CacheMetrics: Performance metrics tracking """ diff --git a/src/cache/cache_metrics.py b/src/cache/cache_metrics.py deleted file mode 100644 index 6abd7084..00000000 --- a/src/cache/cache_metrics.py +++ /dev/null @@ -1,134 +0,0 @@ -""" -Cache Metrics - -Tracks cache performance metrics including hit rates, miss rates, and fetch times. -""" - -import threading -import time -import logging -from typing import Dict, Any, Optional - - -class CacheMetrics: - """Tracks cache performance metrics.""" - - def __init__(self, logger: Optional[logging.Logger] = None) -> None: - """ - Initialize cache metrics tracker. - - Args: - logger: Optional logger instance - """ - self.logger = logger or logging.getLogger(__name__) - self._lock = threading.Lock() - self._metrics: Dict[str, Any] = { - 'hits': 0, - 'misses': 0, - 'api_calls_saved': 0, - 'background_hits': 0, - 'background_misses': 0, - 'total_fetch_time': 0.0, - 'fetch_count': 0, - # Disk cleanup metrics - 'last_disk_cleanup': 0.0, - 'total_files_cleaned': 0, - 'total_space_freed_mb': 0.0, - 'last_cleanup_duration_sec': 0.0 - } - - def record_hit(self, cache_type: str = 'regular') -> None: - """ - Record a cache hit. - - Args: - cache_type: Type of cache hit ('regular' or 'background') - """ - with self._lock: - if cache_type == 'background': - self._metrics['background_hits'] += 1 - else: - self._metrics['hits'] += 1 - - def record_miss(self, cache_type: str = 'regular') -> None: - """ - Record a cache miss. - - Args: - cache_type: Type of cache miss ('regular' or 'background') - """ - with self._lock: - if cache_type == 'background': - self._metrics['background_misses'] += 1 - else: - self._metrics['misses'] += 1 - self._metrics['api_calls_saved'] += 1 - - def record_fetch_time(self, duration: float) -> None: - """ - Record fetch operation duration. - - Args: - duration: Duration in seconds - """ - with self._lock: - self._metrics['total_fetch_time'] += duration - self._metrics['fetch_count'] += 1 - - def record_disk_cleanup(self, files_cleaned: int, space_freed_mb: float, duration_sec: float) -> None: - """ - Record disk cleanup operation results. - - Args: - files_cleaned: Number of files deleted - space_freed_mb: Space freed in megabytes - duration_sec: Duration of cleanup operation in seconds - """ - with self._lock: - self._metrics['last_disk_cleanup'] = time.time() - self._metrics['total_files_cleaned'] += files_cleaned - self._metrics['total_space_freed_mb'] += space_freed_mb - self._metrics['last_cleanup_duration_sec'] = duration_sec - - def get_metrics(self) -> Dict[str, Any]: - """ - Get current cache performance metrics. - - Returns: - Dictionary with cache metrics - """ - with self._lock: - total_hits = self._metrics['hits'] + self._metrics['background_hits'] - total_misses = self._metrics['misses'] + self._metrics['background_misses'] - total_requests = total_hits + total_misses - - avg_fetch_time = (self._metrics['total_fetch_time'] / - self._metrics['fetch_count']) if self._metrics['fetch_count'] > 0 else 0.0 - - return { - 'total_requests': total_requests, - 'cache_hit_rate': total_hits / total_requests if total_requests > 0 else 0.0, - 'background_hit_rate': (self._metrics['background_hits'] / - (self._metrics['background_hits'] + self._metrics['background_misses']) - if (self._metrics['background_hits'] + self._metrics['background_misses']) > 0 else 0.0), - 'api_calls_saved': self._metrics['api_calls_saved'], - 'average_fetch_time': avg_fetch_time, - 'total_fetch_time': self._metrics['total_fetch_time'], - 'fetch_count': self._metrics['fetch_count'], - # Disk cleanup metrics - 'last_disk_cleanup': self._metrics['last_disk_cleanup'], - 'total_files_cleaned': self._metrics['total_files_cleaned'], - 'total_space_freed_mb': self._metrics['total_space_freed_mb'], - 'last_cleanup_duration_sec': self._metrics['last_cleanup_duration_sec'] - } - - def log_metrics(self) -> None: - """Log current cache performance metrics.""" - metrics = self.get_metrics() - self.logger.info("Cache Performance - Hit Rate: %.2f%%, Background Hit Rate: %.2f%%, " - "API Calls Saved: %d, Avg Fetch Time: %.2fs", - metrics['cache_hit_rate'] * 100, - metrics['background_hit_rate'] * 100, - metrics['api_calls_saved'], - metrics['average_fetch_time']) - diff --git a/src/cache/cache_strategy.py b/src/cache/cache_strategy.py index 66962495..6e0556aa 100644 --- a/src/cache/cache_strategy.py +++ b/src/cache/cache_strategy.py @@ -4,7 +4,6 @@ Cache Strategy Manages cache strategies (TTLs) for different data types. """ -import logging from typing import Dict, Any, Optional from datetime import datetime import pytz @@ -13,51 +12,25 @@ import pytz class CacheStrategy: """Manages cache strategies for different data types.""" - def __init__(self, config_manager: Optional[Any] = None, logger: Optional[logging.Logger] = None) -> None: - """ - Initialize cache strategy manager. - - Args: - config_manager: Optional ConfigManager instance. Kept for callers - that pass one; no strategy currently reads it. - logger: Optional logger instance - """ - self.config_manager = config_manager - self.logger = logger or logging.getLogger(__name__) - - def get_sport_live_interval(self, sport_key: str) -> int: - """ - Live-data cache interval, in seconds, for a sport: 60 for every sport. - - This used to read ``live_update_interval`` from a ``_scoreboard`` - config section. Those sections belonged to the built-in scoreboards - that the plugin system replaced; plugin config is keyed by plugin id - (``football-scoreboard``), so the lookup always fell back to 60. - - Args: - sport_key: Sport identifier (e.g., 'nba', 'nfl') - - Returns: - Live update interval in seconds - """ - return 60 - def get_cache_strategy(self, data_type: str, sport_key: Optional[str] = None) -> Dict[str, Any]: """ Get cache strategy for different data types. Args: data_type: Type of data (e.g., 'live_scores', 'stocks', 'weather_current') - sport_key: Optional sport key; for live data it selects the - per-sport interval from :meth:`get_sport_live_interval` - instead of the generic live default. + sport_key: Optional sport key; for live data any sport key + selects a 60s interval instead of the generic live default. + (That used to be a per-sport ``live_update_interval`` from + ``_scoreboard`` config sections, which belonged to the + built-in scoreboards the plugin system replaced, so every + lookup fell back to 60.) Returns: Dictionary with cache strategy (max_age, memory_ttl, etc.) """ live_interval = None if sport_key and data_type in ['sports_live', 'live_scores']: - live_interval = self.get_sport_live_interval(sport_key) + live_interval = 60 strategies = { # Ultra time-sensitive data (live scores, current weather) diff --git a/src/cache/disk_cache.py b/src/cache/disk_cache.py index e66f4b10..eb4bc824 100644 --- a/src/cache/disk_cache.py +++ b/src/cache/disk_cache.py @@ -15,11 +15,14 @@ import tempfile import logging import threading import zlib -from typing import Dict, Any, Optional, Protocol, Tuple +from typing import TYPE_CHECKING, Dict, Any, Optional, Tuple from datetime import datetime from src.common.path_safety import safe_path_component +if TYPE_CHECKING: + from src.cache.cache_strategy import CacheStrategy + try: # optional: large speedup on the cache write path, see _dumps below import orjson except ImportError: # pragma: no cover - exercised on hosts without the wheel @@ -62,23 +65,6 @@ def _filename_stem(key: str) -> str: return f"{prefix}-{digest}" - -class CacheStrategyProtocol(Protocol): - """Protocol for cache strategy objects that categorize cache keys.""" - - def get_data_type_from_key(self, key: str) -> str: - """ - Determine the data type from a cache key. - - Args: - key: Cache key - - Returns: - Data type string for strategy lookup - """ - ... - - class DateTimeEncoder(json.JSONEncoder): """JSON encoder that handles datetime objects. @@ -816,12 +802,12 @@ class DiskCache: # mkstemp's random component. return bool(sep) and len(head) > 1 and bool(suffix) - def cleanup_expired_files(self, cache_strategy: CacheStrategyProtocol, retention_policies: Dict[str, int]) -> Dict[str, Any]: + def cleanup_expired_files(self, cache_strategy: 'CacheStrategy', retention_policies: Dict[str, int]) -> Dict[str, Any]: """ Clean up expired cache files based on retention policies. Args: - cache_strategy: Object implementing CacheStrategyProtocol for categorizing files + cache_strategy: Categorizes files by key (get_data_type_from_key) retention_policies: Dict mapping data types to retention days Returns: diff --git a/src/cache_manager.py b/src/cache_manager.py index b922c3e9..318c9a8b 100644 --- a/src/cache_manager.py +++ b/src/cache_manager.py @@ -35,13 +35,13 @@ import tempfile from src.cache.memory_cache import MemoryCache, default_max_size from src.cache.disk_cache import DiskCache from src.cache.cache_strategy import CacheStrategy -from src.cache.cache_metrics import CacheMetrics from src.logging_config import get_logger # Canonical implementation lives in src.cache.disk_cache; re-exported here # because this module's docstring documents it and external code may import # it from either path. from src.cache.disk_cache import DateTimeEncoder # noqa: F401 - deliberate re-export +from src.deprecation import deprecated # CacheManager.config_manager not built yet (None means "not available"). _UNSET: Any = object() @@ -149,10 +149,7 @@ class CacheManager: max_size=default_max_size(), cleanup_interval=300.0 ) self._disk_cache_component = DiskCache(cache_dir=self.cache_dir, logger=self.logger) - # No config manager: CacheStrategy keeps the parameter for callers but - # reads nothing from it, and passing ours would build it eagerly. - self._strategy_component = CacheStrategy(logger=self.logger) - self._metrics_component = CacheMetrics(logger=self.logger) + self._strategy_component = CacheStrategy() # Disk cleanup configuration self._disk_cleanup_interval_hours = 24 # Run cleanup every 24 hours @@ -398,6 +395,7 @@ class CacheManager: # caller gets as is. self._disk_cache_component.set(key, data) + @deprecated("3.10.0", "use get(key, max_age=3600)") def load_cache(self, key: str) -> Optional[Dict[str, Any]]: """Load data from cache with memory caching.""" # Check memory cache first (1 minute TTL) @@ -607,13 +605,6 @@ class CacheManager: duration = time.time() - start_time space_freed_mb = stats['space_freed_bytes'] / (1024 * 1024) - # Record metrics - self._metrics_component.record_disk_cleanup( - files_cleaned=stats['files_deleted'], - space_freed_mb=space_freed_mb, - duration_sec=duration - ) - # Log summary if stats['files_deleted'] > 0: self.logger.info( @@ -796,6 +787,7 @@ class CacheManager: data_type = self.get_data_type_from_key(key) return self.get_cached_data_with_strategy(key, data_type) + @deprecated("3.10.0") def generate_sport_cache_key(self, sport: str, date_str: Optional[str] = None) -> str: """ Centralized cache key generation for sports data. diff --git a/src/common/api_helper.py b/src/common/api_helper.py index ddb26c26..917b43ad 100644 --- a/src/common/api_helper.py +++ b/src/common/api_helper.py @@ -22,6 +22,7 @@ from typing import TYPE_CHECKING, Any, Dict, Mapping, Optional, cast import requests from urllib3.util.retry import Retry +from src.deprecation import deprecated if TYPE_CHECKING: # What Session() puts in .headers; the stubs only promise a MutableMapping. @@ -171,6 +172,7 @@ class APIHelper: self.logger.error(f"Request failed for {url}: {e}") return None + @deprecated("3.10.0", "use src.common.espn_dates.fetch_espn_scoreboard()") def fetch_espn_scoreboard(self, sport: str, league: str, date: Optional[str] = None, cache_key: Optional[str] = None, @@ -227,6 +229,7 @@ class APIHelper: store_espn_scoreboard_cache(self.cache_manager, shared_key, data) return data + @deprecated("3.10.0", "call get() with the ESPN URL") def fetch_espn_standings(self, sport: str, league: str, cache_key: Optional[str] = None, cache_ttl: int = 3600) -> Optional[Dict]: @@ -249,6 +252,7 @@ class APIHelper: return self.get(url, cache_key=cache_key, cache_ttl=cache_ttl) + @deprecated("3.10.0", "call get() with the ESPN URL") def fetch_espn_rankings(self, sport: str, league: str, cache_key: Optional[str] = None, cache_ttl: int = 3600) -> Optional[Dict]: @@ -311,6 +315,7 @@ class APIHelper: self.logger.error(f"POST request failed for {url}: {e}") return None + @deprecated("3.10.0", "use the plugin's cache_manager") def set_cache(self, key: str, data: Any, ttl: int = 3600) -> None: """ Set cache data. @@ -323,6 +328,7 @@ class APIHelper: """ self._set_cache(key, data, ttl) + @deprecated("3.10.0", "use the plugin's cache_manager") def get_cache(self, key: str) -> Optional[Any]: """ Get cached data. @@ -392,6 +398,7 @@ class APIHelper: self._last_request_monotonic = time.monotonic() self._last_request_time = time.time() + @deprecated("3.10.0") def set_rate_limit(self, min_interval: float) -> None: """ Set minimum interval between requests. @@ -402,6 +409,7 @@ class APIHelper: self._min_request_interval = min_interval self.logger.debug(f"Rate limit set to {min_interval} seconds") + @deprecated("3.10.0") def get_request_stats(self) -> Dict[str, Any]: """ Get request statistics. diff --git a/src/common/sync_manager.py b/src/common/sync_manager.py index ce43de34..876c6a1e 100644 --- a/src/common/sync_manager.py +++ b/src/common/sync_manager.py @@ -141,7 +141,6 @@ class DisplaySyncManager: self._last_leader_frame_time: float = 0.0 self._frame_lock = threading.Lock() self._leader_ip: Optional[str] = None - self._on_new_cycle: Optional[Callable[[], None]] = None # called when leader starts new cycle self._on_scroll_image: Optional[Callable[[Image.Image], None]] = None # called with Image when received self._pending_scroll_image: Optional[Image.Image] = None # image received before callback set self._scroll_image_lock = threading.Lock() # guards _on_scroll_image / _pending_scroll_image @@ -412,17 +411,6 @@ class DisplaySyncManager: except Exception as exc: self.logger.debug("Sync: scroll_x send error: %s", exc) - def send_new_cycle(self) -> None: - """Leader: signal that a new scroll cycle has started so follower rebuilds its image.""" - if self.role != SyncRole.LEADER: - return - if self._leader_state != LeaderState.CONNECTED or not self._peer_ip: - return - try: - self._send_sock.sendto(b'{"t":"nc"}', (self._peer_ip, self.port)) - except Exception as exc: - self.logger.debug("Sync: new_cycle send error: %s", exc) - def send_frame(self, image: Image.Image) -> None: """Leader: send a rendered frame to the follower as raw RGB bytes. Raw format is orders of magnitude faster than PNG on Pi hardware — @@ -506,21 +494,19 @@ class DisplaySyncManager: self._latest_frame = img self._enter_follower_mode(sender_ip) - def _enter_follower_mode(self, sender_ip: str) -> bool: + def _enter_follower_mode(self, sender_ip: str) -> None: """Note that the leader at ``sender_ip`` just sent something, and - switch from standalone to follower mode if not already following. - Returns True if this call made the switch.""" + switch from standalone to follower mode if not already following.""" self._last_leader_frame_time = time.monotonic() self._leader_ip = sender_ip if self._follower_state != FollowerState.STANDALONE: - return False + return self._follower_state = FollowerState.FOLLOWER self.logger.info( "Sync: leader active at %s — switching to follower mode", sender_ip, ) self.write_status_file() - return True def _follower_recv_loop(self) -> None: while self._running: @@ -570,12 +556,9 @@ class DisplaySyncManager: # frame. Read and validate its fields under a guard — # a UDP payload is attacker-shaped, so a non-object # body makes .get() raise AttributeError and an "sx" - # carrying a non-numeric x raises ValueError/TypeError - # — but dispatch the callback *outside* it. Running - # the callback in here would let a fault in someone - # else's code read as a malformed packet and be - # logged as one. - fire_new_cycle = False + # carrying a non-numeric x raises ValueError/TypeError. + # Any other "t" is ignored, including the "nc" (new + # cycle) that older leaders send and no follower used. try: t = msg.get("t") if t == "hello_ack": @@ -601,18 +584,11 @@ class DisplaySyncManager: # back from. Treat it as malformed. raise ValueError(f"non-finite scroll x: {msg['x']!r}") self._latest_scroll_x = scroll_x - if self._enter_follower_mode(sender_ip): - fire_new_cycle = True # build initial scroll image - elif t == "nc": - # Leader started a new scroll cycle — rebuild local image - fire_new_cycle = True + self._enter_follower_mode(sender_ip) except (KeyError, AttributeError, TypeError, ValueError) as exc: self.logger.debug("Sync: malformed control message: %s", exc) continue - if fire_new_cycle and self._on_new_cycle: - self._on_new_cycle() - except socket.timeout: continue except Exception as exc: @@ -679,15 +655,6 @@ class DisplaySyncManager: """Follower: return the most recently received Vegas scroll position, or None.""" return self._latest_scroll_x - def set_on_new_cycle(self, callback: Callable[[], None]) -> None: - """Follower: register a callback fired when the leader starts a new scroll cycle. - - Nothing in core registers one: display_controller follows the leader - through set_on_scroll_image() and the scroll position instead of - rebuilding locally. The hook stays for callers that want the signal. - """ - self._on_new_cycle = callback - def get_latest_frame(self) -> Optional[Image.Image]: """Follower: return the most recently received pixel frame (non-Vegas fallback).""" with self._frame_lock: diff --git a/src/config_manager.py b/src/config_manager.py index 25911d2a..f3da2761 100644 --- a/src/config_manager.py +++ b/src/config_manager.py @@ -45,6 +45,7 @@ from src.common.permission_utils import ( ensure_shared_group_ownership, get_config_dir_mode ) +from src.deprecation import deprecated def _private_copy(config: Dict[str, Any]) -> Dict[str, Any]: @@ -174,6 +175,7 @@ class ConfigManager: return result + @deprecated("3.10.0", "backups are handled by src.backup_manager") def rollback_config(self, backup_version: Optional[str] = None) -> bool: """ Rollback configuration to a previous backup. @@ -198,6 +200,7 @@ class ConfigManager: return success + @deprecated("3.10.0", "backups are handled by src.backup_manager") def list_backups(self) -> List[BackupInfo]: """ List all available configuration backups. @@ -208,6 +211,7 @@ class ConfigManager: atomic_mgr = self._get_atomic_manager() return atomic_mgr.list_backups() + @deprecated("3.10.0") def validate_config_file(self, config_path: Optional[str] = None) -> ValidationResult: """ Validate a configuration file. @@ -404,6 +408,7 @@ class ConfigManager: self.logger.error(error_msg, exc_info=True) raise ConfigError(error_msg, config_path=self.config_path) from e + @deprecated("3.10.0", "secrets are merged into each plugin's config; read them with config.get()") def get_secret(self, key: str) -> Optional[Any]: """Get a secret value by key.""" try: @@ -757,6 +762,7 @@ class ConfigManager: self.logger.error(error_msg, exc_info=True) raise ConfigError(error_msg, config_path=self.config_path, field=plugin_id) from e + @deprecated("3.10.0") def cleanup_orphaned_plugin_configs(self, valid_plugin_ids: List[str]) -> List[str]: """ Remove configuration sections for plugins that are no longer installed. @@ -810,6 +816,7 @@ class ConfigManager: self.logger.error(f"Error cleaning up orphaned plugin configs: {e}") return removed + @deprecated("3.10.0") def validate_all_plugin_configs(self, plugin_schema_manager=None) -> Dict[str, Dict[str, Any]]: """ Validate all plugin configurations against their schemas. diff --git a/src/dynamic_team_resolver.py b/src/dynamic_team_resolver.py index c90c9a4f..ed78ad1c 100644 --- a/src/dynamic_team_resolver.py +++ b/src/dynamic_team_resolver.py @@ -24,6 +24,7 @@ from typing import Any, Dict, List from src.common.api_helper import DEFAULT_HTTP_HEADERS from src.common.json_body import response_json +from src.deprecation import deprecated logger = logging.getLogger(__name__) @@ -201,6 +202,7 @@ class DynamicTeamResolver: DynamicTeamResolver._failure_timestamp = current_time return {} + @deprecated("3.10.0", "use resolve_teams()") def get_available_dynamic_teams(self) -> List[str]: """ Get list of available dynamic team names. @@ -210,6 +212,7 @@ class DynamicTeamResolver: """ return list(self.DYNAMIC_PATTERNS.keys()) + @deprecated("3.10.0", "use resolve_teams()") def is_dynamic_team(self, team_name: str) -> bool: """ Check if a team name is a dynamic team. diff --git a/src/error_aggregator.py b/src/error_aggregator.py index 4d100de9..816aadbb 100644 --- a/src/error_aggregator.py +++ b/src/error_aggregator.py @@ -123,8 +123,7 @@ class ErrorAggregator: self._error_counts: Dict[str, int] = defaultdict(int) self._plugin_error_counts: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._patterns: Dict[str, ErrorPattern] = {} - self._pattern_callbacks: List[Callable[[ErrorPattern], None]] = [] - self._lock = threading.RLock() # RLock: build_snapshot and pattern callbacks re-enter + self._lock = threading.RLock() # RLock: build_snapshot re-enters # Track session start for relative timing self._session_start = datetime.now() @@ -238,13 +237,6 @@ class ErrorAggregator: f"{count} times in last {self.pattern_window}. " f"Affected plugins: {set(affected_plugins) or 'unknown'}" ) - - # Notify callbacks - for callback in self._pattern_callbacks: - try: - callback(pattern) - except Exception as e: - self.logger.error(f"Pattern callback failed: {e}") else: # Update existing pattern self._patterns[pattern_key].count = count @@ -253,15 +245,6 @@ class ErrorAggregator: known = self._patterns[pattern_key].affected_plugins known.extend(p for p in affected_plugins if p not in known) - def on_pattern_detected(self, callback: Callable[[ErrorPattern], None]) -> None: - """ - Register a callback to be called when a new error pattern is detected. - - Args: - callback: Function that takes an ErrorPattern as argument - """ - self._pattern_callbacks.append(callback) - def get_error_summary(self) -> Dict[str, Any]: """ Get summary of all errors for reporting. @@ -325,27 +308,6 @@ class ErrorAggregator: "last_error": recent_plugin_errors[-1].to_dict() if recent_plugin_errors else None } - def clear_old_records(self, max_age_hours: int = 24) -> int: - """ - Clear records older than specified age. - - Args: - max_age_hours: Maximum age in hours - - Returns: - Number of records cleared - """ - with self._lock: - cutoff = datetime.now() - timedelta(hours=max_age_hours) - original_count = len(self._records) - self._records = [r for r in self._records if r.timestamp > cutoff] - cleared = original_count - len(self._records) - - if cleared > 0: - self.logger.info(f"Cleared {cleared} old error records") - - return cleared - @property def version(self) -> int: """Changes whenever the recorded errors do (see ErrorSnapshotPublisher).""" @@ -354,7 +316,7 @@ class ErrorAggregator: def clear_before(self, cutoff: datetime) -> int: """Forget every error recorded at or before ``cutoff``. - Unlike clear_old_records, this also resets what the summary reports: + This also resets what the summary reports: the per-type and per-plugin counts are rebuilt from the records that remain, and detected patterns that began before the cutoff are dropped (one that is still happening is detected again on its next diff --git a/src/font_manager.py b/src/font_manager.py index 17990983..1b3ac19f 100644 --- a/src/font_manager.py +++ b/src/font_manager.py @@ -41,6 +41,7 @@ from PIL import ImageFont from src.common.bdf_font import load_bdf_face, read_bdf_native_size from src.common.font_layout import load_truetype, resolve_asset_path from typing import Dict, Tuple, Optional, Union, Any +from src.deprecation import deprecated logger = logging.getLogger(__name__) @@ -533,6 +534,7 @@ class FontManager: """ return load_bdf_face(font_path, size_px)[0] + @deprecated("3.10.0", "use src.common.bdf_font.read_bdf_native_size()") def get_native_bdf_size(self, family: str) -> Optional[int]: """The one true pixel size of a BDF family in the catalog, or None for scalable (TTF) families / unknown families.""" @@ -553,6 +555,7 @@ class FontManager: # ==================== Font Measurement ==================== + @deprecated("3.10.0", "use src.adaptive_layout.measure_ink()") def measure_text(self, text: str, font: Union[ImageFont.FreeTypeFont, freetype.Face]) -> Tuple[int, int, int]: """ Measure text dimensions and baseline. diff --git a/src/ipc/client.py b/src/ipc/client.py index 17ab5a19..9a3631e6 100644 --- a/src/ipc/client.py +++ b/src/ipc/client.py @@ -288,11 +288,6 @@ def errors_clear(request_id: str, cutoff: float, *, timeout=timeout, paths=paths) -def ping(*, timeout: float = DEFAULT_TIMEOUT_SECONDS, - paths: Optional[Sequence[str]] = None) -> Dict[str, Any]: - return request(Command.PING, {}, timeout=timeout, paths=paths) - - def hello(client: str = 'web', *, timeout: float = DEFAULT_TIMEOUT_SECONDS, paths: Optional[Sequence[str]] = None) -> Dict[str, Any]: """Version negotiation: the result's ``version`` is the one both sides speak.""" diff --git a/src/ipc/contract.py b/src/ipc/contract.py index 004dd8bb..20ba869c 100644 --- a/src/ipc/contract.py +++ b/src/ipc/contract.py @@ -399,9 +399,6 @@ class HelloArgs: versions: Tuple[int, ...] = (PROTOCOL_VERSION,) client: str = '' - def to_dict(self) -> Dict[str, Any]: - return {'versions': list(self.versions), 'client': self.client} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'HelloArgs': versions = args.get('versions', [PROTOCOL_VERSION]) @@ -426,10 +423,6 @@ class OnDemandStartArgs: duration: Optional[float] = None pinned: bool = False - def to_dict(self) -> Dict[str, Any]: - return {'plugin_id': self.plugin_id, 'mode': self.mode, - 'duration': self.duration, 'pinned': self.pinned} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'OnDemandStartArgs': plugin_id = _optional_name(args, 'plugin_id') @@ -449,9 +442,6 @@ class OnDemandStartArgs: class OnDemandStopArgs: """``on_demand.stop``: end the on-demand session and resume rotation.""" - def to_dict(self) -> Dict[str, Any]: - return {} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'OnDemandStopArgs': return cls() @@ -461,9 +451,6 @@ class OnDemandStopArgs: class NoArgs: """``ping`` and ``on_demand.status`` take no arguments (extra ones are ignored).""" - def to_dict(self) -> Dict[str, Any]: - return {} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'NoArgs': return cls() @@ -480,9 +467,6 @@ class BrightnessSetArgs: """ brightness: int - def to_dict(self) -> Dict[str, Any]: - return {'brightness': self.brightness} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'BrightnessSetArgs': value = args.get('brightness') @@ -503,9 +487,6 @@ class PluginReloadArgs: """ plugin_id: str - def to_dict(self) -> Dict[str, Any]: - return {'plugin_id': self.plugin_id} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'PluginReloadArgs': plugin_id = _optional_name(args, 'plugin_id') @@ -546,9 +527,6 @@ class StateGetArgs: since: Optional[int] = None epoch: Optional[str] = None - def to_dict(self) -> Dict[str, Any]: - return {'since': self.since, 'epoch': self.epoch} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'StateGetArgs': return cls(since=_optional_version(args, 'since'), epoch=_optional_epoch(args)) @@ -565,9 +543,6 @@ class StateSubscribeArgs: and a ``tick`` at least every :data:`SUBSCRIBE_KEEPALIVE_SECONDS`. """ - def to_dict(self) -> Dict[str, Any]: - return {} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'StateSubscribeArgs': return cls() @@ -582,9 +557,6 @@ class ErrorsClearArgs: """ cutoff: float - def to_dict(self) -> Dict[str, Any]: - return {'cutoff': self.cutoff} - @classmethod def from_dict(cls, args: Mapping[str, Any]) -> 'ErrorsClearArgs': value = args.get('cutoff') diff --git a/src/logo_downloader.py b/src/logo_downloader.py index 22ab83ab..5b372d08 100644 --- a/src/logo_downloader.py +++ b/src/logo_downloader.py @@ -29,6 +29,7 @@ from src.common.permission_utils import ( get_assets_dir_mode, get_assets_file_mode ) +from src.deprecation import deprecated logger = logging.getLogger(__name__) @@ -471,6 +472,7 @@ class LogoDownloader: logger.info(f"Using dynamic ESPN endpoint for custom soccer league: {league}") return api_url + @deprecated("3.10.0", "download logos one at a time with download_missing_logo()") def fetch_teams_data(self, league: str) -> Optional[Dict]: """Fetch team data from ESPN API for a specific league.""" api_url = self._resolve_api_url(league) @@ -518,6 +520,7 @@ class LogoDownloader: logger.error(f"Error parsing JSON response for {team_id} in {league}: {e}") return None + @deprecated("3.10.0", "download logos one at a time with download_missing_logo()") def extract_teams_from_data(self, data: Dict, league: str) -> List[Dict[str, str]]: """Extract team information from ESPN API response.""" teams = [] @@ -621,6 +624,7 @@ class LogoDownloader: # Default to FBS for unknown conferences return 'FBS' + @deprecated("3.10.0", "download logos one at a time with download_missing_logo()") def download_missing_logos_for_league(self, league: str, force_download: bool = False) -> Tuple[int, int]: """Download missing logos for a specific league.""" logger.info(f"Starting logo download for league: {league}") @@ -675,6 +679,7 @@ class LogoDownloader: logger.info(f"Logo download complete for {league}: {downloaded_count} downloaded, {failed_count} failed") return downloaded_count, failed_count + @deprecated("3.10.0", "download logos one at a time with download_missing_logo()") def download_all_ncaa_football_logos(self, include_fcs: bool = True, force_download: bool = False) -> Tuple[int, int]: """Download all NCAA football team logos including FCS teams.""" logger.info(f"Starting comprehensive NCAA football logo download (FCS: {include_fcs})") @@ -763,6 +768,7 @@ class LogoDownloader: time.sleep(0.1) # Small delay return success + @deprecated("3.10.0", "download logos one at a time with download_missing_logo()") def download_all_missing_logos(self, leagues: List[str] | None = None, force_download: bool = False) -> Dict[str, Tuple[int, int]]: """Download missing logos for all specified leagues.""" if leagues is None: @@ -858,6 +864,7 @@ class LogoDownloader: logger.error(f"Failed to create placeholder logo for {team_abbreviation}: {e}") return False + @deprecated("3.10.0") def convert_image_to_rgba(self, filepath: Path) -> bool: """Convert an image file to RGBA format to avoid PIL warnings.""" try: @@ -875,6 +882,7 @@ class LogoDownloader: logger.error(f"Failed to convert {filepath.name} to RGBA: {e}") return False + @deprecated("3.10.0") def convert_all_logos_to_rgba(self, league: str) -> Tuple[int, int]: """Convert all logos in a league directory to RGBA format.""" logo_dir = Path(self.get_logo_directory(league)) diff --git a/src/plugin_system/operation_history.py b/src/plugin_system/operation_history.py index f20fd2a4..35790895 100644 --- a/src/plugin_system/operation_history.py +++ b/src/plugin_system/operation_history.py @@ -51,44 +51,36 @@ class OperationHistory: def __init__( self, history_file: Optional[str] = None, - max_records: int = 1000, - lazy_load: bool = False + max_records: int = 1000 ): """ - Initialize operation history. - + Initialize operation history. The history file is read on first use, + not here, so constructing this costs the web app's startup nothing. + Args: history_file: Path to file for persisting history max_records: Maximum number of records to keep - lazy_load: If True, defer loading history file until first access """ self.logger = get_logger(__name__) self.history_file = Path(history_file) if history_file else None self.max_records = max_records - self._lazy_load = lazy_load self._history_loaded = False - + # In-memory history self._history: List[OperationRecord] = [] self._lock = threading.RLock() - - # Load history from file if it exists (unless lazy loading) - if not self._lazy_load and self.history_file and self.history_file.exists(): - self._load_history() - self._history_loaded = True - + def _ensure_loaded(self) -> None: - """Ensure history is loaded (for lazy loading).""" + """Load the history file on first use.""" if not self._history_loaded and self.history_file and self.history_file.exists(): self._load_history() self._history_loaded = True - + def record_operation( self, operation_type: str, plugin_id: Optional[str] = None, status: str = "completed", - user: Optional[str] = None, details: Optional[Dict[str, Any]] = None, error: Optional[str] = None, operation_id: Optional[str] = None @@ -100,7 +92,6 @@ class OperationHistory: operation_type: Type of operation (install, update, uninstall, etc.) plugin_id: Plugin identifier status: Operation status - user: User who performed operation details: Optional operation details error: Optional error message operation_id: Optional operation ID @@ -118,7 +109,6 @@ class OperationHistory: plugin_id=plugin_id, timestamp=datetime.now(), status=status, - user=user, details=details, error=error ) diff --git a/src/plugin_system/operation_queue.py b/src/plugin_system/operation_queue.py index 7bd24b78..137dcd87 100644 --- a/src/plugin_system/operation_queue.py +++ b/src/plugin_system/operation_queue.py @@ -2,7 +2,7 @@ Plugin operation queue manager. Serializes plugin operations to prevent conflicts and provides -status tracking and cancellation support. +status tracking. """ import threading @@ -25,8 +25,8 @@ class PluginOperationQueue: - Serialized execution (one operation at a time) - Prevents concurrent operations on same plugin - Operation status tracking - - Operation cancellation - - In-memory history of finished operations + - A bounded in-memory history of finished operations, which also caps + how many finished operations get_operation_status() remembers The history is not persisted. The web UI's operation history comes from OperationHistory (operation_history.py), which has its own file; a copy @@ -133,56 +133,6 @@ class PluginOperationQueue: with self._lock: return self._operations.get(operation_id) - def cancel_operation(self, operation_id: str) -> bool: - """ - Cancel a pending operation. - - Args: - operation_id: Operation identifier - - Returns: - True if operation was cancelled, False if not found or already running - """ - with self._lock: - operation = self._operations.get(operation_id) - if not operation: - return False - - if operation.status == OperationStatus.RUNNING: - self.logger.warning( - f"Cannot cancel running operation {operation_id}" - ) - return False - - if operation.status == OperationStatus.PENDING: - operation.status = OperationStatus.CANCELLED - operation.completed_at = datetime.now() - operation.message = "Operation cancelled by user" - self._add_to_history(operation) - self.logger.info(f"Cancelled operation {operation_id}") - return True - - return False - - def get_operation_history(self, limit: int = 50) -> List[PluginOperation]: - """ - Get operation history. - - Args: - limit: Maximum number of operations to return - - Returns: - List of operations, sorted by creation time (newest first) - """ - with self._lock: - # Sort by creation time (newest first) - history = sorted( - self._operation_history, - key=lambda op: op.created_at, - reverse=True - ) - return history[:limit] - def _start_worker(self) -> None: """Start the worker thread that processes operations.""" if self._worker_thread and self._worker_thread.is_alive(): @@ -207,11 +157,6 @@ class PluginOperationQueue: except queue.Empty: continue - # Check if operation was cancelled - if operation.status == OperationStatus.CANCELLED: - self._operation_queue.task_done() - continue - # Execute operation self._execute_operation(operation) diff --git a/src/plugin_system/operation_types.py b/src/plugin_system/operation_types.py index 950b0fd3..fa220fa7 100644 --- a/src/plugin_system/operation_types.py +++ b/src/plugin_system/operation_types.py @@ -15,11 +15,7 @@ import uuid class OperationType(Enum): """Types of plugin operations.""" INSTALL = "install" - UPDATE = "update" UNINSTALL = "uninstall" - ENABLE = "enable" - DISABLE = "disable" - CONFIGURE = "configure" class OperationStatus(Enum): @@ -28,7 +24,6 @@ class OperationStatus(Enum): RUNNING = "running" COMPLETED = "completed" FAILED = "failed" - CANCELLED = "cancelled" @dataclass @@ -70,29 +65,3 @@ class PluginOperation: 'started_at': self.started_at.isoformat() if self.started_at else None, 'completed_at': self.completed_at.isoformat() if self.completed_at else None, } - - @classmethod - def from_dict(cls, data: Dict[str, Any]) -> 'PluginOperation': - """Create operation from dictionary.""" - op = cls( - operation_type=OperationType(data['operation_type']), - plugin_id=data['plugin_id'], - operation_id=data.get('operation_id', str(uuid.uuid4())), - parameters=data.get('parameters', {}), - status=OperationStatus(data.get('status', 'pending')), - progress=data.get('progress', 0.0), - message=data.get('message', ''), - error=data.get('error'), - result=data.get('result'), - ) - - # Parse datetime fields - if data.get('created_at'): - op.created_at = datetime.fromisoformat(data['created_at']) - if data.get('started_at'): - op.started_at = datetime.fromisoformat(data['started_at']) - if data.get('completed_at'): - op.completed_at = datetime.fromisoformat(data['completed_at']) - - return op - diff --git a/src/plugin_system/plugin_catalog.py b/src/plugin_system/plugin_catalog.py index bd80f2ab..ec6c7c23 100644 --- a/src/plugin_system/plugin_catalog.py +++ b/src/plugin_system/plugin_catalog.py @@ -3,9 +3,10 @@ Plugin catalog: what the web process knows about installed plugins. The web interface and the display run as two processes. Only the display imports plugin code and runs it; the web process reads plugins as files -- -manifest, config schema, the plugin's section of config.json, the installed -version -- and never imports a plugin module, instantiates a plugin class or -calls a plugin lifecycle hook. This class is that read side. +manifest, config schema, the plugin's section of config.json -- and never +imports a plugin module, instantiates a plugin class or calls a plugin +lifecycle hook. This class is the manifest side of that; schemas come from +SchemaManager and config from ConfigManager. It keeps the method names of the read-only part of :class:`PluginManager` (``discover_plugins``, ``plugin_manifests``, ``get_plugin_info``, @@ -25,11 +26,10 @@ machine) the web cannot know, and reports as unknown. See docs/ARCHITECTURE.md ("Web and display processes"). """ -import json import threading import time from pathlib import Path -from typing import Any, Callable, Dict, List, Optional, Union, cast +from typing import Any, Callable, Dict, List, Optional, Union from src.common.permission_utils import ( ensure_directory_permissions, get_plugin_dir_mode, @@ -47,19 +47,16 @@ _RUNTIME_VIEW_TTL_SECONDS = 1.0 class PluginCatalog: - """Manifests, schemas, config and versions of the installed plugins. + """Manifests and directories of the installed plugins. Discovery is explicit and cheap to repeat: :meth:`discover_plugins` rescans the plugins directory and replaces the manifest map, so an uninstalled plugin disappears and a new one appears. """ - def __init__(self, plugins_dir: PathLike, config_manager: Optional[Any] = None, - schema_manager: Optional[Any] = None, + def __init__(self, plugins_dir: PathLike, runtime_source: Optional[Callable[[], Any]] = None) -> None: self.plugins_dir: Path = Path(plugins_dir) - self.config_manager = config_manager - self.schema_manager = schema_manager # Returns the display's PluginRuntimeView # (src/plugin_system/plugin_runtime.py). Its live view carries the # modes the display registered, which the mode lookups below prefer @@ -145,30 +142,6 @@ class PluginCatalog: ids = list(self.plugin_manifests) return [info for info in (self.get_plugin_info(pid) for pid in ids) if info] - def read_manifest(self, plugin_id: str) -> Optional[Dict[str, Any]]: - """The manifest as it is on disk now, not as discovery last saw it. - - For reads that must reflect a change made since the last scan -- the - version just after an update, say. None when the plugin has no - directory or its manifest is missing, unreadable or not an object. - """ - plugin_dir = self.get_plugin_directory(plugin_id) - if plugin_dir is None: - return None - try: - with open(Path(plugin_dir) / 'manifest.json', 'r', encoding='utf-8') as f: - manifest = json.load(f) - except (OSError, ValueError) as exc: - self.logger.debug("Could not read manifest for %s: %s", plugin_id, exc) - return None - return manifest if isinstance(manifest, dict) else None - - def get_installed_version(self, plugin_id: str) -> str: - """The installed version from the on-disk manifest, or ''.""" - manifest = self.read_manifest(plugin_id) or {} - version = manifest.get('version', '') - return version if isinstance(version, str) else str(version) - def get_plugin_directory(self, plugin_id: str) -> Optional[str]: """Where ``plugin_id`` is installed, or None. @@ -252,31 +225,6 @@ class PluginCatalog: return plugin_id return None - # -- schema and config ------------------------------------------------ - - def get_schema(self, plugin_id: str, use_cache: bool = True) -> Optional[Dict[str, Any]]: - """The plugin's config schema through SchemaManager, or None.""" - if self.schema_manager is None: - return None - schema = self.schema_manager.load_schema(plugin_id, use_cache=use_cache) - return cast(Optional[Dict[str, Any]], schema) - - def get_config(self, plugin_id: str) -> Dict[str, Any]: - """The plugin's section of config.json (secrets merged), or {}.""" - if self.config_manager is None: - return {} - section = (self.config_manager.load_config() or {}).get(plugin_id) - return section if isinstance(section, dict) else {} - - def is_enabled(self, plugin_id: str) -> bool: - """Whether config.json enables the plugin, by the display's rule. - - The display loads a plugin only when its section says - ``"enabled": true``; a missing flag or section means disabled - (``DisplayController._reconcile_enabled_plugins``). - """ - return bool(self.get_config(plugin_id).get('enabled', False)) - def display_restart_required(action: str, plugin_enabled: bool, *, changed: bool = True, diff --git a/src/plugin_system/plugin_manager.py b/src/plugin_system/plugin_manager.py index 2ff6fd1f..ad3d5af3 100644 --- a/src/plugin_system/plugin_manager.py +++ b/src/plugin_system/plugin_manager.py @@ -38,6 +38,7 @@ from src.common.permission_utils import ( ensure_directory_permissions, get_plugin_dir_mode ) +from src.deprecation import deprecated class _DeferredConfigChange(NamedTuple): @@ -939,6 +940,7 @@ class PluginManager: """ return self.plugins.get(plugin_id) + @deprecated("3.10.0", "use get_plugin(plugin_id)") def get_all_plugins(self) -> Dict[str, Any]: """ Get all loaded plugins. @@ -948,6 +950,7 @@ class PluginManager: """ return self.plugins.copy() + @deprecated("3.10.0", "read the manifest with src.plugin_system.plugin_catalog.PluginCatalog") def get_plugin_info(self, plugin_id: str) -> Optional[Dict[str, Any]]: """ Get information about a plugin (manifest + runtime info). @@ -985,6 +988,7 @@ class PluginManager: return info + @deprecated("3.10.0", "read manifests with src.plugin_system.plugin_catalog.PluginCatalog") def get_all_plugin_info(self) -> List[Dict[str, Any]]: """ Get information about all plugins. @@ -1025,6 +1029,7 @@ class PluginManager: by_manifest=False) return str(plugin_dir) if plugin_dir is not None else None + @deprecated("3.10.0", "read manifests with src.plugin_system.plugin_catalog.PluginCatalog") def get_plugin_display_modes(self, plugin_id: str) -> List[str]: """ Get display modes provided by a plugin. @@ -1045,6 +1050,7 @@ class PluginManager: return display_modes return [] + @deprecated("3.10.0", "read manifests with src.plugin_system.plugin_catalog.PluginCatalog") def find_plugin_for_mode(self, mode: str) -> Optional[str]: """ Find which plugin provides a given display mode. diff --git a/src/plugin_system/plugin_state.py b/src/plugin_system/plugin_state.py index f89a9b75..f7504b8f 100644 --- a/src/plugin_system/plugin_state.py +++ b/src/plugin_system/plugin_state.py @@ -15,6 +15,7 @@ from datetime import datetime import logging from src.logging_config import get_logger +from src.deprecation import deprecated class PluginState(Enum): @@ -138,6 +139,7 @@ class PluginStateManager: """ return self._states.get(plugin_id, PluginState.UNLOADED) + @deprecated("3.10.0", "use get_state()") def is_loaded(self, plugin_id: str) -> bool: """Check if plugin is loaded.""" state = self.get_state(plugin_id) @@ -148,11 +150,13 @@ class PluginStateManager: state = self.get_state(plugin_id) return state == PluginState.ENABLED + @deprecated("3.10.0", "use get_state()") def is_running(self, plugin_id: str) -> bool: """Check if plugin is currently running.""" state = self.get_state(plugin_id) return state == PluginState.RUNNING + @deprecated("3.10.0", "use get_state()") def is_error(self, plugin_id: str) -> bool: """Check if plugin is in error state.""" state = self.get_state(plugin_id) @@ -197,6 +201,7 @@ class PluginStateManager: state.value, ) + @deprecated("3.10.0") def get_error_info(self, plugin_id: str) -> Optional[Dict[str, Any]]: """ Get error information for a plugin. @@ -287,10 +292,12 @@ class PluginStateManager: """Record that plugin update() was called.""" self._last_update[plugin_id] = datetime.now() + @deprecated("3.10.0") def get_last_update(self, plugin_id: str) -> Optional[datetime]: """Get timestamp of last update() call.""" return self._last_update.get(plugin_id) + @deprecated("3.10.0", "use get_state()") def get_state_info(self, plugin_id: str) -> Dict[str, Any]: """ Get comprehensive state information for a plugin. diff --git a/src/plugin_system/state_reconciliation.py b/src/plugin_system/state_reconciliation.py index 1477984e..72268f49 100644 --- a/src/plugin_system/state_reconciliation.py +++ b/src/plugin_system/state_reconciliation.py @@ -38,7 +38,6 @@ class InconsistencyType(Enum): PLUGIN_MISSING_ON_DISK = "plugin_missing_on_disk" PLUGIN_ENABLED_MISMATCH = "plugin_enabled_mismatch" PLUGIN_VERSION_MISMATCH = "plugin_version_mismatch" - PLUGIN_STATE_CORRUPTED = "plugin_state_corrupted" class FixAction(Enum): @@ -57,7 +56,6 @@ class Inconsistency: fix_action: FixAction current_state: Dict[str, Any] expected_state: Dict[str, Any] - can_auto_fix: bool = False @dataclass @@ -270,7 +268,7 @@ class StateReconciliation: # Attempt to fix auto-fixable inconsistencies for inconsistency in inconsistencies: - if inconsistency.can_auto_fix and inconsistency.fix_action == FixAction.AUTO_FIX: + if inconsistency.fix_action == FixAction.AUTO_FIX: if self._fix_inconsistency(inconsistency): fixed.append(inconsistency) else: @@ -428,7 +426,6 @@ class StateReconciliation: fix_action=FixAction.AUTO_FIX, current_state={'exists_in_config': False}, expected_state={'exists_in_config': True, 'enabled': False}, - can_auto_fix=True )) # Check: Plugin in config but not on disk @@ -459,7 +456,6 @@ class StateReconciliation: fix_action=FixAction.AUTO_FIX if can_repair else FixAction.MANUAL_FIX_REQUIRED, current_state={'exists_on_disk': False}, expected_state={'exists_on_disk': True}, - can_auto_fix=can_repair )) # Observed checks: only against a live snapshot, and only for a plugin @@ -486,7 +482,6 @@ class StateReconciliation: fix_action=FixAction.NO_ACTION, current_state={'loaded': loaded, 'state': runtime.get('state')}, expected_state={'loaded': config_enabled}, - can_auto_fix=False )) loaded_version = runtime.get('loaded_version') disk_version = disk.get('version') @@ -500,7 +495,6 @@ class StateReconciliation: fix_action=FixAction.NO_ACTION, current_state={'version': loaded_version}, expected_state={'version': disk_version}, - can_auto_fix=False )) return inconsistencies diff --git a/src/plugin_system/store_registry.py b/src/plugin_system/store_registry.py index 2270843c..13228695 100644 --- a/src/plugin_system/store_registry.py +++ b/src/plugin_system/store_registry.py @@ -183,16 +183,7 @@ class _RegistryMixin: @staticmethod def _distinct_sequence(values: List[str]) -> List[str]: """Return list preserving order while removing duplicates and falsey entries.""" - seen = set() - ordered = [] - for value in values: - if not value: - continue - if value in seen: - continue - seen.add(value) - ordered.append(value) - return ordered + return list(dict.fromkeys(v for v in values if v)) def _validate_manifest_version_fields(self, manifest: Dict[str, Any]) -> List[str]: """ diff --git a/src/plugin_system/testing/plugin_test_base.py b/src/plugin_system/testing/plugin_test_base.py index 806f3d3f..fcedd01b 100644 --- a/src/plugin_system/testing/plugin_test_base.py +++ b/src/plugin_system/testing/plugin_test_base.py @@ -25,6 +25,7 @@ from src.plugin_system.testing.mocks import ( MockConfigManager, MockPluginManager ) +from src.deprecation import deprecated class PluginTestCase(unittest.TestCase): @@ -34,6 +35,7 @@ class PluginTestCase(unittest.TestCase): Provides common fixtures and helper methods. """ + @deprecated("3.10.0", "use src.plugin_system.testing.harness and the mocks directly") def setUp(self): """Set up test fixtures.""" # Create mock managers diff --git a/src/vegas_mode/config.py b/src/vegas_mode/config.py index affc7a15..9b9f2d29 100644 --- a/src/vegas_mode/config.py +++ b/src/vegas_mode/config.py @@ -291,49 +291,6 @@ class VegasModeConfig: max_cycle_duration=int(get('max_cycle_duration', d.max_cycle_duration)), ) - def to_dict(self) -> Dict[str, Any]: - """Convert config to dictionary for serialization.""" - return { - 'enabled': self.enabled, - 'scroll_speed': self.scroll_speed, - 'separator_width': self.separator_width, - 'intra_plugin_gap': self.intra_plugin_gap, - 'render_width_pct': self.render_width_pct, - 'min_content_separation': self.min_content_separation, - 'min_cut_gap': self.min_cut_gap, - 'smooth_scroll': self.smooth_scroll, - 'sub_pixel_blend': self.sub_pixel_blend, - 'continuous_scroll': self.continuous_scroll, - 'offscreen_prefetch': self.offscreen_prefetch, - 'switch_interval_ms': self.switch_interval_ms, - 'prefetch_gate': self.prefetch_gate, - 'live_refresh': self.live_refresh, - 'live_max_hz': self.live_max_hz, - 'live_min_interval': self.live_min_interval, - 'live_lead_screens': self.live_lead_screens, - 'extend_threshold_screens': self.extend_threshold_screens, - 'auto_trim': self.auto_trim, - 'trim_threshold': self.trim_threshold, - 'content_padding': self.content_padding, - 'min_plugin_width': self.min_plugin_width, - 'lead_in_width': self.lead_in_width, - 'plugins_per_cycle': self.plugins_per_cycle, - 'max_plugin_width_ratio': self.max_plugin_width_ratio, - 'live_in_ticker': self.live_in_ticker, - 'live_weight': self.live_weight, - 'favorite_live_weight': self.favorite_live_weight, - 'overflow_mode': self.overflow_mode, - 'plugin_order': self.plugin_order, - 'excluded_plugins': list(self.excluded_plugins), - 'target_fps': self.target_fps, - 'buffer_ahead': self.buffer_ahead, - 'frame_based_scrolling': self.frame_based_scrolling, - 'scroll_delay': self.scroll_delay, - 'dynamic_duration_enabled': self.dynamic_duration_enabled, - 'min_cycle_duration': self.min_cycle_duration, - 'max_cycle_duration': self.max_cycle_duration, - } - def get_frame_interval(self) -> float: """Get the frame interval in seconds for target FPS.""" return 1.0 / max(1, self.target_fps) diff --git a/src/vegas_mode/coordinator.py b/src/vegas_mode/coordinator.py index 8c296c60..bcdb3794 100644 --- a/src/vegas_mode/coordinator.py +++ b/src/vegas_mode/coordinator.py @@ -182,16 +182,6 @@ class VegasModeCoordinator: self._static_pause_active = False self._saved_scroll_position: Optional[int] = None - # Statistics - self.stats = { - 'total_runtime_seconds': 0.0, - 'cycles_completed': 0, - 'interruptions': 0, - 'config_updates': 0, - 'static_pauses': 0, - } - self._start_time: Optional[float] = None - logger.info( "VegasModeCoordinator initialized: enabled=%s, fps=%d, buffer_ahead=%d", self.vegas_config.enabled, @@ -326,7 +316,6 @@ class VegasModeCoordinator: # new run would have run_frame() refuse every frame. self._is_paused = False self._live_priority_active = False - self._start_time = time.time() # A fresh run starts with a clean health slate: no stale # "was degraded" from the previous run, and a heartbeat that is # due immediately so the first sample confirms the marquee is up. @@ -357,10 +346,6 @@ class VegasModeCoordinator: self._is_paused = False self._live_priority_active = False - if self._start_time: - self.stats['total_runtime_seconds'] += time.time() - self._start_time - self._start_time = None - self._restore_switch_interval() self._remove_render_gate() self._set_live(False, None) @@ -485,7 +470,6 @@ class VegasModeCoordinator: if not self._is_active: return self._is_paused = True - self.stats['interruptions'] += 1 self.display_manager.set_scrolling_state(False) logger.info("Vegas mode paused") @@ -555,9 +539,8 @@ class VegasModeCoordinator: if self.render_pipeline.has_deferred(): self.render_pipeline.drain_deferred() elif self.render_pipeline.needs_extension(): - if self.render_pipeline.extend_scroll_content(): - self.stats['cycles_completed'] += 1 - elif self.render_pipeline.is_cycle_complete(): + if (not self.render_pipeline.extend_scroll_content() + and self.render_pipeline.is_cycle_complete()): # Extension failed and the strip has run out: fall back to # the swap rather than sitting on a dead frame. self.render_pipeline.start_new_cycle() @@ -567,7 +550,6 @@ class VegasModeCoordinator: if not self.render_pipeline.start_new_cycle(): logger.warning("Failed to start new Vegas cycle") return False - self.stats['cycles_completed'] += 1 # Check for hot-swap opportunities if self.render_pipeline.should_recompose(): @@ -847,7 +829,6 @@ class VegasModeCoordinator: self._pending_config_update = True self._pending_config = new_config self._config_version += 1 - self.stats['config_updates'] += 1 logger.debug("Config update queued (version %d)", self._config_version) @@ -922,23 +903,6 @@ class VegasModeCoordinator: self.stream_manager.mark_plugin_updated(plugin_id) self.plugin_adapter.invalidate_cache(plugin_id) - def get_status(self) -> Dict[str, Any]: - """Get comprehensive Vegas mode status.""" - status = { - 'enabled': self.vegas_config.enabled, - 'active': self._is_active, - 'paused': self._is_paused, - 'live_priority_active': self._live_priority_active, - 'config': self.vegas_config.to_dict(), - 'stats': self.stats.copy(), - } - - if self._is_active: - status['render_info'] = self.render_pipeline.get_current_scroll_info() - status['stream_status'] = self.stream_manager.get_buffer_status() - - return status - # ------------------------------------------------------------------------- # Static pause handling (for STATIC display mode) # ------------------------------------------------------------------------- @@ -992,7 +956,6 @@ class VegasModeCoordinator: # Save current scroll position for smooth resume self._saved_scroll_position = self.render_pipeline.get_scroll_position() self._static_pause_active = True - self.stats['static_pauses'] += 1 logger.info("Static pause started for plugin: %s", plugin_id) diff --git a/src/vegas_mode/render_pipeline.py b/src/vegas_mode/render_pipeline.py index 44cd6e90..bb973b84 100644 --- a/src/vegas_mode/render_pipeline.py +++ b/src/vegas_mode/render_pipeline.py @@ -218,7 +218,6 @@ class RenderPipeline: # Render state self._cycle_complete = False - self._segments_in_scroll: List[str] = [] # Plugin IDs in current scroll self._record_by_seq: Dict[int, ElementRecord] = {} # Live updates. _applied: per record, the (epoch, digest) of the # pixels the strip holds. _live_slots / _live_ready: the worker's @@ -235,15 +234,9 @@ class RenderPipeline: self._frame_interval = config.get_frame_interval() self._cycle_start_time = 0.0 - # Statistics - self.stats = { - 'frames_rendered': 0, - 'scroll_cycles': 0, - 'composition_count': 0, - 'hot_swaps': 0, - 'avg_frame_time_ms': 0.0, - } - self._frame_times: Deque[float] = deque(maxlen=100) # Efficient fixed-size buffer + # Read by _measure_refresh (warm-up) and the live integration test. + self.frames_rendered = 0 + self.extensions = 0 logger.info( "RenderPipeline initialized: %dx%d @ %d FPS", @@ -342,7 +335,7 @@ class RenderPipeline: return if getattr(self.display_manager, 'matrix', None) is None: return # No hardware: nothing blocks, so there is nothing to time. - if self.stats['frames_rendered'] < self.REFRESH_WARMUP_FRAMES: + if self.frames_rendered < self.REFRESH_WARMUP_FRAMES: return self._swap_times.append(time.monotonic()) if len(self._swap_times) <= self.REFRESH_SAMPLES: @@ -452,10 +445,6 @@ class RenderPipeline: layouts) self._note_op('compose', self._strip_nbytes()) - # Track which plugins are in this scroll (get safely via buffer status) - self._segments_in_scroll = self.stream_manager.get_active_plugin_ids() - - self.stats['composition_count'] += 1 self._cycle_start_time = time.time() self._cycle_complete = False @@ -865,9 +854,7 @@ class RenderPipeline: # trim's copy, if it made one. self._note_op('extend', moved + (self._copied_bytes() if cut else 0)) - self._segments_in_scroll = [pid for pid, _ in grouped] - self.stats['composition_count'] += 1 - self.stats['extensions'] = self.stats.get('extensions', 0) + 1 + self.extensions += 1 logger.info( "Extended scroll strip with %d plugin block(s), %d rows: " @@ -1150,8 +1137,6 @@ class RenderPipeline: Returns: True if frame was rendered, False if no content """ - frame_start = time.time() - try: if not self.scroll_helper.has_strip(): return False @@ -1200,7 +1185,6 @@ class RenderPipeline: if at_wrap_point or self.scroll_helper.is_scroll_complete(): if not self._cycle_complete: self._cycle_complete = True - self.stats['scroll_cycles'] += 1 logger.info( "Scroll cycle complete after %.1fs", time.time() - self._cycle_start_time @@ -1239,11 +1223,8 @@ class RenderPipeline: # Update scrolling state self.display_manager.set_scrolling_state(True, self._frame_hold) - # Track statistics - self.stats['frames_rendered'] += 1 + self.frames_rendered += 1 self._measure_refresh() - frame_time = time.time() - frame_start - self._track_frame_time(frame_time) return True @@ -1252,15 +1233,6 @@ class RenderPipeline: logger.exception("Error rendering frame") return False - def _track_frame_time(self, frame_time: float) -> None: - """Track frame timing for statistics.""" - self._frame_times.append(frame_time) # deque with maxlen auto-removes old entries - - if self._frame_times: - self.stats['avg_frame_time_ms'] = ( - sum(self._frame_times) / len(self._frame_times) * 1000 - ) - def is_cycle_complete(self) -> bool: """Check if current scroll cycle is complete.""" return self._cycle_complete @@ -1347,7 +1319,6 @@ class RenderPipeline: else: self.scroll_helper.scroll_position = 0.0 - self.stats['hot_swaps'] += 1 logger.debug( "Hot-swap completed: scroll repositioned %.0f→%.0f (%.1f%% of new %dpx image)", old_pos, self.scroll_helper.scroll_position, @@ -1396,8 +1367,6 @@ class RenderPipeline: # transition rather than near-end content wrapping around. self.scroll_helper.scroll_position = float(self.config.lead_in_width) - # Signal follower that a new cycle started (triggers its own rebuild) - self.sync_manager.send_new_cycle() # Push the actual scroll image over TCP so follower has identical pixels. # Done in a background thread to not block the render loop (~15ms transfer). image = self.scroll_helper.cached_image @@ -1410,16 +1379,6 @@ class RenderPipeline: return result - def get_current_scroll_info(self) -> Dict[str, Any]: - """Get current scroll state information.""" - scroll_info = self.scroll_helper.get_scroll_info() - return { - **scroll_info, - 'cycle_complete': self._cycle_complete, - 'plugins_in_scroll': self._segments_in_scroll, - 'stats': self.stats.copy(), - } - def get_scroll_position(self) -> int: """ Get current scroll position. @@ -1465,8 +1424,6 @@ class RenderPipeline: self.scroll_helper.clear_cache() self._cycle_complete = False - self._segments_in_scroll = [] - self._frame_times = deque(maxlen=100) # Content lined up for the old run belongs to it. Left in place, the # first extension after Vegas is switched back on appended that stale diff --git a/src/vegas_mode/stream_manager.py b/src/vegas_mode/stream_manager.py index 46f9de16..ccbf7db2 100644 --- a/src/vegas_mode/stream_manager.py +++ b/src/vegas_mode/stream_manager.py @@ -16,7 +16,7 @@ BasePlugin.get_vegas_participation): import logging import threading import time -from typing import Optional, List, Dict, Any, Deque, Tuple, TYPE_CHECKING +from typing import Optional, List, Dict, Deque, Tuple, TYPE_CHECKING from collections import deque from dataclasses import dataclass, field from PIL import Image @@ -94,13 +94,6 @@ class StreamManager: self._last_refresh: float = 0.0 self._refresh_interval: float = 30.0 # Refresh plugin list every 30s - # Statistics - self.stats = { - 'segments_fetched': 0, - 'segments_served': 0, - 'fetch_errors': 0, - } - logger.info("StreamManager initialized with buffer_ahead=%d", config.buffer_ahead) def initialize(self) -> bool: @@ -143,47 +136,12 @@ class StreamManager: return None segment = self._active_buffer.popleft() - self.stats['segments_served'] += 1 # Trigger prefetch to maintain buffer self._ensure_buffer_filled() return segment - def peek_next_segment(self) -> Optional[ContentSegment]: - """ - Peek at the next segment without removing it. - - Returns: - ContentSegment or None if buffer is empty - """ - with self._buffer_lock: - if self._active_buffer: - return self._active_buffer[0] - return None - - def get_buffer_status(self) -> Dict[str, Any]: - """Get current buffer status for monitoring.""" - with self._buffer_lock: - return { - 'active_count': len(self._active_buffer), - 'total_plugins': len(self._ordered_plugins), - 'prefetch_index': self._prefetch_index, - 'stats': self.stats.copy(), - } - - def get_active_plugin_ids(self) -> List[str]: - """ - Get list of plugin IDs currently in the active buffer. - - Thread-safe accessor for render pipeline. - - Returns: - List of plugin IDs in buffer order - """ - with self._buffer_lock: - return [seg.plugin_id for seg in self._active_buffer] - def mark_plugin_updated(self, plugin_id: str) -> None: """ Mark a plugin as having updated data. @@ -587,7 +545,6 @@ class StreamManager: images=[], # No images needed for static pause display_mode=VegasDisplayMode.STATIC ) - self.stats['segments_fetched'] += 1 logger.debug( "[%s] Created STATIC placeholder (pause trigger)", plugin_id @@ -610,7 +567,6 @@ class StreamManager: display_mode=VegasDisplayMode.SCROLL ) - self.stats['segments_fetched'] += 1 logger.debug( "[%s] Segment: %d image(s), %dpx", plugin_id, len(images), total_width @@ -619,7 +575,6 @@ class StreamManager: except Exception: logger.exception("[%s] ERROR fetching content", plugin_id) - self.stats['fetch_errors'] += 1 return None def _ensure_buffer_filled(self) -> None: @@ -770,10 +725,8 @@ class StreamManager: plugin, plugin_id, offscreen_only=offscreen_only) except Exception: logger.exception("[%s] ERROR fetching content", plugin_id) - self.stats['fetch_errors'] += 1 return None if images: - self.stats['segments_fetched'] += 1 return (plugin_id, images) # Only the old contract hands anything back to the render thread. defer_empty = offscreen_only and not getattr( diff --git a/src/web_interface/api_helpers.py b/src/web_interface/api_helpers.py index b51d2b78..ee2d84cf 100644 --- a/src/web_interface/api_helpers.py +++ b/src/web_interface/api_helpers.py @@ -8,7 +8,6 @@ import time from typing import Any, Optional, Dict, Tuple from flask import jsonify, request -from src.web_interface.error_handler import create_error_response, create_success_response from src.web_interface.errors import ErrorCode, WebInterfaceError @@ -31,7 +30,15 @@ def success_response( Returns: Flask jsonify response """ - response_data = create_success_response(data, message, metadata) + response_data: Dict[str, Any] = {'status': 'success'} + # `is not None` rather than truthiness: "" and {} are values a caller + # chose to send, and dropping them would make the shape depend on the data. + if data is not None: + response_data['data'] = data + if message is not None: + response_data['message'] = message + if metadata is not None: + response_data['metadata'] = metadata for key, value in (extra or {}).items(): response_data.setdefault(key, value) @@ -70,14 +77,14 @@ def error_response( Returns: Flask jsonify response with status code """ - return create_error_response( + error = WebInterfaceError( error_code=error_code, message=message, details=details, - context=context, - suggested_fixes=suggested_fixes, - status_code=status_code + context=context or {}, + suggested_fixes=suggested_fixes ) + return jsonify(error.to_dict()), status_code def exception_error_response( diff --git a/src/web_interface/config_arrays.py b/src/web_interface/config_arrays.py index 0e644a05..bc0eac28 100644 --- a/src/web_interface/config_arrays.py +++ b/src/web_interface/config_arrays.py @@ -12,8 +12,8 @@ from typing import Any, Dict def _schema_type_is(prop: Any, wanted: str) -> bool: """Whether a schema property is of ``wanted`` type, unions included. - Mirrors ``_schema_type_is`` in ``web_interface/blueprints/api_v3`` (kept - here so src/ doesn't import the Flask blueprint). A union such as + The one copy: ``web_interface/blueprints/api_v3`` imports it from here + (src/ must not import the Flask blueprint). A union such as ``["array", "null"]`` -- the per-element style overrides, where null means "inherit" -- is still an array for recombining position-keyed inputs. """ diff --git a/src/web_interface/error_handler.py b/src/web_interface/error_handler.py index c3360def..6f31235f 100644 --- a/src/web_interface/error_handler.py +++ b/src/web_interface/error_handler.py @@ -1,20 +1,13 @@ """ -Centralized error handling for web interface. +Error text and payloads for web interface responses. -Provides helpers for consistent error responses across API endpoints. +Safe exception descriptions and the bodies for exceptions no route handled. +The standard success/error responses are in api_helpers. """ -from typing import Any, Optional -from flask import jsonify - -from src.web_interface.errors import WebInterfaceError, ErrorCode -from src.logging_config import get_logger from src.redaction import redact_credentials -logger = get_logger(__name__) - - # Long enough for an errno string with a path, short enough not to dump a # parser's worth of context into a JSON field. _MAX_DETAIL_LENGTH = 400 @@ -101,72 +94,3 @@ def http_exception_payload(error) -> dict: 'error_code': (error.name or 'HTTP_ERROR').upper().replace(' ', '_'), 'message': error.description, } - - -def create_error_response( - error_code: ErrorCode, - message: str, - details: Optional[str] = None, - context: Optional[dict] = None, - suggested_fixes: Optional[list] = None, - status_code: int = 500 -) -> tuple: - """ - Create a standardized error response. - - Args: - error_code: Error code - message: Error message - details: Optional detailed error information - context: Optional context dictionary - suggested_fixes: Optional list of suggested fixes - status_code: HTTP status code - - Returns: - Tuple of (jsonify response, status_code) - """ - error = WebInterfaceError( - error_code=error_code, - message=message, - details=details, - context=context or {}, - suggested_fixes=suggested_fixes - ) - - return jsonify(error.to_dict()), status_code - - -def create_success_response( - data: Any = None, - message: Optional[str] = None, - metadata: Optional[dict] = None -) -> dict: - """ - Create a standardized success response. - - Args: - data: Response data - message: Optional success message - metadata: Optional metadata (timing, version, etc.) - - Returns: - Dictionary for jsonify - """ - response: dict[str, Any] = { - "status": "success" - } - - # All three use `is not None` rather than truthiness: "" and {} are - # values a caller chose to send, and dropping them silently would make - # the response shape depend on the data. - if data is not None: - response["data"] = data - - if message is not None: - response["message"] = message - - if metadata is not None: - response["metadata"] = metadata - - return response - diff --git a/src/web_interface/errors.py b/src/web_interface/errors.py index 4d3f9fd6..2441ba22 100644 --- a/src/web_interface/errors.py +++ b/src/web_interface/errors.py @@ -61,27 +61,14 @@ class WebInterfaceError: context: Optional[Dict[str, Any]] = None suggested_fixes: Optional[List[str]] = None original_error: Optional[Exception] = None - - def __init__( - self, - error_code: ErrorCode, - message: str, - details: Optional[str] = None, - context: Optional[Dict[str, Any]] = None, - suggested_fixes: Optional[List[str]] = None, - original_error: Optional[Exception] = None - ): - self.error_code = error_code - self.message = message - self.details = details - self.context = context or {} + + def __post_init__(self) -> None: + self.context = self.context or {} # `is None`, not truthiness: an explicit [] means "this caller has # no suggestions to offer", which the default list would override. - self.suggested_fixes = ( - suggested_fixes if suggested_fixes is not None - else self._get_default_suggestions(error_code)) - self.original_error = original_error - + if self.suggested_fixes is None: + self.suggested_fixes = self._get_default_suggestions(self.error_code) + def _get_default_suggestions(self, error_code: ErrorCode) -> List[str]: """Get default suggested fixes for error code.""" suggestions_map = { diff --git a/test/_api_v3_test_helpers.py b/test/_api_v3_test_helpers.py index 4e1439d4..e9ab39ac 100644 --- a/test/_api_v3_test_helpers.py +++ b/test/_api_v3_test_helpers.py @@ -41,8 +41,7 @@ def mock_plugin_catalog(): """ from src.plugin_system.plugin_catalog import PluginCatalog catalog = MagicMock(spec=PluginCatalog) - for name in ('plugins_dir', 'config_manager', 'schema_manager', - 'plugin_manifests', 'plugin_directories'): + for name in ('plugins_dir', 'plugin_manifests', 'plugin_directories'): setattr(catalog, name, MagicMock()) return catalog diff --git a/test/js/unit/test_html_escaping.js b/test/js/unit/test_html_escaping.js index cedc73ea..1b6f8437 100644 --- a/test/js/unit/test_html_escaping.js +++ b/test/js/unit/test_html_escaping.js @@ -12,9 +12,10 @@ // // CodeQL reported 83 js/incomplete-html-attribute-sanitization alerts for // exactly this. The web UI now has one implementation, window.LEDEscape in -// app-early.js, and the old per-file escapers are one-line names for it. This -// suite runs LEDEscape and every one of those names as shipped, and fails if a -// hand-rolled escaper appears anywhere else in web_interface/. +// app-early.js, which every page and widget calls directly (BaseWidget keeps +// an escapeHtml method for plugin widgets). This suite runs LEDEscape and that +// method as shipped, and fails if a hand-rolled escaper appears anywhere else +// in web_interface/. const fs = require('fs'); const path = require('path'); @@ -85,30 +86,6 @@ const ESCAPERS = [ ['app-early.js (LEDEscape.attr)', null, null, 'attr'], ['base-widget.js (BaseWidget.escapeHtml)', 'static/v3/js/widgets/base-widget.js', 'escapeHtml(text) {', 'escapeHtml', true], - ['plugins_manager.js (top-level escapeHtml)', - 'static/v3/plugins_manager.js', 'function escapeHtml(text) {', 'escapeHtml', false], - ['plugins_manager.js (starlark escapeHtml)', - 'static/v3/plugins_manager.js', 'function escapeHtml(str) {', 'escapeHtml', false], - ['json-file-manager.js (_esc)', - 'static/v3/js/widgets/json-file-manager.js', '_esc(str) {', '_esc', true], - ['plugin-file-manager.js (escHtml)', - 'static/v3/js/widgets/plugin-file-manager.js', 'function escHtml(s) {', 'escHtml', false], - ['plugins_manager.js (escapeAttribute)', - 'static/v3/plugins_manager.js', 'function escapeAttribute(text) {', 'escapeAttribute', false], - ['notification.js (escapeHtml)', - 'static/v3/js/widgets/notification.js', 'function escapeHtml(text) {', 'escapeHtml', false], - ['google-calendar-picker.js (escapeHtml)', - 'static/v3/js/widgets/google-calendar-picker.js', 'function escapeHtml(str) {', 'escapeHtml', false], - ['text-input.js (escapeHtml)', - 'static/v3/js/widgets/text-input.js', 'function escapeHtml(text) {', 'escapeHtml', false], - ['slider.js (escapeAttr)', - 'static/v3/js/widgets/slider.js', 'function escapeAttr(text) {', 'escapeAttr', false], - ['tools.html (escHtml)', - 'templates/v3/partials/tools.html', 'function escHtml(s) {', 'escHtml', false], - ['tools.html (phEscape)', - 'templates/v3/partials/tools.html', 'function phEscape(s) {', 'phEscape', false], - ['logs.html (escapeHtml)', - 'templates/v3/partials/logs.html', 'function escapeHtml(text) {', 'escapeHtml', false], // cache.html, backup_restore.html, operation_history.html and display.html // have no script any more (display.html's two escapers were never called): // their js/pages/ modules draw server data with textContent, and each @@ -169,9 +146,7 @@ console.log('\n4b. LEDEscape.jsStringAttr: a JS string literal that survives an console.log('\n4c. no hand-rolled escaper outside app-early.js'); { - const skip = new Set(['static/v3/js/app-early.js', - // documentation example, kept self-contained on purpose - 'static/v3/js/widgets/example-color-picker.js']); + const skip = new Set(['static/v3/js/app-early.js']); const found = []; const walk = dir => fs.readdirSync(dir, { withFileTypes: true }).forEach(e => { const p = path.join(dir, e.name); @@ -308,7 +283,7 @@ console.log('\n6. url-input onInput: previewLink.href is guarded at the sink'); // ── plugin-file-manager: cell edits travel via data-*, not inline handlers ── // A JSON key/day from an uploaded file used to be spliced, HTML-escaped, -// into an oninput="...('${escHtml(col)}'...)" attribute. escHtml neutralises +// into an oninput="...('${escHtml(col)}'...)" attribute. Escaping neutralises // a quote for an ordinary attribute, but here the value also has to survive // as a *JS string literal* -- the browser HTML-decodes the attribute before // running it as script, which turns the escaped quote back into a real one @@ -332,11 +307,10 @@ console.log("\n7. plugin-file-manager: cell edits never go through an inline han process.exit(1); } - const escHtmlFn = loadFn('static/v3/js/widgets/plugin-file-manager.js', 'function escHtml(s) {', 'escHtml', false); const renderEntryTableSrc = extractFn('function renderEntryTable(fieldId, container, content) {'); const calls = []; - const fakeWindow = { _pfmCellEdit: (fieldId, day, col, value) => calls.push({ fieldId, day, col, value }) }; + const fakeWindow = { LEDEscape, _pfmCellEdit: (fieldId, day, col, value) => calls.push({ fieldId, day, col, value }) }; class FakeContainer { constructor() { this._html = ''; this._listeners = {}; } @@ -354,12 +328,11 @@ console.log("\n7. plugin-file-manager: cell edits never go through an inline han } // eslint-disable-next-line no-eval - const renderEntryTable = eval(`(function(getState, escHtml, safeSetHTML, window){ + const renderEntryTable = eval(`(function(getState, safeSetHTML, window){ ${renderEntryTableSrc} return renderEntryTable; })`)( () => ({ entriesPerPage: 20, _tablePage: 1 }), - escHtmlFn, (target, html) => { target.innerHTML = html; }, fakeWindow ); diff --git a/test/js/unit/test_inline_handler_escaping.js b/test/js/unit/test_inline_handler_escaping.js index e32d6d8c..27d992d6 100644 --- a/test/js/unit/test_inline_handler_escaping.js +++ b/test/js/unit/test_inline_handler_escaping.js @@ -62,12 +62,12 @@ global.setGridHtmlIfChanged = (container, html) => { container.innerHTML = html; // eslint-disable-next-line no-eval eval([ - 'function escapeHtml(text) {', 'function escapeAttribute(text) {', 'function jsStringAttr(value) {', + 'function jsStringAttr(value) {', 'function renderPluginStore(plugins) {', 'function renderSavedRepositories(repositories) {', 'function renderCustomRegistryPlugins(plugins, registryUrl) {', -].map(extract).join('\n') + '\nglobal.jsStringAttr = jsStringAttr; global.escapeHtml = escapeHtml;' +].map(extract).join('\n') + '\nglobal.jsStringAttr = jsStringAttr;' + '\nglobal.renderPluginStore = renderPluginStore; global.renderSavedRepositories = renderSavedRepositories;' - + '\nglobal.renderCustomRegistryPlugins = renderCustomRegistryPlugins; global.escapeAttribute = escapeAttribute;'); + + '\nglobal.renderCustomRegistryPlugins = renderCustomRegistryPlugins;'); // ── minimal HTML start-tag tokenizer ─────────────────────────────────────── function decodeEntities(s) { diff --git a/test/js/unit/test_render_cards.js b/test/js/unit/test_render_cards.js index 741a7824..7edecb66 100644 --- a/test/js/unit/test_render_cards.js +++ b/test/js/unit/test_render_cards.js @@ -16,7 +16,7 @@ const container = { innerHTML: '', querySelectorAll: () => [], // no skeletons in this harness }; -// escapeHtml() escapes via a detached element, so mirror what a browser does +// LEDEscape.html() escapes via a detached element, so mirror what a browser does // when you read innerHTML back off textContent: & < > are escaped, quotes are not. class FakeEl { set textContent(v) { this._t = String(v == null ? '' : v); } @@ -36,7 +36,7 @@ global.PLUGIN_DEBUG = false; global.debugLog = () => {}; function setupInstalledEventDelegation() {} // stubbed; tested separately -eval(slice('function escapeHtml(text)', '\nfunction isNewPlugin')); +eval(slice('function jsStringAttr(value)', '\nfunction isNewPlugin')); eval(slice('function renderInstalledCards(plugins, total)', '// Set up event delegation for plugin action buttons')); diff --git a/test/js/unit/test_store_registry_fields.js b/test/js/unit/test_store_registry_fields.js index 968cd35f..99e477cc 100644 --- a/test/js/unit/test_store_registry_fields.js +++ b/test/js/unit/test_store_registry_fields.js @@ -51,7 +51,7 @@ global.installedPlugins = []; // eslint-disable-next-line no-eval eval([ - 'function escapeHtml(text) {', 'function escapeAttribute(text) {', 'function jsStringAttr(value) {', + 'function jsStringAttr(value) {', 'function isStorePluginInstalled(pluginIdOrPlugin) {', 'function findInstalledStorePlugin(pluginIdOrPlugin) {', 'function renderPluginStore(plugins) {', ].map(extract).join('\n') + '\nglobal.renderPluginStore = renderPluginStore;' diff --git a/test/js/unit/test_update_all.js b/test/js/unit/test_update_all.js index b3fbaa79..bd257218 100644 --- a/test/js/unit/test_update_all.js +++ b/test/js/unit/test_update_all.js @@ -253,9 +253,10 @@ const noSleep = { sleep: async () => {} }; for (const endpoint of bad) codes.push(await refusal(endpoint)); ok('an endpoint that could leave the API path is refused before fetch()', codes.every(c => c === 'INVALID_ENDPOINT') && urls.length === 0, { codes, urls }); - await PluginAPI.resetPluginConfig('a/../b&x=1'); + global.debugLog = () => {}; // GETs go through the throttler, which logs + await PluginAPI.getPluginHealth('a/../b&x=1'); ok('a plugin id is encoded into the URL, not spliced into it', - urls[0] === '/api/v3/plugins/config/reset?plugin_id=a%2F..%2Fb%26x%3D1', urls); + urls[0] === '/api/v3/plugins/health/a%2F..%2Fb%26x%3D1', urls); delete global.fetch; } diff --git a/test/test_cache_manager.py b/test/test_cache_manager.py index 9d1195a8..19e4d3fb 100644 --- a/test/test_cache_manager.py +++ b/test/test_cache_manager.py @@ -1,7 +1,7 @@ """ Tests for CacheManager and cache components. -Tests cache functionality including memory cache, disk cache, strategy, and metrics. +Tests cache functionality including memory cache, disk cache, and strategy. """ import pytest @@ -11,7 +11,6 @@ from src.cache_manager import CacheManager from src.cache.memory_cache import MemoryCache from src.cache.disk_cache import DiskCache from src.cache.cache_strategy import CacheStrategy -from src.cache.cache_metrics import CacheMetrics from datetime import datetime @@ -26,7 +25,6 @@ class TestCacheManager: assert hasattr(cm, '_memory_cache_component') assert hasattr(cm, '_disk_cache_component') assert hasattr(cm, '_strategy_component') - assert hasattr(cm, '_metrics_component') def test_set_and_get(self, tmp_path): """Test basic set and get operations.""" @@ -196,50 +194,6 @@ class TestMemoryCache: assert stats["max_size"] == 1000 # default -class TestCacheMetrics: - """Test CacheMetrics functionality.""" - - def test_record_hit(self): - """Test recording cache hit.""" - metrics = CacheMetrics() - metrics.record_hit() - stats = metrics.get_metrics() - - # get_metrics() returns calculated values, not raw hits/misses - assert stats['total_requests'] == 1 - assert stats['cache_hit_rate'] == 1.0 # 1 hit out of 1 request - - def test_record_miss(self): - """Test recording cache miss.""" - metrics = CacheMetrics() - metrics.record_miss() - stats = metrics.get_metrics() - - # get_metrics() returns calculated values, not raw hits/misses - assert stats['total_requests'] == 1 - assert stats['cache_hit_rate'] == 0.0 # 0 hits out of 1 request - - def test_record_fetch_time(self): - """Test recording fetch time.""" - metrics = CacheMetrics() - metrics.record_fetch_time(0.5) - stats = metrics.get_metrics() - - assert stats['fetch_count'] == 1 - assert stats['total_fetch_time'] == 0.5 - assert stats['average_fetch_time'] == 0.5 - - def test_cache_hit_rate(self): - """Test cache hit rate calculation.""" - metrics = CacheMetrics() - metrics.record_hit() - metrics.record_hit() - metrics.record_miss() - - stats = metrics.get_metrics() - assert stats['cache_hit_rate'] == pytest.approx(0.666, abs=0.01) - - class TestDiskCache: """Test DiskCache functionality.""" @@ -370,36 +324,6 @@ class TestDiskCache: # Should handle gracefully assert result is None or isinstance(result, dict) - - def test_record_background_hit(self): - """Test recording background cache hit.""" - metrics = CacheMetrics() - metrics.record_hit(cache_type='background') - stats = metrics.get_metrics() - - assert stats['total_requests'] == 1 - assert stats['background_hit_rate'] == 1.0 - - def test_record_background_miss(self): - """Test recording background cache miss.""" - metrics = CacheMetrics() - metrics.record_miss(cache_type='background') - stats = metrics.get_metrics() - - assert stats['total_requests'] == 1 - assert stats['background_hit_rate'] == 0.0 - - def test_multiple_fetch_times(self): - """Test recording multiple fetch times.""" - metrics = CacheMetrics() - metrics.record_fetch_time(0.5) - metrics.record_fetch_time(1.0) - metrics.record_fetch_time(0.3) - - stats = metrics.get_metrics() - assert stats['fetch_count'] == 3 - assert stats['total_fetch_time'] == 1.8 - assert stats['average_fetch_time'] == pytest.approx(0.6, abs=0.01) class TestDiskCacheWriteEconomy: diff --git a/test/test_cache_strategy_intervals.py b/test/test_cache_strategy_intervals.py index ef489973..e9594ea3 100644 --- a/test/test_cache_strategy_intervals.py +++ b/test/test_cache_strategy_intervals.py @@ -1,12 +1,10 @@ """CacheStrategy intervals, pinned across the whole input grid. The strategy table used to carry a per-sport defaults dict whose every value -was 60, and a soccer branch identical to its else. These tests pin the -returned strategy for every data type x sport key x config shape, so -simplifying the lookup cannot change what any caller gets back. They were -written against the pre-cleanup code and pass on it unchanged, except for -the legacy `_scoreboard` config shape (see below), which that code -still read. +was 60, a soccer branch identical to its else, and a config lookup of +`_scoreboard` sections that only the replaced built-in scoreboards +had. These tests pin the returned strategy for every data type x sport key, +so simplifying the lookup cannot change what any caller gets back. """ import pytest @@ -14,46 +12,6 @@ import pytest from src.cache.cache_strategy import CacheStrategy -class _Cfg: - def __init__(self, config): - self.config = config - - -class _NoConfigAttr: - pass - - -# Plugin config sections are keyed by plugin id. Their intervals belong to the -# plugin, and the strategy table has never read them. -_PLUGIN_ID_CONFIG = { - pid: {"live_update_interval": 5, "recent_update_interval": 7, - "upcoming_update_interval": 9} - for pid in ("football-scoreboard", "basketball-scoreboard", - "baseball-scoreboard", "hockey-scoreboard", "soccer-scoreboard") -} - -# `_scoreboard` sections come from the built-in scoreboards the plugin -# system replaced. An install upgraded from that era can still carry them in -# config.json (nothing deletes them). No current caller passes a sport key to -# the strategy, but a stale section must not steer cache TTLs if one does. -_LEGACY_SCOREBOARD_CONFIG = { - f"{sport}_scoreboard": {"live_update_interval": 5, - "recent_update_interval": 7, - "upcoming_update_interval": 9} - for sport in ("nfl", "nba", "mlb", "nhl", "soccer", "ncaa_fb", - "ncaa_baseball", "ncaam_basketball", "milb") -} - -CONFIG_MANAGERS = { - "no_config_manager": None, - "empty_config": _Cfg({}), - "plugin_id_config": _Cfg(_PLUGIN_ID_CONFIG), - "legacy_scoreboard_config": _Cfg(_LEGACY_SCOREBOARD_CONFIG), - "config_is_none": _Cfg(None), - "config_is_not_a_dict": _Cfg("x"), - "config_manager_without_config": _NoConfigAttr(), -} - SPORT_KEYS = [None, "", "nfl", "nba", "mlb", "nhl", "soccer", "ncaa_fb", "ncaa_baseball", "ncaam_basketball", "milb", "football-scoreboard", "curling"] @@ -93,16 +51,8 @@ def _expected(data_type, sport_key): return FIXED.get(data_type, DEFAULT) -@pytest.mark.parametrize("cm_name", sorted(CONFIG_MANAGERS)) -def test_live_interval_is_60_for_every_sport(cm_name): - strategy = CacheStrategy(config_manager=CONFIG_MANAGERS[cm_name]) - for sport_key in SPORT_KEYS: - assert strategy.get_sport_live_interval(sport_key) == 60, sport_key - - -@pytest.mark.parametrize("cm_name", sorted(CONFIG_MANAGERS)) -def test_strategy_table_for_every_data_type_and_sport(cm_name): - strategy = CacheStrategy(config_manager=CONFIG_MANAGERS[cm_name]) +def test_strategy_table_for_every_data_type_and_sport(): + strategy = CacheStrategy() data_types = ["live_scores", "sports_live", *FIXED, "unknown", ""] for data_type in data_types: for sport_key in SPORT_KEYS: diff --git a/test/test_deprecation.py b/test/test_deprecation.py index 82469291..5bd521f4 100644 --- a/test/test_deprecation.py +++ b/test/test_deprecation.py @@ -32,9 +32,46 @@ DEPRECATED_3_9 = { ], } +#: Deprecated after the October 2026 over-engineering audit, for removal in +#: 3.10.0: nothing in core, the monorepo or the registry's third-party plugins +#: calls them. +DEPRECATED_3_10 = { + "src.logo_downloader.LogoDownloader": [ + "fetch_teams_data", "extract_teams_from_data", "download_missing_logos_for_league", + "download_all_ncaa_football_logos", "download_all_missing_logos", + "convert_image_to_rgba", "convert_all_logos_to_rgba", + ], + "src.config_manager.ConfigManager": [ + "rollback_config", "list_backups", "validate_config_file", "get_secret", + "cleanup_orphaned_plugin_configs", "validate_all_plugin_configs", + ], + "src.common.api_helper.APIHelper": [ + "fetch_espn_scoreboard", "fetch_espn_standings", "fetch_espn_rankings", + "set_cache", "get_cache", "set_rate_limit", "get_request_stats", + ], + "src.plugin_system.testing.plugin_test_base.PluginTestCase": ["setUp"], + "src.background_data_service.BackgroundDataService": [ + "get_result", "is_request_complete", "get_request_status", + ], + "src.plugin_system.plugin_manager.PluginManager": [ + "get_all_plugins", "get_plugin_info", "get_all_plugin_info", + "get_plugin_display_modes", "find_plugin_for_mode", + ], + "src.plugin_system.plugin_state.PluginStateManager": [ + "is_loaded", "is_running", "is_error", "get_last_update", "get_error_info", + "get_state_info", + ], + "src.cache_manager.CacheManager": ["load_cache", "generate_sport_cache_key"], + "src.font_manager.FontManager": ["measure_text", "get_native_bdf_size"], + "src.base_odds_manager.BaseOddsManager": ["get_odds_for_games", "format_odds_summary"], + "src.dynamic_team_resolver.DynamicTeamResolver": [ + "get_available_dynamic_teams", "is_dynamic_team", + ], +} + #: Every pinned marker: (class path, method) -> the release that removes it. PINNED = {(path, name): removal - for removal, table in (("3.9.0", DEPRECATED_3_9),) + for removal, table in (("3.9.0", DEPRECATED_3_9), ("3.10.0", DEPRECATED_3_10)) for path, names in table.items() for name in names} diff --git a/test/test_display_pending_changes.py b/test/test_display_pending_changes.py index 0a2443a7..794aef02 100644 --- a/test/test_display_pending_changes.py +++ b/test/test_display_pending_changes.py @@ -145,7 +145,6 @@ def vegas_coordinator(controller): coord.render_pipeline.target_fps = float(coord.vegas_config.target_fps) coord.stream_manager = MagicMock() coord.display_manager = controller.display_manager - coord.stats = {'cycles_completed': 0, 'interruptions': 0} coord._state_lock = threading.Lock() coord._is_active = True coord._is_paused = False diff --git a/test/test_error_aggregator.py b/test/test_error_aggregator.py index 2d8ca40e..aa20ccef 100644 --- a/test/test_error_aggregator.py +++ b/test/test_error_aggregator.py @@ -218,24 +218,6 @@ class TestPatternDetection: assert pattern is not None assert pattern.severity in ["error", "critical"] - def test_pattern_callback_called(self): - """Pattern detection callback should be called.""" - aggregator = ErrorAggregator(pattern_threshold=2) - - callback_called = [] - - def callback(pattern): - callback_called.append(pattern) - - aggregator.on_pattern_detected(callback) - - # Trigger pattern - for _ in range(3): - aggregator.record_error(error=ValueError("Pattern trigger")) - - assert len(callback_called) == 1 - assert callback_called[0].error_type == "ValueError" - class TestErrorSummary: """Test error summary generation.""" @@ -318,37 +300,6 @@ class TestPluginHealth: assert health["recent_error_count"] == 10 -class TestRecordClearing: - """Test clearing old records.""" - - def test_clear_old_records(self): - """Old records should be cleared.""" - aggregator = ErrorAggregator() - - # Add a record - aggregator.record_error(error=ValueError("Old error")) - - # Manually age the record - aggregator._records[0].timestamp = datetime.now() - timedelta(hours=48) - - # Clear records older than 24 hours - cleared = aggregator.clear_old_records(max_age_hours=24) - - assert cleared == 1 - assert len(aggregator._records) == 0 - - def test_recent_records_not_cleared(self): - """Recent records should not be cleared.""" - aggregator = ErrorAggregator() - - aggregator.record_error(error=ValueError("Recent error")) - - cleared = aggregator.clear_old_records(max_age_hours=24) - - assert cleared == 0 - assert len(aggregator._records) == 1 - - class TestThreadSafety: """Test thread safety of error aggregator.""" diff --git a/test/test_frame_ops.py b/test/test_frame_ops.py index 0fc3f622..dec1d772 100644 --- a/test/test_frame_ops.py +++ b/test/test_frame_ops.py @@ -330,9 +330,6 @@ class _Stream: def get_grouped_content_for_composition(self): return self.groups[0] - def get_active_plugin_ids(self): - return [pid for pid, _ in self.groups[0]] - def take_next_group(self, count=None, offscreen_only=False): if self._i >= len(self.groups): return [] diff --git a/test/test_ipc_contract.py b/test/test_ipc_contract.py index 1847f039..42fb2a24 100644 --- a/test/test_ipc_contract.py +++ b/test/test_ipc_contract.py @@ -60,7 +60,9 @@ class TestRoundTrip: def test_start_args_round_trip(self): args = OnDemandStartArgs(plugin_id='clock', mode='clock_main', duration=45.0, pinned=True) - assert OnDemandStartArgs.from_dict(_wire(args.to_dict())) == args + wire = _wire({'plugin_id': 'clock', 'mode': 'clock_main', 'duration': 45.0, + 'pinned': True}) + assert OnDemandStartArgs.from_dict(wire) == args def test_encoded_messages_are_ascii_single_lines(self): data = c.encode_message({'v': 1, 'id': 'x', 'cmd': 'ping', diff --git a/test/test_ipc_display_stage2.py b/test/test_ipc_display_stage2.py index 2d745665..93d0c085 100644 --- a/test/test_ipc_display_stage2.py +++ b/test/test_ipc_display_stage2.py @@ -349,7 +349,6 @@ class TestVegasChecksEveryFrame: coord.render_pipeline.frame_interval = 0.0 coord.render_pipeline.target_fps = 90 coord.display_manager = MagicMock() - coord.stats = {'cycles_completed': 0, 'interruptions': 0} coord._state_lock = threading.Lock() coord._is_active = True coord._is_paused = False diff --git a/test/test_ipc_server.py b/test/test_ipc_server.py index 4d15e47f..fad5dc4d 100644 --- a/test/test_ipc_server.py +++ b/test/test_ipc_server.py @@ -215,7 +215,7 @@ class TestWhereTheServerListens: assert srv.server_socket_path({}) is None assert ControlServer('x.sock').start() is False with pytest.raises(client.ControlError) as e: - client.ping(paths=['x.sock']) + client.request(Command.PING, paths=['x.sock']) assert e.value.reason == 'unsupported' @@ -271,7 +271,7 @@ def _read_line(s): class TestLiveSocket: def test_client_round_trip(self, live, sock_path): live() - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} assert client.hello(paths=[sock_path])['version'] == 1 assert client.on_demand_status(paths=[sock_path])['current_mode'] == 'clock' @@ -335,7 +335,7 @@ class TestLiveSocket: assert s.recv(10) == b'' # and hung up finally: s.close() - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} def test_a_client_that_hangs_up_mid_message(self, live, sock_path): server = live() @@ -343,7 +343,7 @@ class TestLiveSocket: s.sendall(b'{"v":1,"id":"half","cmd":"on_demand.st') s.close() time.sleep(0.2) - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} assert server.drain() == [] def test_a_slow_client_is_dropped_and_blocks_nobody(self, live, sock_path): @@ -352,7 +352,7 @@ class TestLiveSocket: try: slow.sendall(b'{"v":1,') # ...and never finishes t0 = time.monotonic() - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} assert time.monotonic() - t0 < 0.5, 'a slow client held up another' assert slow.recv(100) == b'' # hung up on, not answered finally: @@ -380,7 +380,7 @@ class TestLiveSocket: for s in held: s.close() time.sleep(0.3) - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} def test_many_concurrent_clients(self, live, sock_path): server = live(queue_size=64) @@ -408,7 +408,7 @@ class TestLiveSocket: s = ControlServer(sock_path, status_provider=lambda: status) try: assert s.start() - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} finally: s.close() @@ -416,7 +416,7 @@ class TestLiveSocket: live() second = ControlServer(sock_path, status_provider=lambda: status) assert second.start() is False - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} def test_a_regular_file_is_never_removed(self, sock_path): with open(sock_path, 'w') as f: @@ -434,24 +434,24 @@ class TestLiveSocket: assert second.start() try: first.close() # must not unlink second's file - assert client.ping(paths=[sock_path]) == {'pong': True} + assert client.request(Command.PING, paths=[sock_path]) == {'pong': True} finally: second.close() def test_client_reasons(self, sock_path, tmp_path): with pytest.raises(client.ControlError) as e: - client.ping(paths=[sock_path]) + client.request(Command.PING, paths=[sock_path]) assert e.value.reason == 'no_socket' dead = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) dead.bind(sock_path) try: with pytest.raises(client.ControlError) as e: - client.ping(paths=[sock_path]) + client.request(Command.PING, paths=[sock_path]) assert e.value.reason == 'refused' finally: dead.close() with pytest.raises(client.ControlError) as e: - client.ping(paths=[]) + client.request(Command.PING, paths=[]) assert e.value.reason == 'disabled' with pytest.raises(client.ControlError) as e: client.on_demand_start('x', None, None, paths=[sock_path]) @@ -464,7 +464,7 @@ class TestLiveSocket: try: t0 = time.monotonic() with pytest.raises(client.ControlError) as e: - client.ping(paths=[sock_path], timeout=0.3) + client.request(Command.PING, paths=[sock_path], timeout=0.3) assert e.value.reason == 'timeout' assert time.monotonic() - t0 < 1.0 finally: @@ -542,7 +542,7 @@ class TestPermissions: os.setgroups([]) os.setgid(gid) os.setuid(nobody.pw_uid) - result = json.dumps(client.ping(paths=[sock_path])) + result = json.dumps(client.request(Command.PING, paths=[sock_path])) except client.ControlError as e: result = 'error:' + e.reason except Exception as e: # report anything else to the parent diff --git a/test/test_ipc_state_stream.py b/test/test_ipc_state_stream.py index 5418a945..6985bbd0 100644 --- a/test/test_ipc_state_stream.py +++ b/test/test_ipc_state_stream.py @@ -522,7 +522,7 @@ class TestLiveStream: # max_clients is 2 and three streams are open: subscribers gave # their request slots back. for _ in range(4): - assert client.ping(paths=[path]) == {'pong': True} + assert client.request(Command.PING, paths=[path]) == {'pong': True} hub.publish('display', _display(mode='weather'), volatile=('last_updated',)) assert _wait_until(lambda: all( s.latest()['state']['display']['mode'] == 'weather' for s in subs)) diff --git a/test/test_operation_queue_pending_and_trim.py b/test/test_operation_queue_pending_and_trim.py index 15559732..e53750da 100644 --- a/test/test_operation_queue_pending_and_trim.py +++ b/test/test_operation_queue_pending_and_trim.py @@ -54,7 +54,7 @@ def test_second_pending_operation_for_a_plugin_is_refused(op_queue): assert _wait_for(lambda: op_queue.get_operation_status(first).status == OperationStatus.COMPLETED) # Once it has finished, the plugin accepts a new operation again. - op_queue.enqueue_operation(OperationType.UPDATE, "demo") + op_queue.enqueue_operation(OperationType.UNINSTALL, "demo") def test_operations_map_is_trimmed_with_history(op_queue): @@ -64,8 +64,8 @@ def test_operations_map_is_trimmed_with_history(op_queue): assert _wait_for(lambda: all( (op_queue.get_operation_status(i) is None or op_queue.get_operation_status(i).status == OperationStatus.COMPLETED) - for i in ids) and len(op_queue.get_operation_history()) == 3) + for i in ids) and len(op_queue._operation_history) == 3) assert len(op_queue._operations) == 3 - kept = {op.operation_id for op in op_queue.get_operation_history()} + kept = {op.operation_id for op in op_queue._operation_history} assert set(op_queue._operations) == kept diff --git a/test/test_render_gate.py b/test/test_render_gate.py index 58f0b5e1..b2f34ea0 100644 --- a/test/test_render_gate.py +++ b/test/test_render_gate.py @@ -404,7 +404,6 @@ class TestVegasWiring: off = VegasModeConfig.from_config( {"display": {"vegas_scroll": {"prefetch_gate": False}}}) assert off.prefetch_gate is False - assert off.to_dict()["prefetch_gate"] is False assert VegasModeConfig.from_config( {"display": {"vegas_scroll": {}}}).prefetch_gate is True diff --git a/test/test_scroll_helper_lazy_image.py b/test/test_scroll_helper_lazy_image.py index e49bc5f0..e785b81d 100644 --- a/test/test_scroll_helper_lazy_image.py +++ b/test/test_scroll_helper_lazy_image.py @@ -187,9 +187,6 @@ def test_vegas_extends_without_building_the_image(no_fromarray): def get_grouped_content_for_composition(self): return groups[0] - def get_active_plugin_ids(self): - return ["a"] - def take_next_group(self, count=None, offscreen_only=False): self.i += 1 return groups[self.i] if self.i < len(groups) else [] diff --git a/test/test_sync_manager.py b/test/test_sync_manager.py index 3b53a2eb..4cd2576e 100644 --- a/test/test_sync_manager.py +++ b/test/test_sync_manager.py @@ -78,7 +78,6 @@ def make_manager(role=SyncRole.STANDALONE, hw_config=None): mgr._last_leader_frame_time = 0.0 mgr._frame_lock = threading.Lock() mgr._leader_ip = None - mgr._on_new_cycle = None mgr._on_scroll_image = None mgr._pending_scroll_image = None mgr._scroll_image_lock = threading.Lock() @@ -495,31 +494,35 @@ class TestFollowerRecvLoop: assert mgr._peer_compatible is True assert mgr.logger.error.called is False - def test_scroll_x_switches_to_follower_and_builds_cycle(self): + def test_scroll_x_switches_to_follower(self): mgr = make_manager(role=SyncRole.FOLLOWER) - calls = [] - mgr._on_new_cycle = lambda: calls.append(1) self._drive(mgr, json.dumps({"t": "sx", "x": 12.34}).encode()) assert mgr._follower_state is FollowerState.FOLLOWER assert mgr.get_latest_scroll_x() == 12.34 - assert calls == [1] - def test_scroll_x_while_already_following_does_not_rebuild(self): + def test_scroll_x_while_already_following_updates_the_position(self): mgr = make_manager(role=SyncRole.FOLLOWER) mgr._follower_state = FollowerState.FOLLOWER - calls = [] - mgr._on_new_cycle = lambda: calls.append(1) self._drive(mgr, json.dumps({"t": "sx", "x": 5.0}).encode()) assert mgr.get_latest_scroll_x() == 5.0 - assert calls == [] - def test_new_cycle_message_triggers_callback(self): + @pytest.mark.parametrize("payload", [{"t": "nc"}, {"t": "some-future-type"}]) + def test_an_older_or_newer_leaders_message_is_ignored(self, payload): + # Older leaders send "nc" at each new cycle. Nothing uses it, and a + # follower must take it -- or any type it does not know -- quietly: + # not as a frame, not as a malformed packet, no error back-off. mgr = make_manager(role=SyncRole.FOLLOWER) mgr._follower_state = FollowerState.FOLLOWER - calls = [] - mgr._on_new_cycle = lambda: calls.append(1) - self._drive(mgr, json.dumps({"t": "nc"}).encode()) - assert calls == [1] + sleeps = MagicMock() + with patch.object(sync_manager, "time", + SimpleNamespace(time=time.time, monotonic=time.monotonic, + sleep=sleeps)): + self._drive(mgr, json.dumps(payload).encode()) + assert mgr._follower_state is FollowerState.FOLLOWER + assert mgr.get_latest_frame() is None + assert mgr.get_latest_scroll_x() is None + assert not mgr.logger.debug.called + sleeps.assert_not_called() def test_non_object_json_does_not_reach_the_outer_handler(self): # A bare JSON scalar parses, then msg.get() raises AttributeError. @@ -547,28 +550,6 @@ class TestFollowerRecvLoop: assert mgr.get_latest_scroll_x() is None sleeps.assert_not_called() - def test_callback_failure_is_not_mistaken_for_a_malformed_packet(self, monkeypatch): - # A payload that parses is a control message, full stop. If the - # callback it triggers raises one of the types the field guard - # catches, that fault belongs to the callback: it must not send - # the packet to the image decoder, which would report it as a - # decode error and bury the real cause. The loop still survives - # it — the outer handler catches it like any other fault. - mgr = make_manager(role=SyncRole.FOLLOWER) - mgr._follower_state = FollowerState.FOLLOWER - - def boom(): - raise ValueError("callback is broken") - - mgr._on_new_cycle = boom - fake_clock(monkeypatch, sleep_fn=MagicMock()) - self._drive(mgr, json.dumps({"t": "nc"}).encode()) - - logged = " | ".join(str(c) for c in mgr.logger.debug.call_args_list) - assert "callback is broken" in logged - assert "frame decode error" not in logged - assert "malformed control message" not in logged - def test_oversized_legacy_frame_is_rejected_before_decode(self, monkeypatch): # The UDP path is reachable by any host on the LAN, so it caps # dimensions before load() just as the TCP image server does. @@ -603,12 +584,9 @@ class TestFollowerRecvLoop: for payload in (b'{"t": "sx", "x": ' + literal.encode() + b'}', json.dumps({"t": "sx", "x": literal}).encode()): mgr = make_manager(role=SyncRole.FOLLOWER) - calls = [] - mgr._on_new_cycle = lambda: calls.append(1) self._drive(mgr, payload) assert mgr.get_latest_scroll_x() is None assert mgr._follower_state is FollowerState.STANDALONE - assert calls == [] def test_non_finite_scroll_x_leaves_a_good_value_in_place(self): # The reject must not clear the last usable position either — a @@ -697,17 +675,10 @@ class TestSendControlMessages: msg = json.loads(mgr._send_sock.sendto.call_args[0][0].decode()) assert msg == {"t": "sx", "x": 3.14} - def test_send_new_cycle(self): - mgr = self._connected_leader() - mgr.send_new_cycle() - msg = json.loads(mgr._send_sock.sendto.call_args[0][0].decode()) - assert msg == {"t": "nc"} - def test_control_messages_noop_when_disconnected(self): mgr = self._connected_leader() mgr._leader_state = LeaderState.NO_PEER mgr.send_scroll_x(1.0) - mgr.send_new_cycle() assert not mgr._send_sock.sendto.called def test_set_leader_width(self): diff --git a/test/test_vegas_config.py b/test/test_vegas_config.py index 3b51903c..b87fdfb8 100644 --- a/test/test_vegas_config.py +++ b/test/test_vegas_config.py @@ -1,10 +1,12 @@ """ Tests for src/vegas_mode/config.py -Covers VegasModeConfig: from_config, to_dict, get_frame_interval, +Covers VegasModeConfig: from_config, get_frame_interval, get_ordered_plugins, validate. """ +import dataclasses + import pytest from src.vegas_mode.config import VegasModeConfig @@ -49,7 +51,8 @@ class TestVegasModeConfigDefaults: (Path(__file__).resolve().parent.parent / "config" / "config.template.json").read_text(encoding="utf-8")) shipped = template["display"]["vegas_scroll"] - defaults = VegasModeConfig().to_dict() + defaults = dataclasses.asdict(VegasModeConfig()) + defaults["excluded_plugins"] = sorted(defaults["excluded_plugins"]) mismatched = {k: (v, defaults[k]) for k, v in shipped.items() if k in defaults and defaults[k] != v} assert not mismatched, f"template vs code default: {mismatched}" @@ -57,9 +60,9 @@ class TestVegasModeConfigDefaults: def test_missing_keys_read_the_field_defaults(self): # from_config used to repeat every default; with no keys set it must # produce exactly the dataclass defaults. - assert VegasModeConfig.from_config({}).to_dict() == VegasModeConfig().to_dict() - assert (VegasModeConfig.from_config({"display": {"vegas_scroll": {}}}).to_dict() - == VegasModeConfig().to_dict()) + assert VegasModeConfig.from_config({}) == VegasModeConfig() + assert (VegasModeConfig.from_config({"display": {"vegas_scroll": {}}}) + == VegasModeConfig()) # --------------------------------------------------------------------------- @@ -114,42 +117,6 @@ class TestFromConfig: assert cfg.frame_based_scrolling is False -# --------------------------------------------------------------------------- -# to_dict -# --------------------------------------------------------------------------- - -class TestToDict: - def test_roundtrip(self): - original = VegasModeConfig( - enabled=True, - scroll_speed=75.0, - separator_width=24, - plugin_order=["a", "b"], - excluded_plugins={"z"}, - target_fps=100, - ) - d = original.to_dict() - assert d["enabled"] is True - assert d["scroll_speed"] == 75.0 - assert d["separator_width"] == 24 - assert d["plugin_order"] == ["a", "b"] - assert "z" in d["excluded_plugins"] - assert d["target_fps"] == 100 - - def test_excluded_plugins_is_list(self): - cfg = VegasModeConfig(excluded_plugins={"x"}) - d = cfg.to_dict() - assert isinstance(d["excluded_plugins"], list) - - def test_all_keys_present(self): - d = VegasModeConfig().to_dict() - for key in ("enabled", "scroll_speed", "separator_width", "plugin_order", - "excluded_plugins", "target_fps", "buffer_ahead", - "frame_based_scrolling", "scroll_delay", - "dynamic_duration_enabled", "min_cycle_duration", "max_cycle_duration"): - assert key in d - - # --------------------------------------------------------------------------- # get_frame_interval # --------------------------------------------------------------------------- diff --git a/test/test_vegas_continuous_refresh.py b/test/test_vegas_continuous_refresh.py index 3b885f1b..7032588e 100644 --- a/test/test_vegas_continuous_refresh.py +++ b/test/test_vegas_continuous_refresh.py @@ -189,7 +189,6 @@ class TestCoordinatorWiring: coordinator.render_pipeline.is_cycle_complete.return_value = False coordinator.render_pipeline.should_recompose.return_value = False coordinator.stream_manager = MagicMock() - coordinator.stats = {'cycles_completed': 0} coordinator._state_lock = threading.Lock() coordinator._is_active = True coordinator._is_paused = False diff --git a/test/test_vegas_coordinator_config.py b/test/test_vegas_coordinator_config.py index c5b1263e..48ace3b0 100644 --- a/test/test_vegas_coordinator_config.py +++ b/test/test_vegas_coordinator_config.py @@ -25,7 +25,6 @@ def _coordinator(active=True): c.render_pipeline.is_cycle_complete.return_value = False c.stream_manager = MagicMock() c.plugin_adapter = MagicMock() - c.stats = {'cycles_completed': 0, 'config_updates': 0} c._state_lock = threading.Lock() c._is_active = active c._is_paused = False diff --git a/test/test_vegas_coordinator_iteration.py b/test/test_vegas_coordinator_iteration.py index 8e73909a..4ed319fb 100644 --- a/test/test_vegas_coordinator_iteration.py +++ b/test/test_vegas_coordinator_iteration.py @@ -27,7 +27,6 @@ def _coordinator(plugins): coord.stream_manager = MagicMock() coord.display_manager = MagicMock() coord.plugin_manager = SimpleNamespace(plugins=plugins, get_plugin=plugins.get) - coord.stats = {'cycles_completed': 0, 'interruptions': 0} coord._state_lock = threading.Lock() coord._is_active = True coord._is_paused = False @@ -99,7 +98,6 @@ def test_vegas_resumes_after_a_live_priority_pause(): def test_stop_clears_a_live_priority_pause(): coord = _live_coordinator(['nfl_live']) - coord._start_time = None coord._restore_switch_interval = lambda: None coord._remove_render_gate = lambda: None coord.run_iteration() diff --git a/test/test_vegas_crisp_pacing.py b/test/test_vegas_crisp_pacing.py index c5b87005..dcd984d6 100644 --- a/test/test_vegas_crisp_pacing.py +++ b/test/test_vegas_crisp_pacing.py @@ -26,9 +26,6 @@ class FakeStream: def get_grouped_content_for_composition(self): return [('a', [Image.new('RGB', (4000, H), (255, 255, 255))])] - def get_active_plugin_ids(self): - return ['a'] - class FakeDM: width = W diff --git a/test/test_vegas_density.py b/test/test_vegas_density.py index 2118cfd5..594492b7 100644 --- a/test/test_vegas_density.py +++ b/test/test_vegas_density.py @@ -3,6 +3,7 @@ Tests for the Vegas mode density work: dead-space trimming in PluginAdapter and the configurable lead-in gap in ScrollHelper. """ +import dataclasses from contextlib import contextmanager import pytest @@ -298,9 +299,6 @@ class TestPluginBoundaryGaps: def get_grouped_content_for_composition(self): return grouped - def get_active_plugin_ids(self): - return [pid for pid, _ in grouped] - class DM: width = DISPLAY_W height = DISPLAY_H @@ -615,10 +613,10 @@ class TestConfigSurface: assert cfg.lead_in_width == 0 assert cfg.content_padding == 8 - def test_round_trips_through_to_dict(self): + def test_round_trips_through_asdict(self): cfg = VegasModeConfig(trim_threshold=20, lead_in_width=64) restored = VegasModeConfig.from_config( - {'display': {'vegas_scroll': cfg.to_dict()}}) + {'display': {'vegas_scroll': dataclasses.asdict(cfg)}}) assert restored.trim_threshold == 20 assert restored.lead_in_width == 64 @@ -698,9 +696,6 @@ class TestMeasuredSeparation: def get_grouped_content_for_composition(self): return grouped - def get_active_plugin_ids(self): - return [pid for pid, _ in grouped] - class DM: width = DISPLAY_W height = DISPLAY_H @@ -816,9 +811,6 @@ class TestCycleEndsBeforeWrap: def get_grouped_content_for_composition(self): return [('a', [Image.new('RGB', (strip_width, DISPLAY_H), (255, 255, 255))])] - def get_active_plugin_ids(self): - return ['a'] - class DM: width = DISPLAY_W height = DISPLAY_H @@ -962,9 +954,6 @@ class TestBudgetUsesMeasuredGaps: def get_grouped_content_for_composition(self): return [('rows', selected)] - def get_active_plugin_ids(self): - return ['rows'] - class DM: width = DISPLAY_W height = DISPLAY_H @@ -1258,9 +1247,6 @@ class TestContinuousExtension: def get_grouped_content_for_composition(self): return groups[0] if groups else [] - def get_active_plugin_ids(self): - return [pid for pid, _ in (groups[0] if groups else [])] - def take_next_group(self, count=None, offscreen_only=False): self.calls.append(offscreen_only) if self._i >= len(groups): @@ -1450,9 +1436,6 @@ class TestDeferredDraining: def get_grouped_content_for_composition(self): return [('seed', [Image.new('RGB', (600, DISPLAY_H), (255, 255, 255))])] - def get_active_plugin_ids(self): - return ['seed'] - def take_next_group(self, count=None, offscreen_only=False): if self._served: return [] diff --git a/test/test_vegas_elements_layout.py b/test/test_vegas_elements_layout.py index 68961405..846e44ef 100644 --- a/test/test_vegas_elements_layout.py +++ b/test/test_vegas_elements_layout.py @@ -49,9 +49,6 @@ class _Stream: def get_grouped_content_for_composition(self): return self.groups[0] - def get_active_plugin_ids(self): - return [pid for pid, _ in self.groups[0]] - def take_next_group(self, count=None, offscreen_only=False): self.i += 1 return self.groups[self.i] if self.i < len(self.groups) else [] diff --git a/test/test_vegas_live_apply.py b/test/test_vegas_live_apply.py index 1be111d6..0b3aa4b6 100644 --- a/test/test_vegas_live_apply.py +++ b/test/test_vegas_live_apply.py @@ -43,9 +43,6 @@ class _Stream: def get_grouped_content_for_composition(self): return self.groups[0] - def get_active_plugin_ids(self): - return ["p"] - def take_next_group(self, count=None, offscreen_only=False): self.i += 1 return self.groups[self.i] if self.i < len(self.groups) else [] diff --git a/test/test_vegas_live_integration.py b/test/test_vegas_live_integration.py index 467f60bc..e186226e 100644 --- a/test/test_vegas_live_integration.py +++ b/test/test_vegas_live_integration.py @@ -189,7 +189,7 @@ def test_with_live_refresh_off_the_same_run_is_plain_content(dm, tmp_path, ticke start_width = pipeline.scroll_helper.total_scroll_width assert _run_until( coordinator, - lambda: pipeline.stats.get('extensions', 0) >= 1, seconds=10.0) + lambda: pipeline.extensions >= 1, seconds=10.0) assert pipeline.live_records() == () assert np.asarray(pipeline.scroll_helper.cached_array).shape[1] > 0 assert start_width > 0 diff --git a/test/test_vegas_live_lifecycle.py b/test/test_vegas_live_lifecycle.py index 8729c00b..f49a4b21 100644 --- a/test/test_vegas_live_lifecycle.py +++ b/test/test_vegas_live_lifecycle.py @@ -46,9 +46,6 @@ class _Stream: def get_grouped_content_for_composition(self): return self.first - def get_active_plugin_ids(self): - return ["p"] - def take_next_group(self, count=None, offscreen_only=False): self.taken += 1 return [("q", [_image(40, 99)])] diff --git a/test/test_vegas_live_property.py b/test/test_vegas_live_property.py index 75fdcfdf..3c035e3e 100644 --- a/test/test_vegas_live_property.py +++ b/test/test_vegas_live_property.py @@ -67,9 +67,6 @@ class _Stream: def get_grouped_content_for_composition(self): return self.groups[0] - def get_active_plugin_ids(self): - return [] - def take_next_group(self, count=None, offscreen_only=False): self.i += 1 return self.groups[self.i] if self.i < len(self.groups) else [] diff --git a/test/test_vegas_prepared_blocks.py b/test/test_vegas_prepared_blocks.py index f9df9289..75e73c63 100644 --- a/test/test_vegas_prepared_blocks.py +++ b/test/test_vegas_prepared_blocks.py @@ -65,9 +65,6 @@ class _Stream: def get_grouped_content_for_composition(self): return self.groups[0] - def get_active_plugin_ids(self): - return [] - def take_next_group(self, count=None, offscreen_only=False): self.i += 1 return self.groups[self.i] if self.i < len(self.groups) else [] diff --git a/test/test_vegas_static_mode.py b/test/test_vegas_static_mode.py index 3bfe1962..f8da3382 100644 --- a/test/test_vegas_static_mode.py +++ b/test/test_vegas_static_mode.py @@ -52,9 +52,6 @@ class FakeStream: def get_static_layout(self): return [(pid, w is None) for pid, w in self.layout] - def get_active_plugin_ids(self): - return [pid for pid, _ in self.layout] - def is_static_plugin(self, plugin_id): return plugin_id in self.statics @@ -163,7 +160,6 @@ class TestStreamManagerSkipsStaticContent: sm.plugin_manager = SimpleNamespace(plugins=plugins) sm.plugin_adapter = MagicMock() sm.plugin_adapter.get_content.return_value = [block(10)] - sm.stats = {'segments_fetched': 0, 'fetch_errors': 0} sm.refresh = lambda: None return sm @@ -213,7 +209,6 @@ class TestCoordinatorStaticPause: coord._live_priority_active = False coord._live_priority_check = None coord._interrupt_check = None - coord.stats = {'static_pauses': 0} return coord def _plugin(self): diff --git a/test/test_web_smoke.py b/test/test_web_smoke.py index 6ccac074..a25708d6 100644 --- a/test/test_web_smoke.py +++ b/test/test_web_smoke.py @@ -174,7 +174,7 @@ def test_widget_bundle_is_served_and_requested(client): assert resp.mimetype == "application/javascript" body = resp.get_data(as_text=True) assert body.index("/* registry.js */") < body.index("/* notification.js */") - assert "/* json-file-manager.js */" in body + assert "/* plugin-loader.js */" in body def test_durations_page_groups_by_plugin(client): diff --git a/test/web_interface/integration/test_plugin_operations.py b/test/web_interface/integration/test_plugin_operations.py index dc9d08e5..b3635020 100644 --- a/test/web_interface/integration/test_plugin_operations.py +++ b/test/web_interface/integration/test_plugin_operations.py @@ -66,31 +66,6 @@ class TestPluginOperationsIntegration(unittest.TestCase): history = self.operation_history.get_history(plugin_id=plugin_id) self.assertEqual([r.operation_type for r in history], ["install"]) - def test_update_operation_flow(self): - """Test complete update operation flow.""" - plugin_id = "test-plugin" - - # Enqueue update operation - operation_id = self.operation_queue.enqueue_operation( - OperationType.UPDATE, - plugin_id, - {"from_version": "1.0.0", "to_version": "2.0.0"} - ) - - self.assertIsNotNone(operation_id) - - # Record in history - self.operation_history.record_operation( - operation_type="update", - plugin_id=plugin_id, - status="in_progress", - operation_id=operation_id - ) - - # Verify history - history = self.operation_history.get_history(plugin_id=plugin_id) - self.assertEqual([r.operation_type for r in history], ["update"]) - def test_uninstall_operation_flow(self): """Test complete uninstall operation flow.""" plugin_id = "test-plugin" @@ -162,7 +137,7 @@ class TestPluginOperationsIntegration(unittest.TestCase): # The prevention only works for truly concurrent (pending/running) operations try: op2_id = self.operation_queue.enqueue_operation( - OperationType.UPDATE, + OperationType.UNINSTALL, plugin_id ) # If no exception, the first operation may have completed already diff --git a/test/web_interface/test_api_v3_helpers.py b/test/web_interface/test_api_v3_helpers.py index 068e97e3..9f5afd6b 100644 --- a/test/web_interface/test_api_v3_helpers.py +++ b/test/web_interface/test_api_v3_helpers.py @@ -22,7 +22,6 @@ from web_interface.blueprints.api_v3 import ( # noqa: E402 _is_plugin_update_available, _coerce_to_bool, deep_merge, - _parse_form_value, _get_schema_property, _set_nested_value, _SKIP_FIELD, @@ -116,44 +115,6 @@ class TestDeepMerge: assert result["keep"] is base["keep"] # untouched subtree is shared -class TestParseFormValue: - def test_boolean_strings(self): - assert _parse_form_value("true") is True - assert _parse_form_value("False") is False - - def test_null_like_strings(self): - assert _parse_form_value("null") is None - assert _parse_form_value("none") is None - assert _parse_form_value("") is None - - def test_none_passthrough(self): - assert _parse_form_value(None) is None - - def test_numbers(self): - assert _parse_form_value("42") == 42 - assert isinstance(_parse_form_value("42"), int) - assert _parse_form_value("3.5") == 3.5 - assert isinstance(_parse_form_value("3.5"), float) - - def test_json_array_parsed_before_numbers(self): - # RGB arrays like "[255, 0, 0]" must come back as lists. - assert _parse_form_value("[255, 0, 0]") == [255, 0, 0] - - def test_json_object(self): - assert _parse_form_value('{"a": 1}') == {"a": 1} - - def test_malformed_json_falls_back_to_string(self): - assert _parse_form_value("[not json") == "[not json" - - def test_plain_string_returned_unstripped(self): - # The original value (not the stripped copy) is returned. - assert _parse_form_value(" hello ") == " hello " - - def test_non_string_passthrough(self): - assert _parse_form_value(7) == 7 - assert _parse_form_value([1, 2]) == [1, 2] - - class TestGetSchemaProperty: SCHEMA: ClassVar[Dict[str, Any]] = { "properties": { diff --git a/test/web_interface/test_error_handler.py b/test/web_interface/test_error_handler.py index 74926c31..1c057c81 100644 --- a/test/web_interface/test_error_handler.py +++ b/test/web_interface/test_error_handler.py @@ -1,24 +1,23 @@ """ -Tests for the response builders in src/web_interface/error_handler.py and -the success path in src/web_interface/api_helpers.py. +Tests for the response builders in src/web_interface/api_helpers.py. -describe_exception() in the same module is already covered by +describe_exception() in error_handler.py is already covered by test/test_web_error_detail.py and is not duplicated here. -Regression coverage for one fixed bug: create_success_response used +Regression coverage for one fixed bug: the success builder used truthiness for `message` and `metadata` while using `is not None` for `data`, so an explicitly-passed "" or {} was silently dropped — -api_helpers.success_response() repeated the same gate, which is the path -every api_v3 endpoint actually calls. +success_response() repeated the same gate, which is the path every api_v3 +endpoint actually calls. """ import pytest from flask import Flask -from src.web_interface.api_helpers import exception_error_response, success_response -from src.web_interface.error_handler import ( - create_error_response, - create_success_response, +from src.web_interface.api_helpers import ( + error_response, + exception_error_response, + success_response, ) from src.web_interface.errors import ErrorCode, WebInterfaceError @@ -28,23 +27,23 @@ def app(): return Flask(__name__) -class TestCreateErrorResponse: +class TestErrorResponse: def test_returns_response_and_status_tuple(self, app): with app.test_request_context(): - response, status = create_error_response( + response, status = error_response( ErrorCode.CONFIG_SAVE_FAILED, "could not save") assert status == 500 assert response.get_json()["message"] == "could not save" def test_status_code_passthrough(self, app): with app.test_request_context(): - _, status = create_error_response( + _, status = error_response( ErrorCode.INVALID_INPUT, "bad", status_code=400) assert status == 400 def test_body_matches_the_error_dataclass(self, app): with app.test_request_context(): - response, _ = create_error_response( + response, _ = error_response( ErrorCode.NETWORK_ERROR, "offline", details="connection refused", context={"url": "http://x"}) expected = WebInterfaceError( @@ -54,12 +53,12 @@ class TestCreateErrorResponse: def test_none_context_produces_no_context_key(self, app): with app.test_request_context(): - response, _ = create_error_response(ErrorCode.SYSTEM_ERROR, "boom") + response, _ = error_response(ErrorCode.SYSTEM_ERROR, "boom") assert "context" not in response.get_json() def test_suggested_fixes_passed_through(self, app): with app.test_request_context(): - response, _ = create_error_response( + response, _ = error_response( ErrorCode.SYSTEM_ERROR, "boom", suggested_fixes=["Try again"]) assert response.get_json()["suggested_fixes"] == ["Try again"] @@ -74,7 +73,6 @@ class TestExceptionErrorResponse: @staticmethod def _by_hand(exc, code, with_context): - from src.web_interface.api_helpers import error_response error = WebInterfaceError.from_exception(exc, code) if with_context: return error_response(error.error_code, error.message, @@ -117,39 +115,45 @@ class TestExceptionErrorResponse: assert "context" not in response.get_json() -class TestCreateSuccessResponse: +def _success_body(**kwargs): + """success_response()'s body, outside any request timing.""" + with Flask(__name__).test_request_context(): + return success_response(**kwargs).get_json() + + +class TestSuccessResponseBody: def test_bare_success(self): - assert create_success_response() == {"status": "success"} + assert _success_body() == {"status": "success"} def test_data_included(self): - assert create_success_response(data={"a": 1})["data"] == {"a": 1} + assert _success_body(data={"a": 1})["data"] == {"a": 1} @pytest.mark.parametrize("falsy", [0, "", False, {}, []]) def test_falsy_data_is_still_included(self, falsy): - assert create_success_response(data=falsy)["data"] == falsy + assert _success_body(data=falsy)["data"] == falsy def test_none_data_omitted(self): - assert "data" not in create_success_response(data=None) + assert "data" not in _success_body(data=None) def test_message_included(self): - assert create_success_response(message="done")["message"] == "done" + assert _success_body(message="done")["message"] == "done" def test_empty_message_is_still_included(self): # Regression: `if message:` dropped an explicitly-passed "". - assert create_success_response(message="")["message"] == "" + assert _success_body(message="")["message"] == "" def test_none_message_omitted(self): - assert "message" not in create_success_response(message=None) + assert "message" not in _success_body(message=None) def test_metadata_included(self): - assert create_success_response(metadata={"v": 1})["metadata"] == {"v": 1} + assert _success_body(metadata={"v": 1})["metadata"] == {"v": 1} def test_empty_metadata_is_still_included(self): # Regression: `if metadata:` dropped an explicitly-passed {}. - assert create_success_response(metadata={})["metadata"] == {} + assert _success_body(metadata={})["metadata"] == {} def test_none_metadata_omitted(self): - assert "metadata" not in create_success_response(metadata=None) + assert "metadata" not in _success_body(metadata=None) class TestSuccessResponseHelper: @@ -162,8 +166,8 @@ class TestSuccessResponseHelper: def test_explicit_empty_metadata_survives_the_wrapper(self, app): # Regression: the wrapper re-gated metadata on truthiness after - # create_success_response had already included it, so {} was - # dropped again on the way out. + # the body builder had already included it, so {} was dropped + # again on the way out. with app.test_request_context(): body = success_response(data=None, metadata={}).get_json() assert body["metadata"] == {} diff --git a/test/web_interface/test_plugin_operation_queue.py b/test/web_interface/test_plugin_operation_queue.py index b68d8f34..e3394775 100644 --- a/test/web_interface/test_plugin_operation_queue.py +++ b/test/web_interface/test_plugin_operation_queue.py @@ -55,7 +55,7 @@ class TestPluginOperationQueue(unittest.TestCase): # behavior may differ. For this test, we'll verify the mechanism exists. try: self.queue.enqueue_operation( - OperationType.UPDATE, + OperationType.UNINSTALL, "test-plugin" ) # If no exception, the first operation may have completed @@ -64,21 +64,6 @@ class TestPluginOperationQueue(unittest.TestCase): # Expected behavior - concurrent operation prevented pass - def test_operation_cancellation(self): - """Test cancelling a pending operation.""" - operation_id = self.queue.enqueue_operation( - OperationType.INSTALL, - "test-plugin" - ) - - # Cancel operation - success = self.queue.cancel_operation(operation_id) - self.assertTrue(success) - - # Check status - operation = self.queue.get_operation_status(operation_id) - self.assertEqual(operation.status, OperationStatus.CANCELLED) - def test_operation_history(self): """Test operation history tracking.""" # Enqueue and complete an operation @@ -92,7 +77,7 @@ class TestPluginOperationQueue(unittest.TestCase): time.sleep(0.5) # Check history - history = self.queue.get_operation_history(limit=10) + history = self.queue._operation_history self.assertGreater(len(history), 0) # Find our operation in history diff --git a/test/web_interface/test_state_reconciliation.py b/test/web_interface/test_state_reconciliation.py index fcab4012..452cd0c0 100644 --- a/test/web_interface/test_state_reconciliation.py +++ b/test/web_interface/test_state_reconciliation.py @@ -100,7 +100,6 @@ class TestStateReconciliation(unittest.TestCase): inconsistency = result.inconsistencies_found[0] self.assertEqual(inconsistency.plugin_id, "plugin1") self.assertEqual(inconsistency.inconsistency_type, InconsistencyType.PLUGIN_MISSING_IN_CONFIG) - self.assertTrue(inconsistency.can_auto_fix) self.assertEqual(inconsistency.fix_action, FixAction.AUTO_FIX) def test_plugin_missing_on_disk(self): @@ -115,7 +114,6 @@ class TestStateReconciliation(unittest.TestCase): inconsistency = result.inconsistencies_found[0] self.assertEqual(inconsistency.plugin_id, "plugin1") self.assertEqual(inconsistency.inconsistency_type, InconsistencyType.PLUGIN_MISSING_ON_DISK) - self.assertFalse(inconsistency.can_auto_fix) self.assertEqual(inconsistency.fix_action, FixAction.MANUAL_FIX_REQUIRED) def test_enabled_but_not_loaded_is_reported_not_fixed(self): @@ -139,7 +137,6 @@ class TestStateReconciliation(unittest.TestCase): inconsistency = result.inconsistencies_found[0] self.assertEqual(inconsistency.inconsistency_type, InconsistencyType.PLUGIN_ENABLED_MISMATCH) self.assertEqual(inconsistency.fix_action, FixAction.NO_ACTION) - self.assertFalse(inconsistency.can_auto_fix) self.assertIn("No module named", inconsistency.description) self.assertEqual(result.inconsistencies_fixed, []) self.assertEqual(result.inconsistencies_manual, []) @@ -410,7 +407,6 @@ class TestStateReconciliationUnrecoverable(unittest.TestCase): # Still one inconsistency, still no install attempt, no new registry fetch self.assertEqual(len(result.inconsistencies_found), 1) inc = result.inconsistencies_found[0] - self.assertFalse(inc.can_auto_fix) self.assertEqual(inc.fix_action, FixAction.MANUAL_FIX_REQUIRED) self.store_manager.install_plugin.assert_not_called() self.store_manager.fetch_registry.assert_not_called() @@ -458,7 +454,6 @@ class TestStateReconciliationUnrecoverable(unittest.TestCase): self.assertEqual(len(result.inconsistencies_found), 1) inc = result.inconsistencies_found[0] - self.assertFalse(inc.can_auto_fix) self.assertEqual(inc.fix_action, FixAction.MANUAL_FIX_REQUIRED) self.store_manager.install_plugin.assert_not_called() diff --git a/test/web_interface/test_web_process_runs_no_plugin_code.py b/test/web_interface/test_web_process_runs_no_plugin_code.py index 32ac40a0..08962153 100644 --- a/test/web_interface/test_web_process_runs_no_plugin_code.py +++ b/test/web_interface/test_web_process_runs_no_plugin_code.py @@ -119,7 +119,7 @@ class Web: self.config_manager.template_path = str(tmp_path / "no-template.json") self.schema_manager = SchemaManager(plugins_dir=self.plugins_dir, project_root=tmp_path, config_manager=self.config_manager) - self.catalog = PluginCatalog(self.plugins_dir, self.config_manager, self.schema_manager) + self.catalog = PluginCatalog(self.plugins_dir) api = self.api = api_v3_module.api_v3 api.config_manager = self.config_manager @@ -242,7 +242,8 @@ class TestTheWebProcessNeverRunsAPlugin: _bump_version(web, "1.1.0") body = web.post("/api/v3/plugins/update", {"plugin_id": PLUGIN_ID}) assert body["data"]["update_status"] == "updated" - assert web.catalog.get_installed_version(PLUGIN_ID) == "1.1.0" + # The route rescans the catalog, which now has the new manifest. + assert web.catalog.get_manifest(PLUGIN_ID)["version"] == "1.1.0" assert web.ran() == [] def test_installing_it(self, web): @@ -497,16 +498,15 @@ class TestCatalogReadsWhatIsInstalled: assert expected, f"no plugins under {root}" before = set(sys.modules) schema_manager = SchemaManager(plugins_dir=root, project_root=PROJECT_ROOT) - catalog = PluginCatalog(root, schema_manager=schema_manager) + catalog = PluginCatalog(root) assert set(catalog.discover_plugins()) == set(expected) for plugin_id, (plugin_dir, manifest) in expected.items(): assert catalog.get_manifest(plugin_id) == manifest assert catalog.get_plugin_directory(plugin_id) == str(plugin_dir) - assert catalog.get_installed_version(plugin_id) == manifest.get("version", "") assert catalog.get_plugin_display_modes(plugin_id) == manifest.get("display_modes", []) if (plugin_dir / "config_schema.json").exists(): - schema = catalog.get_schema(plugin_id, use_cache=False) + schema = schema_manager.load_schema(plugin_id, use_cache=False) assert isinstance(schema, dict) and "properties" in schema, plugin_id imported = [name for name in set(sys.modules) - before @@ -537,16 +537,16 @@ class TestCatalogReadsWhatIsInstalled: assert catalog.discover_plugins() == [] assert catalog.get_manifest("ci-fixture-plugin") is None - def test_enabled_follows_the_display_rule(self, tmp_path): + def test_enabled_follows_the_display_rule(self, api_v3_module): config = MagicMock() config.load_config.return_value = {"a": {"enabled": True}, "b": {}, "c": "junk"} - catalog = PluginCatalog(tmp_path, config_manager=config) - assert catalog.is_enabled("a") is True + api_v3_module.api_v3.config_manager = config + enabled = api_v3_module._plugin_enabled_in_config + assert enabled("a") is True # The display runs a plugin only when its section says so. - assert catalog.is_enabled("b") is False - assert catalog.is_enabled("c") is False - assert catalog.is_enabled("missing") is False - assert catalog.get_config("c") == {} + assert enabled("b") is False + assert enabled("c") is False + assert enabled("missing") is False def test_it_has_nothing_that_runs_a_plugin(self, tmp_path): catalog = PluginCatalog(tmp_path) diff --git a/web_interface/app.py b/web_interface/app.py index 73ae0ca2..d2d4e067 100644 --- a/web_interface/app.py +++ b/web_interface/app.py @@ -168,8 +168,6 @@ def _catalog_runtime_view(): plugin_catalog = PluginCatalog( plugins_dir=plugins_dir, - config_manager=config_manager, - schema_manager=schema_manager, runtime_source=_catalog_runtime_view, ) @@ -181,12 +179,10 @@ operation_queue = PluginOperationQueue(max_history=500) # snapshot the display publishes (src/plugin_system/plugin_runtime.py). An # existing file is left where it is, unread; see docs/ARCHITECTURE.md. -# Initialize operation history -# Use lazy_load=True to defer file loading until first use (improves startup time) +# Initialize operation history (its file is read on first use, not at startup) operation_history = OperationHistory( history_file=str(project_root / "data" / "operation_history.json"), - max_records=1000, - lazy_load=True + max_records=1000 ) # Plugin discovery is deferred until first API request that needs it diff --git a/web_interface/blueprints/api_v3/__init__.py b/web_interface/blueprints/api_v3/__init__.py index b40105be..4410c67a 100644 --- a/web_interface/blueprints/api_v3/__init__.py +++ b/web_interface/blueprints/api_v3/__init__.py @@ -46,6 +46,7 @@ from src.web_interface.error_handler import (describe_exception, http_exception_ redact_text, unhandled_exception_payload) from werkzeug.exceptions import HTTPException from src.plugin_system.operation_types import OperationType +from src.web_interface.config_arrays import _schema_type_is from src.web_interface.validators import ( validate_file_upload ) @@ -881,46 +882,6 @@ def deep_merge(base_dict, update_dict): # For non-dict values or new keys, use the update value result[key] = value return result -def _parse_form_value(value): - """ - Parse a form value into the appropriate Python type. - Handles booleans, numbers, JSON arrays/objects, and strings. - """ - if value is None: - return None - - # Handle string values - if isinstance(value, str): - stripped = value.strip() - - # Check for boolean strings - if stripped.lower() == 'true': - return True - if stripped.lower() == 'false': - return False - if stripped.lower() in ('null', 'none') or stripped == '': - return None - - # Try parsing as JSON (for arrays and objects) - do this BEFORE number parsing - # This handles RGB arrays like "[255, 0, 0]" correctly - if stripped.startswith('[') or stripped.startswith('{'): - try: - return json.loads(stripped) - except json.JSONDecodeError: - pass - - # Try parsing as number - try: - if '.' in stripped: - return float(stripped) - return int(stripped) - except ValueError: - pass - - # Return as string (original value, not stripped) - return value - - return value def _get_schema_property(schema, key_path): """ Get the schema property for a given key path (supports dot notation). @@ -1012,22 +973,6 @@ def _is_field_required(key_path, schema): _SKIP_FIELD = object() -def _schema_type_is(prop, wanted): - """Whether a schema property is of ``wanted`` type. - - JSON Schema allows a union (``["array", "null"]``), which the per-element - style system uses for its per-mode override fields: null there means - "inherit the base", so the type genuinely is "an array or nothing". A - bare ``prop.get('type') == 'array'`` reads False for those, which meant - the indexed colour inputs a form posts as ``...text_color.0/.1/.2`` were - never recombined into a list. - """ - if not isinstance(prop, dict): - return False - declared = prop.get('type') - if isinstance(declared, list): - return wanted in declared - return declared == wanted def _schema_allows_null(prop): diff --git a/web_interface/requirements.txt b/web_interface/requirements.txt index ea140dc7..df4cdb2c 100644 --- a/web_interface/requirements.txt +++ b/web_interface/requirements.txt @@ -8,7 +8,6 @@ werkzeug>=3.1.6,<4.0.0 # Flask transitive; pinned to keep a security floor abov flask-limiter>=3.5.0,<4.0.0 # Rate limiting (prevent accidental abuse) flask-compress>=1.14 # gzip/brotli response compression (big win for the large JS/HTML over WiFi) jinja2>=3.1.6,<4.0.0 # Flask transitive, but imported directly (TemplateNotFound); 3.1.6 is the security floor — earlier 3.1.x has sandbox breakouts -markupsafe>=2.1.0,<4.0.0 # Flask transitive, but imported directly (escape) # WebSocket support: intentionally NOT declared here. The web interface # uses Server-Sent Events, and plugins that need Socket.IO (e.g. diff --git a/web_interface/static/v3/app.js b/web_interface/static/v3/app.js index 0c046c34..27207e87 100644 --- a/web_interface/static/v3/app.js +++ b/web_interface/static/v3/app.js @@ -15,7 +15,7 @@ * is the one Alpine uses) * end of , defer, in this order: app.js, js/tooltips.js, * js/settings-search.js, js/utils/dialog.js, - * js/utils/error_handler.js, js/plugins/api_client.js, + * js/plugins/api_client.js, * state_manager.js, install_manager.js, list_filter.js, * the widget bundle (web_interface/widget_bundle.py), * plugins_manager.js @@ -252,8 +252,7 @@ document.addEventListener('keydown', function(e) { // alone because during the x-transition between tabs both are visible. // requestSubmit() runs validation and onsubmit guards like a real submit; // with no visible form, do nothing. Inside a modal dialog the shortcut is - // the dialog's (json-file-manager saves its file on Ctrl+S), so the tab's - // form is left alone. + // the dialog's, so the tab's form is left alone. if ((e.ctrlKey || e.metaKey) && e.key === 's') { e.preventDefault(); const active = document.activeElement; diff --git a/web_interface/static/v3/js/app-early.js b/web_interface/static/v3/js/app-early.js index d41abf4e..6d5e5be9 100644 --- a/web_interface/static/v3/js/app-early.js +++ b/web_interface/static/v3/js/app-early.js @@ -15,7 +15,7 @@ * is the one Alpine uses) * end of , defer, in this order: app.js, js/tooltips.js, * js/settings-search.js, js/utils/dialog.js, - * js/utils/error_handler.js, js/plugins/api_client.js, + * js/plugins/api_client.js, * state_manager.js, install_manager.js, list_filter.js, * the widget bundle (web_interface/widget_bundle.py), * plugins_manager.js diff --git a/web_interface/static/v3/js/app-shell.js b/web_interface/static/v3/js/app-shell.js index 43e25ebf..db486d7c 100644 --- a/web_interface/static/v3/js/app-shell.js +++ b/web_interface/static/v3/js/app-shell.js @@ -17,7 +17,7 @@ * is the one Alpine uses) * end of , defer, in this order: app.js, js/tooltips.js, * js/settings-search.js, js/utils/dialog.js, - * js/utils/error_handler.js, js/plugins/api_client.js, + * js/plugins/api_client.js, * state_manager.js, install_manager.js, list_filter.js, * the widget bundle (web_interface/widget_bundle.py), * plugins_manager.js diff --git a/web_interface/static/v3/js/plugins/api_client.js b/web_interface/static/v3/js/plugins/api_client.js index 1f9cad65..562d4691 100644 --- a/web_interface/static/v3/js/plugins/api_client.js +++ b/web_interface/static/v3/js/plugins/api_client.js @@ -219,55 +219,6 @@ const PluginAPI = { return []; }, - /** - * Toggle plugin enabled/disabled. - * - * @param {string} pluginId - Plugin identifier - * @param {boolean} enabled - Whether plugin should be enabled - * @returns {Promise} Response data - */ - async togglePlugin(pluginId, enabled) { - return await this.request('/plugins/toggle', 'POST', { - plugin_id: pluginId, - enabled: enabled - }); - }, - - /** - * Get plugin configuration. - * - * @param {string} pluginId - Plugin identifier - * @returns {Promise} Plugin configuration - */ - async getPluginConfig(pluginId) { - const response = await this.request(`/plugins/config?plugin_id=${encodeURIComponent(pluginId)}`); - return response.data || {}; - }, - - /** - * Save plugin configuration. - * - * @param {string} pluginId - Plugin identifier - * @param {Object} config - Configuration data - * @returns {Promise} Response data - */ - async savePluginConfig(pluginId, config) { - return await this.request('/plugins/config', 'POST', { - plugin_id: pluginId, - config: config - }); - }, - - /** - * Reset plugin configuration to defaults. - * - * @param {string} pluginId - Plugin identifier - * @returns {Promise} Response data - */ - async resetPluginConfig(pluginId) { - return await this.request(`/plugins/config/reset?plugin_id=${encodeURIComponent(pluginId)}`, 'POST'); - }, - /** * Update plugin. * diff --git a/web_interface/static/v3/js/plugins/state_manager.js b/web_interface/static/v3/js/plugins/state_manager.js index 30b9282b..d7b3eef8 100644 --- a/web_interface/static/v3/js/plugins/state_manager.js +++ b/web_interface/static/v3/js/plugins/state_manager.js @@ -22,8 +22,9 @@ const PluginStateManager = { window.installedPlugins = plugins; // For backward compatibility return plugins; } catch (error) { - if (window.errorHandler) { - window.errorHandler.displayError(error, 'Failed to load installed plugins'); + if (typeof window.showNotification === 'function') { + window.showNotification('Failed to load installed plugins: ' + + ((error && error.message) || String(error)), 'error'); } throw error; } diff --git a/web_interface/static/v3/js/utils/error_handler.js b/web_interface/static/v3/js/utils/error_handler.js deleted file mode 100644 index f0fd0700..00000000 --- a/web_interface/static/v3/js/utils/error_handler.js +++ /dev/null @@ -1,350 +0,0 @@ -/* global showNotification */ -/** - * Frontend error handling utilities. - * - * Provides user-friendly error formatting and display with enhanced UI. - */ - -/** - * Comprehensive error code to user-friendly message mapping. - * Used only when the server response carries no message of its own. - */ -const ERROR_MESSAGES = { - // Configuration errors - 'CONFIG_SAVE_FAILED': "Couldn't save your settings. Check the values and try again", - 'CONFIG_LOAD_FAILED': "Couldn't load your settings. Reload the page to try again", - 'CONFIG_VALIDATION_FAILED': 'Some settings have invalid values. Fix the highlighted fields and save again', - 'CONFIG_ROLLBACK_FAILED': "Couldn't restore the previous settings. Try again, or restore from Backup & Restore", - - // Plugin errors - 'PLUGIN_NOT_FOUND': "That plugin isn't installed. Check the Plugin Manager", - 'PLUGIN_INSTALL_FAILED': "Couldn't install the plugin. Check the Pi's internet connection and try again", - 'PLUGIN_UPDATE_FAILED': "Couldn't update the plugin. Try again in a moment", - 'PLUGIN_UNINSTALL_FAILED': "Couldn't uninstall the plugin. Try again in a moment", - 'PLUGIN_LOAD_FAILED': "The plugin couldn't start. Check the Logs tab for details", - 'PLUGIN_OPERATION_CONFLICT': 'Another plugin operation is still running. Wait for it to finish, then try again', - - // Validation errors - 'VALIDATION_ERROR': 'Some values are invalid. Fix them and try again', - 'SCHEMA_VALIDATION_FAILED': "Some settings don't match what the plugin expects. Fix them and save again", - 'INVALID_INPUT': "That value isn't valid. Check it and try again", - - // Network errors - 'NETWORK_ERROR': "Couldn't reach the LEDMatrix device. Check that it's on and connected, then try again", - 'API_ERROR': 'The request failed. Try again in a moment', - 'TIMEOUT': 'This took too long to respond. Try again in a moment', - - // Permission errors - 'PERMISSION_DENIED': "LEDMatrix doesn't have permission to do that. See the troubleshooting guide", - 'FILE_PERMISSION_ERROR': "LEDMatrix can't write a file it needs. See the troubleshooting guide", - - // System errors - 'SYSTEM_ERROR': 'Something went wrong on the device. Check the Logs tab for details', - 'SERVICE_UNAVAILABLE': 'The LEDMatrix service is not running. Restart it from the Overview tab', - - // Unknown errors - 'UNKNOWN_ERROR': 'Something went wrong. Try again, and check the Logs tab if it keeps happening' -}; - -/** - * Error code to troubleshooting documentation links. - */ -const TROUBLESHOOTING_URL = 'https://github.com/ChuckBuilds/LEDMatrix/blob/main/docs/TROUBLESHOOTING.md'; -const ERROR_DOCS = { - 'CONFIG_SAVE_FAILED': TROUBLESHOOTING_URL + '#4-check-configuration', - 'CONFIG_VALIDATION_FAILED': TROUBLESHOOTING_URL + '#4-check-configuration', - 'PLUGIN_INSTALL_FAILED': TROUBLESHOOTING_URL + '#plugin-issues', - 'PLUGIN_OPERATION_CONFLICT': TROUBLESHOOTING_URL + '#plugin-issues', - 'PERMISSION_DENIED': TROUBLESHOOTING_URL + '#permission-issues', - 'FILE_PERMISSION_ERROR': TROUBLESHOOTING_URL + '#permission-issues' -}; - -/** - * Format error message for display to user. - * - * @param {Object} error - Error object from API response - * @returns {string} Formatted error message - */ -function formatError(error) { - if (!error) { - return ERROR_MESSAGES.UNKNOWN_ERROR; - } - - // If error is a string, return it - if (typeof error === 'string') { - return error; - } - - // If error has a message, use it - if (error.message) { - return error.message; - } - - // If error has error_code, format it - if (error.error_code) { - const message = ERROR_MESSAGES[error.error_code] || error.message || ERROR_MESSAGES.UNKNOWN_ERROR; - - // Add details if available - if (error.details) { - return `${message}: ${error.details}`; - } - - return message; - } - - return ERROR_MESSAGES.UNKNOWN_ERROR; -} - -/** - * Get suggested fixes for an error. - * - * @param {Object} error - Error object from API response - * @returns {Array} Array of suggested fixes - */ -function getSuggestedFixes(error) { - if (!error || !error.suggested_fixes) { - return []; - } - - return error.suggested_fixes; -} - -/** - * Display error with suggestions in a rich UI. - * - * @param {Object} error - Error object from API response - * @param {string} context - Optional context about what was being done - * @param {Object} options - Display options - * @param {boolean} options.showDetails - Whether to show detailed error modal - * @param {boolean} options.showCopyButton - Whether to show copy button - */ -function displayError(error, context = null, options = {}) { - const message = formatError(error); - const suggestions = getSuggestedFixes(error); - const errorCode = error?.error_code; - const docLink = errorCode ? ERROR_DOCS[errorCode] : null; - - // Build full message - let fullMessage = message; - if (context) { - fullMessage = `${context}: ${message}`; - } - - if (suggestions.length > 0) { - fullMessage += '\n\nSuggested fixes:\n' + suggestions.map(s => `• ${s}`).join('\n'); - } - - // If showDetails is true, show a rich error modal - if (options.showDetails !== false && (suggestions.length > 0 || docLink || error?.details)) { - showErrorModal(error, context, message, suggestions, docLink); - } else { - // Simple notification - showNotification(fullMessage, 'error'); - } -} - -// Release function for the open error modal's focus trap (see utils/dialog.js). -let errorModalRelease = null; - -/** - * Show a rich error modal with details, suggestions, and copy button. - * - * @param {Object} error - Error object - * @param {string} context - Context - * @param {string} message - Formatted message - * @param {Array} suggestions - Suggested fixes - * @param {string} docLink - Documentation link - */ -function showErrorModal(error, context, message, suggestions, docLink) { - error = error || {}; - suggestions = suggestions || []; - - // Create modal container if it doesn't exist - let modalContainer = document.getElementById('error-modal-container'); - if (!modalContainer) { - modalContainer = document.createElement('div'); - modalContainer.id = 'error-modal-container'; - modalContainer.className = 'fixed inset-0 z-50 overflow-y-auto'; - modalContainer.style.display = 'none'; - document.body.appendChild(modalContainer); - } - - // Re-opening while open: release the previous trap before replacing it. - if (errorModalRelease) { - errorModalRelease(); - errorModalRelease = null; - } - - // Build modal content - const contextText = context ? `
${window.LEDEscape.html(context)}
` : ''; - const suggestionsHtml = suggestions.length > 0 ? ` -
-

Suggested fixes:

-
    - ${suggestions.map(s => `
  • ${window.LEDEscape.html(s)}
    • `).join('')} -
-
- ` : ''; - - const docLinkHtml = docLink ? ` -
- - View troubleshooting guide (opens in a new tab) - -
- ` : ''; - - const detailsHtml = error.details ? ` -
-
- Technical details - 
${window.LEDEscape.html(error.details)}
-
-
- ` : ''; - - const errorCodeHtml = error.error_code ? ` -
- Error code: ${window.LEDEscape.html(error.error_code)} -
- ` : ''; - - modalContainer.innerHTML = ` -
- - -
-
-
- -
-

Something went wrong

-
- ${contextText} -

${window.LEDEscape.html(message)}

- ${errorCodeHtml} - ${suggestionsHtml} - ${docLinkHtml} - ${detailsHtml} -
-
-
-
-
- - -
-
-
- `; - - // Attach event listener to copy button - const copyBtn = modalContainer.querySelector('#error-modal-copy-btn'); - if (copyBtn) { - copyBtn.addEventListener('click', () => { - copyErrorDetails(error); - }); - } - - modalContainer.style.display = 'block'; - - const panel = modalContainer.querySelector('#error-modal-panel'); - if (panel) { - panel.setAttribute('aria-describedby', 'error-modal-description'); - if (window.LEDDialog) { - errorModalRelease = window.LEDDialog.trap(panel, { - labelledBy: 'error-modal-title', - initialFocus: '#error-modal-close-btn', - onEscape: closeErrorModal - }); - } - } -} - -/** - * Close the error modal. - */ -function closeErrorModal() { - const modalContainer = document.getElementById('error-modal-container'); - if (modalContainer) { - modalContainer.style.display = 'none'; - } - if (errorModalRelease) { - const release = errorModalRelease; - errorModalRelease = null; - release(); - } -} - -/** - * Copy error details to clipboard. - * - * @param {Object} error - Error object from API response - */ -function copyErrorDetails(error) { - const errorText = JSON.stringify(error, null, 2); - function copyFailed(err) { - console.error('Failed to copy error details:', err); - showNotification("Couldn't copy to the clipboard. Open Technical details and copy the text by hand.", 'warning'); - } - - if (navigator.clipboard && navigator.clipboard.writeText) { - navigator.clipboard.writeText(errorText).then(() => { - showNotification('Error details copied to clipboard', 'success'); - }).catch(copyFailed); - } else { - // Fallback for older browsers - const textArea = document.createElement('textarea'); - textArea.value = errorText; - textArea.style.position = 'fixed'; - textArea.style.opacity = '0'; - // Keep focus inside the open dialog while copying. - const panel = document.getElementById('error-modal-panel'); - const host = panel && panel.offsetParent !== null ? panel : document.body; - const returnFocus = document.activeElement; - host.appendChild(textArea); - textArea.select(); - try { - document.execCommand('copy'); - showNotification('Error details copied to clipboard', 'success'); - } catch (err) { - copyFailed(err); - } - host.removeChild(textArea); - if (returnFocus && typeof returnFocus.focus === 'function') returnFocus.focus(); - } -} - -// Export functions -if (typeof module !== 'undefined' && module.exports) { - module.exports = { - formatError, - getSuggestedFixes, - displayError, - copyErrorDetails, - showErrorModal, - closeErrorModal, - ERROR_MESSAGES, - ERROR_DOCS - }; -} else { - // Make available globally - window.errorHandler = { - formatError, - getSuggestedFixes, - displayError, - copyErrorDetails, - showErrorModal, - closeErrorModal, - ERROR_MESSAGES, - ERROR_DOCS - }; -} diff --git a/web_interface/static/v3/js/widgets/README.md b/web_interface/static/v3/js/widgets/README.md index 649b3bca..cdf2e348 100644 --- a/web_interface/static/v3/js/widgets/README.md +++ b/web_interface/static/v3/js/widgets/README.md @@ -28,7 +28,7 @@ set by `BUNDLE_ORDER` there. | `file-upload-single` | string | One image upload; stores the uploaded file's relative path | | `google-oauth` | string | Step 2 of the calendar plugin's Google sign-in | | `plugin-file-manager` | null | Inline file manager driven by the plugin's `web_ui_actions` | -| `json-file-manager` | null | JSON data-file manager driven by `web_ui_actions` | +| `json-file-manager` | null | Embeds the plugin's own `web_ui/file_manager.html` in an iframe | | `toggle-switch` | boolean | On/off switch | | `slider` | integer / number | Range slider using `minimum` / `maximum` | | `number-input` | integer / number | Number field with min/max check | @@ -51,7 +51,6 @@ Other files here: | `notification.js` | Toast notifications; owns `window.showNotification` | | `plugin-order-list.js` | Drag-and-drop plugin order list used by the Display and Durations tabs (`window.PluginOrderList`) | | `plugin-loader.js` | Loads a plugin-supplied widget on demand | -| `example-color-picker.js` | Example custom widget. Not bundled: it registers `color-picker` and would replace the real one | Each widget file's header comment gives its schema options. The sections below cover the ones that need more than a line. @@ -270,8 +269,6 @@ directory is the only place the core serves plugin widgets from. })(); ``` -[`example-color-picker.js`](example-color-picker.js) is a longer example. - ### 2. Reference it in the schema ```json diff --git a/web_interface/static/v3/js/widgets/color-picker.js b/web_interface/static/v3/js/widgets/color-picker.js index 2ebf6b6c..74031f50 100644 --- a/web_interface/static/v3/js/widgets/color-picker.js +++ b/web_interface/static/v3/js/widgets/color-picker.js @@ -25,8 +25,6 @@ const base = window.BaseWidget ? new window.BaseWidget('ColorPicker', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -141,7 +139,7 @@ html += ''; // Hidden input for form submission - html += ``; + html += ``; // Preset colors - only render valid hex colors if (Array.isArray(presets) && presets.length > 0) { @@ -155,12 +153,12 @@ html += ` `; } diff --git a/web_interface/static/v3/js/widgets/date-picker.js b/web_interface/static/v3/js/widgets/date-picker.js index d9bf36bc..875e78ee 100644 --- a/web_interface/static/v3/js/widgets/date-picker.js +++ b/web_interface/static/v3/js/widgets/date-picker.js @@ -26,8 +26,6 @@ const base = window.BaseWidget ? new window.BaseWidget('DatePicker', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -70,11 +68,11 @@
${escapeHtml(constraintText)}
`; + html += `
${window.LEDEscape.html(constraintText)}
`; } // Error message area diff --git a/web_interface/static/v3/js/widgets/day-selector.js b/web_interface/static/v3/js/widgets/day-selector.js index 5d2cc59d..02c16ca4 100644 --- a/web_interface/static/v3/js/widgets/day-selector.js +++ b/web_interface/static/v3/js/widgets/day-selector.js @@ -50,8 +50,6 @@ // Use BaseWidget utilities if available const base = window.BaseWidget ? new window.BaseWidget('DaySelector', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -100,7 +98,7 @@ // Hidden input to store the value as JSON array // Note: Using single quotes for attribute, JSON uses double quotes, so no escaping needed - html += ``; + html += ``; // Select All toggle if (showSelectAll) { @@ -141,7 +139,7 @@ ${isChecked ? 'checked' : ''} onchange="window.LEDMatrixWidgets.getHandlers('day-selector').onChange('${fieldId}')" class="day-checkbox h-4 w-4 text-blue-600 focus:ring-blue-500 border-gray-300 rounded"> - ${escapeHtml(label)} + ${window.LEDEscape.html(label)} `; } diff --git a/web_interface/static/v3/js/widgets/email-input.js b/web_interface/static/v3/js/widgets/email-input.js index c53aaff1..8014cce1 100644 --- a/web_interface/static/v3/js/widgets/email-input.js +++ b/web_interface/static/v3/js/widgets/email-input.js @@ -24,8 +24,6 @@ const base = window.BaseWidget ? new window.BaseWidget('EmailInput', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -73,9 +71,9 @@ html += ` . - const escapeHtml = (text) => { - const div = document.createElement('div'); - div.textContent = String(text ?? ''); - return div.innerHTML - .replace(/"/g, '"') - .replace(/'/g, '''); - }; - - // Use validated/sanitized hex for style attribute and input values - const safeHex = currentValue; // Already validated above - - container.innerHTML = ` -
-
- - -
-
- - -
-
-
-
-
-
-

Select a color using the color picker or enter a hex code

- `; - - // Get references to elements - const colorInput = container.querySelector('input[type="color"]'); - const hexInput = container.querySelector('input[type="text"]'); - const preview = container.querySelector(`#${fieldId}_preview`); - - // Update hex when color picker changes - colorInput.addEventListener('input', (e) => { - const color = e.target.value; - hexInput.value = color; - if (preview) { - preview.style.backgroundColor = color; - } - this.handlers.onChange(fieldId, color); - }); - - // Update color picker and preview when hex input changes - hexInput.addEventListener('input', (e) => { - const hex = e.target.value; - // Validate hex format - if (/^#[0-9A-Fa-f]{6}$/.test(hex)) { - colorInput.value = hex; - if (preview) { - preview.style.backgroundColor = hex; - } - hexInput.classList.remove('border-red-500'); - hexInput.classList.add('border-gray-300'); - this.handlers.onChange(fieldId, hex); - } else if (hex.length > 0) { - // Show error state for invalid hex - hexInput.classList.remove('border-gray-300'); - hexInput.classList.add('border-red-500'); - } - }); - - // Validate on blur - hexInput.addEventListener('blur', (e) => { - const hex = e.target.value; - if (hex && !/^#[0-9A-Fa-f]{6}$/.test(hex)) { - // Reset to current color picker value - e.target.value = colorInput.value; - e.target.classList.remove('border-red-500'); - e.target.classList.add('border-gray-300'); - } - }); - }, - - /** - * Get current value from widget - * @param {string} fieldId - Field ID - * @returns {string} Current hex color value - */ - getValue: function(fieldId) { - const colorInput = document.querySelector(`#${fieldId}_color`); - return colorInput ? colorInput.value : null; - }, - - /** - * Set value programmatically - * @param {string} fieldId - Field ID - * @param {string} value - Hex color value to set - */ - setValue: function(fieldId, value) { - // Validate hex color format before using - const hexColorRegex = /^#[0-9A-Fa-f]{6}$/; - const safeValue = hexColorRegex.test(value) ? value : '#000000'; - - const colorInput = document.querySelector(`#${fieldId}_color`); - const hexInput = document.querySelector(`#${fieldId}_hex`); - const preview = document.querySelector(`#${fieldId}_preview`); - - if (colorInput && hexInput) { - colorInput.value = safeValue; - hexInput.value = safeValue; - if (preview) { - preview.style.backgroundColor = safeValue; - } - } - }, - - /** - * Event handlers - */ - handlers: { - /** - * Handle color change - * @param {string} fieldId - Field ID - * @param {string} value - New color value - */ - onChange: function(fieldId, value) { - // Trigger form change event for validation and saving - const event = new CustomEvent('widget-change', { - detail: { fieldId, value }, - bubbles: true, - cancelable: true - }); - document.dispatchEvent(event); - - // Also update any hidden input if it exists - const hiddenInput = document.querySelector(`input[name*="${fieldId}"][type="hidden"]`); - if (hiddenInput) { - hiddenInput.value = value; - } - } - } - }); -})(); diff --git a/web_interface/static/v3/js/widgets/file-upload-single.js b/web_interface/static/v3/js/widgets/file-upload-single.js index 1996ed1f..f9c0ee60 100644 --- a/web_interface/static/v3/js/widgets/file-upload-single.js +++ b/web_interface/static/v3/js/widgets/file-upload-single.js @@ -33,8 +33,6 @@ const base = window.BaseWidget ? new window.BaseWidget('FileUploadSingle', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -78,22 +76,22 @@ const currentValue = value || ''; const hasImage = isImagePath(currentValue); - let html = `
`; + let html = `
`; // Hidden input carries the actual string value - html += ``; + html += ``; // Preview area (shown when a value is set) html += `
`; - html += `Preview`; html += ``; html += `
-

${escapeHtml(currentValue.split('/').pop() || '')}

-

${escapeHtml(currentValue)}

+

${window.LEDEscape.html(currentValue.split('/').pop() || '')}

+

${window.LEDEscape.html(currentValue)}

`; html += `` : ''} - -
-
- -
-
Loading…
-
- - ${hasUpload ? ` -
- -
- 📁 -

Drop a JSON file here, or click to browse

- ${this.uploadHint ? `

${this._esc(this.uploadHint)}

` : ''} -
-
` : ''} - - - - - - ${hasDelete ? ` - ` : ''} - - - ${hasCreate ? ` - ` : ''} - -
`; // end #${u} - - // Cache frequently-used elements - this._root = document.getElementById(u); - this._listEl = document.getElementById(`${u}-list`); - this._editorEl = document.getElementById(`${u}-editor`); - this._editModal = document.getElementById(`${u}-edit-modal`); - this._delModal = document.getElementById(`${u}-del-modal`); - this._createModal = document.getElementById(`${u}-create-modal`); - this._dropzone = document.getElementById(`${u}-dropzone`); - this._fileInput = document.getElementById(`${u}-fileinput`); - } - - _css(u) { - // Colors come from the app theme tokens (app.css :root / [data-theme="dark"]) - // with the original light values as fallbacks for pages without app.css. - // Shades the tokens don't cover are local variables with a dark override. - return ``; - } - - // ── Event Binding ──────────────────────────────────────────────────── - - _bind() { - // Delegated clicks on the widget root - this._root.addEventListener('click', this._onClick.bind(this)); - this._root.addEventListener('change', this._onChange.bind(this)); - - // Drag-and-drop on the dropzone - if (this._dropzone) { - this._dropzone.addEventListener('dragover', e => { - e.preventDefault(); - this._dropzone.classList.add('jfm-over'); - }); - this._dropzone.addEventListener('dragleave', () => { - this._dropzone.classList.remove('jfm-over'); - }); - this._dropzone.addEventListener('drop', e => { - e.preventDefault(); - this._dropzone.classList.remove('jfm-over'); - const file = e.dataTransfer?.files[0]; - if (file) this._uploadFile(file); - }); - // Keyboard activation of drop zone - this._dropzone.addEventListener('keydown', e => { - if (e.key === 'Enter' || e.key === ' ') { - e.preventDefault(); - this._fileInput?.click(); - } - }); - } - - // Modal backdrop clicks - [this._editModal, this._delModal, this._createModal].forEach(m => { - if (m) m.addEventListener('click', e => { if (e.target === m) this._closeAll(); }); - }); - - // Editor: char count + Tab indent - if (this._editorEl) { - this._editorEl.addEventListener('input', () => this._updateStat()); - this._editorEl.addEventListener('keydown', e => { - // Plain Tab indents; Shift+Tab is left alone so keyboard - // users can always move focus out of the editor. - if (e.key === 'Tab' && !e.shiftKey && !e.ctrlKey && !e.altKey && !e.metaKey) { - e.preventDefault(); - const s = this._editorEl.selectionStart; - const end = this._editorEl.selectionEnd; - const v = this._editorEl.value; - this._editorEl.value = v.slice(0, s) + ' ' + v.slice(end); - this._editorEl.selectionStart = this._editorEl.selectionEnd = s + 2; - this._updateStat(); - } - }); - } - - // Global keyboard shortcuts - document.addEventListener('keydown', this._keyHandler); - } - - _onKey(e) { - // Escape is handled by the dialog focus trap (see _trap). - const editOpen = this._editModal && !this._editModal.hidden; - if ((e.ctrlKey || e.metaKey) && e.key === 's' && editOpen) { - e.preventDefault(); - this._doSave(); - } - } - - _onClick(e) { - const btn = e.target.closest('[data-jfm]'); - if (!btn) return; - const action = btn.dataset.jfm; - - switch (action) { - case 'refresh': this._loadList(); break; - case 'open-picker': this._fileInput?.click(); break; - case 'open-create': this._openCreate(); break; - case 'close-edit': this._closeEdit(); break; - case 'close-del': this._closeDel(); break; - case 'close-create': this._closeCreate(); break; - case 'fmt': this._formatJson(); break; - case 'validate': this._validateJson(); break; - case 'save': this._doSave(); break; - case 'confirm-del': this._doDelete(); break; - case 'do-create': this._doCreate(); break; - case 'edit-file': { - const card = btn.closest('[data-jfm-file]'); - if (card) this._openEdit(card.dataset.jfmFile); - break; - } - case 'del-file': { - const card = btn.closest('[data-jfm-file]'); - if (card) this._openDel(card.dataset.jfmFile); - break; - } - } - } - - _onChange(e) { - // Toggle checkbox - if (e.target.classList.contains('jfm-toggle-cb')) { - const catName = e.target.dataset.cat; - const enabled = e.target.checked; - this._doToggle(catName, enabled, e.target); - } - // File input - if (e.target === this._fileInput) { - const file = e.target.files?.[0]; - if (file) this._uploadFile(file); - e.target.value = ''; - } - } - - // ── API helper ─────────────────────────────────────────────────────── - - async _api(actionKey, params) { - const actionId = Object.prototype.hasOwnProperty.call(this.actions, actionKey) ? this.actions[actionKey] : undefined; - if (!actionId) throw new Error(`Action "${actionKey}" not configured`); - const body = { plugin_id: this.pluginId, action_id: actionId }; - if (params !== undefined) body.params = params; - const r = await fetch('/api/v3/plugins/action', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(body) - }); - const ct = r.headers.get('content-type') || ''; - if (!r.ok) { - // A failing action script comes back as a 400 whose JSON body - // carries the script's own message; surface it, not the status. - const data = ct.includes('application/json') ? await r.json().catch(() => null) : null; - throw new Error(data?.message || 'Server error ' + r.status); - } - if (!ct.includes('application/json')) { - const txt = await r.text(); - throw new Error('Unexpected response: ' + txt.slice(0, 120)); - } - return r.json(); - } - - // ── File List ──────────────────────────────────────────────────────── - - async _loadList() { - this._listEl.innerHTML = `
Loading…
`; - try { - const data = await this._api('list'); - if (data.status !== 'success') throw new Error(data.message || 'Load failed'); - this._renderList(data.files || []); - } catch (err) { - this._listEl.innerHTML = ` -
-
⚠
-

Failed to load files

-

${this._esc(err.message)}

-
`; - } - } - - _renderList(files) { - if (!files.length) { - this._listEl.innerHTML = ` -
-
📁
-

No files yet

-

Upload or create a JSON file to get started

-
`; - return; - } - this._listEl.innerHTML = files.map(f => this._card(f)).join(''); - } - - _card(f) { - const enabled = f.enabled !== false; - const displayName = this._esc(f.display_name || f.filename); - const filename = this._esc(f.filename); - const catName = this.toggleKey ? this._esc(f[this.toggleKey] || '') : ''; - const showToggle = !!(this.actions.toggle && this.toggleKey && f[this.toggleKey]); - const hasEdit = !!this.actions.get && !!this.actions.save; - const hasDelete = !!this.actions.delete; - - return ` -
-
- ${displayName} - ${showToggle ? ` - ` : ''} -
-
- 📄 ${filename} - 📊 ${f.entry_count ?? 0} entries · ${this._fmtSize(f.size || 0)} - 🕑 ${this._fmtDate(f.modified)} -
-
- ${hasEdit ? `` : ''} - ${hasDelete ? `` : ''} -
-
`; - } - - // ── Edit flow ──────────────────────────────────────────────────────── - - async _openEdit(filename) { - this._editFile = filename; - document.getElementById(`${this._uid}-edit-title`).textContent = `Edit: ${filename}`; - this._clearErr(); - this._editorEl.value = 'Loading…'; - this._updateStat(); - this._editModal.hidden = false; - this._trap('edit', this._editModal, `${this._uid}-edit-title`, this._editorEl, () => this._closeEdit()); - - try { - const data = await this._api('get', { filename }); - if (data.status !== 'success') throw new Error(data.message || 'Load failed'); - this._editorEl.value = JSON.stringify(data.content, null, 2); - this._updateStat(); - this._editorEl.focus(); - this._editorEl.setSelectionRange(0, 0); - this._editorEl.scrollTop = 0; - } catch (err) { - this._showErr(`Couldn't load this file (${err.message}). Close the editor and try again.`); - this._editorEl.value = ''; - } - } - - _closeEdit() { - if (this._editModal) this._editModal.hidden = true; - this._editFile = null; - this._clearErr(); - this._release('edit'); - } - - _formatJson() { - try { - const parsed = JSON.parse(this._editorEl.value); - this._editorEl.value = JSON.stringify(parsed, null, 2); - this._updateStat(); - this._clearErr(); - } catch (err) { - this._showErr('Invalid JSON — ' + err.message); - } - } - - _validateJson() { - try { - const parsed = JSON.parse(this._editorEl.value); - const n = (typeof parsed === 'object' && parsed !== null) ? Object.keys(parsed).length : '?'; - this._clearErr(); - this._notify(`Valid JSON — ${n} top-level keys`, 'success'); - } catch (err) { - this._showErr('Invalid JSON — ' + err.message); - } - } - - async _doSave() { - if (!this._editFile) return; - let contentStr; - try { - const parsed = JSON.parse(this._editorEl.value); - contentStr = JSON.stringify(parsed, null, 2); - } catch (err) { - this._showErr('Cannot save — fix JSON first: ' + err.message); - return; - } - const btn = document.getElementById(`${this._uid}-save-btn`); - this._busy(btn, 'Saving…'); - try { - const data = await this._api('save', { filename: this._editFile, content: contentStr }); - if (data.status !== 'success') throw new Error(data.message || 'Save failed'); - this._notify('File saved', 'success'); - this._closeEdit(); - this._loadList(); - } catch (err) { - this._showErr('Save failed: ' + err.message); - } finally { - this._idle(btn, 'Save'); - } - } - - // ── Delete flow ────────────────────────────────────────────────────── - - _openDel(filename) { - this._deleteFile = filename; - const el = document.getElementById(`${this._uid}-del-name`); - if (el) el.textContent = filename; - if (this._delModal) { - this._delModal.hidden = false; - // Destructive dialog: start on Cancel. - this._trap('del', this._delModal, `${this._uid}-del-title`, - document.getElementById(`${this._uid}-del-cancel`), () => this._closeDel()); - } - } - - _closeDel() { - if (this._delModal) this._delModal.hidden = true; - this._deleteFile = null; - this._release('del'); - } - - async _doDelete() { - if (!this._deleteFile) return; - const btn = document.getElementById(`${this._uid}-del-btn`); - this._busy(btn, 'Deleting…'); - try { - const data = await this._api('delete', { filename: this._deleteFile }); - if (data.status !== 'success') throw new Error(data.message || 'Delete failed'); - this._notify('File deleted', 'success'); - this._closeDel(); - this._loadList(); - } catch (err) { - this._notify('Delete failed: ' + err.message, 'error'); - } finally { - this._idle(btn, 'Delete'); - } - } - - // ── Create flow ────────────────────────────────────────────────────── - - _openCreate() { - if (!this._createModal) return; - this.createFields.forEach(f => { - const el = document.getElementById(`${this._uid}-cf-${f.key}`); - if (el) el.value = ''; - }); - this._createModal.hidden = false; - const first = this.createFields[0]; - const firstInput = first ? document.getElementById(`${this._uid}-cf-${first.key}`) : null; - this._trap('create', this._createModal, `${this._uid}-create-title`, firstInput, () => this._closeCreate()); - } - - _closeCreate() { - if (this._createModal) this._createModal.hidden = true; - this._release('create'); - } - - async _doCreate() { - const params = {}; - for (const f of this.createFields) { - const el = document.getElementById(`${this._uid}-cf-${f.key}`); - const val = (el?.value || '').trim(); - // display_name may be blank — auto-derived from category_name below - if (!val && f.key !== 'display_name') { - this._notify(`"${f.label}" is required`, 'error'); - el?.focus(); - return; - } - if (f.pattern && val && el && el.validity.patternMismatch) { - this._notify(`"${f.label}" format is invalid`, 'error'); - el?.focus(); - return; - } - if (val) params[f.key] = val; - } - // Auto-derive display_name from category_name when left blank - if (!params.display_name && params.category_name) { - params.display_name = params.category_name.replace(/_/g, ' ').replace(/\b\w/g, c => c.toUpperCase()); - } - const btn = document.getElementById(`${this._uid}-create-btn`); - this._busy(btn, 'Creating…'); - try { - const data = await this._api('create', params); - if (data.status !== 'success') throw new Error(data.message || 'Create failed'); - this._notify('File created', 'success'); - this._closeCreate(); - this._loadList(); - } catch (err) { - this._notify('Create failed: ' + err.message, 'error'); - } finally { - this._idle(btn, 'Create'); - } - } - - // ── Upload ─────────────────────────────────────────────────────────── - - async _uploadFile(file) { - if (!file.name.endsWith('.json')) { - this._notify('Please select a .json file', 'error'); - return; - } - let content; - try { - content = await file.text(); - JSON.parse(content); // client-side validation - } catch (err) { - this._notify('Invalid JSON: ' + err.message, 'error'); - return; - } - if (this._dropzone) this._dropzone.style.opacity = '.5'; - try { - const data = await this._api('upload', { filename: file.name, content }); - if (data.status !== 'success') throw new Error(data.message || 'Upload failed'); - this._notify(`"${file.name}" uploaded`, 'success'); - this._loadList(); - } catch (err) { - this._notify('Upload failed: ' + err.message, 'error'); - } finally { - if (this._dropzone) this._dropzone.style.opacity = ''; - } - } - - // ── Toggle ─────────────────────────────────────────────────────────── - - async _doToggle(catName, enabled, checkbox) { - checkbox.disabled = true; - try { - const params = { enabled }; - if (this.toggleKey) params[this.toggleKey] = catName; - const data = await this._api('toggle', params); - if (data.status !== 'success') throw new Error(data.message || 'Toggle failed'); - this._notify(enabled ? 'Category enabled' : 'Category disabled', 'success'); - this._loadList(); - } catch (err) { - this._notify('Toggle failed: ' + err.message, 'error'); - checkbox.checked = !enabled; // revert - checkbox.disabled = false; - } - } - - // ── Helpers ────────────────────────────────────────────────────────── - - _closeAll() { - this._closeEdit(); - this._closeDel(); - this._closeCreate(); - } - - _updateStat() { - const v = this._editorEl?.value || ''; - const lines = v ? v.split('\n').length : 0; - const el = document.getElementById(`${this._uid}-charcount`); - if (el) el.textContent = `${lines.toLocaleString()} lines · ${v.length.toLocaleString()} chars`; - } - - _showErr(msg) { - const el = document.getElementById(`${this._uid}-edit-err`); - if (el) { el.textContent = msg; el.hidden = false; } - } - - _clearErr() { - const el = document.getElementById(`${this._uid}-edit-err`); - if (el) { el.textContent = ''; el.hidden = true; } - } - - _notify(msg, type) { - window.showNotification(msg, type || 'info'); - } - - _busy(btn, label) { - if (!btn) return; - btn._jfmOrigText = btn.textContent; - btn.disabled = true; - btn.textContent = ''; - const spin = document.createElement('span'); - spin.className = 'jfm-spin'; - btn.appendChild(spin); - btn.appendChild(document.createTextNode(' ' + label)); - } - - _idle(btn, label) { - if (!btn) return; - btn.disabled = false; - btn.textContent = btn._jfmOrigText !== undefined ? btn._jfmOrigText : label; - delete btn._jfmOrigText; - } - - _esc(str) { - return window.LEDEscape.html(str); - } - - _fmtSize(bytes) { - if (!bytes) return '0 B'; - const i = Math.min(Math.floor(Math.log2(bytes + 1) / 10), 2); - const unit = ['B', 'KB', 'MB'][i]; - const val = bytes / Math.pow(1024, i); - return (i ? val.toFixed(1) : val) + ' ' + unit; - } - - _fmtDate(str) { - if (!str) return '—'; - try { - return new Date(str).toLocaleDateString(undefined, { - month: 'short', day: 'numeric', year: 'numeric' - }); - } catch { return str; } - } - } - - // ── Widget registry integration ────────────────────────────────────────── - - window.JsonFileManager = JsonFileManager; - - if (typeof window.LEDMatrixWidgets !== 'undefined') { - window.LEDMatrixWidgets.register('json-file-manager', { - name: 'JSON File Manager', - version: '1.0.0', - render(container, config, _value, options) { - new JsonFileManager(container, config || {}, options?.pluginId || ''); - }, - getValue() { return null; }, - setValue() {} - }); - } -})(); diff --git a/web_interface/static/v3/js/widgets/notification.js b/web_interface/static/v3/js/widgets/notification.js index a123eb1d..5f156233 100644 --- a/web_interface/static/v3/js/widgets/notification.js +++ b/web_interface/static/v3/js/widgets/notification.js @@ -161,8 +161,6 @@ setTimeout(() => { region.textContent = text; }, 50); } - function escapeHtml(text) { return window.LEDEscape.html(text); } - function clearTimer(notificationId) { const t = timers.get(notificationId); if (t && t.timer) clearTimeout(t.timer); @@ -294,7 +292,7 @@ html += ``; } - html += `${style.label}: ${escapeHtml(message)}`; + html += `${style.label}: ${window.LEDEscape.html(message)}`; // Optional inline action button (e.g. "Restart Now" on a restart nudge). // The callback is stored by id and invoked via triggerAction, which @@ -306,7 +304,7 @@ onclick="window.LEDMatrixWidgets.get('notification').triggerAction('${notificationId}')" class="flex-shrink-0 ml-2 px-3 py-1 text-xs font-semibold rounded-md bg-white bg-opacity-20 hover:bg-opacity-30 transition-colors duration-150" style="background:rgba(255,255,255,.2);color:inherit;border:0;"> - ${escapeHtml(options.actionLabel)} + ${window.LEDEscape.html(options.actionLabel)} `; } diff --git a/web_interface/static/v3/js/widgets/number-input.js b/web_interface/static/v3/js/widgets/number-input.js index 3fdd039d..437cc24b 100644 --- a/web_interface/static/v3/js/widgets/number-input.js +++ b/web_interface/static/v3/js/widgets/number-input.js @@ -28,8 +28,6 @@ const base = window.BaseWidget ? new window.BaseWidget('NumberInput', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -80,16 +78,16 @@ const currentValue = rawValue === '' ? '' : (isNaN(Number(rawValue)) ? '' : String(Number(rawValue))); // Escape values for safe HTML attribute interpolation - const safeMin = min !== null ? escapeHtml(String(min)) : ''; - const safeMax = max !== null ? escapeHtml(String(max)) : ''; - const safeStep = escapeHtml(String(step)); + const safeMin = min !== null ? window.LEDEscape.html(String(min)) : ''; + const safeMax = max !== null ? window.LEDEscape.html(String(max)) : ''; + const safeStep = window.LEDEscape.html(String(step)); let html = `
`; html += '
'; if (prefix) { - html += `${escapeHtml(prefix)}`; + html += `${window.LEDEscape.html(prefix)}`; } if (showButtons && !disabled) { @@ -108,9 +106,9 @@ html += ` ${escapeHtml(suffix)}`; + html += `${window.LEDEscape.html(suffix)}`; } html += '
'; @@ -142,7 +140,7 @@ const rangeText = min !== null && max !== null ? `${min} - ${max}` : (min !== null ? `Min: ${min}` : `Max: ${max}`); - html += `
${escapeHtml(rangeText)}
`; + html += `
${window.LEDEscape.html(rangeText)}
`; } // Error message area diff --git a/web_interface/static/v3/js/widgets/password-input.js b/web_interface/static/v3/js/widgets/password-input.js index 4b7d051d..ef9bfa9a 100644 --- a/web_interface/static/v3/js/widgets/password-input.js +++ b/web_interface/static/v3/js/widgets/password-input.js @@ -28,8 +28,6 @@ const base = window.BaseWidget ? new window.BaseWidget('PasswordInput', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -120,9 +118,9 @@ html += ` 0 ? `minlength="${sanitizedMinLength}"` : ''} ${disabled ? 'disabled' : ''} ${required ? 'required' : ''} diff --git a/web_interface/static/v3/js/widgets/plugin-file-manager.js b/web_interface/static/v3/js/widgets/plugin-file-manager.js index 8dca0ad5..b8b348c0 100644 --- a/web_interface/static/v3/js/widgets/plugin-file-manager.js +++ b/web_interface/static/v3/js/widgets/plugin-file-manager.js @@ -238,8 +238,6 @@ window.showNotification(msg, type); } - function escHtml(s) { return window.LEDEscape.html(s); } - function formatSize(bytes) { if (bytes >= 1048576) return (bytes / 1048576).toFixed(1) + ' MB'; return (bytes / 1024).toFixed(2) + ' KB'; @@ -399,17 +397,17 @@ modal.className = 'pfm-modal'; safeSetHTML(modal, `
- ${escHtml(filename)} -
-
+
Loading…
- - +
`); @@ -423,7 +421,7 @@ const data = await callAction(st.pluginId, st.actions.get, { filename }).catch(() => null); const body = document.getElementById(`${fieldId}_edit_body`); if (!data || data.status !== 'success' || !body) { - if (body) safeSetHTML(body, `
Couldn't load ${escHtml(filename)}. Close this window and try again.
`); + if (body) safeSetHTML(body, `
Couldn't load ${window.LEDEscape.html(filename)}. Close this window and try again.
`); return; } @@ -438,10 +436,10 @@ // Textarea path: _editData stays null; save() reads from the - `); + + `); } }; @@ -465,7 +463,7 @@ // Delegated listener: day/col reach _pfmCellEdit only through data-* // attributes, never through a JS string spliced into an inline // handler -- the browser HTML-decodes attribute values before - // running them as script, which undoes escHtml's quote escaping and + // running them as script, which undoes LEDEscape's quote escaping and // reopens the exact injection it exists to close. `container` is a // fresh element per modal open (see _pfmOpenEdit), so this attaches // exactly once per table, even though buildPage() re-renders below. @@ -498,21 +496,21 @@ Day - ${cols.map(c => `${escHtml(c.replace(/_/g, ' ').replace(/\b\w/g, l => l.toUpperCase()))}`).join('')} + ${cols.map(c => `${window.LEDEscape.html(c.replace(/_/g, ' ').replace(/\b\w/g, l => l.toUpperCase()))}`).join('')} ${pageEntries.map(([day, val]) => ` - - ${escHtml(day)} + + ${window.LEDEscape.html(day)} ${cols.map(col => { const v = val[col] ?? ''; const isLong = String(v).length > 60 || col === 'description' || col === 'definition' || col === 'content'; return isLong - ? `` - : ``; + ? `` + : ``; }).join('')} `).join('')} @@ -604,20 +602,20 @@ modal.style.maxWidth = '28rem'; safeSetHTML(modal, `
- Delete File -
- ${escHtml(filename)} will be permanently deleted and removed + ${window.LEDEscape.html(filename)} will be permanently deleted and removed from the plugin configuration. This cannot be undone.
- - +
`); @@ -654,25 +652,25 @@ modal.style.maxWidth = '32rem'; safeSetHTML(modal, `
- Create New File -
- + ${fields.map(f => `
- - - ${f.hint ? `
${escHtml(f.hint)}
` : ''} + + + ${f.hint ? `
${window.LEDEscape.html(f.hint)}
` : ''}
`).join('')}
- - +
`); @@ -818,7 +816,7 @@
File Explorer
- ${st.directoryLabel ? `
Manage files in ${escHtml(st.directoryLabel)}
` : ''} + ${st.directoryLabel ? `
Manage files in ${window.LEDEscape.html(st.directoryLabel)}
` : ''}
${actions.create ? ` @@ -841,7 +839,7 @@ onchange="if(this.files[0])window._pfmUpload('${fieldId}',this.files[0]);this.value=''">

Drag and drop or click to upload

- ${escHtml(st.uploadHint)} + ${window.LEDEscape.html(st.uploadHint)}
` : ''}
diff --git a/web_interface/static/v3/js/widgets/radio-group.js b/web_interface/static/v3/js/widgets/radio-group.js index 6971a859..ff9e44c8 100644 --- a/web_interface/static/v3/js/widgets/radio-group.js +++ b/web_interface/static/v3/js/widgets/radio-group.js @@ -33,8 +33,6 @@ const base = window.BaseWidget ? new window.BaseWidget('RadioGroup', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -83,15 +81,15 @@
- ${escapeHtml(label)} - ${description ? `

${escapeHtml(description)}

` : ''} + ${window.LEDEscape.html(label)} + ${description ? `

${window.LEDEscape.html(description)}

` : ''}
`; diff --git a/web_interface/static/v3/js/widgets/schedule-picker.js b/web_interface/static/v3/js/widgets/schedule-picker.js index 69c8cab8..d4cb13e9 100644 --- a/web_interface/static/v3/js/widgets/schedule-picker.js +++ b/web_interface/static/v3/js/widgets/schedule-picker.js @@ -54,8 +54,6 @@ // Use BaseWidget utilities if available const base = window.BaseWidget ? new window.BaseWidget('SchedulePicker', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -239,7 +237,7 @@

When to start displaying content (HH:MM)

@@ -248,7 +246,7 @@

When to stop displaying content (HH:MM)

@@ -279,7 +277,7 @@ // Render each day row DAYS.forEach(day => { const dayConfig = schedule.days[day]; - const dayLabel = escapeHtml(DAY_LABEL_MAP.get(day) || day); + const dayLabel = window.LEDEscape.html(DAY_LABEL_MAP.get(day) || day); const disabled = !dayConfig.enabled; const disabledClass = disabled ? 'bg-gray-100' : ''; @@ -300,7 +298,7 @@ @@ -309,7 +307,7 @@ @@ -345,15 +343,15 @@ html += ``; // Global times (used when mode is global) - html += ``; - html += ``; + html += ``; + html += ``; // Per-day values (used when mode is per_day) DAYS.forEach(day => { const dayConfig = schedule.days[day]; html += ``; - html += ``; - html += ``; + html += ``; + html += ``; }); return html; diff --git a/web_interface/static/v3/js/widgets/select-dropdown.js b/web_interface/static/v3/js/widgets/select-dropdown.js index 30aaa9c5..3e253c1b 100644 --- a/web_interface/static/v3/js/widgets/select-dropdown.js +++ b/web_interface/static/v3/js/widgets/select-dropdown.js @@ -28,8 +28,6 @@ const base = window.BaseWidget ? new window.BaseWidget('SelectDropdown', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -73,7 +71,7 @@ html += ` '; diff --git a/web_interface/static/v3/js/widgets/slider.js b/web_interface/static/v3/js/widgets/slider.js index 08d5a94f..3e20868c 100644 --- a/web_interface/static/v3/js/widgets/slider.js +++ b/web_interface/static/v3/js/widgets/slider.js @@ -28,10 +28,6 @@ const base = window.BaseWidget ? new window.BaseWidget('Slider', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - - function escapeAttr(text) { return window.LEDEscape.attr(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -78,14 +74,14 @@ const currentValue = value !== null && value !== undefined ? value : min; const colorClass = COLOR_CLASSES[color] || COLOR_CLASSES.blue; - let html = `
`; + let html = `
`; // Value display above slider if (showValue) { html += `
- ${escapeHtml(prefix)}${escapeHtml(currentValue)}${escapeHtml(suffix)} + ${window.LEDEscape.html(prefix)}${window.LEDEscape.html(currentValue)}${window.LEDEscape.html(suffix)}
`; @@ -95,11 +91,11 @@ html += ` - ${escapeHtml(prefix)}${escapeHtml(min)}${escapeHtml(suffix)} - ${escapeHtml(prefix)}${escapeHtml(max)}${escapeHtml(suffix)} + ${window.LEDEscape.html(prefix)}${window.LEDEscape.html(min)}${window.LEDEscape.html(suffix)} + ${window.LEDEscape.html(prefix)}${window.LEDEscape.html(max)}${window.LEDEscape.html(suffix)}
`; } diff --git a/web_interface/static/v3/js/widgets/text-input.js b/web_interface/static/v3/js/widgets/text-input.js index 034ce5a9..17c8b176 100644 --- a/web_interface/static/v3/js/widgets/text-input.js +++ b/web_interface/static/v3/js/widgets/text-input.js @@ -29,8 +29,6 @@ const base = window.BaseWidget ? new window.BaseWidget('TextInput', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -80,14 +78,14 @@ const currentValue = value !== null && value !== undefined ? String(value) : ''; - let html = `
`; + let html = `
`; // Container for prefix/input/suffix layout const hasAddons = prefix || suffix || clearable; if (hasAddons) { html += '
'; if (prefix) { - html += `${escapeHtml(prefix)}`; + html += `${window.LEDEscape.html(prefix)}`; } } @@ -98,10 +96,10 @@ html += ` ${escapeHtml(suffix)}`; + html += `${window.LEDEscape.html(suffix)}`; } if (hasAddons) { diff --git a/web_interface/static/v3/js/widgets/textarea.js b/web_interface/static/v3/js/widgets/textarea.js index e4c1108b..c7b49b0c 100644 --- a/web_interface/static/v3/js/widgets/textarea.js +++ b/web_interface/static/v3/js/widgets/textarea.js @@ -26,8 +26,6 @@ const base = window.BaseWidget ? new window.BaseWidget('Textarea', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -75,15 +73,15 @@ html += ` + class="form-textarea w-full rounded-md border-gray-300 shadow-sm focus:border-blue-500 focus:ring-blue-500 ${resizeClass} ${disabled ? 'bg-gray-100 cursor-not-allowed' : 'bg-white'} text-black placeholder:text-gray-400">${window.LEDEscape.html(currentValue)} `; // Character count diff --git a/web_interface/static/v3/js/widgets/time-picker.js b/web_interface/static/v3/js/widgets/time-picker.js index 9ae0ead4..a1ae7c77 100644 --- a/web_interface/static/v3/js/widgets/time-picker.js +++ b/web_interface/static/v3/js/widgets/time-picker.js @@ -25,8 +25,6 @@ const base = window.BaseWidget ? new window.BaseWidget('TimePicker', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -77,9 +75,9 @@
`; // Hidden inputs for form submission - html += ``; - html += ``; + html += ``; + html += ``; html += `
`; // Start time input html += `
- + @@ -170,10 +168,10 @@ // End time input html += `
- + @@ -187,7 +185,7 @@ const duration = calculateDuration(startTime, endTime, allowOvernight); html += `
- Duration: ${escapeHtml(duration)} + Duration: ${window.LEDEscape.html(duration)}
`; } diff --git a/web_interface/static/v3/js/widgets/timezone-selector.js b/web_interface/static/v3/js/widgets/timezone-selector.js index 21cbab08..7675396c 100644 --- a/web_interface/static/v3/js/widgets/timezone-selector.js +++ b/web_interface/static/v3/js/widgets/timezone-selector.js @@ -23,8 +23,6 @@ const base = window.BaseWidget ? new window.BaseWidget('TimezoneSelector', '1.0.0') : null; - function escapeHtml(text) { return window.LEDEscape.html(text); } - function sanitizeId(id) { if (base) return base.sanitizeId(id); return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); @@ -230,7 +228,7 @@ let html = `
`; // Hidden input for form submission - html += ``; + html += ``; html += ` `; + html += ``; html += `
@@ -2779,7 +2779,7 @@ function attachInstallButtonHandler() { debugLog('[attachInstallButtonHandler] Response data:', data); if (data.status === 'success') { if (pluginStatusDiv) { - pluginStatusDiv.innerHTML = `Successfully installed: ${escapeHtml(data.plugin_id)}`; + pluginStatusDiv.innerHTML = `Successfully installed: ${window.LEDEscape.html(data.plugin_id)}`; } pluginUrlInput.value = ''; window.noteRestartRequired(data); @@ -2790,14 +2790,14 @@ function attachInstallButtonHandler() { }, 1000); } else { if (pluginStatusDiv) { - pluginStatusDiv.innerHTML = `${escapeHtml(data.message || 'Installation failed')}`; + pluginStatusDiv.innerHTML = `${window.LEDEscape.html(data.message || 'Installation failed')}`; } } }) .catch(error => { console.error('[attachInstallButtonHandler] Error:', error); if (pluginStatusDiv) { - pluginStatusDiv.innerHTML = `Error: ${escapeHtml(error.message)}`; + pluginStatusDiv.innerHTML = `Error: ${window.LEDEscape.html(error.message)}`; } }) .finally(() => { @@ -2946,7 +2946,7 @@ function setupGitHubInstallHandlers() { } }) .catch(error => { - registryStatusDiv.innerHTML = `Error: ${escapeHtml(error.message)}`; + registryStatusDiv.innerHTML = `Error: ${window.LEDEscape.html(error.message)}`; customRegistryPlugins.classList.add('hidden'); }) .finally(() => { @@ -3045,8 +3045,8 @@ function renderCustomRegistryPlugins(plugins, registryUrl) {
-
${escapeHtml(plugin.name || plugin.id)}
-

${escapeHtml(plugin.description || 'No description')}

+
${window.LEDEscape.html(plugin.name || plugin.id)}
+

${window.LEDEscape.html(plugin.description || 'No description')}

@@ -3080,7 +3080,7 @@ function showInstalledLoadError(message) { content.innerHTML = `
-

${escapeHtml(message)}

+

${window.LEDEscape.html(message)}

`; } @@ -3104,10 +3104,8 @@ function isGithubUrl(url) { } } -// Short local names for window.LEDEscape (app-early.js), which says what each -// one is for. Function declarations, so they are usable anywhere in this IIFE. -function escapeHtml(text) { return window.LEDEscape.html(text); } -function escapeAttribute(text) { return window.LEDEscape.attr(text); } +// Short local name for window.LEDEscape.jsStringAttr (app-early.js). A function +// declaration, so it is usable anywhere in this IIFE. function jsStringAttr(value) { return window.LEDEscape.jsStringAttr(value); } function isNewPlugin(lastUpdated) { @@ -3428,8 +3426,6 @@ document.addEventListener('htmx:afterSettle', function() { let starlarkDataLoaded = false; // ── Helpers ───────────────────────────────────────────────────────────── - function escapeHtml(str) { return window.LEDEscape.html(str); } - function isStarlarkInstalled(appId) { // Check window.installedPlugins (populated by loadInstalledPlugins) if (window.installedPlugins && Array.isArray(window.installedPlugins)) { @@ -3487,7 +3483,7 @@ document.addEventListener('htmx:afterSettle', function() { if (!banner) return; if (data.pixlet_available) { banner.innerHTML = `
- Pixlet available${data.pixlet_version ? ' (' + escapeHtml(data.pixlet_version) + ')' : ''} — ${data.installed_apps || 0} app(s) installed + Pixlet available${data.pixlet_version ? ' (' + window.LEDEscape.html(data.pixlet_version) + ')' : ''} — ${data.installed_apps || 0} app(s) installed
`; } else { banner.innerHTML = `
@@ -3514,7 +3510,7 @@ document.addEventListener('htmx:afterSettle', function() { .then(r => r.json()) .then(data => { if (data.status !== 'success') { - if (grid) grid.innerHTML = `
${escapeHtml(data.message || 'Failed to load')}
`; + if (grid) grid.innerHTML = `
${window.LEDEscape.html(data.message || 'Failed to load')}
`; return; } @@ -3693,22 +3689,22 @@ document.addEventListener('htmx:afterSettle', function() { setGridHtmlIfChanged(grid, apps.map(app => { const installed = isStarlarkInstalled(app.id); return ` -
+
-

${escapeHtml(app.name || app.id)}

+

${window.LEDEscape.html(app.name || app.id)}

Starlark ${installed ? 'Installed' : ''}
- ${app.author ? `

${escapeHtml(app.author)}

` : ''} - ${app.category ? `

${escapeHtml(app.category)}

` : ''} + ${app.author ? `

${window.LEDEscape.html(app.author)}

` : ''} + ${app.category ? `

${window.LEDEscape.html(app.category)}

` : ''}
-

${escapeHtml(app.summary || app.desc || 'No description')}

+

${window.LEDEscape.html(app.summary || app.desc || 'No description')}

diff --git a/web_interface/tailwind/tailwind.config.js b/web_interface/tailwind/tailwind.config.js index 30b5aa44..bfdf8575 100644 --- a/web_interface/tailwind/tailwind.config.js +++ b/web_interface/tailwind/tailwind.config.js @@ -39,6 +39,13 @@ module.exports = { 'sm:hidden', 'sm:inline', 'sm:max-w-4xl', 'md:flex', 'lg:grid-cols-5', 'lg:grid-cols-6', 'lg:gap-x-6', 'xl:grid-cols-5', 'xl:grid-cols-6', 'xl:grid-cols-7', 'xl:grid-cols-8', 'xl:gap-x-8', + // Used only by the removed js/utils/error_handler.js modal; kept for the + // same reason (soccer-scoreboard's custom-leagues widget uses max-h-48). + 'align-bottom', 'bg-opacity-75', 'leading-6', 'list-inside', 'max-h-48', + 'pb-20', 'pt-5', 'transition-opacity', 'focus:ring-indigo-500', + 'sm:align-middle', 'sm:flex', 'sm:flex-row-reverse', 'sm:h-10', 'sm:items-start', + 'sm:max-w-lg', 'sm:ml-3', 'sm:ml-4', 'sm:mt-0', 'sm:mx-0', 'sm:my-8', 'sm:p-0', + 'sm:p-6', 'sm:pb-4', 'sm:text-left', 'sm:w-10', 'sm:w-auto', 'sm:w-full', ], // Rules app.css defines on purpose and Tailwind must not override. Tailwind diff --git a/web_interface/templates/v3/base.html b/web_interface/templates/v3/base.html index 552fd500..ed1ace5d 100644 --- a/web_interface/templates/v3/base.html +++ b/web_interface/templates/v3/base.html @@ -930,7 +930,6 @@ - diff --git a/web_interface/templates/v3/partials/logs.html b/web_interface/templates/v3/partials/logs.html index b7471286..f04eef74 100644 --- a/web_interface/templates/v3/partials/logs.html +++ b/web_interface/templates/v3/partials/logs.html @@ -493,7 +493,7 @@ function updatePluginFilterOptions() { const plugins = Array.from(new Set(window._allLogs.map(log => log.plugin).filter(Boolean))).sort(); pluginFilterEl.innerHTML = '' + - plugins.map(p => ``).join(''); + plugins.map(p => ``).join(''); // Restore previous selection if it's still a valid option if (previousValue && plugins.includes(previousValue)) { @@ -517,14 +517,14 @@ function renderLogs() { const logElement = document.createElement('div'); logElement.className = `log-entry py-1 px-2 hover:bg-gray-800 rounded transition-colors duration-150 ${getLogLevelClass(log.level)}`; const pluginBadge = log.plugin - ? `${escapeHtml(log.plugin)}` + ? `${window.LEDEscape.html(log.plugin)}` : ''; logElement.innerHTML = `
- ${escapeHtml(log.timestamp)} + ${window.LEDEscape.html(log.timestamp)} ${log.level} ${pluginBadge} - ${escapeHtml(log.message)} + ${window.LEDEscape.html(log.message)}
`; if (window._logsContent) { @@ -761,13 +761,11 @@ function showEmptyState() { function showError(message) { if (window._logsContent) { - window._logsContent.innerHTML = `
${escapeHtml(message)}
`; + window._logsContent.innerHTML = `
${window.LEDEscape.html(message)}
`; window._logsContent.classList.remove('hidden'); } } -function escapeHtml(text) { return window.LEDEscape.html(text); } - function refreshCurrentPluginStatus() { fetch('/api/v3/display/current-status') .then(response => response.json()) @@ -843,11 +841,11 @@ function renderPluginErrors(summary) { 'Types'; plugins.forEach(p => { const types = Object.entries(p.types) - .map(([type, n]) => `${escapeHtml(type)} ×${escapeHtml(String(Number(n) || 0))}`) + .map(([type, n]) => `${window.LEDEscape.html(type)} ×${window.LEDEscape.html(String(Number(n) || 0))}`) .join(', '); html += '' + - `${escapeHtml(p.id)}` + - `${escapeHtml(String(p.total))}` + + `${window.LEDEscape.html(p.id)}` + + `${window.LEDEscape.html(String(p.total))}` + `${types}`; }); html += '
'; @@ -862,16 +860,16 @@ function renderPluginErrors(summary) { html += '

Repeating errors

    '; patterns.forEach(p => { const severity = String(p.severity || 'warning'); - const affected = (p.affected_plugins || []).map(id => escapeHtml(id)).join(', ') || 'unknown'; + const affected = (p.affected_plugins || []).map(id => window.LEDEscape.html(id)).join(', ') || 'unknown'; const sample = (p.sample_messages || [])[0]; html += '
  • ' + '
    ' + - `${escapeHtml(severity)}` + - `${escapeHtml(p.error_type || '')}` + - `×${escapeHtml(String(Number(p.count) || 0))}` + - `last seen ${escapeHtml(formatErrorTime(p.last_seen))}
    ` + + `${window.LEDEscape.html(severity)}` + + `${window.LEDEscape.html(p.error_type || '')}` + + `×${window.LEDEscape.html(String(Number(p.count) || 0))}` + + `last seen ${window.LEDEscape.html(formatErrorTime(p.last_seen))}
` + `
Plugins: ${affected}
` + - (sample ? `
${escapeHtml(sample)}
` : '') + + (sample ? `
${window.LEDEscape.html(sample)}
` : '') + ''; }); html += ''; diff --git a/web_interface/templates/v3/partials/tools.html b/web_interface/templates/v3/partials/tools.html index 0875ac40..4814f281 100644 --- a/web_interface/templates/v3/partials/tools.html +++ b/web_interface/templates/v3/partials/tools.html @@ -403,14 +403,14 @@
- ${escHtml(message)} + ${window.LEDEscape.html(message)}
`; if (output) { html += `
Show output - 
${escHtml(output)}
+ 
${window.LEDEscape.html(output)}
`; } @@ -421,10 +421,10 @@ const di = d.ok ? 'fa-check' : 'fa-times'; html += `
  • - ${escHtml(d.plugin)}`; + ${window.LEDEscape.html(d.plugin)}`; if (d.output) { html += `
    output - 
    ${escHtml(d.output)}
    `; + 
    ${window.LEDEscape.html(d.output)}
    `; } html += `
    • `; } @@ -435,8 +435,6 @@ el.innerHTML = html; } - function escHtml(s) { return window.LEDEscape.html(s); } - // ── main action dispatcher ──────────────────────────────────────────────── window.toolsAction = function(action, btnId, resultId, showOutput, showPluginDetails) { @@ -507,7 +505,7 @@ }) .then(d => { if (d.status === 'error') { - panel.innerHTML = `${escHtml(d.message || 'Git info unavailable.')}`; + panel.innerHTML = `${window.LEDEscape.html(d.message || 'Git info unavailable.')}`; return; } @@ -519,45 +517,45 @@ let html = `
    - ${escHtml(d.detached ? (d.version || 'detached') : (d.branch || 'unknown'))} + ${window.LEDEscape.html(d.detached ? (d.version || 'detached') : (d.branch || 'unknown'))} ${dirtyBadge}
    `; if (d.dirty && d.status) { - html += `
    ${escHtml(d.status)}
    `; + html += `
    ${window.LEDEscape.html(d.status)}
    `; } if (d.recent_commits) { html += `

    Recent commits

    - 
    ${escHtml(d.recent_commits)}
    + 
    ${window.LEDEscape.html(d.recent_commits)}
    `; } if (d.detached && d.current_release) { // The stable update channel sits on release tags, with no branch. - html += `

    On release ${escHtml(d.current_release)}, not a branch (stable update channel). Pull Latest follows the update channel set on the General tab.

    `; + html += `

    On release ${window.LEDEscape.html(d.current_release)}, not a branch (stable update channel). Pull Latest follows the update channel set on the General tab.

    `; } else if (d.detached) { // Detached but not on a release: say what the channel does // with it, in the General tab's words. - html += `

    Not on a branch or a release. ${escHtml(d.channel_message || '')} Pull Latest follows the update channel set on the General tab.

    `; + html += `

    Not on a branch or a release. ${window.LEDEscape.html(d.channel_message || '')} Pull Latest follows the update channel set on the General tab.

    `; } else if (d.upstream) { - html += `

    tracking ${escHtml(d.upstream)}

    `; + html += `

    tracking ${window.LEDEscape.html(d.upstream)}

    `; } else if (d.can_pull) { - html += `

    No upstream set; Pull Latest will use origin/${escHtml(d.branch || '')} and set it.

    `; + html += `

    No upstream set; Pull Latest will use origin/${window.LEDEscape.html(d.branch || '')} and set it.

    `; } else { html += `

    No upstream and no matching branch on origin — Pull Latest cannot run. Switch to a branch that exists on the remote.

    `; } if (d.remote_url) { - html += `

    ${escHtml(d.remote_url)}

    `; + html += `

    ${window.LEDEscape.html(d.remote_url)}

    `; } html += `
    `; panel.innerHTML = html; }) .catch(err => { - panel.innerHTML = `Could not load git info: ${escHtml(String(err))}`; + panel.innerHTML = `Could not load git info: ${window.LEDEscape.html(String(err))}`; }); } @@ -574,7 +572,7 @@ .then(r => r.ok ? r.json() : r.json().then(d => Promise.reject(d.message || `HTTP ${r.status}`))) .then(d => { if (d.status === 'error') { - sel.innerHTML = ``; + sel.innerHTML = ``; sel.disabled = true; return; } @@ -592,7 +590,7 @@ if (!sel.options.length) add('', 'no branches found'); }) .catch(err => { - sel.innerHTML = ``; + sel.innerHTML = ``; sel.disabled = true; }); } @@ -680,7 +678,7 @@ let html = `
    - ${escHtml(summaryText)} + ${window.LEDEscape.html(summaryText)}
      `; @@ -688,7 +686,7 @@ const set = !!power[key]; html += `
    • - ${escHtml(label)} + ${window.LEDEscape.html(label)}
      • `; } html += `
    `; @@ -719,9 +717,9 @@
    -
    ${escHtml(label)}
    -
    ${escHtml(value)}
    - ${sub ? `
    ${escHtml(sub)}
    ` : ''} +
    ${window.LEDEscape.html(label)}
    +
    ${window.LEDEscape.html(value)}
    + ${sub ? `
    ${window.LEDEscape.html(sub)}
    ` : ''}
    `; @@ -781,7 +779,7 @@ 'Display Service', d.service_active ? 'Active' : 'Inactive', null); }) .catch(err => { - panel.innerHTML = `
    Diagnostics unavailable: ${escHtml(String(err))}
    `; + panel.innerHTML = `
    Diagnostics unavailable: ${window.LEDEscape.html(String(err))}
    `; }); }; @@ -925,7 +923,6 @@ }; // ── plugin health panel ────────────────────────────────────────────────── - function phEscape(s) { return window.LEDEscape.html(s); } function phFmtSecs(v) { if (typeof v !== 'number' || !isFinite(v)) return '—'; return v.toFixed(3) + 's'; @@ -968,10 +965,10 @@ const lastErr = h.degraded_reason || h.last_error || ''; const calls = (typeof m.call_count === 'number') ? m.call_count : '—'; const errCell = lastErr - ? '' + phEscape(lastErr) + '' + ? '' + window.LEDEscape.html(lastErr) + '' : '—'; rows += '' + - '' + phEscape(id) + '' + + '' + window.LEDEscape.html(id) + '' + '' + st.label + '' + '' + phFmtSecs(m.avg_execution_time) + '' + '' + phFmtSecs(m.max_execution_time) + '' + @@ -982,7 +979,7 @@ tbody.innerHTML = rows; } catch (e) { const emsg = (e && e.message) ? e.message : String(e); - tbody.innerHTML = 'Failed to load plugin health: ' + phEscape(emsg) + ''; + tbody.innerHTML = 'Failed to load plugin health: ' + window.LEDEscape.html(emsg) + ''; } } window.refreshPluginHealth = refreshPluginHealth; @@ -1042,8 +1039,8 @@ const c = data.config || {}; const field = (id, label, value, type, extra) => ` `; @@ -1102,9 +1099,9 @@

    - Saved to ${escHtml(data.config_path)}. + Saved to ${window.LEDEscape.html(data.config_path)}. Any field can also be set as - ${escHtml(data.env_override_prefix)}<KEY>, + ${window.LEDEscape.html(data.env_override_prefix)}<KEY>, which wins over the file.

    `).join('') : `

    No Starlark apps installed. Install one from the Plugins tab first - (looked in ${escHtml(appsDir)}). + (looked in ${window.LEDEscape.html(appsDir)}).

    `; body.innerHTML = active + `
    ${rows}
    `; - // Bound here rather than via an inline onclick: escHtml does not encode - // single quotes, and the id used to be interpolated into a single-quoted + // Bound here rather than via an inline onclick: the escaper used to leave + // single quotes alone, and the id used to be interpolated into a single-quoted // JS string, so an app directory containing an apostrophe could break // out of it and run script. Through dataset the value is only ever // parsed as an HTML attribute, never as JavaScript. @@ -1351,7 +1348,7 @@ }) .catch(err => { const body = document.getElementById('pixlet-editor-body'); - if (body) body.innerHTML = `

    Could not load: ${escHtml(err.message)}

    `; + if (body) body.innerHTML = `

    Could not load: ${window.LEDEscape.html(err.message)}

    `; }); }; diff --git a/web_interface/widget_bundle.py b/web_interface/widget_bundle.py index f37dbcc0..4c541aca 100644 --- a/web_interface/widget_bundle.py +++ b/web_interface/widget_bundle.py @@ -55,16 +55,10 @@ BUNDLE_ORDER = [ "password-input.js", "timezone-selector.js", "plugin-loader.js", - # Reusable JSON file manager (used via x-widget: json-file-manager) - "json-file-manager.js", ] # Widget files that must NOT be bundled, with the reason. -EXCLUDED = { - # Documentation example (docs/widget-guide.md); it registers the name - # 'color-picker' and would shadow the real color-picker.js. - "example-color-picker.js": "documentation example", -} +EXCLUDED = {} _lock = Lock() _cache = {"version": None, "body": None} From 1bcb524fc3876e041503f7259c1abed50996f8e2 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:40:08 -0400 Subject: [PATCH 37/37] feat(scroll): report a panel that cannot reach its refresh cap, and suggest one it can hold (#759) * feat(scroll): report a panel that cannot reach its refresh cap, and suggest one it can hold Scroll speeds are solved against display.hardware.limit_refresh_rate_hz, which is only a ceiling. A panel that cannot reach it still moves whole pixels per frame, but every scroll runs slow by the shortfall and the "smooth" ladder is the cap's, not the panel's. A user rig (Pi 4, 2x128x64, adafruit-hat-pwm, pwm_bits 9, gpio_slowdown 5) measured 107.6-113.1 Hz under a 120 Hz cap: 60 px/s ran at 55, and nothing said why. - scroll_config: refresh_shortfall() (more than 3% under the planned rate), holdable_cap() (a multiple of 10, 5% under the measurement, since the measurement is the fast end of an uncapped panel's drift), and describe_refresh_shortfall(). - FrameTimingRecorder.plan_refresh(): once the measured period has held for three trusted windows, a shortfall is logged once as a warning naming the cap to use. DisplayManager calls it only for a real panel, not the emulator or the fallback canvas. The stats file records planned_refresh_hz (additive). - GET /api/v3/config/refresh-rate, plus a hint under the Display tab's Limit Refresh Rate field (js/pages/display.js) with a button that fills in the suggested cap. - _panel_refresh_hz (behind the Vegas slider's advice) ignores a measurement written under a different cap, so a changed cap stops being advised from the old rate before the display restarts. Verified on ledpi with a temporary 200 Hz cap: the warning logged about a minute after the restart ("about 132 Hz ... Set Limit Refresh Rate to 120 Hz"), the endpoint returned the same shortfall, and the Display tab showed the hint; its button filled in 120. ledpi was restored afterwards. Rebased onto main after the Display tab became an ES-module page (#771); the hint moved from inline script into display.js, with a jsdom test. Co-Authored-By: Claude Sonnet 5.5 * fix(scroll): count only agreeing windows toward the refresh shortfall; ignore a null planned rate Review fixes on #759. - A window the period estimate rejects (more than MAX_REFRESH_DROP faster than the adopted period) no longer counts toward the shortfall check, and it restarts the run. After a loaded start fixed a slow period, later windows at the real, faster rate were rejected yet still counted, so the warning could name the slow rate against a cap the panel was meeting. It now needs REFRESH_CHECK_WINDOWS consecutive windows that agree with the period. - _panel_refresh_hz treats a stats file whose planned_refresh_hz key is present but null as no measurement. The emulator and the fallback canvas write it that way (DisplayManager never plans a refresh for them), and their frame rate says nothing about the cap. A file with no such key (an older display) keeps the old behaviour. Three new tests fail on the previous code and pass now. Co-Authored-By: Claude Sonnet 5.5 --------- Co-authored-by: Claude Sonnet 5.5 --- CHANGELOG.md | 19 +++++ docs/SCROLL_PERFORMANCE.md | 25 ++++++ src/common/frame_timing.py | 49 ++++++++++++ src/common/scroll_config.py | 63 +++++++++++++++ src/display_manager.py | 9 ++- test/fixtures/api_v3_url_map.json | 9 +++ test/js/README.md | 2 +- test/js/dom/test_display_page.js | 11 +++ test/test_frame_timing.py | 77 +++++++++++++++++++ test/test_scroll_config.py | 44 +++++++++++ .../web_interface/test_api_v3_refresh_rate.py | 73 ++++++++++++++++++ web_interface/blueprints/api_v3/config.py | 41 +++++++++- web_interface/static/v3/js/pages/display.js | 40 ++++++++++ .../templates/v3/partials/display.html | 1 + 14 files changed, 458 insertions(+), 5 deletions(-) create mode 100644 test/web_interface/test_api_v3_refresh_rate.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e2012211..d8e6200b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,25 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +### Scroll speed: a panel slower than its refresh cap is reported + +- Scroll speeds are solved against `limit_refresh_rate_hz`, so a panel that + cannot reach its cap ran every scroll slow by the shortfall, with no sign + why (one Pi 4 on a 120 Hz cap refreshed at ~110 Hz: 60 px/s ran at 55). + Once the display has measured the real rate over three windows of + scrolling, a panel more than 3% short of the cap is logged once, as a + warning from `src.common.frame_timing` that names a cap it can hold (a + multiple of 10, 5% under the measurement). The Display tab shows the same + under Limit Refresh Rate, with a button that fills it in, from the new + `GET /api/v3/config/refresh-rate`. Not checked in the emulator or on the + fallback canvas. +- The frame-stats file records `planned_refresh_hz` (additive), and the + scroll-speed advice behind the Vegas slider ignores a measurement written + under a different cap. Until now, after the cap changed, the slider kept + advising from the old rate until the display restarted. +- New in `src.common.scroll_config`: `refresh_shortfall()`, `holdable_cap()` + and `describe_refresh_shortfall()`. + ## 3.8.3 Fresh installs on Raspberry Pi OS Lite work again: 3.8.2's installer reported diff --git a/docs/SCROLL_PERFORMANCE.md b/docs/SCROLL_PERFORMANCE.md index 4c44865a..c980126b 100644 --- a/docs/SCROLL_PERFORMANCE.md +++ b/docs/SCROLL_PERFORMANCE.md @@ -77,6 +77,31 @@ The Vegas **Scroll Speed** slider in the web UI shows the same thing live: a line under it says what your speed will run as on this panel, and links to the nearest smooth speeds. +### A panel that cannot reach its cap + +Speeds are solved against `limit_refresh_rate_hz`, the configured cap, but a +cap is only a ceiling: a long chain, a high `pwm_bits` or a big +`gpio_slowdown` can leave the panel below it. One Pi 4 driving 2×128×64 on +`adafruit-hat-pwm` with `pwm_bits 9` and `gpio_slowdown 5` measured +107.6–113.1 Hz under a 120 Hz cap. Frames still move whole pixels, but +every scroll runs that much slower than configured (60 px/s ran at 55 px/s), +and the smooth speeds are the cap's rather than the panel's. + +The display measures the real rate from its own frames. About a minute +into scrolling, a panel more than 3% short of its cap is logged once: + +``` +WARNING - src.common.frame_timing - The panel refreshes at about 113 Hz, below +the 120 Hz that scroll speeds are planned for ... Set Limit Refresh Rate to +100 Hz (web UI, Display tab), which this panel can hold, and restart. +``` + +The Display tab says the same under **Limit Refresh Rate**, with a button +that fills in the suggested cap (`GET /api/v3/config/refresh-rate`). The +suggestion is a multiple of 10 at least 5% under the measurement, because +an uncapped panel drifts and the measurement is the fast end of it. A cap the +panel holds also stops the drift. + ### How a slow speed stays crisp `SwapOnVSync(canvas, framerate_fraction)` holds each frame for N panel diff --git a/src/common/frame_timing.py b/src/common/frame_timing.py index 4c0edbdc..15df9809 100644 --- a/src/common/frame_timing.py +++ b/src/common/frame_timing.py @@ -135,6 +135,8 @@ import time import traceback from typing import Any, Callable, Dict, List, Optional, Tuple, TypedDict +from src.common import scroll_config + logger = logging.getLogger(__name__) #: Bumped when a field changes meaning, so a reader can refuse stale files. @@ -179,6 +181,12 @@ MAX_REFRESH_DROP = 0.2 #: trusted -- about a second of scrolling. MIN_FRAMES_FOR_REFRESH = 90 +#: Consecutive windows that agree with the adopted refresh period (the one +#: that adopted it counts) before a panel slower than its cap is reported. The +#: estimate can still fall by up to MAX_REFRESH_DROP per window early on; the +#: warning should not name a rate one more window would have corrected. +REFRESH_CHECK_WINDOWS = 3 + FLUSH_INTERVAL = 10.0 #: A scroll's last frame older than this is a stall worth a stack dump. @@ -441,6 +449,12 @@ class FrameTimingRecorder: 1.0 / refresh_hz if refresh_hz and refresh_hz > 0 else None) # The first estimate, until a second window agrees with it. self._refresh_candidate: Optional[float] = None + # The rate scroll speeds are solved against; see plan_refresh(). + self.planned_refresh_hz: Optional[float] = None + # Trusted windows seen since the period was adopted, until the + # shortfall check has run. + self._refresh_windows = 0 + self._shortfall_checked = True self.totals: Dict[str, Any] = { "static_frames": 0, "scroll_frames": 0, @@ -639,7 +653,20 @@ class FrameTimingRecorder: self._refresh_candidate = estimate elif current * (1.0 - MAX_REFRESH_DROP) <= estimate < current: self.refresh_period = estimate + # Only a run of windows that agree with the period counts toward + # the shortfall check. One that disagrees (a faster one than the + # period may fall to, say, after a loaded start fixed a slow one) + # was rejected above, so the period does not reflect it, and a + # warning built on that period would name a rate the panel is not + # at. It resets the run; the check then waits for three that agree. + current = self.refresh_period + if current is not None: + agrees = abs(estimate - current) <= current * MAX_REFRESH_DROP + self._refresh_windows = self._refresh_windows + 1 if agrees else 0 period = self.refresh_period + if (period and not self._shortfall_checked + and self._refresh_windows >= REFRESH_CHECK_WINDOWS): + self._check_refresh_shortfall(1.0 / period) histograms = self.histograms for frame in batch: @@ -688,6 +715,25 @@ class FrameTimingRecorder: elif missed <= -1: totals["early_frames"] += 1 + def plan_refresh(self, hz: Optional[float]) -> None: + """Say what rate scroll speeds are solved against, before frames arrive. + + ``DisplayManager.refresh_hz``: the configured cap. Once the measured + rate has held for :data:`REFRESH_CHECK_WINDOWS` windows in a row, a panel that + falls short of it is logged once, with a cap it can hold (see + :func:`src.common.scroll_config.refresh_shortfall`). The display + manager calls this only for a real panel. + """ + self.planned_refresh_hz = hz + self._shortfall_checked = not hz + + def _check_refresh_shortfall(self, measured_hz: float) -> None: + """Log, once, a panel that cannot reach the rate speeds assume.""" + self._shortfall_checked = True + shortfall = scroll_config.refresh_shortfall(measured_hz, self.planned_refresh_hz) + if shortfall: + logger.warning(scroll_config.describe_refresh_shortfall(shortfall)) + def snapshot(self) -> Dict[str, Any]: """The JSON document: cumulative since this process started.""" if not self._binding_checked: @@ -704,6 +750,9 @@ class FrameTimingRecorder: "bucket_ms": BUCKET_MS, "freeze_seconds": FREEZE_SECONDS, "measured_refresh_hz": round(1.0 / period, 2) if period else None, + # Additive: what scroll speeds were solved against, so a reader can + # tell a stale file (written under another cap) from this one. + "planned_refresh_hz": self.planned_refresh_hz, "binding_releases_gil": self._binding_gil, "info": info, "totals": copy.deepcopy(self.totals), diff --git a/src/common/scroll_config.py b/src/common/scroll_config.py index f45acca1..8d3dc026 100644 --- a/src/common/scroll_config.py +++ b/src/common/scroll_config.py @@ -538,3 +538,66 @@ def speed_advice( "smooth": smooth, "alternatives": [as_dict(c) for c in alternatives], } + + +#: A measured refresh this far below the rate speeds are planned for means +#: the panel cannot reach its cap. Smaller gaps are the cap's own slack and +#: the estimate's: one rig measured 99.95 Hz under a 100 Hz cap. +REFRESH_SHORTFALL = 0.03 + +#: How far under the measured rate a suggested cap sits. The measurement is +#: the fast end of the panel's refreshes (frame_timing takes the 10th +#: percentile of intervals), and an uncapped panel drifts: one read +#: 107.6-113.1 Hz over 15 seconds. A cap inside that band would not hold. +CAP_HEADROOM = 0.05 + + +def holdable_cap(measured_hz: Any) -> Optional[int]: + """A refresh cap the panel can hold: a multiple of 10, 5% under what it measured. + + A multiple of 10 because its whole-pixel speeds are round numbers (a + 100 Hz cap gives 50 and 100 px/s). None without a usable measurement, or + when the panel is too slow for any cap of 10 Hz or more. + """ + hz = _coerce(measured_hz) + if hz is None: + return None + cap = int(hz * (1.0 - CAP_HEADROOM) // 10) * 10 + return cap if cap >= 10 else None + + +def refresh_shortfall(measured_hz: Any, planned_hz: Any) -> Optional[Dict[str, Any]]: + """When the panel refreshes measurably slower than speeds are planned for. + + ``planned_hz`` is what :func:`configure` solves against -- the + ``limit_refresh_rate_hz`` cap, or :data:`DEFAULT_REFRESH_HZ` when it is 0. + A panel that cannot reach it still moves whole pixels per frame, but every + speed runs slow by the shortfall and the ladder of smooth speeds is the + cap's, not the panel's. None when there is no measurement, or the panel + reaches the cap (or beats it, as some do by a few Hz). + """ + measured, planned = _coerce(measured_hz), _coerce(planned_hz) + if measured is None or planned is None: + return None + if measured >= planned * (1.0 - REFRESH_SHORTFALL): + return None + return { + "measured_hz": round(measured, 1), + "planned_hz": round(planned, 1), + "suggested_cap_hz": holdable_cap(measured), + "slow_percent": round((1.0 - measured / planned) * 100), + } + + +def describe_refresh_shortfall(shortfall: Dict[str, Any]) -> str: + """One log line for :func:`refresh_shortfall`'s answer.""" + text = ( + f"The panel refreshes at about {shortfall['measured_hz']:.0f} Hz, below " + f"the {shortfall['planned_hz']:.0f} Hz that scroll speeds are planned " + f"for (display.hardware.limit_refresh_rate_hz), so every scroll runs " + f"about {shortfall['slow_percent']}% slower than configured and the " + f"smooth speeds are worked out for a rate this panel never reaches.") + if shortfall.get("suggested_cap_hz"): + text += (f" Set Limit Refresh Rate to {shortfall['suggested_cap_hz']} Hz " + f"(web UI, Display tab), which this panel can hold, and restart.") + return text diff --git a/src/display_manager.py b/src/display_manager.py index 05b04cb7..dcf3c284 100644 --- a/src/display_manager.py +++ b/src/display_manager.py @@ -393,6 +393,11 @@ class DisplayManager: self._setup_matrix() logger.info("Matrix setup completed in %.3f seconds", time.time() - start_time) + # Only a real panel's swaps wait on its refresh: the emulator and the + # fallback canvas pace themselves, so "slower than the cap" would be + # noise there. + if self.matrix is not None and os.environ.get('EMULATOR', 'false') != 'true': + self.frame_timing.plan_refresh(self.refresh_hz) self._setup_scan_order_compensation() font_time = time.time() @@ -1501,7 +1506,9 @@ class DisplayManager: fractional-pixel motion. See src/common/scroll_config.py. Note this is the configured *cap*, not necessarily what the panel - achieves -- scripts/scroll_speeds.py --measure reports the real rate. + achieves -- scripts/scroll_speeds.py --measure reports the real rate, + and the frame-timing recorder logs a warning, with a cap the panel can + hold, once it has measured a panel that falls short of this. """ hardware = (self.config.get('display') or {}).get('hardware') or {} try: diff --git a/test/fixtures/api_v3_url_map.json b/test/fixtures/api_v3_url_map.json index 707c8774..7beb1c09 100644 --- a/test/fixtures/api_v3_url_map.json +++ b/test/fixtures/api_v3_url_map.json @@ -175,6 +175,15 @@ "POST" ] ], + [ + "/api/v3/config/refresh-rate", + "api_v3.get_refresh_rate", + [ + "GET", + "HEAD", + "OPTIONS" + ] + ], [ "/api/v3/config/schedule", "api_v3.get_schedule_config", diff --git a/test/js/README.md b/test/js/README.md index 1fac3eb0..b4cfeec7 100644 --- a/test/js/README.md +++ b/test/js/README.md @@ -71,7 +71,7 @@ server has none. | `dom/test_raw_json_page.js` | yes | The Config Editor tab (`js/pages/raw-json.js`): one POST per Save after repeated swaps, Format/Validate, invalid JSON never sent, a save survives a swap, the old global entry points | | `dom/test_schedule_page.js` | yes | The Schedule tab (`js/pages/schedule.js`) with the real `schedule-picker` widget: both pickers drawn once per swap from the saved config, one notification per save answer after repeated swaps, the brightness label, a late widget waited for, the old global entry points | | `dom/test_visibility_service.js` | yes (no server) | `js/core/visibility.js` with the real `LEDVisibility` from `app-shell.js` and the real registry: start/stop with the active tab and the browser tab's visibility, no interval while hidden or after a swap-out, registrations independent, the no-`LEDVisibility` fallback | -| `dom/test_display_page.js` | yes | The Display tab (`js/pages/display.js`) with the real `plugin-order-list` widget and `LEDVisibility`: one page, one sync interval and one action per control after repeated swaps, the sync poll only while on screen and never after a swap-out, sync states as text, the debounced scroll-speed hint, `updateSyncUI`'s entry point | +| `dom/test_display_page.js` | yes | The Display tab (`js/pages/display.js`) with the real `plugin-order-list` widget and `LEDVisibility`: one page, one sync interval and one action per control after repeated swaps, the sync poll only while on screen and never after a swap-out, sync states as text, the debounced scroll-speed hint, the refresh-cap hint and its "Use N Hz" button, `updateSyncUI`'s entry point | | `dom/test_general_page.js` | yes | The General tab (`js/pages/general.js`) with the real `timezone-selector` widget: the picker drawn once per swap, one request per Security action after repeated swaps, hostile token names stay text, refused/network/login answers, a write survives a swap, `webLogin`'s entry points | | `dom/test_backup_restore_page.js` | yes | The Backup & Restore tab (`js/pages/backup-restore.js`): one request per action after repeated swaps, the upload and restore options, reads cancelled and writes not on a swap, hostile names stay text, the old global entry points | | `dom/test_tools_sections.js` | yes | The Tools tab's MQTT bridge and Pixlet editor sections: form prefill, the write-only password (blank means unchanged), the running-session banner and countdown, and that the editor link points at the host you loaded the page from | diff --git a/test/js/dom/test_display_page.js b/test/js/dom/test_display_page.js index 2cff377c..906d5bd1 100644 --- a/test/js/dom/test_display_page.js +++ b/test/js/dom/test_display_page.js @@ -88,6 +88,8 @@ const ok = (l, c, x) => c ? (pass++, console.log(' ok ' + l)) let syncAnswer = { status: 'success', data: { role: 'leader', state: 'no_peer' } }; let syncMode = 'ok'; let advice = smooth; + const shortfall = { measured_hz: 110.4, planned_hz: 120, suggested_cap_hz: 100, slow_percent: 8 }; + let refreshAnswer = { status: 'success', data: { planned_hz: 120, measured_hz: 110.4, shortfall } }; const requests = []; function fakeFetch(url, init = {}) { requests.push(url); @@ -99,6 +101,7 @@ const ok = (l, c, x) => c ? (pass++, console.log(' ok ' + l)) }); if (url === '/api/v3/plugins/installed') return respond(200, { status: 'success', data: { plugins } }); if (url.startsWith('/api/v3/config/scroll-speed-advice?')) return respond(200, advice); + if (url === '/api/v3/config/refresh-rate') return respond(200, refreshAnswer); if (url === '/api/v3/sync/status') { if (syncMode === 'network') return Promise.reject(new TypeError('Failed to fetch')); if (syncMode === 'login') return respond(401, { status: 'error' }, { 'X-LEDMatrix-Login': '/login' }); @@ -148,6 +151,14 @@ const ok = (l, c, x) => c ? (pass++, console.log(' ok ' + l)) // ── first load ────────────────────────────────────────────────────────── ok('one plugin-list request on start', count('/api/v3/plugins/installed') === 1, requests); ok('one scroll-speed hint request on start (after the debounce)', count('/api/v3/config/scroll-speed-advice') === 1, requests); + const refreshHint = $('limit_refresh_rate_hz_hint'); + ok('a panel short of its cap says so, as text, with a button for a cap it can hold', + /about 110 Hz, below this 120 Hz cap.*8% slower/.test(refreshHint.textContent) + && refreshHint.querySelector('button').textContent === 'Use 100 Hz', refreshHint.textContent); + refreshHint.querySelector('button').click(); + ok('the button fills the field and says to save and restart', + $('limit_refresh_rate_hz').value === '100' && /Save, then restart/.test(refreshHint.textContent), + [$('limit_refresh_rate_hz').value, refreshHint.textContent]); ok('the saved role is standalone: no sync request, no interval work', $('sync_role').value === 'standalone' && syncPolls() === 0, [$('sync_role').value, syncPolls()]); ok('the sync poll interval runs while the tab is on screen', intervals.size === 1 diff --git a/test/test_frame_timing.py b/test/test_frame_timing.py index e6610ddf..a97998b7 100644 --- a/test/test_frame_timing.py +++ b/test/test_frame_timing.py @@ -807,3 +807,80 @@ def test_a_process_with_the_gc_monitor_exits_cleanly(): assert proc.returncode == 0, proc.stderr assert "Exception ignored" not in proc.stderr assert "installed at exit: False" in proc.stdout + + +SLOW = 1 / 110.0 # a panel that cannot reach a 120 Hz cap + + +def _windows(recorder, n, interval, start=0.0): + for i in range(n): + _feed(recorder, [interval] * 200, start=start + 50.0 * i) + _aggregate(recorder) + + +def _shortfall_warnings(caplog): + return [r for r in caplog.records + if r.name == "src.common.frame_timing" and "Limit Refresh Rate" in r.getMessage()] + + +def test_a_panel_slower_than_its_cap_is_reported_once(tmp_path, caplog): + r = _recorder(tmp_path) + r.plan_refresh(120.0) + caplog.set_level("WARNING") + _windows(r, 3, SLOW) # adopted on the 2nd window, checked on the 4th + assert _shortfall_warnings(caplog) == [] + _windows(r, 3, SLOW, start=1000.0) + warnings = _shortfall_warnings(caplog) + assert len(warnings) == 1 + assert "about 110 Hz" in warnings[0].getMessage() + assert "to 100 Hz" in warnings[0].getMessage() + + +def test_a_panel_that_reaches_its_cap_is_not_reported(tmp_path, caplog): + r = _recorder(tmp_path) + r.plan_refresh(100.0) + caplog.set_level("WARNING") + _windows(r, 6, PERIOD) + assert _shortfall_warnings(caplog) == [] + + +def test_without_a_planned_rate_nothing_is_checked(tmp_path, caplog): + # The emulator and the fallback canvas: DisplayManager never calls + # plan_refresh(), since their frames are not paced by a panel. + r = _recorder(tmp_path) + caplog.set_level("WARNING") + _windows(r, 6, SLOW) + assert _shortfall_warnings(caplog) == [] + + +def test_the_snapshot_records_the_planned_rate(tmp_path): + r = _recorder(tmp_path) + assert r.snapshot()["planned_refresh_hz"] is None + r.plan_refresh(120.0) + assert r.snapshot()["planned_refresh_hz"] == 120.0 + + +def test_windows_the_period_rejected_do_not_count_toward_the_warning(tmp_path, caplog): + # A loaded start fixed 60 Hz (two windows agreed); the panel really runs at + # 100 Hz, but a window that much faster is ignored by the estimate, so the + # period stays 60 Hz. Warning "60 Hz is under your 100 Hz cap" would be wrong. + r = _recorder(tmp_path) + r.plan_refresh(100.0) + caplog.set_level("WARNING") + _windows(r, 2, 1 / 60.0) + assert abs(1.0 / r.refresh_period - 60.0) < 0.5 + _windows(r, 6, PERIOD, start=1000.0) + assert abs(1.0 / r.refresh_period - 60.0) < 0.5 # still ignored + assert _shortfall_warnings(caplog) == [] + + +def test_one_disagreeing_window_restarts_the_run(tmp_path, caplog): + r = _recorder(tmp_path) + r.plan_refresh(120.0) + caplog.set_level("WARNING") + _windows(r, 3, SLOW) # two windows toward three + _windows(r, 1, 1 / 250.0, start=1000.0) # far faster: rejected, resets + _windows(r, 1, SLOW, start=2000.0) + assert _shortfall_warnings(caplog) == [] + _windows(r, 2, SLOW, start=3000.0) # three in a row now + assert len(_shortfall_warnings(caplog)) == 1 diff --git a/test/test_scroll_config.py b/test/test_scroll_config.py index b3c143ca..e14047ff 100644 --- a/test/test_scroll_config.py +++ b/test/test_scroll_config.py @@ -21,6 +21,7 @@ from src.common.scroll_config import ( # noqa: E402 refresh_hz_from_config, resolve, ) +from src.common import scroll_config # noqa: E402 class FakeHelper: @@ -504,3 +505,46 @@ class TestSpeedAdvice: got = solve_crisp(50, 125.74) assert got.steppiness == "smooth" assert got.pixels_per_frame == 1 + + +class TestRefreshShortfall: + """A panel that cannot reach its cap runs every scroll slow.""" + + def test_the_ledmatrix_rig_is_told_to_cap_at_100(self): + # Pi 4, 2x128x64 on adafruit-hat-pwm under a 120 Hz cap: measured + # 107.6-113.1 Hz, and frame_timing reports the fast end. + s = scroll_config.refresh_shortfall(113.1, 120) + assert s == {"measured_hz": 113.1, "planned_hz": 120.0, + "suggested_cap_hz": 100, "slow_percent": 6} + + def test_a_panel_that_holds_its_cap_is_fine(self): + assert scroll_config.refresh_shortfall(99.95, 100) is None + assert scroll_config.refresh_shortfall(97.5, 100) is None + + def test_a_panel_that_beats_its_cap_is_fine(self): + assert scroll_config.refresh_shortfall(125.7, 120) is None + + def test_nothing_measured_says_nothing(self): + assert scroll_config.refresh_shortfall(None, 120) is None + assert scroll_config.refresh_shortfall(0, 120) is None + assert scroll_config.refresh_shortfall("fast", 120) is None + + def test_the_suggestion_leaves_headroom_under_the_measurement(self): + assert scroll_config.holdable_cap(113.1) == 100 + assert scroll_config.holdable_cap(95.0) == 90 + # 5% under 105 is 99.75: 100 would sit inside the panel's drift. + assert scroll_config.holdable_cap(105.0) == 90 + assert scroll_config.holdable_cap(9.0) is None + assert scroll_config.holdable_cap(None) is None + + def test_the_log_line_names_the_cap_to_use(self): + text = scroll_config.describe_refresh_shortfall( + scroll_config.refresh_shortfall(113.1, 120)) + assert "about 113 Hz" in text and "120 Hz" in text + assert "6% slower" in text + assert "Set Limit Refresh Rate to 100 Hz" in text + + def test_no_suggestion_for_a_panel_too_slow_for_any_cap(self): + text = scroll_config.describe_refresh_shortfall( + scroll_config.refresh_shortfall(9.0, 100)) + assert "Set Limit Refresh Rate" not in text diff --git a/test/web_interface/test_api_v3_refresh_rate.py b/test/web_interface/test_api_v3_refresh_rate.py new file mode 100644 index 00000000..5d71151d --- /dev/null +++ b/test/web_interface/test_api_v3_refresh_rate.py @@ -0,0 +1,73 @@ +"""GET /api/v3/config/refresh-rate: the cap, the measured rate, a cap to hold.""" +import json +from unittest.mock import MagicMock + +import pytest +from flask import Flask + +from web_interface.blueprints.api_v3 import api_v3 + + +@pytest.fixture +def client(monkeypatch, tmp_path): + stats = tmp_path / "stats.json" + monkeypatch.setattr("src.common.frame_timing.default_stats_path", lambda: str(stats)) + manager = MagicMock() + manager.load_config.return_value = { + "display": {"hardware": {"limit_refresh_rate_hz": 120}}} + monkeypatch.setattr(api_v3, "config_manager", manager, raising=False) + app = Flask(__name__) + app.register_blueprint(api_v3, url_prefix="/api/v3") + c = app.test_client() + c.stats_path = stats + return c + + +def _get(client): + body = client.get("/api/v3/config/refresh-rate").get_json() + assert body["status"] == "success" + return body["data"] + + +def test_nothing_measured_yet(client): + data = _get(client) + assert data == {"planned_hz": 120.0, "measured_hz": None, "shortfall": None} + + +def test_a_panel_short_of_its_cap_gets_a_cap_it_can_hold(client): + client.stats_path.write_text(json.dumps( + {"measured_refresh_hz": 110.4, "planned_refresh_hz": 120.0})) + data = _get(client) + assert data["measured_hz"] == 110.4 + assert data["shortfall"]["suggested_cap_hz"] == 100 + assert data["shortfall"]["slow_percent"] == 8 + + +def test_a_panel_at_its_cap_has_no_shortfall(client): + client.stats_path.write_text(json.dumps( + {"measured_refresh_hz": 121.3, "planned_refresh_hz": 120.0})) + assert _get(client)["shortfall"] is None + + +def test_a_file_written_under_another_cap_is_stale(client): + # The cap was changed to 120 but the display still runs under 100 Hz. + client.stats_path.write_text(json.dumps( + {"measured_refresh_hz": 99.9, "planned_refresh_hz": 100.0})) + data = _get(client) + assert data["measured_hz"] is None + assert data["shortfall"] is None + + +def test_a_file_from_a_display_too_old_to_record_its_cap_still_counts(client): + client.stats_path.write_text(json.dumps({"measured_refresh_hz": 110.4})) + assert _get(client)["shortfall"]["suggested_cap_hz"] == 100 + + +def test_a_measurement_recorded_without_a_planned_rate_is_not_a_panel(client): + # The emulator and the fallback canvas write the key as null: their frames + # are not paced by a panel, so a rate under the cap is no shortfall. + client.stats_path.write_text(json.dumps( + {"measured_refresh_hz": 60.0, "planned_refresh_hz": None})) + data = _get(client) + assert data["measured_hz"] is None + assert data["shortfall"] is None diff --git a/web_interface/blueprints/api_v3/config.py b/web_interface/blueprints/api_v3/config.py index 91ee73ba..943ace53 100644 --- a/web_interface/blueprints/api_v3/config.py +++ b/web_interface/blueprints/api_v3/config.py @@ -158,11 +158,23 @@ def _panel_refresh_hz(config): cap = scroll_config.refresh_hz_from_config(config) try: with open(frame_timing.default_stats_path(), encoding='utf-8') as fh: - measured = float(json.load(fh).get('measured_refresh_hz') or 0) + stats = json.load(fh) + measured = float(stats.get('measured_refresh_hz') or 0) + recorded = 'planned_refresh_hz' in stats + planned = float(stats.get('planned_refresh_hz') or 0) except (OSError, ValueError, TypeError, AttributeError): + measured = planned = 0.0 + recorded = False + # Reject a stale file from a previous hardware config: one written under + # another cap (the display has not restarted since it changed), or, from + # a display too old to record its cap (no such key), a measurement far + # off this one. A key that is present but null means the display's frames + # are not paced by a panel (the emulator, the fallback canvas): its + # "refresh rate" says nothing about the cap. + if recorded and not planned: + measured = 0.0 + elif planned and abs(planned - cap) > 0.5: measured = 0.0 - # Reject a stale file from a previous hardware config: a measurement far - # off the cap says the config changed since it was written. if measured > 0 and 0.5 * cap <= measured <= 1.5 * cap: return measured, 'measured' return cap, 'configured' @@ -189,6 +201,29 @@ def get_scroll_speed_advice(): return jsonify({'status': 'success', 'data': advice}) +@api_v3.route('/config/refresh-rate', methods=['GET']) +def get_refresh_rate(): + """The refresh cap, what the panel measured, and a cap it can hold. + + Backs the hint under the Display tab's Limit Refresh Rate field. Scroll + speeds are solved against the cap, so a panel that cannot reach it runs + every scroll slow; ``shortfall`` (None when the panel keeps up, or nothing + has been measured yet) says by how much and suggests a cap. + """ + from src.common import scroll_config + if not api_v3.config_manager: + return jsonify({'status': 'error', 'message': 'Config manager not initialized'}), 500 + config = api_v3.config_manager.load_config() + planned = scroll_config.refresh_hz_from_config(config) + hz, source = _panel_refresh_hz(config) + measured = hz if source == 'measured' else None + return jsonify({'status': 'success', 'data': { + 'planned_hz': planned, + 'measured_hz': round(measured, 1) if measured else None, + 'shortfall': scroll_config.refresh_shortfall(measured, planned), + }}) + + @api_v3.route('/config/schedule', methods=['GET']) def get_schedule_config(): """Get current schedule configuration""" diff --git a/web_interface/static/v3/js/pages/display.js b/web_interface/static/v3/js/pages/display.js index 646841b1..5ca6f01b 100644 --- a/web_interface/static/v3/js/pages/display.js +++ b/web_interface/static/v3/js/pages/display.js @@ -92,6 +92,45 @@ function refreshScrollSpeedHint(root, ctx) { }, HINT_DELAY_MS); } +// ── the refresh-cap hint ───────────────────────────────────────────────────── +// Scroll speeds are worked out against the cap, so a panel that cannot reach +// it runs every scroll slow. The display has measured a cap it can hold. +function showRefreshRateHint(root, ctx) { + const hint = root.querySelector('#limit_refresh_rate_hz_hint'); + const input = root.querySelector('#limit_refresh_rate_hz'); + if (!hint || !input) return; + const doc = root.ownerDocument; + const win = doc.defaultView; + ctx.api.get('/api/v3/config/refresh-rate', { signal: ctx.signal }) + .then(function(body) { + const s = body.status === 'success' && body.data.shortfall; + hint.textContent = ''; + if (!s) return; + hint.appendChild(doc.createTextNode( + 'This panel refreshes at about ' + Math.round(s.measured_hz) + + ' Hz, below this ' + Math.round(s.planned_hz) + ' Hz cap, so scrolls run about ' + + s.slow_percent + '% slower than set.' + (s.suggested_cap_hz ? ' ' : ''))); + if (!s.suggested_cap_hz) return; + const btn = doc.createElement('button'); + btn.type = 'button'; + btn.className = 'underline font-medium'; + btn.textContent = 'Use ' + s.suggested_cap_hz + ' Hz'; + btn.addEventListener('click', function() { + input.value = s.suggested_cap_hz; + input.dispatchEvent(new win.Event('input', { bubbles: true })); + input.dispatchEvent(new win.Event('change', { bubbles: true })); + hint.textContent = 'Save, then restart the display, to apply ' + + s.suggested_cap_hz + ' Hz.'; + }); + hint.appendChild(btn); + hint.appendChild(doc.createTextNode(', a cap it can hold.')); + }) + .catch(function(error) { + if (quiet(error) || error.body) return; + hint.textContent = ''; + }); +} + function renderScrollSpeedHint(root, hint, slider, a) { const doc = root.ownerDocument; const win = doc.defaultView; @@ -303,6 +342,7 @@ export function init(root, ctx) { // when it already is), then every 5 s while it stays there. ctx.visibility.every(SYNC_POLL_MS, function() { pollSyncStatus(root, ctx); }); + showRefreshRateHint(root, ctx); startPluginOrder(root, ctx); active = ctx; } diff --git a/web_interface/templates/v3/partials/display.html b/web_interface/templates/v3/partials/display.html index faf79ed2..4c3a0e8e 100644 --- a/web_interface/templates/v3/partials/display.html +++ b/web_interface/templates/v3/partials/display.html @@ -319,6 +319,7 @@ min="0" max="1000" class="form-control"> +