mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-05 14:55:08 +00:00
feat(starlark,on-demand): the third-party fixes worth taking, plus a Home Assistant MQTT bridge (#538)
* feat(starlark,on-demand): the third-party fixes worth taking, plus an MQTT bridge Analysis of ant456/ledmatrix-fixes-repo, a third-party collection of patches and services built while running this project on Starlark apps under MQTT control. Its patches are whole-file copies taken against an older tree, so applying them as written would revert #523's frame pacing, #534's display() bool returns and the GitHub token masking in plugins_manager.js. Three of its claimed fixes are already in main, and its api_v3 Starlark routes are #535's. What follows is the rest -- verified against current code, and reimplemented where the patch's approach did not hold up. **On-demand display.** `pinned` reached the controller from the API, was stored on it and republished in the status payload, but never narrowed the rotation -- a pinned request still cycled every mode its plugin owns. Right for a sports plugin, whose modes are views of one subject; wrong for a plugin whose modes are unrelated, which is every Starlark app. Now honoured, and it survives a restart. Restarting while on-demand was active loaded *only* the on-demand plugin, so normal rotation had nothing to return to for the life of the process -- and a restart mid-session is routine, since that is how an update is applied. The panel came back cycling one plugin's modes with no way out but clearing the cache by hand. Every enabled plugin loads now; on-demand still resumes on its saved mode. Stop requests are exempt from the duplicate guards on purpose, so that a second click stops a mode a race left running -- which means consuming the mailbox is the only thing that ends one. It was never consumed, so the same stop was re-read and re-processed on every poll, forever. Both paths now share one compare-before-delete helper. **Starlark rendering.** `extract_schema` parsed the source with a regex, which can only see option lists written out literally: an app whose dropdown is filled from a live API call inside `get_schema()` came back empty, and the config form offered nothing to pick. Now runs `pixlet schema`, which executes the app, and falls back to the parser when Pixlet is absent, too old for the subcommand, or the app fails to run. The third-party patch replaced the parser outright and hardcoded /usr/local/bin/pixlet; this keeps the fallback and the binary search. A `|` in a config value was dropped by a shell-metacharacter filter, though the command is a list with no shell involved -- and apps do use it as a separator inside one value. The key went missing silently and the app rendered its own "not configured" screen with nothing to say why. And a 0-byte render was reported as success: Pixlet exits 0 and writes nothing when an app has no content, which read downstream as a working app drawing a black panel. **Starlark display.** `display()` ignored the mode it was called with, so a specific app could not be addressed. It now accepts `display_mode` -- which is the whole mechanism, since the controller inspects the signature before passing it. Found while there: `_select_next_app` ran only while `current_app` was unset, so with several apps installed the first was picked once and shown forever while the rest were rendered on schedule and never displayed. And `enable_scrolling` was missing, so multi-frame apps were called once per rotation slot and never advanced past frame one. **GET /api/v3/display/modes.** Every mode that can be requested on-demand, with the plugin that owns it. Nothing exposed this, so anything driving the display from outside the web UI read each plugin's manifest.json off disk and reimplemented PluginManager's fallbacks. It also triggers discovery, which is otherwise lazy and normally happens because a person opened the dashboard. **integrations/mqtt_bridge.** Home Assistant control over MQTT Discovery: a mode select, a stop button, power, brightness. Rewritten against the API rather than the filesystem, so it needs no read access to config.json and cannot drift from the web UI. paho-mqtt 2.x VERSION2, TLS, an availability topic that is also the last will, and secrets from the environment. **Two opt-in extras.** A DNS single-request unit, for glibc's parallel A/AAAA lookup stalling ~5s per name on routers that answer only the A query -- which makes any plugin calling an external API slow and Starlark apps, which have a render timeout, fail outright. And a Pixlet config editor: a script you run and Ctrl+C rather than the third-party version's always-on unauthenticated Flask service, since it stops the display for the length of a session. Neither is installed by default. Long Starlark app names now wrap instead of overflowing their card. 115 new tests across 5 files. Also unblocked test_starlark_display_contract.py, which was silently skipping wherever fcntl is absent. Whole suite: no new failures against main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(mqtt_bridge): the five issues Codacy flagged on this branch All in the new bridge, all real: * requests floor was 2.31.0, which carries CVE-2024-35195, CVE-2024-47081 and CVE-2026-25645. Raised to >=2.33.0,<3.0.0, which is what the project's own requirements.txt already pins. * `import time` was never used. * `"mqtt_password": None` in DEFAULTS read as a hardcoded credential. It is the "no password configured" default; marked nosec B105, the convention used elsewhere in the repo. Also dropped an unused `build_app` from the display-modes test imports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: the review findings on this PR Nine of CodeRabbit's ten, plus the CodeQL alert. The tenth is wrong and is answered below. **One bad config section blanked the whole mode list.** `/display/modes` read `full_config.get(plugin_id, {}).get('enabled')`, so a non-dict under a plugin id -- a shape DisplayController already guards, so it happens -- raised AttributeError mid-loop and answered 500 with no modes at all. Every MQTT bridge entity is built from that list. Now skipped with a warning. **The DNS scripts reported success they had not earned.** Three separate paths: `resolvconf -u` failing was swallowed by `|| true`; the systemd-resolved branch exited 0 without applying anything, so the oneshot unit recorded success while the workaround was inactive; and the installer's `|| echo` turned a failed start into "installation complete." with exit 0. All three now fail loudly. `single-request` is a glibc resolv.conf option with no resolved.conf equivalent, so on those hosts the honest answer is that it cannot be applied. A NetworkManager-generated resolv.conf is regenerated on connection changes, not only at boot, and the unit is oneshot with RemainAfterExit -- so the option can vanish mid-boot with nothing to put it back. Now detected and stated plainly rather than implied to be permanent. **`Before=` does not order a manual restart.** It only orders units already in the same transaction, so `systemctl restart ledmatrix` could bypass the fix. install_dns_fix.sh now writes a ledmatrix.service drop-in with Wants= and After=. Wants=, not Requires=: a DNS workaround failing should not stop the display. **The Pixlet editor's `--lan` is gone.** `pixlet serve` has no authentication, and a printed warning is not access control. Loopback only, with the SSH port-forward in the header where the flag used to be documented -- SSH does the authenticating and nothing is left listening. **The MQTT example config now defaults to TLS** on 8883. The installer copies it verbatim, and without TLS the broker password and every command cross the network in cleartext. A plaintext broker is still supported and documented, and the bridge warns once at startup when a password is configured without TLS. **Not taken: "the upstream Pixlet CLI has no `schema` subcommand."** Upstream tidbyt/pixlet has none, but `scripts/download_pixlet.sh` installs `tronbyt/pixlet`, whose `cmd/schema.go` is `schema [PATH]` -> JSON on stdout, built on `runtime.NewAppletFromPath`, so it does execute `get_schema()`. That is exactly what extract_schema_via_pixlet calls. A binary without the subcommand exits non-zero and falls back to the source parser, which is already covered by a test. **CodeQL stack-trace exposure: not taken either.** I removed `details` first and that broke test_web_error_detail.py::test_no_api_v3_handler_discards_its_exception, which enforces `describe_exception` across all ~75 handlers -- written because a device with failing storage answered "see logs for details" from the log viewer itself. describe_exception redacts credentials; the trade-off is the project's and is already made. Restored, with the reasoning in a comment. 11 new tests. Whole suite: no new failures against main, 4127 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Chuck
@@ -194,6 +194,15 @@ class StarlarkAppsPlugin(BasePlugin):
|
||||
Each installed app becomes a dynamic display mode.
|
||||
"""
|
||||
|
||||
#: Starlark apps are animations: a .webp render carries per-frame delays
|
||||
#: and _display_frame advances at most one frame per call. The controller
|
||||
#: reads this attribute to decide whether a mode needs its high-FPS loop;
|
||||
#: without it display() was called once per rotation slot, so a multi-frame
|
||||
#: app showed a single frame and never moved. static-image is force-run at
|
||||
#: high FPS for the same reason (GIFs), but that plugin is special-cased by
|
||||
#: name in the controller and this one has to declare it.
|
||||
enable_scrolling = True
|
||||
|
||||
def __init__(self, plugin_id: str, config: Dict[str, Any],
|
||||
display_manager, cache_manager, plugin_manager):
|
||||
"""Initialize the Starlark Apps plugin."""
|
||||
@@ -681,13 +690,20 @@ class StarlarkAppsPlugin(BasePlugin):
|
||||
if app.is_enabled() and app.should_render(current_time):
|
||||
self._render_app(app, force=False)
|
||||
|
||||
def display(self, force_clear: bool = False) -> bool:
|
||||
def display(self, display_mode: Optional[str] = None, force_clear: bool = False) -> bool:
|
||||
"""
|
||||
Display current Starlark app.
|
||||
|
||||
This method is called during the display rotation.
|
||||
Displays frames from the currently active app.
|
||||
|
||||
`display_mode` names the app to show when it matches an installed
|
||||
app_id. The controller passes the mode it is rotating to and inspects
|
||||
this signature to decide whether to, so accepting it is what lets a
|
||||
specific app be addressed -- including by an on-demand request pinned
|
||||
to one app. Anything else (the plugin id itself, when the plugin
|
||||
exposes no per-app modes) falls through to normal rotation.
|
||||
|
||||
Returns False when there is no app to show -- which is the state of
|
||||
every install without Pixlet, and of a fresh one before any app is
|
||||
added. The display controller only skips a mode on a boolean False
|
||||
@@ -698,8 +714,15 @@ class StarlarkAppsPlugin(BasePlugin):
|
||||
if force_clear:
|
||||
self.display_manager.clear()
|
||||
|
||||
# If no current app, try to select one
|
||||
if not self.current_app:
|
||||
if display_mode and display_mode in self.apps:
|
||||
self.current_app = self.apps[display_mode]
|
||||
elif force_clear or not self.current_app:
|
||||
# Advance on entry to the mode. _select_next_app only ran when
|
||||
# current_app was unset, so the first enabled app was picked
|
||||
# once and then shown forever -- every other installed app was
|
||||
# rendered on schedule and never displayed. force_clear is the
|
||||
# controller's "we just switched to you" signal (it is reset
|
||||
# immediately after this call), so one app gets each turn.
|
||||
self._select_next_app()
|
||||
|
||||
if not self.current_app:
|
||||
|
||||
@@ -264,10 +264,18 @@ class PixletRenderer:
|
||||
else:
|
||||
value_str = str(value)
|
||||
|
||||
# Validate value doesn't contain dangerous shell metacharacters
|
||||
# Block: backticks, $(), pipes, redirects, semicolons, ampersands, null bytes
|
||||
# Allow: most printable chars including spaces, quotes, brackets, braces
|
||||
if re.search(r'[`$|<>&;\x00]|\$\(', value_str):
|
||||
# Validate value doesn't contain dangerous shell metacharacters.
|
||||
# Kept as defence in depth only: cmd is a list and there is no
|
||||
# shell=True below, so nothing here is ever interpreted by a
|
||||
# shell. That made the list worth trimming rather than growing
|
||||
# -- "|" is a normal character inside a config value, and apps
|
||||
# do use it as a separator (a PennDOT sign id is
|
||||
# "I-476 North|175659"). Blocking it dropped the whole key
|
||||
# silently, and the app then rendered its own "not configured"
|
||||
# screen with nothing to say why.
|
||||
# Block: backticks, $(), redirects, semicolons, ampersands, null bytes
|
||||
# Allow: most printable chars including spaces, quotes, brackets, braces, pipes
|
||||
if re.search(r'[`$<>&;\x00]|\$\(', value_str):
|
||||
logger.warning(f"Skipping config value with unsafe shell characters for key {key}: {value_str}")
|
||||
continue
|
||||
|
||||
@@ -299,13 +307,21 @@ class PixletRenderer:
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
if os.path.isfile(output_path):
|
||||
logger.debug(f"Successfully rendered: {star_file} -> {output_path}")
|
||||
return True, None
|
||||
else:
|
||||
if not os.path.isfile(output_path):
|
||||
error = "Rendering succeeded but output file not found"
|
||||
logger.error(error)
|
||||
return False, error
|
||||
# Pixlet exits 0 and writes a 0-byte file when the app renders
|
||||
# nothing -- an app whose config leaves it with no content to
|
||||
# show does exactly that. Treating existence alone as success
|
||||
# handed the caller a file with no frames in it, which reads
|
||||
# downstream as a working app that draws a black panel.
|
||||
if os.path.getsize(output_path) == 0:
|
||||
error = "Rendering produced an empty (0-byte) file - the app rendered no content"
|
||||
logger.error(error)
|
||||
return False, error
|
||||
logger.debug(f"Successfully rendered: {star_file} -> {output_path}")
|
||||
return True, None
|
||||
else:
|
||||
error = f"Pixlet failed (exit {result.returncode}): {result.stderr}"
|
||||
logger.error(error)
|
||||
@@ -319,11 +335,76 @@ class PixletRenderer:
|
||||
logger.exception("Rendering exception")
|
||||
return False, "Rendering failed - see logs for details"
|
||||
|
||||
#: Schema extraction runs an app's own get_schema(), which may make a
|
||||
#: network call. Short enough that a hung app does not stall an upload,
|
||||
#: long enough for a real API round trip on a slow connection.
|
||||
SCHEMA_TIMEOUT = 20
|
||||
|
||||
def extract_schema_via_pixlet(self, star_file: str) -> Optional[Dict[str, Any]]:
|
||||
"""Ask Pixlet itself for the app's schema, or None if it cannot say.
|
||||
|
||||
`pixlet schema` executes get_schema() instead of reading it, which is
|
||||
the only way to see options an app computes at runtime -- a dropdown
|
||||
whose choices come from a live API call has no option list anywhere in
|
||||
the source for the regex parser below to find, so that parser reports
|
||||
an empty dropdown and the config form offers nothing to pick.
|
||||
|
||||
Pixlet's own field keys are remapped to the ones the rest of this
|
||||
plugin and the config UI already use ("typeOf"/"desc"), so the two
|
||||
extractors return the same shape and callers cannot tell them apart.
|
||||
"""
|
||||
if not self.pixlet_binary:
|
||||
return None
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[self.pixlet_binary, "schema", star_file],
|
||||
capture_output=True, text=True, timeout=self.SCHEMA_TIMEOUT,
|
||||
cwd=self._get_safe_working_directory(star_file),
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning(
|
||||
"pixlet schema timed out after %ss for %s - get_schema() may be "
|
||||
"making a slow network call", self.SCHEMA_TIMEOUT, star_file)
|
||||
return None
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning(f"Could not run pixlet schema for {star_file}: {e}")
|
||||
return None
|
||||
|
||||
if result.returncode != 0:
|
||||
# Not an error worth failing on: older Pixlet builds have no
|
||||
# `schema` subcommand at all, and the source parser still works.
|
||||
logger.debug(
|
||||
"pixlet schema exited %d for %s: %s",
|
||||
result.returncode, star_file, (result.stderr or '').strip()[:300])
|
||||
return None
|
||||
|
||||
try:
|
||||
schema = json.loads(result.stdout)
|
||||
except (json.JSONDecodeError, ValueError) as e:
|
||||
logger.warning(f"pixlet schema returned unparseable output for {star_file}: {e}")
|
||||
return None
|
||||
|
||||
if not isinstance(schema, dict) or not isinstance(schema.get("schema"), list):
|
||||
logger.warning(f"pixlet schema returned an unexpected shape for {star_file}")
|
||||
return None
|
||||
|
||||
for field in schema["schema"]:
|
||||
if not isinstance(field, dict):
|
||||
continue
|
||||
if "type" in field and "typeOf" not in field:
|
||||
field["typeOf"] = field.pop("type")
|
||||
if "description" in field and "desc" not in field:
|
||||
field["desc"] = field.pop("description")
|
||||
return schema
|
||||
|
||||
def extract_schema(self, star_file: str) -> Tuple[bool, Optional[Dict[str, Any]], Optional[str]]:
|
||||
"""
|
||||
Extract configuration schema from a .star file by parsing source code.
|
||||
Extract configuration schema from a .star file.
|
||||
|
||||
Supports:
|
||||
Prefers `pixlet schema`, which runs the app and therefore sees options
|
||||
it computes at runtime. Falls back to parsing the source when Pixlet is
|
||||
unavailable, too old to have the subcommand, or the app fails to run --
|
||||
that parser handles:
|
||||
- Static field definitions (location, text, toggle, dropdown, color, datetime)
|
||||
- Variable-referenced dropdown options
|
||||
- Graceful degradation for unsupported field types
|
||||
@@ -337,6 +418,13 @@ class PixletRenderer:
|
||||
if not os.path.isfile(star_file):
|
||||
return False, None, f"Star file not found: {star_file}"
|
||||
|
||||
schema = self.extract_schema_via_pixlet(star_file)
|
||||
if schema is not None:
|
||||
logger.debug(
|
||||
"Extracted schema with %d field(s) from %s via pixlet schema",
|
||||
len(schema.get('schema', [])), star_file)
|
||||
return True, schema, None
|
||||
|
||||
try:
|
||||
# Read .star file
|
||||
with open(star_file, 'r', encoding='utf-8') as f:
|
||||
|
||||
Reference in New Issue
Block a user