feat(starlark,on-demand): the third-party fixes worth taking, plus a Home Assistant MQTT bridge (#538)

* feat(starlark,on-demand): the third-party fixes worth taking, plus an MQTT bridge

Analysis of ant456/ledmatrix-fixes-repo, a third-party collection of
patches and services built while running this project on Starlark apps
under MQTT control. Its patches are whole-file copies taken against an
older tree, so applying them as written would revert #523's frame
pacing, #534's display() bool returns and the GitHub token masking in
plugins_manager.js. Three of its claimed fixes are already in main, and
its api_v3 Starlark routes are #535's. What follows is the rest --
verified against current code, and reimplemented where the patch's
approach did not hold up.

**On-demand display.** `pinned` reached the controller from the API, was
stored on it and republished in the status payload, but never narrowed
the rotation -- a pinned request still cycled every mode its plugin
owns. Right for a sports plugin, whose modes are views of one subject;
wrong for a plugin whose modes are unrelated, which is every Starlark
app. Now honoured, and it survives a restart.

Restarting while on-demand was active loaded *only* the on-demand
plugin, so normal rotation had nothing to return to for the life of the
process -- and a restart mid-session is routine, since that is how an
update is applied. The panel came back cycling one plugin's modes with
no way out but clearing the cache by hand. Every enabled plugin loads
now; on-demand still resumes on its saved mode.

Stop requests are exempt from the duplicate guards on purpose, so that a
second click stops a mode a race left running -- which means consuming
the mailbox is the only thing that ends one. It was never consumed, so
the same stop was re-read and re-processed on every poll, forever. Both
paths now share one compare-before-delete helper.

**Starlark rendering.** `extract_schema` parsed the source with a regex,
which can only see option lists written out literally: an app whose
dropdown is filled from a live API call inside `get_schema()` came back
empty, and the config form offered nothing to pick. Now runs `pixlet
schema`, which executes the app, and falls back to the parser when
Pixlet is absent, too old for the subcommand, or the app fails to run.
The third-party patch replaced the parser outright and hardcoded
/usr/local/bin/pixlet; this keeps the fallback and the binary search.

A `|` in a config value was dropped by a shell-metacharacter filter,
though the command is a list with no shell involved -- and apps do use
it as a separator inside one value. The key went missing silently and
the app rendered its own "not configured" screen with nothing to say
why. And a 0-byte render was reported as success: Pixlet exits 0 and
writes nothing when an app has no content, which read downstream as a
working app drawing a black panel.

**Starlark display.** `display()` ignored the mode it was called with,
so a specific app could not be addressed. It now accepts `display_mode`
-- which is the whole mechanism, since the controller inspects the
signature before passing it. Found while there: `_select_next_app` ran
only while `current_app` was unset, so with several apps installed the
first was picked once and shown forever while the rest were rendered on
schedule and never displayed. And `enable_scrolling` was missing, so
multi-frame apps were called once per rotation slot and never advanced
past frame one.

**GET /api/v3/display/modes.** Every mode that can be requested
on-demand, with the plugin that owns it. Nothing exposed this, so
anything driving the display from outside the web UI read each plugin's
manifest.json off disk and reimplemented PluginManager's fallbacks. It
also triggers discovery, which is otherwise lazy and normally happens
because a person opened the dashboard.

**integrations/mqtt_bridge.** Home Assistant control over MQTT
Discovery: a mode select, a stop button, power, brightness. Rewritten
against the API rather than the filesystem, so it needs no read access
to config.json and cannot drift from the web UI. paho-mqtt 2.x
VERSION2, TLS, an availability topic that is also the last will, and
secrets from the environment.

**Two opt-in extras.** A DNS single-request unit, for glibc's parallel
A/AAAA lookup stalling ~5s per name on routers that answer only the A
query -- which makes any plugin calling an external API slow and
Starlark apps, which have a render timeout, fail outright. And a Pixlet
config editor: a script you run and Ctrl+C rather than the third-party
version's always-on unauthenticated Flask service, since it stops the
display for the length of a session. Neither is installed by default.

Long Starlark app names now wrap instead of overflowing their card.

115 new tests across 5 files. Also unblocked
test_starlark_display_contract.py, which was silently skipping wherever
fcntl is absent. Whole suite: no new failures against main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(mqtt_bridge): the five issues Codacy flagged on this branch

All in the new bridge, all real:

  * requests floor was 2.31.0, which carries CVE-2024-35195,
    CVE-2024-47081 and CVE-2026-25645. Raised to >=2.33.0,<3.0.0, which
    is what the project's own requirements.txt already pins.
  * `import time` was never used.
  * `"mqtt_password": None` in DEFAULTS read as a hardcoded credential.
    It is the "no password configured" default; marked nosec B105, the
    convention used elsewhere in the repo.

Also dropped an unused `build_app` from the display-modes test imports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: the review findings on this PR

Nine of CodeRabbit's ten, plus the CodeQL alert. The tenth is wrong and
is answered below.

**One bad config section blanked the whole mode list.**
`/display/modes` read `full_config.get(plugin_id, {}).get('enabled')`,
so a non-dict under a plugin id -- a shape DisplayController already
guards, so it happens -- raised AttributeError mid-loop and answered 500
with no modes at all. Every MQTT bridge entity is built from that list.
Now skipped with a warning.

**The DNS scripts reported success they had not earned.** Three separate
paths: `resolvconf -u` failing was swallowed by `|| true`; the
systemd-resolved branch exited 0 without applying anything, so the
oneshot unit recorded success while the workaround was inactive; and the
installer's `|| echo` turned a failed start into "installation
complete." with exit 0. All three now fail loudly. `single-request` is a
glibc resolv.conf option with no resolved.conf equivalent, so on those
hosts the honest answer is that it cannot be applied.

A NetworkManager-generated resolv.conf is regenerated on connection
changes, not only at boot, and the unit is oneshot with RemainAfterExit
-- so the option can vanish mid-boot with nothing to put it back. Now
detected and stated plainly rather than implied to be permanent.

**`Before=` does not order a manual restart.** It only orders units
already in the same transaction, so `systemctl restart ledmatrix` could
bypass the fix. install_dns_fix.sh now writes a ledmatrix.service
drop-in with Wants= and After=. Wants=, not Requires=: a DNS workaround
failing should not stop the display.

**The Pixlet editor's `--lan` is gone.** `pixlet serve` has no
authentication, and a printed warning is not access control. Loopback
only, with the SSH port-forward in the header where the flag used to be
documented -- SSH does the authenticating and nothing is left listening.

**The MQTT example config now defaults to TLS** on 8883. The installer
copies it verbatim, and without TLS the broker password and every
command cross the network in cleartext. A plaintext broker is still
supported and documented, and the bridge warns once at startup when a
password is configured without TLS.

**Not taken: "the upstream Pixlet CLI has no `schema` subcommand."**
Upstream tidbyt/pixlet has none, but `scripts/download_pixlet.sh`
installs `tronbyt/pixlet`, whose `cmd/schema.go` is
`schema [PATH]` -> JSON on stdout, built on
`runtime.NewAppletFromPath`, so it does execute `get_schema()`. That is
exactly what extract_schema_via_pixlet calls. A binary without the
subcommand exits non-zero and falls back to the source parser, which is
already covered by a test.

**CodeQL stack-trace exposure: not taken either.** I removed `details`
first and that broke
test_web_error_detail.py::test_no_api_v3_handler_discards_its_exception,
which enforces `describe_exception` across all ~75 handlers -- written
because a device with failing storage answered "see logs for details"
from the log viewer itself. describe_exception redacts credentials; the
trade-off is the project's and is already made. Restored, with the
reasoning in a comment.

11 new tests. Whole suite: no new failures against main, 4127 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

scripts/utils/README.md

@@ -9,6 +9,8 @@ This directory contains utility scripts for maintenance and system operations.
 - **`wifi_monitor_daemon.py`** - Background daemon that monitors WiFi/Ethernet connection and manages access point mode
 - **`cleanup_venv.sh`** - Cleans up Python virtual environment files
 - **`clear_python_cache.sh`** - Clears Python cache files (__pycache__, *.pyc, etc.)
+- **`pixlet_config_editor.sh`** - Opens Pixlet's own config UI for one installed Starlark app
+- **`apply_dns_single_request.sh`** - Adds `options single-request` to the resolver (run by `ledmatrix-dns-fix.service`)
 
 ## Usage
 
@@ -25,3 +27,31 @@ This script is typically called by the systemd service (`ledmatrix-web.service`)
 ### WiFi Monitor Daemon
 This daemon is typically run as a systemd service (`ledmatrix-wifi-monitor.service`) and automatically manages WiFi access point mode based on network connectivity.
 
+
+### Pixlet Config Editor
+Run it when you want Pixlet's own config form for a Starlark app -- live
+render preview, cascading dropdowns -- rather than the LEDMatrix one.
+
+```bash
+./scripts/utils/pixlet_config_editor.sh                 # list installed apps
+./scripts/utils/pixlet_config_editor.sh penndot_signs   # edit, on localhost:8080
+```
+
+Deliberately not a service. It stops the display for the length of the
+session and `pixlet serve` listens with no authentication, so it should only
+be running while you are actually editing. It backs the config up first and
+restarts the display on exit, however it exits.
+
+It binds loopback only, with no flag to change that: anything that can reach
+`pixlet serve` can rewrite the app's config, and a printed warning is not
+access control. To edit from another machine, forward the port -- SSH does the
+authenticating and nothing is left listening on the LAN:
+
+```bash
+ssh -L 8080:localhost:8080 pi@ledpi.local
+```
+
+### Apply DNS Single-Request Fix
+Installed and run by `ledmatrix-dns-fix.service`; see `systemd/README.md`.
+Safe to run by hand (`sudo ./scripts/utils/apply_dns_single_request.sh`) and
+idempotent.

scripts/utils/apply_dns_single_request.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+#
+# Add `options single-request` to the system resolver configuration.
+#
+# glibc's getaddrinfo() sends the A and AAAA queries for a name in
+# parallel on one socket. Some routers answer the A query and drop the
+# AAAA one, so the resolver waits out its full timeout -- about five
+# seconds -- before returning an address that was already available.
+# Disabling IPv6 in the kernel does not help: the resolver still asks.
+#
+# `single-request` makes it send the two queries one after the other,
+# which those routers answer correctly. Anything on the matrix that
+# calls an external API pays that five seconds per lookup otherwise, and
+# a Starlark app with a render timeout will simply fail instead.
+#
+# Idempotent, and safe to run on a machine that does not need it. Run by
+# ledmatrix-dns-fix.service on every boot, because whatever manages
+# resolv.conf regenerates it and drops the option again.
+#
+# Usage: sudo ./scripts/utils/apply_dns_single_request.sh
+
+set -eu
+
+OPTION="options single-request"
+RESOLVCONF_TAIL="/etc/resolvconf/resolv.conf.d/tail"
+RESOLV_CONF="/etc/resolv.conf"
+
+log() { echo "[dns-single-request] $*"; }
+
+already_applied() {
+    grep -qs "^${OPTION}\$" "$1"
+}
+
+# resolvconf regenerates /etc/resolv.conf from these fragments, so the
+# tail file is the only place an addition survives. Prefer it when the
+# directory exists, whether or not resolvconf has run yet.
+if [ -d "$(dirname "$RESOLVCONF_TAIL")" ]; then
+    if already_applied "$RESOLVCONF_TAIL"; then
+        log "already present in $RESOLVCONF_TAIL"
+    else
+        echo "$OPTION" >> "$RESOLVCONF_TAIL"
+        log "added to $RESOLVCONF_TAIL"
+    fi
+    # Only a missing resolvconf is ignorable. If it is present and the
+    # regeneration fails, /etc/resolv.conf still lacks the option, and
+    # reporting success would be a lie.
+    if command -v resolvconf >/dev/null 2>&1; then
+        if ! resolvconf -u; then
+            log "resolvconf -u failed; $RESOLV_CONF was not regenerated"
+            exit 1
+        fi
+    fi
+fi
+
+# systemd-resolved owns its stub file and rewrites anything appended to it,
+# and `single-request` is a glibc resolv.conf option with no resolved.conf
+# equivalent -- so there is nothing this script can do here. Exit non-zero:
+# the unit would otherwise record success while the workaround is inactive,
+# which is the failure mode this whole script exists to avoid.
+if [ -L "$RESOLV_CONF" ] && readlink -f "$RESOLV_CONF" | grep -q "systemd"; then
+    log "$RESOLV_CONF is managed by systemd-resolved."
+    log "'options single-request' is a glibc resolv.conf option and has no"
+    log "resolved.conf equivalent, so it cannot be applied on this host."
+    log "If external API calls are slow, the workaround is to stop using the"
+    log "systemd-resolved stub (see 'man systemd-resolved', NSS/resolv.conf modes)."
+    exit 1
+fi
+
+if already_applied "$RESOLV_CONF"; then
+    log "already present in $RESOLV_CONF"
+    exit 0
+fi
+
+if [ ! -w "$RESOLV_CONF" ] && [ -e "$RESOLV_CONF" ]; then
+    log "cannot write $RESOLV_CONF (run with sudo?)"
+    exit 1
+fi
+
+# A NetworkManager-generated resolv.conf is regenerated on every connection
+# change, not only at boot -- and this unit is oneshot with RemainAfterExit,
+# so it will not re-run within the same boot to put the option back. Say so
+# rather than implying the fix is permanent. Nothing is silently swallowed:
+# the append below still happens and still works until the next renewal.
+if grep -qs "Generated by NetworkManager" "$RESOLV_CONF" \
+        && [ ! -d "$(dirname "$RESOLVCONF_TAIL")" ]; then
+    log "NOTE: $RESOLV_CONF is generated by NetworkManager and has no"
+    log "resolvconf tail directory to write to. The option is being added, but"
+    log "NetworkManager will drop it on the next connection renewal, and this"
+    log "unit does not run again until the next boot. If lookups go slow again"
+    log "before a reboot, re-run this script."
+fi
+
+echo "$OPTION" >> "$RESOLV_CONF"
+log "added to $RESOLV_CONF"

scripts/utils/pixlet_config_editor.sh

@@ -0,0 +1,143 @@
+#!/bin/bash
+#
+# Edit an installed Starlark app's config in Pixlet's own config UI.
+#
+# `pixlet serve` runs the app for real, so its form has working cascading
+# dropdowns and option lists fetched live -- useful for an app whose choices
+# only exist at runtime, or when you want to see the render change as you
+# type. The LEDMatrix config form now reads the same runtime schema (see
+# PixletRenderer.extract_schema_via_pixlet), so reach for this when you want
+# Pixlet's live preview, not because the normal form is missing options.
+#
+# Deliberately a script you run and then Ctrl+C, not a service: it stops the
+# display for the length of the session, and `pixlet serve` listens on a port
+# with no authentication. Nothing here should be listening when you are not
+# actually editing.
+#
+# Usage:
+#   ./scripts/utils/pixlet_config_editor.sh            # list installed apps
+#   ./scripts/utils/pixlet_config_editor.sh <app_id>   # edit
+#
+# Binds loopback only, and there is deliberately no flag to change that:
+# `pixlet serve` has no authentication, and anything that can reach it can
+# rewrite the app's config. To edit from another machine, forward the port --
+# which authenticates as SSH and leaves nothing listening on the LAN:
+#
+#   ssh -L 8080:localhost:8080 pi@ledpi.local
+
+set -eu
+
+PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd)
+APPS_DIR="$PROJECT_ROOT_DIR/starlark-apps"
+PORT="${PIXLET_EDITOR_PORT:-8080}"
+# Loopback only. See the header: pixlet serve is unauthenticated.
+BIND_HOST="127.0.0.1"
+
+APP_ID="${1:-}"
+
+list_apps() {
+    if [ -d "$APPS_DIR" ]; then
+        find "$APPS_DIR" -maxdepth 1 -mindepth 1 -type d -printf '  %f\n' 2>/dev/null | sort
+    fi
+}
+
+if [ -z "$APP_ID" ]; then
+    echo "Usage: $0 <app_id>"
+    echo ""
+    echo "Installed apps:"
+    list_apps || true
+    [ -n "$(list_apps)" ] || echo "  (none found in $APPS_DIR)"
+    exit 1
+fi
+
+APP_DIR="$APPS_DIR/$APP_ID"
+if [ ! -d "$APP_DIR" ]; then
+    echo "No such app: $APP_ID"
+    echo ""
+    echo "Installed apps:"
+    list_apps
+    exit 1
+fi
+
+STAR_FILE=$(find "$APP_DIR" -maxdepth 1 -iname "*.star" | head -1)
+if [ -z "$STAR_FILE" ]; then
+    echo "No .star file found in $APP_DIR"
+    exit 1
+fi
+
+# Same search order the plugin itself uses: the bundled binary for this
+# architecture first, then PATH -- so this works on an install that never put
+# pixlet on PATH.
+find_pixlet() {
+    local arch bundled
+    case "$(uname -s)-$(uname -m)" in
+        Linux-aarch64|Linux-arm64)  arch="pixlet-linux-arm64" ;;
+        Linux-x86_64|Linux-amd64)   arch="pixlet-linux-amd64" ;;
+        Darwin-arm64)               arch="pixlet-darwin-arm64" ;;
+        Darwin-x86_64)              arch="pixlet-darwin-amd64" ;;
+        *)                          arch="" ;;
+    esac
+    bundled="$PROJECT_ROOT_DIR/bin/pixlet/$arch"
+    if [ -n "$arch" ] && [ -x "$bundled" ]; then
+        echo "$bundled"
+        return 0
+    fi
+    command -v pixlet 2>/dev/null || return 1
+}
+
+PIXLET_BIN=$(find_pixlet) || {
+    echo "Pixlet not found. Install it with:"
+    echo "  ./scripts/download_pixlet.sh"
+    exit 1
+}
+
+CONFIG_FILE="$APP_DIR/config.json"
+if [ -f "$CONFIG_FILE" ]; then
+    cp "$CONFIG_FILE" "$CONFIG_FILE.backup"
+    echo "Backed up existing config to $CONFIG_FILE.backup"
+else
+    echo "{}" > "$CONFIG_FILE"
+fi
+
+DISPLAY_WAS_RUNNING=false
+if systemctl is-active --quiet ledmatrix 2>/dev/null; then
+    DISPLAY_WAS_RUNNING=true
+fi
+
+# Restart the display however this exits -- Ctrl+C, an error, or pixlet
+# dying on its own. Leaving the panel dark because the editor crashed is the
+# failure worth guarding against.
+cleanup() {
+    echo ""
+    if [ "$DISPLAY_WAS_RUNNING" = true ]; then
+        echo "Restarting the display service..."
+        sudo systemctl restart ledmatrix || echo "⚠ Could not restart ledmatrix - do it by hand"
+    fi
+    echo "Your config as it was before this session: $CONFIG_FILE.backup"
+}
+trap cleanup EXIT INT TERM
+
+if [ "$DISPLAY_WAS_RUNNING" = true ]; then
+    echo "Stopping the display service so it does not read config.json mid-write..."
+    sudo systemctl stop ledmatrix
+fi
+
+echo ""
+echo "Editing:  $APP_ID"
+echo "App file: $STAR_FILE"
+echo "URL:      http://localhost:$PORT/"
+echo ""
+echo "Listening on localhost only -- pixlet serve has no authentication."
+echo "From another machine, forward the port:"
+echo "  ssh -L $PORT:localhost:$PORT $(whoami)@$(hostname)"
+echo ""
+echo "Changes save straight to the real config as you make them."
+echo "Press Ctrl+C when finished - the display restarts automatically."
+echo ""
+
+cd "$APP_DIR"
+"$PIXLET_BIN" serve "$(basename "$STAR_FILE")" \
+    --host "$BIND_HOST" \
+    --port "$PORT" \
+    --no-browser \
+    --saveconfig "$CONFIG_FILE"