feat(web): weekly automatic updates with health check and rollback (#581)

* feat(web): weekly automatic updates with health check and rollback

A General-tab toggle (off by default) checks for and installs LEDMatrix and
plugin updates once a week, overnight in the configured timezone.

- Pre-update checks skip (and report) instead of forcing: local edits or
  commits, merge/live rebase, no upstream, low disk, missing health check, or
  a version that was already rolled back. An abandoned rebase (HEAD back on a
  branch) is cleared, since it would otherwise block every pull.
- The pull reuses the Update Code path (now perform_core_update(), which
  reports dependency install failures as data).
- ledmatrix-update-verify.service, started via a .path unit from a request
  file, restarts the services from its own cgroup, requires them to come up
  and stay up, and otherwise resets to the previous commit and reinstalls the
  previous requirements. It runs a copy of the checker taken before the pull.
- No SSH needed: switching the toggle on restarts the display service, which
  (as root) installs the two units from the repo templates for the web user.
  first_time_install.sh installs them too and takes --enable-auto-update /
  LEDMATRIX_AUTO_UPDATE (passed through by one-shot-install.sh).
- Plugins update after the code passes its check; failures, blocks and
  rollbacks raise an Overview banner and show under the toggle.

Tested end to end on a Pi: web-UI setup, a good update, a broken web service
and a broken display (both rolled back), a blocked local edit, and an
abandoned rebase found on the device.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(auto-update): address static-analysis findings

- Replace the subprocess.CompletedProcess the verifier fabricated for a
  command that could not start with a plain namedtuple; nothing is executed
  there, but the scanner flags any CompletedProcess built from variables.
- Mark the subprocess imports with the repo's standard B404 annotation (all
  calls are list-form argv, no shell).
- Mark the rollback-failed message as not SQL (B608 matched its wording).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): CI failures on Linux

- Keep the setup result when chown fails. CI runs as a non-root user, where
  chown to the web user raises; that discarded the result file, so the
  General tab would never learn whether setup worked. Regression test added.
- Register the two new /api/v3/system/auto-update routes in the URL map
  snapshot.
- Use utility classes app.css defines (space-y-1, hover:text-red-600).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): address review feedback

- Health check: a failed restart command no longer lets the check run
  against the still-running old process; it counts as a failure (and after a
  rollback, as a failed rollback). An unreadable restart count is never
  treated as stable, since a crash loop looks healthy between attempts.
- Installer writes the auto_update setting to a temp file and swaps it in,
  keeping mode and owner, so a running config watcher never reads a
  truncated config.json.
- Verify unit quotes its command-line paths (install folders with spaces);
  setup refuses folder names systemd would reinterpret (%, quotes,
  backslashes, control characters) and says so on the General tab.
- The auto-update status route no longer returns exception text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): keep error detail in the status route's 500

test_web_error_detail requires every 5xx handler to log the traceback and
return describe_exception(e), which redacts credentials, so failures are
diagnosable from the web UI. Dropping it for CodeQL broke that policy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): dismiss route rejects non-object JSON with 400

A JSON array or scalar body made `.get('alert_id')` raise, returning 500.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): let the app-wide handler answer status-route errors

CodeQL (py/stack-trace-exposure, #709) flagged the route's own except,
which returned describe_exception(e). web_interface/app.py's error handler
already logs the traceback and returns the same redacted detail for any
unhandled exception, so the local copy is removed: same response, no new
exception-to-response flow, and test_web_error_detail's policy still holds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-15 10:58:57 -04:00
committed by GitHub
co-authored by Claude Opus 5
parent d01da3bd9f
commit 869e36fb2f
25 changed files with 2938 additions and 192 deletions
+54 -3
View File
@@ -220,6 +220,8 @@ SKIP_PERF=${LEDMATRIX_SKIP_PERF:-0}
SKIP_REBOOT_PROMPT=${LEDMATRIX_SKIP_REBOOT_PROMPT:-0}
SKIP_SWAP=${LEDMATRIX_SKIP_SWAP:-0}
BUILD_JOBS_OVERRIDE=${LEDMATRIX_BUILD_JOBS:-}
# Weekly automatic updates: 1 on, 0 off, empty = ask (interactive) or leave as is.
AUTO_UPDATE=${LEDMATRIX_AUTO_UPDATE:-}
usage() {
cat <<USAGE
@@ -234,12 +236,15 @@ Options:
--skip-swap Never add temporary swap for the C++ build
--build-jobs N Compile the C++ library with N parallel jobs
(default: scaled to available RAM)
--enable-auto-update Turn on weekly automatic updates (with health
check and automatic rollback)
--no-auto-update Leave weekly automatic updates off
-h, --help Show this help message and exit
Environment variables (same effect as flags):
LEDMATRIX_ASSUME_YES=1, RPI_RGB_FORCE_REBUILD=1, LEDMATRIX_SKIP_SOUND=1,
LEDMATRIX_SKIP_PERF=1, LEDMATRIX_SKIP_REBOOT_PROMPT=1,
LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N
LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N, LEDMATRIX_AUTO_UPDATE=1|0
Low-memory devices:
On a Pi with under 2GB of RAM the C++ build is limited to fewer parallel
@@ -257,6 +262,8 @@ while [ $# -gt 0 ]; do
--skip-perf) SKIP_PERF=1 ;;
--no-reboot-prompt) SKIP_REBOOT_PROMPT=1 ;;
--skip-swap) SKIP_SWAP=1 ;;
--enable-auto-update) AUTO_UPDATE=1 ;;
--no-auto-update) AUTO_UPDATE=0 ;;
--build-jobs)
shift
if [ $# -eq 0 ]; then echo "--build-jobs requires a number"; usage; exit 1; fi
@@ -863,6 +870,50 @@ else
echo "✓ Main config file already exists"
fi
# Weekly automatic updates (General tab -> Automatic Updates). Off unless asked
# for: --enable-auto-update / LEDMATRIX_AUTO_UPDATE=1, or "y" at the prompt when
# installing interactively. Only an explicit choice changes the setting, so
# re-running the installer with -y never switches it silently.
if [ -z "$AUTO_UPDATE" ] && [ "$ASSUME_YES" != "1" ] && [ -t 0 ]; then
read -p "Automatically check for and install LEDMatrix updates once a week, with automatic rollback if an update breaks something? (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then AUTO_UPDATE=1; else AUTO_UPDATE=0; fi
fi
if [ "$AUTO_UPDATE" = "1" ] || [ "$AUTO_UPDATE" = "0" ]; then
if python3 - "$PROJECT_ROOT_DIR/config/config.json" "$AUTO_UPDATE" <<'PY'
import json, os, sys, tempfile
path, enabled = sys.argv[1], sys.argv[2] == "1"
with open(path, encoding="utf-8") as f:
config = json.load(f)
if not isinstance(config.get("auto_update"), dict):
config["auto_update"] = {}
config["auto_update"]["enabled"] = enabled
# Written beside the original and swapped in whole: the display service's
# config watcher may be running and must never read a half-written file.
original = os.stat(path)
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(path)), prefix=".config.")
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
json.dump(config, f, indent=4)
f.write("\n")
f.flush()
os.fsync(f.fileno())
os.chmod(tmp, original.st_mode & 0o777)
if hasattr(os, "chown"):
os.chown(tmp, original.st_uid, original.st_gid)
os.replace(tmp, path)
except BaseException:
if os.path.exists(tmp):
os.unlink(tmp)
raise
PY
then
if [ "$AUTO_UPDATE" = "1" ]; then echo "✓ Weekly automatic updates enabled"; else echo "✓ Weekly automatic updates off"; fi
else
echo "⚠ Could not set auto_update in config/config.json; turn it on from the General tab instead"
fi
fi
# Create config_secrets.json from template if missing
if [ ! -f "$PROJECT_ROOT_DIR/config/config_secrets.json" ]; then
if [ -f "$PROJECT_ROOT_DIR/config/config_secrets.template.json" ]; then
@@ -1341,7 +1392,7 @@ if [ -f "$PROJECT_ROOT_DIR/scripts/install/install_web_service.sh" ]; then
fi
fi
if [ ! -f "/etc/systemd/system/ledmatrix-web.service" ] || [ "$NEEDS_UPDATE" = true ]; then
if [ ! -f "/etc/systemd/system/ledmatrix-web.service" ] || [ ! -f "/etc/systemd/system/ledmatrix-update-verify.path" ] || [ "$NEEDS_UPDATE" = true ]; then
bash "$PROJECT_ROOT_DIR/scripts/install/install_web_service.sh"
# Ensure systemd sees any new/changed unit files
systemctl daemon-reload || true
@@ -1357,7 +1408,7 @@ echo ""
CURRENT_STEP="Harden systemd unit file permissions"
echo "Step 8.1: Setting systemd unit file permissions..."
echo "-----------------------------------------------"
for unit in "/etc/systemd/system/ledmatrix.service" "/etc/systemd/system/ledmatrix-web.service" "/etc/systemd/system/ledmatrix-wifi-monitor.service"; do
for unit in "/etc/systemd/system/ledmatrix.service" "/etc/systemd/system/ledmatrix-web.service" "/etc/systemd/system/ledmatrix-wifi-monitor.service" "/etc/systemd/system/ledmatrix-update-verify.service" "/etc/systemd/system/ledmatrix-update-verify.path"; do
if [ -f "$unit" ]; then
chown root:root "$unit" || true
chmod 644 "$unit" || true