mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-13 10:36:35 +00:00
Merge remote-tracking branch 'origin/main' into feat/common-sports-helpers
This commit is contained in:
Chuck
@@ -1,5 +1,8 @@
|
||||
{
|
||||
"web_display_autostart": true,
|
||||
"auto_update": {
|
||||
"enabled": false
|
||||
},
|
||||
"schedule": {
|
||||
"enabled": false,
|
||||
"mode": "per-day",
|
||||
|
||||
+46
-11
@@ -203,26 +203,61 @@ configured speed, so position stays proportional to real time.
|
||||
|
||||
## Diagnosing a juddery scroller
|
||||
|
||||
**`Avg FPS` will lie to you.** It is a 100-frame moving average, and a 2 ms
|
||||
duplicate frame plus a 21 ms double-wait average to exactly 10 ms. A ticker
|
||||
that is stalling on half its frames still reports a healthy `100.0`.
|
||||
**An average will lie to you.** A 2 ms duplicate frame and a 21 ms double-wait
|
||||
mean exactly 10 ms, so a ticker stalling on half its frames still averages to a
|
||||
healthy 100 fps. The stats line reports the tail for that reason — read the
|
||||
percentiles, not the fps.
|
||||
|
||||
Look at the **distribution** instead:
|
||||
Every scroller emits one line every 5 seconds covering *every* frame in that
|
||||
window, tagged with the plugin it came from:
|
||||
|
||||
```bash
|
||||
journalctl -u ledmatrix --since "-10min" --no-pager \
|
||||
| grep -oE "Frame time: [0-9.]+ms" | awk '{print $3}' | sed 's/ms//' \
|
||||
| awk '{printf "%.0f\n", $1}' | sort -n | uniq -c
|
||||
journalctl -u ledmatrix --since "-10min" --no-pager | grep "Scroll frame stats"
|
||||
```
|
||||
|
||||
```
|
||||
[Plugin: news] Scroll frame stats - 100.0 fps over 501 frames | median 10.00ms
|
||||
p95 10.11ms max 12.03ms min 7.98ms | stalls 0 (0.0%) skips 0 (0.0%)
|
||||
```
|
||||
|
||||
Reading it, on a 100 Hz panel:
|
||||
|
||||
| you see | it means |
|
||||
|---|---|
|
||||
| everything at 10 ms | healthy |
|
||||
| a mode at ~2 ms | **duplicate frames** — the swap was skipped because the image did not change. The scroller is advancing less than one pixel per frame. |
|
||||
| a mode at 20/30/50 ms | frames missing refreshes — per-frame work is overrunning, or a background thread is holding the GIL |
|
||||
| `Avg FPS` above 100 | duplicates present, unless the scroll cycle has completed and is idling |
|
||||
| median 10 ms, p95 within ~0.5 ms of it | healthy — locked to the panel |
|
||||
| p95 or max at 20/30/50 ms | frames missing refreshes — per-frame work is overrunning, or a background thread is holding the GIL |
|
||||
| non-zero **skips**, or a median *below* 10 ms | **duplicate frames** — the swap was skipped because the image did not change, so the frame never waited on vsync. The scroller is advancing less than one pixel per frame. |
|
||||
| non-zero **stalls** | frames past 1.5× the median, which is the measure of judder that survives averaging |
|
||||
|
||||
`stalls` and `skips` are both counted against that window's own median, so they
|
||||
stay meaningful on a panel running at any refresh rate.
|
||||
|
||||
To rank every scroller at once rather than reading lines one at a time:
|
||||
|
||||
```bash
|
||||
journalctl -u ledmatrix --since "-3h" --no-pager | grep "Scroll frame stats" \
|
||||
| sed -E 's/.*- (\S+) - (\[Plugin: [^]]+\] )?Scroll.*median ([0-9.]+)ms p95 ([0-9.]+)ms.*/\1 \3 \4/' \
|
||||
| awk '$2 < 1000 {n[$1]++; m[$1]+=$2; p[$1]+=$3} END {for (k in n)
|
||||
printf "%-28s %5d windows median %6.2fms p95 %6.2fms\n", k, n[k], m[k]/n[k], p[k]/n[k]}' \
|
||||
| sort -k7 -rn
|
||||
```
|
||||
|
||||
The `$2 < 1000` guard drops windows whose median is a whole second or more.
|
||||
Those are not frames. Until the idle-gap fix in `log_frame_rate()`, the first
|
||||
frame of every scroll was timed against the end of the *previous* scroll, so
|
||||
the gap between them was recorded as one enormous sample — it landed in the
|
||||
`max` field of otherwise healthy windows and counted as one stall per scroll,
|
||||
roughly 0.2% at 500 frames to a window, which is the same order as the real
|
||||
stall rates it sat beside. Current builds emit none, but the guard costs
|
||||
nothing and keeps the command honest against older journals.
|
||||
|
||||
A scroller whose p95 sits several times its median is the one to fix, and it is
|
||||
usually the one doing the most per-frame work rather than the one configured
|
||||
worst. Measured over 20 minutes with two scrollers set identically at 100 px/s,
|
||||
the leaderboard held 10 ms flat while the odds ticker spent ~20% of its frames
|
||||
on duplicates. Same settings, different render cost: odds does more per-frame
|
||||
work, and more variably, so it is first to land a frame that advances less than
|
||||
a whole pixel. Check the render path before the config.
|
||||
|
||||
Then confirm what the plugin actually loaded — config edits do not always reach
|
||||
the running code:
|
||||
|
||||
@@ -96,6 +96,33 @@ Configure basic system settings:
|
||||
- **Plugin System Settings** — including the `plugins_directory` (default
|
||||
`plugin-repos/`) used by the plugin loader
|
||||
- **Autostart** options for the display service
|
||||
- **Automatic updates** — once a week, update LEDMatrix and every installed
|
||||
plugin with a newer version. Off by default. Runs 2–5 AM local time when
|
||||
possible, otherwise within a day of being due. The last result and next
|
||||
check are shown under the toggle, and anything other than success raises a
|
||||
banner on **Overview**.
|
||||
- *Checks first:* the code update is skipped, with the reason shown, if
|
||||
tracked files were edited locally, the checkout has local commits, a
|
||||
rebase/merge is in progress, the branch has no upstream, less than 300 MB
|
||||
is free, or the newest version already failed once. A failed fetch is
|
||||
retried the next day.
|
||||
- *Health check and rollback:* after pulling, `ledmatrix-update-verify.service`
|
||||
restarts the services and checks that the web interface responds and the
|
||||
display (if it was running) stays up. If not — or if the new dependencies
|
||||
failed to install — it resets to the previous commit, reinstalls the
|
||||
previous dependencies and restarts again. A running display is restarted;
|
||||
a stopped one stays stopped.
|
||||
- *Plugins* update through the Plugin Store, which refuses versions that need
|
||||
a newer LEDMatrix and restores the old copy when an install fails. When the
|
||||
code changed, plugins wait until it passes its health check. Plugins are
|
||||
not health-checked after updating.
|
||||
- *Setup needs no SSH.* Turning the toggle on restarts the display service,
|
||||
which installs the health check (`ledmatrix-update-verify.path` and
|
||||
`.service`); the General tab shows when it is ready, or why setup failed.
|
||||
Until then only plugins update. New installs set it up during
|
||||
installation and can switch updates on with
|
||||
`first_time_install.sh --enable-auto-update` (or `LEDMATRIX_AUTO_UPDATE=1`,
|
||||
which `one-shot-install.sh` passes through), or at the installer's prompt.
|
||||
|
||||
Click **Save** to write changes to `config/config.json`. Most changes
|
||||
require a display service restart from **Overview**.
|
||||
|
||||
+54
-3
@@ -220,6 +220,8 @@ SKIP_PERF=${LEDMATRIX_SKIP_PERF:-0}
|
||||
SKIP_REBOOT_PROMPT=${LEDMATRIX_SKIP_REBOOT_PROMPT:-0}
|
||||
SKIP_SWAP=${LEDMATRIX_SKIP_SWAP:-0}
|
||||
BUILD_JOBS_OVERRIDE=${LEDMATRIX_BUILD_JOBS:-}
|
||||
# Weekly automatic updates: 1 on, 0 off, empty = ask (interactive) or leave as is.
|
||||
AUTO_UPDATE=${LEDMATRIX_AUTO_UPDATE:-}
|
||||
|
||||
usage() {
|
||||
cat <<USAGE
|
||||
@@ -234,12 +236,15 @@ Options:
|
||||
--skip-swap Never add temporary swap for the C++ build
|
||||
--build-jobs N Compile the C++ library with N parallel jobs
|
||||
(default: scaled to available RAM)
|
||||
--enable-auto-update Turn on weekly automatic updates (with health
|
||||
check and automatic rollback)
|
||||
--no-auto-update Leave weekly automatic updates off
|
||||
-h, --help Show this help message and exit
|
||||
|
||||
Environment variables (same effect as flags):
|
||||
LEDMATRIX_ASSUME_YES=1, RPI_RGB_FORCE_REBUILD=1, LEDMATRIX_SKIP_SOUND=1,
|
||||
LEDMATRIX_SKIP_PERF=1, LEDMATRIX_SKIP_REBOOT_PROMPT=1,
|
||||
LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N
|
||||
LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N, LEDMATRIX_AUTO_UPDATE=1|0
|
||||
|
||||
Low-memory devices:
|
||||
On a Pi with under 2GB of RAM the C++ build is limited to fewer parallel
|
||||
@@ -257,6 +262,8 @@ while [ $# -gt 0 ]; do
|
||||
--skip-perf) SKIP_PERF=1 ;;
|
||||
--no-reboot-prompt) SKIP_REBOOT_PROMPT=1 ;;
|
||||
--skip-swap) SKIP_SWAP=1 ;;
|
||||
--enable-auto-update) AUTO_UPDATE=1 ;;
|
||||
--no-auto-update) AUTO_UPDATE=0 ;;
|
||||
--build-jobs)
|
||||
shift
|
||||
if [ $# -eq 0 ]; then echo "--build-jobs requires a number"; usage; exit 1; fi
|
||||
@@ -863,6 +870,50 @@ else
|
||||
echo "✓ Main config file already exists"
|
||||
fi
|
||||
|
||||
# Weekly automatic updates (General tab -> Automatic Updates). Off unless asked
|
||||
# for: --enable-auto-update / LEDMATRIX_AUTO_UPDATE=1, or "y" at the prompt when
|
||||
# installing interactively. Only an explicit choice changes the setting, so
|
||||
# re-running the installer with -y never switches it silently.
|
||||
if [ -z "$AUTO_UPDATE" ] && [ "$ASSUME_YES" != "1" ] && [ -t 0 ]; then
|
||||
read -p "Automatically check for and install LEDMatrix updates once a week, with automatic rollback if an update breaks something? (y/N): " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then AUTO_UPDATE=1; else AUTO_UPDATE=0; fi
|
||||
fi
|
||||
if [ "$AUTO_UPDATE" = "1" ] || [ "$AUTO_UPDATE" = "0" ]; then
|
||||
if python3 - "$PROJECT_ROOT_DIR/config/config.json" "$AUTO_UPDATE" <<'PY'
|
||||
import json, os, sys, tempfile
|
||||
path, enabled = sys.argv[1], sys.argv[2] == "1"
|
||||
with open(path, encoding="utf-8") as f:
|
||||
config = json.load(f)
|
||||
if not isinstance(config.get("auto_update"), dict):
|
||||
config["auto_update"] = {}
|
||||
config["auto_update"]["enabled"] = enabled
|
||||
# Written beside the original and swapped in whole: the display service's
|
||||
# config watcher may be running and must never read a half-written file.
|
||||
original = os.stat(path)
|
||||
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(path)), prefix=".config.")
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=4)
|
||||
f.write("\n")
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.chmod(tmp, original.st_mode & 0o777)
|
||||
if hasattr(os, "chown"):
|
||||
os.chown(tmp, original.st_uid, original.st_gid)
|
||||
os.replace(tmp, path)
|
||||
except BaseException:
|
||||
if os.path.exists(tmp):
|
||||
os.unlink(tmp)
|
||||
raise
|
||||
PY
|
||||
then
|
||||
if [ "$AUTO_UPDATE" = "1" ]; then echo "✓ Weekly automatic updates enabled"; else echo "✓ Weekly automatic updates off"; fi
|
||||
else
|
||||
echo "⚠ Could not set auto_update in config/config.json; turn it on from the General tab instead"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Create config_secrets.json from template if missing
|
||||
if [ ! -f "$PROJECT_ROOT_DIR/config/config_secrets.json" ]; then
|
||||
if [ -f "$PROJECT_ROOT_DIR/config/config_secrets.template.json" ]; then
|
||||
@@ -1341,7 +1392,7 @@ if [ -f "$PROJECT_ROOT_DIR/scripts/install/install_web_service.sh" ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ ! -f "/etc/systemd/system/ledmatrix-web.service" ] || [ "$NEEDS_UPDATE" = true ]; then
|
||||
if [ ! -f "/etc/systemd/system/ledmatrix-web.service" ] || [ ! -f "/etc/systemd/system/ledmatrix-update-verify.path" ] || [ "$NEEDS_UPDATE" = true ]; then
|
||||
bash "$PROJECT_ROOT_DIR/scripts/install/install_web_service.sh"
|
||||
# Ensure systemd sees any new/changed unit files
|
||||
systemctl daemon-reload || true
|
||||
@@ -1357,7 +1408,7 @@ echo ""
|
||||
CURRENT_STEP="Harden systemd unit file permissions"
|
||||
echo "Step 8.1: Setting systemd unit file permissions..."
|
||||
echo "-----------------------------------------------"
|
||||
for unit in "/etc/systemd/system/ledmatrix.service" "/etc/systemd/system/ledmatrix-web.service" "/etc/systemd/system/ledmatrix-wifi-monitor.service"; do
|
||||
for unit in "/etc/systemd/system/ledmatrix.service" "/etc/systemd/system/ledmatrix-web.service" "/etc/systemd/system/ledmatrix-wifi-monitor.service" "/etc/systemd/system/ledmatrix-update-verify.service" "/etc/systemd/system/ledmatrix-update-verify.path"; do
|
||||
if [ -f "$unit" ]; then
|
||||
chown root:root "$unit" || true
|
||||
chmod 644 "$unit" || true
|
||||
|
||||
+8
-4
@@ -43,10 +43,14 @@ packaging>=23.0,<27.0
|
||||
# full feature set, or skip them for a minimal install.
|
||||
# ───────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# scipy — sub-pixel interpolation in
|
||||
# src/common/scroll_helper.py for smoother
|
||||
# scrolling. Falls back to a simpler shift algorithm.
|
||||
# pip install 'scipy>=1.10.0,<2.0.0'
|
||||
# scipy — nothing, as of #570. It was listed for the sub-pixel
|
||||
# interpolation path in src/common/scroll_helper.py, but
|
||||
# get_visible_portion never consulted HAS_SCIPY, so that
|
||||
# path was dead before it was deleted. The blend that
|
||||
# replaced it is numpy-only. Do not install it expecting
|
||||
# smoother scrolling: sub-pixel blending is off by default
|
||||
# because it reads worse on a coarse panel, not because it
|
||||
# is missing a library. See docs/SCROLL_PERFORMANCE.md.
|
||||
#
|
||||
# psutil — per-plugin resource monitoring in
|
||||
# src/plugin_system/resource_monitor.py. The monitor
|
||||
|
||||
@@ -95,6 +95,19 @@ else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Health check / rollback units for automatic updates; see install_web_service.sh.
|
||||
for VERIFY_UNIT in ledmatrix-update-verify.service ledmatrix-update-verify.path; do
|
||||
if [ -f "$PROJECT_ROOT_DIR/systemd/$VERIFY_UNIT" ]; then
|
||||
VERIFY_UNIT_TMP=$(mktemp)
|
||||
if sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g; s|__USER__|$ESCAPED_ACTUAL_USER|g" "$PROJECT_ROOT_DIR/systemd/$VERIFY_UNIT" > "$VERIFY_UNIT_TMP"; then
|
||||
sudo cp "$VERIFY_UNIT_TMP" "/etc/systemd/system/$VERIFY_UNIT"
|
||||
else
|
||||
echo "WARNING: failed to render $VERIFY_UNIT; automatic code updates will stay paused." >&2
|
||||
fi
|
||||
rm -f "$VERIFY_UNIT_TMP"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Reloading systemd daemon for web service..."
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
@@ -102,6 +115,11 @@ if [ -f "/etc/systemd/system/ledmatrix-web.service" ]; then
|
||||
echo "Enabling ledmatrix-web.service to start on boot..."
|
||||
sudo systemctl enable ledmatrix-web.service
|
||||
|
||||
if [ -f /etc/systemd/system/ledmatrix-update-verify.path ]; then
|
||||
echo "Enabling ledmatrix-update-verify.path (automatic update health check)..."
|
||||
sudo systemctl enable --now ledmatrix-update-verify.path || echo "WARNING: could not enable ledmatrix-update-verify.path; automatic code updates will stay paused" >&2
|
||||
fi
|
||||
|
||||
echo "Starting ledmatrix-web.service..."
|
||||
sudo systemctl start ledmatrix-web.service
|
||||
|
||||
|
||||
@@ -46,6 +46,22 @@ ESCAPED_ACTUAL_USER=$(sed_escape_replacement "$ACTUAL_USER")
|
||||
sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g; s|__USER__|$ESCAPED_ACTUAL_USER|g" \
|
||||
"$TEMPLATE" > /etc/systemd/system/ledmatrix-web.service
|
||||
|
||||
# Health check and rollback for the web UI's automatic updates. Its own unit so
|
||||
# it survives the web service restart it performs; never enabled -- the web
|
||||
# interface starts it after an update. Without it, automatic code updates
|
||||
# stay paused rather than running with nothing to undo them.
|
||||
for VERIFY_UNIT in ledmatrix-update-verify.service ledmatrix-update-verify.path; do
|
||||
VERIFY_TEMPLATE="$PROJECT_ROOT_DIR/systemd/$VERIFY_UNIT"
|
||||
if [ -f "$VERIFY_TEMPLATE" ]; then
|
||||
echo "Writing unit file to /etc/systemd/system/$VERIFY_UNIT"
|
||||
sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g; s|__USER__|$ESCAPED_ACTUAL_USER|g" \
|
||||
"$VERIFY_TEMPLATE" > "/etc/systemd/system/$VERIFY_UNIT"
|
||||
chmod 644 "/etc/systemd/system/$VERIFY_UNIT"
|
||||
else
|
||||
echo "WARNING: $VERIFY_TEMPLATE not found; automatic code updates will stay paused"
|
||||
fi
|
||||
done
|
||||
|
||||
# Ensure cache directory exists with proper permissions
|
||||
# This is a fallback for older systemd versions that don't support CacheDirectory
|
||||
# Systemd 239+ will automatically create it via CacheDirectory directive
|
||||
@@ -80,6 +96,13 @@ systemctl daemon-reload
|
||||
echo "Enabling ledmatrix-web.service..."
|
||||
systemctl enable ledmatrix-web.service
|
||||
|
||||
# The path unit is what starts the health check after an automatic update.
|
||||
if [ -f /etc/systemd/system/ledmatrix-update-verify.path ]; then
|
||||
echo "Enabling ledmatrix-update-verify.path..."
|
||||
systemctl enable --now ledmatrix-update-verify.path || \
|
||||
echo "WARNING: could not enable ledmatrix-update-verify.path; automatic code updates will stay paused"
|
||||
fi
|
||||
|
||||
# Start the service
|
||||
echo "Starting ledmatrix-web.service..."
|
||||
systemctl start ledmatrix-web.service
|
||||
|
||||
@@ -408,6 +408,7 @@ main() {
|
||||
# which would silently reinstate the duplicate apt update.
|
||||
sudo -E env TMPDIR=/tmp LEDMATRIX_ASSUME_YES=1 \
|
||||
LEDMATRIX_APT_UPDATED="${LEDMATRIX_APT_UPDATED:-0}" \
|
||||
LEDMATRIX_AUTO_UPDATE="${LEDMATRIX_AUTO_UPDATE:-}" \
|
||||
bash ./first_time_install.sh -y </dev/null
|
||||
fi
|
||||
INSTALL_EXIT_CODE=$?
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check that an automatic LEDMatrix update left the device working; roll it back if not.
|
||||
|
||||
Started by the web interface's weekly updater (web_interface/auto_update.py)
|
||||
through ledmatrix-update-verify.service, right after it pulls new code. It has
|
||||
to run outside the web service: checking the update means restarting that
|
||||
service, and a check running inside it would be killed by its own restart.
|
||||
|
||||
It must not be the code it is checking, either. The updater copies this file
|
||||
to data/auto_update_verifier.py *before* pulling and the unit runs that copy,
|
||||
so a broken update cannot break its own rollback. Standard library only for
|
||||
the same reason: the rollback cannot depend on packages the update changed.
|
||||
|
||||
The updater leaves data/auto_update_pending.json:
|
||||
|
||||
{"status": "pending", "old_head": ..., "new_head": ...,
|
||||
"display_was_active": bool, "dependency_failures": [...], "created_at": ...}
|
||||
|
||||
This moves its status to "verifying" and then to one of "success",
|
||||
"rolled_back" or "rollback_failed", with "reason" and "detail" saying why.
|
||||
The web interface reports that outcome and raises a banner for anything but
|
||||
success.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
|
||||
import sys
|
||||
from collections import namedtuple
|
||||
import tempfile
|
||||
import time
|
||||
import traceback
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
PENDING_NAME = 'auto_update_pending.json'
|
||||
REQUIREMENT_FILES = ('requirements.txt', 'web_interface/requirements.txt')
|
||||
WEB_HEALTH_URL = 'http://127.0.0.1:5000/api/v3/system/version'
|
||||
#: How long the services get to come up after a restart...
|
||||
HEALTH_TIMEOUT_SECONDS = 180
|
||||
#: ...and how long they must then stay up. Restart=on-failure makes a crash
|
||||
#: loop look healthy between attempts, so a single "is-active" proves nothing.
|
||||
STABLE_SECONDS = 45
|
||||
POLL_SECONDS = 5
|
||||
PIP_TIMEOUT_SECONDS = 600
|
||||
#: sudoers matches the exact command line, so bash is named by path, the same
|
||||
#: candidates src/common/permission_utils.install_requirements_file tries.
|
||||
BASH_CANDIDATES = ('/usr/bin/bash', '/bin/bash')
|
||||
|
||||
#: What a command that could not run at all reports: its callers only read
|
||||
#: these three fields, the same ones a completed subprocess has.
|
||||
_Failed = namedtuple('_Failed', 'returncode stdout stderr')
|
||||
|
||||
|
||||
def pending_path(project_root):
|
||||
return Path(project_root) / 'data' / PENDING_NAME
|
||||
|
||||
|
||||
def read_pending(path):
|
||||
try:
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def write_pending(path, data):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix='.auto_update_pending_')
|
||||
try:
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
os.replace(tmp, path)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def _web_responds(url=WEB_HEALTH_URL):
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=5) as resp: # nosec B310 - fixed loopback URL
|
||||
return resp.status == 200
|
||||
except (urllib.error.URLError, OSError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def _short(sha):
|
||||
return (sha or 'unknown')[:7]
|
||||
|
||||
|
||||
class Verifier:
|
||||
def __init__(self, project_root, run=subprocess.run, sleep=time.sleep,
|
||||
clock=time.monotonic, web_responds=_web_responds, log=None):
|
||||
self.project_root = Path(project_root)
|
||||
self.pending_file = pending_path(project_root)
|
||||
self.run = run
|
||||
self.sleep = sleep
|
||||
self.clock = clock
|
||||
self.web_responds = web_responds
|
||||
self.log = log or (lambda msg: print(f'[auto-update-verify] {msg}', flush=True))
|
||||
|
||||
def _run(self, args, timeout=60):
|
||||
try:
|
||||
return self.run(args, cwd=str(self.project_root), capture_output=True,
|
||||
text=True, timeout=timeout)
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
return _Failed(returncode=1, stdout='', stderr=str(e))
|
||||
|
||||
# -- services ---------------------------------------------------------
|
||||
|
||||
def service_active(self, unit):
|
||||
return self._run(['systemctl', 'is-active', unit], timeout=10).stdout.strip() == 'active'
|
||||
|
||||
def restart_count(self, unit):
|
||||
out = self._run(['systemctl', 'show', '-p', 'NRestarts', '--value', unit],
|
||||
timeout=10).stdout.strip()
|
||||
return int(out) if out.isdigit() else None
|
||||
|
||||
def restart(self, unit):
|
||||
result = self._run(['sudo', '-n', 'systemctl', 'restart', f'{unit}.service'], timeout=90)
|
||||
if result.returncode != 0:
|
||||
self.log(f'restarting {unit} failed: {(result.stderr or "").strip()}')
|
||||
return result.returncode == 0
|
||||
|
||||
def restart_services(self, display):
|
||||
"""Restart what should be running. False if any restart command failed."""
|
||||
ok = True
|
||||
# A display the user had stopped stays stopped.
|
||||
if display:
|
||||
ok = self.restart('ledmatrix') and ok
|
||||
return self.restart('ledmatrix-web') and ok
|
||||
|
||||
def wait_healthy(self, display):
|
||||
"""None once the services are up and stay up, else what went wrong."""
|
||||
deadline = self.clock() + HEALTH_TIMEOUT_SECONDS + STABLE_SECONDS
|
||||
healthy_since = baseline = None
|
||||
web = disp = False
|
||||
count_known = True
|
||||
while self.clock() < deadline:
|
||||
web = self.web_responds()
|
||||
disp = self.service_active('ledmatrix') if display else True
|
||||
restarts = self.restart_count('ledmatrix') if display else None
|
||||
# Without a restart count a crash loop looks healthy between
|
||||
# attempts, so an unreadable count never counts as stable.
|
||||
count_known = not display or restarts is not None
|
||||
if web and disp and count_known and (healthy_since is None or restarts == baseline):
|
||||
if healthy_since is None:
|
||||
healthy_since, baseline = self.clock(), restarts
|
||||
elif self.clock() - healthy_since >= STABLE_SECONDS:
|
||||
return None
|
||||
else:
|
||||
healthy_since = None
|
||||
self.sleep(POLL_SECONDS)
|
||||
problems = []
|
||||
if not web:
|
||||
problems.append('the web interface did not respond')
|
||||
if not disp:
|
||||
problems.append('the display service did not stay running')
|
||||
if web and disp and not count_known:
|
||||
problems.append("the display service's restart count could not be read")
|
||||
return '; '.join(problems) or 'the display service kept restarting'
|
||||
|
||||
# -- rollback ---------------------------------------------------------
|
||||
|
||||
def changed_requirements(self, old, new):
|
||||
result = self._run(['git', 'diff', '--name-only', old, new])
|
||||
# If the diff is unavailable, reinstall both rather than guess.
|
||||
changed = set(result.stdout.split()) if result.returncode == 0 else set(REQUIREMENT_FILES)
|
||||
return [rel for rel in REQUIREMENT_FILES if rel in changed]
|
||||
|
||||
def install_requirements(self, rel):
|
||||
wrapper = self.project_root / 'scripts' / 'fix_perms' / 'safe_pip_install.sh'
|
||||
req = self.project_root / rel
|
||||
if not req.exists():
|
||||
return True
|
||||
for bash in BASH_CANDIDATES:
|
||||
result = self._run(['sudo', '-n', bash, str(wrapper), str(req)],
|
||||
timeout=PIP_TIMEOUT_SECONDS)
|
||||
if result.returncode == 0:
|
||||
return True
|
||||
return False
|
||||
|
||||
def rollback(self, pending):
|
||||
"""Reset to the previous commit and its dependencies. Returns (ok, detail)."""
|
||||
old, new = pending.get('old_head'), pending.get('new_head')
|
||||
if not old:
|
||||
return False, 'the commit to roll back to is unknown'
|
||||
requirements = self.changed_requirements(old, new) if new else list(REQUIREMENT_FILES)
|
||||
# Safe to --hard: the updater refuses to run with local edits to
|
||||
# tracked files, so the only thing this discards is the update.
|
||||
result = self._run(['git', 'reset', '--hard', old], timeout=120)
|
||||
if result.returncode != 0:
|
||||
return False, (f'"git reset --hard {old}" failed: '
|
||||
f'{(result.stderr or result.stdout or "").strip()}')
|
||||
failed = [rel for rel in requirements if not self.install_requirements(rel)]
|
||||
if failed:
|
||||
return True, ('reinstalling the previous dependencies from ' + ', '.join(failed)
|
||||
+ ' failed; run Install Base Requirements from the Tools tab')
|
||||
return True, ''
|
||||
|
||||
# -- the check itself -------------------------------------------------
|
||||
|
||||
def _finish(self, pending, status, reason=None, detail=None):
|
||||
pending.update({'status': status, 'reason': reason, 'detail': detail or None,
|
||||
'finished_at': time.time()})
|
||||
write_pending(self.pending_file, pending)
|
||||
self.log(' '.join(p for p in (status, reason or '', detail or '') if p))
|
||||
|
||||
def verify(self):
|
||||
pending = read_pending(self.pending_file)
|
||||
if not pending or pending.get('status') != 'pending':
|
||||
self.log('no update is waiting to be verified')
|
||||
return 0
|
||||
pending['status'] = 'verifying'
|
||||
write_pending(self.pending_file, pending)
|
||||
|
||||
display = bool(pending.get('display_was_active'))
|
||||
dependency_failures = pending.get('dependency_failures') or []
|
||||
if dependency_failures:
|
||||
# Never restart onto code whose packages did not install.
|
||||
reason = 'installing its dependencies failed (' + ', '.join(dependency_failures) + ')'
|
||||
elif not self.restart_services(display):
|
||||
# The old process may still be answering; checking it would pass
|
||||
# an update that never started.
|
||||
reason = 'restarting the services failed'
|
||||
else:
|
||||
reason = self.wait_healthy(display)
|
||||
if reason is None:
|
||||
self._finish(pending, 'success')
|
||||
return 0
|
||||
|
||||
self.log(f'update to {_short(pending.get("new_head"))} is unhealthy ({reason}); '
|
||||
f'rolling back to {_short(pending.get("old_head"))}')
|
||||
ok, detail = self.rollback(pending)
|
||||
if not ok:
|
||||
self._finish(pending, 'rollback_failed', reason, detail)
|
||||
return 1
|
||||
still = (self.wait_healthy(display) if self.restart_services(display)
|
||||
else 'restarting the services failed')
|
||||
if still:
|
||||
self._finish(pending, 'rollback_failed', reason,
|
||||
f'still unhealthy after rolling back: {still}'
|
||||
+ (f'; {detail}' if detail else ''))
|
||||
return 1
|
||||
self._finish(pending, 'rolled_back', reason, detail)
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv):
|
||||
if len(argv) != 2:
|
||||
print('usage: auto_update_verify.py PROJECT_ROOT', file=sys.stderr)
|
||||
return 2
|
||||
verifier = Verifier(Path(argv[1]))
|
||||
try:
|
||||
return verifier.verify()
|
||||
except Exception as e:
|
||||
traceback.print_exc()
|
||||
# Whatever happened, the web interface must not be left thinking the
|
||||
# check is still running.
|
||||
try:
|
||||
pending = read_pending(verifier.pending_file) or {}
|
||||
if pending.get('status') in ('pending', 'verifying'):
|
||||
pending.update({'status': 'rollback_failed', 'reason': 'the health check crashed',
|
||||
'detail': str(e), 'finished_at': time.time()})
|
||||
write_pending(verifier.pending_file, pending)
|
||||
except OSError:
|
||||
pass
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main(sys.argv))
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Install the automatic-update health check, from the display service.
|
||||
|
||||
The weekly updater (web_interface/auto_update.py) will not update LEDMatrix
|
||||
code unless ledmatrix-update-verify.path and .service are installed: they
|
||||
restart the services after an update and roll it back if the device is
|
||||
unhealthy. Installing units takes root and the web interface is not root, and
|
||||
"SSH in and run an installer" means most people never get updates with a
|
||||
safety net.
|
||||
|
||||
The display service already runs this repository's code as root, so it
|
||||
installs them -- but only while the user has automatic updates turned on, only
|
||||
these two units, rendered from the repository's templates for the web
|
||||
interface's own user, and it reports what happened in
|
||||
data/auto_update_setup.json for the General tab. It grants nothing new: the
|
||||
units run as the web user, who can already change the code this process runs.
|
||||
|
||||
Called at display startup; the web interface restarts the display service
|
||||
when the toggle is switched on, so setup happens straight away. Refreshing a
|
||||
unit whose template changed happens the same way, which is why this compares
|
||||
content rather than only checking that the files exist.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
SYSTEMD_DIR = Path('/etc/systemd/system')
|
||||
SERVICE_UNIT = 'ledmatrix-update-verify.service'
|
||||
PATH_UNIT = 'ledmatrix-update-verify.path'
|
||||
UNITS = (SERVICE_UNIT, PATH_UNIT)
|
||||
WEB_UNIT = 'ledmatrix-web.service'
|
||||
RESULT_REL = Path('data') / 'auto_update_setup.json'
|
||||
_USER_RE = re.compile(r'^[a-z_][a-z0-9_-]{0,31}$')
|
||||
|
||||
|
||||
class SetupError(Exception):
|
||||
"""A reason setup cannot proceed, worded for the General tab."""
|
||||
|
||||
|
||||
def _is_root():
|
||||
return hasattr(os, 'geteuid') and os.geteuid() == 0
|
||||
|
||||
|
||||
def _lookup_ids(user):
|
||||
try:
|
||||
import pwd
|
||||
entry = pwd.getpwnam(user)
|
||||
return entry.pw_uid, entry.pw_gid
|
||||
except (ImportError, KeyError):
|
||||
return None
|
||||
|
||||
|
||||
def _directive(text, key):
|
||||
match = re.search(rf'^{key}=(.*)$', text or '', re.M)
|
||||
return match.group(1).strip() if match else None
|
||||
|
||||
|
||||
def _read(path):
|
||||
try:
|
||||
return Path(path).read_text(encoding='utf-8')
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def is_enabled(config):
|
||||
return bool((config.get('auto_update') or {}).get('enabled', False))
|
||||
|
||||
|
||||
class UpdateHelperSetup:
|
||||
def __init__(self, project_root=PROJECT_ROOT, systemd_dir=SYSTEMD_DIR, run=subprocess.run,
|
||||
is_root=_is_root, lookup_ids=_lookup_ids, clock=time.time):
|
||||
self.project_root = Path(project_root)
|
||||
self.systemd_dir = Path(systemd_dir)
|
||||
self.run = run
|
||||
self.is_root = is_root
|
||||
self.lookup_ids = lookup_ids
|
||||
self.clock = clock
|
||||
self.result_file = self.project_root / RESULT_REL
|
||||
self._web_ids = None
|
||||
|
||||
def _systemctl(self, *args):
|
||||
return self.run(['systemctl', *args], capture_output=True, text=True, timeout=60)
|
||||
|
||||
def _check(self, result, what):
|
||||
if result.returncode != 0:
|
||||
raise SetupError(f'"{what}" failed: {(result.stderr or result.stdout or "").strip()}')
|
||||
|
||||
def path_active(self):
|
||||
try:
|
||||
return self._systemctl('is-active', PATH_UNIT).stdout.strip() == 'active'
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
|
||||
def ensure(self, config):
|
||||
"""Install or refresh the units while automatic updates are on.
|
||||
|
||||
Returns the result recorded for the General tab, or None when there
|
||||
was nothing to do (updates off, or not a systemd host at all).
|
||||
"""
|
||||
if not is_enabled(config) or not self.systemd_dir.is_dir():
|
||||
return None
|
||||
try:
|
||||
changed = self._install()
|
||||
except SetupError as e:
|
||||
return self._report('failed', str(e))
|
||||
except (OSError, subprocess.SubprocessError) as e:
|
||||
return self._report('failed', f'Could not install the update health check: {e}')
|
||||
if changed:
|
||||
return self._report('installed', 'Installed the update health check.')
|
||||
return self._report('installed', 'The update health check is installed.', quiet=True)
|
||||
|
||||
def _install(self):
|
||||
if not self.is_root():
|
||||
raise SetupError('The display service is not running as root, so it cannot install the '
|
||||
'update health check. Run "sudo ./scripts/install/install_web_service.sh" once.')
|
||||
|
||||
web_text = _read(self.systemd_dir / WEB_UNIT)
|
||||
if web_text is None:
|
||||
raise SetupError('The web interface service (ledmatrix-web.service) is not installed.')
|
||||
user = _directive(web_text, 'User') or 'root'
|
||||
ids = self.lookup_ids(user) if _USER_RE.match(user) else None
|
||||
if ids is None:
|
||||
raise SetupError(f'The web interface runs as "{user}", which is not a usable account.')
|
||||
self._web_ids = ids
|
||||
workdir = _directive(web_text, 'WorkingDirectory')
|
||||
if not workdir or Path(workdir).resolve() != self.project_root.resolve():
|
||||
raise SetupError(f'The web interface service runs from {workdir or "an unknown folder"}, '
|
||||
f'not {self.project_root}.')
|
||||
# Spaces are fine -- the templates quote every command-line path --
|
||||
# but systemd expands % specifiers, and a quote, backslash or line
|
||||
# break would be reinterpreted in a unit file. (On Windows, where the
|
||||
# tests also run, a backslash is the path separator, not a name.)
|
||||
root_text = str(self.project_root)
|
||||
unsafe = set('%"') | ({'\\'} if os.sep == '/' else set())
|
||||
if any(ch in unsafe or ord(ch) < 32 for ch in root_text):
|
||||
raise SetupError(f'LEDMatrix is installed in {root_text!r}, a folder name systemd cannot use '
|
||||
'in a unit file. Move it to a path without %, quotes, backslashes or '
|
||||
'control characters.')
|
||||
|
||||
rendered = {}
|
||||
for name in UNITS:
|
||||
template = _read(self.project_root / 'systemd' / name)
|
||||
if template is None:
|
||||
raise SetupError(f'The unit template systemd/{name} is missing.')
|
||||
rendered[name] = (template.replace('__PROJECT_ROOT_DIR__', str(self.project_root))
|
||||
.replace('__USER__', user))
|
||||
# The templates are ordinary repository files. Whatever they say,
|
||||
# this root process only installs a service that runs as the web user
|
||||
# and a path unit that starts exactly that service.
|
||||
if _directive(rendered[SERVICE_UNIT], 'User') != user:
|
||||
raise SetupError(f'systemd/{SERVICE_UNIT} does not run as the web interface user; '
|
||||
'refusing to install it.')
|
||||
if _directive(rendered[PATH_UNIT], 'Unit') != SERVICE_UNIT:
|
||||
raise SetupError(f'systemd/{PATH_UNIT} does not start {SERVICE_UNIT}; refusing to install it.')
|
||||
|
||||
changed = [name for name in UNITS if _read(self.systemd_dir / name) != rendered[name]]
|
||||
for name in changed:
|
||||
self._write_unit(self.systemd_dir / name, rendered[name])
|
||||
if changed:
|
||||
self._check(self._systemctl('daemon-reload'), 'systemctl daemon-reload')
|
||||
self._check(self._systemctl('enable', PATH_UNIT), f'systemctl enable {PATH_UNIT}')
|
||||
self._check(self._systemctl('restart', PATH_UNIT), f'systemctl restart {PATH_UNIT}')
|
||||
elif not self.path_active():
|
||||
self._check(self._systemctl('enable', '--now', PATH_UNIT), f'systemctl enable --now {PATH_UNIT}')
|
||||
changed = [PATH_UNIT]
|
||||
if not self.path_active():
|
||||
raise SetupError(f'{PATH_UNIT} did not start; see "journalctl -u {PATH_UNIT}".')
|
||||
return bool(changed)
|
||||
|
||||
def _write_unit(self, path, text):
|
||||
fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f'.{path.name}.')
|
||||
try:
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as f:
|
||||
f.write(text)
|
||||
os.chmod(tmp, 0o644)
|
||||
os.replace(tmp, path)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
def _report(self, status, message, quiet=False):
|
||||
previous = None
|
||||
try:
|
||||
previous = json.loads(_read(self.result_file) or 'null')
|
||||
except ValueError:
|
||||
pass
|
||||
if quiet and isinstance(previous, dict) and previous.get('status') == status:
|
||||
return previous # nothing new; don't rewrite it on every boot
|
||||
result = {'status': status, 'message': message, 'at': self.clock()}
|
||||
(logger.info if status == 'installed' else logger.warning)("Automatic update setup: %s", message)
|
||||
try:
|
||||
self.result_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(dir=str(self.result_file.parent), prefix='.auto_update_setup_')
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, indent=2)
|
||||
os.chmod(tmp, 0o644)
|
||||
if self._web_ids and hasattr(os, 'chown'):
|
||||
# Only root can give the file away; the result is readable
|
||||
# (0644) either way, so a failed chown must not lose it.
|
||||
try:
|
||||
os.chown(tmp, *self._web_ids)
|
||||
except OSError:
|
||||
pass
|
||||
os.replace(tmp, self.result_file)
|
||||
except OSError as e:
|
||||
logger.warning("Could not record automatic update setup result: %s", e)
|
||||
return result
|
||||
|
||||
|
||||
def ensure_update_helper(config):
|
||||
return UpdateHelperSetup().ensure(config)
|
||||
@@ -30,6 +30,12 @@ except ImportError:
|
||||
HAS_SCIPY = False
|
||||
|
||||
|
||||
# How often the frame-stats line is emitted, and therefore also the ceiling
|
||||
# on a believable frame time: a scroll that renders at all cannot take this
|
||||
# long over one frame, so a sample this large is an idle gap between scrolls.
|
||||
FPS_LOG_INTERVAL = 5.0
|
||||
|
||||
|
||||
def frame_stats(frame_times: list) -> Dict[str, Any]:
|
||||
"""Summary statistics over one window of frame durations (seconds).
|
||||
|
||||
@@ -158,9 +164,11 @@ class ScrollHelper:
|
||||
self.last_progress_log_time: Optional[float] = None
|
||||
self.progress_log_interval = 5.0 # seconds
|
||||
|
||||
# Frame rate tracking
|
||||
# Frame rate tracking. last_frame_time is None until the first frame
|
||||
# of a scroll is rendered -- see log_frame_rate() for why timing from
|
||||
# construction (or from the end of the previous scroll) is wrong.
|
||||
self.frame_count = 0
|
||||
self.last_frame_time = time.time()
|
||||
self.last_frame_time: Optional[float] = None
|
||||
self.last_fps_log_time = time.time()
|
||||
self.frame_times = []
|
||||
# Every frame time since the last stats line, so the 5s summary can
|
||||
@@ -764,6 +772,10 @@ class ScrollHelper:
|
||||
# Reset last_update_time to prevent large delta_time on next update
|
||||
# This ensures smooth scrolling after reset without jumping ahead
|
||||
self.last_update_time = now
|
||||
# Same reasoning for the frame-rate clock: the first frame after a
|
||||
# reset has no predecessor in this scroll, and timing it against the
|
||||
# last frame of the previous one measures the idle gap between them.
|
||||
self.last_frame_time = None
|
||||
self.logger.debug("Scroll position reset")
|
||||
|
||||
def reset(self) -> None:
|
||||
@@ -963,8 +975,34 @@ class ScrollHelper:
|
||||
"""
|
||||
current_time = time.time()
|
||||
|
||||
# The first frame of a scroll has no predecessor, so it has no frame
|
||||
# time. Measuring one anyway records the whole idle gap since the last
|
||||
# scroll as a single frame: on hardware that produced windows reading
|
||||
# "0.0 fps over 1 frames | median 136776.02ms", and -- worse, because
|
||||
# it is not obviously wrong -- put that gap in the max field of
|
||||
# otherwise healthy windows and counted it as one stall per scroll.
|
||||
# At ~500 frames to a window that is ~0.2%, which is the same order as
|
||||
# the real stall rates being measured, so the number could not be
|
||||
# trusted at all. Seed the clock and take no sample.
|
||||
if self.last_frame_time is None:
|
||||
self.last_frame_time = current_time
|
||||
# Restart the window with the scroll. Otherwise the boundary is
|
||||
# already long overdue when the second frame arrives, and the new
|
||||
# scroll opens by reporting a window of exactly one frame.
|
||||
self.last_fps_log_time = current_time
|
||||
return
|
||||
|
||||
# Calculate instantaneous frame time
|
||||
frame_time = current_time - self.last_frame_time
|
||||
|
||||
# A caller that scrolls without ever calling reset_scroll() never arms
|
||||
# the sentinel above, so catch the same gap by its size. Nothing that
|
||||
# renders a scroll produces a frame longer than the log interval; a
|
||||
# sample that large is an idle period, not a frame.
|
||||
if frame_time >= FPS_LOG_INTERVAL:
|
||||
self.last_frame_time = current_time
|
||||
return
|
||||
|
||||
self.frame_times.append(frame_time)
|
||||
|
||||
# Keep only last 100 frames for average
|
||||
@@ -981,11 +1019,15 @@ class ScrollHelper:
|
||||
self._window.append(frame_time)
|
||||
|
||||
# Log FPS every 5 seconds to avoid spam
|
||||
if current_time - self.last_fps_log_time >= 5.0:
|
||||
self.logger.info(
|
||||
"Scroll frame stats - %s",
|
||||
format_frame_stats(self._window or [frame_time]),
|
||||
)
|
||||
if current_time - self.last_fps_log_time >= FPS_LOG_INTERVAL:
|
||||
# An empty window means every sample in this interval was dropped
|
||||
# as an idle gap. There is nothing to report, and reporting the
|
||||
# gap itself is the bug above.
|
||||
if self._window:
|
||||
self.logger.info(
|
||||
"Scroll frame stats - %s",
|
||||
format_frame_stats(self._window),
|
||||
)
|
||||
self.last_fps_log_time = current_time
|
||||
self.frame_count = 0
|
||||
self._window = []
|
||||
|
||||
@@ -120,6 +120,15 @@ class DisplayController:
|
||||
except Exception as e:
|
||||
logger.warning(f"Startup validation could not be completed: {e}")
|
||||
|
||||
# Automatic updates need their health-check units, and this is the
|
||||
# one root process running project code, so it installs them while
|
||||
# automatic updates are on. See src/auto_update_setup.py.
|
||||
try:
|
||||
from src.auto_update_setup import ensure_update_helper
|
||||
ensure_update_helper(self.config)
|
||||
except Exception as e:
|
||||
logger.warning("Automatic update setup could not be completed: %s", e)
|
||||
|
||||
config_time = time.time()
|
||||
self.display_manager = DisplayManager(self.config)
|
||||
logger.info("DisplayManager initialized in %.3f seconds", time.time() - config_time)
|
||||
|
||||
@@ -81,6 +81,8 @@ class StartupValidator:
|
||||
_UNITS = (
|
||||
("systemd/ledmatrix.service", "/etc/systemd/system/ledmatrix.service"),
|
||||
("systemd/ledmatrix-web.service", "/etc/systemd/system/ledmatrix-web.service"),
|
||||
("systemd/ledmatrix-update-verify.service", "/etc/systemd/system/ledmatrix-update-verify.service"),
|
||||
("systemd/ledmatrix-update-verify.path", "/etc/systemd/system/ledmatrix-update-verify.path"),
|
||||
)
|
||||
|
||||
def _validate_systemd_units(self) -> None:
|
||||
|
||||
+12
-1
@@ -14,6 +14,16 @@ This directory contains systemd service unit files for LEDMatrix services.
|
||||
- Starts automatically on boot if `web_display_autostart` is enabled
|
||||
- Uses `scripts/utils/start_web_conditionally.py`
|
||||
|
||||
- **`ledmatrix-update-verify.service`** / **`.path`** - Automatic update health check and rollback
|
||||
- The path unit starts the service when the web interface creates
|
||||
`data/auto_update_verify.request` after a weekly automatic update
|
||||
- Installed by the installers, or by the display service when automatic
|
||||
updates are turned on in the web UI (`src/auto_update_setup.py`)
|
||||
- Restarts the services, checks they stay up, and otherwise resets to the
|
||||
previous commit and its dependencies
|
||||
- Runs `data/auto_update_verifier.py`, a copy of
|
||||
`scripts/utils/auto_update_verify.py` taken before the update
|
||||
|
||||
- **`ledmatrix-wifi-monitor.service`** - WiFi monitor daemon service
|
||||
- Monitors WiFi/Ethernet connectivity
|
||||
- Automatically enables/disables access point mode
|
||||
@@ -43,7 +53,8 @@ This directory contains systemd service unit files for LEDMatrix services.
|
||||
|
||||
These service files are installed by the installation scripts in `scripts/install/`:
|
||||
- `install_service.sh` installs `ledmatrix.service`
|
||||
- `install_web_service.sh` installs `ledmatrix-web.service`
|
||||
- `install_web_service.sh` installs `ledmatrix-web.service` and the
|
||||
`ledmatrix-update-verify` service and path units
|
||||
- `install_wifi_monitor.sh` installs `ledmatrix-wifi-monitor.service`
|
||||
- `install_dns_fix.sh` installs `ledmatrix-dns-fix.service` (opt-in, not run
|
||||
by the normal installer)
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# This is a template. Do not install it directly -- it is installed with
|
||||
# ledmatrix-update-verify.service by scripts/install/install_web_service.sh,
|
||||
# or by the display service when automatic updates are turned on
|
||||
# (src/auto_update_setup.py). Substitutes:
|
||||
# __PROJECT_ROOT_DIR__ -> the directory the project is installed in
|
||||
#
|
||||
# Starts the automatic update health check when the web interface asks for it
|
||||
# by creating data/auto_update_verify.request. A path unit rather than a
|
||||
# sudoers rule: the web interface needs no extra privilege to trigger it, and
|
||||
# the service it starts runs as the web interface's own user.
|
||||
|
||||
[Unit]
|
||||
Description=Watch for LED Matrix automatic update health check requests
|
||||
|
||||
[Path]
|
||||
PathExists=__PROJECT_ROOT_DIR__/data/auto_update_verify.request
|
||||
Unit=ledmatrix-update-verify.service
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,31 @@
|
||||
# This is a template. Do not install it directly -- run
|
||||
# scripts/install/install_web_service.sh (or install_service.sh); the display
|
||||
# service also installs it when automatic updates are turned on
|
||||
# (src/auto_update_setup.py). Substitutes:
|
||||
# __PROJECT_ROOT_DIR__ -> the directory the project is installed in
|
||||
# __USER__ -> the user the web interface runs as
|
||||
#
|
||||
# Health check and rollback for the web interface's weekly automatic update
|
||||
# (web_interface/auto_update.py). A separate unit because the check restarts
|
||||
# ledmatrix-web.service, which would kill it if it ran there.
|
||||
#
|
||||
# Never enabled itself: ledmatrix-update-verify.path starts it when the web
|
||||
# interface creates data/auto_update_verify.request. It runs as the web
|
||||
# interface's own user, and ExecStart points at the copy the updater makes
|
||||
# before pulling, so the update being checked cannot replace its checker.
|
||||
# Command-line paths are quoted so an install folder with spaces still works.
|
||||
|
||||
[Unit]
|
||||
Description=LED Matrix automatic update health check and rollback
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=__USER__
|
||||
WorkingDirectory=__PROJECT_ROOT_DIR__
|
||||
# Consume the request first, so the path unit cannot start this again.
|
||||
ExecStartPre=/bin/rm -f "__PROJECT_ROOT_DIR__/data/auto_update_verify.request"
|
||||
ExecStart=/usr/bin/python3 "__PROJECT_ROOT_DIR__/data/auto_update_verifier.py" "__PROJECT_ROOT_DIR__"
|
||||
TimeoutStartSec=30min
|
||||
StandardOutput=syslog
|
||||
StandardError=syslog
|
||||
SyslogIdentifier=ledmatrix-update-verify
|
||||
Vendored
+17
@@ -841,6 +841,23 @@
|
||||
"POST"
|
||||
]
|
||||
],
|
||||
[
|
||||
"/api/v3/system/auto-update",
|
||||
"api_v3.get_auto_update_status",
|
||||
[
|
||||
"GET",
|
||||
"HEAD",
|
||||
"OPTIONS"
|
||||
]
|
||||
],
|
||||
[
|
||||
"/api/v3/system/auto-update/dismiss",
|
||||
"api_v3.dismiss_auto_update_alert",
|
||||
[
|
||||
"OPTIONS",
|
||||
"POST"
|
||||
]
|
||||
],
|
||||
[
|
||||
"/api/v3/system/check-update",
|
||||
"api_v3.check_for_update",
|
||||
|
||||
@@ -0,0 +1,683 @@
|
||||
"""Weekly automatic updates (web_interface/auto_update.py).
|
||||
|
||||
The updater pulls code and restarts services unattended. These pin down that
|
||||
it runs only when asked and due, refuses to touch a checkout it could damage,
|
||||
never leaves new code running without a health check, and reports every
|
||||
failure instead of logging it and moving on.
|
||||
|
||||
Git runs for real against a throwaway origin/device clone pair; systemd, sudo
|
||||
and the health check service are faked.
|
||||
"""
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from web_interface import auto_update as au # noqa: E402
|
||||
|
||||
# Local time is pinned to UTC below, so 03:00 is inside the quiet hours.
|
||||
NIGHT = datetime(2026, 9, 14, 3, 0, tzinfo=timezone.utc).timestamp()
|
||||
NOON = datetime(2026, 9, 14, 12, 0, tzinfo=timezone.utc).timestamp()
|
||||
ON = {'auto_update': {'enabled': True}}
|
||||
OFF = {'auto_update': {'enabled': False}}
|
||||
|
||||
needs_git = pytest.mark.skipif(shutil.which('git') is None, reason='git not installed')
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def local_is_utc(monkeypatch):
|
||||
"""Pin 'local time' so quiet-hours checks don't depend on the host zone."""
|
||||
monkeypatch.setattr(au, '_zone', lambda name: timezone.utc)
|
||||
|
||||
|
||||
def git(cwd, *args):
|
||||
result = subprocess.run(['git', '-c', 'user.name=t', '-c', 'user.email=t@example.com', *args],
|
||||
cwd=str(cwd), capture_output=True, text=True)
|
||||
assert result.returncode == 0, result.stderr
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
class Repo:
|
||||
"""An origin, a checkout that publishes to it, and the device's clone."""
|
||||
|
||||
def __init__(self, tmp):
|
||||
self.seed = tmp / 'seed'
|
||||
self.seed.mkdir()
|
||||
git(self.seed, 'init', '-q', '-b', 'main')
|
||||
(self.seed / 'app.py').write_text('v = 1\n')
|
||||
git(self.seed, 'add', '.')
|
||||
git(self.seed, 'commit', '-qm', 'one')
|
||||
self.origin = tmp / 'origin.git'
|
||||
git(tmp, 'clone', '-q', '--bare', str(self.seed), str(self.origin))
|
||||
git(self.seed, 'remote', 'add', 'origin', str(self.origin))
|
||||
self.device = tmp / 'device'
|
||||
git(tmp, 'clone', '-q', str(self.origin), str(self.device))
|
||||
|
||||
def head(self):
|
||||
return git(self.device, 'rev-parse', 'HEAD')
|
||||
|
||||
def publish(self, text='v = 2\n'):
|
||||
(self.seed / 'app.py').write_text(text)
|
||||
git(self.seed, 'commit', '-qam', 'next')
|
||||
git(self.seed, 'push', '-q', 'origin', 'main')
|
||||
return git(self.seed, 'rev-parse', 'HEAD')
|
||||
|
||||
|
||||
def real_pull(device, **extra):
|
||||
def core_update():
|
||||
before = git(device, 'rev-parse', 'HEAD')
|
||||
r = subprocess.run(['git', 'pull', '-q', '--rebase', '--autostash'],
|
||||
cwd=str(device), capture_output=True, text=True)
|
||||
after = git(device, 'rev-parse', 'HEAD')
|
||||
result = {'status': 'success' if r.returncode == 0 else 'error',
|
||||
'message': 'Code updated successfully.' if r.returncode == 0 else r.stderr,
|
||||
'restart_required': before != after, 'dependency_failures': []}
|
||||
result.update(extra)
|
||||
return result
|
||||
return core_update
|
||||
|
||||
|
||||
class FakeStore:
|
||||
def __init__(self, plugins_dir, versions, bump=(), fail=()):
|
||||
self.plugins_dir = str(plugins_dir)
|
||||
self.bump, self.fail = set(bump), set(fail)
|
||||
for pid, version in versions.items():
|
||||
d = Path(plugins_dir) / pid
|
||||
d.mkdir(parents=True)
|
||||
(d / 'manifest.json').write_text(json.dumps({'id': pid, 'version': version}))
|
||||
self.updated_calls = []
|
||||
|
||||
def list_installed_plugins(self):
|
||||
return [p.name for p in Path(self.plugins_dir).iterdir()]
|
||||
|
||||
def _get_local_git_info(self, path):
|
||||
return None
|
||||
|
||||
def update_plugin(self, pid):
|
||||
self.updated_calls.append(pid)
|
||||
if pid in self.fail:
|
||||
return False
|
||||
if pid in self.bump:
|
||||
path = Path(self.plugins_dir) / pid / 'manifest.json'
|
||||
m = json.loads(path.read_text())
|
||||
m['version'] = '9.9.9'
|
||||
path.write_text(json.dumps(m))
|
||||
return True
|
||||
|
||||
|
||||
class Harness:
|
||||
"""An AutoUpdater on a real clone.
|
||||
|
||||
``helper`` is whether the health check's path unit is active; ``pickup``
|
||||
whether the health check actually starts when asked. Starting it is
|
||||
simulated the way systemd does it: the request file is consumed
|
||||
(ExecStartPre) and the verifier marks the pending update "verifying".
|
||||
"""
|
||||
|
||||
def __init__(self, tmp, repo, *, helper=True, pickup=True, core_update=None, store=None,
|
||||
disk_free=10 ** 12, display_active=True, enabled=True, clock=NIGHT):
|
||||
self.sudo, self.restarts, self.now = [], [], clock
|
||||
self.state_at_handoff = None
|
||||
|
||||
def run(args, **kwargs):
|
||||
if args[0] == 'sudo':
|
||||
self.sudo.append(list(args))
|
||||
return subprocess.CompletedProcess(args, 0, stdout='', stderr='')
|
||||
return subprocess.run(args, **kwargs)
|
||||
|
||||
def sleep(seconds):
|
||||
updater = self.updater
|
||||
if self.state_at_handoff is None:
|
||||
self.state_at_handoff = au.load_state(updater.state_file)
|
||||
if pickup and updater.request_file.exists():
|
||||
updater.request_file.unlink()
|
||||
pending = au._read_json(updater.pending_file)
|
||||
pending['status'] = 'verifying'
|
||||
au._write_json(updater.pending_file, pending)
|
||||
|
||||
self.config = {'auto_update': {'enabled': enabled}, 'timezone': 'UTC'}
|
||||
cm = MagicMock()
|
||||
cm.load_config.return_value = self.config
|
||||
self.updater = au.AutoUpdater(
|
||||
config_manager=cm, core_update=core_update or real_pull(repo.device),
|
||||
store_manager=store, project_root=repo.device, clock=lambda: self.now,
|
||||
restart=self.restarts.append, run=run,
|
||||
service_active=lambda unit: display_active, helper_ready=lambda: helper,
|
||||
disk_free=lambda path: disk_free, sleep=sleep)
|
||||
|
||||
@property
|
||||
def state(self):
|
||||
return au.load_state(self.updater.state_file)
|
||||
|
||||
@property
|
||||
def pending(self):
|
||||
return au._read_json(self.updater.pending_file)
|
||||
|
||||
def write_pending(self, **fields):
|
||||
au._write_json(self.updater.pending_file, fields)
|
||||
|
||||
|
||||
def stub_updater(tmp_path, outcome='up_to_date', message='ok', store=None, enabled=True, clock=NIGHT):
|
||||
"""For scheduling and plugin tests: the core update is stubbed out."""
|
||||
cm = MagicMock()
|
||||
cm.load_config.return_value = {'auto_update': {'enabled': enabled}, 'timezone': 'UTC'}
|
||||
restarts = []
|
||||
updater = au.AutoUpdater(config_manager=cm, core_update=MagicMock(), store_manager=store,
|
||||
project_root=tmp_path, clock=lambda: clock,
|
||||
restart=restarts.append)
|
||||
updater.update_core = MagicMock(return_value={'outcome': outcome, 'message': message})
|
||||
return updater, restarts
|
||||
|
||||
|
||||
# -- scheduling ---------------------------------------------------------------
|
||||
|
||||
class TestIsDue:
|
||||
def test_not_before_next_due(self):
|
||||
assert not au.is_due(100, 200, 3)
|
||||
|
||||
def test_in_quiet_hours_once_due(self):
|
||||
assert au.is_due(200, 200, 3)
|
||||
|
||||
def test_waits_for_quiet_hours(self):
|
||||
assert not au.is_due(200, 100, 12)
|
||||
|
||||
def test_gives_up_waiting_after_grace(self):
|
||||
assert au.is_due(100 + au.QUIET_HOURS_GRACE_SECONDS, 100, 12)
|
||||
|
||||
|
||||
class TestTick:
|
||||
def test_disabled_never_runs(self, tmp_path):
|
||||
updater, _ = stub_updater(tmp_path, enabled=False)
|
||||
assert updater.tick() is False
|
||||
updater.update_core.assert_not_called()
|
||||
|
||||
def test_enabling_at_noon_waits_for_the_night(self, tmp_path):
|
||||
updater, _ = stub_updater(tmp_path, clock=NOON)
|
||||
assert updater.tick() is False
|
||||
updater.update_core.assert_not_called()
|
||||
assert au.load_state(updater.state_file)['next_due'] == NOON
|
||||
|
||||
def test_runs_when_due_and_schedules_a_week_out(self, tmp_path):
|
||||
updater, _ = stub_updater(tmp_path)
|
||||
assert updater.tick() is True
|
||||
state = au.load_state(updater.state_file)
|
||||
assert state['next_due'] == NIGHT + au.UPDATE_INTERVAL_SECONDS
|
||||
assert updater.tick() is False, "ran twice in one night"
|
||||
|
||||
def test_transient_error_retries_next_day(self, tmp_path):
|
||||
updater, _ = stub_updater(tmp_path, outcome='error', message='no network')
|
||||
updater.tick()
|
||||
state = au.load_state(updater.state_file)
|
||||
assert state['next_due'] == NIGHT + au.RETRY_AFTER_FAILURE_SECONDS
|
||||
assert state['alert']['message'] == 'no network'
|
||||
|
||||
def test_blocked_keeps_weekly_cadence_but_alerts(self, tmp_path):
|
||||
updater, _ = stub_updater(tmp_path, outcome='blocked', message='local edits')
|
||||
updater.tick()
|
||||
state = au.load_state(updater.state_file)
|
||||
assert state['next_due'] == NIGHT + au.UPDATE_INTERVAL_SECONDS
|
||||
assert 'local edits' in state['alert']['message']
|
||||
|
||||
|
||||
# -- checks before touching the checkout ---------------------------------------
|
||||
|
||||
@needs_git
|
||||
class TestPreflightRefuses:
|
||||
def test_nothing_new_is_up_to_date(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
h = Harness(tmp_path, repo)
|
||||
result = h.updater.run()
|
||||
assert result['core_outcome'] == 'up_to_date'
|
||||
assert result['status'] == 'success'
|
||||
assert h.sudo == [] and 'alert' not in h.state
|
||||
|
||||
def test_without_the_health_check_code_is_not_touched(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
old = repo.head()
|
||||
repo.publish()
|
||||
h = Harness(tmp_path, repo, helper=False)
|
||||
result = h.updater.run()
|
||||
assert result['core_outcome'] == 'blocked'
|
||||
assert repo.head() == old
|
||||
assert 'health check is set up' in result['core_message']
|
||||
assert h.state['alert']
|
||||
|
||||
def test_local_edits_are_never_stashed(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
old = repo.head()
|
||||
repo.publish()
|
||||
(repo.device / 'app.py').write_text('mine\n')
|
||||
h = Harness(tmp_path, repo)
|
||||
result = h.updater.run()
|
||||
assert result['core_outcome'] == 'blocked'
|
||||
assert 'app.py' in result['core_message']
|
||||
assert repo.head() == old
|
||||
assert (repo.device / 'app.py').read_text() == 'mine\n'
|
||||
assert git(repo.device, 'stash', 'list') == ''
|
||||
|
||||
def test_local_commits_block(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
repo.publish()
|
||||
(repo.device / 'extra.txt').write_text('x\n')
|
||||
git(repo.device, 'add', 'extra.txt')
|
||||
git(repo.device, 'commit', '-qm', 'local')
|
||||
mine = repo.head()
|
||||
h = Harness(tmp_path, repo)
|
||||
result = h.updater.run()
|
||||
assert result['core_outcome'] == 'blocked'
|
||||
assert 'local commit' in result['core_message']
|
||||
assert repo.head() == mine
|
||||
|
||||
def test_a_rebase_really_in_progress_blocks(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
repo.publish('v = 2\n')
|
||||
(repo.device / 'app.py').write_text('v = mine\n')
|
||||
git(repo.device, 'commit', '-qam', 'local')
|
||||
git(repo.device, 'fetch', '-q')
|
||||
stopped = subprocess.run(['git', '-c', 'user.name=t', '-c', 'user.email=t@example.com',
|
||||
'rebase', 'origin/main'], cwd=str(repo.device), capture_output=True, text=True)
|
||||
assert stopped.returncode != 0, "the rebase should stop on the conflict"
|
||||
result = Harness(tmp_path, repo).updater.run()
|
||||
assert result['core_outcome'] == 'blocked'
|
||||
assert 'rebase is in progress' in result['core_message']
|
||||
assert (repo.device / '.git' / 'rebase-merge').exists(), "a live rebase must be left alone"
|
||||
|
||||
def test_an_abandoned_rebase_is_cleared_and_the_update_runs(self, tmp_path):
|
||||
"""Found on a real device: a pull stopped mid-rebase a week earlier,
|
||||
HEAD was later checked out to a branch, and the leftover rebase
|
||||
directory would have blocked every automatic update forever."""
|
||||
repo = Repo(tmp_path)
|
||||
leftover = repo.device / '.git' / 'rebase-merge'
|
||||
leftover.mkdir()
|
||||
(leftover / 'head-name').write_text('refs/heads/main\n')
|
||||
new = repo.publish()
|
||||
result = Harness(tmp_path, repo).updater.run()
|
||||
assert not leftover.exists()
|
||||
assert result['core_outcome'] == 'verifying'
|
||||
assert repo.head() == new
|
||||
|
||||
def test_a_merge_in_progress_blocks(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
repo.publish()
|
||||
(repo.device / '.git' / 'MERGE_HEAD').write_text(repo.head() + '\n')
|
||||
result = Harness(tmp_path, repo).updater.run()
|
||||
assert result['core_outcome'] == 'blocked'
|
||||
assert 'merge is in progress' in result['core_message']
|
||||
|
||||
def test_low_disk_blocks(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
repo.publish()
|
||||
result = Harness(tmp_path, repo, disk_free=10 * 1024 * 1024).updater.run()
|
||||
assert result['core_outcome'] == 'blocked'
|
||||
assert 'disk space' in result['core_message']
|
||||
|
||||
def test_unreachable_origin_retries_tomorrow(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
git(repo.device, 'remote', 'set-url', 'origin', str(tmp_path / 'nowhere'))
|
||||
h = Harness(tmp_path, repo)
|
||||
result = h.updater.run()
|
||||
assert result['core_outcome'] == 'error'
|
||||
assert h.state['next_due'] == NIGHT + au.RETRY_AFTER_FAILURE_SECONDS
|
||||
|
||||
def test_version_already_rolled_back_is_not_retried(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
old = repo.head()
|
||||
bad = repo.publish()
|
||||
h = Harness(tmp_path, repo)
|
||||
au.save_state({'rolled_back_head': bad}, h.updater.state_file)
|
||||
result = h.updater.run()
|
||||
assert result['core_outcome'] == 'up_to_date'
|
||||
assert 'rolled back before' in result['core_message']
|
||||
assert repo.head() == old and h.sudo == []
|
||||
|
||||
|
||||
# -- the update and its hand-off to the health check ---------------------------
|
||||
|
||||
@needs_git
|
||||
class TestUpdateIsVerified:
|
||||
def test_pull_hands_off_to_the_health_check(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
old = repo.head()
|
||||
new = repo.publish()
|
||||
store = FakeStore(tmp_path / 'plugins', {'weather': '1.0.0'}, bump={'weather'})
|
||||
h = Harness(tmp_path, repo, store=store)
|
||||
result = h.updater.run()
|
||||
|
||||
assert result['core_outcome'] == 'verifying' and result['status'] == 'pending'
|
||||
assert repo.head() == new
|
||||
assert h.pending == {**h.pending, 'status': 'verifying', 'old_head': old, 'new_head': new,
|
||||
'display_was_active': True, 'dependency_failures': []}
|
||||
assert not h.updater.request_file.exists(), "the request is consumed when the check starts"
|
||||
assert h.sudo == [], "triggering the health check needs no privilege"
|
||||
assert h.restarts == [], "restarts belong to the health check, not the web process"
|
||||
assert store.updated_calls == [], "plugins must wait until the new code is verified"
|
||||
assert h.state['plugins_pending'] is True
|
||||
assert h.updater.verifier_copy.read_text() == au.VERIFIER_SOURCE.read_text()
|
||||
|
||||
def test_state_is_saved_before_the_health_check_restarts_this_process(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
repo.publish()
|
||||
h = Harness(tmp_path, repo)
|
||||
h.updater.run()
|
||||
assert h.state_at_handoff['last_result']['core_outcome'] == 'verifying'
|
||||
assert h.state_at_handoff['plugins_pending'] is True
|
||||
|
||||
def test_dependency_failures_reach_the_health_check(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
repo.publish()
|
||||
h = Harness(tmp_path, repo,
|
||||
core_update=real_pull(repo.device, dependency_failures=['requirements.txt']))
|
||||
h.updater.run()
|
||||
assert h.pending['dependency_failures'] == ['requirements.txt']
|
||||
|
||||
def test_a_health_check_that_never_starts_means_the_update_is_undone(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
old = repo.head()
|
||||
repo.publish()
|
||||
h = Harness(tmp_path, repo, pickup=False)
|
||||
result = h.updater.run()
|
||||
assert repo.head() == old
|
||||
assert result['core_outcome'] == 'blocked'
|
||||
assert 'did not start' in result['core_message']
|
||||
assert h.pending is None and not h.updater.request_file.exists()
|
||||
assert h.state['alert']
|
||||
|
||||
def test_failed_pull_retries_tomorrow(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
old = repo.head()
|
||||
repo.publish()
|
||||
h = Harness(tmp_path, repo,
|
||||
core_update=lambda: {'status': 'error', 'message': 'Update failed: network'})
|
||||
result = h.updater.run()
|
||||
assert result['core_outcome'] == 'error'
|
||||
assert repo.head() == old
|
||||
assert h.state['next_due'] == NIGHT + au.RETRY_AFTER_FAILURE_SECONDS
|
||||
|
||||
def test_failed_pull_that_moved_head_is_rolled_back(self, tmp_path):
|
||||
repo = Repo(tmp_path)
|
||||
old = repo.head()
|
||||
repo.publish()
|
||||
pull = real_pull(repo.device)
|
||||
|
||||
def half_failed():
|
||||
pull()
|
||||
return {'status': 'error', 'message': 'Update failed: interrupted'}
|
||||
result = Harness(tmp_path, repo, core_update=half_failed).updater.run()
|
||||
assert repo.head() == old
|
||||
assert 'rolled back' in result['core_message']
|
||||
|
||||
|
||||
# -- reporting the health check's outcome --------------------------------------
|
||||
|
||||
@needs_git
|
||||
class TestHealthCheckOutcome:
|
||||
def _after_update(self, tmp_path, **pending):
|
||||
repo = Repo(tmp_path)
|
||||
store = FakeStore(tmp_path / 'plugins', {'weather': '1.0.0'}, bump={'weather'})
|
||||
h = Harness(tmp_path, repo, store=store)
|
||||
au.save_state({'plugins_pending': True, 'next_due': NIGHT + au.UPDATE_INTERVAL_SECONDS,
|
||||
'last_result': {'core_outcome': 'verifying', 'status': 'pending',
|
||||
'plugins_deferred': True}}, h.updater.state_file)
|
||||
h.write_pending(old_head='a' * 40, new_head='b' * 40, created_at=NIGHT, **pending)
|
||||
return h, store
|
||||
|
||||
def test_waits_while_the_check_runs(self, tmp_path):
|
||||
h, store = self._after_update(tmp_path, status='verifying')
|
||||
h.now = NIGHT + 60
|
||||
assert h.updater.tick() is False
|
||||
assert store.updated_calls == [] and h.pending
|
||||
|
||||
def test_success_then_runs_the_deferred_plugins(self, tmp_path):
|
||||
h, store = self._after_update(tmp_path, status='success')
|
||||
assert h.updater.tick() is True
|
||||
result = h.state['last_result']
|
||||
assert result['core_outcome'] == 'updated' and result['status'] == 'success'
|
||||
assert result['plugins_updated'] == ['weather']
|
||||
assert h.restarts == ['ledmatrix']
|
||||
assert 'alert' not in h.state and h.pending is None
|
||||
|
||||
def test_rollback_alerts_and_remembers_the_bad_commit(self, tmp_path):
|
||||
h, store = self._after_update(tmp_path, status='rolled_back',
|
||||
reason='the web interface did not respond')
|
||||
h.updater.tick()
|
||||
assert 'rolled back' in h.state['alert']['message']
|
||||
assert 'the web interface did not respond' in h.state['alert']['message']
|
||||
assert h.state['rolled_back_head'] == 'b' * 40
|
||||
assert store.updated_calls == ['weather'], "the old code is healthy; plugins may update"
|
||||
|
||||
def test_failed_rollback_alerts_and_skips_plugins(self, tmp_path):
|
||||
h, store = self._after_update(tmp_path, status='rollback_failed',
|
||||
reason='display down', detail='reset failed')
|
||||
h.updater.tick()
|
||||
assert f'git reset --hard {"a" * 40}' in h.state['alert']['message']
|
||||
assert store.updated_calls == []
|
||||
assert h.state['plugins_pending'] is False
|
||||
|
||||
def test_a_check_that_never_reports_back_alerts(self, tmp_path):
|
||||
h, _ = self._after_update(tmp_path, status='verifying')
|
||||
h.now = NIGHT + au.VERIFY_LOST_SECONDS + 1
|
||||
h.updater.tick()
|
||||
assert h.state['last_result']['core_outcome'] == 'lost'
|
||||
assert 'never reported back' in h.state['alert']['message']
|
||||
|
||||
def test_dismissed_alert_stays_hidden_until_a_new_one(self, tmp_path):
|
||||
h, _ = self._after_update(tmp_path, status='rolled_back', reason='x')
|
||||
h.updater.tick()
|
||||
alert_id = h.state['alert']['id']
|
||||
|
||||
def status():
|
||||
return au.describe_status(h.config, state_file=h.updater.state_file,
|
||||
pending_file=h.updater.pending_file,
|
||||
setup_file=tmp_path / 'no-setup.json', helper=lambda: True)
|
||||
assert status()['alert_id'] == alert_id
|
||||
au.dismiss_alert(alert_id, h.updater.state_file)
|
||||
assert status()['alert'] is None
|
||||
state = h.state
|
||||
state['alert'] = {'id': 'newer', 'message': 'again'}
|
||||
au.save_state(state, h.updater.state_file)
|
||||
assert status()['alert'] == 'again'
|
||||
|
||||
|
||||
def test_describe_status_reports_setup(tmp_path):
|
||||
setup = tmp_path / 'setup.json'
|
||||
setup.write_text(json.dumps({'status': 'failed', 'message': 'not root'}))
|
||||
kwargs = dict(state={}, state_file=tmp_path / 's.json', pending_file=tmp_path / 'p.json',
|
||||
setup_file=setup)
|
||||
on = au.describe_status(ON, helper=lambda: False, **kwargs)
|
||||
assert on['verifier_installed'] is False
|
||||
assert (on['setup_status'], on['setup_message']) == ('failed', 'not root')
|
||||
|
||||
def never(): raise AssertionError("systemctl asked while updates are off")
|
||||
assert au.describe_status(OFF, helper=never, **kwargs)['verifier_installed'] is None
|
||||
|
||||
|
||||
# -- setting it up from the web UI ----------------------------------------------
|
||||
|
||||
class TestSetupFromTheWebUI:
|
||||
def _host(self, monkeypatch, ready=False, active=True, restart_ok=True):
|
||||
restarts = []
|
||||
monkeypatch.setattr(au, 'helper_ready', lambda: ready)
|
||||
monkeypatch.setattr(au, '_service_active', lambda unit: active)
|
||||
monkeypatch.setattr(au, 'restart_service', lambda unit: restarts.append(unit) or restart_ok)
|
||||
return restarts
|
||||
|
||||
def test_switching_on_restarts_the_display_to_finish_setup(self, monkeypatch):
|
||||
restarts = self._host(monkeypatch)
|
||||
note = au.start_setup_if_needed(False, ON)
|
||||
assert restarts == ['ledmatrix']
|
||||
assert 'display is restarting' in note
|
||||
|
||||
@pytest.mark.parametrize('was_enabled, config, ready', [
|
||||
(True, ON, False), # already on: an unrelated save must not restart the display
|
||||
(False, OFF, False), # still off
|
||||
(False, ON, True), # already set up
|
||||
])
|
||||
def test_nothing_to_do(self, monkeypatch, was_enabled, config, ready):
|
||||
restarts = self._host(monkeypatch, ready=ready)
|
||||
assert au.start_setup_if_needed(was_enabled, config) is None
|
||||
assert restarts == []
|
||||
|
||||
def test_a_stopped_display_is_not_started(self, monkeypatch):
|
||||
restarts = self._host(monkeypatch, active=False)
|
||||
assert 'next time the display service starts' in au.start_setup_if_needed(False, ON)
|
||||
assert restarts == []
|
||||
|
||||
def test_a_failed_restart_says_so(self, monkeypatch):
|
||||
self._host(monkeypatch, restart_ok=False)
|
||||
assert 'Could not restart' in au.start_setup_if_needed(False, ON)
|
||||
|
||||
|
||||
# -- plugins --------------------------------------------------------------------
|
||||
|
||||
class TestPlugins:
|
||||
def test_nothing_changed_restarts_nothing(self, tmp_path):
|
||||
store = FakeStore(tmp_path / 'plugins', {'clock': '1.0.0'})
|
||||
updater, restarts = stub_updater(tmp_path, store=store)
|
||||
updater.run()
|
||||
assert store.updated_calls == ['clock']
|
||||
assert restarts == []
|
||||
|
||||
def test_plugin_update_restarts_display_only(self, tmp_path):
|
||||
store = FakeStore(tmp_path / 'plugins', {'clock': '1.0.0', 'weather': '1.0.0'},
|
||||
bump={'weather'})
|
||||
updater, restarts = stub_updater(tmp_path, store=store)
|
||||
result = updater.run()
|
||||
assert result['plugins_updated'] == ['weather']
|
||||
assert restarts == ['ledmatrix']
|
||||
|
||||
def test_plugin_failure_alerts_but_keeps_weekly_cadence(self, tmp_path):
|
||||
store = FakeStore(tmp_path / 'plugins', {'zip-only': '1.0.0'}, fail={'zip-only'})
|
||||
updater, _ = stub_updater(tmp_path, store=store)
|
||||
result = updater.run()
|
||||
assert result['plugins_failed'] == ['zip-only'] and result['status'] == 'error'
|
||||
state = au.load_state(updater.state_file)
|
||||
assert 'zip-only' in state['alert']['message']
|
||||
assert state['next_due'] == NIGHT + au.UPDATE_INTERVAL_SECONDS
|
||||
|
||||
def test_local_only_plugins_are_left_alone(self, tmp_path):
|
||||
store = FakeStore(tmp_path / 'plugins', {'mine': '1.0.0'})
|
||||
path = tmp_path / 'plugins' / 'mine' / 'manifest.json'
|
||||
path.write_text(json.dumps({'id': 'mine', 'version': '1.0.0', 'local_only': True}))
|
||||
updater, _ = stub_updater(tmp_path, store=store)
|
||||
updater.run()
|
||||
assert store.updated_calls == []
|
||||
|
||||
|
||||
def test_restart_service_skips_a_stopped_unit(monkeypatch):
|
||||
calls = []
|
||||
|
||||
def run(args, **kwargs):
|
||||
calls.append(args)
|
||||
return MagicMock(stdout='inactive\n', returncode=3, stderr='')
|
||||
monkeypatch.setattr(au.subprocess, 'run', run)
|
||||
assert au.restart_service('ledmatrix') is False
|
||||
assert calls == [['systemctl', 'is-active', 'ledmatrix']], (
|
||||
"a display the user stopped must not be started by an update")
|
||||
|
||||
|
||||
def test_core_update_names_the_requirement_files_that_failed():
|
||||
"""The health check refuses code whose dependencies did not install, so
|
||||
the failure has to arrive as data, not only as a sentence in the message."""
|
||||
from web_interface.blueprints import api_v3 as pkg
|
||||
from web_interface.blueprints.api_v3 import system
|
||||
|
||||
heads = iter(['aaa111\n', 'bbb222\n'])
|
||||
|
||||
def run(args, **kwargs):
|
||||
if args[:2] == ['git', 'rev-parse'] and args[-1] == 'HEAD':
|
||||
out = next(heads, 'bbb222\n')
|
||||
elif args[:2] == ['git', 'diff']:
|
||||
out = 'requirements.txt\n'
|
||||
elif '@{u}' in args:
|
||||
out = 'origin/main\n'
|
||||
else:
|
||||
out = ''
|
||||
return subprocess.CompletedProcess(args, 0, stdout=out, stderr='')
|
||||
|
||||
pkg.api_v3.plugin_store_manager = None
|
||||
failed_install = subprocess.CompletedProcess([], 1, stdout='', stderr='boom')
|
||||
with patch.object(pkg.subprocess, 'run', run), \
|
||||
patch.object(system, '_pip_install_requirements', return_value=failed_install):
|
||||
result = system.perform_core_update()
|
||||
assert result['status'] == 'success'
|
||||
assert result['dependency_failures'] == ['requirements.txt']
|
||||
|
||||
|
||||
# -- web routes -------------------------------------------------------------------
|
||||
|
||||
@pytest.fixture
|
||||
def api_client(monkeypatch):
|
||||
from flask import Flask
|
||||
from web_interface.blueprints.api_v3 import api_v3
|
||||
app = Flask(__name__)
|
||||
app.register_blueprint(api_v3, url_prefix='/api/v3')
|
||||
cm = MagicMock()
|
||||
cm.load_config.return_value = {'timezone': 'UTC', 'auto_update': {'enabled': True}}
|
||||
cm.get_raw_file_content.return_value = {}
|
||||
cm.save_config_atomic.return_value = MagicMock(status=MagicMock(value='success'), message=None)
|
||||
api_v3.config_manager = cm
|
||||
api_v3.plugin_manager = MagicMock(plugins={})
|
||||
# Never restart a real display from a test run.
|
||||
setup_calls = []
|
||||
monkeypatch.setattr(au, 'start_setup_if_needed',
|
||||
lambda was_enabled, config: setup_calls.append(was_enabled) or None)
|
||||
return app.test_client(), cm, setup_calls
|
||||
|
||||
|
||||
class TestStatusRoutes:
|
||||
def test_status_carries_the_alert(self, api_client, monkeypatch):
|
||||
client, _, _ = api_client
|
||||
monkeypatch.setattr(au, 'describe_status', lambda config: {'alert': 'rolled back', 'alert_id': '7'})
|
||||
data = client.get('/api/v3/system/auto-update').get_json()['data']
|
||||
assert data == {'alert': 'rolled back', 'alert_id': '7'}
|
||||
|
||||
@pytest.mark.parametrize('body', [{}, [1], 'x', 5])
|
||||
def test_dismiss_needs_an_id_in_a_json_object(self, api_client, body):
|
||||
client, _, _ = api_client
|
||||
assert client.post('/api/v3/system/auto-update/dismiss', json=body).status_code == 400
|
||||
|
||||
def test_dismiss(self, api_client, monkeypatch):
|
||||
client, _, _ = api_client
|
||||
calls = []
|
||||
monkeypatch.setattr(au, 'dismiss_alert', calls.append)
|
||||
assert client.post('/api/v3/system/auto-update/dismiss', json={'alert_id': '7'}).status_code == 200
|
||||
assert calls == ['7']
|
||||
|
||||
|
||||
class TestSettingsSave:
|
||||
def _saved(self, cm):
|
||||
return cm.save_config_atomic.call_args[0][0]
|
||||
|
||||
def test_checked_toggle_saves_enabled(self, api_client):
|
||||
client, cm, setup_calls = api_client
|
||||
cm.load_config.return_value = {'timezone': 'UTC'}
|
||||
resp = client.post('/api/v3/config/main', json={'timezone': 'UTC', 'auto_update_enabled': 'true'})
|
||||
assert resp.status_code == 200
|
||||
saved = self._saved(cm)
|
||||
assert saved['auto_update'] == {'enabled': True}
|
||||
assert 'auto_update_enabled' not in saved, "the form field must not leak into config.json"
|
||||
assert setup_calls == [False], "setup is told the toggle was previously off"
|
||||
|
||||
def test_unchecked_toggle_on_general_save_disables(self, api_client):
|
||||
client, cm, setup_calls = api_client
|
||||
resp = client.post('/api/v3/config/main', json={'timezone': 'UTC'})
|
||||
assert resp.status_code == 200
|
||||
assert self._saved(cm)['auto_update'] == {'enabled': False}
|
||||
assert setup_calls == [True]
|
||||
|
||||
def test_setup_note_reaches_the_user(self, api_client, monkeypatch):
|
||||
client, cm, _ = api_client
|
||||
cm.load_config.return_value = {'timezone': 'UTC'}
|
||||
monkeypatch.setattr(au, 'start_setup_if_needed',
|
||||
lambda was_enabled, config: 'Finishing automatic update setup: the display is restarting.')
|
||||
body = client.post('/api/v3/config/main',
|
||||
json={'timezone': 'UTC', 'auto_update_enabled': 'true'}).get_json()
|
||||
assert 'the display is restarting' in body['message']
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Installing the automatic-update health check from the display service.
|
||||
|
||||
src/auto_update_setup.py is how a user who never opens a terminal gets updates
|
||||
with a safety net: it runs as root inside the display service and writes
|
||||
systemd units. So it has to install exactly the right thing when asked, do
|
||||
nothing when not asked, refuse templates that would run as anyone but the web
|
||||
user, and say why whenever it could not finish.
|
||||
"""
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from src import auto_update_setup as aus # noqa: E402
|
||||
|
||||
ON = {'auto_update': {'enabled': True}}
|
||||
|
||||
|
||||
class FakeSystemctl:
|
||||
def __init__(self, fail=(), activates=True):
|
||||
self.calls = []
|
||||
self.fail = set(fail)
|
||||
self.activates = activates
|
||||
self.active = False
|
||||
|
||||
def __call__(self, args, **kwargs):
|
||||
assert args[0] == 'systemctl', args
|
||||
self.calls.append(args[1:])
|
||||
verb = args[1]
|
||||
if verb == 'is-active':
|
||||
return subprocess.CompletedProcess(args, 0 if self.active else 3,
|
||||
stdout='active\n' if self.active else 'inactive\n', stderr='')
|
||||
if verb in self.fail:
|
||||
return subprocess.CompletedProcess(args, 1, stdout='', stderr=f'{verb} refused')
|
||||
if verb in ('restart', 'enable') and self.activates and (verb == 'restart' or '--now' in args):
|
||||
self.active = True
|
||||
return subprocess.CompletedProcess(args, 0, stdout='', stderr='')
|
||||
|
||||
|
||||
def project(tmp_path, user='hdpi', workdir=None, name='LEDMatrix'):
|
||||
root = tmp_path / name
|
||||
(root / 'systemd').mkdir(parents=True)
|
||||
for name in aus.UNITS:
|
||||
shutil.copy(ROOT / 'systemd' / name, root / 'systemd' / name)
|
||||
etc = tmp_path / 'etc'
|
||||
etc.mkdir()
|
||||
(etc / aus.WEB_UNIT).write_text(
|
||||
f'[Service]\nUser={user}\nWorkingDirectory={workdir or root}\n', encoding='utf-8')
|
||||
return root, etc
|
||||
|
||||
|
||||
def setup(root, etc, systemctl, root_user=True):
|
||||
return aus.UpdateHelperSetup(project_root=root, systemd_dir=etc, run=systemctl,
|
||||
is_root=lambda: root_user,
|
||||
lookup_ids=lambda user: None if user == 'ghost' else (1000, 1000),
|
||||
clock=lambda: 1234.0)
|
||||
|
||||
|
||||
def result(root):
|
||||
return json.loads((root / aus.RESULT_REL).read_text())
|
||||
|
||||
|
||||
def test_does_nothing_while_updates_are_off(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
systemctl = FakeSystemctl()
|
||||
assert setup(root, etc, systemctl).ensure({'auto_update': {'enabled': False}}) is None
|
||||
assert systemctl.calls == []
|
||||
assert not (etc / aus.SERVICE_UNIT).exists()
|
||||
assert not (root / aus.RESULT_REL).exists()
|
||||
|
||||
|
||||
def test_installs_both_units_for_the_web_user(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
systemctl = FakeSystemctl()
|
||||
out = setup(root, etc, systemctl).ensure(ON)
|
||||
|
||||
assert out['status'] == 'installed' and result(root)['status'] == 'installed'
|
||||
service = (etc / aus.SERVICE_UNIT).read_text()
|
||||
assert 'User=hdpi' in service
|
||||
assert f'ExecStart=/usr/bin/python3 "{root}/data/auto_update_verifier.py" "{root}"' in service
|
||||
assert f'PathExists={root}/data/auto_update_verify.request' in (etc / aus.PATH_UNIT).read_text()
|
||||
assert '__' not in service
|
||||
assert ['daemon-reload'] in systemctl.calls
|
||||
assert ['enable', aus.PATH_UNIT] in systemctl.calls
|
||||
assert systemctl.active
|
||||
|
||||
|
||||
def test_second_start_changes_nothing(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
systemctl = FakeSystemctl()
|
||||
setup(root, etc, systemctl).ensure(ON)
|
||||
first = result(root)
|
||||
systemctl.calls.clear()
|
||||
setup(root, etc, systemctl).ensure(ON)
|
||||
assert systemctl.calls == [['is-active', aus.PATH_UNIT], ['is-active', aus.PATH_UNIT]]
|
||||
assert result(root) == first, "an unchanged setup must not rewrite its result every boot"
|
||||
|
||||
|
||||
def test_a_changed_template_is_reinstalled(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
systemctl = FakeSystemctl()
|
||||
setup(root, etc, systemctl).ensure(ON)
|
||||
template = root / 'systemd' / aus.SERVICE_UNIT
|
||||
template.write_text(template.read_text() + '\n# newer\n')
|
||||
systemctl.calls.clear()
|
||||
assert setup(root, etc, systemctl).ensure(ON)['message'] == 'Installed the update health check.'
|
||||
assert (etc / aus.SERVICE_UNIT).read_text().endswith('# newer\n')
|
||||
assert ['daemon-reload'] in systemctl.calls
|
||||
|
||||
|
||||
def test_a_template_that_would_not_run_as_the_web_user_is_refused(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
template = root / 'systemd' / aus.SERVICE_UNIT
|
||||
template.write_text(template.read_text().replace('User=__USER__', 'User=root'))
|
||||
systemctl = FakeSystemctl()
|
||||
out = setup(root, etc, systemctl).ensure(ON)
|
||||
assert out['status'] == 'failed' and 'refusing' in out['message']
|
||||
assert not (etc / aus.SERVICE_UNIT).exists()
|
||||
assert systemctl.calls == []
|
||||
|
||||
|
||||
def test_a_path_unit_that_starts_something_else_is_refused(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
template = root / 'systemd' / aus.PATH_UNIT
|
||||
template.write_text(template.read_text().replace('Unit=ledmatrix-update-verify.service', 'Unit=other.service'))
|
||||
out = setup(root, etc, FakeSystemctl()).ensure(ON)
|
||||
assert out['status'] == 'failed' and not (etc / aus.PATH_UNIT).exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('case, expected', [
|
||||
('not_root', 'not running as root'),
|
||||
('no_web_unit', 'not installed'),
|
||||
('other_folder', 'runs from'),
|
||||
('bad_user', 'not a usable account'),
|
||||
])
|
||||
def test_reports_why_it_could_not_install(tmp_path, case, expected):
|
||||
root, etc = project(tmp_path, user='ghost' if case == 'bad_user' else 'hdpi',
|
||||
workdir=tmp_path / 'elsewhere' if case == 'other_folder' else None)
|
||||
if case == 'no_web_unit':
|
||||
(etc / aus.WEB_UNIT).unlink()
|
||||
out = setup(root, etc, FakeSystemctl(), root_user=case != 'not_root').ensure(ON)
|
||||
assert out['status'] == 'failed' and expected in out['message']
|
||||
assert result(root)['message'] == out['message']
|
||||
assert not (etc / aus.SERVICE_UNIT).exists()
|
||||
|
||||
|
||||
def test_a_path_unit_that_will_not_start_is_a_failure(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
out = setup(root, etc, FakeSystemctl(activates=False)).ensure(ON)
|
||||
assert out['status'] == 'failed' and 'did not start' in out['message']
|
||||
|
||||
|
||||
def test_systemctl_errors_are_reported(tmp_path):
|
||||
root, etc = project(tmp_path)
|
||||
out = setup(root, etc, FakeSystemctl(fail={'daemon-reload'})).ensure(ON)
|
||||
assert out['status'] == 'failed' and 'daemon-reload refused' in out['message']
|
||||
|
||||
|
||||
def test_not_a_systemd_host_is_left_alone(tmp_path):
|
||||
root, _ = project(tmp_path)
|
||||
systemctl = FakeSystemctl()
|
||||
assert setup(root, tmp_path / 'no-etc', systemctl).ensure(ON) is None
|
||||
assert systemctl.calls == []
|
||||
|
||||
|
||||
def test_a_folder_with_spaces_is_quoted_in_the_commands(tmp_path):
|
||||
root, etc = project(tmp_path, name='LED Matrix')
|
||||
assert setup(root, etc, FakeSystemctl()).ensure(ON)['status'] == 'installed'
|
||||
service = (etc / aus.SERVICE_UNIT).read_text()
|
||||
assert f'ExecStartPre=/bin/rm -f "{root}/data/auto_update_verify.request"' in service
|
||||
assert f'ExecStart=/usr/bin/python3 "{root}/data/auto_update_verifier.py" "{root}"' in service
|
||||
|
||||
|
||||
def test_a_folder_name_systemd_would_reinterpret_is_refused(tmp_path):
|
||||
root, etc = project(tmp_path, name='LED%Matrix')
|
||||
systemctl = FakeSystemctl()
|
||||
out = setup(root, etc, systemctl).ensure(ON)
|
||||
assert out['status'] == 'failed' and 'systemd cannot use' in out['message']
|
||||
assert not (etc / aus.SERVICE_UNIT).exists() and systemctl.calls == []
|
||||
|
||||
|
||||
def test_installer_sets_the_toggle_without_a_half_written_config(tmp_path):
|
||||
"""first_time_install.sh may run while the display service watches
|
||||
config.json; the file must be replaced whole, never truncated in place."""
|
||||
import re
|
||||
text = (ROOT / 'first_time_install.sh').read_text(encoding='utf-8').replace('\r\n', '\n')
|
||||
script = next(s for s in re.findall(r"<<'PY'\n(.*?)\nPY\n", text, re.S) if 'auto_update' in s)
|
||||
assert 'os.replace(' in script
|
||||
config = tmp_path / 'config.json'
|
||||
config.write_text(json.dumps({'timezone': 'UTC', 'auto_update': {'enabled': False}}))
|
||||
run = subprocess.run([sys.executable, '-', str(config), '1'], input=script, text=True,
|
||||
capture_output=True)
|
||||
assert run.returncode == 0, run.stderr
|
||||
assert json.loads(config.read_text()) == {'timezone': 'UTC', 'auto_update': {'enabled': True}}
|
||||
assert [p.name for p in tmp_path.iterdir()] == ['config.json'], "a temp file was left behind"
|
||||
|
||||
|
||||
def test_the_result_is_kept_when_it_cannot_be_given_to_the_web_user(tmp_path, monkeypatch):
|
||||
"""Caught by CI, which runs as a non-root Linux user: chown needs root, and
|
||||
a failed chown used to discard the result, so the General tab never
|
||||
learned whether setup worked."""
|
||||
def refuse(*args):
|
||||
raise PermissionError('Operation not permitted')
|
||||
monkeypatch.setattr(aus.os, 'chown', refuse, raising=False)
|
||||
root, etc = project(tmp_path)
|
||||
out = setup(root, etc, FakeSystemctl()).ensure(ON)
|
||||
assert out['status'] == 'installed'
|
||||
assert result(root) == out
|
||||
@@ -0,0 +1,248 @@
|
||||
"""The automatic update's health check and rollback (scripts/utils/auto_update_verify.py).
|
||||
|
||||
This is what stands between an unattended update and a device that no longer
|
||||
works, so each way an update can fail is exercised against a real git repo:
|
||||
the new commit is checked out, the services are "restarted", and whether they
|
||||
come up healthy depends on which commit they were restarted onto. Only
|
||||
systemd, sudo and the HTTP check are faked.
|
||||
"""
|
||||
import importlib.util
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
'auto_update_verify', ROOT / 'scripts' / 'utils' / 'auto_update_verify.py')
|
||||
av = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(av)
|
||||
|
||||
pytestmark = pytest.mark.skipif(shutil.which('git') is None, reason='git not installed')
|
||||
|
||||
|
||||
def git(cwd, *args):
|
||||
result = subprocess.run(['git', '-c', 'user.name=t', '-c', 'user.email=t@example.com', *args],
|
||||
cwd=str(cwd), capture_output=True, text=True)
|
||||
assert result.returncode == 0, result.stderr
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def done(args, stdout='', rc=0):
|
||||
return subprocess.CompletedProcess(args, rc, stdout=stdout, stderr='' if rc == 0 else 'failed')
|
||||
|
||||
|
||||
def updated_repo(tmp_path, new_requirements=False):
|
||||
"""A checkout on a new commit, with the commit it came from."""
|
||||
repo = tmp_path / 'LEDMatrix'
|
||||
repo.mkdir()
|
||||
git(repo, 'init', '-q', '-b', 'main')
|
||||
(repo / 'app.py').write_text('v = 1\n')
|
||||
(repo / 'requirements.txt').write_text('requests\n')
|
||||
git(repo, 'add', '.')
|
||||
git(repo, 'commit', '-qm', 'old')
|
||||
old = git(repo, 'rev-parse', 'HEAD')
|
||||
(repo / 'app.py').write_text('v = 2\n')
|
||||
if new_requirements:
|
||||
(repo / 'requirements.txt').write_text('requests\nnewthing\n')
|
||||
git(repo, 'commit', '-qam', 'new')
|
||||
return repo, old, git(repo, 'rev-parse', 'HEAD')
|
||||
|
||||
|
||||
class FakeHost:
|
||||
"""systemd, sudo and the web interface.
|
||||
|
||||
Services run whatever commit was checked out when they were last
|
||||
restarted; ``failure`` says how they misbehave on the new commit
|
||||
("display_down", "web_down", "crash_loop") or on any commit ("always").
|
||||
"""
|
||||
|
||||
def __init__(self, repo, bad_head, failure=None, pip_ok=True, restart_failures=0, count_readable=True):
|
||||
self.repo, self.bad_head, self.failure, self.pip_ok = repo, bad_head, failure, pip_ok
|
||||
self.restart_failures = restart_failures # how many restart commands fail, first to last
|
||||
self.count_readable = count_readable
|
||||
self.running_head = None # not restarted yet: still the old, healthy code
|
||||
self.restarts = [] # (unit, commit it was restarted onto)
|
||||
self.pip_installs = []
|
||||
self.now = 0.0
|
||||
self.nrestarts = 0
|
||||
|
||||
def broken(self, kind):
|
||||
if self.running_head is None:
|
||||
return False
|
||||
return self.failure == 'always' or (self.running_head == self.bad_head and self.failure == kind)
|
||||
|
||||
def run(self, args, **kwargs):
|
||||
if args[0] == 'git':
|
||||
return subprocess.run(args, **kwargs)
|
||||
if args[:2] == ['systemctl', 'is-active']:
|
||||
return done(args, 'failed\n' if self.broken('display_down') else 'active\n')
|
||||
if args[:2] == ['systemctl', 'show']:
|
||||
if not self.count_readable:
|
||||
return done(args, rc=1)
|
||||
if self.broken('crash_loop'):
|
||||
self.nrestarts += 1
|
||||
return done(args, f'{self.nrestarts}\n')
|
||||
if args[0] == 'sudo' and any(a.endswith('safe_pip_install.sh') for a in args):
|
||||
self.pip_installs.append(args[-1])
|
||||
return done(args, rc=0 if self.pip_ok else 1)
|
||||
if args[:4] == ['sudo', '-n', 'systemctl', 'restart']:
|
||||
if self.restart_failures:
|
||||
self.restart_failures -= 1
|
||||
return done(args, rc=1) # the old process keeps running
|
||||
self.running_head = git(self.repo, 'rev-parse', 'HEAD')
|
||||
self.restarts.append((args[4], self.running_head))
|
||||
return done(args)
|
||||
raise AssertionError(f'unexpected command: {args}')
|
||||
|
||||
def web_responds(self):
|
||||
return not self.broken('web_down')
|
||||
|
||||
def sleep(self, seconds):
|
||||
self.now += seconds
|
||||
|
||||
def verifier(self):
|
||||
return av.Verifier(self.repo, run=self.run, sleep=self.sleep, clock=lambda: self.now,
|
||||
web_responds=self.web_responds, log=lambda msg: None)
|
||||
|
||||
|
||||
def check(tmp_path, failure=None, new_requirements=False, pip_ok=True, restart_failures=0,
|
||||
count_readable=True, **pending):
|
||||
repo, old, new = updated_repo(tmp_path, new_requirements)
|
||||
fields = {'status': 'pending', 'old_head': old, 'new_head': new,
|
||||
'display_was_active': True, 'dependency_failures': []}
|
||||
fields.update(pending)
|
||||
av.write_pending(av.pending_path(repo), fields)
|
||||
host = FakeHost(repo, new, failure, pip_ok, restart_failures, count_readable)
|
||||
code = host.verifier().verify()
|
||||
result = av.read_pending(av.pending_path(repo))
|
||||
return code, result, host, git(repo, 'rev-parse', 'HEAD'), old, new
|
||||
|
||||
|
||||
def test_a_healthy_update_is_kept(tmp_path):
|
||||
code, result, host, head, old, new = check(tmp_path)
|
||||
assert code == 0 and result['status'] == 'success'
|
||||
assert head == new
|
||||
assert host.restarts == [('ledmatrix.service', new), ('ledmatrix-web.service', new)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize('failure, reason', [
|
||||
('display_down', 'the display service did not stay running'),
|
||||
('web_down', 'the web interface did not respond'),
|
||||
('crash_loop', 'the display service kept restarting'),
|
||||
])
|
||||
def test_an_unhealthy_update_is_rolled_back(tmp_path, failure, reason):
|
||||
code, result, host, head, old, new = check(tmp_path, failure)
|
||||
assert code == 0
|
||||
assert result['status'] == 'rolled_back' and result['reason'] == reason
|
||||
assert head == old
|
||||
assert host.restarts[-2:] == [('ledmatrix.service', old), ('ledmatrix-web.service', old)]
|
||||
|
||||
|
||||
def test_a_stopped_display_is_not_started(tmp_path):
|
||||
code, result, host, head, old, new = check(tmp_path, display_was_active=False)
|
||||
assert result['status'] == 'success'
|
||||
assert [unit for unit, _ in host.restarts] == ['ledmatrix-web.service']
|
||||
|
||||
|
||||
def test_failed_dependencies_roll_back_before_anything_restarts_onto_them(tmp_path):
|
||||
code, result, host, head, old, new = check(tmp_path, dependency_failures=['requirements.txt'])
|
||||
assert result['status'] == 'rolled_back'
|
||||
assert 'requirements.txt' in result['reason']
|
||||
assert head == old
|
||||
assert all(commit == old for _, commit in host.restarts), host.restarts
|
||||
|
||||
|
||||
def test_rollback_reinstalls_the_previous_dependencies(tmp_path):
|
||||
code, result, host, head, old, new = check(tmp_path, 'display_down', new_requirements=True)
|
||||
assert result['status'] == 'rolled_back' and result['detail'] is None
|
||||
assert host.pip_installs == [str(tmp_path / 'LEDMatrix' / 'requirements.txt')]
|
||||
|
||||
|
||||
def test_a_failed_dependency_reinstall_is_reported(tmp_path):
|
||||
code, result, host, head, old, new = check(tmp_path, 'display_down', new_requirements=True,
|
||||
pip_ok=False)
|
||||
assert result['status'] == 'rolled_back' and head == old
|
||||
assert 'Install Base Requirements' in result['detail']
|
||||
|
||||
|
||||
def test_still_broken_after_rolling_back_is_rollback_failed(tmp_path):
|
||||
code, result, host, head, old, new = check(tmp_path, 'always')
|
||||
assert code == 1 and result['status'] == 'rollback_failed'
|
||||
assert 'still unhealthy after rolling back' in result['detail']
|
||||
|
||||
|
||||
def test_a_failed_restart_is_not_mistaken_for_a_healthy_update(tmp_path):
|
||||
"""When the restart command itself fails the old process keeps answering,
|
||||
so checking it would pass an update that never started."""
|
||||
code, result, host, head, old, new = check(tmp_path, restart_failures=1)
|
||||
assert result['status'] == 'rolled_back'
|
||||
assert result['reason'] == 'restarting the services failed'
|
||||
assert head == old
|
||||
|
||||
|
||||
def test_restarts_that_keep_failing_after_rollback_are_rollback_failed(tmp_path):
|
||||
code, result, host, head, old, new = check(tmp_path, restart_failures=100)
|
||||
assert code == 1 and result['status'] == 'rollback_failed'
|
||||
assert 'restarting the services failed' in result['detail']
|
||||
|
||||
|
||||
def test_an_unreadable_restart_count_is_never_called_stable(tmp_path):
|
||||
"""With no restart count a crash loop looks healthy between attempts."""
|
||||
code, result, host, head, old, new = check(tmp_path, count_readable=False)
|
||||
assert result['status'] != 'success'
|
||||
assert 'restart count could not be read' in result['reason']
|
||||
|
||||
|
||||
def test_nothing_pending_does_nothing(tmp_path):
|
||||
repo, _, _ = updated_repo(tmp_path)
|
||||
host = FakeHost(repo, None)
|
||||
assert host.verifier().verify() == 0
|
||||
assert host.restarts == []
|
||||
|
||||
|
||||
def test_a_crash_is_recorded_not_left_as_running(tmp_path, monkeypatch):
|
||||
repo, old, new = updated_repo(tmp_path)
|
||||
av.write_pending(av.pending_path(repo), {'status': 'pending', 'old_head': old, 'new_head': new})
|
||||
|
||||
def boom(self):
|
||||
raise RuntimeError('boom')
|
||||
monkeypatch.setattr(av.Verifier, 'verify', boom)
|
||||
assert av.main(['auto_update_verify.py', str(repo)]) == 1
|
||||
result = av.read_pending(av.pending_path(repo))
|
||||
assert result['status'] == 'rollback_failed' and result['detail'] == 'boom'
|
||||
|
||||
|
||||
def test_units_installers_and_updater_agree():
|
||||
"""The request file, the copied verifier, the unit names and the places
|
||||
that install them must all line up, or the health check silently never
|
||||
starts on real devices -- and code updates stay paused."""
|
||||
from web_interface import auto_update as au
|
||||
from src import auto_update_setup as aus
|
||||
from src.startup_validator import StartupValidator
|
||||
|
||||
service = (ROOT / 'systemd' / aus.SERVICE_UNIT).read_text(encoding='utf-8')
|
||||
path = (ROOT / 'systemd' / aus.PATH_UNIT).read_text(encoding='utf-8')
|
||||
request = f'__PROJECT_ROOT_DIR__/{au.REQUEST_REL.as_posix()}'
|
||||
assert f'PathExists={request}' in path
|
||||
assert f'Unit={aus.SERVICE_UNIT}' in path
|
||||
assert f'ExecStartPre=/bin/rm -f "{request}"' in service, "the request must be consumed or the path unit re-fires"
|
||||
assert f'"__PROJECT_ROOT_DIR__/{au.VERIFIER_COPY_REL.as_posix()}" "__PROJECT_ROOT_DIR__"' in service
|
||||
assert au.PENDING_REL.name == av.PENDING_NAME
|
||||
assert au.PATH_UNIT == aus.PATH_UNIT and au.SETUP_RESULT_REL == aus.RESULT_REL
|
||||
|
||||
for installer in ('scripts/install/install_web_service.sh', 'scripts/install/install_service.sh'):
|
||||
text = (ROOT / installer).read_text(encoding='utf-8')
|
||||
assert 'ledmatrix-update-verify.service ledmatrix-update-verify.path' in text, installer
|
||||
assert 'enable --now ledmatrix-update-verify.path' in text, installer
|
||||
for unit in aus.UNITS:
|
||||
assert (f'systemd/{unit}', f'/etc/systemd/system/{unit}') in StartupValidator._UNITS
|
||||
|
||||
# Triggering takes no privilege any more; no sudoers rule should linger.
|
||||
for sudoers in ('scripts/install/configure_web_sudo.sh', 'first_time_install.sh'):
|
||||
assert not re.search(r'NOPASSWD:.*update-verify', (ROOT / sudoers).read_text(encoding='utf-8')), sudoers
|
||||
@@ -387,9 +387,81 @@ class TestFrameStatsPercentiles:
|
||||
|
||||
def test_log_frame_rate_emits_the_line_and_clears_the_window(self, helper):
|
||||
helper._window = [0.010] * 20
|
||||
helper.last_frame_time = time.time() # arm the clock; see below
|
||||
helper.last_fps_log_time = 0.0 # force the 5s boundary
|
||||
with patch.object(helper.logger, "info") as info:
|
||||
helper.log_frame_rate()
|
||||
assert info.called
|
||||
assert "Scroll frame stats" in info.call_args[0][0]
|
||||
assert helper._window == []
|
||||
|
||||
|
||||
class TestIdleGapIsNotAFrame:
|
||||
"""The first frame of a scroll has no predecessor, so timing one measures
|
||||
the idle gap since the last scroll rather than a frame.
|
||||
|
||||
Left in, that gap lands in the max field of otherwise healthy windows and
|
||||
counts as one stall per scroll start -- about 0.2% at 500 frames to a
|
||||
window, which is the same order as the real stall rates it sits next to.
|
||||
The stall rate is the number used to judge whether a scroll change worked,
|
||||
so it has to be clean.
|
||||
"""
|
||||
|
||||
def test_clock_starts_unarmed(self, helper):
|
||||
assert helper.last_frame_time is None
|
||||
|
||||
def test_first_frame_seeds_the_clock_without_sampling(self, helper):
|
||||
helper.log_frame_rate()
|
||||
assert helper.last_frame_time is not None
|
||||
assert helper._window == []
|
||||
assert helper.frame_times == []
|
||||
|
||||
def test_second_frame_is_sampled(self, helper):
|
||||
helper.log_frame_rate()
|
||||
helper.log_frame_rate()
|
||||
assert len(helper._window) == 1
|
||||
|
||||
def test_reset_scroll_disarms_the_clock(self, helper):
|
||||
helper.log_frame_rate()
|
||||
assert helper.last_frame_time is not None
|
||||
helper.reset_scroll()
|
||||
assert helper.last_frame_time is None
|
||||
|
||||
def test_gap_longer_than_the_log_interval_is_dropped(self, helper):
|
||||
"""Covers callers that scroll without ever calling reset_scroll()."""
|
||||
helper.last_frame_time = time.time() - 137.0 # a real observed gap
|
||||
helper.log_frame_rate()
|
||||
assert helper._window == []
|
||||
assert helper.frame_times == []
|
||||
|
||||
def test_a_gap_does_not_reach_the_stats_line(self, helper):
|
||||
helper.last_frame_time = time.time() - 137.0
|
||||
helper.last_fps_log_time = 0.0 # the 5s boundary is also due
|
||||
with patch.object(helper.logger, "info") as info:
|
||||
helper.log_frame_rate()
|
||||
assert not info.called, "the idle gap was reported as a frame"
|
||||
|
||||
def test_window_of_only_gaps_logs_nothing(self, helper):
|
||||
"""A window whose every sample was dropped has nothing to report --
|
||||
and reporting the gap itself is the bug this guards."""
|
||||
helper.last_fps_log_time = 0.0
|
||||
with patch.object(helper.logger, "info") as info:
|
||||
helper.log_frame_rate() # seeds
|
||||
helper.last_frame_time = time.time() - 137.0
|
||||
helper.log_frame_rate() # dropped
|
||||
assert not info.called
|
||||
|
||||
def test_seeding_restarts_the_window_timer(self, helper):
|
||||
"""A new scroll should not open by reporting a one-frame window."""
|
||||
helper.last_fps_log_time = 0.0 # boundary long overdue
|
||||
helper.log_frame_rate() # seeds
|
||||
with patch.object(helper.logger, "info") as info:
|
||||
helper.log_frame_rate() # first real frame
|
||||
assert not info.called
|
||||
assert len(helper._window) == 1
|
||||
|
||||
def test_a_normal_frame_still_counts(self, helper):
|
||||
helper.last_frame_time = time.time() - 0.010
|
||||
helper.log_frame_rate()
|
||||
assert len(helper._window) == 1
|
||||
assert helper._window[0] == pytest.approx(0.010, abs=0.005)
|
||||
|
||||
@@ -1047,8 +1047,36 @@ def check_health_monitor():
|
||||
_reconciliation_started = True
|
||||
_threading.Thread(target=_run_startup_reconciliation, daemon=True).start()
|
||||
|
||||
_auto_updater = None
|
||||
|
||||
|
||||
def start_auto_update_scheduler():
|
||||
"""Start the weekly auto-update thread (web_interface/auto_update.py).
|
||||
|
||||
Called by the launchers, not at import: tests and dev tools import this
|
||||
module, and none of them should ever git pull. Idle unless
|
||||
auto_update.enabled is set, so it is safe to always start.
|
||||
"""
|
||||
global _auto_updater
|
||||
if _auto_updater is not None:
|
||||
return _auto_updater
|
||||
from web_interface.auto_update import AutoUpdater
|
||||
from web_interface.blueprints.api_v3.system import perform_core_update
|
||||
_auto_updater = AutoUpdater(
|
||||
config_manager=config_manager,
|
||||
core_update=perform_core_update,
|
||||
store_manager=plugin_store_manager,
|
||||
plugin_manager=plugin_manager,
|
||||
schema_manager=schema_manager,
|
||||
operation_history=operation_history,
|
||||
)
|
||||
_auto_updater.start()
|
||||
return _auto_updater
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
import os as _os
|
||||
start_auto_update_scheduler()
|
||||
# threaded=True is Flask's default since 1.0 but stated explicitly so that
|
||||
# long-lived /api/v3/stream/* SSE connections don't starve other requests.
|
||||
# Debug mode is off by default; opt in with FLASK_DEBUG=1 in the environment.
|
||||
|
||||
@@ -0,0 +1,700 @@
|
||||
"""Weekly automatic updates: LEDMatrix code, then installed plugins.
|
||||
|
||||
Turned on by ``auto_update.enabled`` in config.json (General tab, or the
|
||||
installer's --enable-auto-update). Off by default: an update pulls new code and
|
||||
restarts the display, which nobody should get without asking for it.
|
||||
|
||||
Nothing here is allowed to leave a device broken without saying so:
|
||||
|
||||
* **Checks first.** The code update is skipped, and the reason reported, when
|
||||
the checkout has local edits or commits, a rebase or merge is in progress,
|
||||
the branch has no upstream, disk is low, the newest commit was already
|
||||
rolled back once, or the health check is not set up.
|
||||
* **Verified, and rolled back.** The pull itself is the Overview "Update Code"
|
||||
path (``perform_core_update``). Restarting and checking the result is handed
|
||||
to ledmatrix-update-verify.service (scripts/utils/auto_update_verify.py),
|
||||
started through ledmatrix-update-verify.path by a request file, which
|
||||
survives the web service restart, confirms both services come up and stay
|
||||
up, and otherwise resets to the previous commit and its dependencies.
|
||||
* **Set up without SSH.** Those units are installed by the installer, or by the
|
||||
display service when the toggle is switched on (src/auto_update_setup.py).
|
||||
* **Plugins after the code.** Plugins use the Plugin Store's own update, which
|
||||
refuses versions this core cannot run and restores the old copy when an
|
||||
install fails. When the code changed they wait until it passed its check,
|
||||
so they are never gated against code that is about to be rolled back.
|
||||
* **Loud failures.** Anything other than success is written to the state file,
|
||||
shown under the toggle and raised as a banner on the Overview tab.
|
||||
|
||||
The schedule survives restarts through data/auto_update_state.json (gitignored).
|
||||
"""
|
||||
import importlib.util
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
STATE_REL = Path('data') / 'auto_update_state.json'
|
||||
PENDING_REL = Path('data') / 'auto_update_pending.json'
|
||||
REQUEST_REL = Path('data') / 'auto_update_verify.request'
|
||||
SETUP_RESULT_REL = Path('data') / 'auto_update_setup.json'
|
||||
VERIFIER_COPY_REL = Path('data') / 'auto_update_verifier.py'
|
||||
VERIFIER_SOURCE = PROJECT_ROOT / 'scripts' / 'utils' / 'auto_update_verify.py'
|
||||
STATE_FILE = PROJECT_ROOT / STATE_REL
|
||||
PENDING_FILE = PROJECT_ROOT / PENDING_REL
|
||||
SETUP_RESULT_FILE = PROJECT_ROOT / SETUP_RESULT_REL
|
||||
|
||||
VERIFY_UNIT = 'ledmatrix-update-verify'
|
||||
PATH_UNIT = f'{VERIFY_UNIT}.path'
|
||||
|
||||
UPDATE_INTERVAL_SECONDS = 7 * 24 * 3600
|
||||
#: A failure that is probably transient (usually no network) is retried the
|
||||
#: next day rather than a week later.
|
||||
RETRY_AFTER_FAILURE_SECONDS = 24 * 3600
|
||||
#: Local hours an update prefers, since it restarts the display...
|
||||
QUIET_HOURS = range(2, 5)
|
||||
#: ...but a device that is never powered at night must still get updates.
|
||||
QUIET_HOURS_GRACE_SECONDS = 24 * 3600
|
||||
CHECK_INTERVAL_SECONDS = 30 * 60
|
||||
#: Let the web service settle after boot (and never pull in a crash loop).
|
||||
STARTUP_DELAY_SECONDS = 10 * 60
|
||||
MIN_FREE_BYTES = 300 * 1024 * 1024
|
||||
#: How long the health check gets to pick up a request before the update is undone.
|
||||
HANDOFF_TIMEOUT_SECONDS = 90
|
||||
#: The health check takes a few minutes at most; far past that, it is lost.
|
||||
VERIFY_LOST_SECONDS = 45 * 60
|
||||
|
||||
#: Core outcomes that need the user's attention.
|
||||
ERROR_OUTCOMES = frozenset({'error', 'blocked', 'rolled_back', 'rollback_failed', 'lost'})
|
||||
|
||||
HELPER_MISSING_MESSAGE = (
|
||||
"LEDMatrix code updates are paused until the update health check is set up. "
|
||||
"The display service sets it up when it starts while automatic updates are on, "
|
||||
"so restart the display from the Overview tab. If this keeps happening, the "
|
||||
"setup result under Automatic Updates on the General tab says why. Plugin "
|
||||
"updates still run."
|
||||
)
|
||||
|
||||
_STATE_LOCK = threading.RLock()
|
||||
|
||||
|
||||
def is_enabled(config):
|
||||
return bool((config.get('auto_update') or {}).get('enabled', False))
|
||||
|
||||
|
||||
def _read_json(path):
|
||||
try:
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else None
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _write_json(path, data):
|
||||
path = Path(path)
|
||||
tmp = None
|
||||
try:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f'.{path.stem}_')
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
os.replace(tmp, path)
|
||||
tmp = None
|
||||
finally:
|
||||
if tmp and os.path.exists(tmp):
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _unlink(path):
|
||||
try:
|
||||
Path(path).unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def load_state(state_file=STATE_FILE):
|
||||
return _read_json(state_file) or {}
|
||||
|
||||
|
||||
def save_state(state, state_file=STATE_FILE):
|
||||
with _STATE_LOCK:
|
||||
try:
|
||||
_write_json(state_file, state)
|
||||
except OSError as e:
|
||||
logger.warning("Could not save auto-update state: %s", e)
|
||||
|
||||
|
||||
def dismiss_alert(alert_id, state_file=STATE_FILE):
|
||||
with _STATE_LOCK:
|
||||
state = load_state(state_file)
|
||||
state['alert_dismissed'] = str(alert_id)
|
||||
save_state(state, state_file)
|
||||
|
||||
|
||||
def _zone(tz_name):
|
||||
try:
|
||||
from zoneinfo import ZoneInfo
|
||||
return ZoneInfo(tz_name) if tz_name else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _local_datetime(ts, tz_name):
|
||||
return datetime.fromtimestamp(ts, _zone(tz_name))
|
||||
|
||||
|
||||
def _short(sha):
|
||||
return (sha or 'unknown')[:7]
|
||||
|
||||
|
||||
def is_due(now, next_due, local_hour):
|
||||
"""Due once ``next_due`` has passed, in quiet hours or after the grace period."""
|
||||
if next_due is None or now < next_due:
|
||||
return False
|
||||
return local_hour in QUIET_HOURS or now >= next_due + QUIET_HOURS_GRACE_SECONDS
|
||||
|
||||
|
||||
def _plugin_fingerprint(store_manager, plugin_dir):
|
||||
"""(manifest version, git sha): what changing tells us an update landed."""
|
||||
version = None
|
||||
try:
|
||||
with open(plugin_dir / 'manifest.json', 'r', encoding='utf-8') as f:
|
||||
manifest = json.load(f)
|
||||
if manifest.get('local_only'):
|
||||
return None
|
||||
version = manifest.get('version')
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
sha = None
|
||||
try:
|
||||
git_info = store_manager._get_local_git_info(plugin_dir)
|
||||
sha = git_info.get('sha') if git_info else None
|
||||
except Exception:
|
||||
logger.debug("git info unavailable for %s", plugin_dir, exc_info=True)
|
||||
return (version, sha)
|
||||
|
||||
|
||||
def update_plugins(store_manager, operation_history=None):
|
||||
"""Update every installed plugin that has an update. Returns (updated, failed)."""
|
||||
updated, failed = [], []
|
||||
plugins_dir = Path(store_manager.plugins_dir)
|
||||
for plugin_id in sorted(store_manager.list_installed_plugins()):
|
||||
plugin_dir = plugins_dir / plugin_id
|
||||
before = _plugin_fingerprint(store_manager, plugin_dir)
|
||||
if before is None:
|
||||
continue # local_only: managed by hand, never from the registry
|
||||
try:
|
||||
ok = store_manager.update_plugin(plugin_id)
|
||||
except Exception:
|
||||
logger.exception("Automatic update of plugin %s raised", plugin_id)
|
||||
ok = False
|
||||
if not ok:
|
||||
failed.append(plugin_id)
|
||||
status = 'failed'
|
||||
elif _plugin_fingerprint(store_manager, plugin_dir) != before:
|
||||
updated.append(plugin_id)
|
||||
status = 'success'
|
||||
else:
|
||||
continue # already current; not worth a history entry every week
|
||||
if operation_history:
|
||||
try:
|
||||
operation_history.record_operation(
|
||||
'update', plugin_id=plugin_id, status=status,
|
||||
details={'automatic': True})
|
||||
except Exception:
|
||||
logger.debug("Could not record auto-update history", exc_info=True)
|
||||
return updated, failed
|
||||
|
||||
|
||||
def _service_active(unit):
|
||||
try:
|
||||
result = subprocess.run(['systemctl', 'is-active', unit],
|
||||
capture_output=True, text=True, timeout=5)
|
||||
return result.stdout.strip() == 'active'
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
def helper_ready():
|
||||
"""True when the health check can be triggered (its path unit is watching)."""
|
||||
return _service_active(PATH_UNIT)
|
||||
|
||||
|
||||
def restart_service(unit):
|
||||
"""Restart a unit only if it is running, via the sudoers-allowed command."""
|
||||
if not _service_active(unit):
|
||||
return False
|
||||
try:
|
||||
result = subprocess.run(['sudo', '-n', 'systemctl', 'restart', f'{unit}.service'],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
logger.warning("Auto-update could not restart %s: %s", unit, e)
|
||||
return False
|
||||
if result.returncode != 0:
|
||||
logger.warning("Auto-update restart of %s failed: %s", unit, result.stderr.strip())
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def start_setup_if_needed(was_enabled, config):
|
||||
"""After settings are saved: switching updates on finishes setting them up.
|
||||
|
||||
The health check units are installed by the display service at startup
|
||||
(src/auto_update_setup.py), so switching the toggle on restarts it.
|
||||
Returns a note for the save message, or None.
|
||||
"""
|
||||
if was_enabled or not is_enabled(config) or helper_ready():
|
||||
return None
|
||||
if not _service_active('ledmatrix'):
|
||||
return 'Automatic update setup finishes the next time the display service starts.'
|
||||
if restart_service('ledmatrix'):
|
||||
return 'Finishing automatic update setup: the display is restarting.'
|
||||
return 'Could not restart the display to finish automatic update setup; restart it from the Overview tab.'
|
||||
|
||||
|
||||
def _load_verifier(path):
|
||||
spec = importlib.util.spec_from_file_location('ledmatrix_auto_update_verifier', str(path))
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
class AutoUpdater:
|
||||
"""Decides when an automatic update is due and runs it."""
|
||||
|
||||
def __init__(self, config_manager, core_update, store_manager=None,
|
||||
plugin_manager=None, schema_manager=None, operation_history=None,
|
||||
project_root=PROJECT_ROOT, state_file=None, clock=time.time,
|
||||
restart=restart_service, run=subprocess.run,
|
||||
service_active=_service_active, helper_ready=helper_ready,
|
||||
verifier_source=VERIFIER_SOURCE, disk_free=None, sleep=time.sleep):
|
||||
self.config_manager = config_manager
|
||||
self.core_update = core_update
|
||||
self.store_manager = store_manager
|
||||
self.plugin_manager = plugin_manager
|
||||
self.schema_manager = schema_manager
|
||||
self.operation_history = operation_history
|
||||
self.project_root = Path(project_root)
|
||||
self.state_file = Path(state_file) if state_file else self.project_root / STATE_REL
|
||||
self.pending_file = self.project_root / PENDING_REL
|
||||
self.request_file = self.project_root / REQUEST_REL
|
||||
self.verifier_copy = self.project_root / VERIFIER_COPY_REL
|
||||
self.verifier_source = Path(verifier_source)
|
||||
self.clock = clock
|
||||
self.restart = restart
|
||||
self.run_command = run
|
||||
self.service_active = service_active
|
||||
self.helper_ready = helper_ready
|
||||
self.disk_free = disk_free or (lambda path: shutil.disk_usage(path).free)
|
||||
self.sleep = sleep
|
||||
self._stop = threading.Event()
|
||||
self._thread = None
|
||||
|
||||
# -- scheduling ---------------------------------------------------------
|
||||
|
||||
def tick(self):
|
||||
"""Do whatever is due. Returns True if an update ran."""
|
||||
now = self.clock()
|
||||
state = load_state(self.state_file)
|
||||
# Before the enabled check: a verification started before the user
|
||||
# switched updates off still has to be reported.
|
||||
if self._finalize_verification(state, now) == 'waiting':
|
||||
return False
|
||||
|
||||
try:
|
||||
config = self.config_manager.load_config()
|
||||
except Exception:
|
||||
logger.debug("Auto-update could not load config", exc_info=True)
|
||||
return False
|
||||
if not is_enabled(config):
|
||||
return False
|
||||
|
||||
if state.get('plugins_pending'):
|
||||
self._run_deferred_plugins(state, now)
|
||||
return True
|
||||
|
||||
if state.get('next_due') is None:
|
||||
# Just switched on: due now, which in practice means the next
|
||||
# quiet-hours window rather than the moment Save was clicked.
|
||||
state['next_due'] = now
|
||||
save_state(state, self.state_file)
|
||||
|
||||
local_hour = _local_datetime(now, config.get('timezone')).hour
|
||||
if not is_due(now, state['next_due'], local_hour):
|
||||
return False
|
||||
self.run(state)
|
||||
return True
|
||||
|
||||
def run(self, state=None):
|
||||
state = load_state(self.state_file) if state is None else state
|
||||
logger.info("Automatic update starting")
|
||||
try:
|
||||
core = self.update_core(state)
|
||||
except Exception as e:
|
||||
logger.exception("Automatic core update raised")
|
||||
core = {'outcome': 'error', 'message': f'The LEDMatrix update failed unexpectedly: {e}'}
|
||||
|
||||
deferred = core['outcome'] == 'verifying'
|
||||
updated, failed = ([], []) if deferred else self._update_plugins()
|
||||
self._store_run(state, core, updated, failed)
|
||||
logger.info("Automatic update: core %s (%s); plugins updated=%s failed=%s%s",
|
||||
core['outcome'], core['message'], updated, failed,
|
||||
'; plugins wait for the health check' if deferred else '')
|
||||
|
||||
# Code restarts belong to the health check; this only covers plugins.
|
||||
if updated:
|
||||
self.restart('ledmatrix')
|
||||
return state['last_result']
|
||||
|
||||
def _store_run(self, state, core, updated, failed):
|
||||
now = self.clock()
|
||||
deferred = core['outcome'] == 'verifying'
|
||||
state.update({
|
||||
'last_run': now,
|
||||
'next_due': now + (RETRY_AFTER_FAILURE_SECONDS if core['outcome'] == 'error'
|
||||
else UPDATE_INTERVAL_SECONDS),
|
||||
'plugins_pending': deferred,
|
||||
'last_result': {
|
||||
'core_outcome': core['outcome'],
|
||||
'core_message': core['message'],
|
||||
'plugins_updated': updated,
|
||||
'plugins_failed': failed,
|
||||
'plugins_deferred': deferred,
|
||||
},
|
||||
})
|
||||
self._apply_result(state, now)
|
||||
save_state(state, self.state_file)
|
||||
|
||||
# -- core update --------------------------------------------------------
|
||||
|
||||
def _git(self, *args, timeout=60):
|
||||
return self.run_command(['git', *args], cwd=str(self.project_root),
|
||||
capture_output=True, text=True, timeout=timeout)
|
||||
|
||||
def _count(self, rev_range):
|
||||
out = self._git('rev-list', '--count', rev_range).stdout.strip()
|
||||
return int(out) if out.isdigit() else 0
|
||||
|
||||
def preflight(self, state):
|
||||
"""Decide whether the code update may run.
|
||||
|
||||
Returns ``(outcome, message, info)`` with outcome ``ready``,
|
||||
``up_to_date``, ``blocked`` (needs the user) or ``error`` (retry soon).
|
||||
"""
|
||||
if not self.helper_ready():
|
||||
return 'blocked', HELPER_MISSING_MESSAGE, {}
|
||||
|
||||
git_dir = self._git('rev-parse', '--git-dir')
|
||||
if git_dir.returncode != 0:
|
||||
return 'blocked', 'LEDMatrix is not installed as a git checkout, so it cannot update itself.', {}
|
||||
git_dir = Path(git_dir.stdout.strip())
|
||||
if not git_dir.is_absolute():
|
||||
git_dir = self.project_root / git_dir
|
||||
rebase_dirs = [git_dir / name for name in ('rebase-merge', 'rebase-apply') if (git_dir / name).exists()]
|
||||
if rebase_dirs:
|
||||
# A rebase really in progress leaves HEAD detached. With HEAD back
|
||||
# on a branch it was abandoned -- typically a pull that stopped on
|
||||
# a conflict, then a checkout -- and it makes every later pull
|
||||
# fail. It holds no work (local edits are checked separately
|
||||
# below), so clear it rather than strand a device nobody SSHes into.
|
||||
if self._git('symbolic-ref', '-q', 'HEAD').returncode != 0:
|
||||
return 'blocked', ('A git rebase is in progress in the LEDMatrix folder. '
|
||||
'Finish or abort it; automatic updates will not touch it.'), {}
|
||||
cleared = self._git('rebase', '--quit')
|
||||
if cleared.returncode != 0 or any(path.exists() for path in rebase_dirs):
|
||||
return 'blocked', ('An abandoned git rebase in the LEDMatrix folder could not be cleared: '
|
||||
f'{(cleared.stderr or "").strip() or "unknown error"}.'), {}
|
||||
logger.warning("Cleared an abandoned git rebase from %s", ', '.join(str(p) for p in rebase_dirs))
|
||||
if (git_dir / 'MERGE_HEAD').exists():
|
||||
return 'blocked', ('A git merge is in progress in the LEDMatrix folder. '
|
||||
'Finish or abort it; automatic updates will not touch it.'), {}
|
||||
|
||||
if self._git('rev-parse', '--abbrev-ref', '--symbolic-full-name', '@{u}').returncode != 0:
|
||||
return 'blocked', ('The current branch has no upstream to update from (or HEAD is detached). '
|
||||
'Use Update Code once, or Tools -> Switch branch.'), {}
|
||||
|
||||
# Mode-only changes are ignored: older installers chmod tracked
|
||||
# scripts, and --autostash in the pull carries those across. Plugin
|
||||
# folders are separate installs, as in perform_core_update.
|
||||
status = self._git('-c', 'core.fileMode=false', 'status', '--porcelain', '--untracked-files=no')
|
||||
changed = [line[3:] for line in status.stdout.splitlines()
|
||||
if line.strip() and 'plugins/' not in line and 'plugin-repos/' not in line]
|
||||
if status.returncode != 0 or changed:
|
||||
example = f' (for example {changed[0]})' if changed else ''
|
||||
return 'blocked', (f'{len(changed) or "Some"} tracked file(s) in the LEDMatrix folder were edited '
|
||||
f'locally{example}. Automatic updates will not stash your changes; '
|
||||
'commit or revert them, or update manually with Update Code.'), {}
|
||||
|
||||
free = self.disk_free(str(self.project_root))
|
||||
if free < MIN_FREE_BYTES:
|
||||
return 'blocked', (f'Only {free // (1024 * 1024)} MB of disk space is free; an update '
|
||||
f'needs at least {MIN_FREE_BYTES // (1024 * 1024)} MB.'), {}
|
||||
|
||||
fetch = self._git('fetch', '--quiet', timeout=120)
|
||||
if fetch.returncode != 0:
|
||||
detail = next((ln.strip() for ln in (fetch.stderr or '').splitlines() if ln.strip()), '')
|
||||
return 'error', f'Could not check for LEDMatrix updates: {detail or "git fetch failed"}.', {}
|
||||
|
||||
ahead = self._count('@{u}..HEAD')
|
||||
if ahead:
|
||||
return 'blocked', (f'This checkout has {ahead} local commit(s) that are not upstream. '
|
||||
'Automatic updates will not rebase them; update manually with Update Code.'), {}
|
||||
if not self._count('HEAD..@{u}'):
|
||||
return 'up_to_date', 'LEDMatrix is already up to date.', {}
|
||||
|
||||
upstream = self._git('rev-parse', '@{u}').stdout.strip()
|
||||
if upstream and upstream == state.get('rolled_back_head'):
|
||||
return 'up_to_date', (f'The newest LEDMatrix version ({_short(upstream)}) failed its health '
|
||||
'check and was rolled back before; waiting for a newer one.'), {}
|
||||
head = self._git('rev-parse', 'HEAD').stdout.strip()
|
||||
return 'ready', '', {'old_head': head, 'upstream_head': upstream}
|
||||
|
||||
def update_core(self, state):
|
||||
try:
|
||||
outcome, message, info = self.preflight(state)
|
||||
except (subprocess.SubprocessError, OSError) as e:
|
||||
return {'outcome': 'error', 'message': f'Pre-update checks failed: {e}.'}
|
||||
if outcome != 'ready':
|
||||
return {'outcome': outcome, 'message': message}
|
||||
old_head = info['old_head']
|
||||
|
||||
display_was_active = self.service_active('ledmatrix')
|
||||
try:
|
||||
# Copied before pulling, so the checker is the known-good version.
|
||||
self.verifier_copy.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(self.verifier_source, self.verifier_copy)
|
||||
verifier = _load_verifier(self.verifier_copy)
|
||||
except Exception as e:
|
||||
return {'outcome': 'error', 'message': f'Could not prepare the update health check: {e}.'}
|
||||
|
||||
try:
|
||||
core = self.core_update()
|
||||
except Exception as e:
|
||||
logger.exception("perform_core_update raised")
|
||||
core = {'status': 'error', 'message': f'Update failed: {e}'}
|
||||
new_head = self._git('rev-parse', 'HEAD').stdout.strip()
|
||||
pending = {
|
||||
'status': 'pending',
|
||||
'old_head': old_head,
|
||||
'new_head': new_head,
|
||||
'display_was_active': display_was_active,
|
||||
'dependency_failures': list(core.get('dependency_failures') or []),
|
||||
'created_at': self.clock(),
|
||||
}
|
||||
|
||||
def rollback():
|
||||
return verifier.Verifier(self.project_root, run=self.run_command).rollback(pending)
|
||||
|
||||
if core.get('status') != 'success':
|
||||
message = core.get('message') or 'The LEDMatrix update failed.'
|
||||
if new_head and new_head != old_head:
|
||||
ok, detail = rollback()
|
||||
message += (' The partial update was rolled back.' if ok
|
||||
else f' Rolling back the partial update also failed: {detail}.')
|
||||
if not ok:
|
||||
return {'outcome': 'rollback_failed', 'message': message}
|
||||
return {'outcome': 'error', 'message': message}
|
||||
if new_head == old_head:
|
||||
return {'outcome': 'up_to_date', 'message': core.get('message') or 'LEDMatrix is already up to date.'}
|
||||
|
||||
handoff = {'outcome': 'verifying',
|
||||
'message': (f'Updated LEDMatrix from {_short(old_head)} to {_short(new_head)}; '
|
||||
'restarting and checking the services.')}
|
||||
# Recorded before the handoff: the health check restarts this process.
|
||||
self._store_run(state, handoff, [], [])
|
||||
try:
|
||||
_write_json(self.pending_file, pending)
|
||||
self.request_file.write_text(new_head, encoding='utf-8')
|
||||
picked_up = self._wait_for_pickup()
|
||||
except OSError as e:
|
||||
logger.error("Could not request the update health check: %s", e)
|
||||
picked_up = False
|
||||
if picked_up:
|
||||
return handoff
|
||||
|
||||
# No health check means no update: undo it while the old code is
|
||||
# still the code that is running.
|
||||
logger.error("%s did not start the health check; rolling the update back", PATH_UNIT)
|
||||
_unlink(self.request_file)
|
||||
_unlink(self.pending_file)
|
||||
ok, detail = rollback()
|
||||
if not ok:
|
||||
return {'outcome': 'rollback_failed',
|
||||
'message': (f'The update health check did not start, and rolling back to '
|
||||
f'{_short(old_head)} failed: {detail}.')}
|
||||
return {'outcome': 'blocked',
|
||||
'message': ('The update health check did not start, so the update was undone. '
|
||||
f'Check "systemctl status {PATH_UNIT}".'
|
||||
+ (f' Note: {detail}.' if detail else ''))}
|
||||
|
||||
def _wait_for_pickup(self):
|
||||
for _ in range(HANDOFF_TIMEOUT_SECONDS):
|
||||
pending = _read_json(self.pending_file)
|
||||
if pending and pending.get('status') != 'pending':
|
||||
return True
|
||||
self.sleep(1)
|
||||
pending = _read_json(self.pending_file)
|
||||
return bool(pending and pending.get('status') != 'pending')
|
||||
|
||||
def _finalize_verification(self, state, now):
|
||||
"""Fold the health check's outcome into the state. None, 'waiting' or 'done'."""
|
||||
pending = _read_json(self.pending_file)
|
||||
if pending is None:
|
||||
return None
|
||||
status = pending.get('status')
|
||||
old, new = pending.get('old_head'), pending.get('new_head')
|
||||
reason, detail = pending.get('reason'), pending.get('detail')
|
||||
|
||||
if status in ('pending', 'verifying'):
|
||||
if now - float(pending.get('created_at') or now) < VERIFY_LOST_SECONDS:
|
||||
return 'waiting'
|
||||
outcome = 'lost'
|
||||
message = (f'LEDMatrix was updated to {_short(new)}, but its health check never reported back, '
|
||||
'so it is not known whether the device is healthy. '
|
||||
f'See "journalctl -u {VERIFY_UNIT}".')
|
||||
elif status == 'success':
|
||||
outcome = 'updated'
|
||||
message = (f'Updated LEDMatrix from {_short(old)} to {_short(new)}; '
|
||||
'the services restarted and stayed healthy.')
|
||||
elif status == 'rolled_back':
|
||||
outcome = 'rolled_back'
|
||||
message = (f'The LEDMatrix update to {_short(new)} was rolled back to {_short(old)} because '
|
||||
f'{reason or "it failed its health check"}.' + (f' Note: {detail}.' if detail else ''))
|
||||
state['rolled_back_head'] = new
|
||||
else:
|
||||
outcome = 'rollback_failed'
|
||||
message = (f'The LEDMatrix update to {_short(new)} failed ({reason or "unknown reason"}) and ' # nosec B608 - user-facing message, not SQL # nosemgrep
|
||||
f'could not be rolled back: {detail or "unknown error"}. The device may need '
|
||||
f'attention: run "git reset --hard {old}" in the LEDMatrix folder, then restart '
|
||||
'the display and web services.')
|
||||
|
||||
result = state.setdefault('last_result', {})
|
||||
result.update({'core_outcome': outcome, 'core_message': message})
|
||||
if outcome not in ('updated', 'rolled_back'):
|
||||
# The device is in an unknown state; don't pile plugin changes on it.
|
||||
state['plugins_pending'] = False
|
||||
result['plugins_deferred'] = False
|
||||
self._apply_result(state, now)
|
||||
save_state(state, self.state_file)
|
||||
_unlink(self.pending_file)
|
||||
(logger.info if outcome == 'updated' else logger.error)("Automatic update: %s", message)
|
||||
return 'done'
|
||||
|
||||
# -- plugins and reporting ----------------------------------------------
|
||||
|
||||
def _update_plugins(self):
|
||||
if not self.store_manager:
|
||||
return [], []
|
||||
updated, failed = update_plugins(self.store_manager, self.operation_history)
|
||||
for plugin_id in updated:
|
||||
if self.schema_manager:
|
||||
self.schema_manager.invalidate_cache(plugin_id)
|
||||
if updated and self.plugin_manager:
|
||||
try:
|
||||
self.plugin_manager.discover_plugins()
|
||||
except Exception:
|
||||
logger.debug("discover_plugins after auto-update failed", exc_info=True)
|
||||
return updated, failed
|
||||
|
||||
def _run_deferred_plugins(self, state, now):
|
||||
state['plugins_pending'] = False
|
||||
updated, failed = self._update_plugins()
|
||||
result = state.setdefault('last_result', {})
|
||||
result.update({'plugins_updated': updated, 'plugins_failed': failed, 'plugins_deferred': False})
|
||||
self._apply_result(state, now)
|
||||
save_state(state, self.state_file)
|
||||
if updated:
|
||||
self.restart('ledmatrix')
|
||||
|
||||
@staticmethod
|
||||
def _apply_result(state, now):
|
||||
result = state.get('last_result') or {}
|
||||
outcome = result.get('core_outcome')
|
||||
failed = result.get('plugins_failed') or []
|
||||
if outcome in ERROR_OUTCOMES or failed:
|
||||
result['status'] = 'error'
|
||||
elif outcome == 'verifying':
|
||||
result['status'] = 'pending'
|
||||
else:
|
||||
result['status'] = 'success'
|
||||
|
||||
if result['status'] == 'error':
|
||||
parts = []
|
||||
if outcome in ERROR_OUTCOMES:
|
||||
parts.append(result.get('core_message') or 'The LEDMatrix update failed.')
|
||||
if failed:
|
||||
parts.append('These plugins failed to update: ' + ', '.join(failed) + '.')
|
||||
state['alert'] = {'id': str(int(now * 1000)), 'message': ' '.join(parts)}
|
||||
elif result['status'] == 'success':
|
||||
state.pop('alert', None)
|
||||
|
||||
# -- thread -------------------------------------------------------------
|
||||
|
||||
def start(self):
|
||||
if self._thread and self._thread.is_alive():
|
||||
return
|
||||
self._thread = threading.Thread(target=self._loop, name='auto-update', daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self):
|
||||
self._stop.set()
|
||||
|
||||
def _loop(self):
|
||||
if self._stop.wait(STARTUP_DELAY_SECONDS):
|
||||
return
|
||||
while True:
|
||||
try:
|
||||
self.tick()
|
||||
except Exception:
|
||||
logger.exception("Automatic update check failed")
|
||||
if self._stop.wait(CHECK_INTERVAL_SECONDS):
|
||||
return
|
||||
|
||||
|
||||
def describe_status(config, state=None, state_file=STATE_FILE, pending_file=PENDING_FILE,
|
||||
setup_file=SETUP_RESULT_FILE, helper=None):
|
||||
"""What the General tab and the Overview banner show."""
|
||||
state = load_state(state_file) if state is None else state
|
||||
tz = config.get('timezone')
|
||||
fmt = '%b %d, %Y %I:%M %p'
|
||||
pending = _read_json(pending_file) or {}
|
||||
setup = _read_json(setup_file) or {}
|
||||
enabled = is_enabled(config)
|
||||
out = {
|
||||
'last_run': None, 'summary': None, 'status': None, 'next_due': None,
|
||||
'alert': None, 'alert_id': None,
|
||||
# Only asked while enabled: it runs systemctl, on every page load.
|
||||
'verifier_installed': (helper or helper_ready)() if enabled else None,
|
||||
'setup_status': setup.get('status'),
|
||||
'setup_message': setup.get('message'),
|
||||
'verifying': pending.get('status') in ('pending', 'verifying'),
|
||||
}
|
||||
if state.get('last_run'):
|
||||
out['last_run'] = _local_datetime(state['last_run'], tz).strftime(fmt)
|
||||
result = state.get('last_result') or {}
|
||||
out['status'] = result.get('status')
|
||||
parts = [result.get('core_message') or 'No result recorded.']
|
||||
if result.get('plugins_updated'):
|
||||
parts.append('Plugins updated: ' + ', '.join(result['plugins_updated']) + '.')
|
||||
if result.get('plugins_failed'):
|
||||
parts.append('Plugins that failed to update: ' + ', '.join(result['plugins_failed']) + '.')
|
||||
if result.get('plugins_deferred') and state.get('plugins_pending'):
|
||||
parts.append('Plugin updates run once the code update passes its health check.')
|
||||
out['summary'] = ' '.join(parts)
|
||||
if enabled and state.get('next_due'):
|
||||
out['next_due'] = _local_datetime(state['next_due'], tz).strftime(fmt)
|
||||
alert = state.get('alert')
|
||||
if alert and alert.get('id') != state.get('alert_dismissed'):
|
||||
out['alert'], out['alert_id'] = alert.get('message'), alert.get('id')
|
||||
return out
|
||||
@@ -454,16 +454,21 @@ def save_main_config():
|
||||
|
||||
# Merge with existing config (similar to original implementation)
|
||||
current_config = api_v3.config_manager.load_config()
|
||||
was_auto_update_enabled = bool((current_config.get('auto_update') or {}).get('enabled'))
|
||||
|
||||
# Handle general settings
|
||||
# Note: Checkboxes don't send data when unchecked, so we need to check if we're updating general settings
|
||||
# If any general setting is present, we're updating the general tab
|
||||
is_general_update = any(k in data for k in ['timezone', 'city', 'state', 'country', 'web_display_autostart',
|
||||
'auto_discover', 'auto_load_enabled', 'development_mode', 'plugins_directory'])
|
||||
'auto_discover', 'auto_load_enabled', 'development_mode', 'plugins_directory',
|
||||
'auto_update_enabled'])
|
||||
|
||||
if is_general_update:
|
||||
# For checkbox: if not present in data during general update, it means unchecked
|
||||
current_config['web_display_autostart'] = _coerce_to_bool(data.get('web_display_autostart'))
|
||||
if not isinstance(current_config.get('auto_update'), dict):
|
||||
current_config['auto_update'] = {}
|
||||
current_config['auto_update']['enabled'] = _coerce_to_bool(data.get('auto_update_enabled'))
|
||||
|
||||
if 'timezone' in data:
|
||||
current_config['timezone'] = data['timezone']
|
||||
@@ -1033,7 +1038,7 @@ def save_main_config():
|
||||
if key in ['timezone', 'city', 'state', 'country',
|
||||
'web_display_autostart', 'auto_discover',
|
||||
'auto_load_enabled', 'development_mode',
|
||||
'plugins_directory', 'target_fps']:
|
||||
'plugins_directory', 'target_fps', 'auto_update_enabled']:
|
||||
continue
|
||||
# Skip fields that are already handled above in their own named sections.
|
||||
# Without this, every form field name lands as a top-level config key too.
|
||||
@@ -1068,7 +1073,17 @@ def save_main_config():
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
return success_response(message='Configuration saved successfully')
|
||||
message = 'Configuration saved successfully'
|
||||
# Switching automatic updates on finishes their setup, which needs
|
||||
# the display service to restart (web_interface/auto_update.py).
|
||||
try:
|
||||
from web_interface import auto_update
|
||||
note = auto_update.start_setup_if_needed(was_auto_update_enabled, current_config)
|
||||
if note:
|
||||
message = f'{message}. {note}'
|
||||
except Exception:
|
||||
logger.warning("Automatic update setup could not be started", exc_info=True)
|
||||
return success_response(message=message)
|
||||
except Exception as e:
|
||||
logger.error("Error saving config", exc_info=True)
|
||||
return error_response(
|
||||
|
||||
@@ -12,6 +12,8 @@ from web_interface.blueprints.api_v3 import (
|
||||
get_git_version, jsonify, logger, os, request, resolve_pull_command,
|
||||
shutil, subprocess,
|
||||
)
|
||||
import threading
|
||||
|
||||
import web_interface.blueprints.api_v3 as _pkg
|
||||
# Read through the module rather than bound by value: tests patch these
|
||||
# as module attributes, and a value binding would not see the patch.
|
||||
@@ -120,6 +122,32 @@ def get_system_version():
|
||||
except Exception as e:
|
||||
logger.error("get_system_version failed: %s", e, exc_info=True)
|
||||
return jsonify({'status': 'error', 'message': 'Unable to retrieve version'}), 500
|
||||
@api_v3.route('/system/auto-update', methods=['GET'])
|
||||
def get_auto_update_status():
|
||||
"""Weekly automatic update status: last result, next check, and any alert.
|
||||
|
||||
No local except: a failure falls through to the app-wide handler in
|
||||
web_interface/app.py, which logs the traceback and returns the redacted
|
||||
detail -- the response every route gives, without a second copy here.
|
||||
"""
|
||||
from web_interface import auto_update
|
||||
config = api_v3.config_manager.load_config() if api_v3.config_manager else {}
|
||||
return jsonify({'status': 'success', 'data': auto_update.describe_status(config)})
|
||||
|
||||
|
||||
@api_v3.route('/system/auto-update/dismiss', methods=['POST'])
|
||||
def dismiss_auto_update_alert():
|
||||
"""Hide the current automatic-update banner until a new alert replaces it."""
|
||||
from web_interface import auto_update
|
||||
payload = request.get_json(silent=True)
|
||||
# A JSON array or scalar is a bad request, not a 500.
|
||||
alert_id = str(payload.get('alert_id') or '').strip() if isinstance(payload, dict) else ''
|
||||
if not alert_id:
|
||||
return jsonify({'status': 'error', 'message': 'alert_id required'}), 400
|
||||
auto_update.dismiss_alert(alert_id)
|
||||
return jsonify({'status': 'success'})
|
||||
|
||||
|
||||
@api_v3.route('/system/check-update', methods=['GET'])
|
||||
def check_for_update():
|
||||
"""Check whether a newer LEDMatrix commit is available on origin/main."""
|
||||
@@ -198,6 +226,219 @@ def _sudo_hint_for(text):
|
||||
return None
|
||||
|
||||
|
||||
_core_update_lock = threading.Lock()
|
||||
|
||||
|
||||
def perform_core_update():
|
||||
"""Pull the latest LEDMatrix code and sync its dependencies.
|
||||
|
||||
Shared by the Overview "Update Code" button and the weekly automatic
|
||||
updater (web_interface/auto_update.py), so both take exactly the same
|
||||
path. Returns the JSON-able payload the button has always received:
|
||||
``status``, ``message`` and ``restart_required``.
|
||||
"""
|
||||
# The button and the scheduler can fire together; two pulls racing
|
||||
# over one checkout (and one stash) is how local changes get lost.
|
||||
if not _core_update_lock.acquire(blocking=False):
|
||||
return {'status': 'error', 'restart_required': False,
|
||||
'message': 'An update is already in progress; try again shortly.'}
|
||||
try:
|
||||
return _perform_core_update_locked()
|
||||
finally:
|
||||
_core_update_lock.release()
|
||||
|
||||
|
||||
def _perform_core_update_locked():
|
||||
project_dir = str(PROJECT_ROOT)
|
||||
|
||||
# Decide how to pull BEFORE stashing. If this checkout cannot be
|
||||
# updated at all, stashing first would put the user's local changes
|
||||
# away for an update that was never going to run.
|
||||
pull_args, upstream_note, pull_error = resolve_pull_command(project_dir)
|
||||
if pull_error:
|
||||
logger.warning("git pull not attempted: %s", pull_error)
|
||||
return {'status': 'error', 'message': pull_error, 'restart_required': False}
|
||||
|
||||
# Check if there are local changes that need to be stashed
|
||||
# Exclude plugins directory - plugins are separate repos and shouldn't be stashed with base project
|
||||
# Use --untracked-files=no to skip untracked files check (much faster with symlinked plugins)
|
||||
try:
|
||||
status_result = subprocess.run(
|
||||
['git', 'status', '--porcelain', '--untracked-files=no'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
cwd=project_dir
|
||||
)
|
||||
# Filter out any changes in plugins directory - plugins are separate repositories
|
||||
# Git status format: XY filename (where X is status of index, Y is status of work tree)
|
||||
status_lines = [line for line in status_result.stdout.strip().split('\n')
|
||||
if line.strip() and 'plugins/' not in line]
|
||||
has_changes = bool('\n'.join(status_lines).strip())
|
||||
except subprocess.TimeoutExpired:
|
||||
# If status check times out, assume there might be changes and proceed
|
||||
# This is safer than failing the update
|
||||
has_changes = True
|
||||
status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
|
||||
|
||||
stash_info = ""
|
||||
|
||||
# Stash local changes if they exist (excluding plugins)
|
||||
# Plugins are separate repositories and shouldn't be stashed with base project updates
|
||||
if has_changes:
|
||||
try:
|
||||
# Use pathspec to exclude plugins directory from stash
|
||||
stash_result = subprocess.run(
|
||||
['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--', ':!plugins'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
cwd=project_dir
|
||||
)
|
||||
if stash_result.returncode == 0:
|
||||
logger.debug("git stash: stashed local changes before pull")
|
||||
stash_info = " Local changes were stashed."
|
||||
else:
|
||||
logger.warning("git stash failed before pull (returncode=%d)", stash_result.returncode)
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("git stash timed out, proceeding with pull")
|
||||
|
||||
# Record HEAD before the pull so dependency changes can be detected
|
||||
old_head = None
|
||||
try:
|
||||
_pre = subprocess.run(['git', 'rev-parse', 'HEAD'],
|
||||
capture_output=True, text=True, timeout=10, cwd=project_dir)
|
||||
if _pre.returncode == 0:
|
||||
old_head = _pre.stdout.strip()
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("git rev-parse timed out before pull")
|
||||
|
||||
# Whether the pull actually brought new code in. "Already up to
|
||||
# date" is a success too, and prompting for a restart then would
|
||||
# train users to ignore the prompt.
|
||||
code_changed = False
|
||||
# Requirement files whose install failed. The automatic updater refuses
|
||||
# to restart onto code whose dependencies did not install.
|
||||
dependency_failures = []
|
||||
|
||||
# Perform the git pull. Branches without an upstream were given
|
||||
# an explicit "origin <branch>" above so the update still works.
|
||||
result = subprocess.run(
|
||||
pull_args,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
cwd=project_dir
|
||||
)
|
||||
|
||||
# Give the branch tracking information so the next pull is a plain
|
||||
# `git pull` — otherwise every update repeats the fallback.
|
||||
if result.returncode == 0 and upstream_note:
|
||||
branch = _git_current_branch(project_dir)
|
||||
if branch:
|
||||
try:
|
||||
subprocess.run(
|
||||
['git', 'branch', f'--set-upstream-to=origin/{branch}', branch],
|
||||
capture_output=True, text=True, timeout=10, cwd=project_dir)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("could not set upstream for %s: %s", branch, exc)
|
||||
|
||||
# Return custom response for git_pull
|
||||
if result.returncode == 0:
|
||||
pull_message = "Code updated successfully."
|
||||
if has_changes:
|
||||
pull_message = f"Code updated successfully. Local changes were automatically stashed.{stash_info}"
|
||||
if result.stdout and "Already up to date" not in result.stdout:
|
||||
pull_message = f"Code updated successfully.{stash_info}"
|
||||
if upstream_note:
|
||||
pull_message = f"{pull_message} {upstream_note}"
|
||||
|
||||
# Keep Python dependencies in sync automatically: if the pull
|
||||
# changed a requirements file, install it now — users updating
|
||||
# from the web UI (most of them) never SSH in to pip install.
|
||||
# Installs go through the same root-visible path as the
|
||||
# Tools-tab buttons (_pip_install_requirements).
|
||||
dep_notes = []
|
||||
try:
|
||||
_post = subprocess.run(['git', 'rev-parse', 'HEAD'],
|
||||
capture_output=True, text=True, timeout=10, cwd=project_dir)
|
||||
new_head = _post.stdout.strip() if _post.returncode == 0 else None
|
||||
if old_head and new_head and old_head != new_head:
|
||||
code_changed = True
|
||||
diff = subprocess.run(
|
||||
['git', 'diff', '--name-only', f'{old_head}..{new_head}'],
|
||||
capture_output=True, text=True, timeout=15, cwd=project_dir)
|
||||
changed = set(diff.stdout.split()) if diff.returncode == 0 else set()
|
||||
for rel in ('requirements.txt', 'web_interface/requirements.txt'):
|
||||
req_path = PROJECT_ROOT / rel
|
||||
if rel not in changed or not req_path.exists():
|
||||
continue
|
||||
# Each file's install is isolated: a timeout or
|
||||
# OSError (e.g. the sudo wrapper/interpreter
|
||||
# missing) on one file must not abort the other.
|
||||
try:
|
||||
r = _pip_install_requirements(req_path, timeout=180)
|
||||
if r.returncode == 0:
|
||||
dep_notes.append(f"Dependencies from {rel} updated.")
|
||||
else:
|
||||
dependency_failures.append(rel)
|
||||
dep_notes.append(
|
||||
f"Dependency install from {rel} failed — "
|
||||
"run Install Base Requirements from the Tools tab.")
|
||||
logger.warning("post-update pip install failed for %s: %s",
|
||||
rel, _truncate_output(r.stdout, r.stderr))
|
||||
except subprocess.TimeoutExpired:
|
||||
dependency_failures.append(rel)
|
||||
dep_notes.append(
|
||||
f"Dependency install from {rel} timed out — "
|
||||
"run Install Base Requirements from the Tools tab.")
|
||||
logger.warning("post-update pip install timed out for %s", rel)
|
||||
except OSError as install_err:
|
||||
dependency_failures.append(rel)
|
||||
dep_notes.append(
|
||||
f"Dependency install from {rel} failed — "
|
||||
"run Install Base Requirements from the Tools tab.")
|
||||
logger.warning("post-update pip install errored for %s: %s",
|
||||
rel, install_err)
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("post-update dependency sync timed out")
|
||||
if dep_notes:
|
||||
pull_message += " " + " ".join(dep_notes)
|
||||
# A `git pull` restores built-in plugins (committed under
|
||||
# plugin-repos/) even if the user uninstalled them. Re-remove
|
||||
# any the user previously uninstalled so the update doesn't
|
||||
# resurrect them.
|
||||
if api_v3.plugin_store_manager:
|
||||
try:
|
||||
purged = api_v3.plugin_store_manager.purge_uninstalled_plugins()
|
||||
if purged:
|
||||
logger.info(
|
||||
"Re-removed %d uninstalled plugin(s) restored by update: %s",
|
||||
len(purged), ", ".join(purged),
|
||||
)
|
||||
except (OSError, RuntimeError) as purge_err:
|
||||
logger.warning("Post-update plugin purge failed: %s", purge_err)
|
||||
else:
|
||||
logger.warning("git pull failed (returncode=%d): %s", result.returncode, result.stderr)
|
||||
# Show git's own first line: "check logs" leaves the user with
|
||||
# nothing to act on, and these failures are usually actionable
|
||||
# (conflicting local commits, no upstream, network).
|
||||
detail = next((ln.strip() for ln in (result.stderr or '').splitlines()
|
||||
if ln.strip()), '')
|
||||
pull_message = f"Update failed: {detail}" if detail else "Update failed; check logs for details"
|
||||
|
||||
# Nothing here restarts anything: the pull replaces files on
|
||||
# disk while the display and web services keep running the code
|
||||
# they loaded at boot. Without this the user is told the update
|
||||
# succeeded and sees no change until they happen to reboot.
|
||||
return {
|
||||
'status': 'success' if result.returncode == 0 else 'error',
|
||||
'message': pull_message,
|
||||
'restart_required': bool(result.returncode == 0 and code_changed),
|
||||
'dependency_failures': dependency_failures,
|
||||
}
|
||||
|
||||
|
||||
@api_v3.route('/system/action', methods=['POST'])
|
||||
def execute_system_action():
|
||||
"""Execute system actions (start/stop/reboot/etc)"""
|
||||
@@ -265,188 +506,7 @@ def execute_system_action():
|
||||
result = subprocess.run(['sudo', 'poweroff'],
|
||||
capture_output=True, text=True, timeout=10)
|
||||
elif action == 'git_pull':
|
||||
# Use PROJECT_ROOT instead of hardcoded path
|
||||
project_dir = str(PROJECT_ROOT)
|
||||
|
||||
# Decide how to pull BEFORE stashing. If this checkout cannot be
|
||||
# updated at all, stashing first would put the user's local changes
|
||||
# away for an update that was never going to run.
|
||||
pull_args, upstream_note, pull_error = resolve_pull_command(project_dir)
|
||||
if pull_error:
|
||||
logger.warning("git pull not attempted: %s", pull_error)
|
||||
return jsonify({'status': 'error', 'message': pull_error})
|
||||
|
||||
# Check if there are local changes that need to be stashed
|
||||
# Exclude plugins directory - plugins are separate repos and shouldn't be stashed with base project
|
||||
# Use --untracked-files=no to skip untracked files check (much faster with symlinked plugins)
|
||||
try:
|
||||
status_result = subprocess.run(
|
||||
['git', 'status', '--porcelain', '--untracked-files=no'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
cwd=project_dir
|
||||
)
|
||||
# Filter out any changes in plugins directory - plugins are separate repositories
|
||||
# Git status format: XY filename (where X is status of index, Y is status of work tree)
|
||||
status_lines = [line for line in status_result.stdout.strip().split('\n')
|
||||
if line.strip() and 'plugins/' not in line]
|
||||
has_changes = bool('\n'.join(status_lines).strip())
|
||||
except subprocess.TimeoutExpired:
|
||||
# If status check times out, assume there might be changes and proceed
|
||||
# This is safer than failing the update
|
||||
has_changes = True
|
||||
status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
|
||||
|
||||
stash_info = ""
|
||||
|
||||
# Stash local changes if they exist (excluding plugins)
|
||||
# Plugins are separate repositories and shouldn't be stashed with base project updates
|
||||
if has_changes:
|
||||
try:
|
||||
# Use pathspec to exclude plugins directory from stash
|
||||
stash_result = subprocess.run(
|
||||
['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--', ':!plugins'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
cwd=project_dir
|
||||
)
|
||||
if stash_result.returncode == 0:
|
||||
logger.debug("git stash: stashed local changes before pull")
|
||||
stash_info = " Local changes were stashed."
|
||||
else:
|
||||
logger.warning("git stash failed before pull (returncode=%d)", stash_result.returncode)
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("git stash timed out, proceeding with pull")
|
||||
|
||||
# Record HEAD before the pull so dependency changes can be detected
|
||||
old_head = None
|
||||
try:
|
||||
_pre = subprocess.run(['git', 'rev-parse', 'HEAD'],
|
||||
capture_output=True, text=True, timeout=10, cwd=project_dir)
|
||||
if _pre.returncode == 0:
|
||||
old_head = _pre.stdout.strip()
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("git rev-parse timed out before pull")
|
||||
|
||||
# Whether the pull actually brought new code in. "Already up to
|
||||
# date" is a success too, and prompting for a restart then would
|
||||
# train users to ignore the prompt.
|
||||
code_changed = False
|
||||
|
||||
# Perform the git pull. Branches without an upstream were given
|
||||
# an explicit "origin <branch>" above so the update still works.
|
||||
result = subprocess.run(
|
||||
pull_args,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=60,
|
||||
cwd=project_dir
|
||||
)
|
||||
|
||||
# Give the branch tracking information so the next pull is a plain
|
||||
# `git pull` — otherwise every update repeats the fallback.
|
||||
if result.returncode == 0 and upstream_note:
|
||||
branch = _git_current_branch(project_dir)
|
||||
if branch:
|
||||
try:
|
||||
subprocess.run(
|
||||
['git', 'branch', f'--set-upstream-to=origin/{branch}', branch],
|
||||
capture_output=True, text=True, timeout=10, cwd=project_dir)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("could not set upstream for %s: %s", branch, exc)
|
||||
|
||||
# Return custom response for git_pull
|
||||
if result.returncode == 0:
|
||||
pull_message = "Code updated successfully."
|
||||
if has_changes:
|
||||
pull_message = f"Code updated successfully. Local changes were automatically stashed.{stash_info}"
|
||||
if result.stdout and "Already up to date" not in result.stdout:
|
||||
pull_message = f"Code updated successfully.{stash_info}"
|
||||
if upstream_note:
|
||||
pull_message = f"{pull_message} {upstream_note}"
|
||||
|
||||
# Keep Python dependencies in sync automatically: if the pull
|
||||
# changed a requirements file, install it now — users updating
|
||||
# from the web UI (most of them) never SSH in to pip install.
|
||||
# Installs go through the same root-visible path as the
|
||||
# Tools-tab buttons (_pip_install_requirements).
|
||||
dep_notes = []
|
||||
try:
|
||||
_post = subprocess.run(['git', 'rev-parse', 'HEAD'],
|
||||
capture_output=True, text=True, timeout=10, cwd=project_dir)
|
||||
new_head = _post.stdout.strip() if _post.returncode == 0 else None
|
||||
if old_head and new_head and old_head != new_head:
|
||||
code_changed = True
|
||||
diff = subprocess.run(
|
||||
['git', 'diff', '--name-only', f'{old_head}..{new_head}'],
|
||||
capture_output=True, text=True, timeout=15, cwd=project_dir)
|
||||
changed = set(diff.stdout.split()) if diff.returncode == 0 else set()
|
||||
for rel in ('requirements.txt', 'web_interface/requirements.txt'):
|
||||
req_path = PROJECT_ROOT / rel
|
||||
if rel not in changed or not req_path.exists():
|
||||
continue
|
||||
# Each file's install is isolated: a timeout or
|
||||
# OSError (e.g. the sudo wrapper/interpreter
|
||||
# missing) on one file must not abort the other.
|
||||
try:
|
||||
r = _pip_install_requirements(req_path, timeout=180)
|
||||
if r.returncode == 0:
|
||||
dep_notes.append(f"Dependencies from {rel} updated.")
|
||||
else:
|
||||
dep_notes.append(
|
||||
f"Dependency install from {rel} failed — "
|
||||
"run Install Base Requirements from the Tools tab.")
|
||||
logger.warning("post-update pip install failed for %s: %s",
|
||||
rel, _truncate_output(r.stdout, r.stderr))
|
||||
except subprocess.TimeoutExpired:
|
||||
dep_notes.append(
|
||||
f"Dependency install from {rel} timed out — "
|
||||
"run Install Base Requirements from the Tools tab.")
|
||||
logger.warning("post-update pip install timed out for %s", rel)
|
||||
except OSError as install_err:
|
||||
dep_notes.append(
|
||||
f"Dependency install from {rel} failed — "
|
||||
"run Install Base Requirements from the Tools tab.")
|
||||
logger.warning("post-update pip install errored for %s: %s",
|
||||
rel, install_err)
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("post-update dependency sync timed out")
|
||||
if dep_notes:
|
||||
pull_message += " " + " ".join(dep_notes)
|
||||
# A `git pull` restores built-in plugins (committed under
|
||||
# plugin-repos/) even if the user uninstalled them. Re-remove
|
||||
# any the user previously uninstalled so the update doesn't
|
||||
# resurrect them.
|
||||
if api_v3.plugin_store_manager:
|
||||
try:
|
||||
purged = api_v3.plugin_store_manager.purge_uninstalled_plugins()
|
||||
if purged:
|
||||
logger.info(
|
||||
"Re-removed %d uninstalled plugin(s) restored by update: %s",
|
||||
len(purged), ", ".join(purged),
|
||||
)
|
||||
except (OSError, RuntimeError) as purge_err:
|
||||
logger.warning("Post-update plugin purge failed: %s", purge_err)
|
||||
else:
|
||||
logger.warning("git pull failed (returncode=%d): %s", result.returncode, result.stderr)
|
||||
# Show git's own first line: "check logs" leaves the user with
|
||||
# nothing to act on, and these failures are usually actionable
|
||||
# (conflicting local commits, no upstream, network).
|
||||
detail = next((ln.strip() for ln in (result.stderr or '').splitlines()
|
||||
if ln.strip()), '')
|
||||
pull_message = f"Update failed: {detail}" if detail else "Update failed; check logs for details"
|
||||
|
||||
# Nothing here restarts anything: the pull replaces files on
|
||||
# disk while the display and web services keep running the code
|
||||
# they loaded at boot. Without this the user is told the update
|
||||
# succeeded and sees no change until they happen to reboot.
|
||||
return jsonify({
|
||||
'status': 'success' if result.returncode == 0 else 'error',
|
||||
'message': pull_message,
|
||||
'restart_required': bool(result.returncode == 0 and code_changed),
|
||||
})
|
||||
return jsonify(perform_core_update())
|
||||
elif action == 'checkout_branch':
|
||||
# Switch branches from the Tools tab. Needed because a checkout
|
||||
# that predates tracking (or a restored backup) can leave the pi
|
||||
|
||||
@@ -525,8 +525,15 @@ def _load_general_partial():
|
||||
try:
|
||||
if pages_v3.config_manager:
|
||||
main_config = pages_v3.config_manager.load_config()
|
||||
try:
|
||||
from web_interface.auto_update import describe_status
|
||||
auto_update_status = describe_status(main_config)
|
||||
except Exception:
|
||||
logger.debug("Could not read auto-update status", exc_info=True)
|
||||
auto_update_status = None
|
||||
return render_template('v3/partials/general.html',
|
||||
main_config=main_config)
|
||||
main_config=main_config,
|
||||
auto_update_status=auto_update_status)
|
||||
except Exception as e:
|
||||
logger.error("Error loading partial", exc_info=True)
|
||||
return "Error loading partial", 500
|
||||
|
||||
@@ -100,7 +100,8 @@ def main():
|
||||
werkzeug_logger.error = log_exception_filtered
|
||||
|
||||
# Import and run the Flask app
|
||||
from web_interface.app import app
|
||||
from web_interface.app import app, start_auto_update_scheduler
|
||||
start_auto_update_scheduler()
|
||||
|
||||
print("Starting LED Matrix Web Interface V3...")
|
||||
print("Web server binding to: 0.0.0.0:5000")
|
||||
|
||||
@@ -44,6 +44,44 @@
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<!-- Weekly Automatic Updates -->
|
||||
<div class="form-group" id="setting-general-auto_update" data-setting-key="auto_update.enabled">
|
||||
<label class="flex items-center">
|
||||
<input type="checkbox"
|
||||
name="auto_update_enabled"
|
||||
value="true"
|
||||
{% if (main_config.auto_update or {}).enabled %}checked{% endif %}
|
||||
class="form-control h-4 w-4 text-blue-600 focus:ring-blue-500 border-gray-300 rounded">
|
||||
<span class="ml-2 text-sm font-medium text-gray-900">Automatically check for and install updates once a week</span>
|
||||
{{ ui.help_tip('Once a week, update LEDMatrix and every installed plugin that has a newer version.\nSkipped (and reported) if you have local changes, low disk space, or the rollback service is missing. After updating, the services are restarted and checked; if they do not stay healthy, the update is rolled back automatically.\nRuns overnight (2-5 AM in your timezone) when possible. Problems show as a banner on Overview. A stopped display is not started. Default: off.', 'Automatic Updates') }}
|
||||
</label>
|
||||
{% if auto_update_status and (main_config.auto_update or {}).enabled and auto_update_status.verifier_installed is sameas false %}
|
||||
<p class="mt-1 ml-6 text-xs text-amber-700">
|
||||
{% if auto_update_status.setup_status == 'failed' %}
|
||||
Setting up the update health check failed: {{ auto_update_status.setup_message }}
|
||||
LEDMatrix code updates are paused until it is fixed; plugin updates still run.
|
||||
{% else %}
|
||||
Setting up the update health check. The display service does this when it starts, so it is
|
||||
ready shortly after saving (or once the display is started). LEDMatrix code updates begin after that.
|
||||
{% endif %}
|
||||
</p>
|
||||
{% endif %}
|
||||
{% if auto_update_status and auto_update_status.verifying %}
|
||||
<p class="mt-1 ml-6 text-xs text-blue-700">An update was just installed and its health check is running.</p>
|
||||
{% endif %}
|
||||
{% if auto_update_status and (auto_update_status.last_run or auto_update_status.next_due) %}
|
||||
<div class="mt-1 ml-6 text-xs text-gray-500 space-y-1">
|
||||
{% if auto_update_status.last_run %}
|
||||
<p>Last automatic update: {{ auto_update_status.last_run }}
|
||||
<span class="{{ 'text-red-600' if auto_update_status.status == 'error' else '' }}">— {{ auto_update_status.summary }}</span></p>
|
||||
{% endif %}
|
||||
{% if auto_update_status.next_due %}
|
||||
<p>Next check: {{ auto_update_status.next_due }} or the first overnight window after it</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<!-- Timezone -->
|
||||
<div class="form-group" id="setting-general-timezone" data-setting-key="timezone">
|
||||
<label for="timezone" class="block text-sm font-medium text-gray-700">Timezone{{ ui.help_tip('Time zone used for clocks, schedules, and time-based content.\nChoose the zone where the display physically lives so on/off schedules fire at the correct local time.', 'Timezone') }}</label>
|
||||
|
||||
@@ -1,3 +1,44 @@
|
||||
<!-- Automatic update banner: a weekly update that failed, was blocked or was rolled back -->
|
||||
<div id="auto-update-banner" class="bg-red-50 border border-red-300 rounded-lg p-4 mb-4 flex items-start" style="display:none !important" role="alert">
|
||||
<div class="flex-shrink-0 mr-3 mt-0.5">
|
||||
<i class="fas fa-exclamation-circle text-red-500"></i>
|
||||
</div>
|
||||
<div class="flex-1">
|
||||
<p class="text-sm font-medium text-red-800">Automatic Update Needs Attention</p>
|
||||
<p class="text-sm text-red-700 mt-1" id="auto-update-banner-text"></p>
|
||||
<p class="text-xs text-red-600 mt-1">Details are under Automatic Updates on the General tab.</p>
|
||||
</div>
|
||||
<button type="button" onclick="window.dismissAutoUpdateBanner()" class="ml-4 flex-shrink-0 text-red-500 hover:text-red-600" aria-label="Dismiss">
|
||||
<i class="fas fa-times"></i>
|
||||
</button>
|
||||
</div>
|
||||
<script>
|
||||
(function () {
|
||||
fetch('/api/v3/system/auto-update')
|
||||
.then(function (r) { return r.json(); })
|
||||
.then(function (resp) {
|
||||
var d = resp.data || {};
|
||||
if (!d.alert) return;
|
||||
document.getElementById('auto-update-banner-text').textContent = d.alert;
|
||||
var banner = document.getElementById('auto-update-banner');
|
||||
banner.dataset.alertId = d.alert_id || '';
|
||||
banner.style.setProperty('display', 'flex', 'important');
|
||||
})
|
||||
.catch(function () {});
|
||||
|
||||
// Dismissal is stored server-side, so the banner stays gone on every
|
||||
// device until a new failure raises a new alert.
|
||||
window.dismissAutoUpdateBanner = function () {
|
||||
var banner = document.getElementById('auto-update-banner');
|
||||
banner.style.setProperty('display', 'none', 'important');
|
||||
fetch('/api/v3/system/auto-update/dismiss', {
|
||||
method: 'POST',
|
||||
headers: {'Content-Type': 'application/json'},
|
||||
body: JSON.stringify({alert_id: banner.dataset.alertId})
|
||||
}).catch(function () {});
|
||||
};
|
||||
}());
|
||||
</script>
|
||||
<!-- Reconciliation warning banner: shown when startup reconciliation found stale plugin config entries -->
|
||||
<div id="reconciliation-banner" class="bg-yellow-50 border border-yellow-300 rounded-lg p-4 mb-4 flex items-start" style="display:none !important" role="alert">
|
||||
<div class="flex-shrink-0 mr-3 mt-0.5">
|
||||
|
||||
Reference in New Issue
Block a user