docs(changelog): complete 3.4.0 with weekly auto-updates and scroll timing fix (#584)

* docs(changelog): complete 3.4.0 with weekly auto-updates and scroll timing fix

#581 (weekly automatic updates) and #582 (scroll frame-stats idle gap)
merged after the 3.4.0 section was written in #580. v3.4.0 will be tagged
on main including both, so they belong in 3.4.0.

Also record src.common.font_layout (#539, #565), a src.* module plugins
may import that shipped in 3.4.0 but was never listed, and mark
display_geometry and auto_update_setup as core-internal.

Correct the 3.3.0 historical note: remote tags v3.3.0 (bc2dbf38) and
v3.3.1 (32d637a4) both report "3.3.0" and both ship sports_shared.py.
The "3.2.0" claim came from a stale local tag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(changelog): cover the whole 3.4.0 release since v3.3.1

The 3.4.0 section listed plugin-facing API and per-element customization
but not the rest of what merged since v3.3.1. Group it under subheadings:
Install and updates, Scrolling, Plugins, Web interface, Tools and
security, Fixes, and put the existing customization block under its own
heading.

Omitted on purpose: #569 (fixes a regression and an editor race in the
unreleased per-element framework), #570 (no runtime change), and
test-only, refactor and dev-tooling PRs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -19,25 +19,6 @@ accepts both, but the store flags the old spelling as deprecated
 
 ## Unreleased
 
-New module a plugin may import via `src.*` (floor on the release that ships
-this):
-
-- `src/common/sports_helpers.py` — the helpers the scoreboards' `sports.py`
-  carry byte-identical copies of: `clamp_window`, `clamp_seconds`,
-  `logo_needs_refresh`, `spread_weighted_order` (+ `MIN_WINDOW_DAYS`,
-  `MAX_WINDOW_DAYS`), and `SportsHelpersMixin` with `_mode_customization`,
-  `_setting_int`, `_reset_dwell_on_reentry`, `_next_switch_index`,
-  `_spread_weighted_order`, `_odds_color`, `_upcoming_date_and_time_text` under
-  the plugins' names and signatures, plus the `_favorite_key` override point.
-  No `__init__` and no state; the host contract is in the module docstring.
-  A new module rather than more methods on `sports_shared`: a plugin that
-  deletes a copy and leans on an older module having grown the method fails at
-  runtime with `AttributeError`, which no load-time check sees, while a missing
-  module fails at load. Nothing in core uses it yet.
-- `test/test_common_is_hardware_free.py` — `src/common` must import without
-  `rgbmatrix` and never import `src.base_classes`, `src.display_manager` or
-  `src.plugin_system` at module level.
-
 ## 3.4.0
 
 Plugin-facing changes since 3.3.0 (tag `v3.3.1`) not covered further down:
@@ -60,7 +41,143 @@ Plugin-facing changes since 3.3.0 (tag `v3.3.1`) not covered further down:
 - **Web preview size** now comes from `src/display_geometry.py`, the same
   computation `DisplayManager` uses: double-sided setups preview one screen,
   and a missing `chain_length` defaults to 2 everywhere (the Starlark magnify
-  default and the sync handshake used 1).
+  default and the sync handshake used 1). The module is core-internal: plugins
+  keep reading `display_manager.width`/`height`.
+- `src.common.font_layout` (#539, #565) — `load_truetype()` is
+  `ImageFont.truetype` with the layout engine pinned, so text lays out the same
+  whether or not the host's Pillow was built with libraqm; `crisp_size()` and
+  `FONT_PIXEL_GRID` give the size a bundled face renders on whole pixels at
+  (`sports_card` still re-exports them); `resolve_asset_path()` resolves
+  `assets/fonts/...` against the install root, not the working directory.
+  Floor on 3.4.0 to import it. Relatedly, `DisplayManager` now draws text
+  1-bit (#521), so golden images recorded against 3.3.x may need regenerating.
+
+### Install and updates
+
+**Weekly automatic updates (#581), off by default.** Switching on
+*Automatically check for and install updates once a week* on the General tab
+(or `first_time_install.sh --enable-auto-update` / `LEDMATRIX_AUTO_UPDATE=1`)
+updates the core and then every installed plugin once a week, preferably 2–5 AM
+local time. It follows the branch the checkout tracks — `main` on a standard
+install — so a device gets whatever has merged there, not only tagged releases.
+See `docs/WEB_INTERFACE_GUIDE.md`.
+
+- The core step is skipped, with the reason shown, when the checkout has local
+  edits or commits, a rebase or merge is in progress, the branch has no
+  upstream, less than 300 MB is free, or that commit was already rolled back.
+- After pulling, `ledmatrix-update-verify.service` restarts the services and
+  requires the web interface to answer and the display to stay up. If they
+  don't, or the new requirements fail to install, it resets to the previous
+  commit, reinstalls its requirements and restarts again. Anything but success
+  shows under the toggle and as a banner on Overview.
+- Plugins update through the Plugin Store even when the core step is skipped,
+  fails or is rolled back. A plugin version whose `ledmatrix_min_version` is
+  above the device's core is held back, not installed. When the core did
+  update, plugins wait for its health check, and are left alone if that check
+  never reports or the rollback fails.
+- No SSH is needed: switching the toggle on restarts the display service, which
+  installs the health-check units (`src/auto_update_setup.py`, core-internal
+  and not a plugin API).
+
+Installer and service fixes:
+
+- rgbmatrix builds on ARMv6 boards (Pi Zero, Pi 1); an existing checkout is
+  moved forward to the new pin and no longer left root-owned (#577).
+- `first_time_install.sh` grants the web user `safe_pip_install.sh`, as
+  `configure_web_sudo.sh` already did, so plugin requirements install where
+  the display service can see them (#579).
+- The web interface starts when `web_display_autostart` is missing or
+  `config.json` is unreadable; only an explicit `false` keeps it down (#556).
+- Installers render every systemd unit from its `systemd/` template, so the
+  boot-time unit-drift warning can clear, non-root installs included (#547).
+
+### Scrolling
+
+- **Frame pacing (#523).** The loop waits only for the rest of each panel
+  refresh instead of a flat 8 ms: 44–46 fps → 100 fps, and slow frames 14% →
+  0.02%, on a 2×128×64 chain. Sub-pixel blending is off by default again (it
+  shimmered on pixel fonts; Vegas mode still opts in).
+- **Whole-pixel steps (#545).** At a speed `scroll_config` can render in whole
+  pixels, every frame advances by exactly the same amount, removing about six
+  hitches a second. A loop that can't keep up now scrolls slightly slow rather
+  than jumping.
+- The eight sports scoreboards scroll through `scroll_config` too (#542): the
+  default 50 px/s holds each frame for two refreshes instead of alternating
+  0 px and 1 px steps.
+- **Frame stats ignore the pause between scrolls (#582).** The `Scroll frame
+  stats` log line counted the idle wait before each scroll as one frame,
+  inflating `max` and the stall rate. `docs/SCROLL_PERFORMANCE.md` now
+  describes the line actually logged.
+
+### Plugins
+
+- `FontManager` registers the bundled `tom_thumb` font, so plugins no longer
+  need a private loader (#534).
+- The test harness's `set_scrolling_state()` accepts `frame_hold`, as
+  `DisplayManager`'s does (#534).
+- A `display()` with nothing to draw should return `False`, the only value the
+  controller skips on; starlark-apps now does, rather than holding a black
+  panel (#534).
+- Starlark apps may set `render_width`/`render_height` in their `config.json`
+  to render at their own canvas size instead of Pixlet's 64×32 (#552).
+- `scripts/render_plugin.py --display-mode <mode>` renders one mode of a
+  multi-mode plugin; scoreboards previously rendered blank (#522).
+- Scoreboards resolve their own directory under the real plugin loader
+  (declare `_PLUGIN_DIR`), so 4x6 text snaps to its 7px grid instead of
+  rendering a pixel narrow, and an unreadable schema is logged (#519, #520).
+  `DisplayManager` loads 4x6 on that grid too (#565).
+- The 5x7 BDF face reports a real height, so rows stacked by
+  `get_font_height()` no longer overlap (#539).
+- `LogoHelper` remembers a missing logo instead of warning every rotation
+  (#548), and the decoded sports logo cache is bounded (#559).
+
+### Web interface
+
+- Installed Plugins has search, All / Enabled / Disabled / Updates filters and
+  sort (#540).
+- Hardened and polished per the September 2026 audit (#568): utility classes
+  such as `.hidden` actually exist, focus rings, labels and modal focus
+  trapping, dark theme throughout, no overflow at phone width, and background
+  streams pause when hidden, with first-load JS/CSS down from 1358 KB to 291 KB.
+- WiFi Connect works from the LEDMatrix-Setup hotspot: the page is answered
+  before the hotspot drops, and reopening it shows why an attempt failed (#571).
+- Pixlet install, the Starlark app store and app toggles work again (#535,
+  #537); the store uses the configured GitHub token and reports a rate limit
+  instead of drawing a blank grid (#541).
+- Plugin config: geochron and news saves no longer always fail (#575), the page
+  survives stored values the schema outgrew (#578), the form uses the full page
+  height (#573), and file-manager widgets show the script's error (#574).
+- The live status stream reports real disk usage and available memory (#558);
+  a system action refused for want of passwordless sudo says so and names
+  `configure_web_sudo.sh` (#560).
+
+### Tools and security
+
+- **CodeQL triage (#561):** 129 of 134 alerts fixed. Three were exploitable
+  path-handling flaws in the web interface and are closed; web UI escapers now
+  escape quotes, and URL fields refuse script schemes. Path checks share
+  `src/common/path_safety.py` (core-internal).
+- **Home Assistant MQTT bridge** (`integrations/mqtt_bridge`, #538): mode
+  select, stop, power and brightness over MQTT Discovery.
+- **Tools tab** manages the MQTT bridge and the Pixlet editor (#554); the
+  editor stays on loopback when `PIXLET_EDITOR_HOST` says so.
+
+### Fixes
+
+- Updating a plugin whose directory is named for its manifest id (leaderboard,
+  music, stocks, weather) silently did nothing (#536).
+- Plugin reconciliation no longer reports working plugins as stale or replaces
+  their config with a stub, and the Overview banner advises each case correctly
+  (#557).
+- Two config saves in the same second no longer share one backup, so rollback
+  restores the version asked for (#564).
+- On-demand: a second request is honoured without a restart (#534), a pinned
+  request stays on its mode, and restarting mid-session loads every plugin
+  again (#538).
+- `/health` and `/display/current` report real state, and the preview no longer
+  freezes on a leftover snapshot temp file (#534).
+
+### Per-element display customization
 
 **Per-element display customization, and the last mile of it into the web UI.**
 A user can set the font, size, colour, position, visibility and alignment of
@@ -152,7 +269,7 @@ Removed:
 
 ## 3.3.0
 
-Historical note: tag `v3.3.0` reports `__version__` "3.2.0" and tag `v3.3.1` reports "3.3.0", so a "3.3.0" floor is effectively `v3.3.1`, the first release shipping `src/common/sports_shared.py`.
+Historical note: tags `v3.3.0` and `v3.3.1` both report `__version__` "3.3.0" and both ship `src/common/sports_shared.py`, so a "3.3.0" floor always means a core with `sports_shared`.
 
 **The release the sports scoreboards floor on to delete their bundled copies.**
 3.2.0 shipped the unified sports library and made `ledmatrix_min_version`

config/config.template.json

@@ -1,5 +1,8 @@
 {
     "web_display_autostart": true,
+    "auto_update": {
+        "enabled": false
+    },
     "schedule": {
         "enabled": false,
         "mode": "per-day",

docs/SCROLL_PERFORMANCE.md

@@ -203,26 +203,61 @@ configured speed, so position stays proportional to real time.
 
 ## Diagnosing a juddery scroller
 
-**`Avg FPS` will lie to you.** It is a 100-frame moving average, and a 2 ms
+**An average will lie to you.** A 2 ms duplicate frame and a 21 ms double-wait
-duplicate frame plus a 21 ms double-wait average to exactly 10 ms. A ticker
+mean exactly 10 ms, so a ticker stalling on half its frames still averages to a
-that is stalling on half its frames still reports a healthy `100.0`.
+healthy 100 fps. The stats line reports the tail for that reason — read the
+percentiles, not the fps.
 
-Look at the **distribution** instead:
+Every scroller emits one line every 5 seconds covering *every* frame in that
+window, tagged with the plugin it came from:
 
 ```bash
-journalctl -u ledmatrix --since "-10min" --no-pager \
+journalctl -u ledmatrix --since "-10min" --no-pager | grep "Scroll frame stats"
-  | grep -oE "Frame time: [0-9.]+ms" | awk '{print $3}' | sed 's/ms//' \
+```
-  | awk '{printf "%.0f\n", $1}' | sort -n | uniq -c
+
+```
+[Plugin: news] Scroll frame stats - 100.0 fps over 501 frames | median 10.00ms
+p95 10.11ms max 12.03ms min 7.98ms | stalls 0 (0.0%) skips 0 (0.0%)
 ```
 
 Reading it, on a 100 Hz panel:
 
 | you see | it means |
 |---|---|
-| everything at 10 ms | healthy |
+| median 10 ms, p95 within ~0.5 ms of it | healthy — locked to the panel |
-| a mode at ~2 ms | **duplicate frames** — the swap was skipped because the image did not change. The scroller is advancing less than one pixel per frame. |
+| p95 or max at 20/30/50 ms | frames missing refreshes — per-frame work is overrunning, or a background thread is holding the GIL |
-| a mode at 20/30/50 ms | frames missing refreshes — per-frame work is overrunning, or a background thread is holding the GIL |
+| non-zero **skips**, or a median *below* 10 ms | **duplicate frames** — the swap was skipped because the image did not change, so the frame never waited on vsync. The scroller is advancing less than one pixel per frame. |
-| `Avg FPS` above 100 | duplicates present, unless the scroll cycle has completed and is idling |
+| non-zero **stalls** | frames past 1.5× the median, which is the measure of judder that survives averaging |
+
+`stalls` and `skips` are both counted against that window's own median, so they
+stay meaningful on a panel running at any refresh rate.
+
+To rank every scroller at once rather than reading lines one at a time:
+
+```bash
+journalctl -u ledmatrix --since "-3h" --no-pager | grep "Scroll frame stats" \
+  | sed -E 's/.*- (\S+) - (\[Plugin: [^]]+\] )?Scroll.*median ([0-9.]+)ms p95 ([0-9.]+)ms.*/\1 \3 \4/' \
+  | awk '$2 < 1000 {n[$1]++; m[$1]+=$2; p[$1]+=$3} END {for (k in n)
+        printf "%-28s %5d windows  median %6.2fms  p95 %6.2fms\n", k, n[k], m[k]/n[k], p[k]/n[k]}' \
+  | sort -k7 -rn
+```
+
+The `$2 < 1000` guard drops windows whose median is a whole second or more.
+Those are not frames. Until the idle-gap fix in `log_frame_rate()`, the first
+frame of every scroll was timed against the end of the *previous* scroll, so
+the gap between them was recorded as one enormous sample — it landed in the
+`max` field of otherwise healthy windows and counted as one stall per scroll,
+roughly 0.2% at 500 frames to a window, which is the same order as the real
+stall rates it sat beside. Current builds emit none, but the guard costs
+nothing and keeps the command honest against older journals.
+
+A scroller whose p95 sits several times its median is the one to fix, and it is
+usually the one doing the most per-frame work rather than the one configured
+worst. Measured over 20 minutes with two scrollers set identically at 100 px/s,
+the leaderboard held 10 ms flat while the odds ticker spent ~20% of its frames
+on duplicates. Same settings, different render cost: odds does more per-frame
+work, and more variably, so it is first to land a frame that advances less than
+a whole pixel. Check the render path before the config.
 
 Then confirm what the plugin actually loaded — config edits do not always reach
 the running code:

docs/SPORTS_UNIFICATION.md

@@ -80,30 +80,11 @@ src/base_classes/sports/
 src/common/
   sports_scroll.py       SportsScrollDisplay / …Manager — scroll orchestration
                          (content building stays in the plugins)
-  sports_helpers.py      clamp/logo/rotation free functions + SportsHelpersMixin
-                         (unreleased) — the helpers byte-identical in the
-                         plugins' sports.py, and the _favorite_key seam
 ```
 
 `from src.base_classes.sports import SportsCore` keeps working — the package
 `__init__` re-exports, so the conversion is invisible to every existing importer.
 
-### Converging on `src/common`
-
-The scoreboards do not build on `src/base_classes`; their own `sports.py` copies
-have moved past it. So shared code now lands in hardware-free `src/common`
-modules taken from the plugin copies, each a **new module** rather than growth
-on an existing one: a plugin that deletes a method copy and relies on an older
-module having gained it fails at runtime with an `AttributeError`, while a
-missing module fails at load, where the version checks can see it.
-`sports_helpers.py` is the first (it holds `_favorite_key`, the override point
-listed below, for later phases); its parity test compares every body against
-the plugin copies when `LEDMATRIX_PLUGINS` points at a checkout, and
-`test/test_common_is_hardware_free.py` keeps `src/common` free of
-`rgbmatrix`, `src.base_classes` and `src.plugin_system`. How a plugin adopts a
-module and drops its copy is documented in the plugins repo's
-`docs/plugin-development/08-shared-sports-code.md`.
-
 ## Override points (the plugin-facing seam)
 
 The base class calls these; plugins implement or override them. This table is the

docs/WEB_INTERFACE_GUIDE.md

@@ -96,6 +96,33 @@ Configure basic system settings:
 - **Plugin System Settings** — including the `plugins_directory` (default
   `plugin-repos/`) used by the plugin loader
 - **Autostart** options for the display service
+- **Automatic updates** — once a week, update LEDMatrix and every installed
+  plugin with a newer version. Off by default. Runs 2–5 AM local time when
+  possible, otherwise within a day of being due. The last result and next
+  check are shown under the toggle, and anything other than success raises a
+  banner on **Overview**.
+  - *Checks first:* the code update is skipped, with the reason shown, if
+    tracked files were edited locally, the checkout has local commits, a
+    rebase/merge is in progress, the branch has no upstream, less than 300 MB
+    is free, or the newest version already failed once. A failed fetch is
+    retried the next day.
+  - *Health check and rollback:* after pulling, `ledmatrix-update-verify.service`
+    restarts the services and checks that the web interface responds and the
+    display (if it was running) stays up. If not — or if the new dependencies
+    failed to install — it resets to the previous commit, reinstalls the
+    previous dependencies and restarts again. A running display is restarted;
+    a stopped one stays stopped.
+  - *Plugins* update through the Plugin Store, which refuses versions that need
+    a newer LEDMatrix and restores the old copy when an install fails. When the
+    code changed, plugins wait until it passes its health check. Plugins are
+    not health-checked after updating.
+  - *Setup needs no SSH.* Turning the toggle on restarts the display service,
+    which installs the health check (`ledmatrix-update-verify.path` and
+    `.service`); the General tab shows when it is ready, or why setup failed.
+    Until then only plugins update. New installs set it up during
+    installation and can switch updates on with
+    `first_time_install.sh --enable-auto-update` (or `LEDMATRIX_AUTO_UPDATE=1`,
+    which `one-shot-install.sh` passes through), or at the installer's prompt.
 
 Click **Save** to write changes to `config/config.json`. Most changes
 require a display service restart from **Overview**.

first_time_install.sh

@@ -220,6 +220,8 @@ SKIP_PERF=${LEDMATRIX_SKIP_PERF:-0}
 SKIP_REBOOT_PROMPT=${LEDMATRIX_SKIP_REBOOT_PROMPT:-0}
 SKIP_SWAP=${LEDMATRIX_SKIP_SWAP:-0}
 BUILD_JOBS_OVERRIDE=${LEDMATRIX_BUILD_JOBS:-}
+# Weekly automatic updates: 1 on, 0 off, empty = ask (interactive) or leave as is.
+AUTO_UPDATE=${LEDMATRIX_AUTO_UPDATE:-}
 
 usage() {
     cat <<USAGE
@@ -234,12 +236,15 @@ Options:
       --skip-swap           Never add temporary swap for the C++ build
       --build-jobs N        Compile the C++ library with N parallel jobs
                             (default: scaled to available RAM)
+      --enable-auto-update  Turn on weekly automatic updates (with health
+                            check and automatic rollback)
+      --no-auto-update      Leave weekly automatic updates off
   -h, --help                Show this help message and exit
 
 Environment variables (same effect as flags):
   LEDMATRIX_ASSUME_YES=1, RPI_RGB_FORCE_REBUILD=1, LEDMATRIX_SKIP_SOUND=1,
   LEDMATRIX_SKIP_PERF=1, LEDMATRIX_SKIP_REBOOT_PROMPT=1,
-  LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N
+  LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N, LEDMATRIX_AUTO_UPDATE=1|0
 
 Low-memory devices:
   On a Pi with under 2GB of RAM the C++ build is limited to fewer parallel
@@ -257,6 +262,8 @@ while [ $# -gt 0 ]; do
         --skip-perf) SKIP_PERF=1 ;;
         --no-reboot-prompt) SKIP_REBOOT_PROMPT=1 ;;
         --skip-swap) SKIP_SWAP=1 ;;
+        --enable-auto-update) AUTO_UPDATE=1 ;;
+        --no-auto-update) AUTO_UPDATE=0 ;;
         --build-jobs)
             shift
             if [ $# -eq 0 ]; then echo "--build-jobs requires a number"; usage; exit 1; fi
@@ -863,6 +870,50 @@ else
     echo "✓ Main config file already exists"
 fi
 
+# Weekly automatic updates (General tab -> Automatic Updates). Off unless asked
+# for: --enable-auto-update / LEDMATRIX_AUTO_UPDATE=1, or "y" at the prompt when
+# installing interactively. Only an explicit choice changes the setting, so
+# re-running the installer with -y never switches it silently.
+if [ -z "$AUTO_UPDATE" ] && [ "$ASSUME_YES" != "1" ] && [ -t 0 ]; then
+    read -p "Automatically check for and install LEDMatrix updates once a week, with automatic rollback if an update breaks something? (y/N): " -n 1 -r
+    echo
+    if [[ $REPLY =~ ^[Yy]$ ]]; then AUTO_UPDATE=1; else AUTO_UPDATE=0; fi
+fi
+if [ "$AUTO_UPDATE" = "1" ] || [ "$AUTO_UPDATE" = "0" ]; then
+    if python3 - "$PROJECT_ROOT_DIR/config/config.json" "$AUTO_UPDATE" <<'PY'
+import json, os, sys, tempfile
+path, enabled = sys.argv[1], sys.argv[2] == "1"
+with open(path, encoding="utf-8") as f:
+    config = json.load(f)
+if not isinstance(config.get("auto_update"), dict):
+    config["auto_update"] = {}
+config["auto_update"]["enabled"] = enabled
+# Written beside the original and swapped in whole: the display service's
+# config watcher may be running and must never read a half-written file.
+original = os.stat(path)
+fd, tmp = tempfile.mkstemp(dir=os.path.dirname(os.path.abspath(path)), prefix=".config.")
+try:
+    with os.fdopen(fd, "w", encoding="utf-8") as f:
+        json.dump(config, f, indent=4)
+        f.write("\n")
+        f.flush()
+        os.fsync(f.fileno())
+    os.chmod(tmp, original.st_mode & 0o777)
+    if hasattr(os, "chown"):
+        os.chown(tmp, original.st_uid, original.st_gid)
+    os.replace(tmp, path)
+except BaseException:
+    if os.path.exists(tmp):
+        os.unlink(tmp)
+    raise
+PY
+    then
+        if [ "$AUTO_UPDATE" = "1" ]; then echo "✓ Weekly automatic updates enabled"; else echo "✓ Weekly automatic updates off"; fi
+    else
+        echo "⚠ Could not set auto_update in config/config.json; turn it on from the General tab instead"
+    fi
+fi
+
 # Create config_secrets.json from template if missing
 if [ ! -f "$PROJECT_ROOT_DIR/config/config_secrets.json" ]; then
     if [ -f "$PROJECT_ROOT_DIR/config/config_secrets.template.json" ]; then
@@ -1341,7 +1392,7 @@ if [ -f "$PROJECT_ROOT_DIR/scripts/install/install_web_service.sh" ]; then
         fi
     fi
     
-    if [ ! -f "/etc/systemd/system/ledmatrix-web.service" ] || [ "$NEEDS_UPDATE" = true ]; then
+    if [ ! -f "/etc/systemd/system/ledmatrix-web.service" ] || [ ! -f "/etc/systemd/system/ledmatrix-update-verify.path" ] || [ "$NEEDS_UPDATE" = true ]; then
         bash "$PROJECT_ROOT_DIR/scripts/install/install_web_service.sh"
         # Ensure systemd sees any new/changed unit files
         systemctl daemon-reload || true
@@ -1357,7 +1408,7 @@ echo ""
 CURRENT_STEP="Harden systemd unit file permissions"
 echo "Step 8.1: Setting systemd unit file permissions..."
 echo "-----------------------------------------------"
-for unit in "/etc/systemd/system/ledmatrix.service" "/etc/systemd/system/ledmatrix-web.service" "/etc/systemd/system/ledmatrix-wifi-monitor.service"; do
+for unit in "/etc/systemd/system/ledmatrix.service" "/etc/systemd/system/ledmatrix-web.service" "/etc/systemd/system/ledmatrix-wifi-monitor.service" "/etc/systemd/system/ledmatrix-update-verify.service" "/etc/systemd/system/ledmatrix-update-verify.path"; do
     if [ -f "$unit" ]; then
         chown root:root "$unit" || true
         chmod 644 "$unit" || true

requirements.txt

@@ -43,10 +43,14 @@ packaging>=23.0,<27.0
 # full feature set, or skip them for a minimal install.
 # ───────────────────────────────────────────────────────────────────────
 #
-# scipy             — sub-pixel interpolation in
+# scipy             — nothing, as of #570. It was listed for the sub-pixel
-#                     src/common/scroll_helper.py for smoother
+#                     interpolation path in src/common/scroll_helper.py, but
-#                     scrolling. Falls back to a simpler shift algorithm.
+#                     get_visible_portion never consulted HAS_SCIPY, so that
-#   pip install 'scipy>=1.10.0,<2.0.0'
+#                     path was dead before it was deleted. The blend that
+#                     replaced it is numpy-only. Do not install it expecting
+#                     smoother scrolling: sub-pixel blending is off by default
+#                     because it reads worse on a coarse panel, not because it
+#                     is missing a library. See docs/SCROLL_PERFORMANCE.md.
 #
 # psutil            — per-plugin resource monitoring in
 #                     src/plugin_system/resource_monitor.py. The monitor

scripts/install/install_service.sh

@@ -95,6 +95,19 @@ else
     exit 1
 fi
 
+# Health check / rollback units for automatic updates; see install_web_service.sh.
+for VERIFY_UNIT in ledmatrix-update-verify.service ledmatrix-update-verify.path; do
+    if [ -f "$PROJECT_ROOT_DIR/systemd/$VERIFY_UNIT" ]; then
+        VERIFY_UNIT_TMP=$(mktemp)
+        if sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g; s|__USER__|$ESCAPED_ACTUAL_USER|g" "$PROJECT_ROOT_DIR/systemd/$VERIFY_UNIT" > "$VERIFY_UNIT_TMP"; then
+            sudo cp "$VERIFY_UNIT_TMP" "/etc/systemd/system/$VERIFY_UNIT"
+        else
+            echo "WARNING: failed to render $VERIFY_UNIT; automatic code updates will stay paused." >&2
+        fi
+        rm -f "$VERIFY_UNIT_TMP"
+    fi
+done
+
 echo "Reloading systemd daemon for web service..."
 sudo systemctl daemon-reload
 
@@ -102,6 +115,11 @@ if [ -f "/etc/systemd/system/ledmatrix-web.service" ]; then
     echo "Enabling ledmatrix-web.service to start on boot..."
     sudo systemctl enable ledmatrix-web.service
 
+    if [ -f /etc/systemd/system/ledmatrix-update-verify.path ]; then
+        echo "Enabling ledmatrix-update-verify.path (automatic update health check)..."
+        sudo systemctl enable --now ledmatrix-update-verify.path || echo "WARNING: could not enable ledmatrix-update-verify.path; automatic code updates will stay paused" >&2
+    fi
+
     echo "Starting ledmatrix-web.service..."
     sudo systemctl start ledmatrix-web.service
 

scripts/install/install_web_service.sh

@@ -46,6 +46,22 @@ ESCAPED_ACTUAL_USER=$(sed_escape_replacement "$ACTUAL_USER")
 sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g; s|__USER__|$ESCAPED_ACTUAL_USER|g" \
     "$TEMPLATE" > /etc/systemd/system/ledmatrix-web.service
 
+# Health check and rollback for the web UI's automatic updates. Its own unit so
+# it survives the web service restart it performs; never enabled -- the web
+# interface starts it after an update. Without it, automatic code updates
+# stay paused rather than running with nothing to undo them.
+for VERIFY_UNIT in ledmatrix-update-verify.service ledmatrix-update-verify.path; do
+    VERIFY_TEMPLATE="$PROJECT_ROOT_DIR/systemd/$VERIFY_UNIT"
+    if [ -f "$VERIFY_TEMPLATE" ]; then
+        echo "Writing unit file to /etc/systemd/system/$VERIFY_UNIT"
+        sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g; s|__USER__|$ESCAPED_ACTUAL_USER|g" \
+            "$VERIFY_TEMPLATE" > "/etc/systemd/system/$VERIFY_UNIT"
+        chmod 644 "/etc/systemd/system/$VERIFY_UNIT"
+    else
+        echo "WARNING: $VERIFY_TEMPLATE not found; automatic code updates will stay paused"
+    fi
+done
+
 # Ensure cache directory exists with proper permissions
 # This is a fallback for older systemd versions that don't support CacheDirectory
 # Systemd 239+ will automatically create it via CacheDirectory directive
@@ -80,6 +96,13 @@ systemctl daemon-reload
 echo "Enabling ledmatrix-web.service..."
 systemctl enable ledmatrix-web.service
 
+# The path unit is what starts the health check after an automatic update.
+if [ -f /etc/systemd/system/ledmatrix-update-verify.path ]; then
+    echo "Enabling ledmatrix-update-verify.path..."
+    systemctl enable --now ledmatrix-update-verify.path || \
+        echo "WARNING: could not enable ledmatrix-update-verify.path; automatic code updates will stay paused"
+fi
+
 # Start the service
 echo "Starting ledmatrix-web.service..."
 systemctl start ledmatrix-web.service

scripts/install/one-shot-install.sh

@@ -408,6 +408,7 @@ main() {
         # which would silently reinstate the duplicate apt update.
         sudo -E env TMPDIR=/tmp LEDMATRIX_ASSUME_YES=1 \
             LEDMATRIX_APT_UPDATED="${LEDMATRIX_APT_UPDATED:-0}" \
+            LEDMATRIX_AUTO_UPDATE="${LEDMATRIX_AUTO_UPDATE:-}" \
             bash ./first_time_install.sh -y </dev/null
     fi
     INSTALL_EXIT_CODE=$?

scripts/utils/auto_update_verify.py

@@ -0,0 +1,277 @@
+#!/usr/bin/env python3
+"""Check that an automatic LEDMatrix update left the device working; roll it back if not.
+
+Started by the web interface's weekly updater (web_interface/auto_update.py)
+through ledmatrix-update-verify.service, right after it pulls new code. It has
+to run outside the web service: checking the update means restarting that
+service, and a check running inside it would be killed by its own restart.
+
+It must not be the code it is checking, either. The updater copies this file
+to data/auto_update_verifier.py *before* pulling and the unit runs that copy,
+so a broken update cannot break its own rollback. Standard library only for
+the same reason: the rollback cannot depend on packages the update changed.
+
+The updater leaves data/auto_update_pending.json:
+
+    {"status": "pending", "old_head": ..., "new_head": ...,
+     "display_was_active": bool, "dependency_failures": [...], "created_at": ...}
+
+This moves its status to "verifying" and then to one of "success",
+"rolled_back" or "rollback_failed", with "reason" and "detail" saying why.
+The web interface reports that outcome and raises a banner for anything but
+success.
+"""
+import json
+import os
+import subprocess  # nosec B404 - list-form argv only, no shell  # nosemgrep
+import sys
+from collections import namedtuple
+import tempfile
+import time
+import traceback
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+PENDING_NAME = 'auto_update_pending.json'
+REQUIREMENT_FILES = ('requirements.txt', 'web_interface/requirements.txt')
+WEB_HEALTH_URL = 'http://127.0.0.1:5000/api/v3/system/version'
+#: How long the services get to come up after a restart...
+HEALTH_TIMEOUT_SECONDS = 180
+#: ...and how long they must then stay up. Restart=on-failure makes a crash
+#: loop look healthy between attempts, so a single "is-active" proves nothing.
+STABLE_SECONDS = 45
+POLL_SECONDS = 5
+PIP_TIMEOUT_SECONDS = 600
+#: sudoers matches the exact command line, so bash is named by path, the same
+#: candidates src/common/permission_utils.install_requirements_file tries.
+BASH_CANDIDATES = ('/usr/bin/bash', '/bin/bash')
+
+#: What a command that could not run at all reports: its callers only read
+#: these three fields, the same ones a completed subprocess has.
+_Failed = namedtuple('_Failed', 'returncode stdout stderr')
+
+
+def pending_path(project_root):
+    return Path(project_root) / 'data' / PENDING_NAME
+
+
+def read_pending(path):
+    try:
+        with open(path, 'r', encoding='utf-8') as f:
+            data = json.load(f)
+        return data if isinstance(data, dict) else None
+    except (OSError, ValueError):
+        return None
+
+
+def write_pending(path, data):
+    path = Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix='.auto_update_pending_')
+    try:
+        with os.fdopen(fd, 'w', encoding='utf-8') as f:
+            json.dump(data, f, indent=2)
+        os.replace(tmp, path)
+    except BaseException:
+        try:
+            os.unlink(tmp)
+        except OSError:
+            pass
+        raise
+
+
+def _web_responds(url=WEB_HEALTH_URL):
+    try:
+        with urllib.request.urlopen(url, timeout=5) as resp:  # nosec B310 - fixed loopback URL
+            return resp.status == 200
+    except (urllib.error.URLError, OSError, ValueError):
+        return False
+
+
+def _short(sha):
+    return (sha or 'unknown')[:7]
+
+
+class Verifier:
+    def __init__(self, project_root, run=subprocess.run, sleep=time.sleep,
+                 clock=time.monotonic, web_responds=_web_responds, log=None):
+        self.project_root = Path(project_root)
+        self.pending_file = pending_path(project_root)
+        self.run = run
+        self.sleep = sleep
+        self.clock = clock
+        self.web_responds = web_responds
+        self.log = log or (lambda msg: print(f'[auto-update-verify] {msg}', flush=True))
+
+    def _run(self, args, timeout=60):
+        try:
+            return self.run(args, cwd=str(self.project_root), capture_output=True,
+                            text=True, timeout=timeout)
+        except (subprocess.SubprocessError, OSError) as e:
+            return _Failed(returncode=1, stdout='', stderr=str(e))
+
+    # -- services ---------------------------------------------------------
+
+    def service_active(self, unit):
+        return self._run(['systemctl', 'is-active', unit], timeout=10).stdout.strip() == 'active'
+
+    def restart_count(self, unit):
+        out = self._run(['systemctl', 'show', '-p', 'NRestarts', '--value', unit],
+                        timeout=10).stdout.strip()
+        return int(out) if out.isdigit() else None
+
+    def restart(self, unit):
+        result = self._run(['sudo', '-n', 'systemctl', 'restart', f'{unit}.service'], timeout=90)
+        if result.returncode != 0:
+            self.log(f'restarting {unit} failed: {(result.stderr or "").strip()}')
+        return result.returncode == 0
+
+    def restart_services(self, display):
+        """Restart what should be running. False if any restart command failed."""
+        ok = True
+        # A display the user had stopped stays stopped.
+        if display:
+            ok = self.restart('ledmatrix') and ok
+        return self.restart('ledmatrix-web') and ok
+
+    def wait_healthy(self, display):
+        """None once the services are up and stay up, else what went wrong."""
+        deadline = self.clock() + HEALTH_TIMEOUT_SECONDS + STABLE_SECONDS
+        healthy_since = baseline = None
+        web = disp = False
+        count_known = True
+        while self.clock() < deadline:
+            web = self.web_responds()
+            disp = self.service_active('ledmatrix') if display else True
+            restarts = self.restart_count('ledmatrix') if display else None
+            # Without a restart count a crash loop looks healthy between
+            # attempts, so an unreadable count never counts as stable.
+            count_known = not display or restarts is not None
+            if web and disp and count_known and (healthy_since is None or restarts == baseline):
+                if healthy_since is None:
+                    healthy_since, baseline = self.clock(), restarts
+                elif self.clock() - healthy_since >= STABLE_SECONDS:
+                    return None
+            else:
+                healthy_since = None
+            self.sleep(POLL_SECONDS)
+        problems = []
+        if not web:
+            problems.append('the web interface did not respond')
+        if not disp:
+            problems.append('the display service did not stay running')
+        if web and disp and not count_known:
+            problems.append("the display service's restart count could not be read")
+        return '; '.join(problems) or 'the display service kept restarting'
+
+    # -- rollback ---------------------------------------------------------
+
+    def changed_requirements(self, old, new):
+        result = self._run(['git', 'diff', '--name-only', old, new])
+        # If the diff is unavailable, reinstall both rather than guess.
+        changed = set(result.stdout.split()) if result.returncode == 0 else set(REQUIREMENT_FILES)
+        return [rel for rel in REQUIREMENT_FILES if rel in changed]
+
+    def install_requirements(self, rel):
+        wrapper = self.project_root / 'scripts' / 'fix_perms' / 'safe_pip_install.sh'
+        req = self.project_root / rel
+        if not req.exists():
+            return True
+        for bash in BASH_CANDIDATES:
+            result = self._run(['sudo', '-n', bash, str(wrapper), str(req)],
+                               timeout=PIP_TIMEOUT_SECONDS)
+            if result.returncode == 0:
+                return True
+        return False
+
+    def rollback(self, pending):
+        """Reset to the previous commit and its dependencies. Returns (ok, detail)."""
+        old, new = pending.get('old_head'), pending.get('new_head')
+        if not old:
+            return False, 'the commit to roll back to is unknown'
+        requirements = self.changed_requirements(old, new) if new else list(REQUIREMENT_FILES)
+        # Safe to --hard: the updater refuses to run with local edits to
+        # tracked files, so the only thing this discards is the update.
+        result = self._run(['git', 'reset', '--hard', old], timeout=120)
+        if result.returncode != 0:
+            return False, (f'"git reset --hard {old}" failed: '
+                           f'{(result.stderr or result.stdout or "").strip()}')
+        failed = [rel for rel in requirements if not self.install_requirements(rel)]
+        if failed:
+            return True, ('reinstalling the previous dependencies from ' + ', '.join(failed)
+                          + ' failed; run Install Base Requirements from the Tools tab')
+        return True, ''
+
+    # -- the check itself -------------------------------------------------
+
+    def _finish(self, pending, status, reason=None, detail=None):
+        pending.update({'status': status, 'reason': reason, 'detail': detail or None,
+                        'finished_at': time.time()})
+        write_pending(self.pending_file, pending)
+        self.log(' '.join(p for p in (status, reason or '', detail or '') if p))
+
+    def verify(self):
+        pending = read_pending(self.pending_file)
+        if not pending or pending.get('status') != 'pending':
+            self.log('no update is waiting to be verified')
+            return 0
+        pending['status'] = 'verifying'
+        write_pending(self.pending_file, pending)
+
+        display = bool(pending.get('display_was_active'))
+        dependency_failures = pending.get('dependency_failures') or []
+        if dependency_failures:
+            # Never restart onto code whose packages did not install.
+            reason = 'installing its dependencies failed (' + ', '.join(dependency_failures) + ')'
+        elif not self.restart_services(display):
+            # The old process may still be answering; checking it would pass
+            # an update that never started.
+            reason = 'restarting the services failed'
+        else:
+            reason = self.wait_healthy(display)
+            if reason is None:
+                self._finish(pending, 'success')
+                return 0
+
+        self.log(f'update to {_short(pending.get("new_head"))} is unhealthy ({reason}); '
+                 f'rolling back to {_short(pending.get("old_head"))}')
+        ok, detail = self.rollback(pending)
+        if not ok:
+            self._finish(pending, 'rollback_failed', reason, detail)
+            return 1
+        still = (self.wait_healthy(display) if self.restart_services(display)
+                 else 'restarting the services failed')
+        if still:
+            self._finish(pending, 'rollback_failed', reason,
+                         f'still unhealthy after rolling back: {still}'
+                         + (f'; {detail}' if detail else ''))
+            return 1
+        self._finish(pending, 'rolled_back', reason, detail)
+        return 0
+
+
+def main(argv):
+    if len(argv) != 2:
+        print('usage: auto_update_verify.py PROJECT_ROOT', file=sys.stderr)
+        return 2
+    verifier = Verifier(Path(argv[1]))
+    try:
+        return verifier.verify()
+    except Exception as e:
+        traceback.print_exc()
+        # Whatever happened, the web interface must not be left thinking the
+        # check is still running.
+        try:
+            pending = read_pending(verifier.pending_file) or {}
+            if pending.get('status') in ('pending', 'verifying'):
+                pending.update({'status': 'rollback_failed', 'reason': 'the health check crashed',
+                                'detail': str(e), 'finished_at': time.time()})
+                write_pending(verifier.pending_file, pending)
+        except OSError:
+            pass
+        return 1
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv))

src/auto_update_setup.py

@@ -0,0 +1,221 @@
+"""Install the automatic-update health check, from the display service.
+
+The weekly updater (web_interface/auto_update.py) will not update LEDMatrix
+code unless ledmatrix-update-verify.path and .service are installed: they
+restart the services after an update and roll it back if the device is
+unhealthy. Installing units takes root and the web interface is not root, and
+"SSH in and run an installer" means most people never get updates with a
+safety net.
+
+The display service already runs this repository's code as root, so it
+installs them -- but only while the user has automatic updates turned on, only
+these two units, rendered from the repository's templates for the web
+interface's own user, and it reports what happened in
+data/auto_update_setup.json for the General tab. It grants nothing new: the
+units run as the web user, who can already change the code this process runs.
+
+Called at display startup; the web interface restarts the display service
+when the toggle is switched on, so setup happens straight away. Refreshing a
+unit whose template changed happens the same way, which is why this compares
+content rather than only checking that the files exist.
+"""
+import json
+import logging
+import os
+import re
+import subprocess  # nosec B404 - list-form argv only, no shell  # nosemgrep
+import tempfile
+import time
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+SYSTEMD_DIR = Path('/etc/systemd/system')
+SERVICE_UNIT = 'ledmatrix-update-verify.service'
+PATH_UNIT = 'ledmatrix-update-verify.path'
+UNITS = (SERVICE_UNIT, PATH_UNIT)
+WEB_UNIT = 'ledmatrix-web.service'
+RESULT_REL = Path('data') / 'auto_update_setup.json'
+_USER_RE = re.compile(r'^[a-z_][a-z0-9_-]{0,31}$')
+
+
+class SetupError(Exception):
+    """A reason setup cannot proceed, worded for the General tab."""
+
+
+def _is_root():
+    return hasattr(os, 'geteuid') and os.geteuid() == 0
+
+
+def _lookup_ids(user):
+    try:
+        import pwd
+        entry = pwd.getpwnam(user)
+        return entry.pw_uid, entry.pw_gid
+    except (ImportError, KeyError):
+        return None
+
+
+def _directive(text, key):
+    match = re.search(rf'^{key}=(.*)$', text or '', re.M)
+    return match.group(1).strip() if match else None
+
+
+def _read(path):
+    try:
+        return Path(path).read_text(encoding='utf-8')
+    except OSError:
+        return None
+
+
+def is_enabled(config):
+    return bool((config.get('auto_update') or {}).get('enabled', False))
+
+
+class UpdateHelperSetup:
+    def __init__(self, project_root=PROJECT_ROOT, systemd_dir=SYSTEMD_DIR, run=subprocess.run,
+                 is_root=_is_root, lookup_ids=_lookup_ids, clock=time.time):
+        self.project_root = Path(project_root)
+        self.systemd_dir = Path(systemd_dir)
+        self.run = run
+        self.is_root = is_root
+        self.lookup_ids = lookup_ids
+        self.clock = clock
+        self.result_file = self.project_root / RESULT_REL
+        self._web_ids = None
+
+    def _systemctl(self, *args):
+        return self.run(['systemctl', *args], capture_output=True, text=True, timeout=60)
+
+    def _check(self, result, what):
+        if result.returncode != 0:
+            raise SetupError(f'"{what}" failed: {(result.stderr or result.stdout or "").strip()}')
+
+    def path_active(self):
+        try:
+            return self._systemctl('is-active', PATH_UNIT).stdout.strip() == 'active'
+        except (subprocess.SubprocessError, OSError):
+            return False
+
+    def ensure(self, config):
+        """Install or refresh the units while automatic updates are on.
+
+        Returns the result recorded for the General tab, or None when there
+        was nothing to do (updates off, or not a systemd host at all).
+        """
+        if not is_enabled(config) or not self.systemd_dir.is_dir():
+            return None
+        try:
+            changed = self._install()
+        except SetupError as e:
+            return self._report('failed', str(e))
+        except (OSError, subprocess.SubprocessError) as e:
+            return self._report('failed', f'Could not install the update health check: {e}')
+        if changed:
+            return self._report('installed', 'Installed the update health check.')
+        return self._report('installed', 'The update health check is installed.', quiet=True)
+
+    def _install(self):
+        if not self.is_root():
+            raise SetupError('The display service is not running as root, so it cannot install the '
+                             'update health check. Run "sudo ./scripts/install/install_web_service.sh" once.')
+
+        web_text = _read(self.systemd_dir / WEB_UNIT)
+        if web_text is None:
+            raise SetupError('The web interface service (ledmatrix-web.service) is not installed.')
+        user = _directive(web_text, 'User') or 'root'
+        ids = self.lookup_ids(user) if _USER_RE.match(user) else None
+        if ids is None:
+            raise SetupError(f'The web interface runs as "{user}", which is not a usable account.')
+        self._web_ids = ids
+        workdir = _directive(web_text, 'WorkingDirectory')
+        if not workdir or Path(workdir).resolve() != self.project_root.resolve():
+            raise SetupError(f'The web interface service runs from {workdir or "an unknown folder"}, '
+                             f'not {self.project_root}.')
+        # Spaces are fine -- the templates quote every command-line path --
+        # but systemd expands % specifiers, and a quote, backslash or line
+        # break would be reinterpreted in a unit file. (On Windows, where the
+        # tests also run, a backslash is the path separator, not a name.)
+        root_text = str(self.project_root)
+        unsafe = set('%"') | ({'\\'} if os.sep == '/' else set())
+        if any(ch in unsafe or ord(ch) < 32 for ch in root_text):
+            raise SetupError(f'LEDMatrix is installed in {root_text!r}, a folder name systemd cannot use '
+                             'in a unit file. Move it to a path without %, quotes, backslashes or '
+                             'control characters.')
+
+        rendered = {}
+        for name in UNITS:
+            template = _read(self.project_root / 'systemd' / name)
+            if template is None:
+                raise SetupError(f'The unit template systemd/{name} is missing.')
+            rendered[name] = (template.replace('__PROJECT_ROOT_DIR__', str(self.project_root))
+                              .replace('__USER__', user))
+        # The templates are ordinary repository files. Whatever they say,
+        # this root process only installs a service that runs as the web user
+        # and a path unit that starts exactly that service.
+        if _directive(rendered[SERVICE_UNIT], 'User') != user:
+            raise SetupError(f'systemd/{SERVICE_UNIT} does not run as the web interface user; '
+                             'refusing to install it.')
+        if _directive(rendered[PATH_UNIT], 'Unit') != SERVICE_UNIT:
+            raise SetupError(f'systemd/{PATH_UNIT} does not start {SERVICE_UNIT}; refusing to install it.')
+
+        changed = [name for name in UNITS if _read(self.systemd_dir / name) != rendered[name]]
+        for name in changed:
+            self._write_unit(self.systemd_dir / name, rendered[name])
+        if changed:
+            self._check(self._systemctl('daemon-reload'), 'systemctl daemon-reload')
+            self._check(self._systemctl('enable', PATH_UNIT), f'systemctl enable {PATH_UNIT}')
+            self._check(self._systemctl('restart', PATH_UNIT), f'systemctl restart {PATH_UNIT}')
+        elif not self.path_active():
+            self._check(self._systemctl('enable', '--now', PATH_UNIT), f'systemctl enable --now {PATH_UNIT}')
+            changed = [PATH_UNIT]
+        if not self.path_active():
+            raise SetupError(f'{PATH_UNIT} did not start; see "journalctl -u {PATH_UNIT}".')
+        return bool(changed)
+
+    def _write_unit(self, path, text):
+        fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f'.{path.name}.')
+        try:
+            with os.fdopen(fd, 'w', encoding='utf-8') as f:
+                f.write(text)
+            os.chmod(tmp, 0o644)
+            os.replace(tmp, path)
+        except BaseException:
+            try:
+                os.unlink(tmp)
+            except OSError:
+                pass
+            raise
+
+    def _report(self, status, message, quiet=False):
+        previous = None
+        try:
+            previous = json.loads(_read(self.result_file) or 'null')
+        except ValueError:
+            pass
+        if quiet and isinstance(previous, dict) and previous.get('status') == status:
+            return previous  # nothing new; don't rewrite it on every boot
+        result = {'status': status, 'message': message, 'at': self.clock()}
+        (logger.info if status == 'installed' else logger.warning)("Automatic update setup: %s", message)
+        try:
+            self.result_file.parent.mkdir(parents=True, exist_ok=True)
+            fd, tmp = tempfile.mkstemp(dir=str(self.result_file.parent), prefix='.auto_update_setup_')
+            with os.fdopen(fd, 'w', encoding='utf-8') as f:
+                json.dump(result, f, indent=2)
+            os.chmod(tmp, 0o644)
+            if self._web_ids and hasattr(os, 'chown'):
+                # Only root can give the file away; the result is readable
+                # (0644) either way, so a failed chown must not lose it.
+                try:
+                    os.chown(tmp, *self._web_ids)
+                except OSError:
+                    pass
+            os.replace(tmp, self.result_file)
+        except OSError as e:
+            logger.warning("Could not record automatic update setup result: %s", e)
+        return result
+
+
+def ensure_update_helper(config):
+    return UpdateHelperSetup().ensure(config)

src/common/scroll_helper.py

@@ -30,6 +30,12 @@ except ImportError:
     HAS_SCIPY = False
 
 
+# How often the frame-stats line is emitted, and therefore also the ceiling
+# on a believable frame time: a scroll that renders at all cannot take this
+# long over one frame, so a sample this large is an idle gap between scrolls.
+FPS_LOG_INTERVAL = 5.0
+
+
 def frame_stats(frame_times: list) -> Dict[str, Any]:
     """Summary statistics over one window of frame durations (seconds).
 
@@ -158,9 +164,11 @@ class ScrollHelper:
         self.last_progress_log_time: Optional[float] = None
         self.progress_log_interval = 5.0  # seconds
         
-        # Frame rate tracking
+        # Frame rate tracking. last_frame_time is None until the first frame
+        # of a scroll is rendered -- see log_frame_rate() for why timing from
+        # construction (or from the end of the previous scroll) is wrong.
         self.frame_count = 0
-        self.last_frame_time = time.time()
+        self.last_frame_time: Optional[float] = None
         self.last_fps_log_time = time.time()
         self.frame_times = []
         # Every frame time since the last stats line, so the 5s summary can
@@ -764,6 +772,10 @@ class ScrollHelper:
         # Reset last_update_time to prevent large delta_time on next update
         # This ensures smooth scrolling after reset without jumping ahead
         self.last_update_time = now
+        # Same reasoning for the frame-rate clock: the first frame after a
+        # reset has no predecessor in this scroll, and timing it against the
+        # last frame of the previous one measures the idle gap between them.
+        self.last_frame_time = None
         self.logger.debug("Scroll position reset")
 
     def reset(self) -> None:
@@ -962,11 +974,37 @@ class ScrollHelper:
         Log frame rate statistics for performance monitoring.
         """
         current_time = time.time()
-        
+
+        # The first frame of a scroll has no predecessor, so it has no frame
+        # time. Measuring one anyway records the whole idle gap since the last
+        # scroll as a single frame: on hardware that produced windows reading
+        # "0.0 fps over 1 frames | median 136776.02ms", and -- worse, because
+        # it is not obviously wrong -- put that gap in the max field of
+        # otherwise healthy windows and counted it as one stall per scroll.
+        # At ~500 frames to a window that is ~0.2%, which is the same order as
+        # the real stall rates being measured, so the number could not be
+        # trusted at all. Seed the clock and take no sample.
+        if self.last_frame_time is None:
+            self.last_frame_time = current_time
+            # Restart the window with the scroll. Otherwise the boundary is
+            # already long overdue when the second frame arrives, and the new
+            # scroll opens by reporting a window of exactly one frame.
+            self.last_fps_log_time = current_time
+            return
+
         # Calculate instantaneous frame time
         frame_time = current_time - self.last_frame_time
+
+        # A caller that scrolls without ever calling reset_scroll() never arms
+        # the sentinel above, so catch the same gap by its size. Nothing that
+        # renders a scroll produces a frame longer than the log interval; a
+        # sample that large is an idle period, not a frame.
+        if frame_time >= FPS_LOG_INTERVAL:
+            self.last_frame_time = current_time
+            return
+
         self.frame_times.append(frame_time)
-        
+
         # Keep only last 100 frames for average
         if len(self.frame_times) > 100:
             self.frame_times.pop(0)
@@ -979,17 +1017,21 @@ class ScrollHelper:
         # duplicate and a 21ms double-wait mean exactly 10ms). Chasing scroll
         # judder needs the tail, so keep the window and report percentiles.
         self._window.append(frame_time)
-        
+
         # Log FPS every 5 seconds to avoid spam
-        if current_time - self.last_fps_log_time >= 5.0:
+        if current_time - self.last_fps_log_time >= FPS_LOG_INTERVAL:
-            self.logger.info(
+            # An empty window means every sample in this interval was dropped
-                "Scroll frame stats - %s",
+            # as an idle gap. There is nothing to report, and reporting the
-                format_frame_stats(self._window or [frame_time]),
+            # gap itself is the bug above.
-            )
+            if self._window:
+                self.logger.info(
+                    "Scroll frame stats - %s",
+                    format_frame_stats(self._window),
+                )
             self.last_fps_log_time = current_time
             self.frame_count = 0
             self._window = []
-        
+
         self.last_frame_time = current_time
         self.frame_count += 1
     

src/common/sports_helpers.py

@@ -1,408 +0,0 @@
-"""Sports helpers every scoreboard's ``sports.py`` carries an identical copy of.
-
-Ten helpers are byte-identical (executable AST, docstrings stripped) in the
-scoreboard plugins' ``sports.py`` and have no equivalent elsewhere in
-``src/common``. The bodies below were copied from those plugin copies -- not
-from ``src/base_classes`` -- at ledmatrix-plugins ``f09bff2`` (origin/main,
-2026-09-14):
-
-- In all nine (afl, baseball, basketball, football, hockey, lacrosse, nrl,
-  soccer, ufc): ``_clamp_window``, ``_clamp_seconds``, ``_logo_needs_refresh``
-  (with ``_MIN_WINDOW_DAYS`` / ``_MAX_WINDOW_DAYS``), and the ``SportsCore``
-  methods ``_mode_customization``, ``_setting_int``,
-  ``_reset_dwell_on_reentry`` (with ``_DWELL_REENTRY_GAP_SECONDS``),
-  ``_next_switch_index`` and ``_spread_weighted_order``.
-- In the eight that share a lineage (all but ufc, which has neither):
-  ``_odds_color`` and ``_upcoming_date_and_time_text``.
-
-The module-level helpers become public free functions here (``clamp_window``,
-``clamp_seconds``, ``logo_needs_refresh``, ``spread_weighted_order``,
-``MIN_WINDOW_DAYS``, ``MAX_WINDOW_DAYS``); the methods keep the plugins' exact
-names and signatures on ``SportsHelpersMixin`` so adopting it is deleting the
-copies and adding one base. ``test/test_sports_helpers.py`` compares every body
-here against every plugin copy when a plugins checkout is available
-(``LEDMATRIX_PLUGINS``), so drift in either direction fails a test.
-
-WHY A NEW MODULE, NOT MORE METHODS ON sports_shared
----------------------------------------------------
-A plugin that deletes its copy of a method and relies on an *existing* module
-having grown it fails at runtime on an older core: ``AttributeError`` mid-render,
-usually swallowed into a blank board. The plugin loader cannot see that, and
-neither can the plugins' ``scripts/check_min_core_version.py``, which checks
-modules. A plugin importing a module that does not exist yet fails at load,
-where both can. Plugins import this and floor ``ledmatrix_min_version`` on the
-first core release that ships it (see ``CHANGELOG.md``).
-
-``_favorite_key`` is the one method not taken from the plugins: it is the
-override point from ``src/base_classes/sports/core.py``, carried here so later
-phases (shared celebrations and game selection) have a hardware-free home for
-the seam. No plugin defines it today and nothing in this module calls it.
-
-WHAT A HOST MUST PROVIDE
-------------------------
-Derived by walking every ``self.<attr>`` the mixin reads; the host-contract
-test in ``test/test_sports_helpers.py`` fails if a read is added without being
-listed here.
-
-- ``config`` (dict) -- ``_mode_customization``.
-- ``mode_config`` (dict) and ``logger`` -- ``_setting_int``. ``league`` is
-  read with ``getattr`` for the warning text only.
-- ``games_list`` and ``current_game_index`` -- ``_next_switch_index``; plus
-  ``_is_favorite_game`` (called with a game), which stays per-plugin and is
-  only called when
-  ``favorite_rotation_boost`` is above 1. ``favorite_rotation_boost`` itself
-  defaults to 1 on the mixin.
-- ``last_game_switch`` -- ``_reset_dwell_on_reentry``, read with ``getattr``
-  and written back.
-- ``_card_option``, ``_format_game_date`` and ``_format_game_time`` --
-  ``_upcoming_date_and_time_text``. All three are on
-  ``SportsCoreSharedMixin`` (``src/common/sports_shared.py``).
-- ``_element_color`` -- ``_odds_color``, looked up with ``getattr`` and falling
-  back to green, so a host without it still works. Also on
-  ``SportsCoreSharedMixin``.
-- ``SKIN_MODE`` -- optional, read with ``getattr`` by ``_mode_customization``;
-  without it the mode overrides are simply not merged.
-
-State the mixin keeps on the host, created on first use (all read with
-``getattr``, so nothing needs initialising): ``_last_display_call_monotonic``,
-``_switch_order``, ``_switch_order_key``, ``_switch_position``.
-
-Class-level defaults: ``_DWELL_REENTRY_GAP_SECONDS = 5.0`` and
-``favorite_rotation_boost = 1``, matching the plugins. A host's own value
-shadows them.
-
-MRO
----
-No ``__init__`` and no bare ``super()``, so base order does not change what
-runs. By convention list it after ``SportsCoreSharedMixin``::
-
-    class SportsCore(SportsCoreSharedMixin, SportsHelpersMixin, ABC):
-
-HARDWARE-FREE
--------------
-Nothing here may import ``src.display_manager``, ``src.base_classes``,
-``src.plugin_system`` or anything else that reaches ``rgbmatrix`` at module
-level; ``test/test_common_is_hardware_free.py`` enforces that for all of
-``src/common``. ``logo_needs_refresh`` imports ``src.logo_downloader`` lazily,
-exactly as the plugin copy does.
-"""
-
-from __future__ import annotations
-
-import time
-from typing import Any, ClassVar, Dict, List, Optional, Tuple
-
-__all__ = [
-    "MIN_WINDOW_DAYS",
-    "MAX_WINDOW_DAYS",
-    "clamp_window",
-    "clamp_seconds",
-    "logo_needs_refresh",
-    "spread_weighted_order",
-    "SportsHelpersMixin",
-]
-
-MIN_WINDOW_DAYS = 1
-MAX_WINDOW_DAYS = 60
-
-
-def clamp_window(value: Any, fallback: int) -> int:
-    """Days for one side of the schedule window, or the default if unusable."""
-    try:
-        days = int(value)
-    except (TypeError, ValueError, OverflowError):
-        # OverflowError: json parses a bare Infinity and int(inf) raises,
-        # which crashed manager init from a hand-edited config.
-        return fallback
-    return max(MIN_WINDOW_DAYS, min(MAX_WINDOW_DAYS, days))
-
-
-def clamp_seconds(value: Any, fallback: int, low: int = 5,
-                  high: int = 86400) -> int:
-    """An interval in seconds, or the fallback when the value is unusable."""
-    try:
-        seconds = int(value)
-    except (TypeError, ValueError, OverflowError):
-        # OverflowError: json parses bare Infinity by default and int(inf)
-        # raises -- the same gap clamp_window above already covers.
-        return fallback
-    return max(low, min(high, seconds))
-
-
-def logo_needs_refresh(logo_file) -> bool:
-    """True if this file is a placeholder stale enough to retry the real logo.
-
-    A failed logo download is cached as a placeholder wearing the real logo's
-    filename, so "the file exists" is not proof the logo was ever fetched.
-    Without this check one transient failure leaves a team a grey box forever.
-
-    Returns False on a core that predates placeholder marking, which keeps the
-    previous behaviour rather than breaking the load.
-    """
-    # Imported from the core by its full path, never as a bare name: a
-    # deferred bare-name import can bind another plugin's vendored
-    # logo_downloader once the core isolates top-level plugin modules.
-    try:
-        from src.logo_downloader import (
-            PLACEHOLDER_RETRY_SECONDS,
-            is_placeholder_logo,
-            placeholder_age_seconds,
-        )
-    except ImportError:
-        return False
-
-    try:
-        if not is_placeholder_logo(logo_file):
-            return False
-        age = placeholder_age_seconds(logo_file)
-        return age is None or age >= PLACEHOLDER_RETRY_SECONDS
-    except Exception:
-        return False
-
-
-def spread_weighted_order(weights: List[int]) -> List[int]:
-    """Indices into ``weights``, each repeated by its weight and spread out.
-
-    Each index keeps its own slot and places its extra turns at even
-    fractions of the rotation after it, wrapping round. That keeps the
-    list's schedule order for everything else and spaces a favourite's
-    repeats evenly *around the loop* -- the live rotation's smooth
-    weighted round-robin schedules a boosted game first and last, so a
-    rotation that wraps shows it back to back. Equal weights come back in
-    plain order, so a boost that applies to no card changes nothing.
-
-    Repeats are kept apart only where the ratio leaves room: once one
-    weight exceeds all the others combined, no cyclic order can separate
-    its turns ([3, 1, 1] gives [0, 1, 0, 2, 0]). Each index still gets
-    exactly its weight in turns -- the configured ratio wins over spacing.
-    """
-    count = len(weights)
-    slots = []
-    for index, weight in enumerate(weights):
-        for turn in range(weight):
-            slots.append(((index + turn * count / weight) % count, turn > 0, index))
-    return [index for _, _, index in sorted(slots)]
-
-
-class SportsHelpersMixin:
-    """The ``SportsCore`` methods identical across the scoreboards' sports.py.
-
-    Stateless and constructor-free; see the module docstring for the host
-    contract and base-order guidance.
-    """
-
-    #: Longest gap between two display() calls that still counts as one
-    #: on-screen stint. Frames arrive many times a second while a mode is on
-    #: the panel; between mode blocks the gap is the length of every other
-    #: mode's block -- a minute or more. Anything past a few seconds can only
-    #: be a block boundary, or the very first frame after startup.
-    _DWELL_REENTRY_GAP_SECONDS: ClassVar[float] = 5.0
-
-    #: Turns a favourite's card gets per turn of any other card in switch
-    #: mode. The plugins set it per instance from config; this default keeps
-    #: a host that does not on the plain rotation.
-    favorite_rotation_boost: int = 1
-
-    def _favorite_key(self, game: Dict, side: str) -> Optional[str]:
-        """Override point: which view-model field identifies a team when
-        matching against ``favorite_teams``.
-
-        ``side`` is ``"home"`` or ``"away"``. The default is the team
-        abbreviation -- what eight of the nine scoreboards match on, and what
-        users type into their favorites list.
-
-        NRL needs the team **id** instead, because NRL abbreviations are not
-        unique: "NEW" is both Newcastle Knights and New Zealand Warriors,
-        "CAN" both Canberra Raiders and Canterbury Bulldogs. It is a seam
-        rather than a branch so core never has to learn the string "nrl"::
-
-            def _favorite_key(self, game, side):
-                return str(game.get(f"{side}_id"))
-
-        An override that stringifies should note that a missing id becomes the
-        literal ``"None"``, which would spuriously match a favorites list
-        containing that string. The default returns ``None`` for a missing
-        abbreviation, which never matches.
-
-        Carried from ``src/base_classes/sports/core.py`` for later phases;
-        nothing in this module calls it yet.
-        """
-        return game.get(f"{side}_abbr")
-
-    def _mode_customization(self) -> dict:
-        """``customization`` with this mode's overrides merged over it.
-
-        SportsUpcoming / SportsRecent / SportsLive are separate instances
-        with their own SKIN_MODE, so merging once here makes every
-        per-element lookup mode-aware without changing one of them.
-
-        ``None`` in a mode block means "inherit", which is what lets a user
-        restyle one element on live cards and leave everything else
-        following the settings above. It has to stay distinct from 0: a mode
-        y_offset of 0 means "sit at the base position", not "no preference".
-        """
-        customization = self.config.get('customization', {})
-        if not isinstance(customization, dict):
-            return {}
-        mode = getattr(self, 'SKIN_MODE', None)
-        if not mode:
-            return customization
-        modes = customization.get('modes')
-        block = modes.get(mode) if isinstance(modes, dict) else None
-        if not isinstance(block, dict):
-            return customization
-
-        merged = dict(customization)
-        for element, override in block.items():
-            if element == 'layout' or not isinstance(override, dict):
-                continue
-            base = merged.get(element)
-            base = dict(base) if isinstance(base, dict) else {}
-            base.update({k: v for k, v in override.items() if v is not None})
-            merged[element] = base
-
-        mode_layout = block.get('layout')
-        if isinstance(mode_layout, dict):
-            base_layout = merged.get('layout')
-            new_layout = dict(base_layout) if isinstance(base_layout, dict) else {}
-            for element, axes in mode_layout.items():
-                if not isinstance(axes, dict):
-                    continue
-                current = new_layout.get(element)
-                current = dict(current) if isinstance(current, dict) else {}
-                current.update({k: v for k, v in axes.items() if v is not None})
-                new_layout[element] = current
-            merged['layout'] = new_layout
-        return merged
-
-    def _setting_int(self, key: str, default: int, low: int, high: int) -> int:
-        """A count from config, clamped to the range its schema declares.
-
-        The schema constrains these, but config.json can be hand-edited or
-        written by an older tool, and a string or a negative here does not
-        raise where anyone would see it -- it raises inside update()'s own
-        try/except, which shows up as a mode that silently renders nothing.
-        Same shape as the favorite_live_boost clamp in the plugins.
-        """
-        try:
-            return max(low, min(high, int(self.mode_config.get(key, default))))
-        except (TypeError, ValueError, OverflowError):
-            # OverflowError: json parses a bare Infinity, and int(inf) raises
-            # -- from __init__, outside any try/except, so the manager would
-            # fail to construct instead of falling back.
-            self.logger.warning(
-                "%s: ignoring unusable %s=%r, using %s",
-                getattr(self, "league", "?"), key,
-                self.mode_config.get(key), default,
-            )
-            return default
-
-    def _reset_dwell_on_reentry(self) -> bool:
-        """Give the current card a full turn when this mode (re)takes the panel.
-
-        The dwell clock (last_game_switch) keeps running while the mode is off
-        screen, so on re-entry it was always long expired and the first
-        display() call advanced immediately: the card cut off by the end of
-        the previous block was skipped instead of shown -- measured at one in
-        five card transitions on a 30s block of 15s cards -- and after a
-        service restart the clock started at manager construction, seconds
-        before the first frame, shaving that much off the first card. Both are
-        the same defect: the dwell clock counting time the viewer never saw.
-
-        Returns True when the dwell was reset, so the caller forces a redraw.
-        The one-frame card at the end of a block (the advance that races the
-        controller's mode switch) still renders -- this reset is what turns it
-        into the card that opens the next block with a full turn, instead of
-        one the rotation skipped.
-        """
-        # getattr, and zero treated as "never displayed": the managers are
-        # constructed in several places -- the plugin tests among them -- not
-        # all of which set every attribute, and a freshly booted Pi can reach
-        # the first frame while time.monotonic() itself is still under the
-        # gap threshold, which would make `now - 0.0` look like one stint.
-        last = getattr(self, "_last_display_call_monotonic", 0.0)
-        now = time.monotonic()
-        self._last_display_call_monotonic = now
-        if last > 0.0 and now - last < self._DWELL_REENTRY_GAP_SECONDS:
-            return False
-        if getattr(self, "last_game_switch", 0) <= 0:
-            # Zero is the live screen's "no game shown yet" sentinel with its
-            # own handling; overwriting it here would hide the first game's
-            # arrival from that logic.
-            return False
-        self.last_game_switch = time.time()
-        return True
-
-    _spread_weighted_order = staticmethod(spread_weighted_order)
-
-    def _next_switch_index(self) -> int:
-        """The games_list index switch mode shows next.
-
-        favorite_rotation_boost gives a favourite's card that many turns for
-        every one turn another card gets, spread through the rotation and kept
-        apart wherever the other cards leave room (a boost above the number of
-        other cards makes some repeats adjacent; the ratio is kept either way).
-        games_list itself stays one entry per game -- the
-        cycle-duration count, the scroll strip and the other-games re-cut all
-        read it -- so the weighting is an order walked over it instead.
-
-        The order is rebuilt whenever the list's games change, and the walk
-        resyncs from current_game_index whenever the two disagree: update()
-        and the other-games rotation both set the index directly when they
-        swap a list in, and the card on screen is where the walk resumes.
-
-        Called with _games_lock held and games_list non-empty.
-        """
-        count = len(self.games_list)
-        boost = getattr(self, "favorite_rotation_boost", 1)
-        if boost <= 1 or count < 2:
-            return (self.current_game_index + 1) % count
-        key = (boost, tuple(g.get("id") for g in self.games_list))
-        if getattr(self, "_switch_order_key", None) != key:
-            self._switch_order = self._spread_weighted_order(
-                [boost if self._is_favorite_game(g) else 1 for g in self.games_list]
-            )
-            self._switch_order_key = key
-            self._switch_position = -1
-        order = self._switch_order
-        position = getattr(self, "_switch_position", -1)
-        if not 0 <= position < len(order) or order[position] != self.current_game_index:
-            position = (order.index(self.current_game_index)
-                        if self.current_game_index in order else -1)
-        position = (position + 1) % len(order)
-        self._switch_position = position
-        return order[position]
-
-    def _odds_color(self) -> Tuple[int, int, int]:
-        """Colour for the odds text; the green it always drew unless configured.
-
-        Guarded with getattr because not every class that reaches
-        _draw_dynamic_odds carries the element-colour helper -- the plugins'
-        own test harnesses build minimal manager objects, and a bare
-        AttributeError here is swallowed by the surrounding except, which
-        drops the odds off the card instead of failing loudly.
-        """
-        getter = getattr(self, "_element_color", None)
-        if getter is None:
-            return (0, 255, 0)
-        try:
-            return getter("odds_text", (0, 255, 0))
-        except Exception:
-            return (0, 255, 0)
-
-    def _upcoming_date_and_time_text(self, game_date: str, game_time: str,
-                                     game: Optional[Dict] = None) -> Tuple[str, str]:
-        """The formatted (date, time) pair, blanked by switch_show_date/_time.
-
-        Deliberately not the shared show_date/show_time: those governed only
-        the scroll and Vegas cards before this display read the block, so a
-        config that had turned them off there would silently blank a scorebug
-        that has always drawn both lines. The switch keys default to True for
-        the same reason switch_upcoming_center defaults to "date_time" -- an
-        untouched panel keeps rendering exactly what it rendered before.
-        """
-        date_text = (self._format_game_date(game_date, game)
-                     if self._card_option("switch_show_date", True) else "")
-        time_text = (self._format_game_time(game_time)
-                     if self._card_option("switch_show_time", True) else "")
-        return date_text, time_text

src/display_controller.py

@@ -119,6 +119,15 @@ class DisplayController:
                 # validator.raise_on_errors()  # Uncomment to fail fast on errors
         except Exception as e:
             logger.warning(f"Startup validation could not be completed: {e}")
+
+        # Automatic updates need their health-check units, and this is the
+        # one root process running project code, so it installs them while
+        # automatic updates are on. See src/auto_update_setup.py.
+        try:
+            from src.auto_update_setup import ensure_update_helper
+            ensure_update_helper(self.config)
+        except Exception as e:
+            logger.warning("Automatic update setup could not be completed: %s", e)
         
         config_time = time.time()
         self.display_manager = DisplayManager(self.config)

src/startup_validator.py

@@ -81,6 +81,8 @@ class StartupValidator:
     _UNITS = (
         ("systemd/ledmatrix.service", "/etc/systemd/system/ledmatrix.service"),
         ("systemd/ledmatrix-web.service", "/etc/systemd/system/ledmatrix-web.service"),
+        ("systemd/ledmatrix-update-verify.service", "/etc/systemd/system/ledmatrix-update-verify.service"),
+        ("systemd/ledmatrix-update-verify.path", "/etc/systemd/system/ledmatrix-update-verify.path"),
     )
 
     def _validate_systemd_units(self) -> None:

systemd/README.md

@@ -14,6 +14,16 @@ This directory contains systemd service unit files for LEDMatrix services.
   - Starts automatically on boot if `web_display_autostart` is enabled
   - Uses `scripts/utils/start_web_conditionally.py`
 
+- **`ledmatrix-update-verify.service`** / **`.path`** - Automatic update health check and rollback
+  - The path unit starts the service when the web interface creates
+    `data/auto_update_verify.request` after a weekly automatic update
+  - Installed by the installers, or by the display service when automatic
+    updates are turned on in the web UI (`src/auto_update_setup.py`)
+  - Restarts the services, checks they stay up, and otherwise resets to the
+    previous commit and its dependencies
+  - Runs `data/auto_update_verifier.py`, a copy of
+    `scripts/utils/auto_update_verify.py` taken before the update
+
 - **`ledmatrix-wifi-monitor.service`** - WiFi monitor daemon service
   - Monitors WiFi/Ethernet connectivity
   - Automatically enables/disables access point mode
@@ -43,7 +53,8 @@ This directory contains systemd service unit files for LEDMatrix services.
 
 These service files are installed by the installation scripts in `scripts/install/`:
 - `install_service.sh` installs `ledmatrix.service`
-- `install_web_service.sh` installs `ledmatrix-web.service`
+- `install_web_service.sh` installs `ledmatrix-web.service` and the
+  `ledmatrix-update-verify` service and path units
 - `install_wifi_monitor.sh` installs `ledmatrix-wifi-monitor.service`
 - `install_dns_fix.sh` installs `ledmatrix-dns-fix.service` (opt-in, not run
   by the normal installer)

systemd/ledmatrix-update-verify.path

@@ -0,0 +1,20 @@
+# This is a template. Do not install it directly -- it is installed with
+# ledmatrix-update-verify.service by scripts/install/install_web_service.sh,
+# or by the display service when automatic updates are turned on
+# (src/auto_update_setup.py). Substitutes:
+#   __PROJECT_ROOT_DIR__  -> the directory the project is installed in
+#
+# Starts the automatic update health check when the web interface asks for it
+# by creating data/auto_update_verify.request. A path unit rather than a
+# sudoers rule: the web interface needs no extra privilege to trigger it, and
+# the service it starts runs as the web interface's own user.
+
+[Unit]
+Description=Watch for LED Matrix automatic update health check requests
+
+[Path]
+PathExists=__PROJECT_ROOT_DIR__/data/auto_update_verify.request
+Unit=ledmatrix-update-verify.service
+
+[Install]
+WantedBy=multi-user.target

systemd/ledmatrix-update-verify.service

@@ -0,0 +1,31 @@
+# This is a template. Do not install it directly -- run
+# scripts/install/install_web_service.sh (or install_service.sh); the display
+# service also installs it when automatic updates are turned on
+# (src/auto_update_setup.py). Substitutes:
+#   __PROJECT_ROOT_DIR__  -> the directory the project is installed in
+#   __USER__              -> the user the web interface runs as
+#
+# Health check and rollback for the web interface's weekly automatic update
+# (web_interface/auto_update.py). A separate unit because the check restarts
+# ledmatrix-web.service, which would kill it if it ran there.
+#
+# Never enabled itself: ledmatrix-update-verify.path starts it when the web
+# interface creates data/auto_update_verify.request. It runs as the web
+# interface's own user, and ExecStart points at the copy the updater makes
+# before pulling, so the update being checked cannot replace its checker.
+# Command-line paths are quoted so an install folder with spaces still works.
+
+[Unit]
+Description=LED Matrix automatic update health check and rollback
+
+[Service]
+Type=oneshot
+User=__USER__
+WorkingDirectory=__PROJECT_ROOT_DIR__
+# Consume the request first, so the path unit cannot start this again.
+ExecStartPre=/bin/rm -f "__PROJECT_ROOT_DIR__/data/auto_update_verify.request"
+ExecStart=/usr/bin/python3 "__PROJECT_ROOT_DIR__/data/auto_update_verifier.py" "__PROJECT_ROOT_DIR__"
+TimeoutStartSec=30min
+StandardOutput=syslog
+StandardError=syslog
+SyslogIdentifier=ledmatrix-update-verify

test/fixtures/api_v3_url_map.json

@@ -841,6 +841,23 @@
       "POST"
     ]
   ],
+  [
+    "/api/v3/system/auto-update",
+    "api_v3.get_auto_update_status",
+    [
+      "GET",
+      "HEAD",
+      "OPTIONS"
+    ]
+  ],
+  [
+    "/api/v3/system/auto-update/dismiss",
+    "api_v3.dismiss_auto_update_alert",
+    [
+      "OPTIONS",
+      "POST"
+    ]
+  ],
   [
     "/api/v3/system/check-update",
     "api_v3.check_for_update",

test/test_auto_update.py

@@ -0,0 +1,683 @@
+"""Weekly automatic updates (web_interface/auto_update.py).
+
+The updater pulls code and restarts services unattended. These pin down that
+it runs only when asked and due, refuses to touch a checkout it could damage,
+never leaves new code running without a health check, and reports every
+failure instead of logging it and moving on.
+
+Git runs for real against a throwaway origin/device clone pair; systemd, sudo
+and the health check service are faked.
+"""
+import json
+import shutil
+import subprocess
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT))
+
+from web_interface import auto_update as au  # noqa: E402
+
+# Local time is pinned to UTC below, so 03:00 is inside the quiet hours.
+NIGHT = datetime(2026, 9, 14, 3, 0, tzinfo=timezone.utc).timestamp()
+NOON = datetime(2026, 9, 14, 12, 0, tzinfo=timezone.utc).timestamp()
+ON = {'auto_update': {'enabled': True}}
+OFF = {'auto_update': {'enabled': False}}
+
+needs_git = pytest.mark.skipif(shutil.which('git') is None, reason='git not installed')
+
+
+@pytest.fixture(autouse=True)
+def local_is_utc(monkeypatch):
+    """Pin 'local time' so quiet-hours checks don't depend on the host zone."""
+    monkeypatch.setattr(au, '_zone', lambda name: timezone.utc)
+
+
+def git(cwd, *args):
+    result = subprocess.run(['git', '-c', 'user.name=t', '-c', 'user.email=t@example.com', *args],
+                            cwd=str(cwd), capture_output=True, text=True)
+    assert result.returncode == 0, result.stderr
+    return result.stdout.strip()
+
+
+class Repo:
+    """An origin, a checkout that publishes to it, and the device's clone."""
+
+    def __init__(self, tmp):
+        self.seed = tmp / 'seed'
+        self.seed.mkdir()
+        git(self.seed, 'init', '-q', '-b', 'main')
+        (self.seed / 'app.py').write_text('v = 1\n')
+        git(self.seed, 'add', '.')
+        git(self.seed, 'commit', '-qm', 'one')
+        self.origin = tmp / 'origin.git'
+        git(tmp, 'clone', '-q', '--bare', str(self.seed), str(self.origin))
+        git(self.seed, 'remote', 'add', 'origin', str(self.origin))
+        self.device = tmp / 'device'
+        git(tmp, 'clone', '-q', str(self.origin), str(self.device))
+
+    def head(self):
+        return git(self.device, 'rev-parse', 'HEAD')
+
+    def publish(self, text='v = 2\n'):
+        (self.seed / 'app.py').write_text(text)
+        git(self.seed, 'commit', '-qam', 'next')
+        git(self.seed, 'push', '-q', 'origin', 'main')
+        return git(self.seed, 'rev-parse', 'HEAD')
+
+
+def real_pull(device, **extra):
+    def core_update():
+        before = git(device, 'rev-parse', 'HEAD')
+        r = subprocess.run(['git', 'pull', '-q', '--rebase', '--autostash'],
+                           cwd=str(device), capture_output=True, text=True)
+        after = git(device, 'rev-parse', 'HEAD')
+        result = {'status': 'success' if r.returncode == 0 else 'error',
+                  'message': 'Code updated successfully.' if r.returncode == 0 else r.stderr,
+                  'restart_required': before != after, 'dependency_failures': []}
+        result.update(extra)
+        return result
+    return core_update
+
+
+class FakeStore:
+    def __init__(self, plugins_dir, versions, bump=(), fail=()):
+        self.plugins_dir = str(plugins_dir)
+        self.bump, self.fail = set(bump), set(fail)
+        for pid, version in versions.items():
+            d = Path(plugins_dir) / pid
+            d.mkdir(parents=True)
+            (d / 'manifest.json').write_text(json.dumps({'id': pid, 'version': version}))
+        self.updated_calls = []
+
+    def list_installed_plugins(self):
+        return [p.name for p in Path(self.plugins_dir).iterdir()]
+
+    def _get_local_git_info(self, path):
+        return None
+
+    def update_plugin(self, pid):
+        self.updated_calls.append(pid)
+        if pid in self.fail:
+            return False
+        if pid in self.bump:
+            path = Path(self.plugins_dir) / pid / 'manifest.json'
+            m = json.loads(path.read_text())
+            m['version'] = '9.9.9'
+            path.write_text(json.dumps(m))
+        return True
+
+
+class Harness:
+    """An AutoUpdater on a real clone.
+
+    ``helper`` is whether the health check's path unit is active; ``pickup``
+    whether the health check actually starts when asked. Starting it is
+    simulated the way systemd does it: the request file is consumed
+    (ExecStartPre) and the verifier marks the pending update "verifying".
+    """
+
+    def __init__(self, tmp, repo, *, helper=True, pickup=True, core_update=None, store=None,
+                 disk_free=10 ** 12, display_active=True, enabled=True, clock=NIGHT):
+        self.sudo, self.restarts, self.now = [], [], clock
+        self.state_at_handoff = None
+
+        def run(args, **kwargs):
+            if args[0] == 'sudo':
+                self.sudo.append(list(args))
+                return subprocess.CompletedProcess(args, 0, stdout='', stderr='')
+            return subprocess.run(args, **kwargs)
+
+        def sleep(seconds):
+            updater = self.updater
+            if self.state_at_handoff is None:
+                self.state_at_handoff = au.load_state(updater.state_file)
+            if pickup and updater.request_file.exists():
+                updater.request_file.unlink()
+                pending = au._read_json(updater.pending_file)
+                pending['status'] = 'verifying'
+                au._write_json(updater.pending_file, pending)
+
+        self.config = {'auto_update': {'enabled': enabled}, 'timezone': 'UTC'}
+        cm = MagicMock()
+        cm.load_config.return_value = self.config
+        self.updater = au.AutoUpdater(
+            config_manager=cm, core_update=core_update or real_pull(repo.device),
+            store_manager=store, project_root=repo.device, clock=lambda: self.now,
+            restart=self.restarts.append, run=run,
+            service_active=lambda unit: display_active, helper_ready=lambda: helper,
+            disk_free=lambda path: disk_free, sleep=sleep)
+
+    @property
+    def state(self):
+        return au.load_state(self.updater.state_file)
+
+    @property
+    def pending(self):
+        return au._read_json(self.updater.pending_file)
+
+    def write_pending(self, **fields):
+        au._write_json(self.updater.pending_file, fields)
+
+
+def stub_updater(tmp_path, outcome='up_to_date', message='ok', store=None, enabled=True, clock=NIGHT):
+    """For scheduling and plugin tests: the core update is stubbed out."""
+    cm = MagicMock()
+    cm.load_config.return_value = {'auto_update': {'enabled': enabled}, 'timezone': 'UTC'}
+    restarts = []
+    updater = au.AutoUpdater(config_manager=cm, core_update=MagicMock(), store_manager=store,
+                             project_root=tmp_path, clock=lambda: clock,
+                             restart=restarts.append)
+    updater.update_core = MagicMock(return_value={'outcome': outcome, 'message': message})
+    return updater, restarts
+
+
+# -- scheduling ---------------------------------------------------------------
+
+class TestIsDue:
+    def test_not_before_next_due(self):
+        assert not au.is_due(100, 200, 3)
+
+    def test_in_quiet_hours_once_due(self):
+        assert au.is_due(200, 200, 3)
+
+    def test_waits_for_quiet_hours(self):
+        assert not au.is_due(200, 100, 12)
+
+    def test_gives_up_waiting_after_grace(self):
+        assert au.is_due(100 + au.QUIET_HOURS_GRACE_SECONDS, 100, 12)
+
+
+class TestTick:
+    def test_disabled_never_runs(self, tmp_path):
+        updater, _ = stub_updater(tmp_path, enabled=False)
+        assert updater.tick() is False
+        updater.update_core.assert_not_called()
+
+    def test_enabling_at_noon_waits_for_the_night(self, tmp_path):
+        updater, _ = stub_updater(tmp_path, clock=NOON)
+        assert updater.tick() is False
+        updater.update_core.assert_not_called()
+        assert au.load_state(updater.state_file)['next_due'] == NOON
+
+    def test_runs_when_due_and_schedules_a_week_out(self, tmp_path):
+        updater, _ = stub_updater(tmp_path)
+        assert updater.tick() is True
+        state = au.load_state(updater.state_file)
+        assert state['next_due'] == NIGHT + au.UPDATE_INTERVAL_SECONDS
+        assert updater.tick() is False, "ran twice in one night"
+
+    def test_transient_error_retries_next_day(self, tmp_path):
+        updater, _ = stub_updater(tmp_path, outcome='error', message='no network')
+        updater.tick()
+        state = au.load_state(updater.state_file)
+        assert state['next_due'] == NIGHT + au.RETRY_AFTER_FAILURE_SECONDS
+        assert state['alert']['message'] == 'no network'
+
+    def test_blocked_keeps_weekly_cadence_but_alerts(self, tmp_path):
+        updater, _ = stub_updater(tmp_path, outcome='blocked', message='local edits')
+        updater.tick()
+        state = au.load_state(updater.state_file)
+        assert state['next_due'] == NIGHT + au.UPDATE_INTERVAL_SECONDS
+        assert 'local edits' in state['alert']['message']
+
+
+# -- checks before touching the checkout ---------------------------------------
+
+@needs_git
+class TestPreflightRefuses:
+    def test_nothing_new_is_up_to_date(self, tmp_path):
+        repo = Repo(tmp_path)
+        h = Harness(tmp_path, repo)
+        result = h.updater.run()
+        assert result['core_outcome'] == 'up_to_date'
+        assert result['status'] == 'success'
+        assert h.sudo == [] and 'alert' not in h.state
+
+    def test_without_the_health_check_code_is_not_touched(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        repo.publish()
+        h = Harness(tmp_path, repo, helper=False)
+        result = h.updater.run()
+        assert result['core_outcome'] == 'blocked'
+        assert repo.head() == old
+        assert 'health check is set up' in result['core_message']
+        assert h.state['alert']
+
+    def test_local_edits_are_never_stashed(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        repo.publish()
+        (repo.device / 'app.py').write_text('mine\n')
+        h = Harness(tmp_path, repo)
+        result = h.updater.run()
+        assert result['core_outcome'] == 'blocked'
+        assert 'app.py' in result['core_message']
+        assert repo.head() == old
+        assert (repo.device / 'app.py').read_text() == 'mine\n'
+        assert git(repo.device, 'stash', 'list') == ''
+
+    def test_local_commits_block(self, tmp_path):
+        repo = Repo(tmp_path)
+        repo.publish()
+        (repo.device / 'extra.txt').write_text('x\n')
+        git(repo.device, 'add', 'extra.txt')
+        git(repo.device, 'commit', '-qm', 'local')
+        mine = repo.head()
+        h = Harness(tmp_path, repo)
+        result = h.updater.run()
+        assert result['core_outcome'] == 'blocked'
+        assert 'local commit' in result['core_message']
+        assert repo.head() == mine
+
+    def test_a_rebase_really_in_progress_blocks(self, tmp_path):
+        repo = Repo(tmp_path)
+        repo.publish('v = 2\n')
+        (repo.device / 'app.py').write_text('v = mine\n')
+        git(repo.device, 'commit', '-qam', 'local')
+        git(repo.device, 'fetch', '-q')
+        stopped = subprocess.run(['git', '-c', 'user.name=t', '-c', 'user.email=t@example.com',
+                                  'rebase', 'origin/main'], cwd=str(repo.device), capture_output=True, text=True)
+        assert stopped.returncode != 0, "the rebase should stop on the conflict"
+        result = Harness(tmp_path, repo).updater.run()
+        assert result['core_outcome'] == 'blocked'
+        assert 'rebase is in progress' in result['core_message']
+        assert (repo.device / '.git' / 'rebase-merge').exists(), "a live rebase must be left alone"
+
+    def test_an_abandoned_rebase_is_cleared_and_the_update_runs(self, tmp_path):
+        """Found on a real device: a pull stopped mid-rebase a week earlier,
+        HEAD was later checked out to a branch, and the leftover rebase
+        directory would have blocked every automatic update forever."""
+        repo = Repo(tmp_path)
+        leftover = repo.device / '.git' / 'rebase-merge'
+        leftover.mkdir()
+        (leftover / 'head-name').write_text('refs/heads/main\n')
+        new = repo.publish()
+        result = Harness(tmp_path, repo).updater.run()
+        assert not leftover.exists()
+        assert result['core_outcome'] == 'verifying'
+        assert repo.head() == new
+
+    def test_a_merge_in_progress_blocks(self, tmp_path):
+        repo = Repo(tmp_path)
+        repo.publish()
+        (repo.device / '.git' / 'MERGE_HEAD').write_text(repo.head() + '\n')
+        result = Harness(tmp_path, repo).updater.run()
+        assert result['core_outcome'] == 'blocked'
+        assert 'merge is in progress' in result['core_message']
+
+    def test_low_disk_blocks(self, tmp_path):
+        repo = Repo(tmp_path)
+        repo.publish()
+        result = Harness(tmp_path, repo, disk_free=10 * 1024 * 1024).updater.run()
+        assert result['core_outcome'] == 'blocked'
+        assert 'disk space' in result['core_message']
+
+    def test_unreachable_origin_retries_tomorrow(self, tmp_path):
+        repo = Repo(tmp_path)
+        git(repo.device, 'remote', 'set-url', 'origin', str(tmp_path / 'nowhere'))
+        h = Harness(tmp_path, repo)
+        result = h.updater.run()
+        assert result['core_outcome'] == 'error'
+        assert h.state['next_due'] == NIGHT + au.RETRY_AFTER_FAILURE_SECONDS
+
+    def test_version_already_rolled_back_is_not_retried(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        bad = repo.publish()
+        h = Harness(tmp_path, repo)
+        au.save_state({'rolled_back_head': bad}, h.updater.state_file)
+        result = h.updater.run()
+        assert result['core_outcome'] == 'up_to_date'
+        assert 'rolled back before' in result['core_message']
+        assert repo.head() == old and h.sudo == []
+
+
+# -- the update and its hand-off to the health check ---------------------------
+
+@needs_git
+class TestUpdateIsVerified:
+    def test_pull_hands_off_to_the_health_check(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        new = repo.publish()
+        store = FakeStore(tmp_path / 'plugins', {'weather': '1.0.0'}, bump={'weather'})
+        h = Harness(tmp_path, repo, store=store)
+        result = h.updater.run()
+
+        assert result['core_outcome'] == 'verifying' and result['status'] == 'pending'
+        assert repo.head() == new
+        assert h.pending == {**h.pending, 'status': 'verifying', 'old_head': old, 'new_head': new,
+                             'display_was_active': True, 'dependency_failures': []}
+        assert not h.updater.request_file.exists(), "the request is consumed when the check starts"
+        assert h.sudo == [], "triggering the health check needs no privilege"
+        assert h.restarts == [], "restarts belong to the health check, not the web process"
+        assert store.updated_calls == [], "plugins must wait until the new code is verified"
+        assert h.state['plugins_pending'] is True
+        assert h.updater.verifier_copy.read_text() == au.VERIFIER_SOURCE.read_text()
+
+    def test_state_is_saved_before_the_health_check_restarts_this_process(self, tmp_path):
+        repo = Repo(tmp_path)
+        repo.publish()
+        h = Harness(tmp_path, repo)
+        h.updater.run()
+        assert h.state_at_handoff['last_result']['core_outcome'] == 'verifying'
+        assert h.state_at_handoff['plugins_pending'] is True
+
+    def test_dependency_failures_reach_the_health_check(self, tmp_path):
+        repo = Repo(tmp_path)
+        repo.publish()
+        h = Harness(tmp_path, repo,
+                    core_update=real_pull(repo.device, dependency_failures=['requirements.txt']))
+        h.updater.run()
+        assert h.pending['dependency_failures'] == ['requirements.txt']
+
+    def test_a_health_check_that_never_starts_means_the_update_is_undone(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        repo.publish()
+        h = Harness(tmp_path, repo, pickup=False)
+        result = h.updater.run()
+        assert repo.head() == old
+        assert result['core_outcome'] == 'blocked'
+        assert 'did not start' in result['core_message']
+        assert h.pending is None and not h.updater.request_file.exists()
+        assert h.state['alert']
+
+    def test_failed_pull_retries_tomorrow(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        repo.publish()
+        h = Harness(tmp_path, repo,
+                    core_update=lambda: {'status': 'error', 'message': 'Update failed: network'})
+        result = h.updater.run()
+        assert result['core_outcome'] == 'error'
+        assert repo.head() == old
+        assert h.state['next_due'] == NIGHT + au.RETRY_AFTER_FAILURE_SECONDS
+
+    def test_failed_pull_that_moved_head_is_rolled_back(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        repo.publish()
+        pull = real_pull(repo.device)
+
+        def half_failed():
+            pull()
+            return {'status': 'error', 'message': 'Update failed: interrupted'}
+        result = Harness(tmp_path, repo, core_update=half_failed).updater.run()
+        assert repo.head() == old
+        assert 'rolled back' in result['core_message']
+
+
+# -- reporting the health check's outcome --------------------------------------
+
+@needs_git
+class TestHealthCheckOutcome:
+    def _after_update(self, tmp_path, **pending):
+        repo = Repo(tmp_path)
+        store = FakeStore(tmp_path / 'plugins', {'weather': '1.0.0'}, bump={'weather'})
+        h = Harness(tmp_path, repo, store=store)
+        au.save_state({'plugins_pending': True, 'next_due': NIGHT + au.UPDATE_INTERVAL_SECONDS,
+                       'last_result': {'core_outcome': 'verifying', 'status': 'pending',
+                                       'plugins_deferred': True}}, h.updater.state_file)
+        h.write_pending(old_head='a' * 40, new_head='b' * 40, created_at=NIGHT, **pending)
+        return h, store
+
+    def test_waits_while_the_check_runs(self, tmp_path):
+        h, store = self._after_update(tmp_path, status='verifying')
+        h.now = NIGHT + 60
+        assert h.updater.tick() is False
+        assert store.updated_calls == [] and h.pending
+
+    def test_success_then_runs_the_deferred_plugins(self, tmp_path):
+        h, store = self._after_update(tmp_path, status='success')
+        assert h.updater.tick() is True
+        result = h.state['last_result']
+        assert result['core_outcome'] == 'updated' and result['status'] == 'success'
+        assert result['plugins_updated'] == ['weather']
+        assert h.restarts == ['ledmatrix']
+        assert 'alert' not in h.state and h.pending is None
+
+    def test_rollback_alerts_and_remembers_the_bad_commit(self, tmp_path):
+        h, store = self._after_update(tmp_path, status='rolled_back',
+                                      reason='the web interface did not respond')
+        h.updater.tick()
+        assert 'rolled back' in h.state['alert']['message']
+        assert 'the web interface did not respond' in h.state['alert']['message']
+        assert h.state['rolled_back_head'] == 'b' * 40
+        assert store.updated_calls == ['weather'], "the old code is healthy; plugins may update"
+
+    def test_failed_rollback_alerts_and_skips_plugins(self, tmp_path):
+        h, store = self._after_update(tmp_path, status='rollback_failed',
+                                      reason='display down', detail='reset failed')
+        h.updater.tick()
+        assert f'git reset --hard {"a" * 40}' in h.state['alert']['message']
+        assert store.updated_calls == []
+        assert h.state['plugins_pending'] is False
+
+    def test_a_check_that_never_reports_back_alerts(self, tmp_path):
+        h, _ = self._after_update(tmp_path, status='verifying')
+        h.now = NIGHT + au.VERIFY_LOST_SECONDS + 1
+        h.updater.tick()
+        assert h.state['last_result']['core_outcome'] == 'lost'
+        assert 'never reported back' in h.state['alert']['message']
+
+    def test_dismissed_alert_stays_hidden_until_a_new_one(self, tmp_path):
+        h, _ = self._after_update(tmp_path, status='rolled_back', reason='x')
+        h.updater.tick()
+        alert_id = h.state['alert']['id']
+
+        def status():
+            return au.describe_status(h.config, state_file=h.updater.state_file,
+                                      pending_file=h.updater.pending_file,
+                                      setup_file=tmp_path / 'no-setup.json', helper=lambda: True)
+        assert status()['alert_id'] == alert_id
+        au.dismiss_alert(alert_id, h.updater.state_file)
+        assert status()['alert'] is None
+        state = h.state
+        state['alert'] = {'id': 'newer', 'message': 'again'}
+        au.save_state(state, h.updater.state_file)
+        assert status()['alert'] == 'again'
+
+
+def test_describe_status_reports_setup(tmp_path):
+    setup = tmp_path / 'setup.json'
+    setup.write_text(json.dumps({'status': 'failed', 'message': 'not root'}))
+    kwargs = dict(state={}, state_file=tmp_path / 's.json', pending_file=tmp_path / 'p.json',
+                  setup_file=setup)
+    on = au.describe_status(ON, helper=lambda: False, **kwargs)
+    assert on['verifier_installed'] is False
+    assert (on['setup_status'], on['setup_message']) == ('failed', 'not root')
+
+    def never(): raise AssertionError("systemctl asked while updates are off")
+    assert au.describe_status(OFF, helper=never, **kwargs)['verifier_installed'] is None
+
+
+# -- setting it up from the web UI ----------------------------------------------
+
+class TestSetupFromTheWebUI:
+    def _host(self, monkeypatch, ready=False, active=True, restart_ok=True):
+        restarts = []
+        monkeypatch.setattr(au, 'helper_ready', lambda: ready)
+        monkeypatch.setattr(au, '_service_active', lambda unit: active)
+        monkeypatch.setattr(au, 'restart_service', lambda unit: restarts.append(unit) or restart_ok)
+        return restarts
+
+    def test_switching_on_restarts_the_display_to_finish_setup(self, monkeypatch):
+        restarts = self._host(monkeypatch)
+        note = au.start_setup_if_needed(False, ON)
+        assert restarts == ['ledmatrix']
+        assert 'display is restarting' in note
+
+    @pytest.mark.parametrize('was_enabled, config, ready', [
+        (True, ON, False),   # already on: an unrelated save must not restart the display
+        (False, OFF, False),  # still off
+        (False, ON, True),   # already set up
+    ])
+    def test_nothing_to_do(self, monkeypatch, was_enabled, config, ready):
+        restarts = self._host(monkeypatch, ready=ready)
+        assert au.start_setup_if_needed(was_enabled, config) is None
+        assert restarts == []
+
+    def test_a_stopped_display_is_not_started(self, monkeypatch):
+        restarts = self._host(monkeypatch, active=False)
+        assert 'next time the display service starts' in au.start_setup_if_needed(False, ON)
+        assert restarts == []
+
+    def test_a_failed_restart_says_so(self, monkeypatch):
+        self._host(monkeypatch, restart_ok=False)
+        assert 'Could not restart' in au.start_setup_if_needed(False, ON)
+
+
+# -- plugins --------------------------------------------------------------------
+
+class TestPlugins:
+    def test_nothing_changed_restarts_nothing(self, tmp_path):
+        store = FakeStore(tmp_path / 'plugins', {'clock': '1.0.0'})
+        updater, restarts = stub_updater(tmp_path, store=store)
+        updater.run()
+        assert store.updated_calls == ['clock']
+        assert restarts == []
+
+    def test_plugin_update_restarts_display_only(self, tmp_path):
+        store = FakeStore(tmp_path / 'plugins', {'clock': '1.0.0', 'weather': '1.0.0'},
+                          bump={'weather'})
+        updater, restarts = stub_updater(tmp_path, store=store)
+        result = updater.run()
+        assert result['plugins_updated'] == ['weather']
+        assert restarts == ['ledmatrix']
+
+    def test_plugin_failure_alerts_but_keeps_weekly_cadence(self, tmp_path):
+        store = FakeStore(tmp_path / 'plugins', {'zip-only': '1.0.0'}, fail={'zip-only'})
+        updater, _ = stub_updater(tmp_path, store=store)
+        result = updater.run()
+        assert result['plugins_failed'] == ['zip-only'] and result['status'] == 'error'
+        state = au.load_state(updater.state_file)
+        assert 'zip-only' in state['alert']['message']
+        assert state['next_due'] == NIGHT + au.UPDATE_INTERVAL_SECONDS
+
+    def test_local_only_plugins_are_left_alone(self, tmp_path):
+        store = FakeStore(tmp_path / 'plugins', {'mine': '1.0.0'})
+        path = tmp_path / 'plugins' / 'mine' / 'manifest.json'
+        path.write_text(json.dumps({'id': 'mine', 'version': '1.0.0', 'local_only': True}))
+        updater, _ = stub_updater(tmp_path, store=store)
+        updater.run()
+        assert store.updated_calls == []
+
+
+def test_restart_service_skips_a_stopped_unit(monkeypatch):
+    calls = []
+
+    def run(args, **kwargs):
+        calls.append(args)
+        return MagicMock(stdout='inactive\n', returncode=3, stderr='')
+    monkeypatch.setattr(au.subprocess, 'run', run)
+    assert au.restart_service('ledmatrix') is False
+    assert calls == [['systemctl', 'is-active', 'ledmatrix']], (
+        "a display the user stopped must not be started by an update")
+
+
+def test_core_update_names_the_requirement_files_that_failed():
+    """The health check refuses code whose dependencies did not install, so
+    the failure has to arrive as data, not only as a sentence in the message."""
+    from web_interface.blueprints import api_v3 as pkg
+    from web_interface.blueprints.api_v3 import system
+
+    heads = iter(['aaa111\n', 'bbb222\n'])
+
+    def run(args, **kwargs):
+        if args[:2] == ['git', 'rev-parse'] and args[-1] == 'HEAD':
+            out = next(heads, 'bbb222\n')
+        elif args[:2] == ['git', 'diff']:
+            out = 'requirements.txt\n'
+        elif '@{u}' in args:
+            out = 'origin/main\n'
+        else:
+            out = ''
+        return subprocess.CompletedProcess(args, 0, stdout=out, stderr='')
+
+    pkg.api_v3.plugin_store_manager = None
+    failed_install = subprocess.CompletedProcess([], 1, stdout='', stderr='boom')
+    with patch.object(pkg.subprocess, 'run', run), \
+            patch.object(system, '_pip_install_requirements', return_value=failed_install):
+        result = system.perform_core_update()
+    assert result['status'] == 'success'
+    assert result['dependency_failures'] == ['requirements.txt']
+
+
+# -- web routes -------------------------------------------------------------------
+
+@pytest.fixture
+def api_client(monkeypatch):
+    from flask import Flask
+    from web_interface.blueprints.api_v3 import api_v3
+    app = Flask(__name__)
+    app.register_blueprint(api_v3, url_prefix='/api/v3')
+    cm = MagicMock()
+    cm.load_config.return_value = {'timezone': 'UTC', 'auto_update': {'enabled': True}}
+    cm.get_raw_file_content.return_value = {}
+    cm.save_config_atomic.return_value = MagicMock(status=MagicMock(value='success'), message=None)
+    api_v3.config_manager = cm
+    api_v3.plugin_manager = MagicMock(plugins={})
+    # Never restart a real display from a test run.
+    setup_calls = []
+    monkeypatch.setattr(au, 'start_setup_if_needed',
+                        lambda was_enabled, config: setup_calls.append(was_enabled) or None)
+    return app.test_client(), cm, setup_calls
+
+
+class TestStatusRoutes:
+    def test_status_carries_the_alert(self, api_client, monkeypatch):
+        client, _, _ = api_client
+        monkeypatch.setattr(au, 'describe_status', lambda config: {'alert': 'rolled back', 'alert_id': '7'})
+        data = client.get('/api/v3/system/auto-update').get_json()['data']
+        assert data == {'alert': 'rolled back', 'alert_id': '7'}
+
+    @pytest.mark.parametrize('body', [{}, [1], 'x', 5])
+    def test_dismiss_needs_an_id_in_a_json_object(self, api_client, body):
+        client, _, _ = api_client
+        assert client.post('/api/v3/system/auto-update/dismiss', json=body).status_code == 400
+
+    def test_dismiss(self, api_client, monkeypatch):
+        client, _, _ = api_client
+        calls = []
+        monkeypatch.setattr(au, 'dismiss_alert', calls.append)
+        assert client.post('/api/v3/system/auto-update/dismiss', json={'alert_id': '7'}).status_code == 200
+        assert calls == ['7']
+
+
+class TestSettingsSave:
+    def _saved(self, cm):
+        return cm.save_config_atomic.call_args[0][0]
+
+    def test_checked_toggle_saves_enabled(self, api_client):
+        client, cm, setup_calls = api_client
+        cm.load_config.return_value = {'timezone': 'UTC'}
+        resp = client.post('/api/v3/config/main', json={'timezone': 'UTC', 'auto_update_enabled': 'true'})
+        assert resp.status_code == 200
+        saved = self._saved(cm)
+        assert saved['auto_update'] == {'enabled': True}
+        assert 'auto_update_enabled' not in saved, "the form field must not leak into config.json"
+        assert setup_calls == [False], "setup is told the toggle was previously off"
+
+    def test_unchecked_toggle_on_general_save_disables(self, api_client):
+        client, cm, setup_calls = api_client
+        resp = client.post('/api/v3/config/main', json={'timezone': 'UTC'})
+        assert resp.status_code == 200
+        assert self._saved(cm)['auto_update'] == {'enabled': False}
+        assert setup_calls == [True]
+
+    def test_setup_note_reaches_the_user(self, api_client, monkeypatch):
+        client, cm, _ = api_client
+        cm.load_config.return_value = {'timezone': 'UTC'}
+        monkeypatch.setattr(au, 'start_setup_if_needed',
+                            lambda was_enabled, config: 'Finishing automatic update setup: the display is restarting.')
+        body = client.post('/api/v3/config/main',
+                           json={'timezone': 'UTC', 'auto_update_enabled': 'true'}).get_json()
+        assert 'the display is restarting' in body['message']

test/test_auto_update_setup.py

@@ -0,0 +1,214 @@
+"""Installing the automatic-update health check from the display service.
+
+src/auto_update_setup.py is how a user who never opens a terminal gets updates
+with a safety net: it runs as root inside the display service and writes
+systemd units. So it has to install exactly the right thing when asked, do
+nothing when not asked, refuse templates that would run as anyone but the web
+user, and say why whenever it could not finish.
+"""
+import json
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT))
+
+from src import auto_update_setup as aus  # noqa: E402
+
+ON = {'auto_update': {'enabled': True}}
+
+
+class FakeSystemctl:
+    def __init__(self, fail=(), activates=True):
+        self.calls = []
+        self.fail = set(fail)
+        self.activates = activates
+        self.active = False
+
+    def __call__(self, args, **kwargs):
+        assert args[0] == 'systemctl', args
+        self.calls.append(args[1:])
+        verb = args[1]
+        if verb == 'is-active':
+            return subprocess.CompletedProcess(args, 0 if self.active else 3,
+                                               stdout='active\n' if self.active else 'inactive\n', stderr='')
+        if verb in self.fail:
+            return subprocess.CompletedProcess(args, 1, stdout='', stderr=f'{verb} refused')
+        if verb in ('restart', 'enable') and self.activates and (verb == 'restart' or '--now' in args):
+            self.active = True
+        return subprocess.CompletedProcess(args, 0, stdout='', stderr='')
+
+
+def project(tmp_path, user='hdpi', workdir=None, name='LEDMatrix'):
+    root = tmp_path / name
+    (root / 'systemd').mkdir(parents=True)
+    for name in aus.UNITS:
+        shutil.copy(ROOT / 'systemd' / name, root / 'systemd' / name)
+    etc = tmp_path / 'etc'
+    etc.mkdir()
+    (etc / aus.WEB_UNIT).write_text(
+        f'[Service]\nUser={user}\nWorkingDirectory={workdir or root}\n', encoding='utf-8')
+    return root, etc
+
+
+def setup(root, etc, systemctl, root_user=True):
+    return aus.UpdateHelperSetup(project_root=root, systemd_dir=etc, run=systemctl,
+                                 is_root=lambda: root_user,
+                                 lookup_ids=lambda user: None if user == 'ghost' else (1000, 1000),
+                                 clock=lambda: 1234.0)
+
+
+def result(root):
+    return json.loads((root / aus.RESULT_REL).read_text())
+
+
+def test_does_nothing_while_updates_are_off(tmp_path):
+    root, etc = project(tmp_path)
+    systemctl = FakeSystemctl()
+    assert setup(root, etc, systemctl).ensure({'auto_update': {'enabled': False}}) is None
+    assert systemctl.calls == []
+    assert not (etc / aus.SERVICE_UNIT).exists()
+    assert not (root / aus.RESULT_REL).exists()
+
+
+def test_installs_both_units_for_the_web_user(tmp_path):
+    root, etc = project(tmp_path)
+    systemctl = FakeSystemctl()
+    out = setup(root, etc, systemctl).ensure(ON)
+
+    assert out['status'] == 'installed' and result(root)['status'] == 'installed'
+    service = (etc / aus.SERVICE_UNIT).read_text()
+    assert 'User=hdpi' in service
+    assert f'ExecStart=/usr/bin/python3 "{root}/data/auto_update_verifier.py" "{root}"' in service
+    assert f'PathExists={root}/data/auto_update_verify.request' in (etc / aus.PATH_UNIT).read_text()
+    assert '__' not in service
+    assert ['daemon-reload'] in systemctl.calls
+    assert ['enable', aus.PATH_UNIT] in systemctl.calls
+    assert systemctl.active
+
+
+def test_second_start_changes_nothing(tmp_path):
+    root, etc = project(tmp_path)
+    systemctl = FakeSystemctl()
+    setup(root, etc, systemctl).ensure(ON)
+    first = result(root)
+    systemctl.calls.clear()
+    setup(root, etc, systemctl).ensure(ON)
+    assert systemctl.calls == [['is-active', aus.PATH_UNIT], ['is-active', aus.PATH_UNIT]]
+    assert result(root) == first, "an unchanged setup must not rewrite its result every boot"
+
+
+def test_a_changed_template_is_reinstalled(tmp_path):
+    root, etc = project(tmp_path)
+    systemctl = FakeSystemctl()
+    setup(root, etc, systemctl).ensure(ON)
+    template = root / 'systemd' / aus.SERVICE_UNIT
+    template.write_text(template.read_text() + '\n# newer\n')
+    systemctl.calls.clear()
+    assert setup(root, etc, systemctl).ensure(ON)['message'] == 'Installed the update health check.'
+    assert (etc / aus.SERVICE_UNIT).read_text().endswith('# newer\n')
+    assert ['daemon-reload'] in systemctl.calls
+
+
+def test_a_template_that_would_not_run_as_the_web_user_is_refused(tmp_path):
+    root, etc = project(tmp_path)
+    template = root / 'systemd' / aus.SERVICE_UNIT
+    template.write_text(template.read_text().replace('User=__USER__', 'User=root'))
+    systemctl = FakeSystemctl()
+    out = setup(root, etc, systemctl).ensure(ON)
+    assert out['status'] == 'failed' and 'refusing' in out['message']
+    assert not (etc / aus.SERVICE_UNIT).exists()
+    assert systemctl.calls == []
+
+
+def test_a_path_unit_that_starts_something_else_is_refused(tmp_path):
+    root, etc = project(tmp_path)
+    template = root / 'systemd' / aus.PATH_UNIT
+    template.write_text(template.read_text().replace('Unit=ledmatrix-update-verify.service', 'Unit=other.service'))
+    out = setup(root, etc, FakeSystemctl()).ensure(ON)
+    assert out['status'] == 'failed' and not (etc / aus.PATH_UNIT).exists()
+
+
+@pytest.mark.parametrize('case, expected', [
+    ('not_root', 'not running as root'),
+    ('no_web_unit', 'not installed'),
+    ('other_folder', 'runs from'),
+    ('bad_user', 'not a usable account'),
+])
+def test_reports_why_it_could_not_install(tmp_path, case, expected):
+    root, etc = project(tmp_path, user='ghost' if case == 'bad_user' else 'hdpi',
+                        workdir=tmp_path / 'elsewhere' if case == 'other_folder' else None)
+    if case == 'no_web_unit':
+        (etc / aus.WEB_UNIT).unlink()
+    out = setup(root, etc, FakeSystemctl(), root_user=case != 'not_root').ensure(ON)
+    assert out['status'] == 'failed' and expected in out['message']
+    assert result(root)['message'] == out['message']
+    assert not (etc / aus.SERVICE_UNIT).exists()
+
+
+def test_a_path_unit_that_will_not_start_is_a_failure(tmp_path):
+    root, etc = project(tmp_path)
+    out = setup(root, etc, FakeSystemctl(activates=False)).ensure(ON)
+    assert out['status'] == 'failed' and 'did not start' in out['message']
+
+
+def test_systemctl_errors_are_reported(tmp_path):
+    root, etc = project(tmp_path)
+    out = setup(root, etc, FakeSystemctl(fail={'daemon-reload'})).ensure(ON)
+    assert out['status'] == 'failed' and 'daemon-reload refused' in out['message']
+
+
+def test_not_a_systemd_host_is_left_alone(tmp_path):
+    root, _ = project(tmp_path)
+    systemctl = FakeSystemctl()
+    assert setup(root, tmp_path / 'no-etc', systemctl).ensure(ON) is None
+    assert systemctl.calls == []
+
+
+def test_a_folder_with_spaces_is_quoted_in_the_commands(tmp_path):
+    root, etc = project(tmp_path, name='LED Matrix')
+    assert setup(root, etc, FakeSystemctl()).ensure(ON)['status'] == 'installed'
+    service = (etc / aus.SERVICE_UNIT).read_text()
+    assert f'ExecStartPre=/bin/rm -f "{root}/data/auto_update_verify.request"' in service
+    assert f'ExecStart=/usr/bin/python3 "{root}/data/auto_update_verifier.py" "{root}"' in service
+
+
+def test_a_folder_name_systemd_would_reinterpret_is_refused(tmp_path):
+    root, etc = project(tmp_path, name='LED%Matrix')
+    systemctl = FakeSystemctl()
+    out = setup(root, etc, systemctl).ensure(ON)
+    assert out['status'] == 'failed' and 'systemd cannot use' in out['message']
+    assert not (etc / aus.SERVICE_UNIT).exists() and systemctl.calls == []
+
+
+def test_installer_sets_the_toggle_without_a_half_written_config(tmp_path):
+    """first_time_install.sh may run while the display service watches
+    config.json; the file must be replaced whole, never truncated in place."""
+    import re
+    text = (ROOT / 'first_time_install.sh').read_text(encoding='utf-8').replace('\r\n', '\n')
+    script = next(s for s in re.findall(r"<<'PY'\n(.*?)\nPY\n", text, re.S) if 'auto_update' in s)
+    assert 'os.replace(' in script
+    config = tmp_path / 'config.json'
+    config.write_text(json.dumps({'timezone': 'UTC', 'auto_update': {'enabled': False}}))
+    run = subprocess.run([sys.executable, '-', str(config), '1'], input=script, text=True,
+                         capture_output=True)
+    assert run.returncode == 0, run.stderr
+    assert json.loads(config.read_text()) == {'timezone': 'UTC', 'auto_update': {'enabled': True}}
+    assert [p.name for p in tmp_path.iterdir()] == ['config.json'], "a temp file was left behind"
+
+
+def test_the_result_is_kept_when_it_cannot_be_given_to_the_web_user(tmp_path, monkeypatch):
+    """Caught by CI, which runs as a non-root Linux user: chown needs root, and
+    a failed chown used to discard the result, so the General tab never
+    learned whether setup worked."""
+    def refuse(*args):
+        raise PermissionError('Operation not permitted')
+    monkeypatch.setattr(aus.os, 'chown', refuse, raising=False)
+    root, etc = project(tmp_path)
+    out = setup(root, etc, FakeSystemctl()).ensure(ON)
+    assert out['status'] == 'installed'
+    assert result(root) == out

test/test_auto_update_verify.py

@@ -0,0 +1,248 @@
+"""The automatic update's health check and rollback (scripts/utils/auto_update_verify.py).
+
+This is what stands between an unattended update and a device that no longer
+works, so each way an update can fail is exercised against a real git repo:
+the new commit is checked out, the services are "restarted", and whether they
+come up healthy depends on which commit they were restarted onto. Only
+systemd, sudo and the HTTP check are faked.
+"""
+import importlib.util
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT))
+
+_spec = importlib.util.spec_from_file_location(
+    'auto_update_verify', ROOT / 'scripts' / 'utils' / 'auto_update_verify.py')
+av = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(av)
+
+pytestmark = pytest.mark.skipif(shutil.which('git') is None, reason='git not installed')
+
+
+def git(cwd, *args):
+    result = subprocess.run(['git', '-c', 'user.name=t', '-c', 'user.email=t@example.com', *args],
+                            cwd=str(cwd), capture_output=True, text=True)
+    assert result.returncode == 0, result.stderr
+    return result.stdout.strip()
+
+
+def done(args, stdout='', rc=0):
+    return subprocess.CompletedProcess(args, rc, stdout=stdout, stderr='' if rc == 0 else 'failed')
+
+
+def updated_repo(tmp_path, new_requirements=False):
+    """A checkout on a new commit, with the commit it came from."""
+    repo = tmp_path / 'LEDMatrix'
+    repo.mkdir()
+    git(repo, 'init', '-q', '-b', 'main')
+    (repo / 'app.py').write_text('v = 1\n')
+    (repo / 'requirements.txt').write_text('requests\n')
+    git(repo, 'add', '.')
+    git(repo, 'commit', '-qm', 'old')
+    old = git(repo, 'rev-parse', 'HEAD')
+    (repo / 'app.py').write_text('v = 2\n')
+    if new_requirements:
+        (repo / 'requirements.txt').write_text('requests\nnewthing\n')
+    git(repo, 'commit', '-qam', 'new')
+    return repo, old, git(repo, 'rev-parse', 'HEAD')
+
+
+class FakeHost:
+    """systemd, sudo and the web interface.
+
+    Services run whatever commit was checked out when they were last
+    restarted; ``failure`` says how they misbehave on the new commit
+    ("display_down", "web_down", "crash_loop") or on any commit ("always").
+    """
+
+    def __init__(self, repo, bad_head, failure=None, pip_ok=True, restart_failures=0, count_readable=True):
+        self.repo, self.bad_head, self.failure, self.pip_ok = repo, bad_head, failure, pip_ok
+        self.restart_failures = restart_failures  # how many restart commands fail, first to last
+        self.count_readable = count_readable
+        self.running_head = None  # not restarted yet: still the old, healthy code
+        self.restarts = []        # (unit, commit it was restarted onto)
+        self.pip_installs = []
+        self.now = 0.0
+        self.nrestarts = 0
+
+    def broken(self, kind):
+        if self.running_head is None:
+            return False
+        return self.failure == 'always' or (self.running_head == self.bad_head and self.failure == kind)
+
+    def run(self, args, **kwargs):
+        if args[0] == 'git':
+            return subprocess.run(args, **kwargs)
+        if args[:2] == ['systemctl', 'is-active']:
+            return done(args, 'failed\n' if self.broken('display_down') else 'active\n')
+        if args[:2] == ['systemctl', 'show']:
+            if not self.count_readable:
+                return done(args, rc=1)
+            if self.broken('crash_loop'):
+                self.nrestarts += 1
+            return done(args, f'{self.nrestarts}\n')
+        if args[0] == 'sudo' and any(a.endswith('safe_pip_install.sh') for a in args):
+            self.pip_installs.append(args[-1])
+            return done(args, rc=0 if self.pip_ok else 1)
+        if args[:4] == ['sudo', '-n', 'systemctl', 'restart']:
+            if self.restart_failures:
+                self.restart_failures -= 1
+                return done(args, rc=1)  # the old process keeps running
+            self.running_head = git(self.repo, 'rev-parse', 'HEAD')
+            self.restarts.append((args[4], self.running_head))
+            return done(args)
+        raise AssertionError(f'unexpected command: {args}')
+
+    def web_responds(self):
+        return not self.broken('web_down')
+
+    def sleep(self, seconds):
+        self.now += seconds
+
+    def verifier(self):
+        return av.Verifier(self.repo, run=self.run, sleep=self.sleep, clock=lambda: self.now,
+                           web_responds=self.web_responds, log=lambda msg: None)
+
+
+def check(tmp_path, failure=None, new_requirements=False, pip_ok=True, restart_failures=0,
+          count_readable=True, **pending):
+    repo, old, new = updated_repo(tmp_path, new_requirements)
+    fields = {'status': 'pending', 'old_head': old, 'new_head': new,
+              'display_was_active': True, 'dependency_failures': []}
+    fields.update(pending)
+    av.write_pending(av.pending_path(repo), fields)
+    host = FakeHost(repo, new, failure, pip_ok, restart_failures, count_readable)
+    code = host.verifier().verify()
+    result = av.read_pending(av.pending_path(repo))
+    return code, result, host, git(repo, 'rev-parse', 'HEAD'), old, new
+
+
+def test_a_healthy_update_is_kept(tmp_path):
+    code, result, host, head, old, new = check(tmp_path)
+    assert code == 0 and result['status'] == 'success'
+    assert head == new
+    assert host.restarts == [('ledmatrix.service', new), ('ledmatrix-web.service', new)]
+
+
+@pytest.mark.parametrize('failure, reason', [
+    ('display_down', 'the display service did not stay running'),
+    ('web_down', 'the web interface did not respond'),
+    ('crash_loop', 'the display service kept restarting'),
+])
+def test_an_unhealthy_update_is_rolled_back(tmp_path, failure, reason):
+    code, result, host, head, old, new = check(tmp_path, failure)
+    assert code == 0
+    assert result['status'] == 'rolled_back' and result['reason'] == reason
+    assert head == old
+    assert host.restarts[-2:] == [('ledmatrix.service', old), ('ledmatrix-web.service', old)]
+
+
+def test_a_stopped_display_is_not_started(tmp_path):
+    code, result, host, head, old, new = check(tmp_path, display_was_active=False)
+    assert result['status'] == 'success'
+    assert [unit for unit, _ in host.restarts] == ['ledmatrix-web.service']
+
+
+def test_failed_dependencies_roll_back_before_anything_restarts_onto_them(tmp_path):
+    code, result, host, head, old, new = check(tmp_path, dependency_failures=['requirements.txt'])
+    assert result['status'] == 'rolled_back'
+    assert 'requirements.txt' in result['reason']
+    assert head == old
+    assert all(commit == old for _, commit in host.restarts), host.restarts
+
+
+def test_rollback_reinstalls_the_previous_dependencies(tmp_path):
+    code, result, host, head, old, new = check(tmp_path, 'display_down', new_requirements=True)
+    assert result['status'] == 'rolled_back' and result['detail'] is None
+    assert host.pip_installs == [str(tmp_path / 'LEDMatrix' / 'requirements.txt')]
+
+
+def test_a_failed_dependency_reinstall_is_reported(tmp_path):
+    code, result, host, head, old, new = check(tmp_path, 'display_down', new_requirements=True,
+                                               pip_ok=False)
+    assert result['status'] == 'rolled_back' and head == old
+    assert 'Install Base Requirements' in result['detail']
+
+
+def test_still_broken_after_rolling_back_is_rollback_failed(tmp_path):
+    code, result, host, head, old, new = check(tmp_path, 'always')
+    assert code == 1 and result['status'] == 'rollback_failed'
+    assert 'still unhealthy after rolling back' in result['detail']
+
+
+def test_a_failed_restart_is_not_mistaken_for_a_healthy_update(tmp_path):
+    """When the restart command itself fails the old process keeps answering,
+    so checking it would pass an update that never started."""
+    code, result, host, head, old, new = check(tmp_path, restart_failures=1)
+    assert result['status'] == 'rolled_back'
+    assert result['reason'] == 'restarting the services failed'
+    assert head == old
+
+
+def test_restarts_that_keep_failing_after_rollback_are_rollback_failed(tmp_path):
+    code, result, host, head, old, new = check(tmp_path, restart_failures=100)
+    assert code == 1 and result['status'] == 'rollback_failed'
+    assert 'restarting the services failed' in result['detail']
+
+
+def test_an_unreadable_restart_count_is_never_called_stable(tmp_path):
+    """With no restart count a crash loop looks healthy between attempts."""
+    code, result, host, head, old, new = check(tmp_path, count_readable=False)
+    assert result['status'] != 'success'
+    assert 'restart count could not be read' in result['reason']
+
+
+def test_nothing_pending_does_nothing(tmp_path):
+    repo, _, _ = updated_repo(tmp_path)
+    host = FakeHost(repo, None)
+    assert host.verifier().verify() == 0
+    assert host.restarts == []
+
+
+def test_a_crash_is_recorded_not_left_as_running(tmp_path, monkeypatch):
+    repo, old, new = updated_repo(tmp_path)
+    av.write_pending(av.pending_path(repo), {'status': 'pending', 'old_head': old, 'new_head': new})
+
+    def boom(self):
+        raise RuntimeError('boom')
+    monkeypatch.setattr(av.Verifier, 'verify', boom)
+    assert av.main(['auto_update_verify.py', str(repo)]) == 1
+    result = av.read_pending(av.pending_path(repo))
+    assert result['status'] == 'rollback_failed' and result['detail'] == 'boom'
+
+
+def test_units_installers_and_updater_agree():
+    """The request file, the copied verifier, the unit names and the places
+    that install them must all line up, or the health check silently never
+    starts on real devices -- and code updates stay paused."""
+    from web_interface import auto_update as au
+    from src import auto_update_setup as aus
+    from src.startup_validator import StartupValidator
+
+    service = (ROOT / 'systemd' / aus.SERVICE_UNIT).read_text(encoding='utf-8')
+    path = (ROOT / 'systemd' / aus.PATH_UNIT).read_text(encoding='utf-8')
+    request = f'__PROJECT_ROOT_DIR__/{au.REQUEST_REL.as_posix()}'
+    assert f'PathExists={request}' in path
+    assert f'Unit={aus.SERVICE_UNIT}' in path
+    assert f'ExecStartPre=/bin/rm -f "{request}"' in service, "the request must be consumed or the path unit re-fires"
+    assert f'"__PROJECT_ROOT_DIR__/{au.VERIFIER_COPY_REL.as_posix()}" "__PROJECT_ROOT_DIR__"' in service
+    assert au.PENDING_REL.name == av.PENDING_NAME
+    assert au.PATH_UNIT == aus.PATH_UNIT and au.SETUP_RESULT_REL == aus.RESULT_REL
+
+    for installer in ('scripts/install/install_web_service.sh', 'scripts/install/install_service.sh'):
+        text = (ROOT / installer).read_text(encoding='utf-8')
+        assert 'ledmatrix-update-verify.service ledmatrix-update-verify.path' in text, installer
+        assert 'enable --now ledmatrix-update-verify.path' in text, installer
+    for unit in aus.UNITS:
+        assert (f'systemd/{unit}', f'/etc/systemd/system/{unit}') in StartupValidator._UNITS
+
+    # Triggering takes no privilege any more; no sudoers rule should linger.
+    for sudoers in ('scripts/install/configure_web_sudo.sh', 'first_time_install.sh'):
+        assert not re.search(r'NOPASSWD:.*update-verify', (ROOT / sudoers).read_text(encoding='utf-8')), sudoers

test/test_common_is_hardware_free.py

@@ -1,124 +0,0 @@
-"""src/common must stay importable without display hardware.
-
-Plugins import src.common.sports_* in place of their bundled copies. If any of
-those modules reaches src.display_manager (and through it rgbmatrix) or the
-src.base_classes package (whose core.py imports DisplayManager), a scoreboard
-adopting it acquires a hardware dependency it never had, and the headless
-tooling -- the web preview, check_plugin.py, these tests on a laptop -- stops
-being able to load it.
-
-Two checks, because each misses what the other catches:
-
-1. A real import in a fresh interpreter with rgbmatrix made unimportable.
-   Catches indirect imports (common -> X -> display_manager) that no text scan
-   of src/common would see.
-2. An AST scan of src/common/*.py for module-level imports of the hardware and
-   plugin-system packages. Catches a direct import even where the target
-   happens to import cleanly on this machine.
-"""
-
-import ast
-import subprocess
-import sys
-import textwrap
-from pathlib import Path
-
-REPO_ROOT = Path(__file__).resolve().parents[1]
-COMMON = REPO_ROOT / "src" / "common"
-
-FORBIDDEN = ("src.base_classes", "src.display_manager", "src.plugin_system")
-
-#: Existing module-level violations, by file name, each with the reason it is
-#: tolerated. Empty when this test was added (core 3.4.0 + sports_helpers):
-#: no src/common module imported any of FORBIDDEN at module level. Add to this
-#: only with a comment saying why, never to get a new import past the test.
-ALLOWLIST = {}
-
-
-def test_sports_modules_import_without_hardware():
-    modules = sorted(p.stem for p in COMMON.glob("sports_*.py"))
-    assert "sports_helpers" in modules
-    script = textwrap.dedent(f"""
-        import importlib, json, sys
-        sys.modules["rgbmatrix"] = None   # any import of it now raises
-        importlib.import_module("src.common")
-        for name in {modules!r}:
-            importlib.import_module("src.common." + name)
-        loaded = sorted(m for m in sys.modules
-                        if m.startswith(("src.display_manager", "src.base_classes")))
-        print(json.dumps(loaded))
-    """)
-    result = subprocess.run(
-        [sys.executable, "-c", script], cwd=str(REPO_ROOT),
-        capture_output=True, text=True, timeout=120)
-    assert result.returncode == 0, (
-        f"importing src.common without rgbmatrix failed:\n{result.stderr}")
-    loaded = result.stdout.strip().splitlines()[-1]
-    assert loaded == "[]", f"src.common pulled in hardware modules: {loaded}"
-
-
-def _module_level_imports(tree):
-    """Imports executed at import time: top level, including inside if/try
-    blocks, but not inside functions or classes (those are deferred)."""
-    stack = list(tree.body)
-    while stack:
-        node = stack.pop()
-        if isinstance(node, (ast.Import, ast.ImportFrom)):
-            yield node
-        elif isinstance(node, (ast.If, ast.Try, ast.With)):
-            stack.extend(node.body)
-            stack.extend(getattr(node, "orelse", []))
-            stack.extend(getattr(node, "finalbody", []))
-            for handler in getattr(node, "handlers", []):
-                stack.extend(handler.body)
-
-
-def _targets(node):
-    if isinstance(node, ast.Import):
-        return [alias.name for alias in node.names]
-    if node.module is None:
-        return []
-    if node.module == "src":
-        return [f"src.{alias.name}" for alias in node.names]
-    return [node.module]
-
-
-def _violations():
-    found = {}
-    for path in sorted(COMMON.glob("*.py")):
-        tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
-        for node in _module_level_imports(tree):
-            for target in _targets(node):
-                if target == FORBIDDEN or any(
-                        target == f or target.startswith(f + ".") for f in FORBIDDEN):
-                    found.setdefault(path.name, []).append(f"line {node.lineno}: {target}")
-    return found
-
-
-def test_no_module_level_hardware_or_plugin_system_imports():
-    found = {name: hits for name, hits in _violations().items() if name not in ALLOWLIST}
-    assert found == {}, (
-        "src/common must not import these at module level (defer the import "
-        f"into the function that needs it): {found}")
-
-
-def test_allowlist_has_no_stale_entries():
-    stale = sorted(set(ALLOWLIST) - set(_violations()))
-    assert stale == [], f"allowlisted but no longer violating, remove: {stale}"
-
-
-def test_the_scan_sees_a_direct_import(tmp_path):
-    # Guard the guard: the walker must find imports nested in try/if.
-    tree = ast.parse(textwrap.dedent("""
-        try:
-            from src.display_manager import DisplayManager
-        except ImportError:
-            pass
-        if True:
-            import src.base_classes.sports
-        from src import plugin_system
-        def later():
-            from src.display_manager import DisplayManager
-    """))
-    targets = sorted(t for n in _module_level_imports(tree) for t in _targets(n))
-    assert targets == ["src.base_classes.sports", "src.display_manager", "src.plugin_system"]

test/test_scroll_helper.py

@@ -387,9 +387,81 @@ class TestFrameStatsPercentiles:
 
     def test_log_frame_rate_emits_the_line_and_clears_the_window(self, helper):
         helper._window = [0.010] * 20
+        helper.last_frame_time = time.time()  # arm the clock; see below
         helper.last_fps_log_time = 0.0  # force the 5s boundary
         with patch.object(helper.logger, "info") as info:
             helper.log_frame_rate()
         assert info.called
         assert "Scroll frame stats" in info.call_args[0][0]
         assert helper._window == []
+
+
+class TestIdleGapIsNotAFrame:
+    """The first frame of a scroll has no predecessor, so timing one measures
+    the idle gap since the last scroll rather than a frame.
+
+    Left in, that gap lands in the max field of otherwise healthy windows and
+    counts as one stall per scroll start -- about 0.2% at 500 frames to a
+    window, which is the same order as the real stall rates it sits next to.
+    The stall rate is the number used to judge whether a scroll change worked,
+    so it has to be clean.
+    """
+
+    def test_clock_starts_unarmed(self, helper):
+        assert helper.last_frame_time is None
+
+    def test_first_frame_seeds_the_clock_without_sampling(self, helper):
+        helper.log_frame_rate()
+        assert helper.last_frame_time is not None
+        assert helper._window == []
+        assert helper.frame_times == []
+
+    def test_second_frame_is_sampled(self, helper):
+        helper.log_frame_rate()
+        helper.log_frame_rate()
+        assert len(helper._window) == 1
+
+    def test_reset_scroll_disarms_the_clock(self, helper):
+        helper.log_frame_rate()
+        assert helper.last_frame_time is not None
+        helper.reset_scroll()
+        assert helper.last_frame_time is None
+
+    def test_gap_longer_than_the_log_interval_is_dropped(self, helper):
+        """Covers callers that scroll without ever calling reset_scroll()."""
+        helper.last_frame_time = time.time() - 137.0  # a real observed gap
+        helper.log_frame_rate()
+        assert helper._window == []
+        assert helper.frame_times == []
+
+    def test_a_gap_does_not_reach_the_stats_line(self, helper):
+        helper.last_frame_time = time.time() - 137.0
+        helper.last_fps_log_time = 0.0  # the 5s boundary is also due
+        with patch.object(helper.logger, "info") as info:
+            helper.log_frame_rate()
+        assert not info.called, "the idle gap was reported as a frame"
+
+    def test_window_of_only_gaps_logs_nothing(self, helper):
+        """A window whose every sample was dropped has nothing to report --
+        and reporting the gap itself is the bug this guards."""
+        helper.last_fps_log_time = 0.0
+        with patch.object(helper.logger, "info") as info:
+            helper.log_frame_rate()   # seeds
+            helper.last_frame_time = time.time() - 137.0
+            helper.log_frame_rate()   # dropped
+        assert not info.called
+
+    def test_seeding_restarts_the_window_timer(self, helper):
+        """A new scroll should not open by reporting a one-frame window."""
+        helper.last_fps_log_time = 0.0  # boundary long overdue
+        helper.log_frame_rate()         # seeds
+        with patch.object(helper.logger, "info") as info:
+            helper.log_frame_rate()     # first real frame
+        assert not info.called
+        assert len(helper._window) == 1
+
+    def test_a_normal_frame_still_counts(self, helper):
+        helper.last_frame_time = time.time() - 0.010
+        helper.log_frame_rate()
+        assert len(helper._window) == 1
+        assert helper._window[0] == pytest.approx(0.010, abs=0.005)

test/test_sports_helpers.py

@@ -1,600 +0,0 @@
-"""src.common.sports_helpers: behaviour, host contract, and parity with the plugins.
-
-The bodies were copied from the scoreboard plugins' sports.py, so the plugins'
-own tests already cover them there. They are ported here for two reasons: core
-now owns a copy that can drift on its own, and a plugin that deletes its copy
-loses those tests with it.
-
-The parity class is what keeps "byte-identical" true after this lands. Point
-LEDMATRIX_PLUGINS at a ledmatrix-plugins checkout and every promoted body is
-compared, as a docstring-stripped AST, against every plugin copy that carries
-it. Without the variable it skips rather than fails, since CI has no plugins
-checkout.
-"""
-
-import ast
-import logging
-import math
-import os
-import time
-from pathlib import Path
-
-import pytest
-from PIL import Image
-from PIL.PngImagePlugin import PngInfo
-
-from src.common import sports_helpers
-from src.common.sports_helpers import (
-    MAX_WINDOW_DAYS,
-    MIN_WINDOW_DAYS,
-    SportsHelpersMixin,
-    clamp_seconds,
-    clamp_window,
-    logo_needs_refresh,
-    spread_weighted_order,
-)
-from src.common.sports_shared import SportsCoreSharedMixin
-
-
-# ---------------------------------------------------------------------------
-# Free functions
-# ---------------------------------------------------------------------------
-
-class TestClampWindow:
-    def test_within_range_is_kept(self):
-        assert clamp_window(14, 7) == 14
-        assert clamp_window("30", 7) == 30
-
-    def test_clamped_to_bounds(self):
-        assert (MIN_WINDOW_DAYS, MAX_WINDOW_DAYS) == (1, 60)
-        assert clamp_window(0, 7) == 1
-        assert clamp_window(-5, 7) == 1
-        assert clamp_window(365, 7) == 60
-
-    @pytest.mark.parametrize("bad", [None, "abc", [], {}, float("nan")])
-    def test_unusable_falls_back(self, bad):
-        assert clamp_window(bad, 7) == 7
-
-    @pytest.mark.parametrize("inf", [float("inf"), float("-inf")])
-    def test_infinity_falls_back_instead_of_raising(self, inf):
-        # json.loads accepts a bare Infinity; int(inf) raises OverflowError.
-        with pytest.raises(OverflowError):
-            int(inf)
-        assert clamp_window(inf, 7) == 7
-
-
-class TestClampSeconds:
-    def test_defaults_bound_five_seconds_to_a_day(self):
-        assert clamp_seconds(1, 60) == 5
-        assert clamp_seconds(10 ** 9, 60) == 86400
-        assert clamp_seconds(300, 60) == 300
-
-    def test_custom_bounds(self):
-        assert clamp_seconds(1, 60, low=30, high=900) == 30
-        assert clamp_seconds(5000, 60, low=30, high=900) == 900
-
-    @pytest.mark.parametrize("bad", [None, "x", float("nan"), float("inf")])
-    def test_unusable_falls_back(self, bad):
-        assert clamp_seconds(bad, 60) == 60
-
-
-def _png(path, size=(64, 64), color=(100, 100, 100, 255), marker=None):
-    img = Image.new("RGBA", size, color)
-    info = None
-    if marker is not None:
-        info = PngInfo()
-        info.add_text("ledmatrix_placeholder", marker)
-    img.save(path, "PNG", pnginfo=info)
-    return path
-
-
-class TestLogoNeedsRefresh:
-    def test_stale_placeholder_is_retried(self, tmp_path):
-        from src.logo_downloader import PLACEHOLDER_RETRY_SECONDS
-        stamp = str(time.time() - PLACEHOLDER_RETRY_SECONDS - 60)
-        assert logo_needs_refresh(_png(tmp_path / "A.png", marker=stamp)) is True
-
-    def test_fresh_placeholder_is_trusted(self, tmp_path):
-        assert logo_needs_refresh(_png(tmp_path / "A.png", marker=str(time.time()))) is False
-
-    def test_real_logo_is_never_refreshed(self, tmp_path):
-        assert logo_needs_refresh(_png(tmp_path / "A.png", size=(32, 32),
-                                       color=(200, 0, 0, 255))) is False
-
-    def test_missing_file_is_not_a_placeholder(self, tmp_path):
-        assert logo_needs_refresh(tmp_path / "nope.png") is False
-
-    def test_core_without_placeholder_marking_keeps_old_behaviour(self, monkeypatch, tmp_path):
-        # A None entry in sys.modules makes the import raise ImportError, the
-        # same thing an older core without these names does.
-        monkeypatch.setitem(__import__("sys").modules, "src.logo_downloader", None)
-        stamp = str(time.time() - 10 ** 7)
-        assert logo_needs_refresh(_png(tmp_path / "A.png", marker=stamp)) is False
-
-    def test_the_import_is_deferred(self):
-        tree = ast.parse(Path(sports_helpers.__file__).read_text(encoding="utf-8"))
-        top = [n for n in tree.body if isinstance(n, (ast.Import, ast.ImportFrom))]
-        assert not any(isinstance(n, ast.ImportFrom) and n.module == "src.logo_downloader"
-                       for n in top)
-
-
-def _back_to_back(cycle):
-    return any(cycle[i] == cycle[(i + 1) % len(cycle)] for i in range(len(cycle)))
-
-
-class TestSpreadWeightedOrder:
-    def test_equal_weights_are_plain_order(self):
-        assert spread_weighted_order([1, 1, 1]) == [0, 1, 2]
-        assert spread_weighted_order([]) == []
-
-    def test_a_favourite_at_the_end_wraps_its_extra_turn(self):
-        order = spread_weighted_order([1, 1, 1, 2])
-        assert sorted(order) == [0, 1, 2, 3, 3]
-        assert not _back_to_back(order)
-
-    @pytest.mark.parametrize("weights,expected", [
-        ([3, 1, 1], [0, 1, 0, 2, 0]),
-        ([1, 3, 1], [0, 1, 1, 2, 1]),
-        ([5, 1], None),
-        ([2, 1], None),
-    ])
-    def test_ratio_wins_over_spacing(self, weights, expected):
-        order = spread_weighted_order(weights)
-        assert [order.count(i) for i in range(len(weights))] == weights
-        if expected is not None:
-            assert order == expected
-
-    def test_mixin_exposes_it_as_a_staticmethod(self):
-        assert SportsHelpersMixin._spread_weighted_order([1, 2]) == spread_weighted_order([1, 2])
-        assert isinstance(SportsHelpersMixin.__dict__["_spread_weighted_order"], staticmethod)
-
-
-# ---------------------------------------------------------------------------
-# Mixin methods, against stand-in hosts carrying only what each reads
-# ---------------------------------------------------------------------------
-
-class _Dwell(SportsHelpersMixin):
-    def __init__(self, last_game_switch):
-        self._last_display_call_monotonic = 0.0
-        self.last_game_switch = last_game_switch
-
-
-class TestResetDwellOnReentry:
-    def test_first_frame_resets(self):
-        mgr = _Dwell(time.time() - 6.0)
-        assert mgr._reset_dwell_on_reentry() is True
-        assert time.time() - mgr.last_game_switch < 1.0
-
-    def test_frames_inside_one_stint_do_not(self):
-        mgr = _Dwell(time.time() - 6.0)
-        mgr._reset_dwell_on_reentry()
-        stamp = mgr.last_game_switch
-        assert [mgr._reset_dwell_on_reentry() for _ in range(3)] == [False] * 3
-        assert mgr.last_game_switch == stamp
-
-    def test_returning_after_a_long_gap_resets(self):
-        mgr = _Dwell(time.time() - 150.0)
-        mgr._last_display_call_monotonic = time.monotonic() - 150.0
-        assert mgr._reset_dwell_on_reentry() is True
-
-    def test_just_under_the_threshold_is_the_same_stint(self):
-        mgr = _Dwell(time.time() - 150.0)
-        mgr._last_display_call_monotonic = time.monotonic() - (
-            SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS - 1.0)
-        stamp = mgr.last_game_switch
-        assert mgr._reset_dwell_on_reentry() is False
-        assert mgr.last_game_switch == stamp
-
-    def test_live_zero_sentinel_survives(self):
-        mgr = _Dwell(0)
-        assert mgr._reset_dwell_on_reentry() is False
-        assert mgr.last_game_switch == 0
-
-    def test_host_without_any_state(self):
-        mgr = SportsHelpersMixin()
-        assert mgr._reset_dwell_on_reentry() is False  # no last_game_switch
-        assert mgr._last_display_call_monotonic > 0
-
-
-def _game(gid, home="X", away="Y"):
-    return {"id": gid, "home_abbr": home, "away_abbr": away}
-
-
-class _Switch(SportsHelpersMixin):
-    def __init__(self, games, boost=None, favorites=("UF",)):
-        self.games_list = games
-        self.favorite_teams = list(favorites)
-        if boost is not None:
-            self.favorite_rotation_boost = boost
-        self.current_game_index = 0
-
-    def _is_favorite_game(self, g):
-        return g["home_abbr"] in self.favorite_teams or g["away_abbr"] in self.favorite_teams
-
-    def walk(self, steps):
-        shown = []
-        for _ in range(steps):
-            self.current_game_index = self._next_switch_index()
-            shown.append(self.games_list[self.current_game_index]["id"])
-        return shown
-
-
-class TestNextSwitchIndex:
-    def test_no_boost_attribute_is_the_plain_rotation(self):
-        s = _Switch([_game("a"), _game("b", "UF"), _game("c")])
-        assert s.walk(6) == ["b", "c", "a", "b", "c", "a"]
-
-    def test_boost_without_a_favourite_changes_nothing(self):
-        assert _Switch([_game("a"), _game("b"), _game("c")], boost=3).walk(3) == ["b", "c", "a"]
-
-    def test_boost_two_spreads_the_favourite(self):
-        s = _Switch([_game("a"), _game("uf", "UF"), _game("c"), _game("d")], boost=2)
-        cycle = s.walk(5)
-        assert sorted(cycle) == ["a", "c", "d", "uf", "uf"]
-        assert not _back_to_back(cycle)
-        assert s.walk(5) == cycle
-
-    def test_boost_three_two_favourites(self):
-        s = _Switch([_game("a", "UF"), _game("b"), _game("c", "FSU"), _game("d"), _game("e")],
-                    boost=3, favorites=("UF", "FSU"))
-        cycle = s.walk(9)
-        assert cycle.count("a") == 3 and cycle.count("c") == 3
-
-    def test_index_reset_by_update_is_followed(self):
-        s = _Switch([_game("a"), _game("uf", "UF"), _game("c"), _game("d")], boost=2)
-        s.walk(3)
-        s.current_game_index = 0
-        assert s.walk(1)[0] == "uf"
-
-    def test_single_card_stays_put(self):
-        assert _Switch([_game("uf", "UF")], boost=5).walk(3) == ["uf", "uf", "uf"]
-
-
-class _Settings(SportsHelpersMixin):
-    def __init__(self, mode_config):
-        self.mode_config = mode_config
-        self.logger = logging.getLogger("test.sports_helpers")
-        self.league = "nfl"
-
-
-class TestSettingInt:
-    def test_reads_and_clamps(self):
-        assert _Settings({"n": 3})._setting_int("n", 1, 1, 5) == 3
-        assert _Settings({"n": "4"})._setting_int("n", 1, 1, 5) == 4
-        assert _Settings({"n": 99})._setting_int("n", 1, 1, 5) == 5
-        assert _Settings({"n": -2})._setting_int("n", 1, 1, 5) == 1
-        assert _Settings({})._setting_int("n", 2, 1, 5) == 2
-
-    @pytest.mark.parametrize("bad", ["abc", None, [], float("inf"), float("nan")])
-    def test_unusable_falls_back_with_a_warning(self, bad, caplog):
-        with caplog.at_level(logging.WARNING, logger="test.sports_helpers"):
-            assert _Settings({"n": bad})._setting_int("n", 2, 1, 5) == 2
-        assert "ignoring unusable n" in caplog.text
-
-
-class _Custom(SportsHelpersMixin):
-    def __init__(self, config, mode=None):
-        self.config = config
-        if mode is not None:
-            self.SKIN_MODE = mode
-
-
-class TestModeCustomization:
-    def test_no_mode_returns_block_as_is(self):
-        cust = {"score": {"font_size": 10}}
-        assert _Custom({"customization": cust})._mode_customization() is cust
-
-    def test_non_dict_customization(self):
-        assert _Custom({"customization": "junk"}, "live")._mode_customization() == {}
-        assert _Custom({}, "live")._mode_customization() == {}
-
-    def test_mode_overrides_merge_and_none_inherits(self):
-        cfg = {"customization": {
-            "score": {"font_size": 10, "text_color": [1, 1, 1]},
-            "layout": {"score": {"x_offset": 2, "y_offset": 3}},
-            "modes": {"live": {
-                "score": {"font_size": 12, "text_color": None},
-                "status": "not-a-dict",
-                "layout": {"score": {"y_offset": 0, "x_offset": None}, "bad": 5},
-            }},
-        }}
-        merged = _Custom(cfg, "live")._mode_customization()
-        assert merged["score"] == {"font_size": 12, "text_color": [1, 1, 1]}
-        assert merged["layout"]["score"] == {"x_offset": 2, "y_offset": 0}
-        assert "bad" not in merged["layout"]
-        # The stored config is not mutated.
-        assert cfg["customization"]["score"]["font_size"] == 10
-        assert cfg["customization"]["layout"]["score"]["y_offset"] == 3
-
-    def test_other_mode_is_untouched(self):
-        cust = {"modes": {"live": {"score": {"font_size": 12}}}}
-        assert _Custom({"customization": cust}, "recent")._mode_customization() is cust
-
-
-class TestOddsColor:
-    def test_without_element_color_is_green(self):
-        assert SportsHelpersMixin()._odds_color() == (0, 255, 0)
-
-    def test_raising_getter_is_green(self):
-        class Host(SportsHelpersMixin):
-            def _element_color(self, element, default):
-                raise RuntimeError("boom")
-        assert Host()._odds_color() == (0, 255, 0)
-
-    def test_asks_for_odds_text_with_green_default(self):
-        seen = []
-
-        class Host(SportsHelpersMixin):
-            def _element_color(self, element, default):
-                seen.append((element, default))
-                return (9, 9, 9)
-        assert Host()._odds_color() == (9, 9, 9)
-        assert seen == [("odds_text", (0, 255, 0))]
-
-
-class _Scorebug(SportsHelpersMixin):
-    def __init__(self, scroll_card):
-        self.config = {"scroll_card": scroll_card}
-
-    def _card_option(self, key, default=None):
-        return SportsCoreSharedMixin._card_option(self, key, default)
-
-    def _format_game_date(self, game_date, game=None):
-        return "DATE"
-
-    def _format_game_time(self, game_time):
-        return "TIME"
-
-
-class TestUpcomingDateAndTimeText:
-    def _text(self, block):
-        return _Scorebug(block)._upcoming_date_and_time_text("2026-10-07", "19:00")
-
-    def test_defaults_draw_both(self):
-        assert self._text({}) == ("DATE", "TIME")
-
-    def test_scroll_card_toggles_do_not_reach_the_scorebug(self):
-        assert self._text({"show_date": False, "show_time": False}) == ("DATE", "TIME")
-
-    def test_switch_toggles(self):
-        assert self._text({"switch_show_date": False}) == ("", "TIME")
-        assert self._text({"switch_show_time": False}) == ("DATE", "")
-
-
-class TestFavoriteKey:
-    def test_defaults_to_abbreviation(self):
-        assert SportsHelpersMixin()._favorite_key({"home_abbr": "UF"}, "home") == "UF"
-
-    def test_missing_abbreviation_never_matches(self):
-        assert SportsHelpersMixin()._favorite_key({}, "away") is None
-
-
-# ---------------------------------------------------------------------------
-# Host contract
-# ---------------------------------------------------------------------------
-
-def _self_reads():
-    """Every ``self.X`` / ``getattr(self, "X")`` the mixin reads, by parsing it."""
-    tree = ast.parse(Path(sports_helpers.__file__).read_text(encoding="utf-8"))
-    cls = next(n for n in tree.body
-               if isinstance(n, ast.ClassDef) and n.name == "SportsHelpersMixin")
-    names = set()
-    for node in ast.walk(cls):
-        if (isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load)
-                and isinstance(node.value, ast.Name) and node.value.id == "self"):
-            names.add(node.attr)
-        if (isinstance(node, ast.Call) and isinstance(node.func, ast.Name)
-                and node.func.id == "getattr" and len(node.args) >= 2
-                and isinstance(node.args[0], ast.Name) and node.args[0].id == "self"
-                and isinstance(node.args[1], ast.Constant)):
-            names.add(node.args[1].value)
-    return names
-
-
-class _Host(SportsCoreSharedMixin, SportsHelpersMixin):
-    """The smallest thing a scoreboard's SportsCore has to be for these to run."""
-
-    def __init__(self, config, mode_config=None):
-        self.config = config
-        self.mode_config = mode_config or {}
-        self.logger = logging.getLogger("test.sports_helpers.host")
-        self.games_list = []
-        self.current_game_index = 0
-        self.last_game_switch = 0
-
-    def _is_favorite_game(self, game):
-        return game.get("home_abbr") == "UF"
-
-
-class TestHostContract:
-    def test_every_host_read_is_documented(self):
-        defined = set(dir(SportsHelpersMixin))
-        # Written by the mixin itself before (or instead of) being read.
-        own_state = {"_last_display_call_monotonic", "_switch_order",
-                     "_switch_order_key", "_switch_position"}
-        needed = _self_reads() - defined - own_state
-        doc = sports_helpers.__doc__
-        undocumented = sorted(n for n in needed | own_state if f"``{n}``" not in doc)
-        assert undocumented == [], (
-            f"self attributes read but not in the module docstring's host "
-            f"contract: {undocumented}")
-
-    def test_every_constant_read_has_a_default(self):
-        # SKIN_MODE is the one exception: the host's mode name, read with
-        # getattr(self, 'SKIN_MODE', None) where absence means "no mode". A
-        # default here would say nothing the getattr does not already.
-        missing = sorted(n for n in _self_reads() - {"SKIN_MODE"}
-                         if n.upper() == n and not hasattr(SportsHelpersMixin, n))
-        assert missing == []
-
-    def test_defaults_match_what_the_plugins_ship(self):
-        assert SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS == 5.0
-        assert SportsHelpersMixin.favorite_rotation_boost == 1
-
-    def test_stateless_and_constructor_free(self):
-        assert "__init__" not in SportsHelpersMixin.__dict__
-        assert SportsHelpersMixin().__dict__ == {}
-
-    def test_base_order_does_not_matter(self):
-        class Reversed(SportsHelpersMixin, SportsCoreSharedMixin):
-            __init__ = _Host.__init__
-            _is_favorite_game = _Host._is_favorite_game
-
-        for cls in (_Host, Reversed):
-            h = cls({"scroll_card": {"switch_show_time": False, "time_format": "24h"}})
-            assert h._upcoming_date_and_time_text("10/7", "7:00 PM") == ("10/7", "")
-
-    def test_shared_mixin_supplies_the_card_helpers(self):
-        h = _Host({"scroll_card": {"switch_date_format": "abbrev", "time_format": "24h"}})
-        assert h._upcoming_date_and_time_text("10/7", "7:05 PM") == ("Oct 7", "19:05")
-
-    def test_odds_color_resolves_through_element_style(self):
-        h = _Host({"customization": {"odds_text": {"text_color": [1, 2, 3]}}})
-        assert tuple(h._odds_color()) == (1, 2, 3)
-        assert tuple(_Host({})._odds_color()) == (0, 255, 0)
-
-    def test_rotation_settings_and_dwell_on_one_host(self):
-        h = _Host({}, mode_config={"favorite_rotation_boost": "2"})
-        h.favorite_rotation_boost = h._setting_int("favorite_rotation_boost", 1, 1, 5)
-        h.games_list = [_game("a"), _game("uf", "UF"), _game("c")]
-        seen = []
-        for _ in range(4):
-            h.current_game_index = h._next_switch_index()
-            seen.append(h.games_list[h.current_game_index]["id"])
-        assert sorted(seen) == ["a", "c", "uf", "uf"]
-        h.last_game_switch = time.time() - 100
-        assert h._reset_dwell_on_reentry() is True
-
-
-# ---------------------------------------------------------------------------
-# Parity with the plugin copies
-# ---------------------------------------------------------------------------
-
-SCOREBOARDS = ("afl", "baseball", "basketball", "football", "hockey",
-               "lacrosse", "nrl", "soccer", "ufc")
-
-#: core name -> (where it lives in the plugins' sports.py, plugin name,
-#: plugins expected to carry it).
-_ALL = SCOREBOARDS
-_NOT_UFC = tuple(p for p in SCOREBOARDS if p != "ufc")
-PROMOTED = {
-    "clamp_window": ("module", "_clamp_window", _ALL),
-    "clamp_seconds": ("module", "_clamp_seconds", _ALL),
-    "logo_needs_refresh": ("module", "_logo_needs_refresh", _ALL),
-    "spread_weighted_order": ("SportsCore", "_spread_weighted_order", _ALL),
-    "_mode_customization": ("SportsCore", "_mode_customization", _ALL),
-    "_setting_int": ("SportsCore", "_setting_int", _ALL),
-    "_reset_dwell_on_reentry": ("SportsCore", "_reset_dwell_on_reentry", _ALL),
-    "_next_switch_index": ("SportsCore", "_next_switch_index", _ALL),
-    "_odds_color": ("SportsCore", "_odds_color", _NOT_UFC),
-    "_upcoming_date_and_time_text": ("SportsCore", "_upcoming_date_and_time_text", _NOT_UFC),
-}
-
-#: Public names here that are private in the plugins.
-_RENAMES = {
-    "clamp_window": "_clamp_window",
-    "clamp_seconds": "_clamp_seconds",
-    "logo_needs_refresh": "_logo_needs_refresh",
-    "spread_weighted_order": "_spread_weighted_order",
-    "MIN_WINDOW_DAYS": "_MIN_WINDOW_DAYS",
-    "MAX_WINDOW_DAYS": "_MAX_WINDOW_DAYS",
-}
-
-
-def _plugins_root():
-    raw = os.environ.get("LEDMATRIX_PLUGINS")
-    if not raw:
-        pytest.skip("set LEDMATRIX_PLUGINS to a ledmatrix-plugins checkout to "
-                    "compare sports_helpers against the plugin copies")
-    root = Path(raw)
-    if (root / "plugins").is_dir():
-        root = root / "plugins"
-    if not (root / "football-scoreboard" / "sports.py").is_file():
-        pytest.skip(f"LEDMATRIX_PLUGINS={raw} has no football-scoreboard/sports.py")
-    return root
-
-
-class _Normalise(ast.NodeTransformer):
-    def visit_Name(self, node):
-        node.id = _RENAMES.get(node.id, node.id)
-        return node
-
-    def _func(self, node):
-        node.name = _RENAMES.get(node.name, node.name)
-        node.decorator_list = []
-        body = node.body
-        if (body and isinstance(body[0], ast.Expr)
-                and isinstance(body[0].value, ast.Constant)
-                and isinstance(body[0].value.value, str)):
-            node.body = body[1:] or [ast.Pass()]
-        self.generic_visit(node)
-        return node
-
-    visit_FunctionDef = _func
-
-
-def _dump(node):
-    node = ast.parse(ast.unparse(node)).body[0]   # detach and copy
-    return ast.dump(_Normalise().visit(node))
-
-
-def _definitions(tree):
-    module, core = {}, {}
-    for node in tree.body:
-        if isinstance(node, ast.FunctionDef):
-            module[node.name] = node
-        elif isinstance(node, ast.Assign) and isinstance(node.targets[0], ast.Name):
-            module[node.targets[0].id] = node
-        elif isinstance(node, ast.ClassDef) and node.name == "SportsCore":
-            for item in node.body:
-                if isinstance(item, ast.FunctionDef):
-                    core[item.name] = item
-                elif isinstance(item, (ast.Assign, ast.AnnAssign)):
-                    target = item.targets[0] if isinstance(item, ast.Assign) else item.target
-                    if isinstance(target, ast.Name):
-                        core[target.id] = item
-    return {"module": module, "SportsCore": core}
-
-
-def _core_definitions():
-    tree = ast.parse(Path(sports_helpers.__file__).read_text(encoding="utf-8"))
-    out = {}
-    for node in tree.body:
-        if isinstance(node, ast.FunctionDef):
-            out[node.name] = node
-        elif isinstance(node, ast.ClassDef) and node.name == "SportsHelpersMixin":
-            for item in node.body:
-                if isinstance(item, ast.FunctionDef):
-                    out[item.name] = item
-    return out
-
-
-class TestParityWithPlugins:
-    @pytest.mark.parametrize("name", sorted(PROMOTED))
-    def test_body_matches_every_plugin_copy(self, name):
-        root = _plugins_root()
-        where, plugin_name, carriers = PROMOTED[name]
-        ours = _dump(_core_definitions()[name])
-        drifted, missing = [], []
-        for sport in carriers:
-            defs = _definitions(ast.parse(
-                (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8")))
-            theirs = defs[where].get(plugin_name)
-            if theirs is None:
-                missing.append(sport)
-            elif _dump(theirs) != ours:
-                drifted.append(sport)
-        assert missing == [], f"{plugin_name} no longer in: {missing}"
-        assert drifted == [], (
-            f"{plugin_name} differs from src/common/sports_helpers.py in: {drifted}. "
-            f"Port the change to both, or stop treating it as shared.")
-
-    @pytest.mark.parametrize("sport", SCOREBOARDS)
-    def test_constants_match(self, sport):
-        root = _plugins_root()
-        defs = _definitions(ast.parse(
-            (root / f"{sport}-scoreboard" / "sports.py").read_text(encoding="utf-8")))
-        assert ast.literal_eval(defs["module"]["_MIN_WINDOW_DAYS"].value) == MIN_WINDOW_DAYS
-        assert ast.literal_eval(defs["module"]["_MAX_WINDOW_DAYS"].value) == MAX_WINDOW_DAYS
-        gap = defs["SportsCore"]["_DWELL_REENTRY_GAP_SECONDS"].value
-        assert math.isclose(ast.literal_eval(gap), SportsHelpersMixin._DWELL_REENTRY_GAP_SECONDS)

web_interface/app.py

@@ -1047,8 +1047,36 @@ def check_health_monitor():
             _reconciliation_started = True
             _threading.Thread(target=_run_startup_reconciliation, daemon=True).start()
 
+_auto_updater = None
+
+
+def start_auto_update_scheduler():
+    """Start the weekly auto-update thread (web_interface/auto_update.py).
+
+    Called by the launchers, not at import: tests and dev tools import this
+    module, and none of them should ever git pull. Idle unless
+    auto_update.enabled is set, so it is safe to always start.
+    """
+    global _auto_updater
+    if _auto_updater is not None:
+        return _auto_updater
+    from web_interface.auto_update import AutoUpdater
+    from web_interface.blueprints.api_v3.system import perform_core_update
+    _auto_updater = AutoUpdater(
+        config_manager=config_manager,
+        core_update=perform_core_update,
+        store_manager=plugin_store_manager,
+        plugin_manager=plugin_manager,
+        schema_manager=schema_manager,
+        operation_history=operation_history,
+    )
+    _auto_updater.start()
+    return _auto_updater
+
+
 if __name__ == '__main__':
     import os as _os
+    start_auto_update_scheduler()
     # threaded=True is Flask's default since 1.0 but stated explicitly so that
     # long-lived /api/v3/stream/* SSE connections don't starve other requests.
     # Debug mode is off by default; opt in with FLASK_DEBUG=1 in the environment.

web_interface/auto_update.py

@@ -0,0 +1,700 @@
+"""Weekly automatic updates: LEDMatrix code, then installed plugins.
+
+Turned on by ``auto_update.enabled`` in config.json (General tab, or the
+installer's --enable-auto-update). Off by default: an update pulls new code and
+restarts the display, which nobody should get without asking for it.
+
+Nothing here is allowed to leave a device broken without saying so:
+
+* **Checks first.** The code update is skipped, and the reason reported, when
+  the checkout has local edits or commits, a rebase or merge is in progress,
+  the branch has no upstream, disk is low, the newest commit was already
+  rolled back once, or the health check is not set up.
+* **Verified, and rolled back.** The pull itself is the Overview "Update Code"
+  path (``perform_core_update``). Restarting and checking the result is handed
+  to ledmatrix-update-verify.service (scripts/utils/auto_update_verify.py),
+  started through ledmatrix-update-verify.path by a request file, which
+  survives the web service restart, confirms both services come up and stay
+  up, and otherwise resets to the previous commit and its dependencies.
+* **Set up without SSH.** Those units are installed by the installer, or by the
+  display service when the toggle is switched on (src/auto_update_setup.py).
+* **Plugins after the code.** Plugins use the Plugin Store's own update, which
+  refuses versions this core cannot run and restores the old copy when an
+  install fails. When the code changed they wait until it passed its check,
+  so they are never gated against code that is about to be rolled back.
+* **Loud failures.** Anything other than success is written to the state file,
+  shown under the toggle and raised as a banner on the Overview tab.
+
+The schedule survives restarts through data/auto_update_state.json (gitignored).
+"""
+import importlib.util
+import json
+import logging
+import os
+import shutil
+import subprocess  # nosec B404 - list-form argv only, no shell  # nosemgrep
+import tempfile
+import threading
+import time
+from datetime import datetime
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+STATE_REL = Path('data') / 'auto_update_state.json'
+PENDING_REL = Path('data') / 'auto_update_pending.json'
+REQUEST_REL = Path('data') / 'auto_update_verify.request'
+SETUP_RESULT_REL = Path('data') / 'auto_update_setup.json'
+VERIFIER_COPY_REL = Path('data') / 'auto_update_verifier.py'
+VERIFIER_SOURCE = PROJECT_ROOT / 'scripts' / 'utils' / 'auto_update_verify.py'
+STATE_FILE = PROJECT_ROOT / STATE_REL
+PENDING_FILE = PROJECT_ROOT / PENDING_REL
+SETUP_RESULT_FILE = PROJECT_ROOT / SETUP_RESULT_REL
+
+VERIFY_UNIT = 'ledmatrix-update-verify'
+PATH_UNIT = f'{VERIFY_UNIT}.path'
+
+UPDATE_INTERVAL_SECONDS = 7 * 24 * 3600
+#: A failure that is probably transient (usually no network) is retried the
+#: next day rather than a week later.
+RETRY_AFTER_FAILURE_SECONDS = 24 * 3600
+#: Local hours an update prefers, since it restarts the display...
+QUIET_HOURS = range(2, 5)
+#: ...but a device that is never powered at night must still get updates.
+QUIET_HOURS_GRACE_SECONDS = 24 * 3600
+CHECK_INTERVAL_SECONDS = 30 * 60
+#: Let the web service settle after boot (and never pull in a crash loop).
+STARTUP_DELAY_SECONDS = 10 * 60
+MIN_FREE_BYTES = 300 * 1024 * 1024
+#: How long the health check gets to pick up a request before the update is undone.
+HANDOFF_TIMEOUT_SECONDS = 90
+#: The health check takes a few minutes at most; far past that, it is lost.
+VERIFY_LOST_SECONDS = 45 * 60
+
+#: Core outcomes that need the user's attention.
+ERROR_OUTCOMES = frozenset({'error', 'blocked', 'rolled_back', 'rollback_failed', 'lost'})
+
+HELPER_MISSING_MESSAGE = (
+    "LEDMatrix code updates are paused until the update health check is set up. "
+    "The display service sets it up when it starts while automatic updates are on, "
+    "so restart the display from the Overview tab. If this keeps happening, the "
+    "setup result under Automatic Updates on the General tab says why. Plugin "
+    "updates still run."
+)
+
+_STATE_LOCK = threading.RLock()
+
+
+def is_enabled(config):
+    return bool((config.get('auto_update') or {}).get('enabled', False))
+
+
+def _read_json(path):
+    try:
+        with open(path, 'r', encoding='utf-8') as f:
+            data = json.load(f)
+        return data if isinstance(data, dict) else None
+    except (OSError, ValueError):
+        return None
+
+
+def _write_json(path, data):
+    path = Path(path)
+    tmp = None
+    try:
+        path.parent.mkdir(parents=True, exist_ok=True)
+        fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f'.{path.stem}_')
+        with os.fdopen(fd, 'w', encoding='utf-8') as f:
+            json.dump(data, f, indent=2)
+        os.replace(tmp, path)
+        tmp = None
+    finally:
+        if tmp and os.path.exists(tmp):
+            try:
+                os.unlink(tmp)
+            except OSError:
+                pass
+
+
+def _unlink(path):
+    try:
+        Path(path).unlink()
+    except OSError:
+        pass
+
+
+def load_state(state_file=STATE_FILE):
+    return _read_json(state_file) or {}
+
+
+def save_state(state, state_file=STATE_FILE):
+    with _STATE_LOCK:
+        try:
+            _write_json(state_file, state)
+        except OSError as e:
+            logger.warning("Could not save auto-update state: %s", e)
+
+
+def dismiss_alert(alert_id, state_file=STATE_FILE):
+    with _STATE_LOCK:
+        state = load_state(state_file)
+        state['alert_dismissed'] = str(alert_id)
+        save_state(state, state_file)
+
+
+def _zone(tz_name):
+    try:
+        from zoneinfo import ZoneInfo
+        return ZoneInfo(tz_name) if tz_name else None
+    except Exception:
+        return None
+
+
+def _local_datetime(ts, tz_name):
+    return datetime.fromtimestamp(ts, _zone(tz_name))
+
+
+def _short(sha):
+    return (sha or 'unknown')[:7]
+
+
+def is_due(now, next_due, local_hour):
+    """Due once ``next_due`` has passed, in quiet hours or after the grace period."""
+    if next_due is None or now < next_due:
+        return False
+    return local_hour in QUIET_HOURS or now >= next_due + QUIET_HOURS_GRACE_SECONDS
+
+
+def _plugin_fingerprint(store_manager, plugin_dir):
+    """(manifest version, git sha): what changing tells us an update landed."""
+    version = None
+    try:
+        with open(plugin_dir / 'manifest.json', 'r', encoding='utf-8') as f:
+            manifest = json.load(f)
+        if manifest.get('local_only'):
+            return None
+        version = manifest.get('version')
+    except (OSError, ValueError):
+        pass
+    sha = None
+    try:
+        git_info = store_manager._get_local_git_info(plugin_dir)
+        sha = git_info.get('sha') if git_info else None
+    except Exception:
+        logger.debug("git info unavailable for %s", plugin_dir, exc_info=True)
+    return (version, sha)
+
+
+def update_plugins(store_manager, operation_history=None):
+    """Update every installed plugin that has an update. Returns (updated, failed)."""
+    updated, failed = [], []
+    plugins_dir = Path(store_manager.plugins_dir)
+    for plugin_id in sorted(store_manager.list_installed_plugins()):
+        plugin_dir = plugins_dir / plugin_id
+        before = _plugin_fingerprint(store_manager, plugin_dir)
+        if before is None:
+            continue  # local_only: managed by hand, never from the registry
+        try:
+            ok = store_manager.update_plugin(plugin_id)
+        except Exception:
+            logger.exception("Automatic update of plugin %s raised", plugin_id)
+            ok = False
+        if not ok:
+            failed.append(plugin_id)
+            status = 'failed'
+        elif _plugin_fingerprint(store_manager, plugin_dir) != before:
+            updated.append(plugin_id)
+            status = 'success'
+        else:
+            continue  # already current; not worth a history entry every week
+        if operation_history:
+            try:
+                operation_history.record_operation(
+                    'update', plugin_id=plugin_id, status=status,
+                    details={'automatic': True})
+            except Exception:
+                logger.debug("Could not record auto-update history", exc_info=True)
+    return updated, failed
+
+
+def _service_active(unit):
+    try:
+        result = subprocess.run(['systemctl', 'is-active', unit],
+                                capture_output=True, text=True, timeout=5)
+        return result.stdout.strip() == 'active'
+    except (subprocess.SubprocessError, OSError):
+        return False
+
+
+def helper_ready():
+    """True when the health check can be triggered (its path unit is watching)."""
+    return _service_active(PATH_UNIT)
+
+
+def restart_service(unit):
+    """Restart a unit only if it is running, via the sudoers-allowed command."""
+    if not _service_active(unit):
+        return False
+    try:
+        result = subprocess.run(['sudo', '-n', 'systemctl', 'restart', f'{unit}.service'],
+                                capture_output=True, text=True, timeout=30)
+    except (subprocess.SubprocessError, OSError) as e:
+        logger.warning("Auto-update could not restart %s: %s", unit, e)
+        return False
+    if result.returncode != 0:
+        logger.warning("Auto-update restart of %s failed: %s", unit, result.stderr.strip())
+    return result.returncode == 0
+
+
+def start_setup_if_needed(was_enabled, config):
+    """After settings are saved: switching updates on finishes setting them up.
+
+    The health check units are installed by the display service at startup
+    (src/auto_update_setup.py), so switching the toggle on restarts it.
+    Returns a note for the save message, or None.
+    """
+    if was_enabled or not is_enabled(config) or helper_ready():
+        return None
+    if not _service_active('ledmatrix'):
+        return 'Automatic update setup finishes the next time the display service starts.'
+    if restart_service('ledmatrix'):
+        return 'Finishing automatic update setup: the display is restarting.'
+    return 'Could not restart the display to finish automatic update setup; restart it from the Overview tab.'
+
+
+def _load_verifier(path):
+    spec = importlib.util.spec_from_file_location('ledmatrix_auto_update_verifier', str(path))
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+class AutoUpdater:
+    """Decides when an automatic update is due and runs it."""
+
+    def __init__(self, config_manager, core_update, store_manager=None,
+                 plugin_manager=None, schema_manager=None, operation_history=None,
+                 project_root=PROJECT_ROOT, state_file=None, clock=time.time,
+                 restart=restart_service, run=subprocess.run,
+                 service_active=_service_active, helper_ready=helper_ready,
+                 verifier_source=VERIFIER_SOURCE, disk_free=None, sleep=time.sleep):
+        self.config_manager = config_manager
+        self.core_update = core_update
+        self.store_manager = store_manager
+        self.plugin_manager = plugin_manager
+        self.schema_manager = schema_manager
+        self.operation_history = operation_history
+        self.project_root = Path(project_root)
+        self.state_file = Path(state_file) if state_file else self.project_root / STATE_REL
+        self.pending_file = self.project_root / PENDING_REL
+        self.request_file = self.project_root / REQUEST_REL
+        self.verifier_copy = self.project_root / VERIFIER_COPY_REL
+        self.verifier_source = Path(verifier_source)
+        self.clock = clock
+        self.restart = restart
+        self.run_command = run
+        self.service_active = service_active
+        self.helper_ready = helper_ready
+        self.disk_free = disk_free or (lambda path: shutil.disk_usage(path).free)
+        self.sleep = sleep
+        self._stop = threading.Event()
+        self._thread = None
+
+    # -- scheduling ---------------------------------------------------------
+
+    def tick(self):
+        """Do whatever is due. Returns True if an update ran."""
+        now = self.clock()
+        state = load_state(self.state_file)
+        # Before the enabled check: a verification started before the user
+        # switched updates off still has to be reported.
+        if self._finalize_verification(state, now) == 'waiting':
+            return False
+
+        try:
+            config = self.config_manager.load_config()
+        except Exception:
+            logger.debug("Auto-update could not load config", exc_info=True)
+            return False
+        if not is_enabled(config):
+            return False
+
+        if state.get('plugins_pending'):
+            self._run_deferred_plugins(state, now)
+            return True
+
+        if state.get('next_due') is None:
+            # Just switched on: due now, which in practice means the next
+            # quiet-hours window rather than the moment Save was clicked.
+            state['next_due'] = now
+            save_state(state, self.state_file)
+
+        local_hour = _local_datetime(now, config.get('timezone')).hour
+        if not is_due(now, state['next_due'], local_hour):
+            return False
+        self.run(state)
+        return True
+
+    def run(self, state=None):
+        state = load_state(self.state_file) if state is None else state
+        logger.info("Automatic update starting")
+        try:
+            core = self.update_core(state)
+        except Exception as e:
+            logger.exception("Automatic core update raised")
+            core = {'outcome': 'error', 'message': f'The LEDMatrix update failed unexpectedly: {e}'}
+
+        deferred = core['outcome'] == 'verifying'
+        updated, failed = ([], []) if deferred else self._update_plugins()
+        self._store_run(state, core, updated, failed)
+        logger.info("Automatic update: core %s (%s); plugins updated=%s failed=%s%s",
+                    core['outcome'], core['message'], updated, failed,
+                    '; plugins wait for the health check' if deferred else '')
+
+        # Code restarts belong to the health check; this only covers plugins.
+        if updated:
+            self.restart('ledmatrix')
+        return state['last_result']
+
+    def _store_run(self, state, core, updated, failed):
+        now = self.clock()
+        deferred = core['outcome'] == 'verifying'
+        state.update({
+            'last_run': now,
+            'next_due': now + (RETRY_AFTER_FAILURE_SECONDS if core['outcome'] == 'error'
+                               else UPDATE_INTERVAL_SECONDS),
+            'plugins_pending': deferred,
+            'last_result': {
+                'core_outcome': core['outcome'],
+                'core_message': core['message'],
+                'plugins_updated': updated,
+                'plugins_failed': failed,
+                'plugins_deferred': deferred,
+            },
+        })
+        self._apply_result(state, now)
+        save_state(state, self.state_file)
+
+    # -- core update --------------------------------------------------------
+
+    def _git(self, *args, timeout=60):
+        return self.run_command(['git', *args], cwd=str(self.project_root),
+                                capture_output=True, text=True, timeout=timeout)
+
+    def _count(self, rev_range):
+        out = self._git('rev-list', '--count', rev_range).stdout.strip()
+        return int(out) if out.isdigit() else 0
+
+    def preflight(self, state):
+        """Decide whether the code update may run.
+
+        Returns ``(outcome, message, info)`` with outcome ``ready``,
+        ``up_to_date``, ``blocked`` (needs the user) or ``error`` (retry soon).
+        """
+        if not self.helper_ready():
+            return 'blocked', HELPER_MISSING_MESSAGE, {}
+
+        git_dir = self._git('rev-parse', '--git-dir')
+        if git_dir.returncode != 0:
+            return 'blocked', 'LEDMatrix is not installed as a git checkout, so it cannot update itself.', {}
+        git_dir = Path(git_dir.stdout.strip())
+        if not git_dir.is_absolute():
+            git_dir = self.project_root / git_dir
+        rebase_dirs = [git_dir / name for name in ('rebase-merge', 'rebase-apply') if (git_dir / name).exists()]
+        if rebase_dirs:
+            # A rebase really in progress leaves HEAD detached. With HEAD back
+            # on a branch it was abandoned -- typically a pull that stopped on
+            # a conflict, then a checkout -- and it makes every later pull
+            # fail. It holds no work (local edits are checked separately
+            # below), so clear it rather than strand a device nobody SSHes into.
+            if self._git('symbolic-ref', '-q', 'HEAD').returncode != 0:
+                return 'blocked', ('A git rebase is in progress in the LEDMatrix folder. '
+                                   'Finish or abort it; automatic updates will not touch it.'), {}
+            cleared = self._git('rebase', '--quit')
+            if cleared.returncode != 0 or any(path.exists() for path in rebase_dirs):
+                return 'blocked', ('An abandoned git rebase in the LEDMatrix folder could not be cleared: '
+                                   f'{(cleared.stderr or "").strip() or "unknown error"}.'), {}
+            logger.warning("Cleared an abandoned git rebase from %s", ', '.join(str(p) for p in rebase_dirs))
+        if (git_dir / 'MERGE_HEAD').exists():
+            return 'blocked', ('A git merge is in progress in the LEDMatrix folder. '
+                               'Finish or abort it; automatic updates will not touch it.'), {}
+
+        if self._git('rev-parse', '--abbrev-ref', '--symbolic-full-name', '@{u}').returncode != 0:
+            return 'blocked', ('The current branch has no upstream to update from (or HEAD is detached). '
+                               'Use Update Code once, or Tools -> Switch branch.'), {}
+
+        # Mode-only changes are ignored: older installers chmod tracked
+        # scripts, and --autostash in the pull carries those across. Plugin
+        # folders are separate installs, as in perform_core_update.
+        status = self._git('-c', 'core.fileMode=false', 'status', '--porcelain', '--untracked-files=no')
+        changed = [line[3:] for line in status.stdout.splitlines()
+                   if line.strip() and 'plugins/' not in line and 'plugin-repos/' not in line]
+        if status.returncode != 0 or changed:
+            example = f' (for example {changed[0]})' if changed else ''
+            return 'blocked', (f'{len(changed) or "Some"} tracked file(s) in the LEDMatrix folder were edited '
+                               f'locally{example}. Automatic updates will not stash your changes; '
+                               'commit or revert them, or update manually with Update Code.'), {}
+
+        free = self.disk_free(str(self.project_root))
+        if free < MIN_FREE_BYTES:
+            return 'blocked', (f'Only {free // (1024 * 1024)} MB of disk space is free; an update '
+                               f'needs at least {MIN_FREE_BYTES // (1024 * 1024)} MB.'), {}
+
+        fetch = self._git('fetch', '--quiet', timeout=120)
+        if fetch.returncode != 0:
+            detail = next((ln.strip() for ln in (fetch.stderr or '').splitlines() if ln.strip()), '')
+            return 'error', f'Could not check for LEDMatrix updates: {detail or "git fetch failed"}.', {}
+
+        ahead = self._count('@{u}..HEAD')
+        if ahead:
+            return 'blocked', (f'This checkout has {ahead} local commit(s) that are not upstream. '
+                               'Automatic updates will not rebase them; update manually with Update Code.'), {}
+        if not self._count('HEAD..@{u}'):
+            return 'up_to_date', 'LEDMatrix is already up to date.', {}
+
+        upstream = self._git('rev-parse', '@{u}').stdout.strip()
+        if upstream and upstream == state.get('rolled_back_head'):
+            return 'up_to_date', (f'The newest LEDMatrix version ({_short(upstream)}) failed its health '
+                                  'check and was rolled back before; waiting for a newer one.'), {}
+        head = self._git('rev-parse', 'HEAD').stdout.strip()
+        return 'ready', '', {'old_head': head, 'upstream_head': upstream}
+
+    def update_core(self, state):
+        try:
+            outcome, message, info = self.preflight(state)
+        except (subprocess.SubprocessError, OSError) as e:
+            return {'outcome': 'error', 'message': f'Pre-update checks failed: {e}.'}
+        if outcome != 'ready':
+            return {'outcome': outcome, 'message': message}
+        old_head = info['old_head']
+
+        display_was_active = self.service_active('ledmatrix')
+        try:
+            # Copied before pulling, so the checker is the known-good version.
+            self.verifier_copy.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(self.verifier_source, self.verifier_copy)
+            verifier = _load_verifier(self.verifier_copy)
+        except Exception as e:
+            return {'outcome': 'error', 'message': f'Could not prepare the update health check: {e}.'}
+
+        try:
+            core = self.core_update()
+        except Exception as e:
+            logger.exception("perform_core_update raised")
+            core = {'status': 'error', 'message': f'Update failed: {e}'}
+        new_head = self._git('rev-parse', 'HEAD').stdout.strip()
+        pending = {
+            'status': 'pending',
+            'old_head': old_head,
+            'new_head': new_head,
+            'display_was_active': display_was_active,
+            'dependency_failures': list(core.get('dependency_failures') or []),
+            'created_at': self.clock(),
+        }
+
+        def rollback():
+            return verifier.Verifier(self.project_root, run=self.run_command).rollback(pending)
+
+        if core.get('status') != 'success':
+            message = core.get('message') or 'The LEDMatrix update failed.'
+            if new_head and new_head != old_head:
+                ok, detail = rollback()
+                message += (' The partial update was rolled back.' if ok
+                            else f' Rolling back the partial update also failed: {detail}.')
+                if not ok:
+                    return {'outcome': 'rollback_failed', 'message': message}
+            return {'outcome': 'error', 'message': message}
+        if new_head == old_head:
+            return {'outcome': 'up_to_date', 'message': core.get('message') or 'LEDMatrix is already up to date.'}
+
+        handoff = {'outcome': 'verifying',
+                   'message': (f'Updated LEDMatrix from {_short(old_head)} to {_short(new_head)}; '
+                               'restarting and checking the services.')}
+        # Recorded before the handoff: the health check restarts this process.
+        self._store_run(state, handoff, [], [])
+        try:
+            _write_json(self.pending_file, pending)
+            self.request_file.write_text(new_head, encoding='utf-8')
+            picked_up = self._wait_for_pickup()
+        except OSError as e:
+            logger.error("Could not request the update health check: %s", e)
+            picked_up = False
+        if picked_up:
+            return handoff
+
+        # No health check means no update: undo it while the old code is
+        # still the code that is running.
+        logger.error("%s did not start the health check; rolling the update back", PATH_UNIT)
+        _unlink(self.request_file)
+        _unlink(self.pending_file)
+        ok, detail = rollback()
+        if not ok:
+            return {'outcome': 'rollback_failed',
+                    'message': (f'The update health check did not start, and rolling back to '
+                                f'{_short(old_head)} failed: {detail}.')}
+        return {'outcome': 'blocked',
+                'message': ('The update health check did not start, so the update was undone. '
+                            f'Check "systemctl status {PATH_UNIT}".'
+                            + (f' Note: {detail}.' if detail else ''))}
+
+    def _wait_for_pickup(self):
+        for _ in range(HANDOFF_TIMEOUT_SECONDS):
+            pending = _read_json(self.pending_file)
+            if pending and pending.get('status') != 'pending':
+                return True
+            self.sleep(1)
+        pending = _read_json(self.pending_file)
+        return bool(pending and pending.get('status') != 'pending')
+
+    def _finalize_verification(self, state, now):
+        """Fold the health check's outcome into the state. None, 'waiting' or 'done'."""
+        pending = _read_json(self.pending_file)
+        if pending is None:
+            return None
+        status = pending.get('status')
+        old, new = pending.get('old_head'), pending.get('new_head')
+        reason, detail = pending.get('reason'), pending.get('detail')
+
+        if status in ('pending', 'verifying'):
+            if now - float(pending.get('created_at') or now) < VERIFY_LOST_SECONDS:
+                return 'waiting'
+            outcome = 'lost'
+            message = (f'LEDMatrix was updated to {_short(new)}, but its health check never reported back, '
+                       'so it is not known whether the device is healthy. '
+                       f'See "journalctl -u {VERIFY_UNIT}".')
+        elif status == 'success':
+            outcome = 'updated'
+            message = (f'Updated LEDMatrix from {_short(old)} to {_short(new)}; '
+                       'the services restarted and stayed healthy.')
+        elif status == 'rolled_back':
+            outcome = 'rolled_back'
+            message = (f'The LEDMatrix update to {_short(new)} was rolled back to {_short(old)} because '
+                       f'{reason or "it failed its health check"}.' + (f' Note: {detail}.' if detail else ''))
+            state['rolled_back_head'] = new
+        else:
+            outcome = 'rollback_failed'
+            message = (f'The LEDMatrix update to {_short(new)} failed ({reason or "unknown reason"}) and '  # nosec B608 - user-facing message, not SQL  # nosemgrep
+                       f'could not be rolled back: {detail or "unknown error"}. The device may need '
+                       f'attention: run "git reset --hard {old}" in the LEDMatrix folder, then restart '
+                       'the display and web services.')
+
+        result = state.setdefault('last_result', {})
+        result.update({'core_outcome': outcome, 'core_message': message})
+        if outcome not in ('updated', 'rolled_back'):
+            # The device is in an unknown state; don't pile plugin changes on it.
+            state['plugins_pending'] = False
+            result['plugins_deferred'] = False
+        self._apply_result(state, now)
+        save_state(state, self.state_file)
+        _unlink(self.pending_file)
+        (logger.info if outcome == 'updated' else logger.error)("Automatic update: %s", message)
+        return 'done'
+
+    # -- plugins and reporting ----------------------------------------------
+
+    def _update_plugins(self):
+        if not self.store_manager:
+            return [], []
+        updated, failed = update_plugins(self.store_manager, self.operation_history)
+        for plugin_id in updated:
+            if self.schema_manager:
+                self.schema_manager.invalidate_cache(plugin_id)
+        if updated and self.plugin_manager:
+            try:
+                self.plugin_manager.discover_plugins()
+            except Exception:
+                logger.debug("discover_plugins after auto-update failed", exc_info=True)
+        return updated, failed
+
+    def _run_deferred_plugins(self, state, now):
+        state['plugins_pending'] = False
+        updated, failed = self._update_plugins()
+        result = state.setdefault('last_result', {})
+        result.update({'plugins_updated': updated, 'plugins_failed': failed, 'plugins_deferred': False})
+        self._apply_result(state, now)
+        save_state(state, self.state_file)
+        if updated:
+            self.restart('ledmatrix')
+
+    @staticmethod
+    def _apply_result(state, now):
+        result = state.get('last_result') or {}
+        outcome = result.get('core_outcome')
+        failed = result.get('plugins_failed') or []
+        if outcome in ERROR_OUTCOMES or failed:
+            result['status'] = 'error'
+        elif outcome == 'verifying':
+            result['status'] = 'pending'
+        else:
+            result['status'] = 'success'
+
+        if result['status'] == 'error':
+            parts = []
+            if outcome in ERROR_OUTCOMES:
+                parts.append(result.get('core_message') or 'The LEDMatrix update failed.')
+            if failed:
+                parts.append('These plugins failed to update: ' + ', '.join(failed) + '.')
+            state['alert'] = {'id': str(int(now * 1000)), 'message': ' '.join(parts)}
+        elif result['status'] == 'success':
+            state.pop('alert', None)
+
+    # -- thread -------------------------------------------------------------
+
+    def start(self):
+        if self._thread and self._thread.is_alive():
+            return
+        self._thread = threading.Thread(target=self._loop, name='auto-update', daemon=True)
+        self._thread.start()
+
+    def stop(self):
+        self._stop.set()
+
+    def _loop(self):
+        if self._stop.wait(STARTUP_DELAY_SECONDS):
+            return
+        while True:
+            try:
+                self.tick()
+            except Exception:
+                logger.exception("Automatic update check failed")
+            if self._stop.wait(CHECK_INTERVAL_SECONDS):
+                return
+
+
+def describe_status(config, state=None, state_file=STATE_FILE, pending_file=PENDING_FILE,
+                    setup_file=SETUP_RESULT_FILE, helper=None):
+    """What the General tab and the Overview banner show."""
+    state = load_state(state_file) if state is None else state
+    tz = config.get('timezone')
+    fmt = '%b %d, %Y %I:%M %p'
+    pending = _read_json(pending_file) or {}
+    setup = _read_json(setup_file) or {}
+    enabled = is_enabled(config)
+    out = {
+        'last_run': None, 'summary': None, 'status': None, 'next_due': None,
+        'alert': None, 'alert_id': None,
+        # Only asked while enabled: it runs systemctl, on every page load.
+        'verifier_installed': (helper or helper_ready)() if enabled else None,
+        'setup_status': setup.get('status'),
+        'setup_message': setup.get('message'),
+        'verifying': pending.get('status') in ('pending', 'verifying'),
+    }
+    if state.get('last_run'):
+        out['last_run'] = _local_datetime(state['last_run'], tz).strftime(fmt)
+        result = state.get('last_result') or {}
+        out['status'] = result.get('status')
+        parts = [result.get('core_message') or 'No result recorded.']
+        if result.get('plugins_updated'):
+            parts.append('Plugins updated: ' + ', '.join(result['plugins_updated']) + '.')
+        if result.get('plugins_failed'):
+            parts.append('Plugins that failed to update: ' + ', '.join(result['plugins_failed']) + '.')
+        if result.get('plugins_deferred') and state.get('plugins_pending'):
+            parts.append('Plugin updates run once the code update passes its health check.')
+        out['summary'] = ' '.join(parts)
+    if enabled and state.get('next_due'):
+        out['next_due'] = _local_datetime(state['next_due'], tz).strftime(fmt)
+    alert = state.get('alert')
+    if alert and alert.get('id') != state.get('alert_dismissed'):
+        out['alert'], out['alert_id'] = alert.get('message'), alert.get('id')
+    return out

web_interface/blueprints/api_v3/config.py

@@ -454,16 +454,21 @@ def save_main_config():
 
         # Merge with existing config (similar to original implementation)
         current_config = api_v3.config_manager.load_config()
+        was_auto_update_enabled = bool((current_config.get('auto_update') or {}).get('enabled'))
 
         # Handle general settings
         # Note: Checkboxes don't send data when unchecked, so we need to check if we're updating general settings
         # If any general setting is present, we're updating the general tab
         is_general_update = any(k in data for k in ['timezone', 'city', 'state', 'country', 'web_display_autostart',
-                                                     'auto_discover', 'auto_load_enabled', 'development_mode', 'plugins_directory'])
+                                                     'auto_discover', 'auto_load_enabled', 'development_mode', 'plugins_directory',
+                                                     'auto_update_enabled'])
 
         if is_general_update:
             # For checkbox: if not present in data during general update, it means unchecked
             current_config['web_display_autostart'] = _coerce_to_bool(data.get('web_display_autostart'))
+            if not isinstance(current_config.get('auto_update'), dict):
+                current_config['auto_update'] = {}
+            current_config['auto_update']['enabled'] = _coerce_to_bool(data.get('auto_update_enabled'))
 
         if 'timezone' in data:
             current_config['timezone'] = data['timezone']
@@ -1033,7 +1038,7 @@ def save_main_config():
             if key in ['timezone', 'city', 'state', 'country',
                        'web_display_autostart', 'auto_discover',
                        'auto_load_enabled', 'development_mode',
-                       'plugins_directory', 'target_fps']:
+                       'plugins_directory', 'target_fps', 'auto_update_enabled']:
                 continue
             # Skip fields that are already handled above in their own named sections.
             # Without this, every form field name lands as a top-level config key too.
@@ -1068,7 +1073,17 @@ def save_main_config():
         except ImportError:
             pass
 
-        return success_response(message='Configuration saved successfully')
+        message = 'Configuration saved successfully'
+        # Switching automatic updates on finishes their setup, which needs
+        # the display service to restart (web_interface/auto_update.py).
+        try:
+            from web_interface import auto_update
+            note = auto_update.start_setup_if_needed(was_auto_update_enabled, current_config)
+            if note:
+                message = f'{message}. {note}'
+        except Exception:
+            logger.warning("Automatic update setup could not be started", exc_info=True)
+        return success_response(message=message)
     except Exception as e:
         logger.error("Error saving config", exc_info=True)
         return error_response(

web_interface/blueprints/api_v3/system.py

@@ -12,6 +12,8 @@ from web_interface.blueprints.api_v3 import (
     get_git_version, jsonify, logger, os, request, resolve_pull_command,
     shutil, subprocess,
 )
+import threading
+
 import web_interface.blueprints.api_v3 as _pkg
 # Read through the module rather than bound by value: tests patch these
 # as module attributes, and a value binding would not see the patch.
@@ -120,6 +122,32 @@ def get_system_version():
     except Exception as e:
         logger.error("get_system_version failed: %s", e, exc_info=True)
         return jsonify({'status': 'error', 'message': 'Unable to retrieve version'}), 500
+@api_v3.route('/system/auto-update', methods=['GET'])
+def get_auto_update_status():
+    """Weekly automatic update status: last result, next check, and any alert.
+
+    No local except: a failure falls through to the app-wide handler in
+    web_interface/app.py, which logs the traceback and returns the redacted
+    detail -- the response every route gives, without a second copy here.
+    """
+    from web_interface import auto_update
+    config = api_v3.config_manager.load_config() if api_v3.config_manager else {}
+    return jsonify({'status': 'success', 'data': auto_update.describe_status(config)})
+
+
+@api_v3.route('/system/auto-update/dismiss', methods=['POST'])
+def dismiss_auto_update_alert():
+    """Hide the current automatic-update banner until a new alert replaces it."""
+    from web_interface import auto_update
+    payload = request.get_json(silent=True)
+    # A JSON array or scalar is a bad request, not a 500.
+    alert_id = str(payload.get('alert_id') or '').strip() if isinstance(payload, dict) else ''
+    if not alert_id:
+        return jsonify({'status': 'error', 'message': 'alert_id required'}), 400
+    auto_update.dismiss_alert(alert_id)
+    return jsonify({'status': 'success'})
+
+
 @api_v3.route('/system/check-update', methods=['GET'])
 def check_for_update():
     """Check whether a newer LEDMatrix commit is available on origin/main."""
@@ -198,6 +226,219 @@ def _sudo_hint_for(text):
     return None
 
 
+_core_update_lock = threading.Lock()
+
+
+def perform_core_update():
+    """Pull the latest LEDMatrix code and sync its dependencies.
+
+    Shared by the Overview "Update Code" button and the weekly automatic
+    updater (web_interface/auto_update.py), so both take exactly the same
+    path. Returns the JSON-able payload the button has always received:
+    ``status``, ``message`` and ``restart_required``.
+    """
+    # The button and the scheduler can fire together; two pulls racing
+    # over one checkout (and one stash) is how local changes get lost.
+    if not _core_update_lock.acquire(blocking=False):
+        return {'status': 'error', 'restart_required': False,
+                'message': 'An update is already in progress; try again shortly.'}
+    try:
+        return _perform_core_update_locked()
+    finally:
+        _core_update_lock.release()
+
+
+def _perform_core_update_locked():
+    project_dir = str(PROJECT_ROOT)
+
+    # Decide how to pull BEFORE stashing. If this checkout cannot be
+    # updated at all, stashing first would put the user's local changes
+    # away for an update that was never going to run.
+    pull_args, upstream_note, pull_error = resolve_pull_command(project_dir)
+    if pull_error:
+        logger.warning("git pull not attempted: %s", pull_error)
+        return {'status': 'error', 'message': pull_error, 'restart_required': False}
+
+    # Check if there are local changes that need to be stashed
+    # Exclude plugins directory - plugins are separate repos and shouldn't be stashed with base project
+    # Use --untracked-files=no to skip untracked files check (much faster with symlinked plugins)
+    try:
+        status_result = subprocess.run(
+            ['git', 'status', '--porcelain', '--untracked-files=no'],
+            capture_output=True,
+            text=True,
+            timeout=30,
+            cwd=project_dir
+        )
+        # Filter out any changes in plugins directory - plugins are separate repositories
+        # Git status format: XY filename (where X is status of index, Y is status of work tree)
+        status_lines = [line for line in status_result.stdout.strip().split('\n')
+                       if line.strip() and 'plugins/' not in line]
+        has_changes = bool('\n'.join(status_lines).strip())
+    except subprocess.TimeoutExpired:
+        # If status check times out, assume there might be changes and proceed
+        # This is safer than failing the update
+        has_changes = True
+        status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
+
+    stash_info = ""
+
+    # Stash local changes if they exist (excluding plugins)
+    # Plugins are separate repositories and shouldn't be stashed with base project updates
+    if has_changes:
+        try:
+            # Use pathspec to exclude plugins directory from stash
+            stash_result = subprocess.run(
+                ['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--', ':!plugins'],
+                capture_output=True,
+                text=True,
+                timeout=30,
+                cwd=project_dir
+            )
+            if stash_result.returncode == 0:
+                logger.debug("git stash: stashed local changes before pull")
+                stash_info = " Local changes were stashed."
+            else:
+                logger.warning("git stash failed before pull (returncode=%d)", stash_result.returncode)
+        except subprocess.TimeoutExpired:
+            logger.warning("git stash timed out, proceeding with pull")
+
+    # Record HEAD before the pull so dependency changes can be detected
+    old_head = None
+    try:
+        _pre = subprocess.run(['git', 'rev-parse', 'HEAD'],
+                              capture_output=True, text=True, timeout=10, cwd=project_dir)
+        if _pre.returncode == 0:
+            old_head = _pre.stdout.strip()
+    except subprocess.TimeoutExpired:
+        logger.warning("git rev-parse timed out before pull")
+
+    # Whether the pull actually brought new code in. "Already up to
+    # date" is a success too, and prompting for a restart then would
+    # train users to ignore the prompt.
+    code_changed = False
+    # Requirement files whose install failed. The automatic updater refuses
+    # to restart onto code whose dependencies did not install.
+    dependency_failures = []
+
+    # Perform the git pull. Branches without an upstream were given
+    # an explicit "origin <branch>" above so the update still works.
+    result = subprocess.run(
+        pull_args,
+        capture_output=True,
+        text=True,
+        timeout=60,
+        cwd=project_dir
+    )
+
+    # Give the branch tracking information so the next pull is a plain
+    # `git pull` — otherwise every update repeats the fallback.
+    if result.returncode == 0 and upstream_note:
+        branch = _git_current_branch(project_dir)
+        if branch:
+            try:
+                subprocess.run(
+                    ['git', 'branch', f'--set-upstream-to=origin/{branch}', branch],
+                    capture_output=True, text=True, timeout=10, cwd=project_dir)
+            except (subprocess.TimeoutExpired, OSError) as exc:
+                logger.debug("could not set upstream for %s: %s", branch, exc)
+
+    # Return custom response for git_pull
+    if result.returncode == 0:
+        pull_message = "Code updated successfully."
+        if has_changes:
+            pull_message = f"Code updated successfully. Local changes were automatically stashed.{stash_info}"
+        if result.stdout and "Already up to date" not in result.stdout:
+            pull_message = f"Code updated successfully.{stash_info}"
+        if upstream_note:
+            pull_message = f"{pull_message} {upstream_note}"
+
+        # Keep Python dependencies in sync automatically: if the pull
+        # changed a requirements file, install it now — users updating
+        # from the web UI (most of them) never SSH in to pip install.
+        # Installs go through the same root-visible path as the
+        # Tools-tab buttons (_pip_install_requirements).
+        dep_notes = []
+        try:
+            _post = subprocess.run(['git', 'rev-parse', 'HEAD'],
+                                   capture_output=True, text=True, timeout=10, cwd=project_dir)
+            new_head = _post.stdout.strip() if _post.returncode == 0 else None
+            if old_head and new_head and old_head != new_head:
+                code_changed = True
+                diff = subprocess.run(
+                    ['git', 'diff', '--name-only', f'{old_head}..{new_head}'],
+                    capture_output=True, text=True, timeout=15, cwd=project_dir)
+                changed = set(diff.stdout.split()) if diff.returncode == 0 else set()
+                for rel in ('requirements.txt', 'web_interface/requirements.txt'):
+                    req_path = PROJECT_ROOT / rel
+                    if rel not in changed or not req_path.exists():
+                        continue
+                    # Each file's install is isolated: a timeout or
+                    # OSError (e.g. the sudo wrapper/interpreter
+                    # missing) on one file must not abort the other.
+                    try:
+                        r = _pip_install_requirements(req_path, timeout=180)
+                        if r.returncode == 0:
+                            dep_notes.append(f"Dependencies from {rel} updated.")
+                        else:
+                            dependency_failures.append(rel)
+                            dep_notes.append(
+                                f"Dependency install from {rel} failed — "
+                                "run Install Base Requirements from the Tools tab.")
+                            logger.warning("post-update pip install failed for %s: %s",
+                                           rel, _truncate_output(r.stdout, r.stderr))
+                    except subprocess.TimeoutExpired:
+                        dependency_failures.append(rel)
+                        dep_notes.append(
+                            f"Dependency install from {rel} timed out — "
+                            "run Install Base Requirements from the Tools tab.")
+                        logger.warning("post-update pip install timed out for %s", rel)
+                    except OSError as install_err:
+                        dependency_failures.append(rel)
+                        dep_notes.append(
+                            f"Dependency install from {rel} failed — "
+                            "run Install Base Requirements from the Tools tab.")
+                        logger.warning("post-update pip install errored for %s: %s",
+                                       rel, install_err)
+        except subprocess.TimeoutExpired:
+            logger.warning("post-update dependency sync timed out")
+        if dep_notes:
+            pull_message += " " + " ".join(dep_notes)
+        # A `git pull` restores built-in plugins (committed under
+        # plugin-repos/) even if the user uninstalled them. Re-remove
+        # any the user previously uninstalled so the update doesn't
+        # resurrect them.
+        if api_v3.plugin_store_manager:
+            try:
+                purged = api_v3.plugin_store_manager.purge_uninstalled_plugins()
+                if purged:
+                    logger.info(
+                        "Re-removed %d uninstalled plugin(s) restored by update: %s",
+                        len(purged), ", ".join(purged),
+                    )
+            except (OSError, RuntimeError) as purge_err:
+                logger.warning("Post-update plugin purge failed: %s", purge_err)
+    else:
+        logger.warning("git pull failed (returncode=%d): %s", result.returncode, result.stderr)
+        # Show git's own first line: "check logs" leaves the user with
+        # nothing to act on, and these failures are usually actionable
+        # (conflicting local commits, no upstream, network).
+        detail = next((ln.strip() for ln in (result.stderr or '').splitlines()
+                       if ln.strip()), '')
+        pull_message = f"Update failed: {detail}" if detail else "Update failed; check logs for details"
+
+    # Nothing here restarts anything: the pull replaces files on
+    # disk while the display and web services keep running the code
+    # they loaded at boot. Without this the user is told the update
+    # succeeded and sees no change until they happen to reboot.
+    return {
+        'status': 'success' if result.returncode == 0 else 'error',
+        'message': pull_message,
+        'restart_required': bool(result.returncode == 0 and code_changed),
+        'dependency_failures': dependency_failures,
+    }
+
+
 @api_v3.route('/system/action', methods=['POST'])
 def execute_system_action():
     """Execute system actions (start/stop/reboot/etc)"""
@@ -265,188 +506,7 @@ def execute_system_action():
             result = subprocess.run(['sudo', 'poweroff'],
                                  capture_output=True, text=True, timeout=10)
         elif action == 'git_pull':
-            # Use PROJECT_ROOT instead of hardcoded path
+            return jsonify(perform_core_update())
-            project_dir = str(PROJECT_ROOT)
-
-            # Decide how to pull BEFORE stashing. If this checkout cannot be
-            # updated at all, stashing first would put the user's local changes
-            # away for an update that was never going to run.
-            pull_args, upstream_note, pull_error = resolve_pull_command(project_dir)
-            if pull_error:
-                logger.warning("git pull not attempted: %s", pull_error)
-                return jsonify({'status': 'error', 'message': pull_error})
-
-            # Check if there are local changes that need to be stashed
-            # Exclude plugins directory - plugins are separate repos and shouldn't be stashed with base project
-            # Use --untracked-files=no to skip untracked files check (much faster with symlinked plugins)
-            try:
-                status_result = subprocess.run(
-                    ['git', 'status', '--porcelain', '--untracked-files=no'],
-                    capture_output=True,
-                    text=True,
-                    timeout=30,
-                    cwd=project_dir
-                )
-                # Filter out any changes in plugins directory - plugins are separate repositories
-                # Git status format: XY filename (where X is status of index, Y is status of work tree)
-                status_lines = [line for line in status_result.stdout.strip().split('\n')
-                               if line.strip() and 'plugins/' not in line]
-                has_changes = bool('\n'.join(status_lines).strip())
-            except subprocess.TimeoutExpired:
-                # If status check times out, assume there might be changes and proceed
-                # This is safer than failing the update
-                has_changes = True
-                status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
-
-            stash_info = ""
-
-            # Stash local changes if they exist (excluding plugins)
-            # Plugins are separate repositories and shouldn't be stashed with base project updates
-            if has_changes:
-                try:
-                    # Use pathspec to exclude plugins directory from stash
-                    stash_result = subprocess.run(
-                        ['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--', ':!plugins'],
-                        capture_output=True,
-                        text=True,
-                        timeout=30,
-                        cwd=project_dir
-                    )
-                    if stash_result.returncode == 0:
-                        logger.debug("git stash: stashed local changes before pull")
-                        stash_info = " Local changes were stashed."
-                    else:
-                        logger.warning("git stash failed before pull (returncode=%d)", stash_result.returncode)
-                except subprocess.TimeoutExpired:
-                    logger.warning("git stash timed out, proceeding with pull")
-
-            # Record HEAD before the pull so dependency changes can be detected
-            old_head = None
-            try:
-                _pre = subprocess.run(['git', 'rev-parse', 'HEAD'],
-                                      capture_output=True, text=True, timeout=10, cwd=project_dir)
-                if _pre.returncode == 0:
-                    old_head = _pre.stdout.strip()
-            except subprocess.TimeoutExpired:
-                logger.warning("git rev-parse timed out before pull")
-
-            # Whether the pull actually brought new code in. "Already up to
-            # date" is a success too, and prompting for a restart then would
-            # train users to ignore the prompt.
-            code_changed = False
-
-            # Perform the git pull. Branches without an upstream were given
-            # an explicit "origin <branch>" above so the update still works.
-            result = subprocess.run(
-                pull_args,
-                capture_output=True,
-                text=True,
-                timeout=60,
-                cwd=project_dir
-            )
-
-            # Give the branch tracking information so the next pull is a plain
-            # `git pull` — otherwise every update repeats the fallback.
-            if result.returncode == 0 and upstream_note:
-                branch = _git_current_branch(project_dir)
-                if branch:
-                    try:
-                        subprocess.run(
-                            ['git', 'branch', f'--set-upstream-to=origin/{branch}', branch],
-                            capture_output=True, text=True, timeout=10, cwd=project_dir)
-                    except (subprocess.TimeoutExpired, OSError) as exc:
-                        logger.debug("could not set upstream for %s: %s", branch, exc)
-
-            # Return custom response for git_pull
-            if result.returncode == 0:
-                pull_message = "Code updated successfully."
-                if has_changes:
-                    pull_message = f"Code updated successfully. Local changes were automatically stashed.{stash_info}"
-                if result.stdout and "Already up to date" not in result.stdout:
-                    pull_message = f"Code updated successfully.{stash_info}"
-                if upstream_note:
-                    pull_message = f"{pull_message} {upstream_note}"
-
-                # Keep Python dependencies in sync automatically: if the pull
-                # changed a requirements file, install it now — users updating
-                # from the web UI (most of them) never SSH in to pip install.
-                # Installs go through the same root-visible path as the
-                # Tools-tab buttons (_pip_install_requirements).
-                dep_notes = []
-                try:
-                    _post = subprocess.run(['git', 'rev-parse', 'HEAD'],
-                                           capture_output=True, text=True, timeout=10, cwd=project_dir)
-                    new_head = _post.stdout.strip() if _post.returncode == 0 else None
-                    if old_head and new_head and old_head != new_head:
-                        code_changed = True
-                        diff = subprocess.run(
-                            ['git', 'diff', '--name-only', f'{old_head}..{new_head}'],
-                            capture_output=True, text=True, timeout=15, cwd=project_dir)
-                        changed = set(diff.stdout.split()) if diff.returncode == 0 else set()
-                        for rel in ('requirements.txt', 'web_interface/requirements.txt'):
-                            req_path = PROJECT_ROOT / rel
-                            if rel not in changed or not req_path.exists():
-                                continue
-                            # Each file's install is isolated: a timeout or
-                            # OSError (e.g. the sudo wrapper/interpreter
-                            # missing) on one file must not abort the other.
-                            try:
-                                r = _pip_install_requirements(req_path, timeout=180)
-                                if r.returncode == 0:
-                                    dep_notes.append(f"Dependencies from {rel} updated.")
-                                else:
-                                    dep_notes.append(
-                                        f"Dependency install from {rel} failed — "
-                                        "run Install Base Requirements from the Tools tab.")
-                                    logger.warning("post-update pip install failed for %s: %s",
-                                                   rel, _truncate_output(r.stdout, r.stderr))
-                            except subprocess.TimeoutExpired:
-                                dep_notes.append(
-                                    f"Dependency install from {rel} timed out — "
-                                    "run Install Base Requirements from the Tools tab.")
-                                logger.warning("post-update pip install timed out for %s", rel)
-                            except OSError as install_err:
-                                dep_notes.append(
-                                    f"Dependency install from {rel} failed — "
-                                    "run Install Base Requirements from the Tools tab.")
-                                logger.warning("post-update pip install errored for %s: %s",
-                                               rel, install_err)
-                except subprocess.TimeoutExpired:
-                    logger.warning("post-update dependency sync timed out")
-                if dep_notes:
-                    pull_message += " " + " ".join(dep_notes)
-                # A `git pull` restores built-in plugins (committed under
-                # plugin-repos/) even if the user uninstalled them. Re-remove
-                # any the user previously uninstalled so the update doesn't
-                # resurrect them.
-                if api_v3.plugin_store_manager:
-                    try:
-                        purged = api_v3.plugin_store_manager.purge_uninstalled_plugins()
-                        if purged:
-                            logger.info(
-                                "Re-removed %d uninstalled plugin(s) restored by update: %s",
-                                len(purged), ", ".join(purged),
-                            )
-                    except (OSError, RuntimeError) as purge_err:
-                        logger.warning("Post-update plugin purge failed: %s", purge_err)
-            else:
-                logger.warning("git pull failed (returncode=%d): %s", result.returncode, result.stderr)
-                # Show git's own first line: "check logs" leaves the user with
-                # nothing to act on, and these failures are usually actionable
-                # (conflicting local commits, no upstream, network).
-                detail = next((ln.strip() for ln in (result.stderr or '').splitlines()
-                               if ln.strip()), '')
-                pull_message = f"Update failed: {detail}" if detail else "Update failed; check logs for details"
-
-            # Nothing here restarts anything: the pull replaces files on
-            # disk while the display and web services keep running the code
-            # they loaded at boot. Without this the user is told the update
-            # succeeded and sees no change until they happen to reboot.
-            return jsonify({
-                'status': 'success' if result.returncode == 0 else 'error',
-                'message': pull_message,
-                'restart_required': bool(result.returncode == 0 and code_changed),
-            })
         elif action == 'checkout_branch':
             # Switch branches from the Tools tab. Needed because a checkout
             # that predates tracking (or a restored backup) can leave the pi

web_interface/blueprints/pages_v3.py

@@ -525,8 +525,15 @@ def _load_general_partial():
     try:
         if pages_v3.config_manager:
             main_config = pages_v3.config_manager.load_config()
+            try:
+                from web_interface.auto_update import describe_status
+                auto_update_status = describe_status(main_config)
+            except Exception:
+                logger.debug("Could not read auto-update status", exc_info=True)
+                auto_update_status = None
             return render_template('v3/partials/general.html',
-                                 main_config=main_config)
+                                 main_config=main_config,
+                                 auto_update_status=auto_update_status)
     except Exception as e:
         logger.error("Error loading partial", exc_info=True)
         return "Error loading partial", 500

web_interface/start.py

@@ -100,8 +100,9 @@ def main():
     werkzeug_logger.error = log_exception_filtered
     
     # Import and run the Flask app
-    from web_interface.app import app
+    from web_interface.app import app, start_auto_update_scheduler
-    
+    start_auto_update_scheduler()
+
     print("Starting LED Matrix Web Interface V3...")
     print("Web server binding to: 0.0.0.0:5000")
     

web_interface/templates/v3/partials/general.html

@@ -44,6 +44,44 @@
             </label>
         </div>
 
+        <!-- Weekly Automatic Updates -->
+        <div class="form-group" id="setting-general-auto_update" data-setting-key="auto_update.enabled">
+            <label class="flex items-center">
+                <input type="checkbox"
+                       name="auto_update_enabled"
+                       value="true"
+                       {% if (main_config.auto_update or {}).enabled %}checked{% endif %}
+                       class="form-control h-4 w-4 text-blue-600 focus:ring-blue-500 border-gray-300 rounded">
+                <span class="ml-2 text-sm font-medium text-gray-900">Automatically check for and install updates once a week</span>
+                {{ ui.help_tip('Once a week, update LEDMatrix and every installed plugin that has a newer version.\nSkipped (and reported) if you have local changes, low disk space, or the rollback service is missing. After updating, the services are restarted and checked; if they do not stay healthy, the update is rolled back automatically.\nRuns overnight (2-5 AM in your timezone) when possible. Problems show as a banner on Overview. A stopped display is not started. Default: off.', 'Automatic Updates') }}
+            </label>
+            {% if auto_update_status and (main_config.auto_update or {}).enabled and auto_update_status.verifier_installed is sameas false %}
+            <p class="mt-1 ml-6 text-xs text-amber-700">
+                {% if auto_update_status.setup_status == 'failed' %}
+                Setting up the update health check failed: {{ auto_update_status.setup_message }}
+                LEDMatrix code updates are paused until it is fixed; plugin updates still run.
+                {% else %}
+                Setting up the update health check. The display service does this when it starts, so it is
+                ready shortly after saving (or once the display is started). LEDMatrix code updates begin after that.
+                {% endif %}
+            </p>
+            {% endif %}
+            {% if auto_update_status and auto_update_status.verifying %}
+            <p class="mt-1 ml-6 text-xs text-blue-700">An update was just installed and its health check is running.</p>
+            {% endif %}
+            {% if auto_update_status and (auto_update_status.last_run or auto_update_status.next_due) %}
+            <div class="mt-1 ml-6 text-xs text-gray-500 space-y-1">
+                {% if auto_update_status.last_run %}
+                <p>Last automatic update: {{ auto_update_status.last_run }}
+                   <span class="{{ 'text-red-600' if auto_update_status.status == 'error' else '' }}">— {{ auto_update_status.summary }}</span></p>
+                {% endif %}
+                {% if auto_update_status.next_due %}
+                <p>Next check: {{ auto_update_status.next_due }} or the first overnight window after it</p>
+                {% endif %}
+            </div>
+            {% endif %}
+        </div>
+
         <!-- Timezone -->
         <div class="form-group" id="setting-general-timezone" data-setting-key="timezone">
             <label for="timezone" class="block text-sm font-medium text-gray-700">Timezone{{ ui.help_tip('Time zone used for clocks, schedules, and time-based content.\nChoose the zone where the display physically lives so on/off schedules fire at the correct local time.', 'Timezone') }}</label>

web_interface/templates/v3/partials/overview.html

@@ -1,3 +1,44 @@
+<!-- Automatic update banner: a weekly update that failed, was blocked or was rolled back -->
+<div id="auto-update-banner" class="bg-red-50 border border-red-300 rounded-lg p-4 mb-4 flex items-start" style="display:none !important" role="alert">
+    <div class="flex-shrink-0 mr-3 mt-0.5">
+        <i class="fas fa-exclamation-circle text-red-500"></i>
+    </div>
+    <div class="flex-1">
+        <p class="text-sm font-medium text-red-800">Automatic Update Needs Attention</p>
+        <p class="text-sm text-red-700 mt-1" id="auto-update-banner-text"></p>
+        <p class="text-xs text-red-600 mt-1">Details are under Automatic Updates on the General tab.</p>
+    </div>
+    <button type="button" onclick="window.dismissAutoUpdateBanner()" class="ml-4 flex-shrink-0 text-red-500 hover:text-red-600" aria-label="Dismiss">
+        <i class="fas fa-times"></i>
+    </button>
+</div>
+<script>
+(function () {
+    fetch('/api/v3/system/auto-update')
+        .then(function (r) { return r.json(); })
+        .then(function (resp) {
+            var d = resp.data || {};
+            if (!d.alert) return;
+            document.getElementById('auto-update-banner-text').textContent = d.alert;
+            var banner = document.getElementById('auto-update-banner');
+            banner.dataset.alertId = d.alert_id || '';
+            banner.style.setProperty('display', 'flex', 'important');
+        })
+        .catch(function () {});
+
+    // Dismissal is stored server-side, so the banner stays gone on every
+    // device until a new failure raises a new alert.
+    window.dismissAutoUpdateBanner = function () {
+        var banner = document.getElementById('auto-update-banner');
+        banner.style.setProperty('display', 'none', 'important');
+        fetch('/api/v3/system/auto-update/dismiss', {
+            method: 'POST',
+            headers: {'Content-Type': 'application/json'},
+            body: JSON.stringify({alert_id: banner.dataset.alertId})
+        }).catch(function () {});
+    };
+}());
+</script>
 <!-- Reconciliation warning banner: shown when startup reconciliation found stale plugin config entries -->
 <div id="reconciliation-banner" class="bg-yellow-50 border border-yellow-300 rounded-lg p-4 mb-4 flex items-start" style="display:none !important" role="alert">
     <div class="flex-shrink-0 mr-3 mt-0.5">