feat(install): updates refresh systemd units; new installs run the newest release (#729)

Updates that move HEAD now install changed systemd units through a root-owned helper (/usr/local/sbin/ledmatrix-refresh-units, two literal sudo lines), with a backup restored on rollback; a refresh that fails part-way puts the old units back. Devices without the new sudo rule keep updating and are told to re-run the installer once. The one-shot installer now checks out the newest vX.Y.Z release (LEDMATRIX_CHANNEL=beta keeps main) and never moves an existing checkout backwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitattributes

@@ -10,3 +10,6 @@
 # Generated by scripts/build_css.py; collapsed in diffs, not hand-edited.
 web_interface/static/v3/tailwind.css linguist-generated=true
 web_interface/static/v3/plugin-frame.css linguist-generated=true
+
+# Installed as an executable (its shebang runs it) by install_service.sh.
+scripts/install/ledmatrix_refresh_units.py text eol=lf

CHANGELOG.md

@@ -19,6 +19,49 @@ accepts both, but the store flags the old spelling as deprecated
 
 ## Unreleased
 
+### Updates refresh the systemd units; new installs run the newest release
+
+- **Updates now install changed systemd units.** An update (Update Code, or
+  the weekly automatic update) moved the checkout's `systemd/*.service`
+  templates but never the units systemd runs, so settings added after a
+  device was installed -- #687's render-loop watchdog, for one -- only ever
+  arrived with a reinstall. After an update that moves HEAD, the web
+  interface compares the installed `ledmatrix.service`,
+  `ledmatrix-web.service` and `ledmatrix-update-verify.{service,path}` with
+  the new templates (rendered exactly as `install_service.sh` does, comments
+  ignored as the startup drift warning does) and, when they differ, runs the
+  new root-owned helper `/usr/local/sbin/ledmatrix-refresh-units`
+  (`scripts/install/ledmatrix_refresh_units.py`) through sudo: it installs
+  the changed units and runs `systemctl daemon-reload`, so the restart that
+  follows the update runs under them. Update Code's message says so.
+- **Rollback restores them.** The helper keeps the units it replaced
+  (`/var/lib/ledmatrix/unit-backup`, root only); when the automatic update's
+  health check rolls an update back, it runs `ledmatrix-refresh-units
+  --restore` before restarting the services onto the old code.
+- **The sudo rule needs a reinstall.** `install_service.sh` installs the
+  helper and `lib_sudoers.sh` grants it with exactly two command lines (no
+  arguments, and `--restore`). A device installed before this has neither;
+  its updates keep working, log that the new unit settings need a reinstall
+  and say so in Update Code's message, the same remedy as the startup
+  "unit drift" warning. Re-run `sudo ./first_time_install.sh` once (or
+  `sudo ./scripts/install/install_service.sh` then
+  `./scripts/install/configure_web_sudo.sh`).
+- `install_service.sh` now leaves the units it installs mode `0644`, as
+  `first_time_install.sh` already did; run on its own it left them `0600`.
+- **New installs run the newest release.** The one-shot installer cloned
+  `main`'s tip, so a new device ran unreleased code until the next release.
+  It now checks out the newest `vX.Y.Z` tag after cloning (the same semver
+  rules as `web_interface/update_channel.py`), and that release's own
+  `first_time_install.sh` runs. `LEDMATRIX_CHANNEL=beta` installs `main`
+  instead and records the beta channel; `first_time_install.sh --beta` (or
+  `LEDMATRIX_CHANNEL=beta|stable`) records a channel for a manual install.
+- **Re-running the one-shot never moves backwards.** On an existing stable
+  checkout it moves to the newest release only when that release contains
+  the current commit; a checkout newer than every release keeps its
+  fast-forward pull (on a branch) or stays put (detached), and beta keeps the
+  pull it always had. It used to fast-forward a detached release checkout to
+  `main`'s tip.
+
 ### Control socket stage 3: the display's state over the socket
 
 - Two new commands, still protocol version 1. `state.get` returns a

docs/GETTING_STARTED.md

@@ -39,6 +39,17 @@ Raspberry Pi OS Lite yourself:
    [README Installation Steps / Quick Install](../README.md#installation-steps)
    for full details
 
+   The one-shot installer installs the newest release (the **stable** update
+   channel). To run the newest, unreleased code from `main` instead (the
+   **beta** channel), put `LEDMATRIX_CHANNEL=beta` in front of `bash`:
+   ```bash
+   curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/install/one-shot-install.sh | LEDMATRIX_CHANNEL=beta bash
+   ```
+   A manual clone starts on `main`; add `--beta` to `first_time_install.sh`
+   to stay on it, or leave it off and the first update after the next
+   release moves the device onto releases. You can switch channels later on
+   the General tab.
+
 **Expected Behavior after install:**
 - LED matrix will light up
 - A fresh install ships only the bundled `starlark-apps` and

docs/PERMISSIONS.md

@@ -33,6 +33,9 @@ in again (services pick them up on restart).
 | `/run/ledmatrix/control.sock` | `root` : cache directory's group (`ledmatrix`) | `660` | The display's control socket; only root and that group can connect. See [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md#security-model) |
 | `scripts/fix_perms/safe_plugin_rm.sh`, `safe_pip_install.sh` | `root:root` | `755` | Run as root through sudo, so the web user must not be able to edit them |
 | `/etc/sudoers.d/ledmatrix_web`, `ledmatrix_wifi` | `root` | `440` | |
+| `/usr/local/sbin/ledmatrix-refresh-units` | `root:root` | `755` | Copy of `scripts/install/ledmatrix_refresh_units.py`, installed by `install_service.sh`. Outside the project so the web user cannot edit what sudo runs |
+| `/var/lib/ledmatrix/unit-backup/` | `root` | `700` | The units the last refresh replaced, for the automatic update's rollback |
+| `/etc/systemd/system/ledmatrix*.service`, `.path` | `root:root` | `644` | Readable so the web interface can compare them with the templates after an update |
 
 What keeps it that way at runtime:
 
@@ -86,6 +89,20 @@ password:
 - `journalctl -u ledmatrix.service *`, `-u ledmatrix *`, `-t ledmatrix *`,
   tagged `NOEXEC`: journalctl opens a pager on a terminal, and a shell
   escape from that pager would be a root shell
+- `/usr/local/sbin/ledmatrix-refresh-units ""` and
+  `/usr/local/sbin/ledmatrix-refresh-units --restore` — exactly these two
+  command lines (`""` means "no arguments"). After an update the first
+  installs the systemd units whose templates changed and runs
+  `systemctl daemon-reload`; the automatic update's rollback runs the second
+  to put the previous units back. The helper takes nothing from the caller:
+  the project folder and the web user come from the installed, root-owned
+  `ledmatrix.service` and `ledmatrix-web.service`. It only replaces the four
+  units `install_service.sh` installs, only if they are already installed,
+  and refuses a template that would change a unit's `User=` (root for the
+  display, the web user for the rest) or `WorkingDirectory=`, or that is a
+  symlink, not a regular file, or over 64 KB. It grants nothing new: the
+  templates are files the web user can edit, but so is `run.py`, which the
+  display service already runs as root.
 
 ### `/etc/sudoers.d/ledmatrix_wifi`
 
@@ -136,7 +153,9 @@ directory.
 | `safe_plugin_rm.sh`, `safe_pip_install.sh` | — | Called by the web interface through sudo | Not for manual use |
 
 To reinstall the sudoers rules, run
-`./scripts/install/configure_web_sudo.sh` (web rules) or
+`./scripts/install/configure_web_sudo.sh` (web rules; the
+`ledmatrix-refresh-units` rules also need the helper itself, which
+`sudo ./scripts/install/install_service.sh` installs) or
 `./scripts/install/configure_wifi_permissions.sh` (WiFi rules and polkit) as
 the web user, not with `sudo`.
 

docs/TROUBLESHOOTING.md

@@ -328,6 +328,49 @@ commit, then switches to releases on its own.
 3. **Local changes after a channel switch:** edits that no longer fit the new
    version are kept in the git stash rather than lost; `git stash list`
    shows them as "LEDMatrix autostash before update".
+4. **A new install is on a release, not `main`.** The one-shot installer
+   checks out the newest release. For the newest code instead, install with
+   `LEDMATRIX_CHANNEL=beta`:
+   ```bash
+   curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/install/one-shot-install.sh | LEDMATRIX_CHANNEL=beta bash
+   ```
+
+---
+
+#### Issue: "service settings ... are not applied yet" after an update
+
+**Symptoms:**
+- Update Code's message, or the web interface log, says an update changes
+  service settings that are not applied yet, and to run the installer
+- The display logs `ledmatrix.service differs from systemd/ledmatrix.service`
+  at startup
+
+**Explanation:** updates install the systemd units a new version changes
+through the root helper `/usr/local/sbin/ledmatrix-refresh-units`, which the
+installer sets up and grants to the web user in
+`/etc/sudoers.d/ledmatrix_web`. A device installed before that has neither,
+so the new unit settings (for example the display's watchdog) wait for a
+reinstall. The update itself is fine.
+
+**Solution:** re-run the installer once, as root:
+```bash
+cd ~/LEDMatrix
+sudo ./first_time_install.sh
+# or, lighter: install the units and helper, then the sudo rules
+sudo ./scripts/install/install_service.sh
+./scripts/install/configure_web_sudo.sh
+```
+Check it worked:
+```bash
+ls -l /usr/local/sbin/ledmatrix-refresh-units   # root root, rwxr-xr-x
+sudo -l | grep ledmatrix-refresh-units           # the two rules
+```
+A message that the helper **refused** a unit (`refusing to install it`)
+means a template in `systemd/` was edited so that it would run as another
+account or from another folder. The message names the template. Look at
+what changed with `git diff -- systemd/`, save any edit you want to keep,
+then restore only that file, for example
+`git checkout -- systemd/ledmatrix-web.service`.
 
 ---
 
@@ -590,9 +633,10 @@ stack into the log, so it says which plugin was stuck.
    apart, so a plugin that hangs on every start does not restart the display
    hundreds of times an hour.
 
-4. **Is the watchdog installed?** Installs from before it keep their old unit
+4. **Is the watchdog installed?** Updates install new unit settings once the
-   until the installer is re-run (a startup warning says the unit differs
+   installer has set up `ledmatrix-refresh-units`; installs from before that
-   from its template):
+   keep their old unit until the installer is re-run (a startup warning says
+   the unit differs from its template):
    ```bash
    systemctl show -p WatchdogUSec ledmatrix   # 2min once running; 0 = not installed
    sudo ./scripts/install/install_service.sh

first_time_install.sh

@@ -223,6 +223,8 @@ SKIP_SWAP=${LEDMATRIX_SKIP_SWAP:-0}
 BUILD_JOBS_OVERRIDE=${LEDMATRIX_BUILD_JOBS:-}
 # Weekly automatic updates: 1 on, 0 off, empty = ask (interactive) or leave as is.
 AUTO_UPDATE=${LEDMATRIX_AUTO_UPDATE:-}
+# Update channel written to config.json: stable, beta, or empty = leave as is.
+UPDATE_CHANNEL=$(printf '%s' "${LEDMATRIX_CHANNEL:-}" | tr '[:upper:]' '[:lower:]')
 
 usage() {
     cat <<USAGE
@@ -240,12 +242,18 @@ Options:
       --enable-auto-update  Turn on weekly automatic updates (with health
                             check and automatic rollback)
       --no-auto-update      Leave weekly automatic updates off
+      --beta                Follow main, the newest code (the beta update
+                            channel). Without it, updates follow releases
+                            (stable). It sets the channel; it does not move
+                            this checkout -- the one-shot installer picks the
+                            version, and so does the next update.
   -h, --help                Show this help message and exit
 
 Environment variables (same effect as flags):
   LEDMATRIX_ASSUME_YES=1, RPI_RGB_FORCE_REBUILD=1, LEDMATRIX_SKIP_SOUND=1,
   LEDMATRIX_SKIP_PERF=1, LEDMATRIX_SKIP_REBOOT_PROMPT=1,
-  LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N, LEDMATRIX_AUTO_UPDATE=1|0
+  LEDMATRIX_SKIP_SWAP=1, LEDMATRIX_BUILD_JOBS=N, LEDMATRIX_AUTO_UPDATE=1|0,
+  LEDMATRIX_CHANNEL=stable|beta
 
 Low-memory devices:
   On a Pi with under 2GB of RAM the C++ build is limited to fewer parallel
@@ -265,6 +273,7 @@ while [ $# -gt 0 ]; do
         --skip-swap) SKIP_SWAP=1 ;;
         --enable-auto-update) AUTO_UPDATE=1 ;;
         --no-auto-update) AUTO_UPDATE=0 ;;
+        --beta) UPDATE_CHANNEL=beta ;;
         --build-jobs)
             shift
             if [ $# -eq 0 ]; then echo "--build-jobs requires a number"; usage; exit 1; fi
@@ -873,15 +882,25 @@ if [ -z "$AUTO_UPDATE" ] && [ "$ASSUME_YES" != "1" ] && [ -t 0 ]; then
     echo
     if [[ $REPLY =~ ^[Yy]$ ]]; then AUTO_UPDATE=1; else AUTO_UPDATE=0; fi
 fi
-if [ "$AUTO_UPDATE" = "1" ] || [ "$AUTO_UPDATE" = "0" ]; then
+case "$UPDATE_CHANNEL" in
-    if python3 - "$PROJECT_ROOT_DIR/config/config.json" "$AUTO_UPDATE" <<'PY'
+    stable|beta|"") ;;
+    *) echo "⚠ LEDMATRIX_CHANNEL=$UPDATE_CHANNEL is not stable or beta; leaving the update channel as it is"
+       UPDATE_CHANNEL="" ;;
+esac
+# The update channel, likewise only when asked for (--beta / LEDMATRIX_CHANNEL).
+if [ "$AUTO_UPDATE" = "1" ] || [ "$AUTO_UPDATE" = "0" ] || [ -n "$UPDATE_CHANNEL" ]; then
+    if python3 - "$PROJECT_ROOT_DIR/config/config.json" "$AUTO_UPDATE" "$UPDATE_CHANNEL" <<'PY'
 import json, os, sys, tempfile
-path, enabled = sys.argv[1], sys.argv[2] == "1"
+path, enabled = sys.argv[1], sys.argv[2]
+channel = sys.argv[3] if len(sys.argv) > 3 else ""
 with open(path, encoding="utf-8") as f:
     config = json.load(f)
 if not isinstance(config.get("auto_update"), dict):
     config["auto_update"] = {}
-config["auto_update"]["enabled"] = enabled
+if enabled in ("0", "1"):
+    config["auto_update"]["enabled"] = enabled == "1"
+if channel:
+    config["auto_update"]["channel"] = channel
 # Written beside the original and swapped in whole: the display service's
 # config watcher may be running and must never read a half-written file.
 original = os.stat(path)
@@ -902,9 +921,11 @@ except BaseException:
     raise
 PY
     then
-        if [ "$AUTO_UPDATE" = "1" ]; then echo "✓ Weekly automatic updates enabled"; else echo "✓ Weekly automatic updates off"; fi
+        if [ "$AUTO_UPDATE" = "1" ]; then echo "✓ Weekly automatic updates enabled"
+        elif [ "$AUTO_UPDATE" = "0" ]; then echo "✓ Weekly automatic updates off"; fi
+        if [ -n "$UPDATE_CHANNEL" ]; then echo "✓ Update channel: $UPDATE_CHANNEL"; fi
     else
-        echo "⚠ Could not set auto_update in config/config.json; turn it on from the General tab instead"
+        echo "⚠ Could not set auto_update in config/config.json; set it from the General tab instead"
     fi
 fi
 

scripts/install/README.md

@@ -5,11 +5,18 @@ This directory contains scripts for installing and configuring the LEDMatrix sys
 ## Scripts
 
 - **`one-shot-install.sh`** - Single-command installer; clones the
-  repo, checks prerequisites, then runs `first_time_install.sh`.
+  repo, checks out the newest release (or `main` with
-  Invoked via `curl ... | bash` from the project root README.
+  `LEDMATRIX_CHANNEL=beta`), checks prerequisites, then runs
+  `first_time_install.sh`. Invoked via `curl ... | bash` from the project
+  root README. Re-running it never moves a checkout to an older version.
 - **`install_service.sh`** - Installs, enables and starts the display
   service (`ledmatrix.service`), the web interface service
-  (`ledmatrix-web.service`) and the update-verify units (systemd)
+  (`ledmatrix-web.service`) and the update-verify units (systemd), and
+  installs `/usr/local/sbin/ledmatrix-refresh-units`
+- **`ledmatrix_refresh_units.py`** - Not run from here: `install_service.sh`
+  installs a root-owned copy as `/usr/local/sbin/ledmatrix-refresh-units`,
+  which updates run through sudo to install changed units (and the
+  automatic update's rollback, with `--restore`, to put them back)
 - **`install_web_service.sh`** - Installs only the web interface service
   and the update-verify units (systemd)
 - **`install_wifi_monitor.sh`** - Installs the WiFi monitor daemon service

scripts/install/configure_web_sudo.sh

@@ -138,6 +138,8 @@ echo "- View system logs via journalctl"
 echo "- Reboot and shutdown the system"
 echo "- Remove plugin directories (for update/uninstall when root-owned files block deletion)"
 echo "- Install plugin/base requirements.txt as root (so ledmatrix.service can see them)"
+echo "- Install the LEDMatrix systemd units an update changed, and restore them on rollback"
+echo "  (/usr/local/sbin/ledmatrix-refresh-units, installed by install_service.sh)"
 echo ""
 
 # Ask for confirmation

scripts/install/install_service.sh

@@ -143,6 +143,30 @@ for VERIFY_UNIT in ledmatrix-update-verify.service ledmatrix-update-verify.path;
     fi
 done
 
+# The helper updates run (through sudo, see lib_sudoers.sh) to install these
+# same units when a new version changes their templates, and to put the old
+# ones back if the automatic update rolls back. Root-owned and outside the
+# checkout, so the web user who owns the checkout cannot change what sudo runs.
+# Not fatal: without it, updates leave the units for the next reinstall.
+REFRESH_UNITS_SRC="$PROJECT_ROOT_DIR/scripts/install/ledmatrix_refresh_units.py"
+REFRESH_UNITS_DEST=/usr/local/sbin/ledmatrix-refresh-units
+if [ -f "$REFRESH_UNITS_SRC" ]; then
+    if sudo install -D -o root -g root -m 0755 "$REFRESH_UNITS_SRC" "$REFRESH_UNITS_DEST"; then
+        echo "Installed $REFRESH_UNITS_DEST (lets updates refresh these units)"
+    else
+        echo "WARNING: could not install $REFRESH_UNITS_DEST; updates will not refresh the systemd units" >&2
+    fi
+fi
+# The units above are copied from mktemp files, which are 0600. 0644 is what
+# first_time_install.sh (Step 8.1) sets, and lets the web interface compare
+# them with the templates after an update without root.
+for INSTALLED_UNIT in ledmatrix.service ledmatrix-web.service \
+        ledmatrix-update-verify.service ledmatrix-update-verify.path; do
+    if [ -f "/etc/systemd/system/$INSTALLED_UNIT" ]; then
+        sudo chmod 644 "/etc/systemd/system/$INSTALLED_UNIT" || true
+    fi
+done
+
 echo "Reloading systemd daemon for web service..."
 sudo systemctl daemon-reload
 

scripts/install/ledmatrix_refresh_units.py

@@ -0,0 +1,451 @@
+#!/usr/bin/python3 -I
+"""Refresh the installed LEDMatrix systemd units from the checkout's templates.
+
+Installed by scripts/install/install_service.sh as a root-owned copy,
+/usr/local/sbin/ledmatrix-refresh-units, and granted to the web interface's
+user by /etc/sudoers.d/ledmatrix_web (scripts/install/lib_sudoers.sh) with
+exactly two command lines:
+
+    ledmatrix-refresh-units              (no arguments)
+    ledmatrix-refresh-units --restore
+
+An update (Update Code, or the weekly automatic update) pulls new unit
+templates into systemd/, but the units systemd runs are the copies in
+/etc/systemd/system, which only the installer used to write. So a setting
+added to a template -- the render-loop watchdog, a memory limit -- never
+reached a device that was already installed. After an update the web
+interface runs this, and the next restart picks the new units up.
+
+* **No arguments:** render each installed unit from systemd/<unit> exactly as
+  install_service.sh does (__PROJECT_ROOT_DIR__ and __USER__ replaced
+  literally), and install the ones whose content differs (comments and blank
+  lines aside, as src/startup_validator.py compares them), then
+  ``systemctl daemon-reload``. The units replaced are saved first, so the
+  automatic update's rollback can put them back.
+* ``--restore``: put back the units the last refresh replaced, and
+  daemon-reload. Nothing saved means nothing to do.
+* ``--check``: print the units that would change, one per line. Needs no
+  root and changes nothing.
+
+What it trusts, and why. It takes no other input: the project directory and
+the web interface's user come from the installed, root-owned
+ledmatrix.service and ledmatrix-web.service, not from the caller, and sudo
+strips the caller's environment (``-I`` ignores the PYTHON* variables too).
+It only replaces units that are already installed, only the four
+install_service.sh installs, and only with a rendering that keeps each unit's
+User= (root for the display, the web user for the others) and
+WorkingDirectory=. The templates are files the web user can edit -- but so is
+run.py, which ledmatrix.service already runs as root, so a template grants
+nothing that user did not have; the checks keep a damaged or hostile template
+from changing who a unit runs as, and keep this from reading anything but a
+regular file under the checkout's systemd/ folder.
+
+Standard library only, and no imports from the checkout: the installed copy
+must not run code the web user can change.
+"""
+import json
+import os
+import re
+import stat
+import subprocess  # nosec B404 - fixed argv, no shell  # nosemgrep
+import sys
+import tempfile
+
+SYSTEMD_DIR = '/etc/systemd/system'
+#: Root-only: the units the last refresh replaced, for --restore.
+BACKUP_DIR = '/var/lib/ledmatrix/unit-backup'
+MANIFEST = 'manifest.json'
+INSTALLED_PATH = '/usr/local/sbin/ledmatrix-refresh-units'
+
+DISPLAY_UNIT = 'ledmatrix.service'
+WEB_UNIT = 'ledmatrix-web.service'
+VERIFY_SERVICE = 'ledmatrix-update-verify.service'
+VERIFY_PATH = 'ledmatrix-update-verify.path'
+#: What install_service.sh installs, in its order. Nothing else is touched.
+UNITS = (DISPLAY_UNIT, WEB_UNIT, VERIFY_SERVICE, VERIFY_PATH)
+
+MAX_TEMPLATE_BYTES = 64 * 1024
+_USER_RE = re.compile(r'^[a-z_][a-z0-9_-]{0,31}$')
+#: systemd expands % specifiers, and a quote, backslash or line break would
+#: be reinterpreted in a unit file (src/auto_update_setup.py refuses the same).
+#: (On Windows, where the tests also run, a backslash is the path separator.)
+_UNSAFE_PATH_CHARS = set('%"') | ({'\\'} if os.sep == '/' else set())
+
+EXIT_OK = 0
+EXIT_FAILED = 1
+EXIT_USAGE = 2
+
+
+class RefreshError(Exception):
+    """Why the units were left alone, in words for the web interface's log."""
+
+
+class UnitsUnreadable(RefreshError):
+    """An installed unit is not readable by this (unprivileged) user.
+
+    install_service.sh used to leave units mode 0600 (first_time_install.sh's
+    Step 8.1 makes them 0644), so ``--check`` as the web user cannot always
+    tell; the root helper itself can.
+    """
+
+
+def directive_values(text, key):
+    """Every value of ``key=`` in a unit's text, in order (systemd allows spaces around ``=``)."""
+    return [m.group(1).strip() for m in re.finditer(rf'^[ \t]*{key}[ \t]*=(.*)$', text or '', re.M)]
+
+
+def layout_problem(text, section, keys):
+    """What would make ``directive_values`` misread the unit as systemd reads it, or None.
+
+    A ``User=`` inside a backslash-continued line is part of the line before,
+    and one under [Unit] is ignored, so either could pass a check that systemd
+    then does not apply. Neither appears in the shipped templates.
+    """
+    current = None
+    for raw in (text or '').splitlines():
+        line = raw.strip()
+        if not line or line.startswith(('#', ';')):
+            continue
+        if line.endswith('\\'):
+            return 'continues a line with a backslash'
+        if line.startswith('[') and line.endswith(']'):
+            current = line[1:-1]
+            continue
+        key = line.split('=', 1)[0].strip()
+        if key in keys and current != section:
+            return f'sets {key}= outside [{section}]'
+    return None
+
+
+def unit_body(text):
+    """A unit's meaningful lines in order: no comments, no blank lines.
+
+    The same comparison src/startup_validator.py uses for its drift warning,
+    so what this refreshes is exactly what that warns about.
+    """
+    lines = []
+    for line in (text or '').splitlines():
+        line = line.strip()
+        if line and not line.startswith('#'):
+            lines.append(line)
+    return '\n'.join(lines)
+
+
+def render(template, project_root, user):
+    """install_service.sh's ``sed "s|__PROJECT_ROOT_DIR__|...|g; s|__USER__|...|g"``."""
+    return template.replace('__PROJECT_ROOT_DIR__', project_root).replace('__USER__', user)
+
+
+def _read_regular(path, limit=MAX_TEMPLATE_BYTES, dir_fd=None):
+    """A regular file's text, never through a symlink, a FIFO or a device."""
+    flags = os.O_RDONLY | getattr(os, 'O_NOFOLLOW', 0) | getattr(os, 'O_NONBLOCK', 0)
+    kwargs = {'dir_fd': dir_fd} if dir_fd is not None else {}
+    fd = os.open(path, flags, **kwargs)
+    try:
+        info = os.fstat(fd)
+        if not stat.S_ISREG(info.st_mode):
+            raise RefreshError(f'{path} is not a regular file')
+        if info.st_size > limit:
+            raise RefreshError(f'{path} is larger than {limit} bytes')
+        data = b''
+        while True:
+            chunk = os.read(fd, limit + 1 - len(data))
+            if not chunk:
+                break
+            data += chunk
+            if len(data) > limit:
+                raise RefreshError(f'{path} is larger than {limit} bytes')
+    finally:
+        os.close(fd)
+    if b'\0' in data:
+        raise RefreshError(f'{path} is not a text file')
+    try:
+        return data.decode('utf-8')
+    except UnicodeDecodeError as e:
+        raise RefreshError(f'{path} is not UTF-8') from e
+
+
+def _read_installed(systemd_dir, name):
+    path = os.path.join(systemd_dir, name)
+    try:
+        with open(path, 'r', encoding='utf-8') as f:
+            return f.read()
+    except FileNotFoundError:
+        return None
+    except PermissionError as e:
+        raise UnitsUnreadable(f'cannot read the installed {name}: {e}') from e
+    except (OSError, UnicodeDecodeError) as e:
+        raise RefreshError(f'cannot read the installed {name}: {e}') from e
+
+
+def _lookup_user(user):
+    try:
+        import pwd
+    except ImportError:  # not a POSIX host (the tests on Windows)
+        return True
+    try:
+        pwd.getpwnam(user)
+        return True
+    except KeyError:
+        return False
+
+
+class Refresher:
+    def __init__(self, systemd_dir=SYSTEMD_DIR, backup_dir=BACKUP_DIR, run=subprocess.run,
+                 is_root=None, user_exists=_lookup_user, log=None):
+        self.systemd_dir = systemd_dir
+        self.backup_dir = backup_dir
+        self.run = run
+        self.is_root = is_root or (lambda: hasattr(os, 'geteuid') and os.geteuid() == 0)
+        self.user_exists = user_exists
+        self.log = log or (lambda msg: print(msg, flush=True))
+
+    # -- what the installed units say -------------------------------------
+
+    def context(self, installed):
+        """(project root, web user) from the installed, root-owned units."""
+        display = installed.get(DISPLAY_UNIT)
+        if display is None:
+            raise RefreshError(f'{DISPLAY_UNIT} is not installed; run scripts/install/install_service.sh')
+        roots = directive_values(display, 'WorkingDirectory')
+        if len(roots) != 1:
+            raise RefreshError(f'the installed {DISPLAY_UNIT} does not name one WorkingDirectory')
+        root = roots[0]
+        if (not os.path.isabs(root) or any(ch in _UNSAFE_PATH_CHARS or ord(ch) < 32 for ch in root)
+                or os.path.normpath(root) != root):
+            raise RefreshError(f'the installed {DISPLAY_UNIT} runs from {root!r}, which cannot be used')
+        if not os.path.isdir(root):
+            raise RefreshError(f'{root} (the installed {DISPLAY_UNIT} WorkingDirectory) does not exist')
+
+        user = None
+        web = installed.get(WEB_UNIT)
+        if web is not None:
+            users = directive_values(web, 'User')
+            user = users[0] if len(users) == 1 else ('root' if not users else None)
+            if user is None or not _USER_RE.match(user) or not self.user_exists(user):
+                raise RefreshError(f'the installed {WEB_UNIT} runs as an account that cannot be used')
+            if directive_values(web, 'WorkingDirectory') != [root]:
+                raise RefreshError(f'the installed {WEB_UNIT} and {DISPLAY_UNIT} run from different folders')
+        return root, user
+
+    @staticmethod
+    def expected_user(name, web_user):
+        return 'root' if name == DISPLAY_UNIT else web_user
+
+    def _template(self, root, name):
+        """systemd/<name> under the checkout, as a regular file, never via a symlink."""
+        dir_flags = os.O_RDONLY | getattr(os, 'O_DIRECTORY', 0) | getattr(os, 'O_NOFOLLOW', 0)
+        if os.open in getattr(os, 'supports_dir_fd', set()):
+            try:
+                dfd = os.open(os.path.join(root, 'systemd'), dir_flags)
+            except OSError as e:
+                raise RefreshError(f'cannot open {root}/systemd: {e}') from e
+            try:
+                return _read_regular(name, dir_fd=dfd)
+            except FileNotFoundError:
+                return None
+            except OSError as e:
+                raise RefreshError(f'cannot read systemd/{name}: {e}') from e
+            finally:
+                os.close(dfd)
+        path = os.path.join(root, 'systemd', name)
+        if os.path.islink(os.path.join(root, 'systemd')):
+            raise RefreshError(f'{root}/systemd is a symlink')
+        try:
+            return _read_regular(path)
+        except FileNotFoundError:
+            return None
+        except OSError as e:
+            raise RefreshError(f'cannot read systemd/{name}: {e}') from e
+
+    def _validate(self, name, rendered, root, user):
+        problem = layout_problem(rendered, 'Service', ('User', 'WorkingDirectory'))
+        if problem:
+            raise RefreshError(f'systemd/{name} {problem}; refusing to install it')
+        if directive_values(rendered, 'User') != [user]:
+            raise RefreshError(f'systemd/{name} would not run as {user}; refusing to install it')
+        if directive_values(rendered, 'WorkingDirectory') != [root]:
+            raise RefreshError(f'systemd/{name} would not run from {root}; refusing to install it')
+
+    def plan(self):
+        """{unit: (installed text, new text)} for every installed unit that would change.
+
+        Raises RefreshError, and so changes nothing, if any unit cannot be
+        rendered safely: four units refreshed as a set or not at all.
+        """
+        installed = {name: _read_installed(self.systemd_dir, name) for name in UNITS}
+        root, web_user = self.context(installed)
+        changes = {}
+        for name in UNITS:
+            current = installed[name]
+            if current is None:
+                continue  # never installed here: installing is the installer's job
+            user = self.expected_user(name, web_user)
+            if user is None:
+                continue  # the web unit is not installed, so neither is its user
+            template = self._template(root, name)
+            if template is None:
+                continue  # a version without this unit leaves the installed one alone
+            rendered = render(template, root, user)
+            # A path unit runs nothing itself; what matters is what it starts.
+            if name.endswith('.service'):
+                self._validate(name, rendered, root, user)
+            else:
+                self._validate_path(name, rendered)
+            if unit_body(rendered) != unit_body(current):
+                changes[name] = (current, rendered)
+        return changes
+
+    def _validate_path(self, name, rendered):
+        problem = layout_problem(rendered, 'Path', ('Unit',))
+        if problem:
+            raise RefreshError(f'systemd/{name} {problem}; refusing to install it')
+        if directive_values(rendered, 'Unit') != [VERIFY_SERVICE]:
+            raise RefreshError(f'systemd/{name} does not start {VERIFY_SERVICE}; refusing to install it')
+        if directive_values(rendered, 'User'):
+            raise RefreshError(f'systemd/{name} sets User=; refusing to install it')
+
+    # -- writing ------------------------------------------------------------
+
+    def _write_unit(self, name, text):
+        fd, tmp = tempfile.mkstemp(dir=self.systemd_dir, prefix=f'.{name}.')
+        try:
+            with os.fdopen(fd, 'w', encoding='utf-8', newline='\n') as f:
+                f.write(text)
+            os.chmod(tmp, 0o644)
+            os.replace(tmp, os.path.join(self.systemd_dir, name))
+        except BaseException:
+            try:
+                os.unlink(tmp)
+            except OSError:
+                pass
+            raise
+
+    def _backup_dir(self):
+        """The backup folder, created root-only; refused if it is not a plain folder."""
+        os.makedirs(os.path.dirname(self.backup_dir), mode=0o755, exist_ok=True)
+        try:
+            os.mkdir(self.backup_dir, 0o700)
+        except FileExistsError:
+            pass
+        info = os.lstat(self.backup_dir)
+        if not stat.S_ISDIR(info.st_mode):
+            raise RefreshError(f'{self.backup_dir} is not a folder')
+        if hasattr(os, 'geteuid') and info.st_uid != os.geteuid():
+            raise RefreshError(f'{self.backup_dir} is not owned by root')
+        return self.backup_dir
+
+    def _clear_backup(self, folder):
+        for entry in os.listdir(folder):
+            path = os.path.join(folder, entry)
+            if os.path.isfile(path) or os.path.islink(path):
+                os.unlink(path)
+
+    def _systemctl(self, *args):
+        result = self.run(['systemctl', *args], capture_output=True, text=True, timeout=60)
+        if result.returncode != 0:
+            raise RefreshError(f'"systemctl {" ".join(args)}" failed: '
+                               f'{(result.stderr or result.stdout or "").strip()}')
+
+    def _restart_path_unit_if_active(self, names):
+        """A rewritten path unit watches the old path until it is restarted."""
+        if VERIFY_PATH not in names:
+            return
+        state = self.run(['systemctl', 'is-active', VERIFY_PATH], capture_output=True, text=True, timeout=30)
+        if (state.stdout or '').strip() == 'active':
+            self._systemctl('restart', VERIFY_PATH)
+
+    def refresh(self):
+        if not self.is_root():
+            raise RefreshError('must run as root (sudo)')
+        changes = self.plan()
+        folder = self._backup_dir()
+        # Always reset: the backup belongs to this refresh, so a --restore
+        # after an update that changed nothing restores nothing.
+        self._clear_backup(folder)
+        if not changes:
+            self.log('units: up to date')
+            return []
+        for name, (current, _) in changes.items():
+            with open(os.path.join(folder, name), 'w', encoding='utf-8', newline='\n') as f:
+                f.write(current)
+        with open(os.path.join(folder, MANIFEST), 'w', encoding='utf-8') as f:
+            json.dump({'units': sorted(changes)}, f)
+        try:
+            for name, (_, rendered) in changes.items():
+                self._write_unit(name, rendered)
+            self._systemctl('daemon-reload')
+        except BaseException:
+            # A failed refresh is reported as a failure, so the update records
+            # no units_refreshed and a rollback would not --restore. Put the
+            # replaced units back now, rather than leave a half-written set
+            # under the old code.
+            self._undo(changes, folder)
+            raise
+        self._restart_path_unit_if_active(changes)
+        self.log('units refreshed: ' + ' '.join(sorted(changes)))
+        return sorted(changes)
+
+    def _undo(self, changes, folder):
+        """Best effort: reinstall the units a failed refresh replaced."""
+        undone = True
+        for name, (current, _) in changes.items():
+            try:
+                self._write_unit(name, current)
+            except OSError as e:
+                undone = False
+                self.log(f'units: could not put back {name}: {e}')
+        try:
+            self.run(['systemctl', 'daemon-reload'], capture_output=True, text=True, timeout=60)
+        except (OSError, subprocess.SubprocessError) as e:
+            self.log(f'units: daemon-reload after putting units back failed: {e}')
+        if undone:
+            # Nothing is left to restore; keep the backup only if a unit could
+            # not be put back, so a manual --restore still can.
+            self._clear_backup(folder)
+
+    def restore(self):
+        if not self.is_root():
+            raise RefreshError('must run as root (sudo)')
+        folder = self._backup_dir()
+        try:
+            manifest = json.loads(_read_regular(os.path.join(folder, MANIFEST)))
+        except FileNotFoundError:
+            self.log('units: nothing to restore')
+            return []
+        names = [n for n in (manifest or {}).get('units', []) if n in UNITS]
+        for name in names:
+            self._write_unit(name, _read_regular(os.path.join(folder, name)))
+        self._systemctl('daemon-reload')
+        self._restart_path_unit_if_active(names)
+        self._clear_backup(folder)
+        self.log('units restored: ' + ' '.join(names))
+        return names
+
+
+def main(argv, refresher=None):
+    args = argv[1:]
+    if args not in ([], ['--restore'], ['--check']):
+        print('usage: ledmatrix-refresh-units [--restore | --check]', file=sys.stderr)
+        return EXIT_USAGE
+    refresher = refresher or Refresher()
+    try:
+        if args == ['--check']:
+            for name in sorted(refresher.plan()):
+                print(name)
+        elif args == ['--restore']:
+            refresher.restore()
+        else:
+            refresher.refresh()
+    except (RefreshError, OSError, subprocess.SubprocessError, ValueError) as e:
+        print(f'ledmatrix-refresh-units: {e}', file=sys.stderr)
+        return EXIT_FAILED
+    return EXIT_OK
+
+
+if __name__ == '__main__':
+    # Only as the installed program: sudo already sets a secure PATH, and
+    # this pins the one systemctl comes from. (Not in main(), which the
+    # tests call in-process.)
+    os.environ['PATH'] = '/usr/sbin:/usr/bin:/sbin:/bin'
+    sys.exit(main(sys.argv))

scripts/install/lib_sudoers.sh

@@ -10,6 +10,11 @@
 #
 # Add or remove a grant here and nowhere else.
 
+# Root-owned copy of scripts/install/ledmatrix_refresh_units.py, installed by
+# install_service.sh. Outside the checkout on purpose: the web user owns the
+# checkout, so a granted file inside it could be rewritten and run as root.
+LEDMATRIX_REFRESH_UNITS_PATH=/usr/local/sbin/ledmatrix-refresh-units
+
 # web_sudoers_rules WEB_USER PROJECT_ROOT SYSTEMCTL_PATH BASH_PATH REBOOT_PATH POWEROFF_PATH JOURNALCTL_PATH
 #
 # Print the ledmatrix_web sudoers rules to stdout.
@@ -58,6 +63,10 @@ $WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_ROOT/scripts/fix_perms/safe_pl
 # Install a requirements.txt as root via vetted helper, so packages are visible
 # to root-run ledmatrix.service (not just the web interface's own user).
 $WEB_USER ALL=(ALL) NOPASSWD: $BASH_PATH $PROJECT_ROOT/scripts/fix_perms/safe_pip_install.sh *
+# After an update, install the new systemd units (no arguments: "" allows none)
+# and, on the automatic update's rollback, put the previous ones back.
+$WEB_USER ALL=(ALL) NOPASSWD: $LEDMATRIX_REFRESH_UNITS_PATH ""
+$WEB_USER ALL=(ALL) NOPASSWD: $LEDMATRIX_REFRESH_UNITS_PATH --restore
 EOF
     if [ -n "$JOURNALCTL_PATH" ]; then
         cat << EOF

scripts/install/one-shot-install.sh

@@ -3,6 +3,10 @@
 # LED Matrix One-Shot Installation Script
 # This script provides a single-command installation experience
 # Usage: curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/install/one-shot-install.sh | bash
+#
+# A new install runs the newest release (the stable update channel). For the
+# newest code from main instead (the beta channel), set LEDMATRIX_CHANNEL=beta:
+#   curl -fsSL https://raw.githubusercontent.com/ChuckBuilds/LEDMatrix/main/scripts/install/one-shot-install.sh | LEDMATRIX_CHANNEL=beta bash
 
 set -Eeuo pipefail
 
@@ -205,6 +209,114 @@ check_sudo() {
     print_success "Sudo access confirmed"
 }
 
+# --- release checkout helpers ------------------------------------------------
+# Which version an install runs. The rules are web_interface/update_channel.py's,
+# so the installer and the web interface's updates agree:
+#   stable (default)  the newest vX.Y.Z tag by semantic version; pre-releases
+#                     (v3.8.0-rc1), leading zeros and other tags are ignored
+#   beta              main, the newest code
+# Never backwards: an existing checkout moves to a release only when that
+# release contains its current commit (git merge-base --is-ancestor).
+# Never fatal: whatever goes wrong, the install carries on with the checkout
+# as it is.
+
+# Print "stable" or "beta": LEDMATRIX_CHANNEL when it is set, else the
+# existing install's auto_update.channel (CONFIG_FILE), else stable.
+_lm_channel() {
+    local config_file="${1:-}" value
+    value=$(printf '%s' "${LEDMATRIX_CHANNEL:-}" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')
+    case "$value" in
+        stable|beta) printf '%s\n' "$value"; return 0 ;;
+        "") ;;
+        *) print_warning "LEDMATRIX_CHANNEL=${LEDMATRIX_CHANNEL} is not stable or beta; using stable" >&2
+           printf 'stable\n'; return 0 ;;
+    esac
+    if [ -n "$config_file" ] && [ -f "$config_file" ] && command -v python3 >/dev/null 2>&1; then
+        value=$(python3 - "$config_file" 2>/dev/null <<'PY' || true
+import json, sys
+try:
+    with open(sys.argv[1], encoding="utf-8") as f:
+        section = json.load(f).get("auto_update")
+    value = section.get("channel") if isinstance(section, dict) else None
+    print(value.strip().lower() if isinstance(value, str) else "")
+except Exception:
+    print("")
+PY
+)
+        if [ "$value" = "beta" ]; then
+            printf 'beta\n'
+            return 0
+        fi
+    fi
+    printf 'stable\n'
+}
+
+# Print the newest release tag of the repository in the current directory,
+# or nothing when it has none.
+_lm_newest_release_tag() {
+    git tag --list 'v*' 2>/dev/null \
+        | grep -E '^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$' \
+        | sort -t. -k1.2,1n -k2,2n -k3,3n \
+        | tail -n 1 || true
+}
+
+# A fresh clone (on main): move to the newest release unless beta was asked for.
+_lm_checkout_release_after_clone() {
+    local channel tag
+    channel=$(_lm_channel "")
+    if [ "$channel" = "beta" ]; then
+        print_success "Beta channel: installing the newest code from main"
+        return 0
+    fi
+    tag=$(_lm_newest_release_tag)
+    if [ -z "$tag" ]; then
+        print_warning "No release found; installing the newest code from main"
+        return 0
+    fi
+    if git -c advice.detachedHead=false checkout --quiet --detach "${tag}^{commit}"; then
+        print_success "Installing release $tag (stable channel)"
+    else
+        print_warning "Could not check out release $tag; installing the newest code from main"
+    fi
+    return 0
+}
+
+# An existing checkout: move it forward along its channel, never backwards.
+# Returns 1 when it should be updated the way it always was (a fast-forward
+# pull of its branch): beta, or stable on a branch newer than every release.
+_lm_update_existing_checkout() {
+    local channel tag head tag_sha
+    channel=$(_lm_channel "config/config.json")
+    if [ "$channel" = "beta" ]; then
+        return 1
+    fi
+    if ! git fetch --quiet --tags --force origin >/dev/null 2>&1; then
+        print_warning "Could not fetch release tags; keeping the current version"
+        return 0
+    fi
+    tag=$(_lm_newest_release_tag)
+    head=$(git rev-parse --verify --quiet HEAD 2>/dev/null || true)
+    if [ -n "$tag" ] && [ -n "$head" ] && git merge-base --is-ancestor "$head" "$tag" 2>/dev/null; then
+        tag_sha=$(git rev-parse --verify --quiet "${tag}^{commit}" 2>/dev/null || true)
+        if [ "$head" = "$tag_sha" ]; then
+            print_success "Already on the newest release, $tag"
+        elif git -c advice.detachedHead=false checkout --quiet --detach "${tag}^{commit}"; then
+            print_success "Updated to release $tag (stable channel)"
+        else
+            print_warning "Could not move to release $tag (local changes?); keeping the current version"
+        fi
+        return 0
+    fi
+    if git symbolic-ref --quiet HEAD >/dev/null 2>&1; then
+        # Newer than the newest release (or no release yet): follow the branch
+        # until a release includes this version, as updates do.
+        return 1
+    fi
+    print_success "This checkout is newer than the newest release${tag:+ ($tag)}; leaving it as it is"
+    return 0
+}
+# --- end release checkout helpers --------------------------------------------
+
 # Main installation function
 main() {
     print_step "LED Matrix One-Shot Installation"
@@ -292,7 +404,10 @@ main() {
             
             # Try to safely update current branch first (fast-forward only to avoid unintended merges)
             PULL_SUCCESS=false
-            if git pull --ff-only origin "$CURRENT_BRANCH" >/dev/null 2>&1; then
+            # Stable: the newest release, if it contains this version.
+            if _lm_update_existing_checkout; then
+                PULL_SUCCESS=true
+            elif git pull --ff-only origin "$CURRENT_BRANCH" >/dev/null 2>&1; then
                 print_success "Repository updated successfully (branch: $CURRENT_BRANCH)"
                 PULL_SUCCESS=true
             else
@@ -323,10 +438,12 @@ main() {
             rm -rf "$REPO_DIR"
             print_success "Cloning repository..."
             retry git clone "$REPO_URL" "$REPO_DIR"
+            (cd "$REPO_DIR" && _lm_checkout_release_after_clone) || print_warning "Could not choose a release; installing the newest code from main"
         fi
     else
         print_success "Cloning repository to $REPO_DIR..."
         retry git clone "$REPO_URL" "$REPO_DIR"
+        (cd "$REPO_DIR" && _lm_checkout_release_after_clone) || print_warning "Could not choose a release; installing the newest code from main"
     fi
     
     # Verify repository is accessible
@@ -397,6 +514,7 @@ main() {
         sudo -E env TMPDIR=/tmp LEDMATRIX_ASSUME_YES=1 \
             LEDMATRIX_APT_UPDATED="${LEDMATRIX_APT_UPDATED:-0}" \
             LEDMATRIX_AUTO_UPDATE="${LEDMATRIX_AUTO_UPDATE:-}" \
+            LEDMATRIX_CHANNEL="${LEDMATRIX_CHANNEL:-}" \
             bash ./first_time_install.sh -y </dev/null
     fi
     INSTALL_EXIT_CODE=$?

scripts/utils/auto_update_verify.py

@@ -15,7 +15,8 @@ The updater leaves data/auto_update_pending.json:
 
     {"status": "pending", "old_head": ..., "new_head": ...,
      "old_ref": "main" | "" (detached) | absent (older updaters),
-     "display_was_active": bool, "dependency_failures": [...], "created_at": ...}
+     "display_was_active": bool, "dependency_failures": [...],
+     "units_refreshed": bool (absent from older updaters), "created_at": ...}
 
 This moves its status to "verifying" and then to one of "success",
 "rolled_back" or "rollback_failed", with "reason" and "detail" saying why.
@@ -74,6 +75,10 @@ BASH_CANDIDATES = ('/usr/bin/bash', '/bin/bash')
 #: ...and, like it, moves to the next one only when sudo refused the command
 #: line (permission_utils.SUDO_REFUSAL_PHRASES), never after pip itself ran.
 SUDO_REFUSAL_PHRASES = ('a password is required', 'is not allowed to run', 'no tty present')
+#: The root-owned helper that installed the update's systemd units
+#: (web_interface/unit_refresh.py); ``--restore`` puts the previous ones back.
+REFRESH_UNITS_PATH = '/usr/local/sbin/ledmatrix-refresh-units'
+UNIT_RESTORE_TIMEOUT_SECONDS = 90
 
 #: The longest one health check can take: restart and wait, roll back
 #: (diff, reset, reinstalls), restart and wait again. A wait's last poll can
@@ -81,7 +86,8 @@ SUDO_REFUSAL_PHRASES = ('a password is required', 'is not allowed to run', 'no t
 _WAIT_WORST_SECONDS = (HEALTH_TIMEOUT_SECONDS + STABLE_SECONDS + WEB_CHECK_TIMEOUT_SECONDS
                        + 2 * SYSTEMCTL_QUERY_TIMEOUT_SECONDS + POLL_SECONDS)
 WORST_CASE_SECONDS = (2 * (2 * RESTART_TIMEOUT_SECONDS + _WAIT_WORST_SECONDS)
-                      + GIT_TIMEOUT_SECONDS + GIT_RESET_TIMEOUT_SECONDS + PIP_BUDGET_SECONDS)
+                      + GIT_TIMEOUT_SECONDS + GIT_RESET_TIMEOUT_SECONDS + UNIT_RESTORE_TIMEOUT_SECONDS
+                      + PIP_BUDGET_SECONDS)
 
 #: What a command that could not run at all reports: its callers only read
 #: these three fields, the same ones a completed subprocess has.
@@ -300,12 +306,26 @@ class Verifier:
         if result.returncode != 0:
             return False, (f'"git reset --hard {old}" failed: '
                            f'{(result.stderr or result.stdout or "").strip()}')
+        notes = []
+        # The update also installed its own systemd units: put the previous
+        # ones back before anything restarts onto the rolled-back code.
+        if pending.get('units_refreshed') and not self.restore_units():
+            notes.append('restoring the previous service settings failed; run '
+                         '"sudo ./scripts/install/install_service.sh" in the LEDMatrix folder')
         deadline = self.clock() + PIP_BUDGET_SECONDS
         failed = [rel for rel in requirements if not self.install_requirements(rel, deadline)]
         if failed:
-            return True, ('reinstalling the previous dependencies from ' + ', '.join(failed)
+            notes.append('reinstalling the previous dependencies from ' + ', '.join(failed)
-                          + ' failed; run Install Base Requirements from the Tools tab')
+                         + ' failed; run Install Base Requirements from the Tools tab')
-        return True, ''
+        return True, '; '.join(notes)
+
+    def restore_units(self):
+        """Reinstall the systemd units the update replaced. True on success."""
+        result = self._run(['sudo', '-n', REFRESH_UNITS_PATH, '--restore'],
+                           timeout=UNIT_RESTORE_TIMEOUT_SECONDS)
+        if result.returncode != 0:
+            self.log(f'restoring the previous systemd units failed: {(result.stderr or "").strip()}')
+        return result.returncode == 0
 
     # -- the check itself -------------------------------------------------
 

src/startup_validator.py

@@ -95,11 +95,13 @@ class StartupValidator:
     def _validate_systemd_units(self) -> None:
         """Warn when an installed unit has drifted from the repo's template.
 
-        Nothing re-applies these after the first install. `git pull` -- which is
+        Before updates refreshed units, nothing re-applied these after the
-        what the web UI's update button runs -- brings a new template into the
+        first install: `git pull` brought a new template into the checkout,
-        checkout, but nothing copies it to /etc/systemd/system and nothing runs
+        but nothing copied it to /etc/systemd/system, so the unit that
-        `systemctl daemon-reload`, so the unit that actually runs is whatever
+        actually ran was whatever first_time_install.sh wrote on day one.
-        first_time_install.sh wrote on day one.
+        Updates now install changed units through the root helper
+        ledmatrix-refresh-units (web_interface/unit_refresh.py) -- but only on
+        a device whose installer granted it, so this still catches the rest.
 
         That makes every hardening added to a unit inert on existing installs.
         Measured on one rig: the installed unit was thirteen days older than the
@@ -141,10 +143,11 @@ class StartupValidator:
 
                 if self._unit_body(expected) != self._unit_body(actual):
                     self.warnings.append(
-                        f"{installed.name} differs from {template_rel}; the "
+                        f"{installed.name} differs from {template_rel}, so "
-                        "installed unit is not refreshed by an update, so "
                         "settings added to the template are not in effect. "
-                        "Re-run scripts/install/install_service.sh to apply them."
+                        "Updates apply them only once the installer has granted "
+                        "ledmatrix-refresh-units: re-run "
+                        "scripts/install/install_service.sh (or first_time_install.sh) to apply them."
                     )
         except OSError as e:
             self.logger.debug("Could not compare systemd units: %s", e)

test/conftest.py

@@ -328,6 +328,21 @@ def _hermetic_control_socket(monkeypatch):
     monkeypatch.setenv(SOCKET_PATH_ENV, 'off')
 
 
+@pytest.fixture(autouse=True)
+def _hermetic_unit_refresh(monkeypatch, tmp_path_factory):
+    """Keep updates' systemd unit refresh off the host.
+
+    perform_core_update runs web_interface/unit_refresh.py after any update
+    that moves HEAD, and several tests run the real one against a test clone.
+    On a device -- or a machine where install_service.sh was tried out -- it
+    would compare the clone's templates with the real /etc/systemd/system and
+    run the real sudo helper. Point it at a folder that does not exist: no units
+    installed, nothing to do. The unit refresh tests pass their own.
+    """
+    from web_interface import unit_refresh
+    monkeypatch.setattr(unit_refresh, 'SYSTEMD_DIR', str(tmp_path_factory.getbasetemp() / 'no-systemd'))
+
+
 @pytest.fixture(autouse=True)
 def reset_logging():
     """Reset logging configuration before each test."""

test/test_auto_update.py

@@ -517,6 +517,32 @@ class TestUpdateIsVerified:
         h.updater.run()
         assert h.pending['dependency_failures'] == ['requirements.txt']
 
+    @pytest.mark.parametrize('unit_refresh, expected', [
+        ({'status': 'refreshed', 'message': '', 'units': ['ledmatrix.service']}, True),
+        ({'status': 'needs_reinstall', 'message': '', 'units': ['ledmatrix.service']}, False),
+        (None, False),
+    ])
+    def test_the_health_check_learns_whether_the_update_installed_units(self, tmp_path, unit_refresh,
+                                                                          expected):
+        """Its rollback restores the previous units only when this update replaced them."""
+        repo = Repo(tmp_path)
+        repo.publish()
+        h = Harness(tmp_path, repo, core_update=real_pull(repo.device, unit_refresh=unit_refresh))
+        h.updater.run()
+        assert h.pending['units_refreshed'] is expected
+
+    def test_a_health_check_that_never_starts_also_restores_the_units(self, tmp_path):
+        repo = Repo(tmp_path)
+        old = repo.head()
+        repo.publish()
+        refreshed = {'status': 'refreshed', 'message': '', 'units': ['ledmatrix.service']}
+        h = Harness(tmp_path, repo, pickup=False,
+                    core_update=real_pull(repo.device, unit_refresh=refreshed))
+        h.updater.run()
+        assert repo.head() == old
+        assert [a for a in h.sudo if a[-1] == '--restore'] == [
+            ['sudo', '-n', '/usr/local/sbin/ledmatrix-refresh-units', '--restore']]
+
     def test_a_health_check_that_never_starts_means_the_update_is_undone(self, tmp_path):
         repo = Repo(tmp_path)
         old = repo.head()

test/test_auto_update_verify.py

@@ -80,6 +80,8 @@ class FakeHost:
         self.nrestarts = 0
         self.heartbeat = heartbeat
         self.display_started_at = -1000.0  # the pre-update display, long running
+        self.unit_restores = []  # (argv, commit checked out, restarts so far)
+        self.restore_ok = True
 
     def broken(self, kind):
         if self.running_head is None:
@@ -103,6 +105,10 @@ class FakeHost:
             if self.pip:
                 return self.pip(args, self)
             return done(args, rc=0 if self.pip_ok else 1)
+        if args[:3] == ['sudo', '-n', av.REFRESH_UNITS_PATH]:
+            self.unit_restores.append((list(args), git(self.repo, 'rev-parse', 'HEAD'),
+                                       len(self.restarts)))
+            return done(args, rc=0 if self.restore_ok else 1)
         if args[:4] == ['sudo', '-n', 'systemctl', 'restart']:
             if self.restart_failures:
                 self.restart_failures -= 1
@@ -405,3 +411,45 @@ def test_the_heartbeat_location_and_freshness_match_the_display():
     # window it has to stay healthy for.
     assert av.HEARTBEAT_FRESH_SECONDS + av.POLL_SECONDS < av.STABLE_SECONDS
     assert av.HEARTBEAT_FRESH_SECONDS > display_watchdog.BEAT_INTERVAL_SECONDS * 2
+
+
+# -- systemd units the update installed ------------------------------------------
+
+def test_a_rollback_restores_the_units_the_update_installed(tmp_path):
+    """The update installed new units (web_interface/unit_refresh.py); the
+    rollback puts the old ones back before restarting onto the old code."""
+    code, result, host, head, old, new = check(tmp_path, 'display_down', units_refreshed=True)
+    assert result['status'] == 'rolled_back' and head == old
+    assert len(host.unit_restores) == 1
+    argv, commit, restarts_before = host.unit_restores[0]
+    assert argv == ['sudo', '-n', av.REFRESH_UNITS_PATH, '--restore']
+    assert commit == old, 'restored after the code was rolled back'
+    assert restarts_before == 2, 'restored before the services restart onto the old code'
+    assert host.restarts[-2:] == [('ledmatrix.service', old), ('ledmatrix-web.service', old)]
+    assert result['detail'] is None
+
+
+@pytest.mark.parametrize('pending', [{}, {'units_refreshed': False}])
+def test_a_rollback_leaves_units_alone_when_the_update_did_not_change_them(tmp_path, pending):
+    # {} is what an updater from before this change writes.
+    code, result, host, head, old, new = check(tmp_path, 'display_down', **pending)
+    assert result['status'] == 'rolled_back' and host.unit_restores == []
+
+
+def test_a_healthy_update_keeps_its_new_units(tmp_path):
+    code, result, host, head, old, new = check(tmp_path, units_refreshed=True)
+    assert result['status'] == 'success' and host.unit_restores == []
+
+
+def test_a_failed_unit_restore_is_reported_but_the_rollback_stands(tmp_path):
+    repo, old, new = updated_repo(tmp_path)
+    av.write_pending(av.pending_path(repo), {'status': 'pending', 'old_head': old, 'new_head': new,
+                                             'display_was_active': True, 'dependency_failures': [],
+                                             'units_refreshed': True})
+    host = FakeHost(repo, new, 'display_down')
+    host.restore_ok = False
+    host.verifier().verify()
+    result = av.read_pending(av.pending_path(repo))
+    assert result['status'] == 'rolled_back'
+    assert git(repo, 'rev-parse', 'HEAD') == old
+    assert 'install_service.sh' in result['detail']

test/test_install_release_checkout.py

@@ -0,0 +1,370 @@
+"""New installs run the newest release; re-running the installer never moves backwards.
+
+#684 made devices update along a channel -- stable follows the newest vX.Y.Z
+tag, beta follows main -- but a new install still cloned main's tip, so it ran
+unreleased code until the next release caught up with it. The one-shot
+installer (scripts/install/one-shot-install.sh, which is where the clone
+happens) now checks out the newest release after cloning, unless
+LEDMATRIX_CHANNEL=beta. Re-running it on an existing checkout moves a stable
+device forward to the newest release only when that release contains its
+commit, as update_channel.checkout_release() does, and leaves beta devices
+(and stable ones newer than every release) on the fast-forward pull they
+always had. first_time_install.sh writes an explicitly chosen channel
+(--beta / LEDMATRIX_CHANNEL) into config.json.
+
+These run the installer's own bash, under its strict mode, against real git
+repositories.
+"""
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT))
+ONE_SHOT = ROOT / "scripts" / "install" / "one-shot-install.sh"
+INSTALLER = ROOT / "first_time_install.sh"
+BEGIN = "# --- release checkout helpers"
+END = "# --- end release checkout helpers"
+
+from web_interface import update_channel  # noqa: E402
+
+pytestmark = pytest.mark.skipif(
+    not sys.platform.startswith("linux"), reason="runs the installer's bash under Linux"
+)
+
+
+def helper_block() -> str:
+    text = ONE_SHOT.read_text(encoding="utf-8")
+    assert text.count(BEGIN) == 1 and text.count(END) == 1, "helper block markers missing or duplicated"
+    return text[text.index(BEGIN): text.index(END)]
+
+
+def git(*args, cwd, env):
+    result = subprocess.run(["git", *args], cwd=cwd, env=env, capture_output=True, text=True)
+    assert result.returncode == 0, f"git {' '.join(args)} failed: {result.stderr}"
+    return result.stdout.strip()
+
+
+@pytest.fixture
+def git_env(tmp_path):
+    config = tmp_path / "gitconfig"
+    config.write_text(
+        "[user]\n\tname = t\n\temail = t@t\n"
+        "[protocol \"file\"]\n\tallow = always\n"
+        "[init]\n\tdefaultBranch = main\n"
+        "[advice]\n\tdetachedHead = false\n",
+        encoding="utf-8",
+    )
+    return {
+        "PATH": "/usr/bin:/bin:/usr/sbin:/sbin",
+        "HOME": str(tmp_path),
+        "GIT_CONFIG_GLOBAL": str(config),
+        "GIT_CONFIG_NOSYSTEM": "1",
+    }
+
+
+#: Tags and the commit (index into the history) each points at. The newest
+#: release is v3.10.0: 10 > 8 numerically, the rc and the zero-padded tag are
+#: not releases, and v3.12 and nightly are not vX.Y.Z at all.
+TAGS = {
+    "v3.7.0": 0, "v3.8.0": 1, "v3.10.0": 2,
+    "v3.11.0-rc1": 3, "v03.12.0": 3, "v3.12": 3, "nightly": 3,
+}
+NEWEST = "v3.10.0"
+
+
+@pytest.fixture
+def origin(tmp_path, git_env):
+    """A stand-in for GitHub: five commits on main (the last newer than any release)."""
+    seed = tmp_path / "seed"
+    seed.mkdir()
+    git("init", "-q", ".", cwd=seed, env=git_env)
+    commits = []
+    for i in range(5):
+        (seed / "version.txt").write_text(str(i), encoding="utf-8")
+        git("add", ".", cwd=seed, env=git_env)
+        git("commit", "-qm", f"c{i}", cwd=seed, env=git_env)
+        commits.append(git("rev-parse", "HEAD", cwd=seed, env=git_env))
+    for tag, index in TAGS.items():
+        git("tag", tag, commits[index], cwd=seed, env=git_env)
+    bare = tmp_path / "origin.git"
+    git("clone", "-q", "--bare", str(seed), str(bare), cwd=tmp_path, env=git_env)
+    return bare, commits, seed
+
+
+def run_block(snippet, cwd, env, channel=None):
+    env = dict(env)
+    if channel is not None:
+        env["LEDMATRIX_CHANNEL"] = channel
+    script = (
+        "set -Eeuo pipefail\n"
+        "trap 'echo ERR_TRAP_FIRED >&2; exit 99' ERR\n"
+        'print_success() { echo "OK: $*"; }\n'
+        'print_warning() { echo "W: $*"; }\n'
+        f"{helper_block()}\n"
+        f"{snippet}\n"
+    )
+    result = subprocess.run(["bash", "-c", script], cwd=cwd, capture_output=True, text=True, env=env)
+    assert "ERR_TRAP_FIRED" not in result.stderr, result.stdout + result.stderr
+    return result
+
+
+def clone(origin, tmp_path, env, name="LEDMatrix"):
+    bare, _, _ = origin
+    target = tmp_path / name
+    git("clone", "-q", str(bare), str(target), cwd=tmp_path, env=env)
+    return target
+
+
+def head(repo, env):
+    return git("rev-parse", "HEAD", cwd=repo, env=env)
+
+
+def branch(repo, env):
+    result = subprocess.run(["git", "symbolic-ref", "--quiet", "--short", "HEAD"], cwd=repo, env=env,
+                            capture_output=True, text=True)
+    return result.stdout.strip()
+
+
+def set_channel(repo, channel):
+    (repo / "config").mkdir(exist_ok=True)
+    (repo / "config" / "config.json").write_text(
+        json.dumps({"auto_update": {"enabled": False, "channel": channel}}), encoding="utf-8")
+
+
+# -- a fresh install -------------------------------------------------------------
+
+def test_a_fresh_clone_checks_out_the_newest_release(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = clone(origin, tmp_path, git_env)
+    out = run_block("_lm_checkout_release_after_clone", repo, git_env)
+    assert out.returncode == 0
+    assert head(repo, git_env) == commits[TAGS[NEWEST]]
+    assert branch(repo, git_env) == "", "a release is checked out detached, as Update Code does"
+    assert f"Installing release {NEWEST}" in out.stdout
+
+
+@pytest.mark.parametrize("channel", ["beta", "BETA", " beta "])
+def test_a_fresh_beta_install_stays_on_main(origin, tmp_path, git_env, channel):
+    _, commits, _ = origin
+    repo = clone(origin, tmp_path, git_env)
+    run_block("_lm_checkout_release_after_clone", repo, git_env, channel=channel)
+    assert head(repo, git_env) == commits[-1] and branch(repo, git_env) == "main"
+
+
+def test_an_unknown_channel_falls_back_to_stable_and_says_so(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = clone(origin, tmp_path, git_env)
+    out = run_block("_lm_checkout_release_after_clone", repo, git_env, channel="nightly")
+    assert head(repo, git_env) == commits[TAGS[NEWEST]]
+    assert "not stable or beta" in out.stdout + out.stderr
+
+
+def test_a_repository_without_releases_installs_main(tmp_path, git_env):
+    seed = tmp_path / "seed"
+    seed.mkdir()
+    git("init", "-q", ".", cwd=seed, env=git_env)
+    (seed / "f").write_text("x", encoding="utf-8")
+    git("add", ".", cwd=seed, env=git_env)
+    git("commit", "-qm", "only", cwd=seed, env=git_env)
+    git("tag", "v3.0", cwd=seed, env=git_env)  # not a release tag
+    repo = tmp_path / "LEDMatrix"
+    git("clone", "-q", str(seed), str(repo), cwd=tmp_path, env=git_env)
+    out = run_block("_lm_checkout_release_after_clone", repo, git_env)
+    assert branch(repo, git_env) == "main" and "No release found" in out.stdout
+
+
+# -- re-running on an existing checkout -----------------------------------------------
+
+def existing(origin, tmp_path, env, at, detached):
+    """An installed checkout, at commit index ``at``, on main or detached."""
+    _, commits, _ = origin
+    repo = clone(origin, tmp_path, env)
+    if detached:
+        git("checkout", "-q", "--detach", commits[at], cwd=repo, env=env)
+    else:
+        git("reset", "-q", "--hard", commits[at], cwd=repo, env=env)
+    return repo
+
+
+def update(repo, env, channel=None):
+    out = run_block("if _lm_update_existing_checkout; then echo RESULT=handled; "
+                    "else echo RESULT=pull; fi", repo, env, channel=channel)
+    return re.search(r"RESULT=(\w+)", out.stdout).group(1), out
+
+
+def test_a_device_on_an_older_release_moves_to_the_newest(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS["v3.7.0"], detached=True)
+    result, out = update(repo, git_env)
+    assert result == "handled"
+    assert head(repo, git_env) == commits[TAGS[NEWEST]]
+    assert f"Updated to release {NEWEST}" in out.stdout
+
+
+def test_a_device_already_on_the_newest_release_stays(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS[NEWEST], detached=True)
+    result, out = update(repo, git_env)
+    assert result == "handled" and head(repo, git_env) == commits[TAGS[NEWEST]]
+    assert "Already on the newest release" in out.stdout
+
+
+def test_a_device_on_main_behind_the_newest_release_moves_to_it(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS["v3.8.0"], detached=False)
+    result, _ = update(repo, git_env)
+    assert result == "handled" and head(repo, git_env) == commits[TAGS[NEWEST]]
+
+
+def test_a_device_on_main_newer_than_every_release_is_not_moved_back(origin, tmp_path, git_env):
+    """It keeps the fast-forward pull it always had, and waits for a release to contain it."""
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=4, detached=False)
+    result, _ = update(repo, git_env)
+    assert result == "pull"
+    assert head(repo, git_env) == commits[4] and branch(repo, git_env) == "main"
+
+
+def test_a_detached_device_newer_than_every_release_is_left_alone(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=3, detached=True)
+    result, out = update(repo, git_env)
+    assert result == "handled" and head(repo, git_env) == commits[3]
+    assert "newer than the newest release" in out.stdout
+
+
+def test_a_higher_version_on_an_older_commit_is_not_a_downgrade(origin, tmp_path, git_env):
+    """Newest by version is not newest by history: never move to a tag that does not contain HEAD."""
+    bare, commits, seed = origin
+    git("tag", "v9.0.0", commits[0], cwd=seed, env=git_env)
+    git("push", "-q", str(bare), "v9.0.0", cwd=seed, env=git_env)
+    repo = existing(origin, tmp_path, git_env, at=TAGS[NEWEST], detached=True)
+    result, _ = update(repo, git_env)
+    assert result == "handled" and head(repo, git_env) == commits[TAGS[NEWEST]]
+
+
+def test_a_new_release_published_since_the_clone_is_fetched(origin, tmp_path, git_env):
+    bare, commits, seed = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS[NEWEST], detached=True)
+    git("tag", "v3.11.0", commits[4], cwd=seed, env=git_env)
+    git("push", "-q", str(bare), "v3.11.0", cwd=seed, env=git_env)
+    result, _ = update(repo, git_env)
+    assert result == "handled" and head(repo, git_env) == commits[4]
+
+
+def test_a_beta_device_keeps_its_pull(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS["v3.8.0"], detached=False)
+    set_channel(repo, "beta")
+    result, _ = update(repo, git_env)
+    assert result == "pull" and head(repo, git_env) == commits[TAGS["v3.8.0"]]
+
+
+def test_the_environment_overrides_the_configured_channel(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS["v3.8.0"], detached=False)
+    set_channel(repo, "stable")
+    result, _ = update(repo, git_env, channel="beta")
+    assert result == "pull" and head(repo, git_env) == commits[TAGS["v3.8.0"]]
+
+
+def test_local_edits_that_block_the_move_keep_the_checkout(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS["v3.7.0"], detached=True)
+    (repo / "version.txt").write_text("my edit", encoding="utf-8")
+    result, out = update(repo, git_env)
+    assert result == "handled" and head(repo, git_env) == commits[TAGS["v3.7.0"]]
+    assert (repo / "version.txt").read_text(encoding="utf-8") == "my edit"
+    assert "Could not move to release" in out.stdout
+
+
+def test_an_unreachable_origin_keeps_the_checkout(origin, tmp_path, git_env):
+    _, commits, _ = origin
+    repo = existing(origin, tmp_path, git_env, at=TAGS["v3.7.0"], detached=True)
+    git("remote", "set-url", "origin", str(tmp_path / "gone.git"), cwd=repo, env=git_env)
+    result, out = update(repo, git_env)
+    assert result == "handled" and head(repo, git_env) == commits[TAGS["v3.7.0"]]
+    assert "Could not fetch" in out.stdout
+
+
+# -- same rules as the web interface -------------------------------------------------
+
+NAMES = ["v1.2.3", "v1.10.0", "v1.9.9", "v2.0.0-rc1", "v02.0.0", "v2.0", "v10.0.0", "v9.99.99",
+         "release-11", "v10.0.0+build", "v0.0.0", "v10.0.1", "v1.2.03", "V11.0.0"]
+
+
+@pytest.mark.parametrize("subset", [NAMES, NAMES[:4], ["v2.0", "nightly"], NAMES[::-1][:6]])
+def test_the_newest_tag_matches_update_channel(tmp_path, git_env, subset):
+    repo = tmp_path / "tags"
+    repo.mkdir()
+    git("init", "-q", ".", cwd=repo, env=git_env)
+    git("commit", "-q", "--allow-empty", "-m", "x", cwd=repo, env=git_env)
+    for name in subset:
+        git("tag", name, cwd=repo, env=git_env)
+    out = run_block("_lm_newest_release_tag", repo, git_env).stdout.strip()
+    assert out == (update_channel.newest_release_tag(subset) or "")
+
+
+# -- wiring --------------------------------------------------------------------------
+
+def test_every_clone_is_followed_by_the_release_checkout():
+    text = ONE_SHOT.read_text(encoding="utf-8")
+    clones = [m.start() for m in re.finditer(r'retry git clone "\$REPO_URL" "\$REPO_DIR"\n', text)]
+    assert clones
+    for pos in clones:
+        following = text[pos:].splitlines()[1]
+        assert '_lm_checkout_release_after_clone' in following, following
+
+
+def test_the_existing_checkout_is_handled_before_the_old_pull():
+    text = ONE_SHOT.read_text(encoding="utf-8")
+    assert re.search(r'if _lm_update_existing_checkout; then\n\s+PULL_SUCCESS=true\n'
+                     r'\s+elif git pull --ff-only origin "\$CURRENT_BRANCH"', text)
+
+
+def test_the_one_shot_passes_the_channel_to_the_installer():
+    text = ONE_SHOT.read_text(encoding="utf-8")
+    assert 'LEDMATRIX_CHANNEL="${LEDMATRIX_CHANNEL:-}"' in text
+
+
+# -- first_time_install.sh records the chosen channel ----------------------------------
+
+CHANNEL_BEGIN = 'case "$UPDATE_CHANNEL" in'
+CHANNEL_END = 'set it from the General tab instead"\n    fi\nfi\n'
+
+
+def channel_block():
+    text = INSTALLER.read_text(encoding="utf-8")
+    start = text.index(CHANNEL_BEGIN)
+    return text[start: text.index(CHANNEL_END, start) + len(CHANNEL_END)]
+
+
+@pytest.mark.parametrize("auto_update, channel, expected", [
+    ("", "beta", {"enabled": False, "channel": "beta"}),
+    ("", "stable", {"enabled": False, "channel": "stable"}),
+    ("1", "", {"enabled": True, "channel": "stable"}),
+    ("", "", {"enabled": False, "channel": "stable"}),       # nothing asked: untouched
+    ("", "nightly", {"enabled": False, "channel": "stable"}),  # nonsense: untouched
+])
+def test_the_installer_writes_only_an_explicit_channel(tmp_path, auto_update, channel, expected):
+    (tmp_path / "config").mkdir()
+    config = tmp_path / "config" / "config.json"
+    config.write_text(json.dumps({"auto_update": {"enabled": False, "channel": "stable"}, "x": 1}))
+    script = (f'set -Eeuo pipefail\nPROJECT_ROOT_DIR="{tmp_path}"\nAUTO_UPDATE="{auto_update}"\n'
+              f'UPDATE_CHANNEL="{channel}"\n{channel_block()}')
+    result = subprocess.run(["bash", "-c", script], capture_output=True, text=True)
+    assert result.returncode == 0, result.stdout + result.stderr
+    data = json.loads(config.read_text())
+    assert data["auto_update"] == expected and data["x"] == 1
+
+
+def test_the_installer_accepts_beta_as_a_flag_and_from_the_environment():
+    text = INSTALLER.read_text(encoding="utf-8")
+    assert re.search(r"^\s*--beta\) UPDATE_CHANNEL=beta ;;", text, re.M)
+    assert 'UPDATE_CHANNEL=$(printf \'%s\' "${LEDMATRIX_CHANNEL:-}"' in text
+    assert "LEDMATRIX_CHANNEL=stable|beta" in text, "documented in --help"

test/test_refresh_units.py

@@ -0,0 +1,525 @@
+"""Updates refresh the installed systemd units: the root helper and its callers.
+
+An update moved the checkout, and with it systemd/*.service, but systemd runs
+the copies in /etc/systemd/system, which only the installer wrote. So unit
+settings added after a device was installed (#687's render-loop watchdog)
+never reached it. scripts/install/ledmatrix_refresh_units.py, installed
+root-owned as /usr/local/sbin/ledmatrix-refresh-units and granted to the web
+user by exact command line, now installs changed units after an update, and
+puts the previous ones back when the automatic update rolls back.
+
+The helper runs as root on input the web user can edit (the templates), so
+most of these are about what it refuses.
+"""
+import importlib.util
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT))
+
+_spec = importlib.util.spec_from_file_location(
+    'ledmatrix_refresh_units', ROOT / 'scripts' / 'install' / 'ledmatrix_refresh_units.py')
+ru = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(ru)
+
+from web_interface import unit_refresh  # noqa: E402
+
+try:
+    import pwd
+    # The web side checks the account exists, so use one that does.
+    WEB_USER = pwd.getpwuid(os.getuid()).pw_name
+except ImportError:  # Windows
+    WEB_USER = 'ledpi'
+
+# An account a hostile template switches the web unit to. Root, unless the
+# tests themselves run as root: then root *is* the web user and switching to
+# it changes nothing, so use another account.
+OTHER_USER = 'root' if WEB_USER != 'root' else 'nobody'
+
+
+class Host:
+    """A project checkout, an /etc/systemd/system, and a fake systemctl."""
+
+    def __init__(self, tmp_path, web_user=WEB_USER):
+        self.project = tmp_path / 'LEDMatrix'
+        shutil.copytree(ROOT / 'systemd', self.project / 'systemd')
+        self.systemd = tmp_path / 'etc-systemd-system'
+        self.systemd.mkdir()
+        self.backup = tmp_path / 'var-lib-ledmatrix' / 'unit-backup'
+        self.web_user = web_user
+        self.calls = []
+        self.path_active = True
+        self.root = True
+        for name in ru.UNITS:
+            self.install(name)
+
+    def template(self, name):
+        return (self.project / 'systemd' / name).read_text(encoding='utf-8')
+
+    def set_template(self, name, text):
+        (self.project / 'systemd' / name).write_text(text, encoding='utf-8', newline='\n')
+
+    def rendered(self, name, text=None):
+        user = 'root' if name == ru.DISPLAY_UNIT else self.web_user
+        return ru.render(text if text is not None else self.template(name), str(self.project), user)
+
+    def install(self, name, text=None):
+        (self.systemd / name).write_text(self.rendered(name, text), encoding='utf-8', newline='\n')
+
+    def installed(self, name):
+        path = self.systemd / name
+        return path.read_text(encoding='utf-8') if path.exists() else None
+
+    def run(self, args, **kwargs):
+        self.calls.append(list(args))
+        if args[:2] == ['systemctl', 'is-active']:
+            out = 'active\n' if self.path_active else 'inactive\n'
+            return subprocess.CompletedProcess(args, 0, stdout=out, stderr='')
+        return subprocess.CompletedProcess(args, 0, stdout='', stderr='')
+
+    def refresher(self, log=None):
+        return ru.Refresher(systemd_dir=str(self.systemd), backup_dir=str(self.backup), run=self.run,
+                            is_root=lambda: self.root, user_exists=lambda user: True,
+                            log=log or (lambda m: None))
+
+    @property
+    def reloads(self):
+        return self.calls.count(['systemctl', 'daemon-reload'])
+
+
+def watchdog_added(text):
+    """The kind of change #687 made: a new directive in [Service]."""
+    return text.replace('[Service]\n', '[Service]\nWatchdogSec=60\n', 1)
+
+
+@pytest.fixture
+def host(tmp_path):
+    return Host(tmp_path)
+
+
+# -- refresh ------------------------------------------------------------------
+
+def test_units_that_match_are_left_alone(host):
+    assert host.refresher().refresh() == []
+    assert host.reloads == 0
+
+
+def test_a_changed_template_is_installed_and_systemd_reloaded(host):
+    old = host.installed(ru.DISPLAY_UNIT)
+    host.set_template(ru.DISPLAY_UNIT, watchdog_added(host.template(ru.DISPLAY_UNIT)))
+
+    assert host.refresher().refresh() == [ru.DISPLAY_UNIT]
+    assert 'WatchdogSec=60' in host.installed(ru.DISPLAY_UNIT)
+    assert host.installed(ru.DISPLAY_UNIT) == host.rendered(ru.DISPLAY_UNIT)
+    assert host.reloads == 1
+    # Only the unit that changed is replaced, and the one it replaced is kept.
+    assert (host.backup / ru.DISPLAY_UNIT).read_text(encoding='utf-8') == old
+    assert json.loads((host.backup / ru.MANIFEST).read_text()) == {'units': [ru.DISPLAY_UNIT]}
+
+
+def test_the_web_unit_keeps_the_web_users_account(host):
+    host.set_template(ru.WEB_UNIT, watchdog_added(host.template(ru.WEB_UNIT)))
+    host.refresher().refresh()
+    assert ru.directive_values(host.installed(ru.WEB_UNIT), 'User') == [WEB_USER]
+    assert ru.directive_values(host.installed(ru.DISPLAY_UNIT), 'User') == ['root']
+
+
+def test_comment_only_changes_are_not_a_refresh(host):
+    host.set_template(ru.DISPLAY_UNIT, '# a new comment\n\n' + host.template(ru.DISPLAY_UNIT))
+    assert host.refresher().refresh() == []
+    assert host.reloads == 0
+
+
+def test_a_unit_that_was_never_installed_is_not_installed(host):
+    (host.systemd / ru.VERIFY_SERVICE).unlink()
+    (host.systemd / ru.VERIFY_PATH).unlink()
+    host.set_template(ru.VERIFY_SERVICE, watchdog_added(host.template(ru.VERIFY_SERVICE)))
+    host.refresher().refresh()
+    assert host.installed(ru.VERIFY_SERVICE) is None
+
+
+def test_a_changed_path_unit_is_restarted_so_it_watches_the_new_path(host):
+    host.set_template(ru.VERIFY_PATH, host.template(ru.VERIFY_PATH).replace(
+        '[Path]\n', '[Path]\nMakeDirectory=yes\n'))
+    host.refresher().refresh()
+    assert ['systemctl', 'restart', ru.VERIFY_PATH] in host.calls
+
+
+def test_it_must_run_as_root(host):
+    host.root = False
+    host.set_template(ru.DISPLAY_UNIT, watchdog_added(host.template(ru.DISPLAY_UNIT)))
+    with pytest.raises(ru.RefreshError, match='root'):
+        host.refresher().refresh()
+    assert 'WatchdogSec=60' not in host.installed(ru.DISPLAY_UNIT)
+
+
+def _failing_reload(host):
+    real = host.run
+    def run(args, **kwargs):
+        if args == ['systemctl', 'daemon-reload'] and host.reloads == 0:
+            host.calls.append(list(args))
+            return subprocess.CompletedProcess(args, 1, stdout='', stderr='boom')
+        return real(args, **kwargs)
+    return run
+
+
+def test_a_failed_daemon_reload_puts_the_old_units_back(host):
+    # The web side reports this as a failure and records no units_refreshed,
+    # so a rollback would not --restore: the helper must undo it itself.
+    old = host.installed(ru.DISPLAY_UNIT)
+    host.set_template(ru.DISPLAY_UNIT, watchdog_added(host.template(ru.DISPLAY_UNIT)))
+    host.run = _failing_reload(host)
+
+    with pytest.raises(ru.RefreshError):
+        host.refresher().refresh()
+    assert host.installed(ru.DISPLAY_UNIT) == old
+    assert host.reloads == 2  # the failed one, then one after putting it back
+    assert not (host.backup / ru.MANIFEST).exists()
+
+
+def test_a_failed_write_puts_back_the_units_already_written(host, monkeypatch):
+    old = {name: host.installed(name) for name in (ru.DISPLAY_UNIT, ru.WEB_UNIT)}
+    for name in old:
+        host.set_template(name, watchdog_added(host.template(name)))
+    refresher = host.refresher()
+    real_write = refresher._write_unit
+    writes = []
+
+    def write(name, text):
+        writes.append(name)
+        if len(writes) == 2:
+            raise OSError('disk full')
+        real_write(name, text)
+    monkeypatch.setattr(refresher, '_write_unit', write)
+
+    with pytest.raises(OSError):
+        refresher.refresh()
+    first = writes[0]
+    assert host.installed(first) == old[first]
+    assert all(host.installed(name) == old[name] for name in old)
+
+# -- what it refuses ------------------------------------------------------------
+
+@pytest.mark.parametrize('unit, edit', [
+    # The web interface's unit switched to root by a template edit.
+    (ru.WEB_UNIT, lambda t: t.replace('User=__USER__', f'User={OTHER_USER}')),
+    # The display's unit switched to another account.
+    (ru.DISPLAY_UNIT, lambda t: t.replace('User=root', 'User=nobody')),
+    # A second User= line.
+    (ru.VERIFY_SERVICE, lambda t: t.replace('[Service]\n', '[Service]\nUser=root\n', 1)),
+    # Run from somewhere else.
+    (ru.DISPLAY_UNIT, lambda t: t.replace('WorkingDirectory=__PROJECT_ROOT_DIR__', 'WorkingDirectory=/tmp')),
+    # A second User= written with spaces, which systemd accepts (last one wins).
+    # Placed in [Service] (before [Install]), where it is not a layout problem.
+    (ru.WEB_UNIT, lambda t: t.replace('\n[Install]', 'User = root\n\n[Install]')),
+    # The web user's User= moved to [Unit], where systemd ignores it (so root).
+    (ru.WEB_UNIT, lambda t: t.replace('User=__USER__\n', '').replace('[Unit]\n', '[Unit]\nUser=__USER__\n')),
+    # The User= line hidden inside a continued line, where systemd does not see it.
+    (ru.WEB_UNIT, lambda t: t.replace('User=__USER__\n', '').replace(
+        'Description=LED Matrix Web Interface Service\n',
+        'Description=LED Matrix Web Interface Service \\\\\nUser=__USER__\n')),
+    # A path unit that starts something else.
+    (ru.VERIFY_PATH, lambda t: t.replace('Unit=ledmatrix-update-verify.service', 'Unit=ledmatrix.service')),
+])
+def test_a_template_that_changes_who_or_where_is_refused_and_nothing_changes(host, unit, edit):
+    before = {name: host.installed(name) for name in ru.UNITS}
+    # A legitimate change alongside, which must not go in either.
+    host.set_template(ru.DISPLAY_UNIT, watchdog_added(host.template(ru.DISPLAY_UNIT)))
+    host.set_template(unit, edit(host.template(unit)))
+    with pytest.raises(ru.RefreshError):
+        host.refresher().refresh()
+    assert {name: host.installed(name) for name in ru.UNITS} == before
+    assert host.reloads == 0
+
+
+def test_the_project_folder_comes_from_the_installed_unit_not_the_caller(host, tmp_path):
+    # The installed display unit names the project; a WorkingDirectory that
+    # is not an existing absolute folder is refused before any template is read.
+    host.install(ru.DISPLAY_UNIT, host.template(ru.DISPLAY_UNIT).replace(
+        'WorkingDirectory=__PROJECT_ROOT_DIR__', 'WorkingDirectory=relative/path'))
+    with pytest.raises(ru.RefreshError, match='cannot be used'):
+        host.refresher().plan()
+
+
+def test_without_the_display_unit_installed_nothing_is_done(host):
+    (host.systemd / ru.DISPLAY_UNIT).unlink()
+    with pytest.raises(ru.RefreshError, match='not installed'):
+        host.refresher().plan()
+
+
+@pytest.mark.skipif(not hasattr(os, 'O_NOFOLLOW'), reason='POSIX only')
+def test_a_template_symlink_is_not_followed(host, tmp_path):
+    secret = tmp_path / 'secret'
+    secret.write_text(host.template(ru.DISPLAY_UNIT) + 'Environment=SECRET=1\n', encoding='utf-8')
+    target = host.project / 'systemd' / ru.DISPLAY_UNIT
+    target.unlink()
+    target.symlink_to(secret)
+    with pytest.raises(ru.RefreshError):
+        host.refresher().plan()
+
+
+@pytest.mark.skipif(not hasattr(os, 'O_NOFOLLOW'), reason='POSIX only')
+def test_a_symlinked_systemd_folder_is_not_followed(host, tmp_path):
+    elsewhere = tmp_path / 'elsewhere'
+    shutil.move(str(host.project / 'systemd'), str(elsewhere))
+    (host.project / 'systemd').symlink_to(elsewhere, target_is_directory=True)
+    with pytest.raises(ru.RefreshError):
+        host.refresher().plan()
+
+
+def test_an_oversized_template_is_refused(host):
+    host.set_template(ru.DISPLAY_UNIT, host.template(ru.DISPLAY_UNIT) + '#' * (ru.MAX_TEMPLATE_BYTES + 1))
+    with pytest.raises(ru.RefreshError, match='larger'):
+        host.refresher().plan()
+
+
+def test_main_leaves_the_callers_environment_alone(host):
+    """main() runs in-process in these tests; pinning PATH belongs to the installed program."""
+    before = os.environ.get('PATH')
+    ru.main(['ledmatrix-refresh-units', '--check'], refresher=host.refresher())
+    assert os.environ.get('PATH') == before
+
+
+@pytest.mark.parametrize('argv', [
+    ['--restore', 'x'], ['--refresh'], ['/etc/passwd'], ['--check', '--restore'], ['']])
+def test_any_other_command_line_is_refused(argv):
+    class Boom:
+        def __getattr__(self, name):
+            raise AssertionError('must not run')
+    assert ru.main(['ledmatrix-refresh-units', *argv], refresher=Boom()) == ru.EXIT_USAGE
+
+
+def test_main_reports_a_refusal_as_a_failure(host, capsys):
+    host.set_template(ru.WEB_UNIT, host.template(ru.WEB_UNIT).replace('User=__USER__', f'User={OTHER_USER}'))
+    assert ru.main(['ledmatrix-refresh-units'], refresher=host.refresher()) == ru.EXIT_FAILED
+    assert 'refusing' in capsys.readouterr().err
+
+
+# -- restore ------------------------------------------------------------------
+
+def test_restore_puts_back_exactly_what_the_refresh_replaced(host):
+    # A hand-edited installed unit: the rollback must give back this file,
+    # not a rendering of the old template.
+    hand_edited = host.installed(ru.DISPLAY_UNIT) + '# edited by hand\n'
+    (host.systemd / ru.DISPLAY_UNIT).write_text(hand_edited, encoding='utf-8', newline='\n')
+    untouched = host.installed(ru.WEB_UNIT)
+    host.set_template(ru.DISPLAY_UNIT, watchdog_added(host.template(ru.DISPLAY_UNIT)))
+    host.refresher().refresh()
+
+    assert host.refresher().restore() == [ru.DISPLAY_UNIT]
+    assert host.installed(ru.DISPLAY_UNIT) == hand_edited
+    assert host.installed(ru.WEB_UNIT) == untouched
+    assert host.reloads == 2
+    assert not (host.backup / ru.MANIFEST).exists(), 'a second restore must not repeat it'
+    assert host.refresher().restore() == []
+
+
+def test_restore_after_an_update_that_changed_no_units_restores_nothing(host):
+    host.set_template(ru.DISPLAY_UNIT, watchdog_added(host.template(ru.DISPLAY_UNIT)))
+    host.refresher().refresh()          # an earlier update...
+    newer = host.installed(ru.DISPLAY_UNIT)
+    host.refresher().refresh()          # ...then one that changed no units
+    assert host.refresher().restore() == []
+    assert host.installed(ru.DISPLAY_UNIT) == newer
+
+
+def test_restore_must_run_as_root(host):
+    host.root = False
+    with pytest.raises(ru.RefreshError, match='root'):
+        host.refresher().restore()
+
+
+# -- the real templates and install_service.sh ----------------------------------
+
+def test_every_shipped_template_passes_the_helpers_checks(tmp_path):
+    host = Host(tmp_path)
+    for name in ru.UNITS:
+        host.set_template(name, watchdog_added(host.template(name)) if name.endswith('.service')
+                          else host.template(name))
+    assert host.refresher().refresh() == sorted(n for n in ru.UNITS if n.endswith('.service'))
+
+
+@pytest.mark.skipif(not sys.platform.startswith('linux'), reason='runs sed as install_service.sh does')
+def test_rendering_matches_install_service_sh(tmp_path):
+    """Same text as the installer's sed, including characters sed treats specially."""
+    lib = ROOT / 'scripts' / 'install' / 'lib_systemd_render.sh'
+    for project in ('/home/pi/LEDMatrix', '/opt/led matrix&co'):
+        for name in ru.UNITS:
+            user = 'root' if name == ru.DISPLAY_UNIT else 'pi'
+            script = (f'source "{lib}"; R=$(sed_escape_replacement "$1"); U=$(sed_escape_replacement "$2"); '
+                      f'sed "s|__PROJECT_ROOT_DIR__|$R|g; s|__USER__|$U|g" "$3"')
+            out = subprocess.run(['bash', '-c', script, 'x', project, user, str(ROOT / 'systemd' / name)],
+                                 capture_output=True, text=True, check=True).stdout
+            template = (ROOT / 'systemd' / name).read_text(encoding='utf-8')
+            assert ru.render(template, project, user) == out, name
+
+
+def test_install_service_installs_the_helper_root_owned_at_the_granted_path():
+    text = (ROOT / 'scripts' / 'install' / 'install_service.sh').read_text(encoding='utf-8')
+    assert 'scripts/install/ledmatrix_refresh_units.py' in text
+    m = re.search(r'install -D -o root -g root -m 0755 "\$REFRESH_UNITS_SRC" "\$REFRESH_UNITS_DEST"', text)
+    assert m, 'install_service.sh must install the helper root:root 0755'
+    assert f'REFRESH_UNITS_DEST={ru.INSTALLED_PATH}' in text
+
+
+def test_every_caller_names_the_same_helper_path():
+    lib = (ROOT / 'scripts' / 'install' / 'lib_sudoers.sh').read_text(encoding='utf-8')
+    verifier = (ROOT / 'scripts' / 'utils' / 'auto_update_verify.py').read_text(encoding='utf-8')
+    assert f'LEDMATRIX_REFRESH_UNITS_PATH={ru.INSTALLED_PATH}' in lib
+    assert unit_refresh.HELPER_PATH == ru.INSTALLED_PATH
+    assert f"REFRESH_UNITS_PATH = '{ru.INSTALLED_PATH}'" in verifier
+    assert ru.INSTALLED_PATH.startswith('/usr/local/sbin/'), 'must live outside the user-owned checkout'
+
+
+def test_the_helper_imports_nothing_from_the_checkout():
+    source = (ROOT / 'scripts' / 'install' / 'ledmatrix_refresh_units.py').read_text(encoding='utf-8')
+    imports = re.findall(r'^\s*(?:from|import)\s+([\w.]+)', source, re.M)
+    assert not [m for m in imports if m.split('.')[0] in ('src', 'web_interface', 'scripts')]
+    assert source.startswith('#!/usr/bin/python3 -I\n'), 'isolated mode: no PYTHON* env, no user site'
+
+
+# -- the web interface's side (web_interface/unit_refresh.py) ---------------------
+
+class Sudo:
+    """sudo: refuses (``rc``/``stderr``), or runs the real helper as root against ``host``."""
+
+    def __init__(self, host=None, rc=0, stderr=''):
+        self.host, self.rc, self.stderr, self.calls = host, rc, stderr, []
+        self.as_root = False
+
+    def __call__(self, args, **kwargs):
+        self.calls.append(list(args))
+        if self.rc or self.host is None:
+            return subprocess.CompletedProcess(args, self.rc, stdout='', stderr=self.stderr)
+        lines = []
+        self.as_root = True
+        try:
+            rc = ru.main(['ledmatrix-refresh-units', *args[3:]], refresher=self.host.refresher(lines.append))
+        finally:
+            self.as_root = False
+        return subprocess.CompletedProcess(args, rc, stdout='\n'.join(lines) + '\n', stderr='')
+
+
+def _web(host, tmp_path, sudo, helper_installed=True):
+    helper = tmp_path / 'usr-local-sbin' / 'ledmatrix-refresh-units'
+    if helper_installed:
+        helper.parent.mkdir(exist_ok=True)
+        helper.write_text('#!/bin/true\n')
+    return unit_refresh.refresh_after_update(run=sudo, systemd_dir=str(host.systemd),
+                                             helper_path=str(helper))
+
+
+def _stale(host):
+    # The web side compares the installed units with the checkout's templates.
+    host.set_template(ru.DISPLAY_UNIT, watchdog_added(host.template(ru.DISPLAY_UNIT)))
+
+
+def test_web_side_does_nothing_when_the_units_match(host, tmp_path):
+    sudo = Sudo(host)
+    result = _web(host, tmp_path, sudo)
+    assert result['status'] == unit_refresh.CURRENT and sudo.calls == []
+    assert result['message'] == ''
+
+
+def test_web_side_runs_the_helper_through_sudo_with_no_arguments(host, tmp_path):
+    _stale(host)
+    sudo = Sudo(host)
+    result = _web(host, tmp_path, sudo)
+    assert result['status'] == unit_refresh.REFRESHED
+    assert result['units'] == [ru.DISPLAY_UNIT]
+    assert len(sudo.calls) == 1 and sudo.calls[0][:2] == ['sudo', '-n'] and len(sudo.calls[0]) == 3
+    assert 'ledmatrix.service' in result['message']
+    assert 'WatchdogSec=60' in host.installed(ru.DISPLAY_UNIT)
+
+
+@pytest.fixture
+def unreadable(monkeypatch, host):
+    """Installed units only root can read (install_service.sh used to leave them 0600)."""
+    real = ru._read_installed
+    sudo = Sudo(host)
+
+    def read(systemd_dir, name):
+        if not sudo.as_root:
+            raise ru.UnitsUnreadable(f'cannot read the installed {name}: Permission denied')
+        return real(systemd_dir, name)
+    monkeypatch.setattr(ru, '_read_installed', read)
+    monkeypatch.setattr(unit_refresh, '_load_helper', lambda path=None: ru)
+    return sudo
+
+
+def test_web_side_lets_the_helper_decide_when_it_cannot_read_the_units(host, tmp_path, unreadable):
+    _stale(host)
+    result = _web(host, tmp_path, unreadable)
+    assert len(unreadable.calls) == 1
+    assert result['status'] == unit_refresh.REFRESHED and result['units'] == [ru.DISPLAY_UNIT]
+
+
+def test_web_side_unreadable_and_already_current_is_current(host, tmp_path, unreadable):
+    result = _web(host, tmp_path, unreadable)
+    assert result['status'] == unit_refresh.CURRENT and result['message'] == ''
+
+
+def test_web_side_unreadable_without_the_rule_asks_for_a_reinstall(host, tmp_path, unreadable, caplog):
+    unreadable.rc, unreadable.stderr = 1, 'sudo: a password is required'
+    result = _web(host, tmp_path, unreadable)
+    assert result['status'] == unit_refresh.NEEDS_REINSTALL
+    assert 'could not be checked' in result['message']
+
+
+def test_web_side_trusts_the_helper_about_what_changed(host, tmp_path):
+    """An older installed helper that renders differently changed nothing: nothing to roll back."""
+    _stale(host)
+
+    def older_helper(args, **kwargs):
+        return subprocess.CompletedProcess(args, 0, stdout='units: up to date\n', stderr='')
+    result = _web(host, tmp_path, older_helper)
+    assert result['status'] == unit_refresh.CURRENT
+
+
+def test_web_side_without_the_helper_asks_for_a_reinstall(host, tmp_path, caplog):
+    _stale(host)
+    sudo = Sudo()
+    result = _web(host, tmp_path, sudo, helper_installed=False)
+    assert result['status'] == unit_refresh.NEEDS_REINSTALL and sudo.calls == []
+    assert 'first_time_install.sh' in result['message']
+    assert 'reinstall' in caplog.text
+
+
+@pytest.mark.parametrize('stderr', [
+    'sudo: a password is required',
+    'Sorry, user ledpi is not allowed to run \'/usr/local/sbin/ledmatrix-refresh-units\' as root on ledpi.',
+])
+def test_web_side_without_the_sudo_rule_asks_for_a_reinstall(host, tmp_path, stderr, caplog):
+    _stale(host)
+    result = _web(host, tmp_path, Sudo(rc=1, stderr=stderr))
+    assert result['status'] == unit_refresh.NEEDS_REINSTALL
+    assert 'configure_web_sudo.sh' in result['message']
+    assert 'no sudo rule' in caplog.text
+
+
+def test_web_side_reports_a_helper_refusal_as_a_failure(host, tmp_path):
+    _stale(host)
+    result = _web(host, tmp_path, Sudo(rc=1, stderr='ledmatrix-refresh-units: systemd/x refusing'))
+    assert result['status'] == unit_refresh.FAILED
+    assert 'refusing' in result['message']
+
+
+def test_web_side_on_a_machine_without_the_units_does_nothing(tmp_path):
+    sudo = Sudo()
+    result = unit_refresh.refresh_after_update(run=sudo, systemd_dir=str(tmp_path))
+    assert result['status'] == unit_refresh.SKIPPED and sudo.calls == []
+
+
+def test_web_side_never_raises_on_a_broken_template(host, tmp_path):
+    host.set_template(ru.WEB_UNIT, host.template(ru.WEB_UNIT).replace('User=__USER__', f'User={OTHER_USER}'))
+    sudo = Sudo()
+    result = _web(host, tmp_path, sudo)
+    assert result['status'] == unit_refresh.FAILED and sudo.calls == []

test/test_update_prompts_restart.py

@@ -82,3 +82,37 @@ class TestRestartIsRequestedWhenCodeChanged:
             data = _pull(client)
         assert data['status'] == 'error'
         assert data['restart_required'] is False
+
+
+class TestUnitsAreRefreshedWithTheCode:
+    """New code may bring new systemd unit settings; installing them is part of the update."""
+
+    @pytest.fixture
+    def refresh(self, monkeypatch):
+        from web_interface import unit_refresh
+        calls = []
+
+        def fake():
+            calls.append(True)
+            return {'status': 'refreshed', 'message': 'Service settings updated (ledmatrix.service).',
+                    'units': ['ledmatrix.service']}
+        monkeypatch.setattr(unit_refresh, 'refresh_after_update', fake)
+        return calls
+
+    def test_an_update_that_moved_head_refreshes_the_units(self, client, refresh):
+        with patch.object(mod.subprocess, 'run', _git(['aaa111', 'bbb222'])):
+            data = _pull(client)
+        assert refresh == [True]
+        assert data['unit_refresh']['status'] == 'refreshed'
+        assert 'Service settings updated' in data['message']
+
+    def test_nothing_new_touches_no_units(self, client, refresh):
+        with patch.object(mod.subprocess, 'run',
+                          _git(['aaa111', 'aaa111'], pull_out='Already up to date.\n')):
+            data = _pull(client)
+        assert refresh == [] and data['unit_refresh'] is None
+
+    def test_a_failed_pull_touches_no_units(self, client, refresh):
+        with patch.object(mod.subprocess, 'run', _git(['aaa111'], pull_rc=1)):
+            data = _pull(client)
+        assert refresh == [] and data['unit_refresh'] is None

test/test_web_sudoers_installers_agree.py

@@ -54,6 +54,10 @@ EXPECTED_GRANTS = frozenset({
     ("NOPASSWD:", "$SYSTEMCTL_PATH restart ledmatrix-web.service"),
     ("NOPASSWD:", "$BASH_PATH $PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh *"),
     ("NOPASSWD:", "$BASH_PATH $PROJECT_ROOT/scripts/fix_perms/safe_pip_install.sh *"),
+    # The unit refresh helper (scripts/install/ledmatrix_refresh_units.py),
+    # with no arguments (`""`; RULE below drops the closing quote) or --restore.
+    ("NOPASSWD:", '$LEDMATRIX_REFRESH_UNITS_PATH "'),
+    ("NOPASSWD:", "$LEDMATRIX_REFRESH_UNITS_PATH --restore"),
     ("NOPASSWD:NOEXEC:", "$JOURNALCTL_PATH -u ledmatrix.service *"),
     ("NOPASSWD:NOEXEC:", "$JOURNALCTL_PATH -u ledmatrix *"),
     ("NOPASSWD:NOEXEC:", "$JOURNALCTL_PATH -t ledmatrix *"),
@@ -145,7 +149,8 @@ def test_installer_call_renders_the_expected_rules(installer, tmp_path):
             rendered.add((m.group(2), m.group(3)))
     subst = {"$SYSTEMCTL_PATH": "/x/systemctl", "$BASH_PATH": "/x/bash",
              "$REBOOT_PATH": "/x/reboot", "$POWEROFF_PATH": "/x/poweroff",
-             "$JOURNALCTL_PATH": "/x/journalctl", "$PROJECT_ROOT": "/srv/led root"}
+             "$JOURNALCTL_PATH": "/x/journalctl", "$PROJECT_ROOT": "/srv/led root",
+             "$LEDMATRIX_REFRESH_UNITS_PATH": "/usr/local/sbin/ledmatrix-refresh-units"}
     expected = set()
     for tags, command in EXPECTED_GRANTS:
         for var, value in subst.items():

web_interface/auto_update.py

@@ -603,6 +603,9 @@ class AutoUpdater:
             'release': info.get('release') if new_head == info.get('upstream_head') else None,
             'display_was_active': display_was_active,
             'dependency_failures': list(core.get('dependency_failures') or []),
+            # The update installed new systemd units: a rollback puts the
+            # previous ones back (ledmatrix-refresh-units --restore).
+            'units_refreshed': (core.get('unit_refresh') or {}).get('status') == 'refreshed',
             'created_at': self.clock(),
         }
 

web_interface/blueprints/api_v3/system.py

@@ -415,6 +415,7 @@ def _perform_core_update_locked(stash_local_changes=True):
     # date" is a success too, and prompting for a restart then would
     # train users to ignore the prompt.
     code_changed = False
+    unit_result = None
     # Requirement files whose install failed. The automatic updater refuses
     # to restart onto code whose dependencies did not install.
     dependency_failures = []
@@ -537,6 +538,14 @@ def _perform_core_update_locked(stash_local_changes=True):
                     )
             except (OSError, RuntimeError) as purge_err:
                 logger.warning("Post-update plugin purge failed: %s", purge_err)
+        # The new code may come with new systemd unit settings (systemd/*).
+        # Install them now, so the restart that follows runs under them; the
+        # automatic update's rollback puts the old ones back.
+        if code_changed:
+            from web_interface import unit_refresh
+            unit_result = unit_refresh.refresh_after_update()
+            if unit_result['message']:
+                pull_message += " " + unit_result['message']
     else:
         logger.warning("git pull failed (returncode=%d): %s", result.returncode, result.stderr)
         # Show git's own first line: "check logs" leaves the user with
@@ -556,6 +565,8 @@ def _perform_core_update_locked(stash_local_changes=True):
         'restart_required': bool(result.returncode == 0 and code_changed),
         'dependency_failures': dependency_failures,
         'channel': channel.channel,
+        # web_interface/unit_refresh.py's result, or None when no code changed.
+        'unit_refresh': unit_result,
     }
 
 

web_interface/unit_refresh.py

@@ -0,0 +1,133 @@
+"""After an update, bring the installed systemd units in line with the new templates.
+
+An update moves the checkout, and with it systemd/*.service, but systemd runs
+the copies in /etc/systemd/system, which used to be written only by the
+installer. Settings added to a template (the render-loop watchdog, a memory
+limit) therefore never reached a device that was already installed.
+
+Updates now run the root-owned helper /usr/local/sbin/ledmatrix-refresh-units
+(scripts/install/ledmatrix_refresh_units.py) through sudo when the rendered
+units differ from the installed ones. The services pick the new units up at
+the restart that follows the update. The automatic update's rollback runs the
+same helper with ``--restore`` (scripts/utils/auto_update_verify.py).
+
+The sudo rule is written by the installer, so a device installed before it
+existed cannot run the helper. That is reported, not fatal: the update
+itself stands, and the message says to re-run the installer once, the same
+remedy as the display's startup "unit drift" warning.
+"""
+import importlib.util
+import logging
+import os
+import subprocess  # nosec B404 - list-form argv only, no shell  # nosemgrep
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+HELPER_SOURCE = PROJECT_ROOT / 'scripts' / 'install' / 'ledmatrix_refresh_units.py'
+#: The installed, root-owned copy that sudo is allowed to run.
+HELPER_PATH = '/usr/local/sbin/ledmatrix-refresh-units'
+SYSTEMD_DIR = '/etc/systemd/system'
+TIMEOUT_SECONDS = 90
+
+#: Outcomes (``result['status']``).
+CURRENT = 'current'            # nothing differs
+REFRESHED = 'refreshed'        # installed the new units
+NEEDS_REINSTALL = 'needs_reinstall'  # the helper or its sudo rule is missing
+FAILED = 'failed'              # the helper ran and refused or failed
+SKIPPED = 'skipped'            # not a systemd install (dev machine, emulator)
+
+REINSTALL_HINT = ('Run "sudo ./first_time_install.sh" in the LEDMatrix folder once '
+                  '(or "sudo ./scripts/install/install_service.sh" followed by '
+                  '"./scripts/install/configure_web_sudo.sh") so updates can apply them.')
+
+#: sudo's words for "this command line is not allowed without a password".
+_SUDO_REFUSED = ('a password is required', 'is not allowed to run', 'no tty present',
+                 'a terminal is required', 'command not found')
+
+
+def _load_helper(path=HELPER_SOURCE):
+    spec = importlib.util.spec_from_file_location('ledmatrix_refresh_units', str(path))
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def stale_units(systemd_dir=None, helper_source=None):
+    """Installed units whose rendering from the checkout differs. Needs no root.
+
+    Returns a sorted list, or None when this is not a systemd install.
+    Raises the helper's RefreshError when a unit cannot be rendered safely.
+    """
+    systemd_dir = systemd_dir or SYSTEMD_DIR
+    helper_source = helper_source or HELPER_SOURCE
+    if not os.path.isfile(os.path.join(systemd_dir, 'ledmatrix.service')):
+        return None
+    helper = _load_helper(helper_source)
+    return sorted(helper.Refresher(systemd_dir=systemd_dir).plan())
+
+
+def _result(status, message, units=()):
+    return {'status': status, 'message': message, 'units': list(units)}
+
+
+def refresh_after_update(run=None, systemd_dir=None, helper_source=None, helper_path=None):
+    """Install the units an update changed. Never raises.
+
+    Returns ``{'status', 'message', 'units'}``; ``message`` is '' when there
+    is nothing to tell the user. The defaults are the module's constants,
+    read at call time (the test suite points SYSTEMD_DIR away from the host).
+    """
+    run = run or subprocess.run
+    helper_path = helper_path or HELPER_PATH
+    try:
+        stale = stale_units(systemd_dir, helper_source)
+    except Exception as e:  # a broken template must not fail the update itself
+        if type(e).__name__ != 'UnitsUnreadable':
+            logger.warning("Could not compare the installed systemd units with the new templates: %s", e)
+            return _result(FAILED, f'The service settings could not be checked: {e}.')
+        # Units installed mode 0600 (install_service.sh run on its own, before
+        # it set 0644): only root can compare them, so let the helper decide.
+        stale = []
+        unknown = True
+    else:
+        unknown = False
+        if stale is None:
+            return _result(SKIPPED, '')
+        if not stale:
+            return _result(CURRENT, '')
+
+    names = ', '.join(stale) or 'the LEDMatrix units'
+    changes = (f'This update changes service settings ({names}) that are not applied yet. ' if not unknown
+               else 'Service settings this update may change could not be checked or applied. ')
+    if not os.path.isfile(helper_path):
+        logger.warning("Updates cannot install systemd unit changes (%s): %s is not installed; "
+                       "they take effect only after a reinstall. %s", names, helper_path, REINSTALL_HINT)
+        return _result(NEEDS_REINSTALL, changes + REINSTALL_HINT, stale)
+    try:
+        result = run(['sudo', '-n', helper_path], capture_output=True, text=True,
+                     timeout=TIMEOUT_SECONDS)
+    except (subprocess.SubprocessError, OSError) as e:
+        logger.warning("Refreshing the systemd units failed: %s", e)
+        return _result(FAILED, f'Updating the service settings ({names}) failed: {e}.', stale)
+    if result.returncode == 0:
+        # The helper says what it did: "units refreshed: a b" or "units: up to date".
+        done = next((line.split(':', 1)[1].split() for line in (result.stdout or '').splitlines()
+                     if line.startswith('units refreshed:')), None)
+        if not done:
+            # Nothing replaced, so nothing for a rollback to restore.
+            if stale:
+                logger.warning("The unit helper found nothing to change in %s; the installed "
+                               "helper may be older than this version", names)
+            return _result(CURRENT, '')
+        logger.info("Refreshed systemd units after the update: %s", ', '.join(done))
+        return _result(REFRESHED, f'Service settings updated ({", ".join(done)}).', done)
+    detail = (result.stderr or result.stdout or '').strip()
+    if any(phrase in detail.lower() for phrase in _SUDO_REFUSED):
+        logger.warning("Updates cannot install systemd unit changes (%s): no sudo rule for %s; "
+                       "they take effect only after a reinstall. %s", names, helper_path, REINSTALL_HINT)
+        return _result(NEEDS_REINSTALL, changes + REINSTALL_HINT, stale)
+    logger.warning("Refreshing the systemd units failed (exit %s): %s", result.returncode, detail)
+    return _result(FAILED, f'Updating the service settings ({names}) failed: {detail or "unknown error"}.',
+                   stale)