mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-04 14:25:08 +00:00
feat(web): weekly automatic updates with health check and rollback (#581)
* feat(web): weekly automatic updates with health check and rollback A General-tab toggle (off by default) checks for and installs LEDMatrix and plugin updates once a week, overnight in the configured timezone. - Pre-update checks skip (and report) instead of forcing: local edits or commits, merge/live rebase, no upstream, low disk, missing health check, or a version that was already rolled back. An abandoned rebase (HEAD back on a branch) is cleared, since it would otherwise block every pull. - The pull reuses the Update Code path (now perform_core_update(), which reports dependency install failures as data). - ledmatrix-update-verify.service, started via a .path unit from a request file, restarts the services from its own cgroup, requires them to come up and stay up, and otherwise resets to the previous commit and reinstalls the previous requirements. It runs a copy of the checker taken before the pull. - No SSH needed: switching the toggle on restarts the display service, which (as root) installs the two units from the repo templates for the web user. first_time_install.sh installs them too and takes --enable-auto-update / LEDMATRIX_AUTO_UPDATE (passed through by one-shot-install.sh). - Plugins update after the code passes its check; failures, blocks and rollbacks raise an Overview banner and show under the toggle. Tested end to end on a Pi: web-UI setup, a good update, a broken web service and a broken display (both rolled back), a blocked local edit, and an abandoned rebase found on the device. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(auto-update): address static-analysis findings - Replace the subprocess.CompletedProcess the verifier fabricated for a command that could not start with a plain namedtuple; nothing is executed there, but the scanner flags any CompletedProcess built from variables. - Mark the subprocess imports with the repo's standard B404 annotation (all calls are list-form argv, no shell). - Mark the rollback-failed message as not SQL (B608 matched its wording). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(auto-update): CI failures on Linux - Keep the setup result when chown fails. CI runs as a non-root user, where chown to the web user raises; that discarded the result file, so the General tab would never learn whether setup worked. Regression test added. - Register the two new /api/v3/system/auto-update routes in the URL map snapshot. - Use utility classes app.css defines (space-y-1, hover:text-red-600). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(auto-update): address review feedback - Health check: a failed restart command no longer lets the check run against the still-running old process; it counts as a failure (and after a rollback, as a failed rollback). An unreadable restart count is never treated as stable, since a crash loop looks healthy between attempts. - Installer writes the auto_update setting to a temp file and swaps it in, keeping mode and owner, so a running config watcher never reads a truncated config.json. - Verify unit quotes its command-line paths (install folders with spaces); setup refuses folder names systemd would reinterpret (%, quotes, backslashes, control characters) and says so on the General tab. - The auto-update status route no longer returns exception text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(auto-update): keep error detail in the status route's 500 test_web_error_detail requires every 5xx handler to log the traceback and return describe_exception(e), which redacts credentials, so failures are diagnosable from the web UI. Dropping it for CodeQL broke that policy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(auto-update): dismiss route rejects non-object JSON with 400 A JSON array or scalar body made `.get('alert_id')` raise, returning 500. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(auto-update): let the app-wide handler answer status-route errors CodeQL (py/stack-trace-exposure, #709) flagged the route's own except, which returned describe_exception(e). web_interface/app.py's error handler already logs the traceback and returns the same redacted detail for any unhandled exception, so the local copy is removed: same response, no new exception-to-response flow, and test_web_error_detail's policy still holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Chuck
@@ -0,0 +1,221 @@
|
||||
"""Install the automatic-update health check, from the display service.
|
||||
|
||||
The weekly updater (web_interface/auto_update.py) will not update LEDMatrix
|
||||
code unless ledmatrix-update-verify.path and .service are installed: they
|
||||
restart the services after an update and roll it back if the device is
|
||||
unhealthy. Installing units takes root and the web interface is not root, and
|
||||
"SSH in and run an installer" means most people never get updates with a
|
||||
safety net.
|
||||
|
||||
The display service already runs this repository's code as root, so it
|
||||
installs them -- but only while the user has automatic updates turned on, only
|
||||
these two units, rendered from the repository's templates for the web
|
||||
interface's own user, and it reports what happened in
|
||||
data/auto_update_setup.json for the General tab. It grants nothing new: the
|
||||
units run as the web user, who can already change the code this process runs.
|
||||
|
||||
Called at display startup; the web interface restarts the display service
|
||||
when the toggle is switched on, so setup happens straight away. Refreshing a
|
||||
unit whose template changed happens the same way, which is why this compares
|
||||
content rather than only checking that the files exist.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
SYSTEMD_DIR = Path('/etc/systemd/system')
|
||||
SERVICE_UNIT = 'ledmatrix-update-verify.service'
|
||||
PATH_UNIT = 'ledmatrix-update-verify.path'
|
||||
UNITS = (SERVICE_UNIT, PATH_UNIT)
|
||||
WEB_UNIT = 'ledmatrix-web.service'
|
||||
RESULT_REL = Path('data') / 'auto_update_setup.json'
|
||||
_USER_RE = re.compile(r'^[a-z_][a-z0-9_-]{0,31}$')
|
||||
|
||||
|
||||
class SetupError(Exception):
|
||||
"""A reason setup cannot proceed, worded for the General tab."""
|
||||
|
||||
|
||||
def _is_root():
|
||||
return hasattr(os, 'geteuid') and os.geteuid() == 0
|
||||
|
||||
|
||||
def _lookup_ids(user):
|
||||
try:
|
||||
import pwd
|
||||
entry = pwd.getpwnam(user)
|
||||
return entry.pw_uid, entry.pw_gid
|
||||
except (ImportError, KeyError):
|
||||
return None
|
||||
|
||||
|
||||
def _directive(text, key):
|
||||
match = re.search(rf'^{key}=(.*)$', text or '', re.M)
|
||||
return match.group(1).strip() if match else None
|
||||
|
||||
|
||||
def _read(path):
|
||||
try:
|
||||
return Path(path).read_text(encoding='utf-8')
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def is_enabled(config):
|
||||
return bool((config.get('auto_update') or {}).get('enabled', False))
|
||||
|
||||
|
||||
class UpdateHelperSetup:
|
||||
def __init__(self, project_root=PROJECT_ROOT, systemd_dir=SYSTEMD_DIR, run=subprocess.run,
|
||||
is_root=_is_root, lookup_ids=_lookup_ids, clock=time.time):
|
||||
self.project_root = Path(project_root)
|
||||
self.systemd_dir = Path(systemd_dir)
|
||||
self.run = run
|
||||
self.is_root = is_root
|
||||
self.lookup_ids = lookup_ids
|
||||
self.clock = clock
|
||||
self.result_file = self.project_root / RESULT_REL
|
||||
self._web_ids = None
|
||||
|
||||
def _systemctl(self, *args):
|
||||
return self.run(['systemctl', *args], capture_output=True, text=True, timeout=60)
|
||||
|
||||
def _check(self, result, what):
|
||||
if result.returncode != 0:
|
||||
raise SetupError(f'"{what}" failed: {(result.stderr or result.stdout or "").strip()}')
|
||||
|
||||
def path_active(self):
|
||||
try:
|
||||
return self._systemctl('is-active', PATH_UNIT).stdout.strip() == 'active'
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
|
||||
def ensure(self, config):
|
||||
"""Install or refresh the units while automatic updates are on.
|
||||
|
||||
Returns the result recorded for the General tab, or None when there
|
||||
was nothing to do (updates off, or not a systemd host at all).
|
||||
"""
|
||||
if not is_enabled(config) or not self.systemd_dir.is_dir():
|
||||
return None
|
||||
try:
|
||||
changed = self._install()
|
||||
except SetupError as e:
|
||||
return self._report('failed', str(e))
|
||||
except (OSError, subprocess.SubprocessError) as e:
|
||||
return self._report('failed', f'Could not install the update health check: {e}')
|
||||
if changed:
|
||||
return self._report('installed', 'Installed the update health check.')
|
||||
return self._report('installed', 'The update health check is installed.', quiet=True)
|
||||
|
||||
def _install(self):
|
||||
if not self.is_root():
|
||||
raise SetupError('The display service is not running as root, so it cannot install the '
|
||||
'update health check. Run "sudo ./scripts/install/install_web_service.sh" once.')
|
||||
|
||||
web_text = _read(self.systemd_dir / WEB_UNIT)
|
||||
if web_text is None:
|
||||
raise SetupError('The web interface service (ledmatrix-web.service) is not installed.')
|
||||
user = _directive(web_text, 'User') or 'root'
|
||||
ids = self.lookup_ids(user) if _USER_RE.match(user) else None
|
||||
if ids is None:
|
||||
raise SetupError(f'The web interface runs as "{user}", which is not a usable account.')
|
||||
self._web_ids = ids
|
||||
workdir = _directive(web_text, 'WorkingDirectory')
|
||||
if not workdir or Path(workdir).resolve() != self.project_root.resolve():
|
||||
raise SetupError(f'The web interface service runs from {workdir or "an unknown folder"}, '
|
||||
f'not {self.project_root}.')
|
||||
# Spaces are fine -- the templates quote every command-line path --
|
||||
# but systemd expands % specifiers, and a quote, backslash or line
|
||||
# break would be reinterpreted in a unit file. (On Windows, where the
|
||||
# tests also run, a backslash is the path separator, not a name.)
|
||||
root_text = str(self.project_root)
|
||||
unsafe = set('%"') | ({'\\'} if os.sep == '/' else set())
|
||||
if any(ch in unsafe or ord(ch) < 32 for ch in root_text):
|
||||
raise SetupError(f'LEDMatrix is installed in {root_text!r}, a folder name systemd cannot use '
|
||||
'in a unit file. Move it to a path without %, quotes, backslashes or '
|
||||
'control characters.')
|
||||
|
||||
rendered = {}
|
||||
for name in UNITS:
|
||||
template = _read(self.project_root / 'systemd' / name)
|
||||
if template is None:
|
||||
raise SetupError(f'The unit template systemd/{name} is missing.')
|
||||
rendered[name] = (template.replace('__PROJECT_ROOT_DIR__', str(self.project_root))
|
||||
.replace('__USER__', user))
|
||||
# The templates are ordinary repository files. Whatever they say,
|
||||
# this root process only installs a service that runs as the web user
|
||||
# and a path unit that starts exactly that service.
|
||||
if _directive(rendered[SERVICE_UNIT], 'User') != user:
|
||||
raise SetupError(f'systemd/{SERVICE_UNIT} does not run as the web interface user; '
|
||||
'refusing to install it.')
|
||||
if _directive(rendered[PATH_UNIT], 'Unit') != SERVICE_UNIT:
|
||||
raise SetupError(f'systemd/{PATH_UNIT} does not start {SERVICE_UNIT}; refusing to install it.')
|
||||
|
||||
changed = [name for name in UNITS if _read(self.systemd_dir / name) != rendered[name]]
|
||||
for name in changed:
|
||||
self._write_unit(self.systemd_dir / name, rendered[name])
|
||||
if changed:
|
||||
self._check(self._systemctl('daemon-reload'), 'systemctl daemon-reload')
|
||||
self._check(self._systemctl('enable', PATH_UNIT), f'systemctl enable {PATH_UNIT}')
|
||||
self._check(self._systemctl('restart', PATH_UNIT), f'systemctl restart {PATH_UNIT}')
|
||||
elif not self.path_active():
|
||||
self._check(self._systemctl('enable', '--now', PATH_UNIT), f'systemctl enable --now {PATH_UNIT}')
|
||||
changed = [PATH_UNIT]
|
||||
if not self.path_active():
|
||||
raise SetupError(f'{PATH_UNIT} did not start; see "journalctl -u {PATH_UNIT}".')
|
||||
return bool(changed)
|
||||
|
||||
def _write_unit(self, path, text):
|
||||
fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f'.{path.name}.')
|
||||
try:
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as f:
|
||||
f.write(text)
|
||||
os.chmod(tmp, 0o644)
|
||||
os.replace(tmp, path)
|
||||
except BaseException:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
def _report(self, status, message, quiet=False):
|
||||
previous = None
|
||||
try:
|
||||
previous = json.loads(_read(self.result_file) or 'null')
|
||||
except ValueError:
|
||||
pass
|
||||
if quiet and isinstance(previous, dict) and previous.get('status') == status:
|
||||
return previous # nothing new; don't rewrite it on every boot
|
||||
result = {'status': status, 'message': message, 'at': self.clock()}
|
||||
(logger.info if status == 'installed' else logger.warning)("Automatic update setup: %s", message)
|
||||
try:
|
||||
self.result_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(dir=str(self.result_file.parent), prefix='.auto_update_setup_')
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as f:
|
||||
json.dump(result, f, indent=2)
|
||||
os.chmod(tmp, 0o644)
|
||||
if self._web_ids and hasattr(os, 'chown'):
|
||||
# Only root can give the file away; the result is readable
|
||||
# (0644) either way, so a failed chown must not lose it.
|
||||
try:
|
||||
os.chown(tmp, *self._web_ids)
|
||||
except OSError:
|
||||
pass
|
||||
os.replace(tmp, self.result_file)
|
||||
except OSError as e:
|
||||
logger.warning("Could not record automatic update setup result: %s", e)
|
||||
return result
|
||||
|
||||
|
||||
def ensure_update_helper(config):
|
||||
return UpdateHelperSetup().ensure(config)
|
||||
@@ -119,6 +119,15 @@ class DisplayController:
|
||||
# validator.raise_on_errors() # Uncomment to fail fast on errors
|
||||
except Exception as e:
|
||||
logger.warning(f"Startup validation could not be completed: {e}")
|
||||
|
||||
# Automatic updates need their health-check units, and this is the
|
||||
# one root process running project code, so it installs them while
|
||||
# automatic updates are on. See src/auto_update_setup.py.
|
||||
try:
|
||||
from src.auto_update_setup import ensure_update_helper
|
||||
ensure_update_helper(self.config)
|
||||
except Exception as e:
|
||||
logger.warning("Automatic update setup could not be completed: %s", e)
|
||||
|
||||
config_time = time.time()
|
||||
self.display_manager = DisplayManager(self.config)
|
||||
|
||||
@@ -81,6 +81,8 @@ class StartupValidator:
|
||||
_UNITS = (
|
||||
("systemd/ledmatrix.service", "/etc/systemd/system/ledmatrix.service"),
|
||||
("systemd/ledmatrix-web.service", "/etc/systemd/system/ledmatrix-web.service"),
|
||||
("systemd/ledmatrix-update-verify.service", "/etc/systemd/system/ledmatrix-update-verify.service"),
|
||||
("systemd/ledmatrix-update-verify.path", "/etc/systemd/system/ledmatrix-update-verify.path"),
|
||||
)
|
||||
|
||||
def _validate_systemd_units(self) -> None:
|
||||
|
||||
Reference in New Issue
Block a user