feat(web): weekly automatic updates with health check and rollback (#581)

* feat(web): weekly automatic updates with health check and rollback

A General-tab toggle (off by default) checks for and installs LEDMatrix and
plugin updates once a week, overnight in the configured timezone.

- Pre-update checks skip (and report) instead of forcing: local edits or
  commits, merge/live rebase, no upstream, low disk, missing health check, or
  a version that was already rolled back. An abandoned rebase (HEAD back on a
  branch) is cleared, since it would otherwise block every pull.
- The pull reuses the Update Code path (now perform_core_update(), which
  reports dependency install failures as data).
- ledmatrix-update-verify.service, started via a .path unit from a request
  file, restarts the services from its own cgroup, requires them to come up
  and stay up, and otherwise resets to the previous commit and reinstalls the
  previous requirements. It runs a copy of the checker taken before the pull.
- No SSH needed: switching the toggle on restarts the display service, which
  (as root) installs the two units from the repo templates for the web user.
  first_time_install.sh installs them too and takes --enable-auto-update /
  LEDMATRIX_AUTO_UPDATE (passed through by one-shot-install.sh).
- Plugins update after the code passes its check; failures, blocks and
  rollbacks raise an Overview banner and show under the toggle.

Tested end to end on a Pi: web-UI setup, a good update, a broken web service
and a broken display (both rolled back), a blocked local edit, and an
abandoned rebase found on the device.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(auto-update): address static-analysis findings

- Replace the subprocess.CompletedProcess the verifier fabricated for a
  command that could not start with a plain namedtuple; nothing is executed
  there, but the scanner flags any CompletedProcess built from variables.
- Mark the subprocess imports with the repo's standard B404 annotation (all
  calls are list-form argv, no shell).
- Mark the rollback-failed message as not SQL (B608 matched its wording).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): CI failures on Linux

- Keep the setup result when chown fails. CI runs as a non-root user, where
  chown to the web user raises; that discarded the result file, so the
  General tab would never learn whether setup worked. Regression test added.
- Register the two new /api/v3/system/auto-update routes in the URL map
  snapshot.
- Use utility classes app.css defines (space-y-1, hover:text-red-600).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): address review feedback

- Health check: a failed restart command no longer lets the check run
  against the still-running old process; it counts as a failure (and after a
  rollback, as a failed rollback). An unreadable restart count is never
  treated as stable, since a crash loop looks healthy between attempts.
- Installer writes the auto_update setting to a temp file and swaps it in,
  keeping mode and owner, so a running config watcher never reads a
  truncated config.json.
- Verify unit quotes its command-line paths (install folders with spaces);
  setup refuses folder names systemd would reinterpret (%, quotes,
  backslashes, control characters) and says so on the General tab.
- The auto-update status route no longer returns exception text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): keep error detail in the status route's 500

test_web_error_detail requires every 5xx handler to log the traceback and
return describe_exception(e), which redacts credentials, so failures are
diagnosable from the web UI. Dropping it for CodeQL broke that policy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): dismiss route rejects non-object JSON with 400

A JSON array or scalar body made `.get('alert_id')` raise, returning 500.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(auto-update): let the app-wide handler answer status-route errors

CodeQL (py/stack-trace-exposure, #709) flagged the route's own except,
which returned describe_exception(e). web_interface/app.py's error handler
already logs the traceback and returns the same redacted detail for any
unhandled exception, so the local copy is removed: same response, no new
exception-to-response flow, and test_web_error_detail's policy still holds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

web_interface/app.py

@@ -1047,8 +1047,36 @@ def check_health_monitor():
             _reconciliation_started = True
             _threading.Thread(target=_run_startup_reconciliation, daemon=True).start()
 
+_auto_updater = None
+
+
+def start_auto_update_scheduler():
+    """Start the weekly auto-update thread (web_interface/auto_update.py).
+
+    Called by the launchers, not at import: tests and dev tools import this
+    module, and none of them should ever git pull. Idle unless
+    auto_update.enabled is set, so it is safe to always start.
+    """
+    global _auto_updater
+    if _auto_updater is not None:
+        return _auto_updater
+    from web_interface.auto_update import AutoUpdater
+    from web_interface.blueprints.api_v3.system import perform_core_update
+    _auto_updater = AutoUpdater(
+        config_manager=config_manager,
+        core_update=perform_core_update,
+        store_manager=plugin_store_manager,
+        plugin_manager=plugin_manager,
+        schema_manager=schema_manager,
+        operation_history=operation_history,
+    )
+    _auto_updater.start()
+    return _auto_updater
+
+
 if __name__ == '__main__':
     import os as _os
+    start_auto_update_scheduler()
     # threaded=True is Flask's default since 1.0 but stated explicitly so that
     # long-lived /api/v3/stream/* SSE connections don't starve other requests.
     # Debug mode is off by default; opt in with FLASK_DEBUG=1 in the environment.

web_interface/auto_update.py

@@ -0,0 +1,700 @@
+"""Weekly automatic updates: LEDMatrix code, then installed plugins.
+
+Turned on by ``auto_update.enabled`` in config.json (General tab, or the
+installer's --enable-auto-update). Off by default: an update pulls new code and
+restarts the display, which nobody should get without asking for it.
+
+Nothing here is allowed to leave a device broken without saying so:
+
+* **Checks first.** The code update is skipped, and the reason reported, when
+  the checkout has local edits or commits, a rebase or merge is in progress,
+  the branch has no upstream, disk is low, the newest commit was already
+  rolled back once, or the health check is not set up.
+* **Verified, and rolled back.** The pull itself is the Overview "Update Code"
+  path (``perform_core_update``). Restarting and checking the result is handed
+  to ledmatrix-update-verify.service (scripts/utils/auto_update_verify.py),
+  started through ledmatrix-update-verify.path by a request file, which
+  survives the web service restart, confirms both services come up and stay
+  up, and otherwise resets to the previous commit and its dependencies.
+* **Set up without SSH.** Those units are installed by the installer, or by the
+  display service when the toggle is switched on (src/auto_update_setup.py).
+* **Plugins after the code.** Plugins use the Plugin Store's own update, which
+  refuses versions this core cannot run and restores the old copy when an
+  install fails. When the code changed they wait until it passed its check,
+  so they are never gated against code that is about to be rolled back.
+* **Loud failures.** Anything other than success is written to the state file,
+  shown under the toggle and raised as a banner on the Overview tab.
+
+The schedule survives restarts through data/auto_update_state.json (gitignored).
+"""
+import importlib.util
+import json
+import logging
+import os
+import shutil
+import subprocess  # nosec B404 - list-form argv only, no shell  # nosemgrep
+import tempfile
+import threading
+import time
+from datetime import datetime
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+STATE_REL = Path('data') / 'auto_update_state.json'
+PENDING_REL = Path('data') / 'auto_update_pending.json'
+REQUEST_REL = Path('data') / 'auto_update_verify.request'
+SETUP_RESULT_REL = Path('data') / 'auto_update_setup.json'
+VERIFIER_COPY_REL = Path('data') / 'auto_update_verifier.py'
+VERIFIER_SOURCE = PROJECT_ROOT / 'scripts' / 'utils' / 'auto_update_verify.py'
+STATE_FILE = PROJECT_ROOT / STATE_REL
+PENDING_FILE = PROJECT_ROOT / PENDING_REL
+SETUP_RESULT_FILE = PROJECT_ROOT / SETUP_RESULT_REL
+
+VERIFY_UNIT = 'ledmatrix-update-verify'
+PATH_UNIT = f'{VERIFY_UNIT}.path'
+
+UPDATE_INTERVAL_SECONDS = 7 * 24 * 3600
+#: A failure that is probably transient (usually no network) is retried the
+#: next day rather than a week later.
+RETRY_AFTER_FAILURE_SECONDS = 24 * 3600
+#: Local hours an update prefers, since it restarts the display...
+QUIET_HOURS = range(2, 5)
+#: ...but a device that is never powered at night must still get updates.
+QUIET_HOURS_GRACE_SECONDS = 24 * 3600
+CHECK_INTERVAL_SECONDS = 30 * 60
+#: Let the web service settle after boot (and never pull in a crash loop).
+STARTUP_DELAY_SECONDS = 10 * 60
+MIN_FREE_BYTES = 300 * 1024 * 1024
+#: How long the health check gets to pick up a request before the update is undone.
+HANDOFF_TIMEOUT_SECONDS = 90
+#: The health check takes a few minutes at most; far past that, it is lost.
+VERIFY_LOST_SECONDS = 45 * 60
+
+#: Core outcomes that need the user's attention.
+ERROR_OUTCOMES = frozenset({'error', 'blocked', 'rolled_back', 'rollback_failed', 'lost'})
+
+HELPER_MISSING_MESSAGE = (
+    "LEDMatrix code updates are paused until the update health check is set up. "
+    "The display service sets it up when it starts while automatic updates are on, "
+    "so restart the display from the Overview tab. If this keeps happening, the "
+    "setup result under Automatic Updates on the General tab says why. Plugin "
+    "updates still run."
+)
+
+_STATE_LOCK = threading.RLock()
+
+
+def is_enabled(config):
+    return bool((config.get('auto_update') or {}).get('enabled', False))
+
+
+def _read_json(path):
+    try:
+        with open(path, 'r', encoding='utf-8') as f:
+            data = json.load(f)
+        return data if isinstance(data, dict) else None
+    except (OSError, ValueError):
+        return None
+
+
+def _write_json(path, data):
+    path = Path(path)
+    tmp = None
+    try:
+        path.parent.mkdir(parents=True, exist_ok=True)
+        fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=f'.{path.stem}_')
+        with os.fdopen(fd, 'w', encoding='utf-8') as f:
+            json.dump(data, f, indent=2)
+        os.replace(tmp, path)
+        tmp = None
+    finally:
+        if tmp and os.path.exists(tmp):
+            try:
+                os.unlink(tmp)
+            except OSError:
+                pass
+
+
+def _unlink(path):
+    try:
+        Path(path).unlink()
+    except OSError:
+        pass
+
+
+def load_state(state_file=STATE_FILE):
+    return _read_json(state_file) or {}
+
+
+def save_state(state, state_file=STATE_FILE):
+    with _STATE_LOCK:
+        try:
+            _write_json(state_file, state)
+        except OSError as e:
+            logger.warning("Could not save auto-update state: %s", e)
+
+
+def dismiss_alert(alert_id, state_file=STATE_FILE):
+    with _STATE_LOCK:
+        state = load_state(state_file)
+        state['alert_dismissed'] = str(alert_id)
+        save_state(state, state_file)
+
+
+def _zone(tz_name):
+    try:
+        from zoneinfo import ZoneInfo
+        return ZoneInfo(tz_name) if tz_name else None
+    except Exception:
+        return None
+
+
+def _local_datetime(ts, tz_name):
+    return datetime.fromtimestamp(ts, _zone(tz_name))
+
+
+def _short(sha):
+    return (sha or 'unknown')[:7]
+
+
+def is_due(now, next_due, local_hour):
+    """Due once ``next_due`` has passed, in quiet hours or after the grace period."""
+    if next_due is None or now < next_due:
+        return False
+    return local_hour in QUIET_HOURS or now >= next_due + QUIET_HOURS_GRACE_SECONDS
+
+
+def _plugin_fingerprint(store_manager, plugin_dir):
+    """(manifest version, git sha): what changing tells us an update landed."""
+    version = None
+    try:
+        with open(plugin_dir / 'manifest.json', 'r', encoding='utf-8') as f:
+            manifest = json.load(f)
+        if manifest.get('local_only'):
+            return None
+        version = manifest.get('version')
+    except (OSError, ValueError):
+        pass
+    sha = None
+    try:
+        git_info = store_manager._get_local_git_info(plugin_dir)
+        sha = git_info.get('sha') if git_info else None
+    except Exception:
+        logger.debug("git info unavailable for %s", plugin_dir, exc_info=True)
+    return (version, sha)
+
+
+def update_plugins(store_manager, operation_history=None):
+    """Update every installed plugin that has an update. Returns (updated, failed)."""
+    updated, failed = [], []
+    plugins_dir = Path(store_manager.plugins_dir)
+    for plugin_id in sorted(store_manager.list_installed_plugins()):
+        plugin_dir = plugins_dir / plugin_id
+        before = _plugin_fingerprint(store_manager, plugin_dir)
+        if before is None:
+            continue  # local_only: managed by hand, never from the registry
+        try:
+            ok = store_manager.update_plugin(plugin_id)
+        except Exception:
+            logger.exception("Automatic update of plugin %s raised", plugin_id)
+            ok = False
+        if not ok:
+            failed.append(plugin_id)
+            status = 'failed'
+        elif _plugin_fingerprint(store_manager, plugin_dir) != before:
+            updated.append(plugin_id)
+            status = 'success'
+        else:
+            continue  # already current; not worth a history entry every week
+        if operation_history:
+            try:
+                operation_history.record_operation(
+                    'update', plugin_id=plugin_id, status=status,
+                    details={'automatic': True})
+            except Exception:
+                logger.debug("Could not record auto-update history", exc_info=True)
+    return updated, failed
+
+
+def _service_active(unit):
+    try:
+        result = subprocess.run(['systemctl', 'is-active', unit],
+                                capture_output=True, text=True, timeout=5)
+        return result.stdout.strip() == 'active'
+    except (subprocess.SubprocessError, OSError):
+        return False
+
+
+def helper_ready():
+    """True when the health check can be triggered (its path unit is watching)."""
+    return _service_active(PATH_UNIT)
+
+
+def restart_service(unit):
+    """Restart a unit only if it is running, via the sudoers-allowed command."""
+    if not _service_active(unit):
+        return False
+    try:
+        result = subprocess.run(['sudo', '-n', 'systemctl', 'restart', f'{unit}.service'],
+                                capture_output=True, text=True, timeout=30)
+    except (subprocess.SubprocessError, OSError) as e:
+        logger.warning("Auto-update could not restart %s: %s", unit, e)
+        return False
+    if result.returncode != 0:
+        logger.warning("Auto-update restart of %s failed: %s", unit, result.stderr.strip())
+    return result.returncode == 0
+
+
+def start_setup_if_needed(was_enabled, config):
+    """After settings are saved: switching updates on finishes setting them up.
+
+    The health check units are installed by the display service at startup
+    (src/auto_update_setup.py), so switching the toggle on restarts it.
+    Returns a note for the save message, or None.
+    """
+    if was_enabled or not is_enabled(config) or helper_ready():
+        return None
+    if not _service_active('ledmatrix'):
+        return 'Automatic update setup finishes the next time the display service starts.'
+    if restart_service('ledmatrix'):
+        return 'Finishing automatic update setup: the display is restarting.'
+    return 'Could not restart the display to finish automatic update setup; restart it from the Overview tab.'
+
+
+def _load_verifier(path):
+    spec = importlib.util.spec_from_file_location('ledmatrix_auto_update_verifier', str(path))
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+class AutoUpdater:
+    """Decides when an automatic update is due and runs it."""
+
+    def __init__(self, config_manager, core_update, store_manager=None,
+                 plugin_manager=None, schema_manager=None, operation_history=None,
+                 project_root=PROJECT_ROOT, state_file=None, clock=time.time,
+                 restart=restart_service, run=subprocess.run,
+                 service_active=_service_active, helper_ready=helper_ready,
+                 verifier_source=VERIFIER_SOURCE, disk_free=None, sleep=time.sleep):
+        self.config_manager = config_manager
+        self.core_update = core_update
+        self.store_manager = store_manager
+        self.plugin_manager = plugin_manager
+        self.schema_manager = schema_manager
+        self.operation_history = operation_history
+        self.project_root = Path(project_root)
+        self.state_file = Path(state_file) if state_file else self.project_root / STATE_REL
+        self.pending_file = self.project_root / PENDING_REL
+        self.request_file = self.project_root / REQUEST_REL
+        self.verifier_copy = self.project_root / VERIFIER_COPY_REL
+        self.verifier_source = Path(verifier_source)
+        self.clock = clock
+        self.restart = restart
+        self.run_command = run
+        self.service_active = service_active
+        self.helper_ready = helper_ready
+        self.disk_free = disk_free or (lambda path: shutil.disk_usage(path).free)
+        self.sleep = sleep
+        self._stop = threading.Event()
+        self._thread = None
+
+    # -- scheduling ---------------------------------------------------------
+
+    def tick(self):
+        """Do whatever is due. Returns True if an update ran."""
+        now = self.clock()
+        state = load_state(self.state_file)
+        # Before the enabled check: a verification started before the user
+        # switched updates off still has to be reported.
+        if self._finalize_verification(state, now) == 'waiting':
+            return False
+
+        try:
+            config = self.config_manager.load_config()
+        except Exception:
+            logger.debug("Auto-update could not load config", exc_info=True)
+            return False
+        if not is_enabled(config):
+            return False
+
+        if state.get('plugins_pending'):
+            self._run_deferred_plugins(state, now)
+            return True
+
+        if state.get('next_due') is None:
+            # Just switched on: due now, which in practice means the next
+            # quiet-hours window rather than the moment Save was clicked.
+            state['next_due'] = now
+            save_state(state, self.state_file)
+
+        local_hour = _local_datetime(now, config.get('timezone')).hour
+        if not is_due(now, state['next_due'], local_hour):
+            return False
+        self.run(state)
+        return True
+
+    def run(self, state=None):
+        state = load_state(self.state_file) if state is None else state
+        logger.info("Automatic update starting")
+        try:
+            core = self.update_core(state)
+        except Exception as e:
+            logger.exception("Automatic core update raised")
+            core = {'outcome': 'error', 'message': f'The LEDMatrix update failed unexpectedly: {e}'}
+
+        deferred = core['outcome'] == 'verifying'
+        updated, failed = ([], []) if deferred else self._update_plugins()
+        self._store_run(state, core, updated, failed)
+        logger.info("Automatic update: core %s (%s); plugins updated=%s failed=%s%s",
+                    core['outcome'], core['message'], updated, failed,
+                    '; plugins wait for the health check' if deferred else '')
+
+        # Code restarts belong to the health check; this only covers plugins.
+        if updated:
+            self.restart('ledmatrix')
+        return state['last_result']
+
+    def _store_run(self, state, core, updated, failed):
+        now = self.clock()
+        deferred = core['outcome'] == 'verifying'
+        state.update({
+            'last_run': now,
+            'next_due': now + (RETRY_AFTER_FAILURE_SECONDS if core['outcome'] == 'error'
+                               else UPDATE_INTERVAL_SECONDS),
+            'plugins_pending': deferred,
+            'last_result': {
+                'core_outcome': core['outcome'],
+                'core_message': core['message'],
+                'plugins_updated': updated,
+                'plugins_failed': failed,
+                'plugins_deferred': deferred,
+            },
+        })
+        self._apply_result(state, now)
+        save_state(state, self.state_file)
+
+    # -- core update --------------------------------------------------------
+
+    def _git(self, *args, timeout=60):
+        return self.run_command(['git', *args], cwd=str(self.project_root),
+                                capture_output=True, text=True, timeout=timeout)
+
+    def _count(self, rev_range):
+        out = self._git('rev-list', '--count', rev_range).stdout.strip()
+        return int(out) if out.isdigit() else 0
+
+    def preflight(self, state):
+        """Decide whether the code update may run.
+
+        Returns ``(outcome, message, info)`` with outcome ``ready``,
+        ``up_to_date``, ``blocked`` (needs the user) or ``error`` (retry soon).
+        """
+        if not self.helper_ready():
+            return 'blocked', HELPER_MISSING_MESSAGE, {}
+
+        git_dir = self._git('rev-parse', '--git-dir')
+        if git_dir.returncode != 0:
+            return 'blocked', 'LEDMatrix is not installed as a git checkout, so it cannot update itself.', {}
+        git_dir = Path(git_dir.stdout.strip())
+        if not git_dir.is_absolute():
+            git_dir = self.project_root / git_dir
+        rebase_dirs = [git_dir / name for name in ('rebase-merge', 'rebase-apply') if (git_dir / name).exists()]
+        if rebase_dirs:
+            # A rebase really in progress leaves HEAD detached. With HEAD back
+            # on a branch it was abandoned -- typically a pull that stopped on
+            # a conflict, then a checkout -- and it makes every later pull
+            # fail. It holds no work (local edits are checked separately
+            # below), so clear it rather than strand a device nobody SSHes into.
+            if self._git('symbolic-ref', '-q', 'HEAD').returncode != 0:
+                return 'blocked', ('A git rebase is in progress in the LEDMatrix folder. '
+                                   'Finish or abort it; automatic updates will not touch it.'), {}
+            cleared = self._git('rebase', '--quit')
+            if cleared.returncode != 0 or any(path.exists() for path in rebase_dirs):
+                return 'blocked', ('An abandoned git rebase in the LEDMatrix folder could not be cleared: '
+                                   f'{(cleared.stderr or "").strip() or "unknown error"}.'), {}
+            logger.warning("Cleared an abandoned git rebase from %s", ', '.join(str(p) for p in rebase_dirs))
+        if (git_dir / 'MERGE_HEAD').exists():
+            return 'blocked', ('A git merge is in progress in the LEDMatrix folder. '
+                               'Finish or abort it; automatic updates will not touch it.'), {}
+
+        if self._git('rev-parse', '--abbrev-ref', '--symbolic-full-name', '@{u}').returncode != 0:
+            return 'blocked', ('The current branch has no upstream to update from (or HEAD is detached). '
+                               'Use Update Code once, or Tools -> Switch branch.'), {}
+
+        # Mode-only changes are ignored: older installers chmod tracked
+        # scripts, and --autostash in the pull carries those across. Plugin
+        # folders are separate installs, as in perform_core_update.
+        status = self._git('-c', 'core.fileMode=false', 'status', '--porcelain', '--untracked-files=no')
+        changed = [line[3:] for line in status.stdout.splitlines()
+                   if line.strip() and 'plugins/' not in line and 'plugin-repos/' not in line]
+        if status.returncode != 0 or changed:
+            example = f' (for example {changed[0]})' if changed else ''
+            return 'blocked', (f'{len(changed) or "Some"} tracked file(s) in the LEDMatrix folder were edited '
+                               f'locally{example}. Automatic updates will not stash your changes; '
+                               'commit or revert them, or update manually with Update Code.'), {}
+
+        free = self.disk_free(str(self.project_root))
+        if free < MIN_FREE_BYTES:
+            return 'blocked', (f'Only {free // (1024 * 1024)} MB of disk space is free; an update '
+                               f'needs at least {MIN_FREE_BYTES // (1024 * 1024)} MB.'), {}
+
+        fetch = self._git('fetch', '--quiet', timeout=120)
+        if fetch.returncode != 0:
+            detail = next((ln.strip() for ln in (fetch.stderr or '').splitlines() if ln.strip()), '')
+            return 'error', f'Could not check for LEDMatrix updates: {detail or "git fetch failed"}.', {}
+
+        ahead = self._count('@{u}..HEAD')
+        if ahead:
+            return 'blocked', (f'This checkout has {ahead} local commit(s) that are not upstream. '
+                               'Automatic updates will not rebase them; update manually with Update Code.'), {}
+        if not self._count('HEAD..@{u}'):
+            return 'up_to_date', 'LEDMatrix is already up to date.', {}
+
+        upstream = self._git('rev-parse', '@{u}').stdout.strip()
+        if upstream and upstream == state.get('rolled_back_head'):
+            return 'up_to_date', (f'The newest LEDMatrix version ({_short(upstream)}) failed its health '
+                                  'check and was rolled back before; waiting for a newer one.'), {}
+        head = self._git('rev-parse', 'HEAD').stdout.strip()
+        return 'ready', '', {'old_head': head, 'upstream_head': upstream}
+
+    def update_core(self, state):
+        try:
+            outcome, message, info = self.preflight(state)
+        except (subprocess.SubprocessError, OSError) as e:
+            return {'outcome': 'error', 'message': f'Pre-update checks failed: {e}.'}
+        if outcome != 'ready':
+            return {'outcome': outcome, 'message': message}
+        old_head = info['old_head']
+
+        display_was_active = self.service_active('ledmatrix')
+        try:
+            # Copied before pulling, so the checker is the known-good version.
+            self.verifier_copy.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(self.verifier_source, self.verifier_copy)
+            verifier = _load_verifier(self.verifier_copy)
+        except Exception as e:
+            return {'outcome': 'error', 'message': f'Could not prepare the update health check: {e}.'}
+
+        try:
+            core = self.core_update()
+        except Exception as e:
+            logger.exception("perform_core_update raised")
+            core = {'status': 'error', 'message': f'Update failed: {e}'}
+        new_head = self._git('rev-parse', 'HEAD').stdout.strip()
+        pending = {
+            'status': 'pending',
+            'old_head': old_head,
+            'new_head': new_head,
+            'display_was_active': display_was_active,
+            'dependency_failures': list(core.get('dependency_failures') or []),
+            'created_at': self.clock(),
+        }
+
+        def rollback():
+            return verifier.Verifier(self.project_root, run=self.run_command).rollback(pending)
+
+        if core.get('status') != 'success':
+            message = core.get('message') or 'The LEDMatrix update failed.'
+            if new_head and new_head != old_head:
+                ok, detail = rollback()
+                message += (' The partial update was rolled back.' if ok
+                            else f' Rolling back the partial update also failed: {detail}.')
+                if not ok:
+                    return {'outcome': 'rollback_failed', 'message': message}
+            return {'outcome': 'error', 'message': message}
+        if new_head == old_head:
+            return {'outcome': 'up_to_date', 'message': core.get('message') or 'LEDMatrix is already up to date.'}
+
+        handoff = {'outcome': 'verifying',
+                   'message': (f'Updated LEDMatrix from {_short(old_head)} to {_short(new_head)}; '
+                               'restarting and checking the services.')}
+        # Recorded before the handoff: the health check restarts this process.
+        self._store_run(state, handoff, [], [])
+        try:
+            _write_json(self.pending_file, pending)
+            self.request_file.write_text(new_head, encoding='utf-8')
+            picked_up = self._wait_for_pickup()
+        except OSError as e:
+            logger.error("Could not request the update health check: %s", e)
+            picked_up = False
+        if picked_up:
+            return handoff
+
+        # No health check means no update: undo it while the old code is
+        # still the code that is running.
+        logger.error("%s did not start the health check; rolling the update back", PATH_UNIT)
+        _unlink(self.request_file)
+        _unlink(self.pending_file)
+        ok, detail = rollback()
+        if not ok:
+            return {'outcome': 'rollback_failed',
+                    'message': (f'The update health check did not start, and rolling back to '
+                                f'{_short(old_head)} failed: {detail}.')}
+        return {'outcome': 'blocked',
+                'message': ('The update health check did not start, so the update was undone. '
+                            f'Check "systemctl status {PATH_UNIT}".'
+                            + (f' Note: {detail}.' if detail else ''))}
+
+    def _wait_for_pickup(self):
+        for _ in range(HANDOFF_TIMEOUT_SECONDS):
+            pending = _read_json(self.pending_file)
+            if pending and pending.get('status') != 'pending':
+                return True
+            self.sleep(1)
+        pending = _read_json(self.pending_file)
+        return bool(pending and pending.get('status') != 'pending')
+
+    def _finalize_verification(self, state, now):
+        """Fold the health check's outcome into the state. None, 'waiting' or 'done'."""
+        pending = _read_json(self.pending_file)
+        if pending is None:
+            return None
+        status = pending.get('status')
+        old, new = pending.get('old_head'), pending.get('new_head')
+        reason, detail = pending.get('reason'), pending.get('detail')
+
+        if status in ('pending', 'verifying'):
+            if now - float(pending.get('created_at') or now) < VERIFY_LOST_SECONDS:
+                return 'waiting'
+            outcome = 'lost'
+            message = (f'LEDMatrix was updated to {_short(new)}, but its health check never reported back, '
+                       'so it is not known whether the device is healthy. '
+                       f'See "journalctl -u {VERIFY_UNIT}".')
+        elif status == 'success':
+            outcome = 'updated'
+            message = (f'Updated LEDMatrix from {_short(old)} to {_short(new)}; '
+                       'the services restarted and stayed healthy.')
+        elif status == 'rolled_back':
+            outcome = 'rolled_back'
+            message = (f'The LEDMatrix update to {_short(new)} was rolled back to {_short(old)} because '
+                       f'{reason or "it failed its health check"}.' + (f' Note: {detail}.' if detail else ''))
+            state['rolled_back_head'] = new
+        else:
+            outcome = 'rollback_failed'
+            message = (f'The LEDMatrix update to {_short(new)} failed ({reason or "unknown reason"}) and '  # nosec B608 - user-facing message, not SQL  # nosemgrep
+                       f'could not be rolled back: {detail or "unknown error"}. The device may need '
+                       f'attention: run "git reset --hard {old}" in the LEDMatrix folder, then restart '
+                       'the display and web services.')
+
+        result = state.setdefault('last_result', {})
+        result.update({'core_outcome': outcome, 'core_message': message})
+        if outcome not in ('updated', 'rolled_back'):
+            # The device is in an unknown state; don't pile plugin changes on it.
+            state['plugins_pending'] = False
+            result['plugins_deferred'] = False
+        self._apply_result(state, now)
+        save_state(state, self.state_file)
+        _unlink(self.pending_file)
+        (logger.info if outcome == 'updated' else logger.error)("Automatic update: %s", message)
+        return 'done'
+
+    # -- plugins and reporting ----------------------------------------------
+
+    def _update_plugins(self):
+        if not self.store_manager:
+            return [], []
+        updated, failed = update_plugins(self.store_manager, self.operation_history)
+        for plugin_id in updated:
+            if self.schema_manager:
+                self.schema_manager.invalidate_cache(plugin_id)
+        if updated and self.plugin_manager:
+            try:
+                self.plugin_manager.discover_plugins()
+            except Exception:
+                logger.debug("discover_plugins after auto-update failed", exc_info=True)
+        return updated, failed
+
+    def _run_deferred_plugins(self, state, now):
+        state['plugins_pending'] = False
+        updated, failed = self._update_plugins()
+        result = state.setdefault('last_result', {})
+        result.update({'plugins_updated': updated, 'plugins_failed': failed, 'plugins_deferred': False})
+        self._apply_result(state, now)
+        save_state(state, self.state_file)
+        if updated:
+            self.restart('ledmatrix')
+
+    @staticmethod
+    def _apply_result(state, now):
+        result = state.get('last_result') or {}
+        outcome = result.get('core_outcome')
+        failed = result.get('plugins_failed') or []
+        if outcome in ERROR_OUTCOMES or failed:
+            result['status'] = 'error'
+        elif outcome == 'verifying':
+            result['status'] = 'pending'
+        else:
+            result['status'] = 'success'
+
+        if result['status'] == 'error':
+            parts = []
+            if outcome in ERROR_OUTCOMES:
+                parts.append(result.get('core_message') or 'The LEDMatrix update failed.')
+            if failed:
+                parts.append('These plugins failed to update: ' + ', '.join(failed) + '.')
+            state['alert'] = {'id': str(int(now * 1000)), 'message': ' '.join(parts)}
+        elif result['status'] == 'success':
+            state.pop('alert', None)
+
+    # -- thread -------------------------------------------------------------
+
+    def start(self):
+        if self._thread and self._thread.is_alive():
+            return
+        self._thread = threading.Thread(target=self._loop, name='auto-update', daemon=True)
+        self._thread.start()
+
+    def stop(self):
+        self._stop.set()
+
+    def _loop(self):
+        if self._stop.wait(STARTUP_DELAY_SECONDS):
+            return
+        while True:
+            try:
+                self.tick()
+            except Exception:
+                logger.exception("Automatic update check failed")
+            if self._stop.wait(CHECK_INTERVAL_SECONDS):
+                return
+
+
+def describe_status(config, state=None, state_file=STATE_FILE, pending_file=PENDING_FILE,
+                    setup_file=SETUP_RESULT_FILE, helper=None):
+    """What the General tab and the Overview banner show."""
+    state = load_state(state_file) if state is None else state
+    tz = config.get('timezone')
+    fmt = '%b %d, %Y %I:%M %p'
+    pending = _read_json(pending_file) or {}
+    setup = _read_json(setup_file) or {}
+    enabled = is_enabled(config)
+    out = {
+        'last_run': None, 'summary': None, 'status': None, 'next_due': None,
+        'alert': None, 'alert_id': None,
+        # Only asked while enabled: it runs systemctl, on every page load.
+        'verifier_installed': (helper or helper_ready)() if enabled else None,
+        'setup_status': setup.get('status'),
+        'setup_message': setup.get('message'),
+        'verifying': pending.get('status') in ('pending', 'verifying'),
+    }
+    if state.get('last_run'):
+        out['last_run'] = _local_datetime(state['last_run'], tz).strftime(fmt)
+        result = state.get('last_result') or {}
+        out['status'] = result.get('status')
+        parts = [result.get('core_message') or 'No result recorded.']
+        if result.get('plugins_updated'):
+            parts.append('Plugins updated: ' + ', '.join(result['plugins_updated']) + '.')
+        if result.get('plugins_failed'):
+            parts.append('Plugins that failed to update: ' + ', '.join(result['plugins_failed']) + '.')
+        if result.get('plugins_deferred') and state.get('plugins_pending'):
+            parts.append('Plugin updates run once the code update passes its health check.')
+        out['summary'] = ' '.join(parts)
+    if enabled and state.get('next_due'):
+        out['next_due'] = _local_datetime(state['next_due'], tz).strftime(fmt)
+    alert = state.get('alert')
+    if alert and alert.get('id') != state.get('alert_dismissed'):
+        out['alert'], out['alert_id'] = alert.get('message'), alert.get('id')
+    return out

web_interface/blueprints/api_v3/config.py

@@ -454,16 +454,21 @@ def save_main_config():
 
         # Merge with existing config (similar to original implementation)
         current_config = api_v3.config_manager.load_config()
+        was_auto_update_enabled = bool((current_config.get('auto_update') or {}).get('enabled'))
 
         # Handle general settings
         # Note: Checkboxes don't send data when unchecked, so we need to check if we're updating general settings
         # If any general setting is present, we're updating the general tab
         is_general_update = any(k in data for k in ['timezone', 'city', 'state', 'country', 'web_display_autostart',
-                                                     'auto_discover', 'auto_load_enabled', 'development_mode', 'plugins_directory'])
+                                                     'auto_discover', 'auto_load_enabled', 'development_mode', 'plugins_directory',
+                                                     'auto_update_enabled'])
 
         if is_general_update:
             # For checkbox: if not present in data during general update, it means unchecked
             current_config['web_display_autostart'] = _coerce_to_bool(data.get('web_display_autostart'))
+            if not isinstance(current_config.get('auto_update'), dict):
+                current_config['auto_update'] = {}
+            current_config['auto_update']['enabled'] = _coerce_to_bool(data.get('auto_update_enabled'))
 
         if 'timezone' in data:
             current_config['timezone'] = data['timezone']
@@ -1033,7 +1038,7 @@ def save_main_config():
             if key in ['timezone', 'city', 'state', 'country',
                        'web_display_autostart', 'auto_discover',
                        'auto_load_enabled', 'development_mode',
-                       'plugins_directory', 'target_fps']:
+                       'plugins_directory', 'target_fps', 'auto_update_enabled']:
                 continue
             # Skip fields that are already handled above in their own named sections.
             # Without this, every form field name lands as a top-level config key too.
@@ -1068,7 +1073,17 @@ def save_main_config():
         except ImportError:
             pass
 
-        return success_response(message='Configuration saved successfully')
+        message = 'Configuration saved successfully'
+        # Switching automatic updates on finishes their setup, which needs
+        # the display service to restart (web_interface/auto_update.py).
+        try:
+            from web_interface import auto_update
+            note = auto_update.start_setup_if_needed(was_auto_update_enabled, current_config)
+            if note:
+                message = f'{message}. {note}'
+        except Exception:
+            logger.warning("Automatic update setup could not be started", exc_info=True)
+        return success_response(message=message)
     except Exception as e:
         logger.error("Error saving config", exc_info=True)
         return error_response(

web_interface/blueprints/api_v3/system.py

@@ -12,6 +12,8 @@ from web_interface.blueprints.api_v3 import (
     get_git_version, jsonify, logger, os, request, resolve_pull_command,
     shutil, subprocess,
 )
+import threading
+
 import web_interface.blueprints.api_v3 as _pkg
 # Read through the module rather than bound by value: tests patch these
 # as module attributes, and a value binding would not see the patch.
@@ -120,6 +122,32 @@ def get_system_version():
     except Exception as e:
         logger.error("get_system_version failed: %s", e, exc_info=True)
         return jsonify({'status': 'error', 'message': 'Unable to retrieve version'}), 500
+@api_v3.route('/system/auto-update', methods=['GET'])
+def get_auto_update_status():
+    """Weekly automatic update status: last result, next check, and any alert.
+
+    No local except: a failure falls through to the app-wide handler in
+    web_interface/app.py, which logs the traceback and returns the redacted
+    detail -- the response every route gives, without a second copy here.
+    """
+    from web_interface import auto_update
+    config = api_v3.config_manager.load_config() if api_v3.config_manager else {}
+    return jsonify({'status': 'success', 'data': auto_update.describe_status(config)})
+
+
+@api_v3.route('/system/auto-update/dismiss', methods=['POST'])
+def dismiss_auto_update_alert():
+    """Hide the current automatic-update banner until a new alert replaces it."""
+    from web_interface import auto_update
+    payload = request.get_json(silent=True)
+    # A JSON array or scalar is a bad request, not a 500.
+    alert_id = str(payload.get('alert_id') or '').strip() if isinstance(payload, dict) else ''
+    if not alert_id:
+        return jsonify({'status': 'error', 'message': 'alert_id required'}), 400
+    auto_update.dismiss_alert(alert_id)
+    return jsonify({'status': 'success'})
+
+
 @api_v3.route('/system/check-update', methods=['GET'])
 def check_for_update():
     """Check whether a newer LEDMatrix commit is available on origin/main."""
@@ -198,6 +226,219 @@ def _sudo_hint_for(text):
     return None
 
 
+_core_update_lock = threading.Lock()
+
+
+def perform_core_update():
+    """Pull the latest LEDMatrix code and sync its dependencies.
+
+    Shared by the Overview "Update Code" button and the weekly automatic
+    updater (web_interface/auto_update.py), so both take exactly the same
+    path. Returns the JSON-able payload the button has always received:
+    ``status``, ``message`` and ``restart_required``.
+    """
+    # The button and the scheduler can fire together; two pulls racing
+    # over one checkout (and one stash) is how local changes get lost.
+    if not _core_update_lock.acquire(blocking=False):
+        return {'status': 'error', 'restart_required': False,
+                'message': 'An update is already in progress; try again shortly.'}
+    try:
+        return _perform_core_update_locked()
+    finally:
+        _core_update_lock.release()
+
+
+def _perform_core_update_locked():
+    project_dir = str(PROJECT_ROOT)
+
+    # Decide how to pull BEFORE stashing. If this checkout cannot be
+    # updated at all, stashing first would put the user's local changes
+    # away for an update that was never going to run.
+    pull_args, upstream_note, pull_error = resolve_pull_command(project_dir)
+    if pull_error:
+        logger.warning("git pull not attempted: %s", pull_error)
+        return {'status': 'error', 'message': pull_error, 'restart_required': False}
+
+    # Check if there are local changes that need to be stashed
+    # Exclude plugins directory - plugins are separate repos and shouldn't be stashed with base project
+    # Use --untracked-files=no to skip untracked files check (much faster with symlinked plugins)
+    try:
+        status_result = subprocess.run(
+            ['git', 'status', '--porcelain', '--untracked-files=no'],
+            capture_output=True,
+            text=True,
+            timeout=30,
+            cwd=project_dir
+        )
+        # Filter out any changes in plugins directory - plugins are separate repositories
+        # Git status format: XY filename (where X is status of index, Y is status of work tree)
+        status_lines = [line for line in status_result.stdout.strip().split('\n')
+                       if line.strip() and 'plugins/' not in line]
+        has_changes = bool('\n'.join(status_lines).strip())
+    except subprocess.TimeoutExpired:
+        # If status check times out, assume there might be changes and proceed
+        # This is safer than failing the update
+        has_changes = True
+        status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
+
+    stash_info = ""
+
+    # Stash local changes if they exist (excluding plugins)
+    # Plugins are separate repositories and shouldn't be stashed with base project updates
+    if has_changes:
+        try:
+            # Use pathspec to exclude plugins directory from stash
+            stash_result = subprocess.run(
+                ['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--', ':!plugins'],
+                capture_output=True,
+                text=True,
+                timeout=30,
+                cwd=project_dir
+            )
+            if stash_result.returncode == 0:
+                logger.debug("git stash: stashed local changes before pull")
+                stash_info = " Local changes were stashed."
+            else:
+                logger.warning("git stash failed before pull (returncode=%d)", stash_result.returncode)
+        except subprocess.TimeoutExpired:
+            logger.warning("git stash timed out, proceeding with pull")
+
+    # Record HEAD before the pull so dependency changes can be detected
+    old_head = None
+    try:
+        _pre = subprocess.run(['git', 'rev-parse', 'HEAD'],
+                              capture_output=True, text=True, timeout=10, cwd=project_dir)
+        if _pre.returncode == 0:
+            old_head = _pre.stdout.strip()
+    except subprocess.TimeoutExpired:
+        logger.warning("git rev-parse timed out before pull")
+
+    # Whether the pull actually brought new code in. "Already up to
+    # date" is a success too, and prompting for a restart then would
+    # train users to ignore the prompt.
+    code_changed = False
+    # Requirement files whose install failed. The automatic updater refuses
+    # to restart onto code whose dependencies did not install.
+    dependency_failures = []
+
+    # Perform the git pull. Branches without an upstream were given
+    # an explicit "origin <branch>" above so the update still works.
+    result = subprocess.run(
+        pull_args,
+        capture_output=True,
+        text=True,
+        timeout=60,
+        cwd=project_dir
+    )
+
+    # Give the branch tracking information so the next pull is a plain
+    # `git pull` — otherwise every update repeats the fallback.
+    if result.returncode == 0 and upstream_note:
+        branch = _git_current_branch(project_dir)
+        if branch:
+            try:
+                subprocess.run(
+                    ['git', 'branch', f'--set-upstream-to=origin/{branch}', branch],
+                    capture_output=True, text=True, timeout=10, cwd=project_dir)
+            except (subprocess.TimeoutExpired, OSError) as exc:
+                logger.debug("could not set upstream for %s: %s", branch, exc)
+
+    # Return custom response for git_pull
+    if result.returncode == 0:
+        pull_message = "Code updated successfully."
+        if has_changes:
+            pull_message = f"Code updated successfully. Local changes were automatically stashed.{stash_info}"
+        if result.stdout and "Already up to date" not in result.stdout:
+            pull_message = f"Code updated successfully.{stash_info}"
+        if upstream_note:
+            pull_message = f"{pull_message} {upstream_note}"
+
+        # Keep Python dependencies in sync automatically: if the pull
+        # changed a requirements file, install it now — users updating
+        # from the web UI (most of them) never SSH in to pip install.
+        # Installs go through the same root-visible path as the
+        # Tools-tab buttons (_pip_install_requirements).
+        dep_notes = []
+        try:
+            _post = subprocess.run(['git', 'rev-parse', 'HEAD'],
+                                   capture_output=True, text=True, timeout=10, cwd=project_dir)
+            new_head = _post.stdout.strip() if _post.returncode == 0 else None
+            if old_head and new_head and old_head != new_head:
+                code_changed = True
+                diff = subprocess.run(
+                    ['git', 'diff', '--name-only', f'{old_head}..{new_head}'],
+                    capture_output=True, text=True, timeout=15, cwd=project_dir)
+                changed = set(diff.stdout.split()) if diff.returncode == 0 else set()
+                for rel in ('requirements.txt', 'web_interface/requirements.txt'):
+                    req_path = PROJECT_ROOT / rel
+                    if rel not in changed or not req_path.exists():
+                        continue
+                    # Each file's install is isolated: a timeout or
+                    # OSError (e.g. the sudo wrapper/interpreter
+                    # missing) on one file must not abort the other.
+                    try:
+                        r = _pip_install_requirements(req_path, timeout=180)
+                        if r.returncode == 0:
+                            dep_notes.append(f"Dependencies from {rel} updated.")
+                        else:
+                            dependency_failures.append(rel)
+                            dep_notes.append(
+                                f"Dependency install from {rel} failed — "
+                                "run Install Base Requirements from the Tools tab.")
+                            logger.warning("post-update pip install failed for %s: %s",
+                                           rel, _truncate_output(r.stdout, r.stderr))
+                    except subprocess.TimeoutExpired:
+                        dependency_failures.append(rel)
+                        dep_notes.append(
+                            f"Dependency install from {rel} timed out — "
+                            "run Install Base Requirements from the Tools tab.")
+                        logger.warning("post-update pip install timed out for %s", rel)
+                    except OSError as install_err:
+                        dependency_failures.append(rel)
+                        dep_notes.append(
+                            f"Dependency install from {rel} failed — "
+                            "run Install Base Requirements from the Tools tab.")
+                        logger.warning("post-update pip install errored for %s: %s",
+                                       rel, install_err)
+        except subprocess.TimeoutExpired:
+            logger.warning("post-update dependency sync timed out")
+        if dep_notes:
+            pull_message += " " + " ".join(dep_notes)
+        # A `git pull` restores built-in plugins (committed under
+        # plugin-repos/) even if the user uninstalled them. Re-remove
+        # any the user previously uninstalled so the update doesn't
+        # resurrect them.
+        if api_v3.plugin_store_manager:
+            try:
+                purged = api_v3.plugin_store_manager.purge_uninstalled_plugins()
+                if purged:
+                    logger.info(
+                        "Re-removed %d uninstalled plugin(s) restored by update: %s",
+                        len(purged), ", ".join(purged),
+                    )
+            except (OSError, RuntimeError) as purge_err:
+                logger.warning("Post-update plugin purge failed: %s", purge_err)
+    else:
+        logger.warning("git pull failed (returncode=%d): %s", result.returncode, result.stderr)
+        # Show git's own first line: "check logs" leaves the user with
+        # nothing to act on, and these failures are usually actionable
+        # (conflicting local commits, no upstream, network).
+        detail = next((ln.strip() for ln in (result.stderr or '').splitlines()
+                       if ln.strip()), '')
+        pull_message = f"Update failed: {detail}" if detail else "Update failed; check logs for details"
+
+    # Nothing here restarts anything: the pull replaces files on
+    # disk while the display and web services keep running the code
+    # they loaded at boot. Without this the user is told the update
+    # succeeded and sees no change until they happen to reboot.
+    return {
+        'status': 'success' if result.returncode == 0 else 'error',
+        'message': pull_message,
+        'restart_required': bool(result.returncode == 0 and code_changed),
+        'dependency_failures': dependency_failures,
+    }
+
+
 @api_v3.route('/system/action', methods=['POST'])
 def execute_system_action():
     """Execute system actions (start/stop/reboot/etc)"""
@@ -265,188 +506,7 @@ def execute_system_action():
             result = subprocess.run(['sudo', 'poweroff'],
                                  capture_output=True, text=True, timeout=10)
         elif action == 'git_pull':
-            # Use PROJECT_ROOT instead of hardcoded path
-            project_dir = str(PROJECT_ROOT)
-
-            # Decide how to pull BEFORE stashing. If this checkout cannot be
-            # updated at all, stashing first would put the user's local changes
-            # away for an update that was never going to run.
-            pull_args, upstream_note, pull_error = resolve_pull_command(project_dir)
-            if pull_error:
-                logger.warning("git pull not attempted: %s", pull_error)
-                return jsonify({'status': 'error', 'message': pull_error})
-
-            # Check if there are local changes that need to be stashed
-            # Exclude plugins directory - plugins are separate repos and shouldn't be stashed with base project
-            # Use --untracked-files=no to skip untracked files check (much faster with symlinked plugins)
-            try:
-                status_result = subprocess.run(
-                    ['git', 'status', '--porcelain', '--untracked-files=no'],
-                    capture_output=True,
-                    text=True,
-                    timeout=30,
-                    cwd=project_dir
-                )
-                # Filter out any changes in plugins directory - plugins are separate repositories
-                # Git status format: XY filename (where X is status of index, Y is status of work tree)
-                status_lines = [line for line in status_result.stdout.strip().split('\n')
-                               if line.strip() and 'plugins/' not in line]
-                has_changes = bool('\n'.join(status_lines).strip())
-            except subprocess.TimeoutExpired:
-                # If status check times out, assume there might be changes and proceed
-                # This is safer than failing the update
-                has_changes = True
-                status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
-
-            stash_info = ""
-
-            # Stash local changes if they exist (excluding plugins)
-            # Plugins are separate repositories and shouldn't be stashed with base project updates
-            if has_changes:
-                try:
-                    # Use pathspec to exclude plugins directory from stash
-                    stash_result = subprocess.run(
-                        ['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--', ':!plugins'],
-                        capture_output=True,
-                        text=True,
-                        timeout=30,
-                        cwd=project_dir
-                    )
-                    if stash_result.returncode == 0:
-                        logger.debug("git stash: stashed local changes before pull")
-                        stash_info = " Local changes were stashed."
-                    else:
-                        logger.warning("git stash failed before pull (returncode=%d)", stash_result.returncode)
-                except subprocess.TimeoutExpired:
-                    logger.warning("git stash timed out, proceeding with pull")
-
-            # Record HEAD before the pull so dependency changes can be detected
-            old_head = None
-            try:
-                _pre = subprocess.run(['git', 'rev-parse', 'HEAD'],
-                                      capture_output=True, text=True, timeout=10, cwd=project_dir)
-                if _pre.returncode == 0:
-                    old_head = _pre.stdout.strip()
-            except subprocess.TimeoutExpired:
-                logger.warning("git rev-parse timed out before pull")
-
-            # Whether the pull actually brought new code in. "Already up to
-            # date" is a success too, and prompting for a restart then would
-            # train users to ignore the prompt.
-            code_changed = False
-
-            # Perform the git pull. Branches without an upstream were given
-            # an explicit "origin <branch>" above so the update still works.
-            result = subprocess.run(
-                pull_args,
-                capture_output=True,
-                text=True,
-                timeout=60,
-                cwd=project_dir
-            )
-
-            # Give the branch tracking information so the next pull is a plain
-            # `git pull` — otherwise every update repeats the fallback.
-            if result.returncode == 0 and upstream_note:
-                branch = _git_current_branch(project_dir)
-                if branch:
-                    try:
-                        subprocess.run(
-                            ['git', 'branch', f'--set-upstream-to=origin/{branch}', branch],
-                            capture_output=True, text=True, timeout=10, cwd=project_dir)
-                    except (subprocess.TimeoutExpired, OSError) as exc:
-                        logger.debug("could not set upstream for %s: %s", branch, exc)
-
-            # Return custom response for git_pull
-            if result.returncode == 0:
-                pull_message = "Code updated successfully."
-                if has_changes:
-                    pull_message = f"Code updated successfully. Local changes were automatically stashed.{stash_info}"
-                if result.stdout and "Already up to date" not in result.stdout:
-                    pull_message = f"Code updated successfully.{stash_info}"
-                if upstream_note:
-                    pull_message = f"{pull_message} {upstream_note}"
-
-                # Keep Python dependencies in sync automatically: if the pull
-                # changed a requirements file, install it now — users updating
-                # from the web UI (most of them) never SSH in to pip install.
-                # Installs go through the same root-visible path as the
-                # Tools-tab buttons (_pip_install_requirements).
-                dep_notes = []
-                try:
-                    _post = subprocess.run(['git', 'rev-parse', 'HEAD'],
-                                           capture_output=True, text=True, timeout=10, cwd=project_dir)
-                    new_head = _post.stdout.strip() if _post.returncode == 0 else None
-                    if old_head and new_head and old_head != new_head:
-                        code_changed = True
-                        diff = subprocess.run(
-                            ['git', 'diff', '--name-only', f'{old_head}..{new_head}'],
-                            capture_output=True, text=True, timeout=15, cwd=project_dir)
-                        changed = set(diff.stdout.split()) if diff.returncode == 0 else set()
-                        for rel in ('requirements.txt', 'web_interface/requirements.txt'):
-                            req_path = PROJECT_ROOT / rel
-                            if rel not in changed or not req_path.exists():
-                                continue
-                            # Each file's install is isolated: a timeout or
-                            # OSError (e.g. the sudo wrapper/interpreter
-                            # missing) on one file must not abort the other.
-                            try:
-                                r = _pip_install_requirements(req_path, timeout=180)
-                                if r.returncode == 0:
-                                    dep_notes.append(f"Dependencies from {rel} updated.")
-                                else:
-                                    dep_notes.append(
-                                        f"Dependency install from {rel} failed — "
-                                        "run Install Base Requirements from the Tools tab.")
-                                    logger.warning("post-update pip install failed for %s: %s",
-                                                   rel, _truncate_output(r.stdout, r.stderr))
-                            except subprocess.TimeoutExpired:
-                                dep_notes.append(
-                                    f"Dependency install from {rel} timed out — "
-                                    "run Install Base Requirements from the Tools tab.")
-                                logger.warning("post-update pip install timed out for %s", rel)
-                            except OSError as install_err:
-                                dep_notes.append(
-                                    f"Dependency install from {rel} failed — "
-                                    "run Install Base Requirements from the Tools tab.")
-                                logger.warning("post-update pip install errored for %s: %s",
-                                               rel, install_err)
-                except subprocess.TimeoutExpired:
-                    logger.warning("post-update dependency sync timed out")
-                if dep_notes:
-                    pull_message += " " + " ".join(dep_notes)
-                # A `git pull` restores built-in plugins (committed under
-                # plugin-repos/) even if the user uninstalled them. Re-remove
-                # any the user previously uninstalled so the update doesn't
-                # resurrect them.
-                if api_v3.plugin_store_manager:
-                    try:
-                        purged = api_v3.plugin_store_manager.purge_uninstalled_plugins()
-                        if purged:
-                            logger.info(
-                                "Re-removed %d uninstalled plugin(s) restored by update: %s",
-                                len(purged), ", ".join(purged),
-                            )
-                    except (OSError, RuntimeError) as purge_err:
-                        logger.warning("Post-update plugin purge failed: %s", purge_err)
-            else:
-                logger.warning("git pull failed (returncode=%d): %s", result.returncode, result.stderr)
-                # Show git's own first line: "check logs" leaves the user with
-                # nothing to act on, and these failures are usually actionable
-                # (conflicting local commits, no upstream, network).
-                detail = next((ln.strip() for ln in (result.stderr or '').splitlines()
-                               if ln.strip()), '')
-                pull_message = f"Update failed: {detail}" if detail else "Update failed; check logs for details"
-
-            # Nothing here restarts anything: the pull replaces files on
-            # disk while the display and web services keep running the code
-            # they loaded at boot. Without this the user is told the update
-            # succeeded and sees no change until they happen to reboot.
-            return jsonify({
-                'status': 'success' if result.returncode == 0 else 'error',
-                'message': pull_message,
-                'restart_required': bool(result.returncode == 0 and code_changed),
-            })
+            return jsonify(perform_core_update())
         elif action == 'checkout_branch':
             # Switch branches from the Tools tab. Needed because a checkout
             # that predates tracking (or a restored backup) can leave the pi

web_interface/blueprints/pages_v3.py

@@ -525,8 +525,15 @@ def _load_general_partial():
     try:
         if pages_v3.config_manager:
             main_config = pages_v3.config_manager.load_config()
+            try:
+                from web_interface.auto_update import describe_status
+                auto_update_status = describe_status(main_config)
+            except Exception:
+                logger.debug("Could not read auto-update status", exc_info=True)
+                auto_update_status = None
             return render_template('v3/partials/general.html',
-                                 main_config=main_config)
+                                 main_config=main_config,
+                                 auto_update_status=auto_update_status)
     except Exception as e:
         logger.error("Error loading partial", exc_info=True)
         return "Error loading partial", 500

web_interface/start.py

@@ -100,8 +100,9 @@ def main():
     werkzeug_logger.error = log_exception_filtered
     
     # Import and run the Flask app
-    from web_interface.app import app
-    
+    from web_interface.app import app, start_auto_update_scheduler
+    start_auto_update_scheduler()
+
     print("Starting LED Matrix Web Interface V3...")
     print("Web server binding to: 0.0.0.0:5000")
     

web_interface/templates/v3/partials/general.html

@@ -44,6 +44,44 @@
             </label>
         </div>
 
+        <!-- Weekly Automatic Updates -->
+        <div class="form-group" id="setting-general-auto_update" data-setting-key="auto_update.enabled">
+            <label class="flex items-center">
+                <input type="checkbox"
+                       name="auto_update_enabled"
+                       value="true"
+                       {% if (main_config.auto_update or {}).enabled %}checked{% endif %}
+                       class="form-control h-4 w-4 text-blue-600 focus:ring-blue-500 border-gray-300 rounded">
+                <span class="ml-2 text-sm font-medium text-gray-900">Automatically check for and install updates once a week</span>
+                {{ ui.help_tip('Once a week, update LEDMatrix and every installed plugin that has a newer version.\nSkipped (and reported) if you have local changes, low disk space, or the rollback service is missing. After updating, the services are restarted and checked; if they do not stay healthy, the update is rolled back automatically.\nRuns overnight (2-5 AM in your timezone) when possible. Problems show as a banner on Overview. A stopped display is not started. Default: off.', 'Automatic Updates') }}
+            </label>
+            {% if auto_update_status and (main_config.auto_update or {}).enabled and auto_update_status.verifier_installed is sameas false %}
+            <p class="mt-1 ml-6 text-xs text-amber-700">
+                {% if auto_update_status.setup_status == 'failed' %}
+                Setting up the update health check failed: {{ auto_update_status.setup_message }}
+                LEDMatrix code updates are paused until it is fixed; plugin updates still run.
+                {% else %}
+                Setting up the update health check. The display service does this when it starts, so it is
+                ready shortly after saving (or once the display is started). LEDMatrix code updates begin after that.
+                {% endif %}
+            </p>
+            {% endif %}
+            {% if auto_update_status and auto_update_status.verifying %}
+            <p class="mt-1 ml-6 text-xs text-blue-700">An update was just installed and its health check is running.</p>
+            {% endif %}
+            {% if auto_update_status and (auto_update_status.last_run or auto_update_status.next_due) %}
+            <div class="mt-1 ml-6 text-xs text-gray-500 space-y-1">
+                {% if auto_update_status.last_run %}
+                <p>Last automatic update: {{ auto_update_status.last_run }}
+                   <span class="{{ 'text-red-600' if auto_update_status.status == 'error' else '' }}">— {{ auto_update_status.summary }}</span></p>
+                {% endif %}
+                {% if auto_update_status.next_due %}
+                <p>Next check: {{ auto_update_status.next_due }} or the first overnight window after it</p>
+                {% endif %}
+            </div>
+            {% endif %}
+        </div>
+
         <!-- Timezone -->
         <div class="form-group" id="setting-general-timezone" data-setting-key="timezone">
             <label for="timezone" class="block text-sm font-medium text-gray-700">Timezone{{ ui.help_tip('Time zone used for clocks, schedules, and time-based content.\nChoose the zone where the display physically lives so on/off schedules fire at the correct local time.', 'Timezone') }}</label>

web_interface/templates/v3/partials/overview.html

@@ -1,3 +1,44 @@
+<!-- Automatic update banner: a weekly update that failed, was blocked or was rolled back -->
+<div id="auto-update-banner" class="bg-red-50 border border-red-300 rounded-lg p-4 mb-4 flex items-start" style="display:none !important" role="alert">
+    <div class="flex-shrink-0 mr-3 mt-0.5">
+        <i class="fas fa-exclamation-circle text-red-500"></i>
+    </div>
+    <div class="flex-1">
+        <p class="text-sm font-medium text-red-800">Automatic Update Needs Attention</p>
+        <p class="text-sm text-red-700 mt-1" id="auto-update-banner-text"></p>
+        <p class="text-xs text-red-600 mt-1">Details are under Automatic Updates on the General tab.</p>
+    </div>
+    <button type="button" onclick="window.dismissAutoUpdateBanner()" class="ml-4 flex-shrink-0 text-red-500 hover:text-red-600" aria-label="Dismiss">
+        <i class="fas fa-times"></i>
+    </button>
+</div>
+<script>
+(function () {
+    fetch('/api/v3/system/auto-update')
+        .then(function (r) { return r.json(); })
+        .then(function (resp) {
+            var d = resp.data || {};
+            if (!d.alert) return;
+            document.getElementById('auto-update-banner-text').textContent = d.alert;
+            var banner = document.getElementById('auto-update-banner');
+            banner.dataset.alertId = d.alert_id || '';
+            banner.style.setProperty('display', 'flex', 'important');
+        })
+        .catch(function () {});
+
+    // Dismissal is stored server-side, so the banner stays gone on every
+    // device until a new failure raises a new alert.
+    window.dismissAutoUpdateBanner = function () {
+        var banner = document.getElementById('auto-update-banner');
+        banner.style.setProperty('display', 'none', 'important');
+        fetch('/api/v3/system/auto-update/dismiss', {
+            method: 'POST',
+            headers: {'Content-Type': 'application/json'},
+            body: JSON.stringify({alert_id: banner.dataset.alertId})
+        }).catch(function () {});
+    };
+}());
+</script>
 <!-- Reconciliation warning banner: shown when startup reconciliation found stale plugin config entries -->
 <div id="reconciliation-banner" class="bg-yellow-50 border border-yellow-300 rounded-lg p-4 mb-4 flex items-start" style="display:none !important" role="alert">
     <div class="flex-shrink-0 mr-3 mt-0.5">