fix(startup): warn when an installed systemd unit has drifted from the repo's

Nothing re-applies systemd units after the first install. `git pull` -- which
is what the web UI's update button runs -- brings a new template into the
checkout, but no code in web_interface/ or src/ copies it to
/etc/systemd/system, and nothing anywhere runs `systemctl daemon-reload`. The
unit that actually runs is whatever first_time_install.sh wrote on day one.

So every hardening added to a unit is inert on existing installs, silently.
Measured on a live rig:

    installed  /etc/systemd/system/ledmatrix.service   2026-08-06
    template   systemd/ledmatrix.service               2026-08-19
    contents                                           differ

with the practical result that the MemoryMax=85% the repo's template specifies
was not being enforced at all -- `systemctl show` reported
MemoryMax=infinity. Anyone reading the template would reasonably believe the
service was capped.

Startup now compares each installed unit against its substituted template and
warns when they differ, naming install_service.sh as the remedy.

A warning, not an error, and deliberately not a silent rewrite: editing files
under /etc and restarting services is the installer's job, not something a
display process should do to a machine while it is booting. Making it fatal
would also brick every development checkout whose unit is legitimately absent
or hand-edited.

Comparison ignores comments, blank lines and ordering. The template carries
explanatory comments the installed copy will not have, and systemd does not
care about order within a section, so a literal comparison would warn on every
boot and be ignored within a week.

Mutation-checked three ways: never reporting drift fails, making it fatal
fails, and -- after the first attempt missed it -- comparing raw text now fails
too. That last gap is worth noting: the comment-insensitivity tests originally
exercised the helper directly, so a comparison that stopped calling the helper
passed them all. The test that catches it goes through _validate_systemd_units.

29 startup-validator tests pass.

docs/LOW_MEMORY_BOARDS.md

@@ -0,0 +1,115 @@
+# Running on Low-Memory Boards
+
+Applies to the Pi Zero 2 W (512 MB), Pi 3 / 3B+ (1 GB), and the 1 GB Pi 4.
+If your board has 2 GB or more you can skip this document.
+
+## The failure this prevents
+
+The display process is the largest thing on the board. On a 1 GB Pi 3B+ with
+around 20 plugins enabled it settles near **600 MB of 905 MB usable**, leaving
+under 200 MB of headroom for everything else.
+
+When that headroom runs out, the board does not crash cleanly. `fork()` starts
+failing, and because a new process is needed to do almost anything, the
+symptoms look nothing like "out of memory":
+
+| What you see | Why |
+|---|---|
+| SSH accepts the connection then closes it instantly, before any banner | `sshd` forks a session per connection; the fork fails |
+| The web UI still responds quickly | Already running, serves from existing threads, forks nothing |
+| Ping is perfect, 0% loss | Handled entirely in the kernel |
+| The panel is dark | The display process was killed and cannot be respawned |
+| The clock is wrong after the next boot | `fake-hwclock`'s periodic save is a scheduled job, and it cannot fork either |
+
+The board looks healthy from the outside and cannot be logged into. Only a
+power cycle clears it. If you are here because SSH stopped working, also see
+[SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md), which
+covers the more common cause (AP mode).
+
+## Check your headroom
+
+```bash
+free -m
+ps -eo rss,comm --sort=-rss | head -5
+```
+
+If `MemAvailable` is under ~150 MB while the display is running, you are close
+to the edge. To watch it over time:
+
+```bash
+watch -n 30 'free -m | head -2'
+```
+
+Available memory that falls steadily rather than holding flat means you will
+reach the wall; it is a question of when.
+
+## What to do
+
+**1. Enable the memory cgroup controller.** Without it, the `MemoryMax=85%` in
+`systemd/ledmatrix.service` is accepted by systemd and silently ignored, so the
+service has no ceiling and a runaway takes the whole board down instead of just
+restarting. Raspberry Pi firmware disables this controller by default.
+
+`first_time_install.sh` does this for you. To check it took effect:
+
+```bash
+grep memory /sys/fs/cgroup/cgroup.controllers
+```
+
+If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the
+kernel command line and reboot. Edit whichever file your image uses —
+`/boot/firmware/cmdline.txt` on current Raspberry Pi OS, `/boot/cmdline.txt` on
+older layouts (the installer checks the first and falls back to the second).
+Everything must stay on a single line.
+
+This changes the failure mode from "the board becomes unreachable" to "the
+display service restarts". It is a safety net, not a fix.
+
+**2. Run fewer plugins.** This is the actual remedy. Every enabled plugin costs
+memory permanently — its module, its parsed config, and its cached API
+responses. On a 512 MB or 1 GB board, keep the enabled set small and prefer
+plugins that poll infrequently.
+
+**3. Lower the cache ceiling.** The in-memory cache is sized from total RAM
+(150 entries at 1 GB and below, up to 1500 at 8 GB). To go lower still:
+
+```ini
+# /etc/systemd/system/ledmatrix.service.d/override.conf
+[Service]
+Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75
+```
+
+Writing the file does not change the running service. Reload systemd and
+restart it:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl restart ledmatrix
+```
+
+Fewer entries means more API calls, so lower this only while you are actually
+short of memory.
+
+**4. Consider `MemoryHigh`.** `MemoryMax` kills and restarts. `MemoryHigh`
+throttles and reclaims instead, which is gentler — but on a board where the
+process genuinely wants more than the limit, sustained reclaim can stall the
+render loop and show as visible stutter on the panel. Add it only if you prefer
+degraded output to a restart:
+
+```ini
+[Service]
+MemoryHigh=70%
+```
+
+## Keep your logs
+
+These images default to volatile journald storage, so every reboot destroys the
+logs — including the ones explaining why the board rebooted. `first_time_install.sh`
+enables persistent storage capped at 64 MB. To confirm:
+
+```bash
+journalctl --list-boots
+```
+
+More than one boot listed means logs are surviving reboots. If only one is
+listed, journald is still writing to `/run` (tmpfs).

docs/README.md

@@ -14,6 +14,7 @@ the one-shot installer. The pages here go deeper.
 5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes
 6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install
 7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems
+8. [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) — Pi Zero 2 W / 3B+ / 1GB Pi 4 memory limits
 
 ## I want to write a plugin
 

docs/SSH_UNAVAILABLE_AFTER_INSTALL.md

@@ -20,7 +20,22 @@ The installation script:
 - Installs and configures `dnsmasq` (DHCP server for AP mode)
 - These services can interfere with normal WiFi client mode
 
-### 3. Reboot After Installation
+### 3. The Board Ran Out of Memory
+
+On a 512MB or 1GB board, memory exhaustion stops `sshd` being able to fork a
+session process. The connection is accepted and then closed immediately, before
+any banner:
+
+```text
+kex_exchange_identification: Connection closed by remote host
+```
+
+The giveaway is that the board is otherwise healthy — ping is clean and the web
+UI still responds — but nothing that needs to start a new process works, and
+the panel is usually dark. Only a power cycle clears it. See
+[LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md).
+
+### 4. Reboot After Installation
 
 If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode.
 
@@ -190,11 +205,23 @@ The web interface allows you to:
 
 ## Summary
 
-**SSH becomes unavailable because**:
+**SSH becomes unavailable because** — two unrelated causes, and they need
+different responses:
+
+*AP mode (most common):*
 - WiFi monitor service enables AP mode when WiFi disconnects
 - AP mode switches WiFi from client to access point mode
 - Pi loses connection to your original network
 
+*Memory exhaustion (low-memory boards):*
+- The board runs out of memory, so `sshd` cannot fork a session process
+- The connection is accepted and closed before any banner
+- Ping still answers and the web UI still responds, so it looks healthy
+- The panel is usually dark and the service cannot restart
+- **Only a power cycle clears this** — there is no remote recovery, because
+  every remote route needs a new process
+- Prevention and tuning: [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md)
+
 **To regain SSH**:
 1. Connect to **LEDMatrix-Setup** AP network (password: `ledmatrix123`)
 2. SSH to `192.168.4.1`

first_time_install.sh

@@ -1688,6 +1688,94 @@ else
     echo "✗ $CMDLINE_FILE not found; skipping isolcpus optimization"
 fi
 
+# Enable the memory cgroup controller (idempotent).
+# The Pi firmware boots with cgroup_disable=memory, so systemd's MemoryMax= is
+# accepted and silently ignored — the display service then has no ceiling, and
+# a runaway takes the whole board down (sshd can no longer fork, the panel goes
+# dark) rather than just restarting the one service.
+if [ "$SKIP_PERF" != "1" ] && [ -f "$CMDLINE_FILE" ]; then
+    # Both parameters are required for the memory controller, and they can get
+    # separated -- an image, another tool or a half-applied earlier run can
+    # leave one without the other. Checking only cgroup_enable=memory would
+    # report success while MemoryMax= silently does nothing, so each is checked
+    # and appended independently.
+    cgroup_missing=""
+    for cgroup_param in cgroup_enable=memory cgroup_memory=1; do
+        if ! grep -qw "$cgroup_param" "$CMDLINE_FILE"; then
+            cgroup_missing="$cgroup_missing $cgroup_param"
+        fi
+    done
+    if [ -z "$cgroup_missing" ]; then
+        echo "cgroup memory parameters already present in $CMDLINE_FILE"
+    else
+        echo "Adding${cgroup_missing} to $CMDLINE_FILE..."
+        cp "$CMDLINE_FILE" "$CMDLINE_FILE.bak" 2>/dev/null || true
+        # The kernel command line must stay on one line.
+        sed -i "1 s|\$|${cgroup_missing}|" "$CMDLINE_FILE"
+        echo "  Takes effect after reboot. Verify with:"
+        echo "    grep memory /sys/fs/cgroup/cgroup.controllers"
+    fi
+fi
+
+# Persist the journal (idempotent).
+# These images default to volatile storage: journald keeps everything in /run
+# (tmpfs), so every reboot destroys the logs — including the ones that would
+# explain why the board rebooted. Capped so an SD card is not worn out by logs.
+# A non-empty /var/log/journal does not prove journald is configured the way
+# this needs: the directory survives a switch back to volatile storage, and it
+# says nothing about whether a size cap is set. Read the effective
+# configuration instead, and only write the keys the user has not set
+# themselves so an explicit local limit is preserved.
+journald_effective() {
+    # systemd-analyze merges journald.conf with every drop-in; grep is the
+    # fallback for images that ship without it.
+    if command -v systemd-analyze >/dev/null 2>&1 &&
+       systemd-analyze cat-config systemd/journald.conf >/dev/null 2>&1; then
+        systemd-analyze cat-config systemd/journald.conf 2>/dev/null
+    else
+        cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null
+    fi
+}
+journald_conf="$(journald_effective)"
+journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
+journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
+
+if [ "$journald_storage" = "persistent" ] && [ -n "$journald_cap" ]; then
+    echo "Persistent journald storage already configured (SystemMaxUse=$journald_cap)"
+else
+    echo "Enabling persistent journald storage..."
+    mkdir -p /etc/systemd/journald.conf.d
+    {
+        echo "# Installed by LEDMatrix first_time_install.sh"
+        echo "[Journal]"
+        echo "Storage=persistent"
+        if [ -n "$journald_cap" ]; then
+            echo "# SystemMaxUse left to your existing setting ($journald_cap)"
+        else
+            # Capped so logs cannot wear out or fill an SD card.
+            echo "SystemMaxUse=64M"
+        fi
+    } > /etc/systemd/journald.conf.d/ledmatrix-persistent.conf
+    mkdir -p /var/log/journal
+    systemd-tmpfiles --create --prefix /var/log/journal >/dev/null 2>&1 || true
+    systemctl restart systemd-journald >/dev/null 2>&1 || true
+
+    # Drop-ins are applied in lexical order, so a locally added file that sorts
+    # after ledmatrix-persistent.conf (zz-local.conf and friends) still wins.
+    # Writing the file is not evidence it took effect -- re-read and say so
+    # plainly rather than reporting success we cannot confirm.
+    journald_now="$(journald_effective | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
+    if [ "$journald_now" = "persistent" ]; then
+        echo "  Persistent journald storage active"
+    else
+        echo "  WARNING: journald storage is still '${journald_now:-unset}' after"
+        echo "  writing /etc/systemd/journald.conf.d/ledmatrix-persistent.conf."
+        echo "  Another drop-in that sorts later is overriding it. Check:"
+        echo "    systemd-analyze cat-config systemd/journald.conf | grep -n Storage="
+        echo "  Logs will not survive a reboot until that is resolved."
+    fi
+fi
+
 # Ensure dtparam=audio=off in config.txt (idempotent)
 if [ "$SKIP_PERF" = "1" ]; then
     : # skipped

src/cache/memory_cache.py

@@ -4,11 +4,58 @@ Memory Cache
 Handles in-memory caching with TTL support, size limits, and automatic cleanup.
 """
 
+import os
 import time
 import threading
 import logging
 from typing import Dict, Any, Optional
 
+# Historical fixed ceiling, kept as the fallback when RAM cannot be read.
+DEFAULT_MAX_SIZE = 1000
+
+
+def _total_memory_mb() -> Optional[float]:
+    """Physical RAM in MB, or None where /proc/meminfo is unavailable."""
+    try:
+        with open('/proc/meminfo', 'r', encoding='utf-8') as fh:
+            for line in fh:
+                if line.startswith('MemTotal:'):
+                    return int(line.split()[1]) / 1024
+    except (OSError, ValueError, IndexError):
+        return None
+    return None
+
+
+def default_max_size() -> int:
+    """Entry ceiling scaled to this machine's RAM.
+
+    One fixed ceiling cannot serve both a 512 MB Pi Zero 2 W and an 8 GB Pi 5.
+    Entries here are parsed API payloads that routinely run tens of kilobytes
+    each, so a thousand of them is a comfortable cache on a large board and a
+    substantial fraction of total RAM on a small one — where the process
+    competing for that RAM is also driving the panel. Set
+    LEDMATRIX_CACHE_MAX_ENTRIES to override.
+    """
+    override = os.environ.get('LEDMATRIX_CACHE_MAX_ENTRIES')
+    if override:
+        try:
+            value = int(override)
+            if value > 0:
+                return value
+        except ValueError:
+            pass
+
+    total_mb = _total_memory_mb()
+    if total_mb is None:
+        return DEFAULT_MAX_SIZE
+    if total_mb < 1536:      # 512 MB and 1 GB boards
+        return 150
+    if total_mb < 3072:      # 2 GB
+        return 400
+    if total_mb < 6144:      # 4 GB
+        return 800
+    return 1500              # 8 GB and up
+
 
 class MemoryCache:
     """Manages in-memory cache with TTL and size limits."""
@@ -87,6 +134,32 @@ class MemoryCache:
         with self._lock:
             self._cache[key] = value
             self._timestamps[key] = time.time()
+            # Enforce the ceiling here rather than leaving it to the periodic
+            # cleanup, which only runs every cleanup_interval seconds (300 by
+            # default). A burst of inserts between two sweeps could otherwise
+            # take the cache far past _max_size, which is the memory growth this
+            # limit exists to prevent -- and on a 1GB board that is the
+            # difference between a bounded cache and an unreachable Pi.
+            self._evict_over_limit_locked()
+
+    def _evict_over_limit_locked(self) -> int:
+        """Drop oldest entries until the cache is within _max_size.
+
+        Caller must hold self._lock. Returns the number of entries removed.
+        """
+        excess = len(self._cache) - self._max_size
+        if excess <= 0:
+            return 0
+        oldest = sorted(
+            self._timestamps.items(),
+            key=lambda item: float(item[1]) if isinstance(item[1], (int, float)) else 0.0
+        )
+        removed = 0
+        for key, _ in oldest[:excess]:
+            self._cache.pop(key, None)
+            self._timestamps.pop(key, None)
+            removed += 1
+        return removed
     
     def clear(self, key: Optional[str] = None) -> None:
         """
@@ -143,22 +216,8 @@ class MemoryCache:
                 self._timestamps.pop(key, None)
                 removed_count += 1
             
-            # Enforce size limit by removing oldest entries if cache is too large
+            # Same ceiling enforcement set() uses, so the two cannot drift.
-            if len(self._cache) > self._max_size:
+            removed_count += self._evict_over_limit_locked()
-                # Sort by timestamp (oldest first)
-                sorted_entries = sorted(
-                    self._timestamps.items(),
-                    key=lambda x: float(x[1]) if isinstance(x[1], (int, float)) else 0
-                )
-                
-                # Remove oldest entries until we're under the limit
-                excess_count = len(self._cache) - self._max_size
-                for i in range(excess_count):
-                    if i < len(sorted_entries):
-                        key = sorted_entries[i][0]
-                        self._cache.pop(key, None)
-                        self._timestamps.pop(key, None)
-                        removed_count += 1
             
             self._last_cleanup = current_time
             

src/cache_manager.py

@@ -33,7 +33,7 @@ import logging
 import threading
 import tempfile
 from src.exceptions import CacheError
-from src.cache.memory_cache import MemoryCache
+from src.cache.memory_cache import MemoryCache, default_max_size
 from src.cache.disk_cache import DiskCache
 from src.cache.cache_strategy import CacheStrategy
 from src.cache.cache_metrics import CacheMetrics
@@ -84,7 +84,9 @@ class CacheManager:
             self.logger.warning("ConfigManager not available, using default cache intervals")
         
         # Initialize cache components using composition
-        self._memory_cache_component = MemoryCache(max_size=1000, cleanup_interval=300.0)
+        self._memory_cache_component = MemoryCache(
+            max_size=default_max_size(), cleanup_interval=300.0
+        )
         self._disk_cache_component = DiskCache(cache_dir=self.cache_dir, logger=self.logger)
         self._strategy_component = CacheStrategy(config_manager=self.config_manager, logger=self.logger)
         self._metrics_component = CacheMetrics(logger=self.logger)

src/plugin_system/plugin_health.py

@@ -7,7 +7,7 @@ and circuit breaker state. Provides automatic recovery mechanisms.
 
 import time
 import logging
-from typing import Dict, Optional, Any
+from typing import Dict, Optional, Any, Tuple
 from enum import Enum
 
 
@@ -64,10 +64,48 @@ class PluginHealthTracker:
             cache_key, max_age=None, memory_ttl=0 if force_reload else None
         )
 
-        if cached:
+        if isinstance(cached, dict) and cached:
-            return cached
+            # Complete it rather than trusting it: a persisted record can be
-        
+            # missing fields the callers index directly (a partial write, a
-        # Default state
+            # restored backup, an older schema), and returning it verbatim makes
+            # record_success / record_failure raise KeyError, which takes the
+            # display down in a restart loop that survives reboots because the
+            # bad entry is on disk.
+            state, repaired = self._repair_health_state(cached)
+            if repaired:
+                self.logger.warning(
+                    f"Repaired health state for {plugin_id}: "
+                    f"{sorted(repaired)} missing or invalid, using defaults for those."
+                )
+            return state
+
+        # Not a dict at all: written by something other than
+        # _save_health_state (a key collision, a corrupted entry). Nothing to
+        # salvage.
+        if cached is not None and not isinstance(cached, dict):
+            self.logger.warning(
+                f"Discarding malformed health state for {plugin_id}: expected "
+                f"dict, got {type(cached).__name__}. Falling back to defaults."
+            )
+
+        return self._default_health_state()
+    
+    def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None:
+        """Save health state to cache."""
+        cache_key = self._get_health_key(plugin_id)
+        self.cache_manager.set(cache_key, state)  # Persist indefinitely
+        self._health_state[plugin_id] = state
+    
+    # The fields callers index directly (state['circuit_state'] and friends).
+    # A cached dict missing any of them raises KeyError deep in record_success /
+    # record_failure, so the value is completed before it is handed out.
+    _COUNTER_FIELDS = ('consecutive_failures', 'total_failures', 'total_successes')
+    _TIMESTAMP_FIELDS = ('last_success_time', 'last_failure_time',
+                         'circuit_opened_time', 'half_open_start_time')
+
+    @staticmethod
+    def _default_health_state() -> Dict[str, Any]:
+        """A fresh state with every field the callers expect."""
         return {
             'consecutive_failures': 0,
             'total_failures': 0,
@@ -77,15 +115,56 @@ class PluginHealthTracker:
             'circuit_state': CircuitState.CLOSED.value,
             'circuit_opened_time': None,
             'half_open_start_time': None,
-            'last_error': None
+            'last_error': None,
         }
-    
+
-    def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None:
+    @classmethod
-        """Save health state to cache."""
+    def _repair_health_state(cls, cached: Dict[str, Any]) -> Tuple[Dict[str, Any], list]:
-        cache_key = self._get_health_key(plugin_id)
+        """Return `cached` completed against the defaults, plus what was repaired.
-        self.cache_manager.set(cache_key, state)  # Persist indefinitely
+
-        self._health_state[plugin_id] = state
+        Per-field rather than all-or-nothing: a record that has real failure
-    
+        counts but is missing `last_error` should keep the counts, not be reset
+        to healthy. Only values that are absent or the wrong type fall back to
+        the default, so a partial or older-schema record survives with whatever
+        it does carry, while every field the callers index is guaranteed present
+        and of a usable type.
+        """
+        state = cls._default_health_state()
+        repaired = []
+        for field, default in state.items():
+            if field not in cached:
+                repaired.append(field)
+                continue
+            value = cached[field]
+            if field in cls._COUNTER_FIELDS:
+                ok = isinstance(value, int) and not isinstance(value, bool) and value >= 0
+            elif field in cls._TIMESTAMP_FIELDS:
+                # bool is a subclass of int, so True would pass as a timestamp
+                # and then compare as 1.0 -- expiring a cooldown the instant it
+                # opens, or (False) making the elapsed check never fire.
+                ok = value is None or (
+                    isinstance(value, (int, float)) and not isinstance(value, bool)
+                )
+            elif field == 'circuit_state':
+                # Membership first requires the value to be hashable: a list or
+                # dict here would raise TypeError out of the repair itself,
+                # which is the crash this whole path exists to prevent.
+                ok = isinstance(value, str) and value in {
+                    member.value for member in CircuitState
+                }
+            else:  # last_error
+                ok = value is None or isinstance(value, str)
+            if ok:
+                state[field] = value
+            else:
+                repaired.append(field)
+        # Anything the schema has since grown (degraded, degraded_reason) is
+        # read with .get() by its callers, so carry it through untouched.
+        for field, value in cached.items():
+            if field not in state:
+                state[field] = value
+        return state, repaired
+
     def get_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]:
         """Get current health state for a plugin.
 

src/plugin_system/plugin_loader.py

@@ -14,7 +14,7 @@ import sys
 import subprocess
 import threading
 from pathlib import Path
-from typing import Dict, Any, Optional, Tuple, Type
+from typing import Dict, Any, List, Optional, Tuple, Type
 import logging
 
 from packaging.requirements import InvalidRequirement, Requirement
@@ -45,6 +45,76 @@ def requirements_has_real_deps(requirements_file: str) -> bool:
     return False
 
 
+def _extra_dependencies(dist_name: str, extras) -> Optional[List[Requirement]]:
+    """Dependencies a distribution declares *only* behind the given extras.
+
+    Returns None when the installed metadata cannot be read or parsed, so the
+    caller can fall back to running pip rather than assuming anything.
+    """
+    try:
+        meta = importlib.metadata.metadata(dist_name)
+    except importlib.metadata.PackageNotFoundError:
+        return None
+
+    gated: List[Requirement] = []
+    for raw in meta.get_all('Requires-Dist') or []:
+        try:
+            dep = Requirement(raw)
+        except InvalidRequirement:
+            return None
+        if dep.marker is None:
+            continue
+        # Keep only what the distribution gates behind an extra we asked for:
+        # satisfied when `extra` is that name, but not when no extra is
+        # requested. A marker that holds either way (python_version, sys_platform)
+        # belongs to the base install and is already covered by the version check.
+        if dep.marker.evaluate({'extra': ''}):
+            continue
+        if any(dep.marker.evaluate({'extra': extra}) for extra in extras):
+            gated.append(dep)
+    return gated
+
+
+def _extras_are_satisfied(req: Requirement, _visited: Optional[set] = None) -> bool:
+    """Check the dependencies pulled in by req's extras are installed.
+
+    Follows extras through nested extras. A gated dependency can itself request
+    one (`requests[socks]`), and checking only that `requests` is installed at
+    an acceptable version says nothing about whether the socks extra's own
+    dependency is there -- so the caller would skip pip and the plugin would
+    fail at import instead. Plain dependencies are still checked one level
+    deep, which is all that is needed to tell "the extra was installed" from
+    "the extra was never installed".
+
+    `_visited` carries the (distribution, extras) pairs already seen, so a
+    dependency cycle between extras terminates instead of recursing forever.
+    Anything unreadable returns False, so the caller still falls through to pip.
+    """
+    if _visited is None:
+        _visited = set()
+    marker = (req.name.lower(), frozenset(e.lower() for e in req.extras))
+    if marker in _visited:
+        # Already accounted for higher up the chain; treating a cycle as
+        # satisfied here is safe because the outer frame still has to pass.
+        return True
+    _visited.add(marker)
+
+    gated = _extra_dependencies(req.name, req.extras)
+    if gated is None:
+        return False
+
+    for dep in gated:
+        try:
+            dep_version = importlib.metadata.version(dep.name)
+        except importlib.metadata.PackageNotFoundError:
+            return False
+        if dep.specifier and not dep.specifier.contains(dep_version, prereleases=True):
+            return False
+        if dep.extras and not _extras_are_satisfied(dep, _visited):
+            return False
+    return True
+
+
 def requirements_are_satisfied(requirements_file: str) -> bool:
     """
     Check whether every real requirement line in requirements.txt is already
@@ -76,9 +146,6 @@ def requirements_are_satisfied(requirements_file: str) -> bool:
         except InvalidRequirement:
             return False
 
-        if req.extras:
-            return False  # verifying extras' sub-dependencies isn't worth it here
-
         if req.marker is not None and not req.marker.evaluate():
             continue  # not applicable on this platform/interpreter
 
@@ -90,6 +157,9 @@ def requirements_are_satisfied(requirements_file: str) -> bool:
         if req.specifier and not req.specifier.contains(installed_version, prereleases=True):
             return False
 
+        if req.extras and not _extras_are_satisfied(req):
+            return False
+
     return True
 
 

src/startup_validator.py

@@ -62,6 +62,9 @@ class StartupValidator:
         # Validate plugins if plugin manager is available
         if self.plugin_manager:
             self._validate_plugins()
+
+        # Warn when the running systemd unit no longer matches the repo's
+        self._validate_systemd_units()
         
         is_valid = len(self.errors) == 0
         
@@ -74,6 +77,71 @@ class StartupValidator:
         
         return (is_valid, self.errors.copy(), self.warnings.copy())
     
+    #: Units this project installs, and where each is installed to.
+    _UNITS = (
+        ("systemd/ledmatrix.service", "/etc/systemd/system/ledmatrix.service"),
+        ("systemd/ledmatrix-web.service", "/etc/systemd/system/ledmatrix-web.service"),
+    )
+
+    def _validate_systemd_units(self) -> None:
+        """Warn when an installed unit has drifted from the repo's template.
+
+        Nothing re-applies these after the first install. `git pull` -- which is
+        what the web UI's update button runs -- brings a new template into the
+        checkout, but nothing copies it to /etc/systemd/system and nothing runs
+        `systemctl daemon-reload`, so the unit that actually runs is whatever
+        first_time_install.sh wrote on day one.
+
+        That makes every hardening added to a unit inert on existing installs.
+        Measured on one rig: the installed unit was thirteen days older than the
+        repo's and differed in content, so a MemoryMax the repo had specified
+        was not being enforced at all -- `systemctl show` reported
+        MemoryMax=infinity.
+
+        A warning rather than an error, and certainly not a silent rewrite:
+        editing files under /etc and restarting services is the installer's job,
+        not something a display process should do to a machine while it boots.
+        The remedy is to re-run scripts/install/install_service.sh.
+        """
+        try:
+            project_root = Path(__file__).resolve().parent.parent
+            for template_rel, installed_path in self._UNITS:
+                template = project_root / template_rel
+                installed = Path(installed_path)
+                if not template.is_file() or not installed.is_file():
+                    continue
+
+                # The template carries placeholders the installer substitutes,
+                # so compare the substituted form rather than the raw file.
+                expected = template.read_text(encoding="utf-8")
+                expected = expected.replace("__PROJECT_ROOT_DIR__", str(project_root))
+                expected = expected.replace("__USER__", "root")
+
+                try:
+                    actual = installed.read_text(encoding="utf-8")
+                except PermissionError:
+                    continue
+
+                if self._unit_body(expected) != self._unit_body(actual):
+                    self.warnings.append(
+                        f"{installed.name} differs from {template_rel}; the "
+                        "installed unit is not refreshed by an update, so "
+                        "settings added to the template are not in effect. "
+                        "Re-run scripts/install/install_service.sh to apply them."
+                    )
+        except OSError as e:
+            self.logger.debug("Could not compare systemd units: %s", e)
+
+    @staticmethod
+    def _unit_body(text: str) -> str:
+        """A unit's meaningful lines: no comments, no blanks, no ordering noise."""
+        lines = []
+        for line in text.splitlines():
+            line = line.strip()
+            if line and not line.startswith("#"):
+                lines.append(line)
+        return "\n".join(sorted(lines))
+
     def _validate_config(self) -> None:
         """Validate configuration files."""
         try:

systemd/ledmatrix-wifi-monitor.service

@@ -10,8 +10,8 @@ WorkingDirectory=__PROJECT_ROOT_DIR__
 ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/scripts/utils/wifi_monitor_daemon.py --interval 30
 Restart=on-failure
 RestartSec=10
-StandardOutput=syslog
+StandardOutput=journal
-StandardError=syslog
+StandardError=journal
 SyslogIdentifier=ledmatrix-wifi-monitor
 
 [Install]

systemd/ledmatrix.service

@@ -9,8 +9,25 @@ User=root
 WorkingDirectory=__PROJECT_ROOT_DIR__
 Environment=PYTHONDONTWRITEBYTECODE=1
 ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py
-Restart=on-failure
+# Restart=always, not on-failure: run.py exiting 0 (a clean shutdown path taken
+# for a reason that no longer applies, e.g. a config reload) would otherwise leave
+# the service stopped and the panel dark indefinitely, with systemd considering
+# that a successful outcome and never bringing it back.
+Restart=always
 RestartSec=10
+# Memory ceiling as a share of physical RAM, so one unit file suits a 512 MB
+# Pi Zero 2 W and an 8 GB Pi 5 alike. This is a backstop, not a tuning knob: it
+# turns "the board runs out of memory, stops being able to fork, and takes sshd
+# and the panel down together until someone pulls the plug" into "this one
+# service restarts".
+#
+# NOTE: Raspberry Pi firmware boots the kernel with cgroup_disable=memory, and
+# systemd accepts this setting and then silently ignores it. Verify with:
+#     grep memory /sys/fs/cgroup/cgroup.controllers
+# If that prints nothing, add "cgroup_enable=memory cgroup_memory=1" to
+# /boot/firmware/cmdline.txt (all on line 1) and reboot. first_time_install.sh
+# does this for you.
+MemoryMax=85%
 StandardOutput=journal
 StandardError=journal
 SyslogIdentifier=ledmatrix

test/test_cache_manager.py

@@ -458,3 +458,26 @@ class TestDiskCacheWriteEconomy:
         cache = DiskCache(cache_dir=str(tmp_path))
         cache.set("k", {"when": datetime(2026, 7, 12, 10, 30)})
         assert cache.get("k") == {"when": "2026-07-12T10:30:00"}
+
+
+# --- the ceiling has to hold between cleanup sweeps ---------------------------
+
+def test_memory_cache_enforces_ceiling_on_every_write():
+    """_cleanup_memory_cache only runs every cleanup_interval seconds (300 by
+    default). If set() accepted entries without bound in between, a burst could
+    take the cache far past max_size -- which is the unbounded growth the limit
+    exists to prevent, and on a 1GB board the difference between a bounded cache
+    and a Pi that cannot fork.
+    """
+    from src.cache.memory_cache import MemoryCache
+
+    cache = MemoryCache(max_size=150, cleanup_interval=300.0)
+    for i in range(1000):
+        cache.set(f"k{i}", {"v": i})
+
+    assert len(cache._cache) <= 150
+    # The timestamp map has to be evicted alongside the values, or it becomes
+    # the leak instead.
+    assert len(cache._timestamps) <= 150
+    assert cache.get("k999") is not None, "the newest write must survive"
+    assert cache.get("k0") is None, "the oldest must be the one evicted"

test/test_checkbox_group_stale_values.py

@@ -0,0 +1,113 @@
+"""A checkbox group must not post back options it cannot show.
+
+The enum that lets the widget draw checkboxes is also what validates the
+saved value. When a league retires a team code -- OAK for the Athletics, ARI
+for the Coyotes -- or a schema drops an option, a config that still holds the
+old value has nothing to render for it. The value stayed in the hidden
+``_data`` input regardless, because that input is seeded from the stored array
+and only rebuilt by ``updateCheckboxGroupData()`` on change. Editing any other
+field on that plugin therefore posted the stale value back, the schema
+rejected it, and the save endpoint returned 400
+``CONFIG_VALIDATION_FAILED`` -- so the whole plugin became uneditable until
+the user worked out which invisible entry was at fault.
+
+Runtime was never affected: plugin loading treats schema violations as
+warn/degrade, and the stale code already matched no team. Only the web UI
+blocked.
+
+These tests render the checkbox-group block lifted *out of the shipped
+template*, following test_enum_option_labels.py, so they exercise the
+production expression rather than a copy that could drift from it.
+"""
+import json
+import re
+from pathlib import Path
+
+from jinja2 import DictLoader, Environment
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+CONFIG_FORM = (PROJECT_ROOT / 'web_interface' / 'templates' / 'v3' / 'partials'
+               / 'plugin_config.html')
+
+# The checkbox-group branch: from its `{% elif %}` guard through the sentinel
+# hidden input that closes it. Anchored on the guard so the match cannot run on
+# into a neighbouring widget branch.
+BLOCK_RE = re.compile(
+    r"\{%\s*elif x_widget == 'checkbox-group'\s*%\}(.*?)"
+    r"<input type=\"hidden\" name=\"\{\{ full_key \}\}\[\]\" value=\"\">",
+    re.S,
+)
+
+
+def _shipped_block() -> str:
+    """Return the live checkbox-group block lifted from plugin_config.html."""
+    source = CONFIG_FORM.read_text(encoding='utf-8')
+    match = BLOCK_RE.search(source)
+    assert match, (
+        'could not find the checkbox-group block in plugin_config.html — the '
+        'template changed shape and this guard needs updating'
+    )
+    block = match.group(1)
+    assert 'data-option-value' in block, 'extracted the wrong branch'
+    assert '{% elif' not in block, 'extraction ran past the checkbox-group branch'
+    return block
+
+
+def _render(prop: dict, value=None) -> str:
+    env = Environment(loader=DictLoader({'f': _shipped_block()}), autoescape=True)
+    return env.get_template('f').render(
+        prop=prop, value=value, field_id='fid', full_key='k'
+    )
+
+
+def _submitted(html: str) -> list:
+    """The array the form will actually post: the hidden _data input."""
+    match = re.search(r'id="fid_data"[^>]*\svalue=\'([^\']*)\'', html)
+    assert match, f'hidden _data input not found in:\n{html}'
+    return json.loads(match.group(1).replace('&#39;', "'"))
+
+
+def _checked(html: str) -> list:
+    return re.findall(r'data-option-value="([^"]+)"[^>]*checked', html)
+
+
+MLB = {'type': 'array', 'items': {'type': 'string', 'enum': ['NYY', 'BOS', 'ATH']},
+       'x-widget': 'checkbox-group'}
+
+
+def test_a_retired_code_is_not_posted_back() -> None:
+    """The regression: OAK became ATH, and OAK used to ride along on save."""
+    html = _render(MLB, ['NYY', 'OAK'])
+    assert _submitted(html) == ['NYY'], 'stale value would still be submitted'
+
+
+def test_the_dropped_value_is_named_rather_than_vanishing() -> None:
+    html = _render(MLB, ['NYY', 'OAK'])
+    assert 'OAK' in html
+    assert 'data-stale-options' in html
+
+
+def test_valid_values_are_untouched_and_still_checked() -> None:
+    html = _render(MLB, ['NYY', 'ATH'])
+    assert _submitted(html) == ['NYY', 'ATH']
+    assert sorted(_checked(html)) == ['ATH', 'NYY']
+    assert 'data-stale-options' not in html
+
+
+def test_an_all_stale_selection_clears_rather_than_blocking() -> None:
+    html = _render(MLB, ['OAK', 'SD'])
+    assert _submitted(html) == []
+
+
+def test_an_empty_enum_leaves_the_value_alone() -> None:
+    """No options means nothing to validate against — filtering would wipe it."""
+    prop = {'type': 'array', 'items': {'type': 'string'}, 'x-widget': 'checkbox-group'}
+    html = _render(prop, ['ANYTHING', 'GOES'])
+    assert _submitted(html) == ['ANYTHING', 'GOES']
+
+
+def test_unset_value_falls_back_to_the_default() -> None:
+    prop = dict(MLB, default=['BOS'])
+    html = _render(prop, None)
+    assert _submitted(html) == ['BOS']
+    assert _checked(html) == ['BOS']

test/test_onboarding_checklist.py

@@ -0,0 +1,241 @@
+"""
+Getting Started checklist: what the server decides, and what it must not.
+
+The timezone step used to tick server-side when the saved timezone differed
+from the shipped default, OR-ed with the saved city. That made the step
+unsatisfiable for anyone genuinely in the default zone (the card nagged
+forever), and let a saved city tick it off while the timezone was still wrong.
+The step is now verified in the browser against its own zone, so the server's
+only job is to hand over the configured value and stay out of the decision.
+
+These tests pin that contract: the panel-size step still reflects config, the
+timezone step never pre-ticks, it carries the configured zone, and the city
+has no influence on it.
+"""
+
+import copy
+import re
+import sys
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import pytest
+from flask import Flask
+
+PROJECT_ROOT = Path(__file__).parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+BASE_CONFIG = {
+    "timezone": "America/New_York",
+    "location": {"city": "Tampa", "state": "Florida", "country": "US"},
+    "display": {
+        "hardware": {"rows": 32, "cols": 64, "chain_length": 2, "parallel": 1},
+        "runtime": {},
+        "double_sided": {"enabled": False},
+        "vegas_scroll": {"plugin_order": [], "excluded_plugins": []},
+        "plugin_rotation_order": [],
+    },
+    "plugin_system": {},
+    "schedule": {},
+    "dim_schedule": {},
+    "sync": {},
+}
+
+
+def render(config):
+    """Render the overview partial against one config, as app.py would."""
+    base = PROJECT_ROOT / "web_interface"
+    app = Flask(
+        __name__,
+        template_folder=str(base / "templates"),
+        static_folder=str(base / "static"),
+    )
+    app.config["TESTING"] = True
+
+    from web_interface.blueprints import pages_v3 as pv
+
+    # pages_v3 is a module-level singleton shared across the test process;
+    # restore whatever the previous test left on it.
+    original_cm = getattr(pv.pages_v3, "config_manager", None)
+    original_pm = getattr(pv.pages_v3, "plugin_manager", None)
+
+    mock_cm = MagicMock()
+    mock_cm.load_config.return_value = config
+    mock_cm.get_raw_file_content.return_value = config
+    pv.pages_v3.config_manager = mock_cm
+
+    mock_pm = MagicMock()
+    mock_pm.plugins = {}
+    mock_pm.get_all_plugin_info.return_value = []
+    mock_pm.get_plugin_display_modes.side_effect = lambda pid: []
+    pv.pages_v3.plugin_manager = mock_pm
+
+    app.register_blueprint(pv.pages_v3, url_prefix="")
+    try:
+        resp = app.test_client().get("/partials/overview")
+        assert resp.status_code == 200, resp.status_code
+        return resp.get_data(as_text=True)
+    finally:
+        pv.pages_v3.config_manager = original_cm
+        pv.pages_v3.plugin_manager = original_pm
+
+
+def timezone_step(body):
+    """The checklist <button> for the timezone step."""
+    match = re.search(r"<button[^>]*data-check=\"timezone\"[^>]*>", body)
+    assert match, "timezone step not found in the rendered checklist"
+    return match.group(0)
+
+
+def config_with(**overrides):
+    config = copy.deepcopy(BASE_CONFIG)
+    for key, value in overrides.items():
+        config[key] = value
+    return config
+
+
+@pytest.mark.parametrize(
+    "timezone",
+    ["America/New_York", "America/Los_Angeles", "Europe/Madrid", "Asia/Kolkata"],
+)
+def test_timezone_step_never_pre_ticks_server_side(timezone):
+    """The browser owns this decision; the server must not pre-empt it.
+
+    The default zone is in the list deliberately: that is the case the old
+    default-comparison could never tick.
+    """
+    step = timezone_step(render(config_with(timezone=timezone)))
+    assert 'data-done="0"' in step, step
+
+
+@pytest.mark.parametrize(
+    "timezone",
+    ["America/New_York", "Europe/Madrid", "Pacific/Auckland"],
+)
+def test_timezone_step_carries_the_configured_zone(timezone):
+    """JS compares data-tz against the browser, so it has to be the real value."""
+    assert f'data-tz="{timezone}"' in timezone_step(render(config_with(timezone=timezone)))
+
+
+def test_city_does_not_influence_the_timezone_step():
+    """The coupling this change removes: city said nothing about the timezone,
+    and OR-ing it let a saved city tick the step off with the zone still wrong.
+
+    timezone_step() returns the opening tag only, so this compares the state
+    the step is in -- data-done and data-tz -- and not the label, which does
+    still show the configured city as context and so differs between the two.
+    """
+    tampa = timezone_step(render(config_with(
+        location={"city": "Tampa", "state": "Florida", "country": "US"})))
+    seattle = timezone_step(render(config_with(
+        location={"city": "Seattle", "state": "Washington", "country": "US"})))
+    assert tampa == seattle
+
+
+def test_missing_timezone_leaves_the_step_open():
+    """Nothing saved means nothing to verify: the step stays unticked and the
+    JS bails on the empty value rather than comparing against ''."""
+    step = timezone_step(render(config_with(timezone="")))
+    assert 'data-tz=""' in step
+    assert 'data-done="0"' in step
+
+
+def test_zone_comparison_asks_for_the_time_of_day():
+    """Guard on the Intl options, which look like a stylistic choice.
+
+    dateStyle/timeStyle are late additions (Firefox shipped them in 91). An
+    implementation that does not know them ignores them and formats the date
+    alone -- which compares New York, Chicago and Madrid as equal and ticks
+    the step for a timezone that is plainly wrong. Explicit numeric fields
+    have been in Intl since ECMA-402 v1.
+    """
+    template = (PROJECT_ROOT / "web_interface" / "templates" / "v3"
+                / "partials" / "overview.html").read_text()
+    body = template[template.index("function sameZone"):]
+    body = body[:body.index("}())")]
+    # The comment above the options names dateStyle/timeStyle to explain why
+    # they are not used, so match on code only.
+    body = "\n".join(line for line in body.splitlines()
+                     if not line.lstrip().startswith("//"))
+    assert "dateStyle" not in body and "timeStyle" not in body, (
+        "zone comparison must not depend on dateStyle/timeStyle")
+    for field in ("hour:", "minute:", "year:", "month:", "day:"):
+        assert field in body, f"zone comparison dropped {field!r}"
+
+
+def test_zone_comparison_samples_both_sides_of_dst():
+    """One instant is not enough, and the shortfall is invisible for months.
+
+    America/New_York and America/Lima hold the same offset all winter, so a
+    check against now alone ticks the step in January for a panel that runs an
+    hour off from March. The comparison has to sample instants either side of
+    DST -- mid-January and mid-July, which covers both hemispheres.
+    """
+    template = (PROJECT_ROOT / "web_interface" / "templates" / "v3"
+                / "partials" / "overview.html").read_text()
+    body = template[template.index("function sameZone"):]
+    body = body[:body.index("}())")]
+    code = "\n".join(line for line in body.splitlines()
+                     if not line.lstrip().startswith("//"))
+    assert "Date.UTC" in code, (
+        "zone comparison samples only the current instant, so zones that "
+        "coincide seasonally would read as equal")
+    assert code.count("Date.UTC") >= 2, "expected an instant either side of DST"
+
+
+def _stamp(zone, instant):
+    """The JS comparison's algorithm, for pinning what it must decide.
+
+    There is no JS runtime here (and the repo has no JS test infra), so this
+    mirrors sameZone rather than executing it: same instants, same wall-clock
+    equality. It records the verdicts the shipped code has to reach.
+    """
+    from zoneinfo import ZoneInfo
+    return instant.astimezone(ZoneInfo(zone)).strftime("%m/%d/%Y %H:%M")
+
+
+@pytest.mark.parametrize(
+    "left,right,equivalent",
+    [
+        # Aliases: one zone under two names.
+        ("Asia/Calcutta", "Asia/Kolkata", True),
+        ("Europe/Kiev", "Europe/Kyiv", True),
+        # Same rules year-round: either renders the same times, so a panel set
+        # to one and browsed from the other is correctly configured.
+        ("America/New_York", "America/Toronto", True),
+        # Coincide in winter only -- the case a single-instant check gets wrong.
+        ("America/New_York", "America/Lima", False),
+        ("America/Phoenix", "America/Los_Angeles", False),
+        ("Australia/Sydney", "Pacific/Guadalcanal", False),
+        # Plainly different.
+        ("America/New_York", "America/Chicago", False),
+        ("America/New_York", "Europe/Madrid", False),
+    ],
+)
+def test_which_zone_pairs_must_count_as_the_same(left, right, equivalent):
+    from datetime import datetime
+    from zoneinfo import ZoneInfo
+
+    year = 2026
+    instants = [datetime(year, 1, 15, 12, tzinfo=ZoneInfo("UTC")),
+                datetime(year, 7, 15, 12, tzinfo=ZoneInfo("UTC"))]
+    matched = all(_stamp(left, at) == _stamp(right, at) for at in instants)
+    assert matched is equivalent, (
+        f"{left} vs {right}: sampling both seasons gave {matched}")
+
+
+@pytest.mark.parametrize(
+    "hardware,expected",
+    [
+        ({"rows": 32, "cols": 64, "chain_length": 2, "parallel": 1}, "1"),
+        ({"rows": 0, "cols": 0, "chain_length": 0, "parallel": 1}, "0"),
+    ],
+)
+def test_panel_size_step_still_reflects_config(hardware, expected):
+    """Regression guard: the hardware step is still decided server-side."""
+    config = config_with()
+    config["display"]["hardware"] = hardware
+    body = render(config)
+    match = re.search(r"<button[^>]*data-tab=\"display\"[^>]*>", body)
+    assert match, "panel-size step not found"
+    assert f'data-done="{expected}"' in match.group(0), match.group(0)

test/test_plugin_health.py

@@ -91,3 +91,101 @@ def test_force_reload_refreshes_stale_in_memory_snapshot():
 
     # and it asked the cache to bypass the in-memory tier (memory_ttl=0).
     assert any(c.kwargs.get("memory_ttl") == 0 for c in cache.get.call_args_list)
+
+
+# --- persisted state that does not match the current schema -------------------
+#
+# A record on disk can be missing fields the callers index directly: a partial
+# write, a restored backup, or a state written by an older schema. Returning it
+# verbatim raises KeyError inside record_success / record_failure, which takes
+# the display down in a restart loop that survives reboots, because the bad
+# entry is on disk and gets read again on the way back up. Observed in the wild
+# as `plugin clock-simple operation failed: 'circuit_state'`, repeating ~50x a
+# minute with the panel frozen.
+
+_INDEXED_FIELDS = (
+    "consecutive_failures", "total_failures", "total_successes",
+    "last_success_time", "last_failure_time", "circuit_state",
+    "circuit_opened_time", "half_open_start_time", "last_error",
+)
+
+
+def _tracker_reading(persisted):
+    cache = _cache()
+    cache.get.return_value = persisted
+    return PluginHealthTracker(cache)
+
+
+def test_partial_state_is_completed_not_returned_raw():
+    """The shape seen in the wild: one field, everything else absent."""
+    state = _tracker_reading({"circuit_state": "closed"}).get_health_state("p")
+    for field in _INDEXED_FIELDS:
+        assert field in state, f"{field} missing; callers index it directly"
+
+
+def test_repair_keeps_real_failure_history():
+    """A record with genuine counts must not be reset to healthy just because
+    an optional field is absent -- that would clear a tripped breaker."""
+    state = _tracker_reading({
+        "consecutive_failures": 5,
+        "total_failures": 5,
+        "circuit_state": "open",
+    }).get_health_state("p")
+    assert state["consecutive_failures"] == 5
+    assert state["total_failures"] == 5
+    assert state["circuit_state"] == "open"
+
+
+def test_wrong_types_fall_back_per_field():
+    """A counter persisted as a string would pass a membership check and then
+    fail on the first += 1; an unknown circuit_state would take a branch the
+    breaker has no handling for."""
+    state = _tracker_reading({
+        "consecutive_failures": "3",
+        "circuit_state": "melted",
+        "total_failures": 7,
+    }).get_health_state("p")
+    assert state["consecutive_failures"] == 0
+    assert state["circuit_state"] == CircuitState.CLOSED.value
+    assert state["total_failures"] == 7, "valid neighbours must survive"
+
+
+def test_newer_fields_are_carried_through():
+    """degraded/degraded_reason are read with .get() and are not part of the
+    indexed set; repairing must not drop them."""
+    state = _tracker_reading({
+        "circuit_state": "closed", "degraded": True, "degraded_reason": "x",
+    }).get_health_state("p")
+    assert state["degraded"] is True
+    assert state["degraded_reason"] == "x"
+
+
+def test_recording_against_a_repaired_state_does_not_raise():
+    """The actual failure: record_failure indexing a field that was not there.
+
+    The seed deliberately omits circuit_state. Seeding a record that *has* it
+    would pass against the old raw-return behaviour too -- the counters are
+    read with .get(), so circuit_state is the only field whose absence used to
+    raise.
+    """
+    tracker = _tracker_reading({"total_failures": 2, "total_successes": 1})
+    tracker.record_failure("p", Exception("boom"))
+    tracker.record_success("p")
+
+
+def test_unhashable_or_boolean_values_are_repaired():
+    """Values that break the repair itself rather than a later caller.
+
+    An unhashable circuit_state raises TypeError inside a set membership test,
+    and bool is a subclass of int, so True would pass as a timestamp and then
+    compare as 1.0 -- expiring a cooldown the moment it opens.
+    """
+    for bad_state in ({"circuit_state": []}, {"circuit_state": {}}):
+        state = _tracker_reading(bad_state).get_health_state("p")
+        assert state["circuit_state"] == CircuitState.CLOSED.value
+
+    state = _tracker_reading({
+        "circuit_opened_time": True, "last_success_time": False,
+    }).get_health_state("p")
+    assert state["circuit_opened_time"] is None
+    assert state["last_success_time"] is None

test/test_systemd_unit_drift.py

@@ -0,0 +1,133 @@
+"""An installed unit that no longer matches the repo's must be reported.
+
+Nothing re-applies systemd units after the first install. `git pull` -- what
+the web UI's update button runs -- brings a new template into the checkout, but
+no code in web_interface/ or src/ copies it to /etc/systemd/system or runs
+`systemctl daemon-reload`. The unit that actually runs is whatever
+first_time_install.sh wrote on day one.
+
+So every hardening added to a unit is inert on existing installs. Measured on a
+live rig: the installed unit was dated 2026-08-06 and the repo's 2026-08-19,
+and they differed -- with the result that a MemoryMax=85% present in the repo's
+template was not being enforced at all. `systemctl show` reported
+MemoryMax=infinity.
+
+This is a warning, not an error, and deliberately not a silent rewrite:
+editing files under /etc and restarting services is the installer's job, not
+something a display process should do to a machine while it boots.
+"""
+import logging
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import pytest
+
+from src.startup_validator import StartupValidator
+
+
+@pytest.fixture
+def validator():
+    v = StartupValidator(config_manager=MagicMock())
+    v.logger = logging.getLogger("test")
+    v.warnings = []
+    v.errors = []
+    return v
+
+
+def test_a_matching_unit_produces_no_warning(validator, tmp_path):
+    """The installed unit, substituted exactly as the installer would."""
+    project_root = Path("src/startup_validator.py").resolve().parent.parent
+    template_rel = "systemd/ledmatrix.service"
+    template = project_root / template_rel
+    if not template.is_file():
+        pytest.skip("repo unit template not present")
+
+    installed = tmp_path / "ledmatrix.service"
+    installed.write_text(
+        template.read_text(encoding="utf-8")
+        .replace("__PROJECT_ROOT_DIR__", str(project_root))
+        .replace("__USER__", "root"),
+        encoding="utf-8")
+
+    validator._UNITS = ((template_rel, str(installed)),)
+    validator._validate_systemd_units()
+    assert not validator.warnings, f"a matching unit warned: {validator.warnings}"
+    assert not validator.errors
+
+
+def test_comments_and_blank_lines_are_not_drift():
+    """Otherwise every comment the repo adds would look like a changed unit."""
+    a = "[Service]\n# explains a setting\nExecStart=/x\nRestart=always\n"
+    b = "[Service]\nExecStart=/x\n\nRestart=always\n"
+    assert StartupValidator._unit_body(a) == StartupValidator._unit_body(b)
+
+
+def test_a_changed_directive_is_drift():
+    a = "[Service]\nExecStart=/x\nMemoryMax=85%\n"
+    b = "[Service]\nExecStart=/x\n"
+    assert StartupValidator._unit_body(a) != StartupValidator._unit_body(b)
+
+
+def test_reordered_directives_are_not_drift():
+    """systemd does not care about order within a section, so neither should this."""
+    a = "[Service]\nExecStart=/x\nRestart=always\n"
+    b = "[Service]\nRestart=always\nExecStart=/x\n"
+    assert StartupValidator._unit_body(a) == StartupValidator._unit_body(b)
+
+
+def test_cosmetic_differences_do_not_warn(validator, tmp_path):
+    """Through the real comparison, not the helper.
+
+    The repo's template carries explanatory comments the installed copy may not
+    have, and the installer does not preserve ordering or blank lines. If those
+    counted as drift, every boot would warn and the warning would be ignored.
+    Asserting this on _unit_body alone would not catch a comparison that stopped
+    calling it -- which is exactly what a careless edit does.
+    """
+    project_root = Path("src/startup_validator.py").resolve().parent.parent
+    template_rel = "systemd/ledmatrix.service"
+    template = project_root / template_rel
+    if not template.is_file():
+        pytest.skip("repo unit template not present")
+
+    substituted = (template.read_text(encoding="utf-8")
+                   .replace("__PROJECT_ROOT_DIR__", str(project_root))
+                   .replace("__USER__", "root"))
+    # Same directives, stripped of comments and blank lines and reordered.
+    directives = sorted(line.strip() for line in substituted.splitlines()
+                        if line.strip() and not line.strip().startswith("#"))
+    installed = tmp_path / "ledmatrix.service"
+    installed.write_text("\n".join(reversed(directives)) + "\n", encoding="utf-8")
+
+    validator._UNITS = ((template_rel, str(installed)),)
+    validator._validate_systemd_units()
+    assert not validator.warnings, (
+        f"cosmetic-only difference reported as drift: {validator.warnings}")
+
+
+def test_drift_is_reported_as_a_warning(validator, tmp_path):
+    """The whole point: a real difference must surface, and only as a warning."""
+    installed = tmp_path / "ledmatrix.service"
+    installed.write_text("[Service]\nExecStart=/usr/bin/python3 /x/run.py\n")
+
+    project_root = Path("src/startup_validator.py").resolve().parent.parent
+    template_rel = "systemd/ledmatrix.service"
+    template = project_root / template_rel
+    if not template.is_file():
+        pytest.skip("repo unit template not present")
+
+    validator._UNITS = ((template_rel, str(installed)),)
+    validator._validate_systemd_units()
+
+    assert validator.warnings, "a differing unit produced no warning"
+    assert "install_service.sh" in validator.warnings[0], (
+        "the warning does not tell the user how to fix it")
+    assert not validator.errors, "drift must not be fatal at startup"
+
+
+def test_a_missing_installed_unit_is_silent(validator, tmp_path):
+    """Development checkouts have no /etc/systemd unit; that is not drift."""
+    validator._UNITS = (("systemd/ledmatrix.service", str(tmp_path / "absent.service")),)
+    validator._validate_systemd_units()
+    assert not validator.warnings
+    assert not validator.errors

web_interface/templates/v3/partials/overview.html

@@ -63,13 +63,13 @@
 
 <!-- Getting Started checklist: non-gating, dismissible (localStorage), items
      auto-check from existing config/endpoints — no new persisted state.
-     Known heuristic limits (acceptable, disclosed): values left at legitimate
+     The timezone step is verified against the browser's own zone rather than
-     defaults (e.g. a user actually in Tampa) read as "not done". -->
+     compared to the shipped default; see the data-check="timezone" block below
+     for why. -->
 {% set _hw = main_config.display.hardware if main_config and main_config.display else {} %}
 {% set _hw_done = (_hw.rows or 0) > 0 and (_hw.cols or 0) > 0 and (_hw.chain_length or 0) > 0 %}
 {% set _loc = main_config.location if main_config and main_config.location else {} %}
-{% set _loc_done = (main_config.timezone and main_config.timezone != 'America/New_York')
+{% set _tz = (main_config.timezone if main_config else '') or '' %}
-                   or (_loc.city and _loc.city != 'Tampa') %}
 <div id="getting-started-card" class="bg-blue-50 border border-blue-200 rounded-lg p-4 mb-4" style="display:none" role="region" aria-label="Getting started checklist">
     <div class="flex items-start justify-between">
         <div class="flex-1">
@@ -78,8 +78,8 @@
             <ul class="space-y-1 text-sm" id="getting-started-items">
                 <li><button type="button" class="gs-item text-left w-full" data-done="{{ '1' if _hw_done else '0' }}" data-tab="display">
                     <i class="far fa-square mr-2"></i>Set your panel size (Display tab)</button></li>
-                <li><button type="button" class="gs-item text-left w-full" data-done="{{ '1' if _loc_done else '0' }}" data-tab="general">
+                <li><button type="button" class="gs-item text-left w-full" data-done="0" data-check="timezone" data-tz="{{ _tz }}" data-tab="general">
-                    <i class="far fa-square mr-2"></i>Set your timezone and location (General tab)</button></li>
+                    <i class="far fa-square mr-2"></i>Set your timezone{% if _tz %} — currently {{ _tz }}{% if _loc.city %}, {{ _loc.city }}{% endif %}{% endif %} (General tab)<span data-gs-tz-note class="text-xs"></span></button></li>
                 <li><button type="button" class="gs-item text-left w-full" data-done="0" data-check="installed" data-tab="plugins">
                     <i class="far fa-square mr-2"></i>Install a plugin from the Plugin Store</button></li>
                 <li><button type="button" class="gs-item text-left w-full" data-done="0" data-check="enabled" data-tab="plugins">
@@ -165,6 +165,91 @@
     });
     maybeAutoHide();
 
+    // Timezone: verified against the browser's own zone.
+    //
+    // This step used to tick when the saved timezone differed from the value
+    // config.template.json ships (America/New_York), with the saved city
+    // OR-ed in. Two things were wrong with that. "Differs from the default"
+    // answers "did somebody edit this?", but what the checklist needs to know
+    // is whether the value is RIGHT — so anyone who genuinely lives in the
+    // default zone could never satisfy it and the card nagged forever. And
+    // the city has no bearing on whether the timezone is set: because the two
+    // were OR-ed, saving a city ticked the step off with the timezone still
+    // wrong, which is the direction that actually breaks displays (event
+    // times render in the wrong zone).
+    //
+    // The browser already knows its zone, so compare against that: no new
+    // persisted state, no network, and it catches the reverse case too — a
+    // panel still set to the old zone after a move now stays unticked, where
+    // the old test ticked it the moment the value stopped being the default.
+    function sameZone(a, b) {
+        if (a === b) return true;
+        // Compare the wall-clock time each zone yields, not the identifiers:
+        // aliases (Asia/Calcutta vs Asia/Kolkata, Europe/Kiev vs Europe/Kyiv)
+        // name one zone and must not read as a mismatch.
+        //
+        // Sampled at three instants, all of which have to agree. Checking only
+        // now is not enough: America/New_York and America/Lima hold the same
+        // offset all winter, so a panel set to the wrong one of those would
+        // tick in January and then run an hour off from March. Mid-January and
+        // mid-July sit either side of DST in both hemispheres, so only zones
+        // that agree year-round match -- while Toronto still matches New York,
+        // which is right, since either renders the same times.
+        try {
+            var now = new Date();
+            var year = now.getUTCFullYear();
+            var instants = [now,
+                            new Date(Date.UTC(year, 0, 15, 12)),
+                            new Date(Date.UTC(year, 6, 15, 12))];
+            var stamp = function (tz, at) {
+                // Explicit numeric fields rather than dateStyle/timeStyle:
+                // those are late additions to Intl (Firefox shipped them in
+                // 91), and an implementation that does not know them ignores
+                // them and formats the date alone. That would compare
+                // New York, Chicago and Madrid as equal and tick the step for
+                // a timezone that is plainly wrong -- the exact failure this
+                // check exists to catch. These options have been in Intl
+                // since ECMA-402 v1.
+                return new Intl.DateTimeFormat('en-US', {
+                    timeZone: tz, year: 'numeric', month: '2-digit',
+                    day: '2-digit', hour: '2-digit', minute: '2-digit',
+                    hour12: false
+                }).format(at);
+            };
+            for (var i = 0; i < instants.length; i++) {
+                if (stamp(a, instants[i]) !== stamp(b, instants[i])) {
+                    return false;
+                }
+            }
+            return true;
+        } catch (e) {
+            // An unparseable zone in the config is worth surfacing, not hiding.
+            return false;
+        }
+    }
+
+    (function () {
+        var tzBtn = card.querySelector('[data-check="timezone"]');
+        if (!tzBtn) return;
+        var configured = tzBtn.dataset.tz || '';
+        if (!configured) return;          // nothing saved yet: leave it open
+        var local = '';
+        try {
+            local = (Intl.DateTimeFormat().resolvedOptions().timeZone) || '';
+        } catch (e) {
+            return;                        // no Intl: leave it to the manual tick
+        }
+        if (!local) return;
+        if (sameZone(configured, local)) {
+            markDone(tzBtn);
+            return;
+        }
+        // Unticked on its own says "wrong" without saying why; name the zone
+        // the browser is in so the step is actionable.
+        var note = tzBtn.querySelector('[data-gs-tz-note]');
+        if (note) note.textContent = ' — this browser is in ' + local;
+    }());
+
     // Plugin-derived states from the existing installed-plugins endpoint.
     fetch('/api/v3/plugins/installed')
         .then(function (r) { return r.json(); })

web_interface/templates/v3/partials/plugin_config.html

@@ -296,7 +296,27 @@
                 {% set enum_items = items_schema.get('enum') or [] %}
                 {% set x_options = prop.get('x-options') or {} %}
                 {% set labels = x_options.get('labels') or {} %}
-                
+                {# A saved value that is no longer one of the options -- a team
+                   code the league retired, an option dropped from the schema --
+                   has no checkbox to render, so it would sit unseen in the
+                   hidden input below and be posted back on save. The schema
+                   rejects it and the save endpoint returns 400, which blocks
+                   editing any other field on the plugin until the stale entry
+                   is found and removed. Drop them here instead, and say which,
+                   so the value is not lost silently. Only when the widget
+                   actually has options: an empty enum means nothing to check
+                   against, and filtering on it would wipe the field. #}
+                {% set stale_values = (array_value | reject('in', enum_items) | list) if enum_items else [] %}
+                {% set array_value = (array_value | select('in', enum_items) | list) if enum_items else array_value %}
+
+                {% if stale_values %}
+                <div class="mt-1 mb-2 rounded border border-amber-300 bg-amber-50 px-3 py-2 text-sm text-amber-800"
+                     data-stale-options="{{ field_id }}">
+                    No longer offered, and will be removed when you save:
+                    <span class="font-mono">{{ stale_values | join(', ') }}</span>.
+                </div>
+                {% endif %}
+
                 <div class="mt-1 space-y-2">
                     {% for option in enum_items %}
                     {% set is_checked = option in array_value %}