fix(service): survive corrupt health cache and clean exits

Three independent failure modes that each end with a dark panel and no automatic recovery. 1. PluginHealthTracker._load_health_state returned the cached value verbatim. If that value is not a dict, every caller raises AttributeError: 'list' object has no attribute 'get' — during DisplayController.__init__, so the process dies before the display loop starts. systemd restarts it, the same bad entry is read back from disk, and it dies again: an unattended restart loop that survives reboots because the cause is persisted. Observed in the field with plugin_health:<id> holding an unrelated plugin's list payload. Now non-dict entries are discarded with a warning and the defaults are rebuilt. 2. ledmatrix.service used Restart=on-failure, so any exit with status 0 left the unit stopped and the panel dark indefinitely — systemd treats it as success and never brings it back. Restart=always. 3. ledmatrix-wifi-monitor.service used StandardOutput=syslog, which systemd has marked obsolete; it warns and rewrites it to journal on every load. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-19 17:39:06 +00:00 · 2026-08-18 19:04:57 -04:00
13 changed files with 47 additions and 937 deletions
@@ -1,115 +0,0 @@
 # Running on Low-Memory Boards
 Applies to the Pi Zero 2 W (512 MB), Pi 3 / 3B+ (1 GB), and the 1 GB Pi 4.
 If your board has 2 GB or more you can skip this document.
 ## The failure this prevents
 The display process is the largest thing on the board. On a 1 GB Pi 3B+ with
 around 20 plugins enabled it settles near **600 MB of 905 MB usable**, leaving
 under 200 MB of headroom for everything else.
 When that headroom runs out, the board does not crash cleanly. `fork()` starts
 failing, and because a new process is needed to do almost anything, the
 symptoms look nothing like "out of memory":
 | What you see | Why |
 |---|---|
 | SSH accepts the connection then closes it instantly, before any banner | `sshd` forks a session per connection; the fork fails |
 | The web UI still responds quickly | Already running, serves from existing threads, forks nothing |
 | Ping is perfect, 0% loss | Handled entirely in the kernel |
 | The panel is dark | The display process was killed and cannot be respawned |
 | The clock is wrong after the next boot | `fake-hwclock`'s periodic save is a scheduled job, and it cannot fork either |
 The board looks healthy from the outside and cannot be logged into. Only a
 power cycle clears it. If you are here because SSH stopped working, also see
 [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md), which
 covers the more common cause (AP mode).
 ## Check your headroom
 ```bash
 free -m
 ps -eo rss,comm --sort=-rss | head -5
 ```
 If `MemAvailable` is under ~150 MB while the display is running, you are close
 to the edge. To watch it over time:
 ```bash
 watch -n 30 'free -m | head -2'
 ```
 Available memory that falls steadily rather than holding flat means you will
 reach the wall; it is a question of when.
 ## What to do
 **1. Enable the memory cgroup controller.** Without it, the `MemoryMax=85%` in
 `systemd/ledmatrix.service` is accepted by systemd and silently ignored, so the
 service has no ceiling and a runaway takes the whole board down instead of just
 restarting. Raspberry Pi firmware disables this controller by default.
 `first_time_install.sh` does this for you. To check it took effect:
 ```bash
 grep memory /sys/fs/cgroup/cgroup.controllers
 ```
 If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the
 kernel command line and reboot. Edit whichever file your image uses —
 `/boot/firmware/cmdline.txt` on current Raspberry Pi OS, `/boot/cmdline.txt` on
 older layouts (the installer checks the first and falls back to the second).
 Everything must stay on a single line.
 This changes the failure mode from "the board becomes unreachable" to "the
 display service restarts". It is a safety net, not a fix.
 **2. Run fewer plugins.** This is the actual remedy. Every enabled plugin costs
 memory permanently — its module, its parsed config, and its cached API
 responses. On a 512 MB or 1 GB board, keep the enabled set small and prefer
 plugins that poll infrequently.
 **3. Lower the cache ceiling.** The in-memory cache is sized from total RAM
 (150 entries at 1 GB and below, up to 1500 at 8 GB). To go lower still:
 ```ini
 # /etc/systemd/system/ledmatrix.service.d/override.conf
 [Service]
 Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75
 ```
 Writing the file does not change the running service. Reload systemd and
 restart it:
 ```bash
 sudo systemctl daemon-reload
 sudo systemctl restart ledmatrix
 ```
 Fewer entries means more API calls, so lower this only while you are actually
 short of memory.
 **4. Consider `MemoryHigh`.** `MemoryMax` kills and restarts. `MemoryHigh`
 throttles and reclaims instead, which is gentler — but on a board where the
 process genuinely wants more than the limit, sustained reclaim can stall the
 render loop and show as visible stutter on the panel. Add it only if you prefer
 degraded output to a restart:
 ```ini
 [Service]
 MemoryHigh=70%
 ```
 ## Keep your logs
 These images default to volatile journald storage, so every reboot destroys the
 logs — including the ones explaining why the board rebooted. `first_time_install.sh`
 enables persistent storage capped at 64 MB. To confirm:
 ```bash
 journalctl --list-boots
 ```
 More than one boot listed means logs are surviving reboots. If only one is
 listed, journald is still writing to `/run` (tmpfs).
@@ -14,7 +14,6 @@ the one-shot installer. The pages here go deeper.
 5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes
 6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install
 7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems
 8. [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) — Pi Zero 2 W / 3B+ / 1GB Pi 4 memory limits
 ## I want to write a plugin
@@ -20,22 +20,7 @@ The installation script:
 - Installs and configures `dnsmasq` (DHCP server for AP mode)
 - These services can interfere with normal WiFi client mode
-### 3. The Board Ran Out of Memory
+### 3. Reboot After Installation
 On a 512MB or 1GB board, memory exhaustion stops `sshd` being able to fork a
 session process. The connection is accepted and then closed immediately, before
 any banner:
 ```text
 kex_exchange_identification: Connection closed by remote host
 ```
 The giveaway is that the board is otherwise healthy — ping is clean and the web
 UI still responds — but nothing that needs to start a new process works, and
 the panel is usually dark. Only a power cycle clears it. See
 [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md).
 ### 4. Reboot After Installation
 If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode.
@@ -205,23 +190,11 @@ The web interface allows you to:
 ## Summary
-**SSH becomes unavailable because** — two unrelated causes, and they need
+**SSH becomes unavailable because**:
 different responses:
 *AP mode (most common):*
 - WiFi monitor service enables AP mode when WiFi disconnects
 - AP mode switches WiFi from client to access point mode
 - Pi loses connection to your original network
 *Memory exhaustion (low-memory boards):*
 - The board runs out of memory, so `sshd` cannot fork a session process
 - The connection is accepted and closed before any banner
 - Ping still answers and the web UI still responds, so it looks healthy
 - The panel is usually dark and the service cannot restart
 - **Only a power cycle clears this** — there is no remote recovery, because
  every remote route needs a new process
 - Prevention and tuning: [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md)
 **To regain SSH**:
 1. Connect to **LEDMatrix-Setup** AP network (password: `ledmatrix123`)
 2. SSH to `192.168.4.1`
@@ -1688,94 +1688,6 @@ else
    echo "✗ $CMDLINE_FILE not found; skipping isolcpus optimization"
 fi
 # Enable the memory cgroup controller (idempotent).
 # The Pi firmware boots with cgroup_disable=memory, so systemd's MemoryMax= is
 # accepted and silently ignored — the display service then has no ceiling, and
 # a runaway takes the whole board down (sshd can no longer fork, the panel goes
 # dark) rather than just restarting the one service.
 if [ "$SKIP_PERF" != "1" ] && [ -f "$CMDLINE_FILE" ]; then
    # Both parameters are required for the memory controller, and they can get
    # separated -- an image, another tool or a half-applied earlier run can
    # leave one without the other. Checking only cgroup_enable=memory would
    # report success while MemoryMax= silently does nothing, so each is checked
    # and appended independently.
    cgroup_missing=""
    for cgroup_param in cgroup_enable=memory cgroup_memory=1; do
        if ! grep -qw "$cgroup_param" "$CMDLINE_FILE"; then
            cgroup_missing="$cgroup_missing $cgroup_param"
        fi
    done
    if [ -z "$cgroup_missing" ]; then
        echo "cgroup memory parameters already present in $CMDLINE_FILE"
    else
        echo "Adding${cgroup_missing} to $CMDLINE_FILE..."
        cp "$CMDLINE_FILE" "$CMDLINE_FILE.bak" 2>/dev/null || true
        # The kernel command line must stay on one line.
        sed -i "1 s|\$|${cgroup_missing}|" "$CMDLINE_FILE"
        echo "  Takes effect after reboot. Verify with:"
        echo "    grep memory /sys/fs/cgroup/cgroup.controllers"
    fi
 fi
 # Persist the journal (idempotent).
 # These images default to volatile storage: journald keeps everything in /run
 # (tmpfs), so every reboot destroys the logs — including the ones that would
 # explain why the board rebooted. Capped so an SD card is not worn out by logs.
 # A non-empty /var/log/journal does not prove journald is configured the way
 # this needs: the directory survives a switch back to volatile storage, and it
 # says nothing about whether a size cap is set. Read the effective
 # configuration instead, and only write the keys the user has not set
 # themselves so an explicit local limit is preserved.
 journald_effective() {
    # systemd-analyze merges journald.conf with every drop-in; grep is the
    # fallback for images that ship without it.
    if command -v systemd-analyze >/dev/null 2>&1 &&
       systemd-analyze cat-config systemd/journald.conf >/dev/null 2>&1; then
        systemd-analyze cat-config systemd/journald.conf 2>/dev/null
    else
        cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null
    fi
 }
 journald_conf="$(journald_effective)"
 journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
 journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
 if [ "$journald_storage" = "persistent" ] && [ -n "$journald_cap" ]; then
    echo "Persistent journald storage already configured (SystemMaxUse=$journald_cap)"
 else
    echo "Enabling persistent journald storage..."
    mkdir -p /etc/systemd/journald.conf.d
    {
        echo "# Installed by LEDMatrix first_time_install.sh"
        echo "[Journal]"
        echo "Storage=persistent"
        if [ -n "$journald_cap" ]; then
            echo "# SystemMaxUse left to your existing setting ($journald_cap)"
        else
            # Capped so logs cannot wear out or fill an SD card.
            echo "SystemMaxUse=64M"
        fi
    } > /etc/systemd/journald.conf.d/ledmatrix-persistent.conf
    mkdir -p /var/log/journal
    systemd-tmpfiles --create --prefix /var/log/journal >/dev/null 2>&1 || true
    systemctl restart systemd-journald >/dev/null 2>&1 || true
    # Drop-ins are applied in lexical order, so a locally added file that sorts
    # after ledmatrix-persistent.conf (zz-local.conf and friends) still wins.
    # Writing the file is not evidence it took effect -- re-read and say so
    # plainly rather than reporting success we cannot confirm.
    journald_now="$(journald_effective | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
    if [ "$journald_now" = "persistent" ]; then
        echo "  Persistent journald storage active"
    else
        echo "  WARNING: journald storage is still '${journald_now:-unset}' after"
        echo "  writing /etc/systemd/journald.conf.d/ledmatrix-persistent.conf."
        echo "  Another drop-in that sorts later is overriding it. Check:"
        echo "    systemd-analyze cat-config systemd/journald.conf | grep -n Storage="
        echo "  Logs will not survive a reboot until that is resolved."
    fi
 fi
 # Ensure dtparam=audio=off in config.txt (idempotent)
 if [ "$SKIP_PERF" = "1" ]; then
    : # skipped
@@ -4,58 +4,11 @@ Memory Cache
 Handles in-memory caching with TTL support, size limits, and automatic cleanup.
 """
 import os
 import time
 import threading
 import logging
 from typing import Dict, Any, Optional
 # Historical fixed ceiling, kept as the fallback when RAM cannot be read.
 DEFAULT_MAX_SIZE = 1000
 def _total_memory_mb() -> Optional[float]:
    """Physical RAM in MB, or None where /proc/meminfo is unavailable."""
    try:
        with open('/proc/meminfo', 'r', encoding='utf-8') as fh:
            for line in fh:
                if line.startswith('MemTotal:'):
                    return int(line.split()[1]) / 1024
    except (OSError, ValueError, IndexError):
        return None
    return None
 def default_max_size() -> int:
    """Entry ceiling scaled to this machine's RAM.
    One fixed ceiling cannot serve both a 512 MB Pi Zero 2 W and an 8 GB Pi 5.
    Entries here are parsed API payloads that routinely run tens of kilobytes
    each, so a thousand of them is a comfortable cache on a large board and a
    substantial fraction of total RAM on a small one — where the process
    competing for that RAM is also driving the panel. Set
    LEDMATRIX_CACHE_MAX_ENTRIES to override.
    """
    override = os.environ.get('LEDMATRIX_CACHE_MAX_ENTRIES')
    if override:
        try:
            value = int(override)
            if value > 0:
                return value
        except ValueError:
            pass
    total_mb = _total_memory_mb()
    if total_mb is None:
        return DEFAULT_MAX_SIZE
    if total_mb < 1536:      # 512 MB and 1 GB boards
        return 150
    if total_mb < 3072:      # 2 GB
        return 400
    if total_mb < 6144:      # 4 GB
        return 800
    return 1500              # 8 GB and up
 class MemoryCache:
    """Manages in-memory cache with TTL and size limits."""
@@ -134,32 +87,6 @@ class MemoryCache:
        with self._lock:
            self._cache[key] = value
            self._timestamps[key] = time.time()
            # Enforce the ceiling here rather than leaving it to the periodic
            # cleanup, which only runs every cleanup_interval seconds (300 by
            # default). A burst of inserts between two sweeps could otherwise
            # take the cache far past _max_size, which is the memory growth this
            # limit exists to prevent -- and on a 1GB board that is the
            # difference between a bounded cache and an unreachable Pi.
            self._evict_over_limit_locked()
    def _evict_over_limit_locked(self) -> int:
        """Drop oldest entries until the cache is within _max_size.
        Caller must hold self._lock. Returns the number of entries removed.
        """
        excess = len(self._cache) - self._max_size
        if excess <= 0:
            return 0
        oldest = sorted(
            self._timestamps.items(),
            key=lambda item: float(item[1]) if isinstance(item[1], (int, float)) else 0.0
        )
        removed = 0
        for key, _ in oldest[:excess]:
            self._cache.pop(key, None)
            self._timestamps.pop(key, None)
            removed += 1
        return removed
    def clear(self, key: Optional[str] = None) -> None:
        """
@@ -216,8 +143,22 @@ class MemoryCache:
                self._timestamps.pop(key, None)
                removed_count += 1
-            # Same ceiling enforcement set() uses, so the two cannot drift.
+            # Enforce size limit by removing oldest entries if cache is too large
-            removed_count += self._evict_over_limit_locked()
+            if len(self._cache) > self._max_size:
                # Sort by timestamp (oldest first)
                sorted_entries = sorted(
                    self._timestamps.items(),
                    key=lambda x: float(x[1]) if isinstance(x[1], (int, float)) else 0
                )
                # Remove oldest entries until we're under the limit
                excess_count = len(self._cache) - self._max_size
                for i in range(excess_count):
                    if i < len(sorted_entries):
                        key = sorted_entries[i][0]
                        self._cache.pop(key, None)
                        self._timestamps.pop(key, None)
                        removed_count += 1
            self._last_cleanup = current_time
@@ -33,7 +33,7 @@ import logging
 import threading
 import tempfile
 from src.exceptions import CacheError
-from src.cache.memory_cache import MemoryCache, default_max_size
+from src.cache.memory_cache import MemoryCache
 from src.cache.disk_cache import DiskCache
 from src.cache.cache_strategy import CacheStrategy
 from src.cache.cache_metrics import CacheMetrics
@@ -84,9 +84,7 @@ class CacheManager:
            self.logger.warning("ConfigManager not available, using default cache intervals")
        # Initialize cache components using composition
-        self._memory_cache_component = MemoryCache(
+        self._memory_cache_component = MemoryCache(max_size=1000, cleanup_interval=300.0)
            max_size=default_max_size(), cleanup_interval=300.0
        )
        self._disk_cache_component = DiskCache(cache_dir=self.cache_dir, logger=self.logger)
        self._strategy_component = CacheStrategy(config_manager=self.config_manager, logger=self.logger)
        self._metrics_component = CacheMetrics(logger=self.logger)
@@ -7,7 +7,7 @@ and circuit breaker state. Provides automatic recovery mechanisms.
 import time
 import logging
-from typing import Dict, Optional, Any, Tuple
+from typing import Dict, Optional, Any
 from enum import Enum
@@ -65,47 +65,20 @@ class PluginHealthTracker:
        )
        if isinstance(cached, dict) and cached:
-            # Complete it rather than trusting it: a persisted record can be
+            return cached
            # missing fields the callers index directly (a partial write, a
            # restored backup, an older schema), and returning it verbatim makes
            # record_success / record_failure raise KeyError, which takes the
            # display down in a restart loop that survives reboots because the
            # bad entry is on disk.
            state, repaired = self._repair_health_state(cached)
            if repaired:
                self.logger.warning(
                    f"Repaired health state for {plugin_id}: "
                    f"{sorted(repaired)} missing or invalid, using defaults for those."
                )
            return state
-        # Not a dict at all: written by something other than
+        # A cache entry that is not a dict means the persisted state was written
-        # _save_health_state (a key collision, a corrupted entry). Nothing to
+        # by something other than _save_health_state (a key collision, a partial
-        # salvage.
+        # write, a restored backup). Returning it verbatim makes every caller
        # blow up on .get(), which takes the display down in a restart loop that
        # survives reboots because the bad entry is on disk. Discard and rebuild.
        if cached is not None and not isinstance(cached, dict):
            self.logger.warning(
                f"Discarding malformed health state for {plugin_id}: expected "
                f"dict, got {type(cached).__name__}. Falling back to defaults."
            )
-
+        
-        return self._default_health_state()
+        # Default state
    def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None:
        """Save health state to cache."""
        cache_key = self._get_health_key(plugin_id)
        self.cache_manager.set(cache_key, state)  # Persist indefinitely
        self._health_state[plugin_id] = state
    # The fields callers index directly (state['circuit_state'] and friends).
    # A cached dict missing any of them raises KeyError deep in record_success /
    # record_failure, so the value is completed before it is handed out.
    _COUNTER_FIELDS = ('consecutive_failures', 'total_failures', 'total_successes')
    _TIMESTAMP_FIELDS = ('last_success_time', 'last_failure_time',
                         'circuit_opened_time', 'half_open_start_time')
    @staticmethod
    def _default_health_state() -> Dict[str, Any]:
        """A fresh state with every field the callers expect."""
        return {
            'consecutive_failures': 0,
            'total_failures': 0,
@@ -115,56 +88,15 @@ class PluginHealthTracker:
            'circuit_state': CircuitState.CLOSED.value,
            'circuit_opened_time': None,
            'half_open_start_time': None,
-            'last_error': None,
+            'last_error': None
        }
-
+    
-    @classmethod
+    def _save_health_state(self, plugin_id: str, state: Dict[str, Any]) -> None:
-    def _repair_health_state(cls, cached: Dict[str, Any]) -> Tuple[Dict[str, Any], list]:
+        """Save health state to cache."""
-        """Return `cached` completed against the defaults, plus what was repaired.
+        cache_key = self._get_health_key(plugin_id)
-
+        self.cache_manager.set(cache_key, state)  # Persist indefinitely
-        Per-field rather than all-or-nothing: a record that has real failure
+        self._health_state[plugin_id] = state
-        counts but is missing `last_error` should keep the counts, not be reset
+    
        to healthy. Only values that are absent or the wrong type fall back to
        the default, so a partial or older-schema record survives with whatever
        it does carry, while every field the callers index is guaranteed present
        and of a usable type.
        """
        state = cls._default_health_state()
        repaired = []
        for field, default in state.items():
            if field not in cached:
                repaired.append(field)
                continue
            value = cached[field]
            if field in cls._COUNTER_FIELDS:
                ok = isinstance(value, int) and not isinstance(value, bool) and value >= 0
            elif field in cls._TIMESTAMP_FIELDS:
                # bool is a subclass of int, so True would pass as a timestamp
                # and then compare as 1.0 -- expiring a cooldown the instant it
                # opens, or (False) making the elapsed check never fire.
                ok = value is None or (
                    isinstance(value, (int, float)) and not isinstance(value, bool)
                )
            elif field == 'circuit_state':
                # Membership first requires the value to be hashable: a list or
                # dict here would raise TypeError out of the repair itself,
                # which is the crash this whole path exists to prevent.
                ok = isinstance(value, str) and value in {
                    member.value for member in CircuitState
                }
            else:  # last_error
                ok = value is None or isinstance(value, str)
            if ok:
                state[field] = value
            else:
                repaired.append(field)
        # Anything the schema has since grown (degraded, degraded_reason) is
        # read with .get() by its callers, so carry it through untouched.
        for field, value in cached.items():
            if field not in state:
                state[field] = value
        return state, repaired
    def get_health_state(self, plugin_id: str, force_reload: bool = False) -> Dict[str, Any]:
        """Get current health state for a plugin.
@@ -14,7 +14,7 @@ import sys
 import subprocess
 import threading
 from pathlib import Path
-from typing import Dict, Any, List, Optional, Tuple, Type
+from typing import Dict, Any, Optional, Tuple, Type
 import logging
 from packaging.requirements import InvalidRequirement, Requirement
@@ -45,76 +45,6 @@ def requirements_has_real_deps(requirements_file: str) -> bool:
    return False
 def _extra_dependencies(dist_name: str, extras) -> Optional[List[Requirement]]:
    """Dependencies a distribution declares *only* behind the given extras.
    Returns None when the installed metadata cannot be read or parsed, so the
    caller can fall back to running pip rather than assuming anything.
    """
    try:
        meta = importlib.metadata.metadata(dist_name)
    except importlib.metadata.PackageNotFoundError:
        return None
    gated: List[Requirement] = []
    for raw in meta.get_all('Requires-Dist') or []:
        try:
            dep = Requirement(raw)
        except InvalidRequirement:
            return None
        if dep.marker is None:
            continue
        # Keep only what the distribution gates behind an extra we asked for:
        # satisfied when `extra` is that name, but not when no extra is
        # requested. A marker that holds either way (python_version, sys_platform)
        # belongs to the base install and is already covered by the version check.
        if dep.marker.evaluate({'extra': ''}):
            continue
        if any(dep.marker.evaluate({'extra': extra}) for extra in extras):
            gated.append(dep)
    return gated
 def _extras_are_satisfied(req: Requirement, _visited: Optional[set] = None) -> bool:
    """Check the dependencies pulled in by req's extras are installed.
    Follows extras through nested extras. A gated dependency can itself request
    one (`requests[socks]`), and checking only that `requests` is installed at
    an acceptable version says nothing about whether the socks extra's own
    dependency is there -- so the caller would skip pip and the plugin would
    fail at import instead. Plain dependencies are still checked one level
    deep, which is all that is needed to tell "the extra was installed" from
    "the extra was never installed".
    `_visited` carries the (distribution, extras) pairs already seen, so a
    dependency cycle between extras terminates instead of recursing forever.
    Anything unreadable returns False, so the caller still falls through to pip.
    """
    if _visited is None:
        _visited = set()
    marker = (req.name.lower(), frozenset(e.lower() for e in req.extras))
    if marker in _visited:
        # Already accounted for higher up the chain; treating a cycle as
        # satisfied here is safe because the outer frame still has to pass.
        return True
    _visited.add(marker)
    gated = _extra_dependencies(req.name, req.extras)
    if gated is None:
        return False
    for dep in gated:
        try:
            dep_version = importlib.metadata.version(dep.name)
        except importlib.metadata.PackageNotFoundError:
            return False
        if dep.specifier and not dep.specifier.contains(dep_version, prereleases=True):
            return False
        if dep.extras and not _extras_are_satisfied(dep, _visited):
            return False
    return True
 def requirements_are_satisfied(requirements_file: str) -> bool:
    """
    Check whether every real requirement line in requirements.txt is already
@@ -146,6 +76,9 @@ def requirements_are_satisfied(requirements_file: str) -> bool:
        except InvalidRequirement:
            return False
        if req.extras:
            return False  # verifying extras' sub-dependencies isn't worth it here
        if req.marker is not None and not req.marker.evaluate():
            continue  # not applicable on this platform/interpreter
@@ -157,9 +90,6 @@ def requirements_are_satisfied(requirements_file: str) -> bool:
        if req.specifier and not req.specifier.contains(installed_version, prereleases=True):
            return False
        if req.extras and not _extras_are_satisfied(req):
            return False
    return True
@@ -15,19 +15,6 @@ ExecStart=/usr/bin/python3 __PROJECT_ROOT_DIR__/run.py
 # that a successful outcome and never bringing it back.
 Restart=always
 RestartSec=10
 # Memory ceiling as a share of physical RAM, so one unit file suits a 512 MB
 # Pi Zero 2 W and an 8 GB Pi 5 alike. This is a backstop, not a tuning knob: it
 # turns "the board runs out of memory, stops being able to fork, and takes sshd
 # and the panel down together until someone pulls the plug" into "this one
 # service restarts".
 #
 # NOTE: Raspberry Pi firmware boots the kernel with cgroup_disable=memory, and
 # systemd accepts this setting and then silently ignores it. Verify with:
 #     grep memory /sys/fs/cgroup/cgroup.controllers
 # If that prints nothing, add "cgroup_enable=memory cgroup_memory=1" to
 # /boot/firmware/cmdline.txt (all on line 1) and reboot. first_time_install.sh
 # does this for you.
 MemoryMax=85%
 StandardOutput=journal
 StandardError=journal
 SyslogIdentifier=ledmatrix
@@ -458,26 +458,3 @@ class TestDiskCacheWriteEconomy:
        cache = DiskCache(cache_dir=str(tmp_path))
        cache.set("k", {"when": datetime(2026, 7, 12, 10, 30)})
        assert cache.get("k") == {"when": "2026-07-12T10:30:00"}
 # --- the ceiling has to hold between cleanup sweeps ---------------------------
 def test_memory_cache_enforces_ceiling_on_every_write():
    """_cleanup_memory_cache only runs every cleanup_interval seconds (300 by
    default). If set() accepted entries without bound in between, a burst could
    take the cache far past max_size -- which is the unbounded growth the limit
    exists to prevent, and on a 1GB board the difference between a bounded cache
    and a Pi that cannot fork.
    """
    from src.cache.memory_cache import MemoryCache
    cache = MemoryCache(max_size=150, cleanup_interval=300.0)
    for i in range(1000):
        cache.set(f"k{i}", {"v": i})
    assert len(cache._cache) <= 150
    # The timestamp map has to be evicted alongside the values, or it becomes
    # the leak instead.
    assert len(cache._timestamps) <= 150
    assert cache.get("k999") is not None, "the newest write must survive"
    assert cache.get("k0") is None, "the oldest must be the one evicted"
@@ -1,241 +0,0 @@
 """
 Getting Started checklist: what the server decides, and what it must not.
 The timezone step used to tick server-side when the saved timezone differed
 from the shipped default, OR-ed with the saved city. That made the step
 unsatisfiable for anyone genuinely in the default zone (the card nagged
 forever), and let a saved city tick it off while the timezone was still wrong.
 The step is now verified in the browser against its own zone, so the server's
 only job is to hand over the configured value and stay out of the decision.
 These tests pin that contract: the panel-size step still reflects config, the
 timezone step never pre-ticks, it carries the configured zone, and the city
 has no influence on it.
 """
 import copy
 import re
 import sys
 from pathlib import Path
 from unittest.mock import MagicMock
 import pytest
 from flask import Flask
 PROJECT_ROOT = Path(__file__).parent.parent
 sys.path.insert(0, str(PROJECT_ROOT))
 BASE_CONFIG = {
    "timezone": "America/New_York",
    "location": {"city": "Tampa", "state": "Florida", "country": "US"},
    "display": {
        "hardware": {"rows": 32, "cols": 64, "chain_length": 2, "parallel": 1},
        "runtime": {},
        "double_sided": {"enabled": False},
        "vegas_scroll": {"plugin_order": [], "excluded_plugins": []},
        "plugin_rotation_order": [],
    },
    "plugin_system": {},
    "schedule": {},
    "dim_schedule": {},
    "sync": {},
 }
 def render(config):
    """Render the overview partial against one config, as app.py would."""
    base = PROJECT_ROOT / "web_interface"
    app = Flask(
        __name__,
        template_folder=str(base / "templates"),
        static_folder=str(base / "static"),
    )
    app.config["TESTING"] = True
    from web_interface.blueprints import pages_v3 as pv
    # pages_v3 is a module-level singleton shared across the test process;
    # restore whatever the previous test left on it.
    original_cm = getattr(pv.pages_v3, "config_manager", None)
    original_pm = getattr(pv.pages_v3, "plugin_manager", None)
    mock_cm = MagicMock()
    mock_cm.load_config.return_value = config
    mock_cm.get_raw_file_content.return_value = config
    pv.pages_v3.config_manager = mock_cm
    mock_pm = MagicMock()
    mock_pm.plugins = {}
    mock_pm.get_all_plugin_info.return_value = []
    mock_pm.get_plugin_display_modes.side_effect = lambda pid: []
    pv.pages_v3.plugin_manager = mock_pm
    app.register_blueprint(pv.pages_v3, url_prefix="")
    try:
        resp = app.test_client().get("/partials/overview")
        assert resp.status_code == 200, resp.status_code
        return resp.get_data(as_text=True)
    finally:
        pv.pages_v3.config_manager = original_cm
        pv.pages_v3.plugin_manager = original_pm
 def timezone_step(body):
    """The checklist <button> for the timezone step."""
    match = re.search(r"<button[^>]*data-check=\"timezone\"[^>]*>", body)
    assert match, "timezone step not found in the rendered checklist"
    return match.group(0)
 def config_with(**overrides):
    config = copy.deepcopy(BASE_CONFIG)
    for key, value in overrides.items():
        config[key] = value
    return config
@pytest.mark.parametrize(
    "timezone",
    ["America/New_York", "America/Los_Angeles", "Europe/Madrid", "Asia/Kolkata"],
 )
 def test_timezone_step_never_pre_ticks_server_side(timezone):
    """The browser owns this decision; the server must not pre-empt it.
    The default zone is in the list deliberately: that is the case the old
    default-comparison could never tick.
    """
    step = timezone_step(render(config_with(timezone=timezone)))
    assert 'data-done="0"' in step, step
@pytest.mark.parametrize(
    "timezone",
    ["America/New_York", "Europe/Madrid", "Pacific/Auckland"],
 )
 def test_timezone_step_carries_the_configured_zone(timezone):
    """JS compares data-tz against the browser, so it has to be the real value."""
    assert f'data-tz="{timezone}"' in timezone_step(render(config_with(timezone=timezone)))
 def test_city_does_not_influence_the_timezone_step():
    """The coupling this change removes: city said nothing about the timezone,
    and OR-ing it let a saved city tick the step off with the zone still wrong.
    timezone_step() returns the opening tag only, so this compares the state
    the step is in -- data-done and data-tz -- and not the label, which does
    still show the configured city as context and so differs between the two.
    """
    tampa = timezone_step(render(config_with(
        location={"city": "Tampa", "state": "Florida", "country": "US"})))
    seattle = timezone_step(render(config_with(
        location={"city": "Seattle", "state": "Washington", "country": "US"})))
    assert tampa == seattle
 def test_missing_timezone_leaves_the_step_open():
    """Nothing saved means nothing to verify: the step stays unticked and the
    JS bails on the empty value rather than comparing against ''."""
    step = timezone_step(render(config_with(timezone="")))
    assert 'data-tz=""' in step
    assert 'data-done="0"' in step
 def test_zone_comparison_asks_for_the_time_of_day():
    """Guard on the Intl options, which look like a stylistic choice.
    dateStyle/timeStyle are late additions (Firefox shipped them in 91). An
    implementation that does not know them ignores them and formats the date
    alone -- which compares New York, Chicago and Madrid as equal and ticks
    the step for a timezone that is plainly wrong. Explicit numeric fields
    have been in Intl since ECMA-402 v1.
    """
    template = (PROJECT_ROOT / "web_interface" / "templates" / "v3"
                / "partials" / "overview.html").read_text()
    body = template[template.index("function sameZone"):]
    body = body[:body.index("}())")]
    # The comment above the options names dateStyle/timeStyle to explain why
    # they are not used, so match on code only.
    body = "\n".join(line for line in body.splitlines()
                     if not line.lstrip().startswith("//"))
    assert "dateStyle" not in body and "timeStyle" not in body, (
        "zone comparison must not depend on dateStyle/timeStyle")
    for field in ("hour:", "minute:", "year:", "month:", "day:"):
        assert field in body, f"zone comparison dropped {field!r}"
 def test_zone_comparison_samples_both_sides_of_dst():
    """One instant is not enough, and the shortfall is invisible for months.
    America/New_York and America/Lima hold the same offset all winter, so a
    check against now alone ticks the step in January for a panel that runs an
    hour off from March. The comparison has to sample instants either side of
    DST -- mid-January and mid-July, which covers both hemispheres.
    """
    template = (PROJECT_ROOT / "web_interface" / "templates" / "v3"
                / "partials" / "overview.html").read_text()
    body = template[template.index("function sameZone"):]
    body = body[:body.index("}())")]
    code = "\n".join(line for line in body.splitlines()
                     if not line.lstrip().startswith("//"))
    assert "Date.UTC" in code, (
        "zone comparison samples only the current instant, so zones that "
        "coincide seasonally would read as equal")
    assert code.count("Date.UTC") >= 2, "expected an instant either side of DST"
 def _stamp(zone, instant):
    """The JS comparison's algorithm, for pinning what it must decide.
    There is no JS runtime here (and the repo has no JS test infra), so this
    mirrors sameZone rather than executing it: same instants, same wall-clock
    equality. It records the verdicts the shipped code has to reach.
    """
    from zoneinfo import ZoneInfo
    return instant.astimezone(ZoneInfo(zone)).strftime("%m/%d/%Y %H:%M")
@pytest.mark.parametrize(
    "left,right,equivalent",
    [
        # Aliases: one zone under two names.
        ("Asia/Calcutta", "Asia/Kolkata", True),
        ("Europe/Kiev", "Europe/Kyiv", True),
        # Same rules year-round: either renders the same times, so a panel set
        # to one and browsed from the other is correctly configured.
        ("America/New_York", "America/Toronto", True),
        # Coincide in winter only -- the case a single-instant check gets wrong.
        ("America/New_York", "America/Lima", False),
        ("America/Phoenix", "America/Los_Angeles", False),
        ("Australia/Sydney", "Pacific/Guadalcanal", False),
        # Plainly different.
        ("America/New_York", "America/Chicago", False),
        ("America/New_York", "Europe/Madrid", False),
    ],
 )
 def test_which_zone_pairs_must_count_as_the_same(left, right, equivalent):
    from datetime import datetime
    from zoneinfo import ZoneInfo
    year = 2026
    instants = [datetime(year, 1, 15, 12, tzinfo=ZoneInfo("UTC")),
                datetime(year, 7, 15, 12, tzinfo=ZoneInfo("UTC"))]
    matched = all(_stamp(left, at) == _stamp(right, at) for at in instants)
    assert matched is equivalent, (
        f"{left} vs {right}: sampling both seasons gave {matched}")
@pytest.mark.parametrize(
    "hardware,expected",
    [
        ({"rows": 32, "cols": 64, "chain_length": 2, "parallel": 1}, "1"),
        ({"rows": 0, "cols": 0, "chain_length": 0, "parallel": 1}, "0"),
    ],
 )
 def test_panel_size_step_still_reflects_config(hardware, expected):
    """Regression guard: the hardware step is still decided server-side."""
    config = config_with()
    config["display"]["hardware"] = hardware
    body = render(config)
    match = re.search(r"<button[^>]*data-tab=\"display\"[^>]*>", body)
    assert match, "panel-size step not found"
    assert f'data-done="{expected}"' in match.group(0), match.group(0)
@@ -91,101 +91,3 @@ def test_force_reload_refreshes_stale_in_memory_snapshot():
    # and it asked the cache to bypass the in-memory tier (memory_ttl=0).
    assert any(c.kwargs.get("memory_ttl") == 0 for c in cache.get.call_args_list)
 # --- persisted state that does not match the current schema -------------------
 #
 # A record on disk can be missing fields the callers index directly: a partial
 # write, a restored backup, or a state written by an older schema. Returning it
 # verbatim raises KeyError inside record_success / record_failure, which takes
 # the display down in a restart loop that survives reboots, because the bad
 # entry is on disk and gets read again on the way back up. Observed in the wild
 # as `plugin clock-simple operation failed: 'circuit_state'`, repeating ~50x a
 # minute with the panel frozen.
 _INDEXED_FIELDS = (
    "consecutive_failures", "total_failures", "total_successes",
    "last_success_time", "last_failure_time", "circuit_state",
    "circuit_opened_time", "half_open_start_time", "last_error",
 )
 def _tracker_reading(persisted):
    cache = _cache()
    cache.get.return_value = persisted
    return PluginHealthTracker(cache)
 def test_partial_state_is_completed_not_returned_raw():
    """The shape seen in the wild: one field, everything else absent."""
    state = _tracker_reading({"circuit_state": "closed"}).get_health_state("p")
    for field in _INDEXED_FIELDS:
        assert field in state, f"{field} missing; callers index it directly"
 def test_repair_keeps_real_failure_history():
    """A record with genuine counts must not be reset to healthy just because
    an optional field is absent -- that would clear a tripped breaker."""
    state = _tracker_reading({
        "consecutive_failures": 5,
        "total_failures": 5,
        "circuit_state": "open",
    }).get_health_state("p")
    assert state["consecutive_failures"] == 5
    assert state["total_failures"] == 5
    assert state["circuit_state"] == "open"
 def test_wrong_types_fall_back_per_field():
    """A counter persisted as a string would pass a membership check and then
    fail on the first += 1; an unknown circuit_state would take a branch the
    breaker has no handling for."""
    state = _tracker_reading({
        "consecutive_failures": "3",
        "circuit_state": "melted",
        "total_failures": 7,
    }).get_health_state("p")
    assert state["consecutive_failures"] == 0
    assert state["circuit_state"] == CircuitState.CLOSED.value
    assert state["total_failures"] == 7, "valid neighbours must survive"
 def test_newer_fields_are_carried_through():
    """degraded/degraded_reason are read with .get() and are not part of the
    indexed set; repairing must not drop them."""
    state = _tracker_reading({
        "circuit_state": "closed", "degraded": True, "degraded_reason": "x",
    }).get_health_state("p")
    assert state["degraded"] is True
    assert state["degraded_reason"] == "x"
 def test_recording_against_a_repaired_state_does_not_raise():
    """The actual failure: record_failure indexing a field that was not there.
    The seed deliberately omits circuit_state. Seeding a record that *has* it
    would pass against the old raw-return behaviour too -- the counters are
    read with .get(), so circuit_state is the only field whose absence used to
    raise.
    """
    tracker = _tracker_reading({"total_failures": 2, "total_successes": 1})
    tracker.record_failure("p", Exception("boom"))
    tracker.record_success("p")
 def test_unhashable_or_boolean_values_are_repaired():
    """Values that break the repair itself rather than a later caller.
    An unhashable circuit_state raises TypeError inside a set membership test,
    and bool is a subclass of int, so True would pass as a timestamp and then
    compare as 1.0 -- expiring a cooldown the moment it opens.
    """
    for bad_state in ({"circuit_state": []}, {"circuit_state": {}}):
        state = _tracker_reading(bad_state).get_health_state("p")
        assert state["circuit_state"] == CircuitState.CLOSED.value
    state = _tracker_reading({
        "circuit_opened_time": True, "last_success_time": False,
    }).get_health_state("p")
    assert state["circuit_opened_time"] is None
    assert state["last_success_time"] is None
@@ -63,13 +63,13 @@
 <!-- Getting Started checklist: non-gating, dismissible (localStorage), items
     auto-check from existing config/endpoints — no new persisted state.
-     The timezone step is verified against the browser's own zone rather than
+     Known heuristic limits (acceptable, disclosed): values left at legitimate
-     compared to the shipped default; see the data-check="timezone" block below
+     defaults (e.g. a user actually in Tampa) read as "not done". -->
     for why. -->
 {% set _hw = main_config.display.hardware if main_config and main_config.display else {} %}
 {% set _hw_done = (_hw.rows or 0) > 0 and (_hw.cols or 0) > 0 and (_hw.chain_length or 0) > 0 %}
 {% set _loc = main_config.location if main_config and main_config.location else {} %}
-{% set _tz = (main_config.timezone if main_config else '') or '' %}
+{% set _loc_done = (main_config.timezone and main_config.timezone != 'America/New_York')
                   or (_loc.city and _loc.city != 'Tampa') %}
 <div id="getting-started-card" class="bg-blue-50 border border-blue-200 rounded-lg p-4 mb-4" style="display:none" role="region" aria-label="Getting started checklist">
    <div class="flex items-start justify-between">
        <div class="flex-1">
@@ -78,8 +78,8 @@
            <ul class="space-y-1 text-sm" id="getting-started-items">
                <li><button type="button" class="gs-item text-left w-full" data-done="{{ '1' if _hw_done else '0' }}" data-tab="display">
                    <i class="far fa-square mr-2"></i>Set your panel size (Display tab)</button></li>
-                <li><button type="button" class="gs-item text-left w-full" data-done="0" data-check="timezone" data-tz="{{ _tz }}" data-tab="general">
+                <li><button type="button" class="gs-item text-left w-full" data-done="{{ '1' if _loc_done else '0' }}" data-tab="general">
-                    <i class="far fa-square mr-2"></i>Set your timezone{% if _tz %} — currently {{ _tz }}{% if _loc.city %}, {{ _loc.city }}{% endif %}{% endif %} (General tab)<span data-gs-tz-note class="text-xs"></span></button></li>
+                    <i class="far fa-square mr-2"></i>Set your timezone and location (General tab)</button></li>
                <li><button type="button" class="gs-item text-left w-full" data-done="0" data-check="installed" data-tab="plugins">
                    <i class="far fa-square mr-2"></i>Install a plugin from the Plugin Store</button></li>
                <li><button type="button" class="gs-item text-left w-full" data-done="0" data-check="enabled" data-tab="plugins">
@@ -165,91 +165,6 @@
    });
    maybeAutoHide();
    // Timezone: verified against the browser's own zone.
    //
    // This step used to tick when the saved timezone differed from the value
    // config.template.json ships (America/New_York), with the saved city
    // OR-ed in. Two things were wrong with that. "Differs from the default"
    // answers "did somebody edit this?", but what the checklist needs to know
    // is whether the value is RIGHT — so anyone who genuinely lives in the
    // default zone could never satisfy it and the card nagged forever. And
    // the city has no bearing on whether the timezone is set: because the two
    // were OR-ed, saving a city ticked the step off with the timezone still
    // wrong, which is the direction that actually breaks displays (event
    // times render in the wrong zone).
    //
    // The browser already knows its zone, so compare against that: no new
    // persisted state, no network, and it catches the reverse case too — a
    // panel still set to the old zone after a move now stays unticked, where
    // the old test ticked it the moment the value stopped being the default.
    function sameZone(a, b) {
        if (a === b) return true;
        // Compare the wall-clock time each zone yields, not the identifiers:
        // aliases (Asia/Calcutta vs Asia/Kolkata, Europe/Kiev vs Europe/Kyiv)
        // name one zone and must not read as a mismatch.
        //
        // Sampled at three instants, all of which have to agree. Checking only
        // now is not enough: America/New_York and America/Lima hold the same
        // offset all winter, so a panel set to the wrong one of those would
        // tick in January and then run an hour off from March. Mid-January and
        // mid-July sit either side of DST in both hemispheres, so only zones
        // that agree year-round match -- while Toronto still matches New York,
        // which is right, since either renders the same times.
        try {
            var now = new Date();
            var year = now.getUTCFullYear();
            var instants = [now,
                            new Date(Date.UTC(year, 0, 15, 12)),
                            new Date(Date.UTC(year, 6, 15, 12))];
            var stamp = function (tz, at) {
                // Explicit numeric fields rather than dateStyle/timeStyle:
                // those are late additions to Intl (Firefox shipped them in
                // 91), and an implementation that does not know them ignores
                // them and formats the date alone. That would compare
                // New York, Chicago and Madrid as equal and tick the step for
                // a timezone that is plainly wrong -- the exact failure this
                // check exists to catch. These options have been in Intl
                // since ECMA-402 v1.
                return new Intl.DateTimeFormat('en-US', {
                    timeZone: tz, year: 'numeric', month: '2-digit',
                    day: '2-digit', hour: '2-digit', minute: '2-digit',
                    hour12: false
                }).format(at);
            };
            for (var i = 0; i < instants.length; i++) {
                if (stamp(a, instants[i]) !== stamp(b, instants[i])) {
                    return false;
                }
            }
            return true;
        } catch (e) {
            // An unparseable zone in the config is worth surfacing, not hiding.
            return false;
        }
    }
    (function () {
        var tzBtn = card.querySelector('[data-check="timezone"]');
        if (!tzBtn) return;
        var configured = tzBtn.dataset.tz || '';
        if (!configured) return;          // nothing saved yet: leave it open
        var local = '';
        try {
            local = (Intl.DateTimeFormat().resolvedOptions().timeZone) || '';
        } catch (e) {
            return;                        // no Intl: leave it to the manual tick
        }
        if (!local) return;
        if (sameZone(configured, local)) {
            markDone(tzBtn);
            return;
        }
        // Unticked on its own says "wrong" without saying why; name the zone
        // the browser is in so the step is actionable.
        var note = tzBtn.querySelector('[data-gs-tz-note]');
        if (note) note.textContent = ' — this browser is in ' + local;
    }());
    // Plugin-derived states from the existing installed-plugins endpoint.
    fetch('/api/v3/plugins/installed')
        .then(function (r) { return r.json(); })