fix: address review findings on the low-memory work

Nine CodeRabbit findings, five in code.

**Health state (the one that matters).** The non-dict guard did not cover a
dict missing fields the callers index directly, which is the shape actually
seen in the wild: a record carrying only circuit_state produced
`plugin clock-simple operation failed: 'circuit_state'` about fifty times a
minute with the panel frozen. The record is now completed against the
defaults per field rather than trusted or discarded wholesale. Per field
matters: a first pass rejected any incomplete record outright, which reset a
tripped breaker and real failure counts to healthy because one optional
field was absent -- an existing test caught it. Values of the wrong type
(a counter persisted as a string, an unknown circuit_state) fall back
individually, valid neighbours survive, and newer fields the schema has
grown since (degraded, degraded_reason) are carried through untouched.

**Cache ceiling.** MemoryCache.set() accepted entries without bound between
cleanup sweeps, which run every 300s by default, so a burst could take the
cache far past max_size -- the unbounded growth the limit exists to stop.
Eviction now runs under the same lock on every write, sharing one helper
with the periodic sweep so the two cannot drift.

**Installer, cgroups.** Only cgroup_enable=memory was checked, so a board
carrying that without cgroup_memory=1 reported success and got no change,
leaving MemoryMax= inert. Each parameter is now checked and appended
independently; verified against all four combinations, single line preserved.

**Installer, journald.** Persistence was inferred from /var/log/journal being
non-empty, which proves neither Storage=persistent nor a size cap -- the
directory survives a switch back to volatile. The effective configuration is
read instead (systemd-analyze cat-config, falling back to the conf files),
and an explicitly configured SystemMaxUse is preserved rather than
overwritten. Verified across volatile, persistent-without-cap,
persistent-with-user-cap, cap-without-storage, and commented-only configs.

**Dependency extras.** _extras_are_satisfied stopped at one level, so a
gated dependency that itself requests an extra (requests[socks]) passed on
the base distribution's version while the extra's own dependency was
missing, and pip was skipped. It now recurses, with a visited
(distribution, extras) set so a cycle terminates.

Docs: both kernel command-line paths documented (the installer falls back to
/boot/cmdline.txt), daemon-reload and restart added after the systemd
override example, memory exhaustion added to the SSH summary with its
power-cycle-only recovery, and a language on the fenced block for MD040.

Tests: five for the health-state repair including the exact wild shape and
that record_failure/record_success no longer raise against it, and one for
the cache ceiling. Both mutation-checked. Full suite 2927 passed, with the
one pre-existing tmpfs failure that also fails on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW
This commit is contained in:
ChuckBuilds
2026-08-19 09:28:02 -04:00
co-authored by Claude Opus 5
parent ef1e9e0eee
commit 34a7414275
8 changed files with 297 additions and 53 deletions
+12 -1
View File
@@ -57,7 +57,10 @@ grep memory /sys/fs/cgroup/cgroup.controllers
```
If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the
single line in `/boot/firmware/cmdline.txt` and reboot.
kernel command line and reboot. Edit whichever file your image uses —
`/boot/firmware/cmdline.txt` on current Raspberry Pi OS, `/boot/cmdline.txt` on
older layouts (the installer checks the first and falls back to the second).
Everything must stay on a single line.
This changes the failure mode from "the board becomes unreachable" to "the
display service restarts". It is a safety net, not a fix.
@@ -76,6 +79,14 @@ plugins that poll infrequently.
Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75
```
Writing the file does not change the running service. Reload systemd and
restart it:
```bash
sudo systemctl daemon-reload
sudo systemctl restart ledmatrix
```
Fewer entries means more API calls, so lower this only while you are actually
short of memory.
+14 -2
View File
@@ -26,7 +26,7 @@ On a 512MB or 1GB board, memory exhaustion stops `sshd` being able to fork a
session process. The connection is accepted and then closed immediately, before
any banner:
```
```text
kex_exchange_identification: Connection closed by remote host
```
@@ -205,11 +205,23 @@ The web interface allows you to:
## Summary
**SSH becomes unavailable because**:
**SSH becomes unavailable because** — two unrelated causes, and they need
different responses:
*AP mode (most common):*
- WiFi monitor service enables AP mode when WiFi disconnects
- AP mode switches WiFi from client to access point mode
- Pi loses connection to your original network
*Memory exhaustion (low-memory boards):*
- The board runs out of memory, so `sshd` cannot fork a session process
- The connection is accepted and closed before any banner
- Ping still answers and the web UI still responds, so it looks healthy
- The panel is usually dark and the service cannot restart
- **Only a power cycle clears this** — there is no remote recovery, because
every remote route needs a new process
- Prevention and tuning: [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md)
**To regain SSH**:
1. Connect to **LEDMatrix-Setup** AP network (password: `ledmatrix123`)
2. SSH to `192.168.4.1`