mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-08-19 17:39:06 +00:00
0c5b9c57d3
* fix(service): survive corrupt health cache and clean exits Three independent failure modes that each end with a dark panel and no automatic recovery. 1. PluginHealthTracker._load_health_state returned the cached value verbatim. If that value is not a dict, every caller raises AttributeError: 'list' object has no attribute 'get' — during DisplayController.__init__, so the process dies before the display loop starts. systemd restarts it, the same bad entry is read back from disk, and it dies again: an unattended restart loop that survives reboots because the cause is persisted. Observed in the field with plugin_health:<id> holding an unrelated plugin's list payload. Now non-dict entries are discarded with a warning and the defaults are rebuilt. 2. ledmatrix.service used Restart=on-failure, so any exit with status 0 left the unit stopped and the panel dark indefinitely — systemd treats it as success and never brings it back. Restart=always. 3. ledmatrix-wifi-monitor.service used StandardOutput=syslog, which systemd has marked obsolete; it warns and rewrites it to journal on every load. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(memory): size the cache to the board and stop reinstalling deps On a 1GB Pi 3B+ the display process settles around 600MB RSS of 905MB total. When the remaining headroom runs out the failure is not a clean crash: fork() starts returning ENOMEM, so sshd accepts connections and closes them before its banner, timer jobs stop running, and the panel goes dark, while already-resident processes keep serving normally. The board looks healthy from outside and cannot be logged into. Only a power cycle clears it. Three contributing causes: - MemoryCache had a fixed 1000-entry ceiling. Entries are parsed API payloads of tens of KB, so one ceiling cannot serve both a 512MB Zero 2 W and an 8GB Pi 5. Now scaled from MemTotal (150 entries at <=1GB, 1500 at >=8GB), overridable with LEDMATRIX_CACHE_MAX_ENTRIES. - requirements_are_satisfied() returned False for any requirement with extras, so a plugin depending on python-socketio[client] re-ran pip on every single start: ~8s, a network dependency, and a 100-200MB spike at the least convenient moment. During a restart loop it repeats for each restart. Extras are now resolved one level deep against installed metadata, keeping the conservative "anything unverifiable falls through to pip" contract. - ledmatrix.service had no memory ceiling. MemoryMax=85% expressed as a percentage so one unit file suits every board. Note this needs the memory cgroup controller, which Pi firmware disables by default; first_time_install.sh now adds cgroup_enable=memory to cmdline.txt, and the unit file documents how to verify it took effect. first_time_install.sh also enables persistent journald storage (capped at 64M). Default storage is volatile, so every reboot destroys the logs that would explain why the board rebooted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: guidance for 512MB and 1GB boards Documents the memory ceiling on small boards and, more usefully, what running into it actually looks like: sshd accepting connections and closing them before the banner, the web UI still responding normally, clean ping, a dark panel, and a wrong clock after the next boot. None of those read as "out of memory", which makes the failure hard to identify from the symptoms. Cross-referenced from SSH_UNAVAILABLE_AFTER_INSTALL.md, since "I can't SSH in any more" is how most people will first meet this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: address review findings on the low-memory work Nine CodeRabbit findings, five in code. **Health state (the one that matters).** The non-dict guard did not cover a dict missing fields the callers index directly, which is the shape actually seen in the wild: a record carrying only circuit_state produced `plugin clock-simple operation failed: 'circuit_state'` about fifty times a minute with the panel frozen. The record is now completed against the defaults per field rather than trusted or discarded wholesale. Per field matters: a first pass rejected any incomplete record outright, which reset a tripped breaker and real failure counts to healthy because one optional field was absent -- an existing test caught it. Values of the wrong type (a counter persisted as a string, an unknown circuit_state) fall back individually, valid neighbours survive, and newer fields the schema has grown since (degraded, degraded_reason) are carried through untouched. **Cache ceiling.** MemoryCache.set() accepted entries without bound between cleanup sweeps, which run every 300s by default, so a burst could take the cache far past max_size -- the unbounded growth the limit exists to stop. Eviction now runs under the same lock on every write, sharing one helper with the periodic sweep so the two cannot drift. **Installer, cgroups.** Only cgroup_enable=memory was checked, so a board carrying that without cgroup_memory=1 reported success and got no change, leaving MemoryMax= inert. Each parameter is now checked and appended independently; verified against all four combinations, single line preserved. **Installer, journald.** Persistence was inferred from /var/log/journal being non-empty, which proves neither Storage=persistent nor a size cap -- the directory survives a switch back to volatile. The effective configuration is read instead (systemd-analyze cat-config, falling back to the conf files), and an explicitly configured SystemMaxUse is preserved rather than overwritten. Verified across volatile, persistent-without-cap, persistent-with-user-cap, cap-without-storage, and commented-only configs. **Dependency extras.** _extras_are_satisfied stopped at one level, so a gated dependency that itself requests an extra (requests[socks]) passed on the base distribution's version while the extra's own dependency was missing, and pip was skipped. It now recurses, with a visited (distribution, extras) set so a cycle terminates. Docs: both kernel command-line paths documented (the installer falls back to /boot/cmdline.txt), daemon-reload and restart added after the systemd override example, memory exhaustion added to the SSH summary with its power-cycle-only recovery, and a language on the fenced block for MD040. Tests: five for the health-state repair including the exact wild shape and that record_failure/record_success no longer raise against it, and one for the cache ceiling. Both mutation-checked. Full suite 2927 passed, with the one pre-existing tmpfs failure that also fails on main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW * fix: harden the health-state repair and confirm journald took effect Second review round; all three findings were valid and two were bugs in the repair added last commit. The repair could raise out of itself. An unhashable circuit_state (a list or dict on disk) hit `value in {...}` and raised TypeError -- from the code whose whole job is to stop a malformed record crashing the caller. It now requires a str before the membership test. bool is a subclass of int, so True passed the timestamp check and then compared as 1.0: enough to expire a cooldown the instant the breaker opened, while False would stop the elapsed check firing at all. Timestamps now exclude bool explicitly. The regression test for the original crash was seeded with a record that *contained* circuit_state, so it passed against the old raw-return behaviour too -- the counters are read with .get(), so circuit_state is the only field whose absence used to raise. Reseeded to omit it, and it now fails against raw-return as intended. journald: drop-ins apply in lexical order, so a local file sorting after ledmatrix-persistent.conf still wins and writing ours proves nothing. The effective Storage is re-read afterwards and a warning naming the diagnostic command is printed if persistence is still not active, rather than reporting a success that was not verified. Full suite 2934 passed, same single pre-existing tmpfs failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
116 lines
4.2 KiB
Markdown
116 lines
4.2 KiB
Markdown
# Running on Low-Memory Boards
|
|
|
|
Applies to the Pi Zero 2 W (512 MB), Pi 3 / 3B+ (1 GB), and the 1 GB Pi 4.
|
|
If your board has 2 GB or more you can skip this document.
|
|
|
|
## The failure this prevents
|
|
|
|
The display process is the largest thing on the board. On a 1 GB Pi 3B+ with
|
|
around 20 plugins enabled it settles near **600 MB of 905 MB usable**, leaving
|
|
under 200 MB of headroom for everything else.
|
|
|
|
When that headroom runs out, the board does not crash cleanly. `fork()` starts
|
|
failing, and because a new process is needed to do almost anything, the
|
|
symptoms look nothing like "out of memory":
|
|
|
|
| What you see | Why |
|
|
|---|---|
|
|
| SSH accepts the connection then closes it instantly, before any banner | `sshd` forks a session per connection; the fork fails |
|
|
| The web UI still responds quickly | Already running, serves from existing threads, forks nothing |
|
|
| Ping is perfect, 0% loss | Handled entirely in the kernel |
|
|
| The panel is dark | The display process was killed and cannot be respawned |
|
|
| The clock is wrong after the next boot | `fake-hwclock`'s periodic save is a scheduled job, and it cannot fork either |
|
|
|
|
The board looks healthy from the outside and cannot be logged into. Only a
|
|
power cycle clears it. If you are here because SSH stopped working, also see
|
|
[SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md), which
|
|
covers the more common cause (AP mode).
|
|
|
|
## Check your headroom
|
|
|
|
```bash
|
|
free -m
|
|
ps -eo rss,comm --sort=-rss | head -5
|
|
```
|
|
|
|
If `MemAvailable` is under ~150 MB while the display is running, you are close
|
|
to the edge. To watch it over time:
|
|
|
|
```bash
|
|
watch -n 30 'free -m | head -2'
|
|
```
|
|
|
|
Available memory that falls steadily rather than holding flat means you will
|
|
reach the wall; it is a question of when.
|
|
|
|
## What to do
|
|
|
|
**1. Enable the memory cgroup controller.** Without it, the `MemoryMax=85%` in
|
|
`systemd/ledmatrix.service` is accepted by systemd and silently ignored, so the
|
|
service has no ceiling and a runaway takes the whole board down instead of just
|
|
restarting. Raspberry Pi firmware disables this controller by default.
|
|
|
|
`first_time_install.sh` does this for you. To check it took effect:
|
|
|
|
```bash
|
|
grep memory /sys/fs/cgroup/cgroup.controllers
|
|
```
|
|
|
|
If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the
|
|
kernel command line and reboot. Edit whichever file your image uses —
|
|
`/boot/firmware/cmdline.txt` on current Raspberry Pi OS, `/boot/cmdline.txt` on
|
|
older layouts (the installer checks the first and falls back to the second).
|
|
Everything must stay on a single line.
|
|
|
|
This changes the failure mode from "the board becomes unreachable" to "the
|
|
display service restarts". It is a safety net, not a fix.
|
|
|
|
**2. Run fewer plugins.** This is the actual remedy. Every enabled plugin costs
|
|
memory permanently — its module, its parsed config, and its cached API
|
|
responses. On a 512 MB or 1 GB board, keep the enabled set small and prefer
|
|
plugins that poll infrequently.
|
|
|
|
**3. Lower the cache ceiling.** The in-memory cache is sized from total RAM
|
|
(150 entries at 1 GB and below, up to 1500 at 8 GB). To go lower still:
|
|
|
|
```ini
|
|
# /etc/systemd/system/ledmatrix.service.d/override.conf
|
|
[Service]
|
|
Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75
|
|
```
|
|
|
|
Writing the file does not change the running service. Reload systemd and
|
|
restart it:
|
|
|
|
```bash
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart ledmatrix
|
|
```
|
|
|
|
Fewer entries means more API calls, so lower this only while you are actually
|
|
short of memory.
|
|
|
|
**4. Consider `MemoryHigh`.** `MemoryMax` kills and restarts. `MemoryHigh`
|
|
throttles and reclaims instead, which is gentler — but on a board where the
|
|
process genuinely wants more than the limit, sustained reclaim can stall the
|
|
render loop and show as visible stutter on the panel. Add it only if you prefer
|
|
degraded output to a restart:
|
|
|
|
```ini
|
|
[Service]
|
|
MemoryHigh=70%
|
|
```
|
|
|
|
## Keep your logs
|
|
|
|
These images default to volatile journald storage, so every reboot destroys the
|
|
logs — including the ones explaining why the board rebooted. `first_time_install.sh`
|
|
enables persistent storage capped at 64 MB. To confirm:
|
|
|
|
```bash
|
|
journalctl --list-boots
|
|
```
|
|
|
|
More than one boot listed means logs are surviving reboots. If only one is
|
|
listed, journald is still writing to `/run` (tmpfs).
|