From ef1e9e0eee5ac4cdba865bc76b6ad9402340c6ab Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Tue, 18 Aug 2026 19:20:49 -0400 Subject: [PATCH] docs: guidance for 512MB and 1GB boards Documents the memory ceiling on small boards and, more usefully, what running into it actually looks like: sshd accepting connections and closing them before the banner, the web UI still responding normally, clean ping, a dark panel, and a wrong clock after the next boot. None of those read as "out of memory", which makes the failure hard to identify from the symptoms. Cross-referenced from SSH_UNAVAILABLE_AFTER_INSTALL.md, since "I can't SSH in any more" is how most people will first meet this. Co-Authored-By: Claude Opus 5 --- docs/LOW_MEMORY_BOARDS.md | 104 ++++++++++++++++++++++++++ docs/README.md | 1 + docs/SSH_UNAVAILABLE_AFTER_INSTALL.md | 17 ++++- 3 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 docs/LOW_MEMORY_BOARDS.md diff --git a/docs/LOW_MEMORY_BOARDS.md b/docs/LOW_MEMORY_BOARDS.md new file mode 100644 index 00000000..98741f2b --- /dev/null +++ b/docs/LOW_MEMORY_BOARDS.md @@ -0,0 +1,104 @@ +# Running on Low-Memory Boards + +Applies to the Pi Zero 2 W (512 MB), Pi 3 / 3B+ (1 GB), and the 1 GB Pi 4. +If your board has 2 GB or more you can skip this document. + +## The failure this prevents + +The display process is the largest thing on the board. On a 1 GB Pi 3B+ with +around 20 plugins enabled it settles near **600 MB of 905 MB usable**, leaving +under 200 MB of headroom for everything else. + +When that headroom runs out, the board does not crash cleanly. `fork()` starts +failing, and because a new process is needed to do almost anything, the +symptoms look nothing like "out of memory": + +| What you see | Why | +|---|---| +| SSH accepts the connection then closes it instantly, before any banner | `sshd` forks a session per connection; the fork fails | +| The web UI still responds quickly | Already running, serves from existing threads, forks nothing | +| Ping is perfect, 0% loss | Handled entirely in the kernel | +| The panel is dark | The display process was killed and cannot be respawned | +| The clock is wrong after the next boot | `fake-hwclock`'s periodic save is a scheduled job, and it cannot fork either | + +The board looks healthy from the outside and cannot be logged into. Only a +power cycle clears it. If you are here because SSH stopped working, also see +[SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md), which +covers the more common cause (AP mode). + +## Check your headroom + +```bash +free -m +ps -eo rss,comm --sort=-rss | head -5 +``` + +If `MemAvailable` is under ~150 MB while the display is running, you are close +to the edge. To watch it over time: + +```bash +watch -n 30 'free -m | head -2' +``` + +Available memory that falls steadily rather than holding flat means you will +reach the wall; it is a question of when. + +## What to do + +**1. Enable the memory cgroup controller.** Without it, the `MemoryMax=85%` in +`systemd/ledmatrix.service` is accepted by systemd and silently ignored, so the +service has no ceiling and a runaway takes the whole board down instead of just +restarting. Raspberry Pi firmware disables this controller by default. + +`first_time_install.sh` does this for you. To check it took effect: + +```bash +grep memory /sys/fs/cgroup/cgroup.controllers +``` + +If that prints nothing, add `cgroup_enable=memory cgroup_memory=1` to the +single line in `/boot/firmware/cmdline.txt` and reboot. + +This changes the failure mode from "the board becomes unreachable" to "the +display service restarts". It is a safety net, not a fix. + +**2. Run fewer plugins.** This is the actual remedy. Every enabled plugin costs +memory permanently — its module, its parsed config, and its cached API +responses. On a 512 MB or 1 GB board, keep the enabled set small and prefer +plugins that poll infrequently. + +**3. Lower the cache ceiling.** The in-memory cache is sized from total RAM +(150 entries at 1 GB and below, up to 1500 at 8 GB). To go lower still: + +```ini +# /etc/systemd/system/ledmatrix.service.d/override.conf +[Service] +Environment=LEDMATRIX_CACHE_MAX_ENTRIES=75 +``` + +Fewer entries means more API calls, so lower this only while you are actually +short of memory. + +**4. Consider `MemoryHigh`.** `MemoryMax` kills and restarts. `MemoryHigh` +throttles and reclaims instead, which is gentler — but on a board where the +process genuinely wants more than the limit, sustained reclaim can stall the +render loop and show as visible stutter on the panel. Add it only if you prefer +degraded output to a restart: + +```ini +[Service] +MemoryHigh=70% +``` + +## Keep your logs + +These images default to volatile journald storage, so every reboot destroys the +logs — including the ones explaining why the board rebooted. `first_time_install.sh` +enables persistent storage capped at 64 MB. To confirm: + +```bash +journalctl --list-boots +``` + +More than one boot listed means logs are surviving reboots. If only one is +listed, journald is still writing to `/run` (tmpfs). diff --git a/docs/README.md b/docs/README.md index acb2d944..63128da6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,6 +14,7 @@ the one-shot installer. The pages here go deeper. 5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes 6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install 7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems +8. [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) — Pi Zero 2 W / 3B+ / 1GB Pi 4 memory limits ## I want to write a plugin diff --git a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md index d9830165..82f98a70 100644 --- a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md +++ b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md @@ -20,7 +20,22 @@ The installation script: - Installs and configures `dnsmasq` (DHCP server for AP mode) - These services can interfere with normal WiFi client mode -### 3. Reboot After Installation +### 3. The Board Ran Out of Memory + +On a 512MB or 1GB board, memory exhaustion stops `sshd` being able to fork a +session process. The connection is accepted and then closed immediately, before +any banner: + +``` +kex_exchange_identification: Connection closed by remote host +``` + +The giveaway is that the board is otherwise healthy — ping is clean and the web +UI still responds — but nothing that needs to start a new process works, and +the panel is usually dark. Only a power cycle clears it. See +[LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md). + +### 4. Reboot After Installation If the script reboots the Pi (which it recommends), network services may restart in a different state, potentially triggering AP mode.