fix: address review findings on the low-memory work

Nine CodeRabbit findings, five in code.

**Health state (the one that matters).** The non-dict guard did not cover a
dict missing fields the callers index directly, which is the shape actually
seen in the wild: a record carrying only circuit_state produced
`plugin clock-simple operation failed: 'circuit_state'` about fifty times a
minute with the panel frozen. The record is now completed against the
defaults per field rather than trusted or discarded wholesale. Per field
matters: a first pass rejected any incomplete record outright, which reset a
tripped breaker and real failure counts to healthy because one optional
field was absent -- an existing test caught it. Values of the wrong type
(a counter persisted as a string, an unknown circuit_state) fall back
individually, valid neighbours survive, and newer fields the schema has
grown since (degraded, degraded_reason) are carried through untouched.

**Cache ceiling.** MemoryCache.set() accepted entries without bound between
cleanup sweeps, which run every 300s by default, so a burst could take the
cache far past max_size -- the unbounded growth the limit exists to stop.
Eviction now runs under the same lock on every write, sharing one helper
with the periodic sweep so the two cannot drift.

**Installer, cgroups.** Only cgroup_enable=memory was checked, so a board
carrying that without cgroup_memory=1 reported success and got no change,
leaving MemoryMax= inert. Each parameter is now checked and appended
independently; verified against all four combinations, single line preserved.

**Installer, journald.** Persistence was inferred from /var/log/journal being
non-empty, which proves neither Storage=persistent nor a size cap -- the
directory survives a switch back to volatile. The effective configuration is
read instead (systemd-analyze cat-config, falling back to the conf files),
and an explicitly configured SystemMaxUse is preserved rather than
overwritten. Verified across volatile, persistent-without-cap,
persistent-with-user-cap, cap-without-storage, and commented-only configs.

**Dependency extras.** _extras_are_satisfied stopped at one level, so a
gated dependency that itself requests an extra (requests[socks]) passed on
the base distribution's version while the extra's own dependency was
missing, and pip was skipped. It now recurses, with a visited
(distribution, extras) set so a cycle terminates.

Docs: both kernel command-line paths documented (the installer falls back to
/boot/cmdline.txt), daemon-reload and restart added after the systemd
override example, memory exhaustion added to the SSH summary with its
power-cycle-only recovery, and a language on the fenced block for MD040.

Tests: five for the health-state repair including the exact wild shape and
that record_failure/record_success no longer raise against it, and one for
the cache ceiling. Both mutation-checked. Full suite 2927 passed, with the
one pre-existing tmpfs failure that also fails on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW
This commit is contained in:
ChuckBuilds
2026-08-19 09:28:02 -04:00
co-authored by Claude Opus 5
parent ef1e9e0eee
commit 34a7414275
8 changed files with 297 additions and 53 deletions
+48 -12
View File
@@ -1694,12 +1694,24 @@ fi
# a runaway takes the whole board down (sshd can no longer fork, the panel goes
# dark) rather than just restarting the one service.
if [ "$SKIP_PERF" != "1" ] && [ -f "$CMDLINE_FILE" ]; then
if grep -q 'cgroup_enable=memory' "$CMDLINE_FILE"; then
echo "cgroup_enable=memory already present in $CMDLINE_FILE"
# Both parameters are required for the memory controller, and they can get
# separated -- an image, another tool or a half-applied earlier run can
# leave one without the other. Checking only cgroup_enable=memory would
# report success while MemoryMax= silently does nothing, so each is checked
# and appended independently.
cgroup_missing=""
for cgroup_param in cgroup_enable=memory cgroup_memory=1; do
if ! grep -qw "$cgroup_param" "$CMDLINE_FILE"; then
cgroup_missing="$cgroup_missing $cgroup_param"
fi
done
if [ -z "$cgroup_missing" ]; then
echo "cgroup memory parameters already present in $CMDLINE_FILE"
else
echo "Adding cgroup_enable=memory to $CMDLINE_FILE..."
echo "Adding${cgroup_missing} to $CMDLINE_FILE..."
cp "$CMDLINE_FILE" "$CMDLINE_FILE.bak" 2>/dev/null || true
sed -i '1 s/$/ cgroup_enable=memory cgroup_memory=1/' "$CMDLINE_FILE"
# The kernel command line must stay on one line.
sed -i "1 s|\$|${cgroup_missing}|" "$CMDLINE_FILE"
echo " Takes effect after reboot. Verify with:"
echo " grep memory /sys/fs/cgroup/cgroup.controllers"
fi
@@ -1709,17 +1721,41 @@ fi
# These images default to volatile storage: journald keeps everything in /run
# (tmpfs), so every reboot destroys the logs — including the ones that would
# explain why the board rebooted. Capped so an SD card is not worn out by logs.
if [ -d /var/log/journal ] && [ -n "$(ls -A /var/log/journal 2>/dev/null)" ]; then
echo "Persistent journald storage already enabled"
# A non-empty /var/log/journal does not prove journald is configured the way
# this needs: the directory survives a switch back to volatile storage, and it
# says nothing about whether a size cap is set. Read the effective
# configuration instead, and only write the keys the user has not set
# themselves so an explicit local limit is preserved.
journald_effective() {
# systemd-analyze merges journald.conf with every drop-in; grep is the
# fallback for images that ship without it.
if command -v systemd-analyze >/dev/null 2>&1 &&
systemd-analyze cat-config systemd/journald.conf >/dev/null 2>&1; then
systemd-analyze cat-config systemd/journald.conf 2>/dev/null
else
cat /etc/systemd/journald.conf /etc/systemd/journald.conf.d/*.conf 2>/dev/null
fi
}
journald_conf="$(journald_effective)"
journald_storage="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*Storage=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
journald_cap="$(printf '%s\n' "$journald_conf" | grep -E '^[[:space:]]*SystemMaxUse=' | tail -n1 | cut -d= -f2 | tr -d '[:space:]')"
if [ "$journald_storage" = "persistent" ] && [ -n "$journald_cap" ]; then
echo "Persistent journald storage already configured (SystemMaxUse=$journald_cap)"
else
echo "Enabling persistent journald storage..."
mkdir -p /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/ledmatrix-persistent.conf <<'JOURNALD'
# Installed by LEDMatrix first_time_install.sh
[Journal]
Storage=persistent
SystemMaxUse=64M
JOURNALD
{
echo "# Installed by LEDMatrix first_time_install.sh"
echo "[Journal]"
echo "Storage=persistent"
if [ -n "$journald_cap" ]; then
echo "# SystemMaxUse left to your existing setting ($journald_cap)"
else
# Capped so logs cannot wear out or fill an SD card.
echo "SystemMaxUse=64M"
fi
} > /etc/systemd/journald.conf.d/ledmatrix-persistent.conf
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal >/dev/null 2>&1 || true
systemctl restart systemd-journald >/dev/null 2>&1 || true