fix(plugins): cap the per-plugin state transition history (#501)

* fix(plugins): cap the per-plugin state transition history

PluginStateManager recorded every state transition in a per-plugin list
and never trimmed it. The only code that removed entries was
clear_state(), called solely from PluginManager.unload_plugin(), so a
plugin that stays loaded -- normal operation -- never released one.

The list is written on the hot scheduling path. Every update cycle
appends twice: _reserve_for_update() sets RUNNING and _finish() sets
ENABLED back again. At the default 60s update interval that is 2,880
entries per plugin per day, and nothing reads them -- get_state_info()
only takes their len(). Pure dead weight.

Measured against the unpatched class, ten plugins on a 60s interval:

    sim uptime   history entries   heap growth
          1 day           28,810        7.7 MB
          7 days         201,610       53.9 MB
         30 days         864,010      230.9 MB   (still climbing)

With the cap it is flat at 2,000 entries / 0.5 MB from day one.

On a 1 GB board 231 MB of garbage is fatal on its own, and the failure
is not a clean OOM: once MemAvailable falls far enough fork() starts
returning ENOMEM, so sshd accepts connections and closes them before its
banner while the kernel still answers pings. The board looks like a
hardware fault and needs a power cycle. Same family as the ceilings
added in #464.

Retain the most recent 200 transitions per plugin in a deque and let the
rest age out. state_history_count is surfaced through the web API, so
the lifetime total is tracked separately rather than plateauing at the
cap. get_state_history() now returns a copy under the lock; it was
handing out the manager's own list, which a caller could mutate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(plugins): copy history entries out, lock clear_state

Review follow-ups on the transition history.

get_state_history() copied only the outer list, so a caller holding a
returned transition could rewrite the manager's record of what happened
-- which contradicted the defensive-copy guarantee in its own docstring.
Copy each entry too. Every value in a transition is immutable, so a
shallow copy per entry is enough. test_get_state_history_entries_are_copies
pins it; without the change it fails with 'tampered' == 'enabled'.

clear_state() mutated five shared dicts without holding _lock, while
every other mutator takes it. A concurrent set_state() could interleave
and leave a plugin with history but no state. Drop the five as one unit.

This does not close the wider unload-vs-worker race, which lives in
PluginManager.unload_plugin() and predates this change: an update worker
still in flight can call set_state() after clear_state() returns and
recreate the entry. Serialising that needs the per-plugin lock held
across worker join in unload_plugin(), which is a separate change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

src/background_data_service.py

@@ -57,7 +57,15 @@ class FetchRequest:
 
 @dataclass
 class FetchResult:
-    """Result of a background fetch operation."""
+    """Result of a background fetch operation.
+
+    ``data`` survives on the stored result only for requests submitted without
+    a ``callback``, where polling ``get_result()`` is the sole way to collect
+    it. When a callback was given, the payload has already been delivered and
+    the service releases it -- see :meth:`BackgroundDataService._release_payload`.
+    Either way the data remains in the cache under the request's ``cache_key``,
+    which is where consumers read it from.
+    """
     request_id: str
     success: bool
     data: Optional[Any] = None
@@ -191,14 +199,19 @@ class BackgroundDataService:
                     cached=True,
                     fetch_time=0.0
                 )
+                # Filed before the callback runs, as it always was: a callback
+                # that queries get_result()/is_request_complete() for its own
+                # request must still find it. Releasing afterwards mutates the
+                # same object the dict holds.
                 self.completed_requests[request_id] = result
-                
+
                 if callback:
                     try:
                         callback(result)
                     except Exception as e:
                         logger.error(f"Error in callback for request {request_id}: {e}")
-                
+                    self._release_payload(result)
+
                 logger.debug(f"Cache hit for {sport} {year} data")
                 return request_id
         
@@ -333,8 +346,29 @@ class BackgroundDataService:
                     request.callback(result)
                 except Exception as e:
                     logger.error(f"Error in callback for request {request.id}: {e}")
-        
+                # Delivered. Drop both references -- they point at the same
+                # object, so one survivor keeps the whole payload resident.
+                self._release_payload(result)
+                request.result = None
+
         return result
+
+    @staticmethod
+    def _release_payload(result: FetchResult) -> None:
+        """Drop a delivered payload, keeping the result's status and timings.
+
+        Only called once a callback has been handed the data. Consumers read
+        fetched data back from the cache under ``cache_key``; the copy carried
+        here was pinning a parsed season schedule -- 946 games for NCAA
+        football, roughly a tenth of total RAM on a 1GB Pi -- in memory until
+        the hourly sweep.
+
+        The cache-hit path matters most: it runs once per update interval per
+        sport, mints a fresh request_id each time, and a memory-tier miss
+        re-parses the payload from disk. Those were genuinely separate copies
+        accumulating toward the 500-entry cap, not shared references.
+        """
+        result.data = None
     
     def _make_request_with_retry(self, request: FetchRequest) -> requests.Response:
         """

src/plugin_system/plugin_state.py

@@ -6,14 +6,24 @@ with state transitions and queries.
 """
 
 import threading
+from collections import deque
 from enum import Enum
-from typing import Optional, Dict, Any
+from typing import Optional, Dict, Any, Deque, List
 from datetime import datetime
 import logging
 
 from src.logging_config import get_logger
 
 
+# Transitions retained per plugin. The history is diagnostic only -- nothing
+# reads the entries themselves, just their count -- but it is appended to on the
+# hot scheduling path: every update cycle records RUNNING on reserve and ENABLED
+# on finish. Unbounded, that is 2,880 entries per plugin per day at the default
+# 60s interval, which on a 1 GB Pi exhausts memory in weeks. Keep the recent
+# tail for debugging and let the rest age out.
+MAX_STATE_HISTORY_PER_PLUGIN = 200
+
+
 class PluginState(Enum):
     """Plugin state enumeration."""
     UNLOADED = "unloaded"  # Plugin not loaded
@@ -37,11 +47,34 @@ class PluginStateManager:
         self.logger = logger or get_logger(__name__)
         self._lock = threading.RLock()
         self._states: Dict[str, PluginState] = {}
-        self._state_history: Dict[str, list] = {}
+        self._state_history: Dict[str, Deque[Dict[str, Any]]] = {}
+        # Lifetime transition totals, kept separately so the count reported by
+        # get_state_info() stays truthful once the history above starts rolling.
+        self._state_transition_counts: Dict[str, int] = {}
         self._error_info: Dict[str, Dict[str, Any]] = {}
         self._last_update: Dict[str, datetime] = {}
         self._last_display: Dict[str, datetime] = {}
     
+    def _record_transition(
+        self,
+        plugin_id: str,
+        transition: Dict[str, Any]
+    ) -> None:
+        """Append a transition to the plugin's bounded history.
+
+        Callers must already hold ``_lock``. The deque discards its oldest
+        entry once it is full, so the history cannot grow without bound; the
+        lifetime total is tracked separately for get_state_info().
+        """
+        history = self._state_history.get(plugin_id)
+        if history is None:
+            history = deque(maxlen=MAX_STATE_HISTORY_PER_PLUGIN)
+            self._state_history[plugin_id] = history
+        history.append(transition)
+        self._state_transition_counts[plugin_id] = (
+            self._state_transition_counts.get(plugin_id, 0) + 1
+        )
+
     def set_state(
         self,
         plugin_id: str,
@@ -60,16 +93,13 @@ class PluginStateManager:
             old_state = self._states.get(plugin_id, PluginState.UNLOADED)
             self._states[plugin_id] = state
 
-            if plugin_id not in self._state_history:
-                self._state_history[plugin_id] = []
-
             transition = {
                 'timestamp': datetime.now(),
                 'from': old_state.value,
                 'to': state.value,
                 'error': str(error) if error else None
             }
-            self._state_history[plugin_id].append(transition)
+            self._record_transition(plugin_id, transition)
 
             # Store error info if transitioning to ERROR state
             if state == PluginState.ERROR and error:
@@ -126,17 +156,27 @@ class PluginStateManager:
         state = self.get_state(plugin_id)
         return state == PluginState.ENABLED
     
-    def get_state_history(self, plugin_id: str) -> list:
+    def get_state_history(self, plugin_id: str) -> List[Dict[str, Any]]:
         """
         Get state transition history for a plugin.
-        
+
+        Only the most recent MAX_STATE_HISTORY_PER_PLUGIN transitions are
+        retained; older ones age out.
+
         Args:
             plugin_id: Plugin identifier
-            
+
         Returns:
-            List of state transitions
+            List of recent state transitions, oldest first. Both the list and
+            the transition dicts are copies, so callers cannot mutate the
+            manager's own history. The values inside a transition are all
+            immutable, so a shallow copy per entry is enough.
         """
-        return self._state_history.get(plugin_id, [])
+        with self._lock:
+            return [
+                dict(transition)
+                for transition in self._state_history.get(plugin_id, ())
+            ]
     
     def set_error_info(self, plugin_id: str, error_info: Dict[str, Any]) -> None:
         """
@@ -179,9 +219,7 @@ class PluginStateManager:
             old_state = self._states.get(plugin_id, PluginState.UNLOADED)
             self._states[plugin_id] = state
 
-            if plugin_id not in self._state_history:
-                self._state_history[plugin_id] = []
-            self._state_history[plugin_id].append({
+            self._record_transition(plugin_id, {
                 'timestamp': datetime.now(),
                 'from': old_state.value,
                 'to': state.value,
@@ -252,15 +290,22 @@ class PluginStateManager:
             'last_update': self.get_last_update(plugin_id),
             'last_display': self.get_last_display(plugin_id),
             'error_info': self.get_error_info(plugin_id),
-            'state_history_count': len(self.get_state_history(plugin_id))
+            'state_history_count': self._state_transition_counts.get(plugin_id, 0)
         }
         return info
     
     def clear_state(self, plugin_id: str) -> None:
-        """Clear all state information for a plugin."""
-        self._states.pop(plugin_id, None)
-        self._state_history.pop(plugin_id, None)
-        self._error_info.pop(plugin_id, None)
-        self._last_update.pop(plugin_id, None)
-        self._last_display.pop(plugin_id, None)
+        """Clear all state information for a plugin.
+
+        Held under ``_lock`` so the five dicts are dropped as one unit: every
+        other mutator takes the lock, and without it a concurrent set_state()
+        could interleave and leave a plugin with history but no state.
+        """
+        with self._lock:
+            self._states.pop(plugin_id, None)
+            self._state_history.pop(plugin_id, None)
+            self._state_transition_counts.pop(plugin_id, None)
+            self._error_info.pop(plugin_id, None)
+            self._last_update.pop(plugin_id, None)
+            self._last_display.pop(plugin_id, None)
 

test/test_background_payload_release.py

@@ -0,0 +1,187 @@
+"""A delivered fetch payload must not stay resident on the stored result.
+
+BackgroundDataService kept the fetched body on the FetchResult it filed in
+`completed_requests`, which is swept only hourly and capped at 500 entries by
+count. For status records that is free; for a season schedule it is not. NCAA
+football's 2026 schedule is 946 games, and on a 1GB Pi 3B+ the parsed payload
+measured ~90MB -- a tenth of the board's memory, pinned for an hour after the
+consumer had already been handed it.
+
+The cache-hit path was the worse of the two. It runs once per update interval
+per sport, mints a fresh request_id each time, and hands back whatever the
+cache returns -- so a memory-tier miss (the tier is capped at 150 entries)
+re-parses the payload from disk into a genuinely new object. Those accumulate
+as separate copies rather than shared references, which is the staircase seen
+in the field: RSS stepping up ~90MB per sport as seasons loaded and never
+coming back down.
+
+Releasing is safe because the payload is written to the cache under the
+request's cache_key before the result is built, and that is where consumers
+read it from -- the callback is handed the object directly and the plugins use
+it only in passing before reading the cache back.
+
+Requests submitted *without* a callback keep their payload: polling
+get_result() is then the only way to collect it, so releasing would break that
+contract.
+"""
+
+import time
+import pytest
+from unittest.mock import MagicMock, Mock, patch
+
+from src.background_data_service import BackgroundDataService
+
+
+PAYLOAD = {"events": [{"id": f"g{i}"} for i in range(50)]}
+
+
+@pytest.fixture
+def cache():
+    m = MagicMock()
+    m.get.return_value = None
+    m.set.return_value = None
+    return m
+
+
+@pytest.fixture
+def service(cache):
+    svc = BackgroundDataService(cache, max_workers=2, request_timeout=5)
+    yield svc
+    svc.shutdown(wait=False)
+
+
+def _wait(service, req_id, timeout=5):
+    """Wait for the result to be FILED.
+
+    Enough for anything that is true by the time the worker stores the result:
+    its success flag, its error, the cache write that happened during the
+    fetch.
+    """
+    deadline = time.time() + timeout
+    while not service.is_request_complete(req_id) and time.time() < deadline:
+        time.sleep(0.02)
+
+
+def _wait_for_release(service, req_id, timeout=5):
+    """Wait for the payload to be RELEASED, which is strictly later.
+
+    The worker files the result, then runs the callback, then releases. So
+    is_request_complete() goes true while the callback still has not run --
+    waiting on it alone leaves a window in which `seen` is empty and the
+    payload is still resident, and the assertions race the worker. It passes
+    in practice only because a one-line callback usually beats the 20ms poll.
+
+    Release happens after the callback returns, so a released payload also
+    means the callback has finished: one wait covers both.
+    """
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        result = service.get_result(req_id)
+        if result is not None and result.data is None:
+            return
+        time.sleep(0.02)
+    raise AssertionError(
+        f"payload for {req_id} was never released (callback may not have run)")
+
+
+def _resp():
+    r = Mock()
+    r.json.return_value = PAYLOAD
+    r.raise_for_status.return_value = None
+    return r
+
+
+class TestFetchPath:
+    def test_callback_receives_the_payload_then_it_is_released(self, service, cache):
+        seen = {}
+
+        def callback(result):
+            # The consumer's one look at the data happens here.
+            seen['events'] = len(result.data['events'])
+
+        with patch.object(service.session, "get", return_value=_resp()):
+            req_id = service.submit_fetch_request(
+                sport="ncaa_fb", year=2026, url="https://example.com/s",
+                cache_key="ncaa_fb_2026", callback=callback, max_retries=0,
+            )
+            _wait_for_release(service, req_id)
+
+        assert seen['events'] == 50, "callback must still be handed the payload"
+
+        stored = service.get_result(req_id)
+        assert stored is not None
+        assert stored.success is True
+        assert stored.data is None, "payload must not stay on the stored result"
+
+    def test_nothing_is_lost_the_cache_holds_it(self, service, cache):
+        with patch.object(service.session, "get", return_value=_resp()):
+            req_id = service.submit_fetch_request(
+                sport="ncaa_fb", year=2026, url="https://example.com/s",
+                cache_key="ncaa_fb_2026", callback=lambda r: None, max_retries=0,
+            )
+            _wait(service, req_id)
+
+        cache.set.assert_called_once()
+        key, written = cache.set.call_args[0][:2]
+        assert key == "ncaa_fb_2026"
+        assert written == PAYLOAD, "the payload must be persisted before release"
+
+    def test_without_a_callback_the_payload_is_kept(self, service, cache):
+        # Polling get_result() is then the only delivery mechanism.
+        with patch.object(service.session, "get", return_value=_resp()):
+            req_id = service.submit_fetch_request(
+                sport="nfl", year=2026, url="https://example.com/s",
+                cache_key="nfl_2026", max_retries=0,
+            )
+            _wait(service, req_id)
+
+        assert service.get_result(req_id).data == PAYLOAD
+
+    def test_a_failed_fetch_still_records_its_error(self, service, cache):
+        with patch.object(service.session, "get", side_effect=Exception("boom")):
+            req_id = service.submit_fetch_request(
+                sport="nfl", year=2026, url="https://example.com/s",
+                cache_key="nfl_2026", callback=lambda r: None, max_retries=0,
+            )
+            _wait(service, req_id)
+
+        stored = service.get_result(req_id)
+        assert stored.success is False
+        assert stored.error is not None
+
+
+class TestCacheHitPath:
+    def test_cache_hit_releases_after_the_callback(self, service, cache):
+        cache.get.return_value = PAYLOAD
+        seen = {}
+
+        req_id = service.submit_fetch_request(
+            sport="ncaa_fb", year=2026, url="https://example.com/s",
+            cache_key="ncaa_fb_2026",
+            callback=lambda r: seen.update(events=len(r.data['events'])),
+        )
+
+        assert seen['events'] == 50
+        assert service.get_result(req_id).data is None
+
+    def test_repeated_cache_hits_do_not_accumulate_payloads(self, service, cache):
+        # The staircase: one entry per update interval per sport, each one
+        # potentially a freshly parsed copy after a memory-tier miss.
+        cache.get.return_value = PAYLOAD
+
+        for _ in range(25):
+            service.submit_fetch_request(
+                sport="ncaa_fb", year=2026, url="https://example.com/s",
+                cache_key="ncaa_fb_2026", callback=lambda r: None,
+            )
+
+        retained = [r for r in service.completed_requests.values() if r.data is not None]
+        assert retained == [], f"{len(retained)} payloads still resident"
+
+    def test_cache_hit_without_a_callback_is_unchanged(self, service, cache):
+        cache.get.return_value = PAYLOAD
+        req_id = service.submit_fetch_request(
+            sport="nfl", year=2026, url="https://example.com/s",
+            cache_key="nfl_2026",
+        )
+        assert service.get_result(req_id).data == PAYLOAD

test/test_plugin_state_history_cap.py

@@ -0,0 +1,166 @@
+"""Plugin state history must not grow without bound.
+
+`PluginStateManager` recorded every state transition in a per-plugin list and
+never trimmed it. The only code that removed entries was `clear_state()`, called
+solely from `PluginManager.unload_plugin()`, so a plugin that stays loaded --
+i.e. normal operation -- never released a single entry.
+
+The list is written on the hot scheduling path. Every update cycle appends
+twice: `_reserve_for_update()` sets RUNNING and `_finish()` sets ENABLED back
+again. At the default 60-second update interval that is 2,880 entries per
+plugin per day, and nothing ever reads the entries -- `get_state_info()` only
+takes their `len()`. It is pure dead weight.
+
+Measured against the unpatched class, ten plugins on a 60s interval retain
+864,010 transitions after thirty simulated days, for 231 MB of heap. On a 1 GB
+Pi that is fatal on its own, and the failure is not a clean OOM: once
+MemAvailable falls far enough, fork() starts returning ENOMEM, so sshd accepts
+connections and closes them before its banner while the kernel still answers
+pings. The board looks like a hardware fault and needs a power cycle.
+
+These tests pin the cap, the retention order, and the one piece of behaviour the
+cap must not change: `state_history_count` is surfaced through the web API, so
+it has to keep reporting the lifetime total rather than plateauing at the cap.
+"""
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
+from src.plugin_system.plugin_state import (  # noqa: E402
+    MAX_STATE_HISTORY_PER_PLUGIN,
+    PluginState,
+    PluginStateManager,
+)
+
+
+def _cycle_updates(manager, plugin_id, cycles):
+    """Drive the real scheduling path: RUNNING on reserve, ENABLED on finish."""
+    for _ in range(cycles):
+        manager.set_state(plugin_id, PluginState.RUNNING)
+        manager.set_state(plugin_id, PluginState.ENABLED)
+
+
+def test_state_history_is_capped():
+    """A day of updates must not retain a day of transitions."""
+    manager = PluginStateManager()
+    manager.set_state("clock", PluginState.ENABLED)
+
+    # One simulated day at the default 60s update interval.
+    _cycle_updates(manager, "clock", 1440)
+
+    history = manager.get_state_history("clock")
+    assert len(history) <= MAX_STATE_HISTORY_PER_PLUGIN, (
+        f"history grew to {len(history)} entries; it is never trimmed"
+    )
+
+
+def test_state_history_keeps_the_most_recent_transitions():
+    """Trimming drops the oldest entries, not the newest."""
+    manager = PluginStateManager()
+    manager.set_state("clock", PluginState.ENABLED)
+    _cycle_updates(manager, "clock", MAX_STATE_HISTORY_PER_PLUGIN)
+
+    history = manager.get_state_history("clock")
+
+    # The scheduling cycle ends on ENABLED, so the newest entry is the
+    # RUNNING -> ENABLED half of the last cycle.
+    assert history[-1]["from"] == PluginState.RUNNING.value
+    assert history[-1]["to"] == PluginState.ENABLED.value
+
+    # And the very first ENABLED transition has aged out.
+    assert history[0]["from"] != PluginState.UNLOADED.value
+
+
+def test_state_history_count_reports_lifetime_total():
+    """The count exposed through the API must not plateau at the cap.
+
+    `get_state_info()['state_history_count']` is surfaced by the web UI. Capping
+    the retained list must not turn it into "entries we happen to still hold".
+    """
+    manager = PluginStateManager()
+    manager.set_state("clock", PluginState.ENABLED)
+    total = 1
+
+    cycles = MAX_STATE_HISTORY_PER_PLUGIN * 2
+    _cycle_updates(manager, "clock", cycles)
+    total += cycles * 2
+
+    info = manager.get_state_info("clock")
+    assert info["state_history_count"] == total
+    assert len(manager.get_state_history("clock")) <= MAX_STATE_HISTORY_PER_PLUGIN
+
+
+def test_error_transitions_are_capped_too():
+    """set_state_with_error() appends to the same list and needs the same cap."""
+    manager = PluginStateManager()
+    manager.set_state("clock", PluginState.ENABLED)
+
+    for _ in range(MAX_STATE_HISTORY_PER_PLUGIN * 2):
+        manager.set_state_with_error(
+            "clock",
+            PluginState.ENABLED,
+            {"reason": "update timeout"},
+            error=RuntimeError("boom"),
+        )
+
+    assert len(manager.get_state_history("clock")) <= MAX_STATE_HISTORY_PER_PLUGIN
+
+
+def test_history_is_isolated_per_plugin():
+    """The cap is per plugin, not shared across the manager."""
+    manager = PluginStateManager()
+    for plugin_id in ("clock", "weather"):
+        manager.set_state(plugin_id, PluginState.ENABLED)
+        _cycle_updates(manager, plugin_id, 50)
+
+    assert len(manager.get_state_history("clock")) == 101
+    assert len(manager.get_state_history("weather")) == 101
+
+
+def test_get_state_history_returns_a_copy():
+    """Callers must not be able to mutate the manager's internal history."""
+    manager = PluginStateManager()
+    manager.set_state("clock", PluginState.ENABLED)
+
+    history = manager.get_state_history("clock")
+    history.clear()
+
+    assert len(manager.get_state_history("clock")) == 1
+
+
+def test_get_state_history_entries_are_copies():
+    """Copying the outer list is not enough -- the entries are handed out too.
+
+    A caller holding a returned transition must not be able to rewrite the
+    manager's record of what happened.
+    """
+    manager = PluginStateManager()
+    manager.set_state("clock", PluginState.ENABLED)
+
+    entry = manager.get_state_history("clock")[0]
+    entry["to"] = "tampered"
+    entry["error"] = "injected"
+
+    stored = manager.get_state_history("clock")[0]
+    assert stored["to"] == PluginState.ENABLED.value
+    assert stored["error"] is None
+
+
+def test_clear_state_drops_history():
+    """Unloading a plugin still releases everything it accumulated."""
+    manager = PluginStateManager()
+    manager.set_state("clock", PluginState.ENABLED)
+    _cycle_updates(manager, "clock", 10)
+
+    manager.clear_state("clock")
+
+    assert manager.get_state_history("clock") == []
+    assert manager.get_state_info("clock")["state_history_count"] == 0
+
+
+if __name__ == "__main__":
+    sys.exit(pytest.main([__file__, "-v"]))

test/test_system_status_available_memory.py

@@ -0,0 +1,92 @@
+"""/api/v3/system/status must report MemAvailable, not just used/total.
+
+"Memory used %" cannot tell a healthy board from one about to fail. Page cache
+counts as used and is reclaimable on demand, so a Pi can read 70% used and be
+perfectly fine, or read the same and be minutes from trouble. MemAvailable is
+the kernel's own estimate of what a new allocation can actually obtain, and it
+is the number that tracked the failure on a 1GB Pi 3B+: healthy running sat at
+500MB+, the crash happened at 73MB, and by then fork() was failing -- sshd
+could not spawn a session and systemd could not respawn the display, while the
+kernel carried on answering pings.
+
+psutil.virtual_memory().available is MemAvailable on Linux. total - used is not
+a substitute: they diverge exactly when unreclaimable memory (shmem, tmpfs) is
+in play, which is when the distinction matters.
+"""
+
+import json
+import sys
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+from flask import Flask
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+MB = 1024 * 1024
+
+
+@pytest.fixture
+def client():
+    pytest.importorskip("psutil")
+    app = Flask(__name__)
+    app.config["TESTING"] = True
+    from web_interface.blueprints.api_v3 import api_v3
+    for attr in ("config_manager", "plugin_manager", "cache_manager"):
+        setattr(api_v3, attr, MagicMock())
+    if "api_v3" not in app.blueprints:
+        app.register_blueprint(api_v3, url_prefix="/api/v3")
+    return app.test_client()
+
+
+def _memory(total_mb, used_mb, available_mb):
+    m = MagicMock()
+    m.total = total_mb * MB
+    m.used = used_mb * MB
+    m.available = available_mb * MB
+    m.percent = round(used_mb / total_mb * 100, 1)
+    return m
+
+
+def _get_status(client, memory):
+    # The endpoint caches for 10s; bypass so each case is measured fresh.
+    with patch("web_interface.cache.get_cached", return_value=None), \
+         patch("psutil.virtual_memory", return_value=memory), \
+         patch("psutil.cpu_percent", return_value=5.0), \
+         patch("psutil.boot_time", return_value=0.0):
+        resp = client.get("/api/v3/system/status")
+    assert resp.status_code == 200, resp.data
+    return json.loads(resp.data)["data"]
+
+
+def test_available_memory_is_reported(client):
+    data = _get_status(client, _memory(total_mb=905, used_mb=620, available_mb=284))
+    assert "memory_available_mb" in data
+    assert data["memory_available_mb"] == pytest.approx(284, abs=0.5)
+
+
+def test_available_is_not_total_minus_used(client):
+    # The case the readout exists for: 600MB is "not used", but only 300MB can
+    # actually be allocated. Reporting used% alone would call this healthy.
+    data = _get_status(client, _memory(total_mb=1000, used_mb=400, available_mb=300))
+
+    derived = data["memory_total_mb"] - data["memory_used_mb"]
+    assert derived == pytest.approx(600, abs=1)
+    assert data["memory_available_mb"] == pytest.approx(300, abs=0.5)
+    assert data["memory_available_mb"] != pytest.approx(derived, abs=1), \
+        "available must come from MemAvailable, not be derived from used"
+
+
+def test_existing_memory_fields_are_unchanged(client):
+    data = _get_status(client, _memory(total_mb=905, used_mb=620, available_mb=284))
+    assert data["memory_total_mb"] == pytest.approx(905, abs=0.5)
+    assert data["memory_used_mb"] == pytest.approx(620, abs=0.5)
+    assert "memory_used_percent" in data
+
+
+def test_a_nearly_exhausted_board_reports_a_small_number(client):
+    # 73MB available is what the board actually read when it stopped being able
+    # to fork. The readout has to surface that rather than round it away.
+    data = _get_status(client, _memory(total_mb=905, used_mb=800, available_mb=73))
+    assert data["memory_available_mb"] == pytest.approx(73, abs=0.5)

web_interface/blueprints/api_v3.py

@@ -1563,6 +1563,11 @@ def get_system_status():
             'memory_used_percent': round(memory_percent, 1),
             'memory_total_mb': round(memory.total / (1024 * 1024), 1),
             'memory_used_mb': round(memory.used / (1024 * 1024), 1),
+            # MemAvailable, not total-minus-used: it accounts for reclaimable
+            # page cache, so it is what actually predicts memory trouble. A
+            # board can read 70% "used" and be fine, or read the same and be
+            # about to fail fork(), and only this number tells them apart.
+            'memory_available_mb': round(memory.available / (1024 * 1024), 1),
             'cpu_temp': round(cpu_temp, 1) if cpu_temp is not None else None,
             'disk_used_percent': round(disk_percent, 1),
             'disk_total_gb': round(disk.total / (1024 * 1024 * 1024), 1),

web_interface/templates/v3/partials/tools.html

@@ -687,12 +687,35 @@
                 const mUsedGb = d.memory_used_mb != null ? (d.memory_used_mb / 1024).toFixed(1) : null;
                 const mTotGb  = d.memory_total_mb != null ? (d.memory_total_mb / 1024).toFixed(1) : null;
                 const temp    = d.cpu_temp != null ? d.cpu_temp + '°C' : 'N/A';
+                // Available memory is the number that predicts trouble. When it
+                // runs out the board does not fail cleanly: fork() starts
+                // returning ENOMEM, so sshd cannot spawn a session and systemd
+                // cannot respawn the display, while the kernel keeps answering
+                // pings. Thresholds are drawn from that failure -- it was
+                // measured at 73MB free, and healthy running sits well above.
+                // Round ONCE, then colour and label off the same number.
+                // The API sends one decimal place, so classifying the raw
+                // value and displaying the rounded one disagreed at the
+                // boundaries: 149.6 rendered as "150 MB" in red, and 299.6 as
+                // "300 MB" in amber, both contradicting the threshold the
+                // colour claims to apply. Which side of the line a spare
+                // 0.4MB falls on does not matter; the tile agreeing with
+                // itself does.
+                const availMb = d.memory_available_mb == null
+                              ? null : Math.round(d.memory_available_mb);
+                const availColor = availMb == null ? 'text-gray-400'
+                                 : availMb < 150   ? 'text-red-600'
+                                 : availMb < 300   ? 'text-amber-500'
+                                 : 'text-green-600';
                 panel.innerHTML =
                     diagTile('fa-microchip', 'text-blue-600', 'CPU Usage',
                         (d.cpu_percent != null ? d.cpu_percent : '--') + '%', null) +
                     diagTile('fa-memory', 'text-green-600', 'Memory',
                         (d.memory_used_percent != null ? d.memory_used_percent : '--') + '%',
                         (mUsedGb && mTotGb) ? `${mUsedGb} / ${mTotGb} GB` : null) +
+                    diagTile('fa-memory', availColor, 'Available Memory',
+                        availMb != null ? `${availMb} MB` : '--',
+                        mTotGb ? `of ${mTotGb} GB total` : null) +
                     diagTile('fa-thermometer-half', 'text-red-600', 'CPU Temp', temp, null) +
                     diagTile('fa-hdd', 'text-indigo-600', 'Disk',
                         (d.disk_used_percent != null ? d.disk_used_percent : '--') + '%',