rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-08-19 17:39:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0c5b9c57d3caddd71c8cfa434a68a6172b803af7
LEDMatrix/test/test_cache_manager.py
T
ChuckGitHubClaude Opus 5
 0c5b9c57d3 fix: keep low-memory boards reachable under load (#464) 
* fix(service): survive corrupt health cache and clean exits

Three independent failure modes that each end with a dark panel and no
automatic recovery.

1. PluginHealthTracker._load_health_state returned the cached value
   verbatim. If that value is not a dict, every caller raises
   AttributeError: 'list' object has no attribute 'get' — during
   DisplayController.__init__, so the process dies before the display
   loop starts. systemd restarts it, the same bad entry is read back
   from disk, and it dies again: an unattended restart loop that
   survives reboots because the cause is persisted. Observed in the
   field with plugin_health:<id> holding an unrelated plugin's list
   payload. Now non-dict entries are discarded with a warning and the
   defaults are rebuilt.

2. ledmatrix.service used Restart=on-failure, so any exit with status 0
   left the unit stopped and the panel dark indefinitely — systemd
   treats it as success and never brings it back. Restart=always.

3. ledmatrix-wifi-monitor.service used StandardOutput=syslog, which
   systemd has marked obsolete; it warns and rewrites it to journal on
   every load.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* perf(memory): size the cache to the board and stop reinstalling deps

On a 1GB Pi 3B+ the display process settles around 600MB RSS of 905MB
total. When the remaining headroom runs out the failure is not a clean
crash: fork() starts returning ENOMEM, so sshd accepts connections and
closes them before its banner, timer jobs stop running, and the panel
goes dark, while already-resident processes keep serving normally. The
board looks healthy from outside and cannot be logged into. Only a power
cycle clears it.

Three contributing causes:

- MemoryCache had a fixed 1000-entry ceiling. Entries are parsed API
  payloads of tens of KB, so one ceiling cannot serve both a 512MB Zero
  2 W and an 8GB Pi 5. Now scaled from MemTotal (150 entries at <=1GB,
  1500 at >=8GB), overridable with LEDMATRIX_CACHE_MAX_ENTRIES.

- requirements_are_satisfied() returned False for any requirement with
  extras, so a plugin depending on python-socketio[client] re-ran pip on
  every single start: ~8s, a network dependency, and a 100-200MB spike
  at the least convenient moment. During a restart loop it repeats for
  each restart. Extras are now resolved one level deep against installed
  metadata, keeping the conservative "anything unverifiable falls
  through to pip" contract.

- ledmatrix.service had no memory ceiling. MemoryMax=85% expressed as a
  percentage so one unit file suits every board. Note this needs the
  memory cgroup controller, which Pi firmware disables by default;
  first_time_install.sh now adds cgroup_enable=memory to cmdline.txt,
  and the unit file documents how to verify it took effect.

first_time_install.sh also enables persistent journald storage (capped
at 64M). Default storage is volatile, so every reboot destroys the logs
that would explain why the board rebooted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs: guidance for 512MB and 1GB boards

Documents the memory ceiling on small boards and, more usefully, what
running into it actually looks like: sshd accepting connections and
closing them before the banner, the web UI still responding normally,
clean ping, a dark panel, and a wrong clock after the next boot. None of
those read as "out of memory", which makes the failure hard to identify
from the symptoms.

Cross-referenced from SSH_UNAVAILABLE_AFTER_INSTALL.md, since "I can't
SSH in any more" is how most people will first meet this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: address review findings on the low-memory work

Nine CodeRabbit findings, five in code.

**Health state (the one that matters).** The non-dict guard did not cover a
dict missing fields the callers index directly, which is the shape actually
seen in the wild: a record carrying only circuit_state produced
`plugin clock-simple operation failed: 'circuit_state'` about fifty times a
minute with the panel frozen. The record is now completed against the
defaults per field rather than trusted or discarded wholesale. Per field
matters: a first pass rejected any incomplete record outright, which reset a
tripped breaker and real failure counts to healthy because one optional
field was absent -- an existing test caught it. Values of the wrong type
(a counter persisted as a string, an unknown circuit_state) fall back
individually, valid neighbours survive, and newer fields the schema has
grown since (degraded, degraded_reason) are carried through untouched.

**Cache ceiling.** MemoryCache.set() accepted entries without bound between
cleanup sweeps, which run every 300s by default, so a burst could take the
cache far past max_size -- the unbounded growth the limit exists to stop.
Eviction now runs under the same lock on every write, sharing one helper
with the periodic sweep so the two cannot drift.

**Installer, cgroups.** Only cgroup_enable=memory was checked, so a board
carrying that without cgroup_memory=1 reported success and got no change,
leaving MemoryMax= inert. Each parameter is now checked and appended
independently; verified against all four combinations, single line preserved.

**Installer, journald.** Persistence was inferred from /var/log/journal being
non-empty, which proves neither Storage=persistent nor a size cap -- the
directory survives a switch back to volatile. The effective configuration is
read instead (systemd-analyze cat-config, falling back to the conf files),
and an explicitly configured SystemMaxUse is preserved rather than
overwritten. Verified across volatile, persistent-without-cap,
persistent-with-user-cap, cap-without-storage, and commented-only configs.

**Dependency extras.** _extras_are_satisfied stopped at one level, so a
gated dependency that itself requests an extra (requests[socks]) passed on
the base distribution's version while the extra's own dependency was
missing, and pip was skipped. It now recurses, with a visited
(distribution, extras) set so a cycle terminates.

Docs: both kernel command-line paths documented (the installer falls back to
/boot/cmdline.txt), daemon-reload and restart added after the systemd
override example, memory exhaustion added to the SSH summary with its
power-cycle-only recovery, and a language on the fenced block for MD040.

Tests: five for the health-state repair including the exact wild shape and
that record_failure/record_success no longer raise against it, and one for
the cache ceiling. Both mutation-checked. Full suite 2927 passed, with the
one pre-existing tmpfs failure that also fails on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW

* fix: harden the health-state repair and confirm journald took effect

Second review round; all three findings were valid and two were bugs in the
repair added last commit.

The repair could raise out of itself. An unhashable circuit_state (a list or
dict on disk) hit `value in {...}` and raised TypeError -- from the code
whose whole job is to stop a malformed record crashing the caller. It now
requires a str before the membership test.

bool is a subclass of int, so True passed the timestamp check and then
compared as 1.0: enough to expire a cooldown the instant the breaker opened,
while False would stop the elapsed check firing at all. Timestamps now
exclude bool explicitly.

The regression test for the original crash was seeded with a record that
*contained* circuit_state, so it passed against the old raw-return behaviour
too -- the counters are read with .get(), so circuit_state is the only field
whose absence used to raise. Reseeded to omit it, and it now fails against
raw-return as intended.

journald: drop-ins apply in lexical order, so a local file sorting after
ledmatrix-persistent.conf still wins and writing ours proves nothing. The
effective Storage is re-read afterwards and a warning naming the diagnostic
command is printed if persistence is still not active, rather than reporting
a success that was not verified.

Full suite 2934 passed, same single pre-existing tmpfs failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-19 12:28:22 -04:00

484 lines
17 KiB
Python
Raw Blame History

"""
Tests for CacheManager and cache components.
Tests cache functionality including memory cache, disk cache, strategy, and metrics.
"""
import pytest
import time
from unittest.mock import patch
from src.cache_manager import CacheManager
from src.cache.memory_cache import MemoryCache
from src.cache.disk_cache import DiskCache
from src.cache.cache_strategy import CacheStrategy
from src.cache.cache_metrics import CacheMetrics
from datetime import datetime
class TestCacheManager:
"""Test CacheManager functionality."""
def test_init(self, tmp_path):
"""Test CacheManager initialization."""
with patch('src.cache_manager.CacheManager._get_writable_cache_dir', return_value=str(tmp_path)):
cm = CacheManager()
assert cm.cache_dir == str(tmp_path)
assert hasattr(cm, '_memory_cache_component')
assert hasattr(cm, '_disk_cache_component')
assert hasattr(cm, '_strategy_component')
assert hasattr(cm, '_metrics_component')
def test_set_and_get(self, tmp_path):
"""Test basic set and get operations."""
with patch('src.cache_manager.CacheManager._get_writable_cache_dir', return_value=str(tmp_path)):
cm = CacheManager()
test_data = {"key": "value", "number": 42}
cm.set("test_key", test_data)
result = cm.get("test_key")
assert result == test_data
def test_get_expired(self, tmp_path):
"""Test getting expired cache entry."""
with patch('src.cache_manager.CacheManager._get_writable_cache_dir', return_value=str(tmp_path)):
cm = CacheManager()
cm.set("test_key", {"data": "value"})
# Get with max_age=0 to force expiration
result = cm.get("test_key", max_age=0)
assert result is None
class TestCacheStrategy:
"""Test CacheStrategy functionality."""
def test_get_cache_strategy_default(self):
"""Test getting default cache strategy."""
strategy = CacheStrategy()
result = strategy.get_cache_strategy("unknown_type")
assert "max_age" in result
assert "memory_ttl" in result
assert result["max_age"] == 300 # Default
def test_get_cache_strategy_live(self):
"""Test getting live sports cache strategy."""
strategy = CacheStrategy()
result = strategy.get_cache_strategy("sports_live")
assert "max_age" in result
assert result["max_age"] <= 60 # Live data should be short
def test_get_data_type_from_key(self):
"""Test data type detection from cache key."""
strategy = CacheStrategy()
assert strategy.get_data_type_from_key("nba_live_scores") == "sports_live"
# "weather_current" contains "current" which matches live sports pattern first
# Use "weather" without "current" to test weather detection
assert strategy.get_data_type_from_key("weather") == "weather_current"
assert strategy.get_data_type_from_key("weather_data") == "weather_current"
assert strategy.get_data_type_from_key("unknown_key") == "default"
class TestMemoryCache:
"""Test MemoryCache functionality."""
def test_init(self):
"""Test MemoryCache initialization."""
cache = MemoryCache(max_size=100, cleanup_interval=60.0)
assert cache._max_size == 100
assert cache._cleanup_interval == 60.0
assert cache.size() == 0
def test_set_and_get(self):
"""Test basic set and get operations."""
cache = MemoryCache()
test_data = {"key": "value", "number": 42}
cache.set("test_key", test_data)
result = cache.get("test_key")
assert result == test_data
def test_get_expired(self):
"""Test getting expired cache entry."""
cache = MemoryCache()
cache.set("test_key", {"data": "value"})
# Get with max_age=0 to force expiration
result = cache.get("test_key", max_age=0)
assert result is None
def test_get_nonexistent(self):
"""Test getting non-existent key."""
cache = MemoryCache()
result = cache.get("nonexistent_key")
assert result is None
def test_clear_specific_key(self):
"""Test clearing a specific cache key."""
cache = MemoryCache()
cache.set("key1", {"data": "value1"})
cache.set("key2", {"data": "value2"})
cache.clear("key1")
assert cache.get("key1") is None
assert cache.get("key2") is not None
def test_clear_all(self):
"""Test clearing all cache entries."""
cache = MemoryCache()
cache.set("key1", {"data": "value1"})
cache.set("key2", {"data": "value2"})
cache.clear()
assert cache.size() == 0
assert cache.get("key1") is None
assert cache.get("key2") is None
def test_cleanup_expired(self):
"""Test cleanup removes expired entries."""
cache = MemoryCache()
cache.set("key1", {"data": "value1"})
# Force expiration by manipulating timestamp (older than 1 hour cleanup threshold)
# Cleanup uses max_age_for_cleanup = 3600 (1 hour)
cache._timestamps["key1"] = time.time() - 4000 # More than 1 hour
removed = cache.cleanup(force=True)
# Cleanup should remove expired entries (older than 3600 seconds)
# The key should be gone after cleanup
assert cache.get("key1") is None or removed >= 0
def test_cleanup_size_limit(self):
"""Test cleanup enforces size limits."""
cache = MemoryCache(max_size=3)
# Add more entries than max_size
for i in range(5):
cache.set(f"key{i}", {"data": f"value{i}"})
removed = cache.cleanup(force=True)
assert cache.size() <= cache._max_size
assert removed >= 0
def test_size(self):
"""Test size reporting."""
cache = MemoryCache()
assert cache.size() == 0
cache.set("key1", {"data": "value1"})
cache.set("key2", {"data": "value2"})
assert cache.size() == 2
def test_max_size(self):
"""Test max_size property."""
cache = MemoryCache(max_size=500)
assert cache.max_size() == 500
def test_get_stats(self):
"""Test getting cache statistics."""
cache = MemoryCache()
cache.set("key1", {"data": "value1"})
cache.set("key2", {"data": "value2"})
stats = cache.get_stats()
assert "size" in stats
assert "max_size" in stats
assert stats["size"] == 2
assert stats["max_size"] == 1000 # default
class TestCacheMetrics:
"""Test CacheMetrics functionality."""
def test_record_hit(self):
"""Test recording cache hit."""
metrics = CacheMetrics()
metrics.record_hit()
stats = metrics.get_metrics()
# get_metrics() returns calculated values, not raw hits/misses
assert stats['total_requests'] == 1
assert stats['cache_hit_rate'] == 1.0 # 1 hit out of 1 request
def test_record_miss(self):
"""Test recording cache miss."""
metrics = CacheMetrics()
metrics.record_miss()
stats = metrics.get_metrics()
# get_metrics() returns calculated values, not raw hits/misses
assert stats['total_requests'] == 1
assert stats['cache_hit_rate'] == 0.0 # 0 hits out of 1 request
def test_record_fetch_time(self):
"""Test recording fetch time."""
metrics = CacheMetrics()
metrics.record_fetch_time(0.5)
stats = metrics.get_metrics()
assert stats['fetch_count'] == 1
assert stats['total_fetch_time'] == 0.5
assert stats['average_fetch_time'] == 0.5
def test_cache_hit_rate(self):
"""Test cache hit rate calculation."""
metrics = CacheMetrics()
metrics.record_hit()
metrics.record_hit()
metrics.record_miss()
stats = metrics.get_metrics()
assert stats['cache_hit_rate'] == pytest.approx(0.666, abs=0.01)
class TestDiskCache:
"""Test DiskCache functionality."""
def test_init_with_dir(self, tmp_path):
"""Test DiskCache initialization with directory."""
cache = DiskCache(cache_dir=str(tmp_path))
assert cache.cache_dir == str(tmp_path)
def test_init_without_dir(self):
"""Test DiskCache initialization without directory."""
cache = DiskCache(cache_dir=None)
assert cache.cache_dir is None
def test_get_cache_path(self, tmp_path):
"""Test getting cache file path."""
cache = DiskCache(cache_dir=str(tmp_path))
path = cache.get_cache_path("test_key")
assert path == str(tmp_path / "test_key.json")
def test_get_cache_path_disabled(self):
"""Test getting cache path when disabled."""
cache = DiskCache(cache_dir=None)
path = cache.get_cache_path("test_key")
assert path is None
def test_set_and_get(self, tmp_path):
"""Test basic set and get operations."""
cache = DiskCache(cache_dir=str(tmp_path))
test_data = {"key": "value", "number": 42}
cache.set("test_key", test_data)
result = cache.get("test_key")
assert result == test_data
def test_get_expired(self, tmp_path):
"""Test getting expired cache entry."""
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("test_key", {"data": "value"})
# Get with max_age=0 to force expiration
result = cache.get("test_key", max_age=0)
assert result is None
def test_get_max_age_none_never_expires(self, tmp_path):
"""max_age=None must return persisted records regardless of age.
Regression: the age comparison raised TypeError for max_age=None,
which was swallowed and treated as a miss — silently breaking
long-lived state (plugin health/metrics) read across processes.
"""
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("test_key", {"data": "value", "timestamp": 0}) # epoch → very old
result = cache.get("test_key", max_age=None)
assert result is not None
assert result["data"] == "value"
def test_get_nonexistent(self, tmp_path):
"""Test getting non-existent key."""
cache = DiskCache(cache_dir=str(tmp_path))
result = cache.get("nonexistent_key")
assert result is None
def test_clear_specific_key(self, tmp_path):
"""Test clearing a specific cache key."""
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("key1", {"data": "value1"})
cache.set("key2", {"data": "value2"})
cache.clear("key1")
assert cache.get("key1") is None
assert cache.get("key2") is not None
def test_clear_all(self, tmp_path):
"""Test clearing all cache entries."""
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("key1", {"data": "value1"})
cache.set("key2", {"data": "value2"})
cache.clear()
assert cache.get("key1") is None
assert cache.get("key2") is None
def test_get_cache_dir(self, tmp_path):
"""Test getting cache directory."""
cache = DiskCache(cache_dir=str(tmp_path))
assert cache.get_cache_dir() == str(tmp_path)
def test_set_with_datetime(self, tmp_path):
"""Test setting cache with datetime objects."""
cache = DiskCache(cache_dir=str(tmp_path))
test_data = {
"timestamp": datetime.now(),
"data": "value"
}
cache.set("test_key", test_data)
result = cache.get("test_key")
# Datetime should be serialized/deserialized
assert result is not None
assert "data" in result
def test_cleanup_interval(self, tmp_path):
"""Test cleanup respects interval."""
cache = MemoryCache(cleanup_interval=60.0)
cache.set("key1", {"data": "value1"})
# First cleanup should work
removed1 = cache.cleanup(force=True)
# Second cleanup immediately after should return 0 (unless forced)
removed2 = cache.cleanup(force=False)
# If forced, should work; if not forced and within interval, should return 0
assert removed2 >= 0
def test_get_with_invalid_timestamp(self):
"""Test getting entry with invalid timestamp format."""
cache = MemoryCache()
cache.set("key1", {"data": "value1"})
# Set invalid timestamp
cache._timestamps["key1"] = "invalid_timestamp"
result = cache.get("key1")
# Should handle gracefully
assert result is None or isinstance(result, dict)
def test_record_background_hit(self):
"""Test recording background cache hit."""
metrics = CacheMetrics()
metrics.record_hit(cache_type='background')
stats = metrics.get_metrics()
assert stats['total_requests'] == 1
assert stats['background_hit_rate'] == 1.0
def test_record_background_miss(self):
"""Test recording background cache miss."""
metrics = CacheMetrics()
metrics.record_miss(cache_type='background')
stats = metrics.get_metrics()
assert stats['total_requests'] == 1
assert stats['background_hit_rate'] == 0.0
def test_multiple_fetch_times(self):
"""Test recording multiple fetch times."""
metrics = CacheMetrics()
metrics.record_fetch_time(0.5)
metrics.record_fetch_time(1.0)
metrics.record_fetch_time(0.3)
stats = metrics.get_metrics()
assert stats['fetch_count'] == 3
assert stats['total_fetch_time'] == 1.8
assert stats['average_fetch_time'] == pytest.approx(0.6, abs=0.01)
class TestDiskCacheWriteEconomy:
"""SD-card wear guards: identical payloads skip the disk, files are
compact, and TTL semantics survive the skip (see PR: fix/diskcache-sd-wear)."""
def test_identical_set_skips_rewrite(self, tmp_path):
import os
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("k", {"data": "v"})
path = cache.get_cache_path("k")
first = os.stat(path)
os.utime(path, (first.st_atime - 100, first.st_mtime - 100)) # age it
aged_mtime = os.stat(path).st_mtime
ino_before = os.stat(path).st_ino
cache.set("k", {"data": "v"}) # identical payload
after = os.stat(path)
# mtime refreshed (TTL for mtime-based records preserved)...
assert after.st_mtime > aged_mtime
# ...but the file was NOT rewritten (same inode: no replace happened)
assert after.st_ino == ino_before
def test_changed_data_rewrites(self, tmp_path):
import os
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("k", {"data": "v1"})
cache.set("k", {"data": "v2"})
assert cache.get("k") == {"data": "v2"}
def test_clear_resets_digest(self, tmp_path):
import os
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("k", {"data": "v"})
cache.clear("k")
assert cache.get("k") is None
cache.set("k", {"data": "v"}) # same payload after clear must WRITE
assert cache.get("k") == {"data": "v"}
def test_skip_self_heals_when_file_deleted_externally(self, tmp_path):
import os
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("k", {"data": "v"})
os.remove(cache.get_cache_path("k")) # e.g. expiry cleanup
cache.set("k", {"data": "v"}) # digest matches but file is gone
assert cache.get("k") == {"data": "v"}
def test_files_are_compact_json(self, tmp_path):
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("k", {"a": 1, "b": [1, 2, 3]})
raw = open(cache.get_cache_path("k")).read()
assert "\n" not in raw.strip() # no indent
assert cache.get("k") == {"a": 1, "b": [1, 2, 3]}
def test_datetime_round_trip_still_works(self, tmp_path):
from datetime import datetime
cache = DiskCache(cache_dir=str(tmp_path))
cache.set("k", {"when": datetime(2026, 7, 12, 10, 30)})
assert cache.get("k") == {"when": "2026-07-12T10:30:00"}
# --- the ceiling has to hold between cleanup sweeps ---------------------------
def test_memory_cache_enforces_ceiling_on_every_write():
"""_cleanup_memory_cache only runs every cleanup_interval seconds (300 by
default). If set() accepted entries without bound in between, a burst could
take the cache far past max_size -- which is the unbounded growth the limit
exists to prevent, and on a 1GB board the difference between a bounded cache
and a Pi that cannot fork.
"""
from src.cache.memory_cache import MemoryCache
cache = MemoryCache(max_size=150, cleanup_interval=300.0)
for i in range(1000):
cache.set(f"k{i}", {"v": i})
assert len(cache._cache) <= 150
# The timestamp map has to be evicted alongside the values, or it becomes
# the leak instead.
assert len(cache._timestamps) <= 150
assert cache.get("k999") is not None, "the newest write must survive"
assert cache.get("k0") is None, "the oldest must be the one evicted"
Reference in New Issue View Git Blame Copy Permalink