rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-10-04 06:15:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(cache): remove the cache layer's duplicate cleanup and dead lookups (#613)

Browse Source 
* refactor(cache): collapse CacheStrategy's all-60 defaults table and twin soccer branch

get_sport_live_interval() without a config manager looked the sport up in
a table where every value was 60, with 60 as the fallback; it now returns
60. get_data_type_from_key() had an `if 'soccer'` branch returning the
same 'sports_live' as its else.

test_cache_strategy_intervals pins the returned strategy for every data
type x sport key x config-manager shape; it passes unchanged on the old
code. A 2,544-entry dump of every CacheStrategy method over a wider grid
is identical before and after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cache): drop CacheStrategy's `<sport>_scoreboard` config lookup

get_sport_live_interval() and get_cache_strategy() read live/recent/
upcoming intervals from config[f"{sport}_scoreboard"]. Those sections
belonged to the built-in scoreboards the plugin system replaced; plugin
config is keyed by plugin id ("football-scoreboard"), so on a current
config the lookup always fell through to the defaults (60 live, 1800
recent, 10800 upcoming), which are now returned directly.

The one input where this differs: a config.json upgraded from the
pre-plugin era that still carries e.g. an "nfl_scoreboard" section (no
code removes them), queried with an explicit sport key. No caller in core
or the plugin monorepo passes a sport key here -- get_with_auto_strategy
only derives one for keys classed sports_live/live_scores, and its callers
(odds managers, odds-ticker) use odds keys -- so the stale section was
unreachable in practice. A dump of every CacheStrategy method over 2,544
inputs differs from the previous commit only in those 45 legacy-config
entries; the test grid now includes that shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(cache): list cache files without holding the memory-tier lock

CacheManager.list_cache_files() held the in-memory cache's lock while it
listed and stat'd the whole cache directory -- 8,864 files on a real rig
-- so every get()/set() from the display loop and plugins waited out the
scan. The lock never protected the disk: DiskCache writes and deletes
under their own lock, and a file vanishing between listdir and stat was
already handled (logged and skipped). The body is unchanged apart from
the dedent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(cache): delegate memory-tier cleanup and stats to MemoryCache

CacheManager._cleanup_memory_cache() was a line-for-line copy of
MemoryCache.cleanup(), and get_memory_cache_stats() a copy of
MemoryCache.get_stats(), both reaching into the component's private
_cache/_timestamps/_lock through "backward compatibility" aliases bound
in __init__. So the component's own cleanup and stats only ever ran in
tests, and the aliases went stale whenever the component was swapped
(test_cache_ttl_honoured does). Both now delegate, and the aliases are
gone: nothing in core, the tests, or the ledmatrix-plugins monorepo reads
them.

Behaviour is the same. Compared line by line, the two cleanups differ
only in the sort key's fallback (0 vs 0.0, which orders identically),
range+bounds check vs slice for the eviction, and the logger name on the
DEBUG summary line (src.cache_manager -> src.cache.memory_cache). A
differential run over 20,000 random memory states (str/None/garbage/
future timestamps, orphan keys, sizes 0-12, forced and throttled runs)
gives identical removed counts, resulting dicts and last-cleanup times;
the same harness catches each of three seeded mutations of
MemoryCache.cleanup. The throttle clock also moves with it:
CacheManager kept its own copy of last-cleanup, the component's is used
now, and they started equal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(background): inline the sport cache key and drop the unused request queue

get_sport_cache_key() constructed a whole CacheManager -- ConfigManager,
config parse, cache-dir probing with test-file writes -- to return
f"{sport}_{date}". It now builds the key itself in the same format as
CacheManager.generate_sport_cache_key() (UTC date, %Y%m%d); tests check
the two agree for explicit dates and, with a frozen clock at 03:30 UTC,
for the default date. Median per call on Windows: ~0.6 ms -> ~2 us
(alternating runs); on a Pi the old path also wrote a probe file per call.

request_queue was a PriorityQueue nothing ever put into: requests go
straight to the executor, so `priority` never did anything. The queue is
gone; the `priority` parameter and FetchRequest field stay (every
monorepo scoreboard passes priority=) and are documented as ignored, and
get_statistics() keeps reporting queue_size, now a literal 0 as it
always was in practice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-23 12:54:07 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 269385c97c
commit 9d024f24ef
6 changed files with 416 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/background_data_service.py
+16 -9
View File
@@ -16,14 +16,15 @@ Key Features:
import itertools
import time
from datetime import datetime
import logging
import threading
import requests
from typing import Dict, Any, Optional, Callable, List
from dataclasses import dataclass, field
from enum import Enum
import queue
from concurrent.futures import ThreadPoolExecutor
import pytz
from src.cache_manager import CacheManager
from src.common.espn_dates import (
RANGE_RETRY_SECONDS,
@@ -57,7 +58,9 @@ class FetchRequest:
timeout: int = 30
retry_count: int = 0
max_retries: int = 3
priority: int = 1 # Higher number = higher priority
# Recorded but not acted on: requests go straight to the thread pool in
# submission order. Kept because plugins pass it through.
priority: int = 1
callback: Optional[Callable] = None
# Callbacks from submitters that JOINED this fetch instead of starting a
# duplicate one. The primary `callback` above belongs to whoever created
@@ -143,7 +146,6 @@ class BackgroundDataService:
self._request_seq = itertools.count()
self.active_requests: Dict[str, FetchRequest] = {}
self.completed_requests: Dict[str, FetchResult] = {}
self.request_queue = queue.PriorityQueue()
# Thread safety
self._lock = threading.RLock()
@@ -187,10 +189,12 @@ class BackgroundDataService:
This ensures Recent/Upcoming managers and background service
use the same cache keys.
"""
# Use the centralized cache key generation from CacheManager
from src.cache_manager import CacheManager
cache_manager = CacheManager()
return cache_manager.generate_sport_cache_key(sport, date_str)
# Same format as CacheManager.generate_sport_cache_key(). This used to
# build a whole CacheManager to call it -- config load, cache-dir
# probing with test writes -- on every submit without a cache_key.
if date_str is None:
date_str = datetime.now(pytz.utc).strftime('%Y%m%d')
return f"{sport}_{date_str}"
def submit_fetch_request(self,
sport: str,
@@ -215,7 +219,8 @@ class BackgroundDataService:
headers: HTTP headers
timeout: Request timeout
max_retries: Maximum number of retries
priority: Request priority (higher = more important)
priority: Accepted for compatibility and ignored; requests run in
submission order.
callback: Optional callback function when request completes
Returns:
@@ -719,7 +724,9 @@ class BackgroundDataService:
'completed_requests_count': len(self.completed_requests),
'max_completed_requests': self._max_completed_requests,
'completed_requests_usage_percent': (len(self.completed_requests) / self._max_completed_requests * 100) if self._max_completed_requests > 0 else 0,
'queue_size': self.request_queue.qsize(),
# Nothing is queued outside the executor; kept for callers
# that read the key.
'queue_size': 0,
'last_cleanup': self._last_completed_requests_cleanup,
'cleanup_interval': self._completed_requests_cleanup_interval
}
src/cache/cache_strategy.py
Vendored
+23 -55
View File
@@ -1,7 +1,7 @@
"""
Cache Strategy
Manages cache strategies for different data types with sport-specific configurations.
Manages cache strategies (TTLs) for different data types.
"""
import logging
@@ -18,7 +18,8 @@ class CacheStrategy:
Initialize cache strategy manager.
Args:
config_manager: Optional ConfigManager instance for sport-specific configs
config_manager: Optional ConfigManager instance. Kept for callers
that pass one; no strategy currently reads it.
logger: Optional logger instance
"""
self.config_manager = config_manager
@@ -26,69 +27,38 @@ class CacheStrategy:
def get_sport_live_interval(self, sport_key: str) -> int:
"""
Get the live_update_interval for a specific sport from config.
Falls back to default values if config is not available.
Live-data cache interval, in seconds, for a sport: 60 for every sport.
This used to read ``live_update_interval`` from a ``<sport>_scoreboard``
config section. Those sections belonged to the built-in scoreboards
that the plugin system replaced; plugin config is keyed by plugin id
(``football-scoreboard``), so the lookup always fell back to 60.
Args:
sport_key: Sport identifier (e.g., 'nba', 'nfl')
Returns:
Live update interval in seconds
"""
if not self.config_manager:
# Default intervals - all sports use 60 seconds as default
default_intervals = {
'soccer': 60,
'nfl': 60,
'nhl': 60,
'nba': 60,
'mlb': 60,
'milb': 60,
'ncaa_fb': 60,
'ncaa_baseball': 60,
'ncaam_basketball': 60,
}
return default_intervals.get(sport_key, 60)
try:
config = self.config_manager.config
# All sports now use _scoreboard suffix
sport_config = config.get(f"{sport_key}_scoreboard", {})
return sport_config.get("live_update_interval", 60) # Default to 60 seconds
except (KeyError, AttributeError, TypeError) as e:
self.logger.warning("Could not get live_update_interval for %s: %s", sport_key, e, exc_info=True)
return 60 # Default to 60 seconds
return 60
def get_cache_strategy(self, data_type: str, sport_key: Optional[str] = None) -> Dict[str, Any]:
"""
Get cache strategy for different data types.
Now respects sport-specific live_update_interval configurations.
Args:
data_type: Type of data (e.g., 'live_scores', 'stocks', 'weather_current')
sport_key: Optional sport key for sport-specific intervals
sport_key: Optional sport key; for live data it selects the
per-sport interval from :meth:`get_sport_live_interval`
instead of the generic live default.
Returns:
Dictionary with cache strategy (max_age, memory_ttl, etc.)
"""
# Get sport-specific live interval if provided
live_interval = None
if sport_key and data_type in ['sports_live', 'live_scores']:
live_interval = self.get_sport_live_interval(sport_key)
# Try to read sport-specific config for recent/upcoming
recent_interval = None
upcoming_interval = None
if self.config_manager and sport_key:
try:
# All sports now use _scoreboard suffix
sport_cfg = self.config_manager.config.get(f"{sport_key}_scoreboard", {})
recent_interval = sport_cfg.get('recent_update_interval')
upcoming_interval = sport_cfg.get('upcoming_update_interval')
except (KeyError, AttributeError, TypeError) as e:
self.logger.debug("Could not read sport-specific recent/upcoming intervals for %s: %s",
sport_key, e, exc_info=True)
strategies = {
# Ultra time-sensitive data (live scores, current weather)
'live_scores': {
@@ -122,13 +92,13 @@ class CacheStrategy:
# Sports data
'sports_recent': {
'max_age': recent_interval or 1800, # 30 minutes default; override by config
'memory_ttl': (recent_interval or 1800) * 2,
'max_age': 1800, # 30 minutes
'memory_ttl': 3600,
'force_refresh': False
},
'sports_upcoming': {
'max_age': upcoming_interval or 10800, # 3 hours default; override by config
'memory_ttl': (upcoming_interval or 10800) * 2,
'max_age': 10800, # 3 hours
'memory_ttl': 21600,
'force_refresh': False
},
'sports_schedules': {
@@ -206,8 +176,6 @@ class CacheStrategy:
# Live sports data
if any(x in key_lower for x in ['live', 'current', 'scoreboard']):
if 'soccer' in key_lower:
return 'sports_live' # Soccer live data is very time-sensitive
return 'sports_live'
# Weather data
src/cache_manager.py
+55 -124
View File
@@ -92,14 +92,6 @@ class CacheManager:
self._strategy_component = CacheStrategy(config_manager=self.config_manager, logger=self.logger)
self._metrics_component = CacheMetrics(logger=self.logger)
# Keep old attributes for backward compatibility (delegated to components)
self._memory_cache = self._memory_cache_component._cache
self._memory_cache_timestamps = self._memory_cache_component._timestamps
self._cache_lock = self._memory_cache_component._lock
self._max_memory_cache_size = self._memory_cache_component._max_size
self._memory_cache_cleanup_interval = self._memory_cache_component._cleanup_interval
self._last_memory_cache_cleanup = self._memory_cache_component._last_cleanup
# Disk cleanup configuration
self._disk_cleanup_interval_hours = 24 # Run cleanup every 24 hours
self._disk_cleanup_interval = 3600.0 # Minimum interval between cleanups (1 hour) for throttle
@@ -229,70 +221,14 @@ class CacheManager:
return None
def _cleanup_memory_cache(self, force: bool = False) -> int:
"""
Clean up expired entries from memory cache and enforce size limits.
Args:
force: If True, perform cleanup regardless of time interval
"""Sweep the memory tier: drop entries older than an hour and trim it
to its size ceiling, at most once per cleanup interval unless forced.
Returns:
Number of entries removed
"""
now = time.time()
# Check if cleanup is needed
if not force and (now - self._last_memory_cache_cleanup) < self._memory_cache_cleanup_interval:
return 0
with self._cache_lock:
removed_count = 0
current_time = time.time()
# Remove expired entries (entries older than 1 hour without access are considered expired)
# We use a conservative TTL of 1 hour for cleanup
max_age_for_cleanup = 3600 # 1 hour
expired_keys = []
for key, timestamp in list(self._memory_cache_timestamps.items()):
if isinstance(timestamp, str):
try:
timestamp = float(timestamp)
except ValueError:
timestamp = None
if timestamp is None or (current_time - timestamp) > max_age_for_cleanup:
expired_keys.append(key)
# Remove expired entries
for key in expired_keys:
self._memory_cache.pop(key, None)
self._memory_cache_timestamps.pop(key, None)
removed_count += 1
# Enforce size limit by removing oldest entries if cache is too large
if len(self._memory_cache) > self._max_memory_cache_size:
# Sort by timestamp (oldest first)
sorted_entries = sorted(
self._memory_cache_timestamps.items(),
key=lambda x: float(x[1]) if isinstance(x[1], (int, float)) else 0
)
# Remove oldest entries until we're under the limit
excess_count = len(self._memory_cache) - self._max_memory_cache_size
for i in range(excess_count):
if i < len(sorted_entries):
key = sorted_entries[i][0]
self._memory_cache.pop(key, None)
self._memory_cache_timestamps.pop(key, None)
removed_count += 1
self._last_memory_cache_cleanup = current_time
if removed_count > 0:
self.logger.debug(f"Memory cache cleanup: removed {removed_count} entries (current size: {len(self._memory_cache)})")
return removed_count
return self._memory_cache_component.cleanup(force=force)
def _get_cache_path(self, key: str) -> Optional[str]:
"""Get the path for a cache file."""
return self._disk_cache_component.get_cache_path(key)
@@ -412,56 +348,58 @@ class CacheManager:
current_time = time.time()
try:
with self._cache_lock:
for filename in os.listdir(self.cache_dir):
if not filename.endswith('.json'):
continue
# No lock: this is disk-only work, and the memory-tier lock it used
# to hold would stall every get/set while thousands of files are
# stat'd. A file deleted mid-scan is skipped below.
for filename in os.listdir(self.cache_dir):
if not filename.endswith('.json'):
continue
# Extract key from filename (remove .json extension)
key = filename[:-5] # Remove '.json'
file_path = os.path.join(self.cache_dir, filename)
try:
# Get file stats
stat_info = os.stat(file_path)
size_bytes = stat_info.st_size
modified_time = stat_info.st_mtime
age_seconds = current_time - modified_time
# Extract key from filename (remove .json extension)
key = filename[:-5] # Remove '.json'
# Format age display
if age_seconds < 60:
age_display = f"{int(age_seconds)}s"
elif age_seconds < 3600:
age_display = f"{int(age_seconds / 60)}m"
elif age_seconds < 86400:
age_display = f"{int(age_seconds / 3600)}h"
else:
age_display = f"{int(age_seconds / 86400)}d"
file_path = os.path.join(self.cache_dir, filename)
# Format size display
if size_bytes < 1024:
size_display = f"{size_bytes}B"
elif size_bytes < 1024 * 1024:
size_display = f"{size_bytes / 1024:.1f}KB"
else:
size_display = f"{size_bytes / (1024 * 1024):.1f}MB"
cache_files.append({
'key': key,
'filename': filename,
'age_seconds': age_seconds,
'age_display': age_display,
'size_bytes': size_bytes,
'size_display': size_display,
'path': file_path,
'modified_time': modified_time,
'modified_datetime': datetime.fromtimestamp(modified_time).isoformat()
})
except OSError as e:
self.logger.warning(f"Error getting stats for cache file {filename} at {file_path}: {e}", exc_info=True)
continue
try:
# Get file stats
stat_info = os.stat(file_path)
size_bytes = stat_info.st_size
modified_time = stat_info.st_mtime
age_seconds = current_time - modified_time
# Format age display
if age_seconds < 60:
age_display = f"{int(age_seconds)}s"
elif age_seconds < 3600:
age_display = f"{int(age_seconds / 60)}m"
elif age_seconds < 86400:
age_display = f"{int(age_seconds / 3600)}h"
else:
age_display = f"{int(age_seconds / 86400)}d"
# Format size display
if size_bytes < 1024:
size_display = f"{size_bytes}B"
elif size_bytes < 1024 * 1024:
size_display = f"{size_bytes / 1024:.1f}KB"
else:
size_display = f"{size_bytes / (1024 * 1024):.1f}MB"
cache_files.append({
'key': key,
'filename': filename,
'age_seconds': age_seconds,
'age_display': age_display,
'size_bytes': size_bytes,
'size_display': size_display,
'path': file_path,
'modified_time': modified_time,
'modified_datetime': datetime.fromtimestamp(modified_time).isoformat()
})
except OSError as e:
self.logger.warning(f"Error getting stats for cache file {filename} at {file_path}: {e}", exc_info=True)
continue
except OSError as e:
self.logger.error(f"Error listing cache directory {self.cache_dir}: {e}", exc_info=True)
return []
@@ -1001,14 +939,7 @@ class CacheManager:
Returns:
Dictionary with memory cache statistics
"""
with self._cache_lock:
return {
'size': len(self._memory_cache),
'max_size': self._max_memory_cache_size,
'usage_percent': (len(self._memory_cache) / self._max_memory_cache_size * 100) if self._max_memory_cache_size > 0 else 0,
'last_cleanup': self._last_memory_cache_cleanup,
'cleanup_interval': self._memory_cache_cleanup_interval
}
return self._memory_cache_component.get_stats()
def log_memory_cache_stats(self) -> None:
"""Log current memory cache statistics."""
test/test_background_data_service.py
+68
View File
@@ -297,3 +297,71 @@ class TestGetBackgroundService:
shutdown_background_service()
with pytest.raises(ValueError):
get_background_service()
# ---------------------------------------------------------------------------
# Sport cache keys
# ---------------------------------------------------------------------------
class _FrozenDatetime:
"""Stands in for the datetime class at a fixed instant. now() without a
tz answers in a UTC-4 local zone, so a key built from local time shows."""
def __init__(self, moment):
self._moment = moment
def now(self, tz=None):
from datetime import timedelta, timezone
if tz is None:
return self._moment.astimezone(timezone(timedelta(hours=-4))).replace(tzinfo=None)
return self._moment.astimezone(tz)
class TestSportCacheKey:
"""get_sport_cache_key() must produce CacheManager's key format without
building a CacheManager (config load + cache-dir probing) to do it."""
@pytest.mark.parametrize("sport,date_str", [
("nfl", "20260922"), ("ncaa_fb", "20251231"), ("soccer", ""),
])
def test_explicit_date_matches_cache_manager(self, service, sport, date_str):
from src.cache_manager import CacheManager
expected = CacheManager.generate_sport_cache_key(None, sport, date_str)
assert service.get_sport_cache_key(sport, date_str) == expected
def test_default_date_is_today_in_utc_like_cache_manager(self, service):
from datetime import datetime, timezone
import src.cache_manager as cm_module
# 23:30 on the 21st in New York is already the 22nd in UTC.
frozen = _FrozenDatetime(datetime(2026, 9, 22, 3, 30, tzinfo=timezone.utc))
with patch.object(bds_module, "datetime", frozen), \
patch.object(cm_module, "datetime", frozen):
ours = service.get_sport_cache_key("nba")
theirs = cm_module.CacheManager.generate_sport_cache_key(None, "nba")
assert ours == theirs == "nba_20260922"
def test_does_not_construct_a_cache_manager(self, service):
with patch("src.cache_manager.CacheManager.__init__",
side_effect=AssertionError("CacheManager constructed")):
assert service.get_sport_cache_key("nhl", "20260101") == "nhl_20260101"
assert service.get_sport_cache_key("nhl").startswith("nhl_")
def test_submit_without_cache_key_looks_up_todays_sport_key(
self, service, mock_cache_manager):
mock_cache_manager.get.return_value = {"events": []}
with patch("src.cache_manager.CacheManager.__init__",
side_effect=AssertionError("CacheManager constructed")):
service.submit_fetch_request("mlb", 2026, "http://example.invalid/x")
key = mock_cache_manager.get.call_args[0][0]
assert key == service.get_sport_cache_key("mlb")
class TestPriorityIsAcceptedAndIgnored:
def test_priority_keyword_is_accepted(self, service, mock_cache_manager):
mock_cache_manager.get.return_value = {"cached": True}
rid = service.submit_fetch_request(
"nfl", 2026, "http://example.invalid/x", cache_key="k", priority=5)
assert service.get_result(rid).cached is True
def test_statistics_still_report_an_empty_queue(self, service):
assert service.get_statistics()["queue_size"] == 0
test/test_cache_manager_memory_tier.py
+118
View File
@@ -0,0 +1,118 @@
"""CacheManager's memory tier is MemoryCache's, not a copy of it.
CacheManager used to re-implement MemoryCache.cleanup() line for line and
read the component's private dicts and lock through aliases bound at
construction. Those aliases went stale the moment the component was replaced
(tests do exactly that), and the listing of the cache *directory* held the
memory tier's lock for the whole scan.
"""
import os
import time
from unittest.mock import patch
import pytest
from src.cache.memory_cache import MemoryCache
from src.cache_manager import CacheManager
@pytest.fixture
def cm(tmp_path):
with patch('src.cache_manager.CacheManager._get_writable_cache_dir',
return_value=str(tmp_path)):
manager = CacheManager()
# The disk sweep thread stats this directory too; keep it out of the
# os.stat spies below.
manager.stop_cleanup_thread()
yield manager
def test_cleanup_and_stats_follow_a_replaced_component(cm):
"""Replace the component the way test_cache_ttl_honoured does; cleanup and
stats must act on the new one, not on dicts captured at construction."""
cm._memory_cache_component = MemoryCache(max_size=7, cleanup_interval=11.0)
cm._memory_cache_component.set("fresh", {"v": 1})
cm._memory_cache_component.set("stale", {"v": 2})
cm._memory_cache_component._timestamps["stale"] = time.time() - 4000
assert cm._cleanup_memory_cache(force=True) == 1
assert cm._memory_cache_component.get("stale") is None
assert cm._memory_cache_component.get("fresh") == {"v": 1}
stats = cm.get_memory_cache_stats()
assert stats["size"] == 1
assert stats["max_size"] == 7
assert stats["cleanup_interval"] == 11.0
assert stats["usage_percent"] == pytest.approx(100 / 7)
def test_periodic_cleanup_is_throttled_and_records_its_run(cm):
mem = cm._memory_cache_component
mem.set("stale", {"v": 1})
mem._timestamps["stale"] = time.time() - 4000
# Within the interval: nothing runs, even through the get path.
assert cm._cleanup_memory_cache() == 0
assert mem.size() == 1
mem._last_cleanup = time.time() - mem._cleanup_interval - 1
before = time.time()
cm.get_cached_data("missing") # triggers the periodic sweep
assert mem.size() == 0
assert cm.get_memory_cache_stats()["last_cleanup"] >= before
def test_stats_have_the_documented_shape(cm):
cm.set("k", {"v": 1})
stats = cm.get_memory_cache_stats()
assert set(stats) == {"size", "max_size", "usage_percent",
"last_cleanup", "cleanup_interval"}
assert stats["size"] == 1
assert stats["max_size"] == cm._memory_cache_component.max_size()
def test_listing_the_cache_dir_does_not_hold_the_memory_lock(cm, tmp_path):
"""8,864 files on a real rig: every get/set used to wait out the scan."""
for name in ("a", "b"):
(tmp_path / f"{name}.json").write_text("{}")
(tmp_path / "notes.txt").write_text("x")
lock = cm._memory_cache_component._lock
held_during_stat = []
real_stat = os.stat
def spying_stat(path, *args, **kwargs):
held_during_stat.append(lock.locked())
return real_stat(path, *args, **kwargs)
with patch('src.cache_manager.os.stat', side_effect=spying_stat):
files = cm.list_cache_files()
assert held_during_stat and not any(held_during_stat)
assert sorted(f["key"] for f in files) == ["a", "b"]
def test_listing_skips_a_file_deleted_mid_scan(cm, tmp_path):
for name in ("a", "b"):
(tmp_path / f"{name}.json").write_text("{}")
real_stat = os.stat
def vanishing_stat(path, *args, **kwargs):
if str(path).endswith("a.json"):
raise FileNotFoundError(path)
return real_stat(path, *args, **kwargs)
with patch('src.cache_manager.os.stat', side_effect=vanishing_stat):
files = cm.list_cache_files()
assert [f["key"] for f in files] == ["b"]
def test_listing_is_newest_first(cm, tmp_path):
now = time.time()
for i, name in enumerate(("old", "mid", "new")):
p = tmp_path / f"{name}.json"
p.write_text("{}")
os.utime(p, (now - 300 + i * 100, now - 300 + i * 100))
assert [f["key"] for f in cm.list_cache_files()] == ["new", "mid", "old"]
test/test_cache_strategy_intervals.py
+136
View File
@@ -0,0 +1,136 @@
"""CacheStrategy intervals, pinned across the whole input grid.
The strategy table used to carry a per-sport defaults dict whose every value
was 60, and a soccer branch identical to its else. These tests pin the
returned strategy for every data type x sport key x config shape, so
simplifying the lookup cannot change what any caller gets back. They were
written against the pre-cleanup code and pass on it unchanged, except for
the legacy `<sport>_scoreboard` config shape (see below), which that code
still read.
"""
import pytest
from src.cache.cache_strategy import CacheStrategy
class _Cfg:
def __init__(self, config):
self.config = config
class _NoConfigAttr:
pass
# Plugin config sections are keyed by plugin id. Their intervals belong to the
# plugin, and the strategy table has never read them.
_PLUGIN_ID_CONFIG = {
pid: {"live_update_interval": 5, "recent_update_interval": 7,
"upcoming_update_interval": 9}
for pid in ("football-scoreboard", "basketball-scoreboard",
"baseball-scoreboard", "hockey-scoreboard", "soccer-scoreboard")
}
# `<sport>_scoreboard` sections come from the built-in scoreboards the plugin
# system replaced. An install upgraded from that era can still carry them in
# config.json (nothing deletes them). No current caller passes a sport key to
# the strategy, but a stale section must not steer cache TTLs if one does.
_LEGACY_SCOREBOARD_CONFIG = {
f"{sport}_scoreboard": {"live_update_interval": 5,
"recent_update_interval": 7,
"upcoming_update_interval": 9}
for sport in ("nfl", "nba", "mlb", "nhl", "soccer", "ncaa_fb",
"ncaa_baseball", "ncaam_basketball", "milb")
}
CONFIG_MANAGERS = {
"no_config_manager": None,
"empty_config": _Cfg({}),
"plugin_id_config": _Cfg(_PLUGIN_ID_CONFIG),
"legacy_scoreboard_config": _Cfg(_LEGACY_SCOREBOARD_CONFIG),
"config_is_none": _Cfg(None),
"config_is_not_a_dict": _Cfg("x"),
"config_manager_without_config": _NoConfigAttr(),
}
SPORT_KEYS = [None, "", "nfl", "nba", "mlb", "nhl", "soccer", "ncaa_fb",
"ncaa_baseball", "ncaam_basketball", "milb",
"football-scoreboard", "curling"]
def _fixed(max_age, memory_ttl, **extra):
return {"max_age": max_age, "memory_ttl": memory_ttl,
"force_refresh": False, **extra}
DEFAULT = _fixed(300, 600)
FIXED = {
"weather_current": _fixed(300, 600),
"stocks": _fixed(600, 1200, market_hours_only=True),
"crypto": _fixed(300, 600),
"sports_recent": _fixed(1800, 3600),
"sports_upcoming": _fixed(10800, 21600),
"sports_schedules": _fixed(86400, 172800),
"leaderboard": _fixed(604800, 1209600),
"news": _fixed(3600, 7200),
"odds": _fixed(1800, 3600),
"odds_live": _fixed(120, 240),
"team_info": _fixed(604800, 1209600),
"logos": _fixed(2592000, 5184000),
"default": DEFAULT,
}
def _expected(data_type, sport_key):
if data_type in ("live_scores", "sports_live"):
if sport_key:
interval = 60
else:
interval = 15 if data_type == "live_scores" else 30
return {"max_age": interval, "memory_ttl": interval * 2,
"force_refresh": True}
return FIXED.get(data_type, DEFAULT)
@pytest.mark.parametrize("cm_name", sorted(CONFIG_MANAGERS))
def test_live_interval_is_60_for_every_sport(cm_name):
strategy = CacheStrategy(config_manager=CONFIG_MANAGERS[cm_name])
for sport_key in SPORT_KEYS:
assert strategy.get_sport_live_interval(sport_key) == 60, sport_key
@pytest.mark.parametrize("cm_name", sorted(CONFIG_MANAGERS))
def test_strategy_table_for_every_data_type_and_sport(cm_name):
strategy = CacheStrategy(config_manager=CONFIG_MANAGERS[cm_name])
data_types = ["live_scores", "sports_live", *FIXED, "unknown", ""]
for data_type in data_types:
for sport_key in SPORT_KEYS:
got = strategy.get_cache_strategy(data_type, sport_key)
assert got == _expected(data_type, sport_key), (data_type, sport_key)
@pytest.mark.parametrize("key", [
"soccer_live", "soccer_current", "soccer_scoreboard", "SOCCER_LIVE",
"nfl_live", "live", "hockey_current", "nba_live_scores",
])
def test_live_keys_including_soccer_are_sports_live(key):
assert CacheStrategy().get_data_type_from_key(key) == "sports_live"
@pytest.mark.parametrize("key,data_type", [
("odds_soccer_live", "odds_live"),
("odds_x", "odds"),
("weather", "weather_current"),
("crypto_stock", "crypto"),
("stock", "stocks"),
("news_soccer", "news"),
("soccer_schedule", "sports_schedules"),
("soccer_recent", "sports_recent"),
("soccer_upcoming", "sports_upcoming"),
("soccer_logo", "team_info"),
("soccer", "default"),
("", "default"),
])
def test_non_live_keys_keep_their_data_type(key, data_type):
assert CacheStrategy().get_data_type_from_key(key) == data_type