mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-08-12 06:08:05 +00:00
* refactor(web): use canonical secret helpers in api_v3; make ConfigManager secret strip/merge array-aware
api_v3.py carried three inline nested copies of find_secret_fields/
separate_secrets (main-config save, plugin-config save, plugin-config
reset). They drifted from each other (one lacked isinstance guards) and
none supported the canonical module's array-item secrets
(accounts[].token). All three endpoints now import from
src/web_interface/secret_helpers.
Adopting the canonical behavior makes array-item secrets reachable, and
their parallel-placeholder shape ([{'token': ...}, {}] alongside the
regular list) was not survivable by ConfigManager's round-trip:
_strip_secrets_recursive dropped the whole key (losing the regular
fields from config.json) and _deep_merge replaced the regular list
wholesale on load. Both are now array-aware:
- strip removes the secret fields from each item and ALWAYS keeps the
list so indices survive for merge-on-load; whole-key secrets (scalar
lists, shape mismatches) still drop the key entirely — never leak.
- merge folds each secrets item into the config item at the same index,
skipping {} placeholders. The regular list's length is authoritative
in both directions: a user deleting an array item never has it
resurrected from a stale secrets entry (extras warn and are ignored).
api_v3's own deep_merge intentionally still replaces lists wholesale —
form posts carry complete arrays and index-merging would resurrect
deleted items; a comment now documents that.
Tests: the parity guard flips from 'exactly 3 inline copies' to 'zero,
and the canonical import must exist'; TestArraySecretStripAndMerge
covers the new strip/merge semantics incl. length-mismatch contracts;
new test_api_v3_secret_roundtrip.py drives all three endpoints through
a Flask client with a REAL ConfigManager+SchemaManager over tmp_path,
proving secrets land in config_secrets.json, config.json stays clean,
and a fresh load merges them back into the right array items.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh
* fix: repair broken helper paths across display, cache, odds, logging, resolver, repos, config, validator
Nine fixes for bugs surfaced while writing coverage for previously
untested modules (plus the bool-duration quirk pinned in PR #441):
- base_plugin.get_display_duration: exclude bools from both numeric
branches — display_duration=True no longer reads as a 1-second slot;
it falls through to config, then the 15.0 default.
- display_helper: draw_error_message/draw_no_data_message called
_draw_centered_text with the wrong arguments and crashed with
AttributeError — both now delegate to draw_centered_text.
draw_scorebug_layout drew status and clock at the same y, overprinting
each other — they now share one combined top line.
draw_ticker_layout drew its text starting at x=display_width (fully
off-canvas), returning a blank frame every time — now draws at x=0;
scroll_speed stays accepted-but-unused and is documented as such.
- api_helper.clear_cache guarded on a nonexistent CacheManager.clear()
method, silently never clearing anything; it now uses the real surface
(clear_cache/delete/list_cache_files) and no-ops safely otherwise.
- base_odds_manager._extract_espn_data raised AttributeError when ESPN
sent explicit JSON nulls ("homeTeamOdds": null) — every level now
null-safes with 'or {}'. format_odds_summary gated on
is_odds_available, which deliberately ignores money lines, so
ML-only odds formatted as "No odds available" — it now gates only on
empty/no_odds data and formats money lines.
- logging_config.ContextualFormatter mutated record.msg in place, so a
second handler prepended the context prefix twice; it now formats a
copy. log_error hardcoded exc_info=True and raised TypeError when the
caller passed exc_info — now kwargs.setdefault.
- dynamic_team_resolver wrote its "shared" class cache through self,
creating instance shadows — the cache was per-instance and every
scoreboard refetched rankings. Writes now go through the class.
- saved_repositories cleaned URLs with an unanchored .replace('.git','')
that mangled URLs merely containing '.git' (my.github.io -> myhub.io);
now strips only a trailing suffix. add/remove also roll back the
in-memory list when the save fails, so memory always matches disk.
- config_helper.merge_configs shallow-copied the base, aliasing every
un-overridden nested dict into the result — now deep-copies.
- startup_validator.validate_all accumulated errors/warnings across
calls — now resets both lists per run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh
* test: cover the previously untested modules
Nine new suites plus an extension, asserting the Phase-1b fixed behavior
and pinning the quirks deliberately left alone:
- test_logging_config.py: formatters (JSON shape, no record mutation,
single prefix through two handlers), PluginLoggerAdapter precedence,
setup_logging handler hygiene and LEDMATRIX_DEBUG, log_error exc_info.
- test_startup_validator.py: exact messages, error-vs-warning split,
accessor split (load_config vs get_config), cache-dir branches with
os.access monkeypatched (root can write anything in CI), idempotence,
raise_on_errors classification precedence.
- test_config_helper.py (full): load/save round trips, dot-notation
get/set incl. silent-failure contract, post-fix no-aliasing merge,
schema validation branches, the '{id}_config' key pin, default-enabled
pin.
- test_saved_repositories.py: three load shapes, bare-list rewrite pin,
trailing-only .git strip (my.github.io regression), save-failure
rollback, type-classification case-sensitivity pin.
- test_api_helper.py: rate-limit math, cache-hit short circuit, ESPN
URL/key formats, exact User-Agent guard, retry adapter, post-fix
clear_cache against the real CacheManager surface, ttl-dropped pin.
- test_base_odds_manager.py: cache-key/URL construction, no_odds
sentinel round trip, stale-cache fallback, null-safe extraction,
ML-only formatting, is_odds_available truth table (ML-blind by
contract), config key/attr mismatch pin.
- test_dynamic_team_resolver.py: expansion/dedup/slicing, dropped
unknown-dynamic names (TOP_ substring hazard pinned), genuinely
shared class cache (second instance: zero HTTP), TTL expiry,
failure degradation without raising.
- test_display_helper.py (full): the fixed error/no-data renders,
combined scorebug top line, non-blank ticker with scroll_speed
no-op pin, composite upconversion, logo bleed positions, square
orientation pin.
- test_skin_runtime_cache.py: discovery-cache hit/invalidation
semantics (manifest mtime, .py edits pinned as non-invalidating),
sys.modules namespacing contract incl. bare-name restore and stdlib
shadowing, entry-module execute-once, API minor-version tolerance,
skin_matches_target table.
- test_sports_capabilities.py (extended): _draw_celebration_layout
executed for real (flash window, matrix-dims fallback, highlight
alternation, logo-failure isolation), _should_celebrate_for direct,
strict duration boundary, score_to_int edges, both-teams-score
precedence, expired-coalesce refire, disabled-win baseline
preservation, id-less prune.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh
* test: real schedule/dim coverage for DisplayController; fix two vacuous schedule tests
New test_display_controller_schedule.py drives _check_schedule and
_check_dim_schedule on a bare controller stub: same-day and
midnight-crossing windows with inclusive boundaries, global vs per-day vs
legacy-inferred modes (and dim's global-only default — no legacy
inference), per-day disabled days, invalid %H:%M fallbacks, unknown
timezone -> UTC, dim_brightness default 30, inactive-display short
circuit, and the _was_display_active/_was_dimmed transition flags.
test_display_controller.py's test_schedule_disabled and
test_active_hours patched config_service.get_config — which
_check_schedule never reads — so both asserted the init-default value
and could not fail. Rewritten on the test_inactive_hours pattern
(inject controller.config['schedule'], reset the minute gate, flip the
flag to the opposite state first so the assertion has teeth).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh
* ci: raise coverage floor to 48%
Measured 50% with the new suites in place (was 47% baseline when the
gate was introduced at 45); floor stays two points under measured.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh
* fix: address CodeQL alert and review findings
- config_manager: the "secrets list longer than config list" warning now
interpolates only config-side data (no key name or secrets-derived
values), resolving the CodeQL clear-text-logging alert.
- base_plugin: validate_config rejects bool display_duration, matching
get_display_duration (bool is an int subclass and would otherwise pass
as a positive number).
- config_helper: merge_configs deep-copies override values in the
non-recursive branch so mutating the merged result cannot reach back
into override_config.
- saved_repositories: saves are atomic (temp file + fsync + os.replace),
so a failed write can no longer truncate saved_repositories.json.
- tests: regression cases for each fix, plus a pin that whole-item
array secrets (key[] + key[].field both marked) strip to empty {}
skeletons — no secret values can reach config.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh
---------
Co-authored-by: Claude <noreply@anthropic.com>
349 lines
12 KiB
Python
349 lines
12 KiB
Python
"""
|
|
API Helper
|
|
|
|
Handles HTTP requests, caching, and ESPN API integration for LED matrix plugins.
|
|
Extracted from LEDMatrix core to provide reusable functionality for plugins.
|
|
"""
|
|
|
|
import logging
|
|
import time
|
|
from datetime import datetime
|
|
from typing import Any, Dict, Optional
|
|
|
|
import requests
|
|
from requests.adapters import HTTPAdapter
|
|
from urllib3.util.retry import Retry
|
|
|
|
|
|
class APIHelper:
|
|
"""
|
|
Helper class for HTTP requests, caching, and ESPN API integration.
|
|
|
|
Provides functionality for:
|
|
- HTTP requests with retry logic and timeouts
|
|
- Response caching with TTL support
|
|
- ESPN API integration for sports data
|
|
- Request rate limiting and throttling
|
|
"""
|
|
|
|
def __init__(self, cache_manager=None, default_timeout: int = 30,
|
|
max_retries: int = 3, logger: Optional[logging.Logger] = None):
|
|
"""
|
|
Initialize the APIHelper.
|
|
|
|
Args:
|
|
cache_manager: Optional cache manager for response caching
|
|
default_timeout: Default timeout for requests in seconds
|
|
max_retries: Maximum number of retry attempts
|
|
logger: Optional logger instance
|
|
"""
|
|
self.cache_manager = cache_manager
|
|
self.default_timeout = default_timeout
|
|
self.max_retries = max_retries
|
|
self.logger = logger or logging.getLogger(__name__)
|
|
|
|
# Setup session with retry strategy
|
|
self.session = requests.Session()
|
|
retry_strategy = Retry(
|
|
total=max_retries,
|
|
backoff_factor=1,
|
|
status_forcelist=[429, 500, 502, 503, 504],
|
|
allowed_methods=["GET", "HEAD", "OPTIONS"]
|
|
)
|
|
adapter = HTTPAdapter(max_retries=retry_strategy)
|
|
self.session.mount("https://", adapter)
|
|
self.session.mount("http://", adapter)
|
|
|
|
# Default headers
|
|
self.session.headers.update({
|
|
# Identifies the client and links to it: ESPN began 403ing bare
|
|
# custom tokens (and browser strings) around 2026-08-04.
|
|
'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)',
|
|
'Accept': 'application/json',
|
|
'Accept-Language': 'en-US,en;q=0.9',
|
|
'Accept-Encoding': 'gzip, deflate, br',
|
|
'Connection': 'keep-alive'
|
|
})
|
|
|
|
# Rate limiting
|
|
self._last_request_time = 0
|
|
self._min_request_interval = 1.0 # Minimum seconds between requests
|
|
|
|
def get(self, url: str, params: Optional[Dict] = None,
|
|
headers: Optional[Dict] = None, timeout: Optional[int] = None,
|
|
cache_key: Optional[str] = None, cache_ttl: int = 3600) -> Optional[Dict]:
|
|
"""
|
|
Make a GET request with optional caching.
|
|
|
|
Args:
|
|
url: URL to request
|
|
params: Query parameters
|
|
headers: Additional headers
|
|
timeout: Request timeout (uses default if None)
|
|
cache_key: Key for caching response
|
|
cache_ttl: Cache time-to-live in seconds
|
|
|
|
Returns:
|
|
Response data as dictionary or None if request fails
|
|
"""
|
|
# Check cache first
|
|
if cache_key and self.cache_manager:
|
|
cached = self._get_from_cache(cache_key)
|
|
if cached is not None:
|
|
self.logger.debug(f"Using cached response for {cache_key}")
|
|
return cached
|
|
|
|
# Rate limiting
|
|
self._enforce_rate_limit()
|
|
|
|
try:
|
|
# Prepare request
|
|
request_headers = self.session.headers.copy()
|
|
if headers:
|
|
request_headers.update(headers)
|
|
|
|
# Make request
|
|
response = self.session.get(
|
|
url,
|
|
params=params,
|
|
headers=request_headers,
|
|
timeout=timeout or self.default_timeout
|
|
)
|
|
response.raise_for_status()
|
|
|
|
# Parse JSON response
|
|
data = response.json()
|
|
|
|
# Cache response if cache key provided
|
|
if cache_key and self.cache_manager:
|
|
self._set_cache(cache_key, data, cache_ttl)
|
|
|
|
self.logger.debug(f"Successfully fetched {url}")
|
|
return data
|
|
|
|
except requests.exceptions.RequestException as e:
|
|
self.logger.error(f"Request failed for {url}: {e}")
|
|
return None
|
|
|
|
def fetch_espn_scoreboard(self, sport: str, league: str,
|
|
date: Optional[str] = None,
|
|
cache_key: Optional[str] = None,
|
|
cache_ttl: int = 300) -> Optional[Dict]:
|
|
"""
|
|
Fetch ESPN scoreboard data for a specific sport and league.
|
|
|
|
Args:
|
|
sport: Sport name (e.g., 'basketball', 'football')
|
|
league: League name (e.g., 'nba', 'nfl')
|
|
date: Date in YYYYMMDD format (defaults to today)
|
|
cache_key: Cache key for response
|
|
cache_ttl: Cache time-to-live in seconds
|
|
|
|
Returns:
|
|
ESPN API response data or None if request fails
|
|
"""
|
|
if date is None:
|
|
date = datetime.now().strftime('%Y%m%d')
|
|
|
|
# Build URL
|
|
url = f"https://site.api.espn.com/apis/site/v2/sports/{sport}/{league}/scoreboard"
|
|
|
|
# Build cache key if not provided
|
|
if cache_key is None:
|
|
cache_key = f"espn_{sport}_{league}_{date}"
|
|
|
|
# Set parameters
|
|
params = {
|
|
'dates': date,
|
|
'limit': 1000
|
|
}
|
|
|
|
return self.get(url, params=params, cache_key=cache_key, cache_ttl=cache_ttl)
|
|
|
|
def fetch_espn_standings(self, sport: str, league: str,
|
|
cache_key: Optional[str] = None,
|
|
cache_ttl: int = 3600) -> Optional[Dict]:
|
|
"""
|
|
Fetch ESPN standings data for a specific sport and league.
|
|
|
|
Args:
|
|
sport: Sport name
|
|
league: League name
|
|
cache_key: Cache key for response
|
|
cache_ttl: Cache time-to-live in seconds
|
|
|
|
Returns:
|
|
ESPN standings data or None if request fails
|
|
"""
|
|
url = f"https://site.api.espn.com/apis/site/v2/sports/{sport}/{league}/standings"
|
|
|
|
if cache_key is None:
|
|
cache_key = f"espn_standings_{sport}_{league}"
|
|
|
|
return self.get(url, cache_key=cache_key, cache_ttl=cache_ttl)
|
|
|
|
def fetch_espn_rankings(self, sport: str, league: str,
|
|
cache_key: Optional[str] = None,
|
|
cache_ttl: int = 3600) -> Optional[Dict]:
|
|
"""
|
|
Fetch ESPN rankings data for a specific sport and league.
|
|
|
|
Args:
|
|
sport: Sport name
|
|
league: League name
|
|
cache_key: Cache key for response
|
|
cache_ttl: Cache time-to-live in seconds
|
|
|
|
Returns:
|
|
ESPN rankings data or None if request fails
|
|
"""
|
|
url = f"https://site.api.espn.com/apis/site/v2/sports/{sport}/{league}/rankings"
|
|
|
|
if cache_key is None:
|
|
cache_key = f"espn_rankings_{sport}_{league}"
|
|
|
|
return self.get(url, cache_key=cache_key, cache_ttl=cache_ttl)
|
|
|
|
def post(self, url: str, data: Optional[Dict] = None,
|
|
json_data: Optional[Dict] = None,
|
|
headers: Optional[Dict] = None,
|
|
timeout: Optional[int] = None) -> Optional[Dict]:
|
|
"""
|
|
Make a POST request.
|
|
|
|
Args:
|
|
url: URL to request
|
|
data: Form data
|
|
json_data: JSON data
|
|
headers: Additional headers
|
|
timeout: Request timeout
|
|
|
|
Returns:
|
|
Response data as dictionary or None if request fails
|
|
"""
|
|
self._enforce_rate_limit()
|
|
|
|
try:
|
|
request_headers = self.session.headers.copy()
|
|
if headers:
|
|
request_headers.update(headers)
|
|
|
|
response = self.session.post(
|
|
url,
|
|
data=data,
|
|
json=json_data,
|
|
headers=request_headers,
|
|
timeout=timeout or self.default_timeout
|
|
)
|
|
response.raise_for_status()
|
|
|
|
return response.json()
|
|
|
|
except requests.exceptions.RequestException as e:
|
|
self.logger.error(f"POST request failed for {url}: {e}")
|
|
return None
|
|
|
|
def set_cache(self, key: str, data: Any, ttl: int = 3600) -> None:
|
|
"""
|
|
Set cache data.
|
|
|
|
Args:
|
|
key: Cache key
|
|
data: Data to cache
|
|
ttl: Time-to-live in seconds (ignored - CacheManager doesn't support TTL)
|
|
"""
|
|
if self.cache_manager:
|
|
self.cache_manager.set(key, data)
|
|
|
|
def get_cache(self, key: str) -> Optional[Any]:
|
|
"""
|
|
Get cached data.
|
|
|
|
Args:
|
|
key: Cache key
|
|
|
|
Returns:
|
|
Cached data or None if not found
|
|
"""
|
|
if self.cache_manager:
|
|
return self.cache_manager.get(key)
|
|
return None
|
|
|
|
def clear_cache(self, pattern: Optional[str] = None) -> None:
|
|
"""
|
|
Clear cache data.
|
|
|
|
Uses CacheManager's real surface (clear_cache / delete /
|
|
list_cache_files); safely no-ops on managers without it. The old
|
|
implementation guarded on a nonexistent ``clear`` method, so it
|
|
silently never cleared anything.
|
|
|
|
Args:
|
|
pattern: Optional substring to match cache keys; only matching
|
|
entries are deleted.
|
|
"""
|
|
if not self.cache_manager:
|
|
return
|
|
if pattern:
|
|
if (hasattr(self.cache_manager, 'list_cache_files')
|
|
and hasattr(self.cache_manager, 'delete')):
|
|
for entry in self.cache_manager.list_cache_files():
|
|
key = entry.get('key') if isinstance(entry, dict) else None
|
|
if key and pattern in key:
|
|
self.cache_manager.delete(key)
|
|
else:
|
|
self.logger.debug(
|
|
"Cache manager lacks list_cache_files/delete; "
|
|
"cannot clear by pattern")
|
|
elif hasattr(self.cache_manager, 'clear_cache'):
|
|
self.cache_manager.clear_cache()
|
|
elif hasattr(self.cache_manager, 'clear'):
|
|
self.cache_manager.clear()
|
|
else:
|
|
self.logger.debug("Cache manager exposes no clear method; no-op")
|
|
|
|
def _get_from_cache(self, key: str) -> Optional[Any]:
|
|
"""Get data from cache."""
|
|
if self.cache_manager:
|
|
return self.cache_manager.get(key)
|
|
return None
|
|
|
|
def _set_cache(self, key: str, data: Any, ttl: int) -> None:
|
|
"""Set data in cache."""
|
|
if self.cache_manager:
|
|
self.cache_manager.set(key, data)
|
|
|
|
def _enforce_rate_limit(self) -> None:
|
|
"""Enforce rate limiting between requests."""
|
|
current_time = time.time()
|
|
time_since_last = current_time - self._last_request_time
|
|
|
|
if time_since_last < self._min_request_interval:
|
|
sleep_time = self._min_request_interval - time_since_last
|
|
time.sleep(sleep_time)
|
|
|
|
self._last_request_time = time.time()
|
|
|
|
def set_rate_limit(self, min_interval: float) -> None:
|
|
"""
|
|
Set minimum interval between requests.
|
|
|
|
Args:
|
|
min_interval: Minimum seconds between requests
|
|
"""
|
|
self._min_request_interval = min_interval
|
|
self.logger.debug(f"Rate limit set to {min_interval} seconds")
|
|
|
|
def get_request_stats(self) -> Dict[str, Any]:
|
|
"""
|
|
Get request statistics.
|
|
|
|
Returns:
|
|
Dictionary with request statistics
|
|
"""
|
|
return {
|
|
'min_request_interval': self._min_request_interval,
|
|
'last_request_time': self._last_request_time,
|
|
'time_since_last_request': time.time() - self._last_request_time
|
|
}
|