Files
LEDMatrix/src/common/api_helper.py
T
ee59caa577 Follow-ups from #441: secret-helper migration, ten more bug fixes, and coverage for every remaining untested module (#444)
* refactor(web): use canonical secret helpers in api_v3; make ConfigManager secret strip/merge array-aware

api_v3.py carried three inline nested copies of find_secret_fields/
separate_secrets (main-config save, plugin-config save, plugin-config
reset). They drifted from each other (one lacked isinstance guards) and
none supported the canonical module's array-item secrets
(accounts[].token). All three endpoints now import from
src/web_interface/secret_helpers.

Adopting the canonical behavior makes array-item secrets reachable, and
their parallel-placeholder shape ([{'token': ...}, {}] alongside the
regular list) was not survivable by ConfigManager's round-trip:
_strip_secrets_recursive dropped the whole key (losing the regular
fields from config.json) and _deep_merge replaced the regular list
wholesale on load. Both are now array-aware:

- strip removes the secret fields from each item and ALWAYS keeps the
  list so indices survive for merge-on-load; whole-key secrets (scalar
  lists, shape mismatches) still drop the key entirely — never leak.
- merge folds each secrets item into the config item at the same index,
  skipping {} placeholders. The regular list's length is authoritative
  in both directions: a user deleting an array item never has it
  resurrected from a stale secrets entry (extras warn and are ignored).

api_v3's own deep_merge intentionally still replaces lists wholesale —
form posts carry complete arrays and index-merging would resurrect
deleted items; a comment now documents that.

Tests: the parity guard flips from 'exactly 3 inline copies' to 'zero,
and the canonical import must exist'; TestArraySecretStripAndMerge
covers the new strip/merge semantics incl. length-mismatch contracts;
new test_api_v3_secret_roundtrip.py drives all three endpoints through
a Flask client with a REAL ConfigManager+SchemaManager over tmp_path,
proving secrets land in config_secrets.json, config.json stays clean,
and a fresh load merges them back into the right array items.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: repair broken helper paths across display, cache, odds, logging, resolver, repos, config, validator

Nine fixes for bugs surfaced while writing coverage for previously
untested modules (plus the bool-duration quirk pinned in PR #441):

- base_plugin.get_display_duration: exclude bools from both numeric
  branches — display_duration=True no longer reads as a 1-second slot;
  it falls through to config, then the 15.0 default.
- display_helper: draw_error_message/draw_no_data_message called
  _draw_centered_text with the wrong arguments and crashed with
  AttributeError — both now delegate to draw_centered_text.
  draw_scorebug_layout drew status and clock at the same y, overprinting
  each other — they now share one combined top line.
  draw_ticker_layout drew its text starting at x=display_width (fully
  off-canvas), returning a blank frame every time — now draws at x=0;
  scroll_speed stays accepted-but-unused and is documented as such.
- api_helper.clear_cache guarded on a nonexistent CacheManager.clear()
  method, silently never clearing anything; it now uses the real surface
  (clear_cache/delete/list_cache_files) and no-ops safely otherwise.
- base_odds_manager._extract_espn_data raised AttributeError when ESPN
  sent explicit JSON nulls ("homeTeamOdds": null) — every level now
  null-safes with 'or {}'. format_odds_summary gated on
  is_odds_available, which deliberately ignores money lines, so
  ML-only odds formatted as "No odds available" — it now gates only on
  empty/no_odds data and formats money lines.
- logging_config.ContextualFormatter mutated record.msg in place, so a
  second handler prepended the context prefix twice; it now formats a
  copy. log_error hardcoded exc_info=True and raised TypeError when the
  caller passed exc_info — now kwargs.setdefault.
- dynamic_team_resolver wrote its "shared" class cache through self,
  creating instance shadows — the cache was per-instance and every
  scoreboard refetched rankings. Writes now go through the class.
- saved_repositories cleaned URLs with an unanchored .replace('.git','')
  that mangled URLs merely containing '.git' (my.github.io -> myhub.io);
  now strips only a trailing suffix. add/remove also roll back the
  in-memory list when the save fails, so memory always matches disk.
- config_helper.merge_configs shallow-copied the base, aliasing every
  un-overridden nested dict into the result — now deep-copies.
- startup_validator.validate_all accumulated errors/warnings across
  calls — now resets both lists per run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: cover the previously untested modules

Nine new suites plus an extension, asserting the Phase-1b fixed behavior
and pinning the quirks deliberately left alone:

- test_logging_config.py: formatters (JSON shape, no record mutation,
  single prefix through two handlers), PluginLoggerAdapter precedence,
  setup_logging handler hygiene and LEDMATRIX_DEBUG, log_error exc_info.
- test_startup_validator.py: exact messages, error-vs-warning split,
  accessor split (load_config vs get_config), cache-dir branches with
  os.access monkeypatched (root can write anything in CI), idempotence,
  raise_on_errors classification precedence.
- test_config_helper.py (full): load/save round trips, dot-notation
  get/set incl. silent-failure contract, post-fix no-aliasing merge,
  schema validation branches, the '{id}_config' key pin, default-enabled
  pin.
- test_saved_repositories.py: three load shapes, bare-list rewrite pin,
  trailing-only .git strip (my.github.io regression), save-failure
  rollback, type-classification case-sensitivity pin.
- test_api_helper.py: rate-limit math, cache-hit short circuit, ESPN
  URL/key formats, exact User-Agent guard, retry adapter, post-fix
  clear_cache against the real CacheManager surface, ttl-dropped pin.
- test_base_odds_manager.py: cache-key/URL construction, no_odds
  sentinel round trip, stale-cache fallback, null-safe extraction,
  ML-only formatting, is_odds_available truth table (ML-blind by
  contract), config key/attr mismatch pin.
- test_dynamic_team_resolver.py: expansion/dedup/slicing, dropped
  unknown-dynamic names (TOP_ substring hazard pinned), genuinely
  shared class cache (second instance: zero HTTP), TTL expiry,
  failure degradation without raising.
- test_display_helper.py (full): the fixed error/no-data renders,
  combined scorebug top line, non-blank ticker with scroll_speed
  no-op pin, composite upconversion, logo bleed positions, square
  orientation pin.
- test_skin_runtime_cache.py: discovery-cache hit/invalidation
  semantics (manifest mtime, .py edits pinned as non-invalidating),
  sys.modules namespacing contract incl. bare-name restore and stdlib
  shadowing, entry-module execute-once, API minor-version tolerance,
  skin_matches_target table.
- test_sports_capabilities.py (extended): _draw_celebration_layout
  executed for real (flash window, matrix-dims fallback, highlight
  alternation, logo-failure isolation), _should_celebrate_for direct,
  strict duration boundary, score_to_int edges, both-teams-score
  precedence, expired-coalesce refire, disabled-win baseline
  preservation, id-less prune.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: real schedule/dim coverage for DisplayController; fix two vacuous schedule tests

New test_display_controller_schedule.py drives _check_schedule and
_check_dim_schedule on a bare controller stub: same-day and
midnight-crossing windows with inclusive boundaries, global vs per-day vs
legacy-inferred modes (and dim's global-only default — no legacy
inference), per-day disabled days, invalid %H:%M fallbacks, unknown
timezone -> UTC, dim_brightness default 30, inactive-display short
circuit, and the _was_display_active/_was_dimmed transition flags.

test_display_controller.py's test_schedule_disabled and
test_active_hours patched config_service.get_config — which
_check_schedule never reads — so both asserted the init-default value
and could not fail. Rewritten on the test_inactive_hours pattern
(inject controller.config['schedule'], reset the minute gate, flip the
flag to the opposite state first so the assertion has teeth).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* ci: raise coverage floor to 48%

Measured 50% with the new suites in place (was 47% baseline when the
gate was introduced at 45); floor stays two points under measured.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: address CodeQL alert and review findings

- config_manager: the "secrets list longer than config list" warning now
  interpolates only config-side data (no key name or secrets-derived
  values), resolving the CodeQL clear-text-logging alert.
- base_plugin: validate_config rejects bool display_duration, matching
  get_display_duration (bool is an int subclass and would otherwise pass
  as a positive number).
- config_helper: merge_configs deep-copies override values in the
  non-recursive branch so mutating the merged result cannot reach back
  into override_config.
- saved_repositories: saves are atomic (temp file + fsync + os.replace),
  so a failed write can no longer truncate saved_repositories.json.
- tests: regression cases for each fix, plus a pin that whole-item
  array secrets (key[] + key[].field both marked) strip to empty {}
  skeletons — no secret values can reach config.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-07 16:17:11 -04:00

349 lines
12 KiB
Python

"""
API Helper
Handles HTTP requests, caching, and ESPN API integration for LED matrix plugins.
Extracted from LEDMatrix core to provide reusable functionality for plugins.
"""
import logging
import time
from datetime import datetime
from typing import Any, Dict, Optional
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
class APIHelper:
"""
Helper class for HTTP requests, caching, and ESPN API integration.
Provides functionality for:
- HTTP requests with retry logic and timeouts
- Response caching with TTL support
- ESPN API integration for sports data
- Request rate limiting and throttling
"""
def __init__(self, cache_manager=None, default_timeout: int = 30,
max_retries: int = 3, logger: Optional[logging.Logger] = None):
"""
Initialize the APIHelper.
Args:
cache_manager: Optional cache manager for response caching
default_timeout: Default timeout for requests in seconds
max_retries: Maximum number of retry attempts
logger: Optional logger instance
"""
self.cache_manager = cache_manager
self.default_timeout = default_timeout
self.max_retries = max_retries
self.logger = logger or logging.getLogger(__name__)
# Setup session with retry strategy
self.session = requests.Session()
retry_strategy = Retry(
total=max_retries,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET", "HEAD", "OPTIONS"]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
self.session.mount("https://", adapter)
self.session.mount("http://", adapter)
# Default headers
self.session.headers.update({
# Identifies the client and links to it: ESPN began 403ing bare
# custom tokens (and browser strings) around 2026-08-04.
'User-Agent': 'LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)',
'Accept': 'application/json',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive'
})
# Rate limiting
self._last_request_time = 0
self._min_request_interval = 1.0 # Minimum seconds between requests
def get(self, url: str, params: Optional[Dict] = None,
headers: Optional[Dict] = None, timeout: Optional[int] = None,
cache_key: Optional[str] = None, cache_ttl: int = 3600) -> Optional[Dict]:
"""
Make a GET request with optional caching.
Args:
url: URL to request
params: Query parameters
headers: Additional headers
timeout: Request timeout (uses default if None)
cache_key: Key for caching response
cache_ttl: Cache time-to-live in seconds
Returns:
Response data as dictionary or None if request fails
"""
# Check cache first
if cache_key and self.cache_manager:
cached = self._get_from_cache(cache_key)
if cached is not None:
self.logger.debug(f"Using cached response for {cache_key}")
return cached
# Rate limiting
self._enforce_rate_limit()
try:
# Prepare request
request_headers = self.session.headers.copy()
if headers:
request_headers.update(headers)
# Make request
response = self.session.get(
url,
params=params,
headers=request_headers,
timeout=timeout or self.default_timeout
)
response.raise_for_status()
# Parse JSON response
data = response.json()
# Cache response if cache key provided
if cache_key and self.cache_manager:
self._set_cache(cache_key, data, cache_ttl)
self.logger.debug(f"Successfully fetched {url}")
return data
except requests.exceptions.RequestException as e:
self.logger.error(f"Request failed for {url}: {e}")
return None
def fetch_espn_scoreboard(self, sport: str, league: str,
date: Optional[str] = None,
cache_key: Optional[str] = None,
cache_ttl: int = 300) -> Optional[Dict]:
"""
Fetch ESPN scoreboard data for a specific sport and league.
Args:
sport: Sport name (e.g., 'basketball', 'football')
league: League name (e.g., 'nba', 'nfl')
date: Date in YYYYMMDD format (defaults to today)
cache_key: Cache key for response
cache_ttl: Cache time-to-live in seconds
Returns:
ESPN API response data or None if request fails
"""
if date is None:
date = datetime.now().strftime('%Y%m%d')
# Build URL
url = f"https://site.api.espn.com/apis/site/v2/sports/{sport}/{league}/scoreboard"
# Build cache key if not provided
if cache_key is None:
cache_key = f"espn_{sport}_{league}_{date}"
# Set parameters
params = {
'dates': date,
'limit': 1000
}
return self.get(url, params=params, cache_key=cache_key, cache_ttl=cache_ttl)
def fetch_espn_standings(self, sport: str, league: str,
cache_key: Optional[str] = None,
cache_ttl: int = 3600) -> Optional[Dict]:
"""
Fetch ESPN standings data for a specific sport and league.
Args:
sport: Sport name
league: League name
cache_key: Cache key for response
cache_ttl: Cache time-to-live in seconds
Returns:
ESPN standings data or None if request fails
"""
url = f"https://site.api.espn.com/apis/site/v2/sports/{sport}/{league}/standings"
if cache_key is None:
cache_key = f"espn_standings_{sport}_{league}"
return self.get(url, cache_key=cache_key, cache_ttl=cache_ttl)
def fetch_espn_rankings(self, sport: str, league: str,
cache_key: Optional[str] = None,
cache_ttl: int = 3600) -> Optional[Dict]:
"""
Fetch ESPN rankings data for a specific sport and league.
Args:
sport: Sport name
league: League name
cache_key: Cache key for response
cache_ttl: Cache time-to-live in seconds
Returns:
ESPN rankings data or None if request fails
"""
url = f"https://site.api.espn.com/apis/site/v2/sports/{sport}/{league}/rankings"
if cache_key is None:
cache_key = f"espn_rankings_{sport}_{league}"
return self.get(url, cache_key=cache_key, cache_ttl=cache_ttl)
def post(self, url: str, data: Optional[Dict] = None,
json_data: Optional[Dict] = None,
headers: Optional[Dict] = None,
timeout: Optional[int] = None) -> Optional[Dict]:
"""
Make a POST request.
Args:
url: URL to request
data: Form data
json_data: JSON data
headers: Additional headers
timeout: Request timeout
Returns:
Response data as dictionary or None if request fails
"""
self._enforce_rate_limit()
try:
request_headers = self.session.headers.copy()
if headers:
request_headers.update(headers)
response = self.session.post(
url,
data=data,
json=json_data,
headers=request_headers,
timeout=timeout or self.default_timeout
)
response.raise_for_status()
return response.json()
except requests.exceptions.RequestException as e:
self.logger.error(f"POST request failed for {url}: {e}")
return None
def set_cache(self, key: str, data: Any, ttl: int = 3600) -> None:
"""
Set cache data.
Args:
key: Cache key
data: Data to cache
ttl: Time-to-live in seconds (ignored - CacheManager doesn't support TTL)
"""
if self.cache_manager:
self.cache_manager.set(key, data)
def get_cache(self, key: str) -> Optional[Any]:
"""
Get cached data.
Args:
key: Cache key
Returns:
Cached data or None if not found
"""
if self.cache_manager:
return self.cache_manager.get(key)
return None
def clear_cache(self, pattern: Optional[str] = None) -> None:
"""
Clear cache data.
Uses CacheManager's real surface (clear_cache / delete /
list_cache_files); safely no-ops on managers without it. The old
implementation guarded on a nonexistent ``clear`` method, so it
silently never cleared anything.
Args:
pattern: Optional substring to match cache keys; only matching
entries are deleted.
"""
if not self.cache_manager:
return
if pattern:
if (hasattr(self.cache_manager, 'list_cache_files')
and hasattr(self.cache_manager, 'delete')):
for entry in self.cache_manager.list_cache_files():
key = entry.get('key') if isinstance(entry, dict) else None
if key and pattern in key:
self.cache_manager.delete(key)
else:
self.logger.debug(
"Cache manager lacks list_cache_files/delete; "
"cannot clear by pattern")
elif hasattr(self.cache_manager, 'clear_cache'):
self.cache_manager.clear_cache()
elif hasattr(self.cache_manager, 'clear'):
self.cache_manager.clear()
else:
self.logger.debug("Cache manager exposes no clear method; no-op")
def _get_from_cache(self, key: str) -> Optional[Any]:
"""Get data from cache."""
if self.cache_manager:
return self.cache_manager.get(key)
return None
def _set_cache(self, key: str, data: Any, ttl: int) -> None:
"""Set data in cache."""
if self.cache_manager:
self.cache_manager.set(key, data)
def _enforce_rate_limit(self) -> None:
"""Enforce rate limiting between requests."""
current_time = time.time()
time_since_last = current_time - self._last_request_time
if time_since_last < self._min_request_interval:
sleep_time = self._min_request_interval - time_since_last
time.sleep(sleep_time)
self._last_request_time = time.time()
def set_rate_limit(self, min_interval: float) -> None:
"""
Set minimum interval between requests.
Args:
min_interval: Minimum seconds between requests
"""
self._min_request_interval = min_interval
self.logger.debug(f"Rate limit set to {min_interval} seconds")
def get_request_stats(self) -> Dict[str, Any]:
"""
Get request statistics.
Returns:
Dictionary with request statistics
"""
return {
'min_request_interval': self._min_request_interval,
'last_request_time': self._last_request_time,
'time_since_last_request': time.time() - self._last_request_time
}