Files
LEDMatrix/src/common/config_helper.py
T
ee59caa577 Follow-ups from #441: secret-helper migration, ten more bug fixes, and coverage for every remaining untested module (#444)
* refactor(web): use canonical secret helpers in api_v3; make ConfigManager secret strip/merge array-aware

api_v3.py carried three inline nested copies of find_secret_fields/
separate_secrets (main-config save, plugin-config save, plugin-config
reset). They drifted from each other (one lacked isinstance guards) and
none supported the canonical module's array-item secrets
(accounts[].token). All three endpoints now import from
src/web_interface/secret_helpers.

Adopting the canonical behavior makes array-item secrets reachable, and
their parallel-placeholder shape ([{'token': ...}, {}] alongside the
regular list) was not survivable by ConfigManager's round-trip:
_strip_secrets_recursive dropped the whole key (losing the regular
fields from config.json) and _deep_merge replaced the regular list
wholesale on load. Both are now array-aware:

- strip removes the secret fields from each item and ALWAYS keeps the
  list so indices survive for merge-on-load; whole-key secrets (scalar
  lists, shape mismatches) still drop the key entirely — never leak.
- merge folds each secrets item into the config item at the same index,
  skipping {} placeholders. The regular list's length is authoritative
  in both directions: a user deleting an array item never has it
  resurrected from a stale secrets entry (extras warn and are ignored).

api_v3's own deep_merge intentionally still replaces lists wholesale —
form posts carry complete arrays and index-merging would resurrect
deleted items; a comment now documents that.

Tests: the parity guard flips from 'exactly 3 inline copies' to 'zero,
and the canonical import must exist'; TestArraySecretStripAndMerge
covers the new strip/merge semantics incl. length-mismatch contracts;
new test_api_v3_secret_roundtrip.py drives all three endpoints through
a Flask client with a REAL ConfigManager+SchemaManager over tmp_path,
proving secrets land in config_secrets.json, config.json stays clean,
and a fresh load merges them back into the right array items.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: repair broken helper paths across display, cache, odds, logging, resolver, repos, config, validator

Nine fixes for bugs surfaced while writing coverage for previously
untested modules (plus the bool-duration quirk pinned in PR #441):

- base_plugin.get_display_duration: exclude bools from both numeric
  branches — display_duration=True no longer reads as a 1-second slot;
  it falls through to config, then the 15.0 default.
- display_helper: draw_error_message/draw_no_data_message called
  _draw_centered_text with the wrong arguments and crashed with
  AttributeError — both now delegate to draw_centered_text.
  draw_scorebug_layout drew status and clock at the same y, overprinting
  each other — they now share one combined top line.
  draw_ticker_layout drew its text starting at x=display_width (fully
  off-canvas), returning a blank frame every time — now draws at x=0;
  scroll_speed stays accepted-but-unused and is documented as such.
- api_helper.clear_cache guarded on a nonexistent CacheManager.clear()
  method, silently never clearing anything; it now uses the real surface
  (clear_cache/delete/list_cache_files) and no-ops safely otherwise.
- base_odds_manager._extract_espn_data raised AttributeError when ESPN
  sent explicit JSON nulls ("homeTeamOdds": null) — every level now
  null-safes with 'or {}'. format_odds_summary gated on
  is_odds_available, which deliberately ignores money lines, so
  ML-only odds formatted as "No odds available" — it now gates only on
  empty/no_odds data and formats money lines.
- logging_config.ContextualFormatter mutated record.msg in place, so a
  second handler prepended the context prefix twice; it now formats a
  copy. log_error hardcoded exc_info=True and raised TypeError when the
  caller passed exc_info — now kwargs.setdefault.
- dynamic_team_resolver wrote its "shared" class cache through self,
  creating instance shadows — the cache was per-instance and every
  scoreboard refetched rankings. Writes now go through the class.
- saved_repositories cleaned URLs with an unanchored .replace('.git','')
  that mangled URLs merely containing '.git' (my.github.io -> myhub.io);
  now strips only a trailing suffix. add/remove also roll back the
  in-memory list when the save fails, so memory always matches disk.
- config_helper.merge_configs shallow-copied the base, aliasing every
  un-overridden nested dict into the result — now deep-copies.
- startup_validator.validate_all accumulated errors/warnings across
  calls — now resets both lists per run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: cover the previously untested modules

Nine new suites plus an extension, asserting the Phase-1b fixed behavior
and pinning the quirks deliberately left alone:

- test_logging_config.py: formatters (JSON shape, no record mutation,
  single prefix through two handlers), PluginLoggerAdapter precedence,
  setup_logging handler hygiene and LEDMATRIX_DEBUG, log_error exc_info.
- test_startup_validator.py: exact messages, error-vs-warning split,
  accessor split (load_config vs get_config), cache-dir branches with
  os.access monkeypatched (root can write anything in CI), idempotence,
  raise_on_errors classification precedence.
- test_config_helper.py (full): load/save round trips, dot-notation
  get/set incl. silent-failure contract, post-fix no-aliasing merge,
  schema validation branches, the '{id}_config' key pin, default-enabled
  pin.
- test_saved_repositories.py: three load shapes, bare-list rewrite pin,
  trailing-only .git strip (my.github.io regression), save-failure
  rollback, type-classification case-sensitivity pin.
- test_api_helper.py: rate-limit math, cache-hit short circuit, ESPN
  URL/key formats, exact User-Agent guard, retry adapter, post-fix
  clear_cache against the real CacheManager surface, ttl-dropped pin.
- test_base_odds_manager.py: cache-key/URL construction, no_odds
  sentinel round trip, stale-cache fallback, null-safe extraction,
  ML-only formatting, is_odds_available truth table (ML-blind by
  contract), config key/attr mismatch pin.
- test_dynamic_team_resolver.py: expansion/dedup/slicing, dropped
  unknown-dynamic names (TOP_ substring hazard pinned), genuinely
  shared class cache (second instance: zero HTTP), TTL expiry,
  failure degradation without raising.
- test_display_helper.py (full): the fixed error/no-data renders,
  combined scorebug top line, non-blank ticker with scroll_speed
  no-op pin, composite upconversion, logo bleed positions, square
  orientation pin.
- test_skin_runtime_cache.py: discovery-cache hit/invalidation
  semantics (manifest mtime, .py edits pinned as non-invalidating),
  sys.modules namespacing contract incl. bare-name restore and stdlib
  shadowing, entry-module execute-once, API minor-version tolerance,
  skin_matches_target table.
- test_sports_capabilities.py (extended): _draw_celebration_layout
  executed for real (flash window, matrix-dims fallback, highlight
  alternation, logo-failure isolation), _should_celebrate_for direct,
  strict duration boundary, score_to_int edges, both-teams-score
  precedence, expired-coalesce refire, disabled-win baseline
  preservation, id-less prune.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: real schedule/dim coverage for DisplayController; fix two vacuous schedule tests

New test_display_controller_schedule.py drives _check_schedule and
_check_dim_schedule on a bare controller stub: same-day and
midnight-crossing windows with inclusive boundaries, global vs per-day vs
legacy-inferred modes (and dim's global-only default — no legacy
inference), per-day disabled days, invalid %H:%M fallbacks, unknown
timezone -> UTC, dim_brightness default 30, inactive-display short
circuit, and the _was_display_active/_was_dimmed transition flags.

test_display_controller.py's test_schedule_disabled and
test_active_hours patched config_service.get_config — which
_check_schedule never reads — so both asserted the init-default value
and could not fail. Rewritten on the test_inactive_hours pattern
(inject controller.config['schedule'], reset the minute gate, flip the
flag to the opposite state first so the assertion has teeth).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* ci: raise coverage floor to 48%

Measured 50% with the new suites in place (was 47% baseline when the
gate was introduced at 45); floor stays two points under measured.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: address CodeQL alert and review findings

- config_manager: the "secrets list longer than config list" warning now
  interpolates only config-side data (no key name or secrets-derived
  values), resolving the CodeQL clear-text-logging alert.
- base_plugin: validate_config rejects bool display_duration, matching
  get_display_duration (bool is an int subclass and would otherwise pass
  as a positive number).
- config_helper: merge_configs deep-copies override values in the
  non-recursive branch so mutating the merged result cannot reach back
  into override_config.
- saved_repositories: saves are atomic (temp file + fsync + os.replace),
  so a failed write can no longer truncate saved_repositories.json.
- tests: regression cases for each fix, plus a pin that whole-item
  array secrets (key[] + key[].field both marked) strip to empty {}
  skeletons — no secret values can reach config.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-07 16:17:11 -04:00

362 lines
12 KiB
Python

"""
Config Helper
Handles configuration management and validation for LED matrix plugins.
Extracted from LEDMatrix core to provide reusable functionality for plugins.
"""
import copy
import json
import logging
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
class ConfigHelper:
"""
Helper class for configuration management and validation.
Provides functionality for:
- Loading and saving configuration files
- Validating configuration against schemas
- Merging configurations
- Getting configuration values with defaults
- Configuration schema validation
"""
def __init__(self, logger: Optional[logging.Logger] = None):
"""
Initialize the ConfigHelper.
Args:
logger: Optional logger instance
"""
self.logger = logger or logging.getLogger(__name__)
def load_config(self, config_path: Union[str, Path]) -> Dict[str, Any]:
"""
Load configuration from a JSON file.
Args:
config_path: Path to configuration file
Returns:
Configuration dictionary
"""
config_path = Path(config_path)
try:
if not config_path.exists():
self.logger.warning(f"Configuration file not found: {config_path}")
return {}
with open(config_path, 'r', encoding='utf-8') as f:
config = json.load(f)
self.logger.debug(f"Loaded configuration from {config_path}")
return config
except json.JSONDecodeError as e:
self.logger.error(f"Invalid JSON in configuration file {config_path}: {e}")
return {}
except Exception as e:
self.logger.error(f"Error loading configuration from {config_path}: {e}")
return {}
def save_config(self, config: Dict[str, Any], config_path: Union[str, Path]) -> bool:
"""
Save configuration to a JSON file.
Args:
config: Configuration dictionary to save
config_path: Path to save configuration file
Returns:
True if successful, False otherwise
"""
config_path = Path(config_path)
try:
# Ensure directory exists
config_path.parent.mkdir(parents=True, exist_ok=True)
with open(config_path, 'w', encoding='utf-8') as f:
json.dump(config, f, indent=2, ensure_ascii=False)
self.logger.debug(f"Saved configuration to {config_path}")
return True
except Exception as e:
self.logger.error(f"Error saving configuration to {config_path}: {e}")
return False
def get_config_value(self, config: Dict[str, Any], key: str,
default: Any = None, required: bool = False) -> Any:
"""
Get a configuration value with optional default.
Args:
config: Configuration dictionary
key: Configuration key (supports dot notation like 'display.width')
default: Default value if key not found
required: If True, raise error if key not found
Returns:
Configuration value or default
"""
try:
# Support dot notation for nested keys
keys = key.split('.')
value = config
for k in keys:
if isinstance(value, dict) and k in value:
value = value[k]
else:
if required:
raise KeyError(f"Required configuration key not found: {key}")
return default
return value
except Exception as e:
if required:
raise
self.logger.warning(f"Error getting config value for {key}: {e}")
return default
def set_config_value(self, config: Dict[str, Any], key: str, value: Any) -> None:
"""
Set a configuration value.
Args:
config: Configuration dictionary to modify
key: Configuration key (supports dot notation)
value: Value to set
"""
try:
# Support dot notation for nested keys
keys = key.split('.')
current = config
# Navigate to parent of target key
for k in keys[:-1]:
if k not in current:
current[k] = {}
current = current[k]
# Set the value
current[keys[-1]] = value
except Exception as e:
self.logger.error(f"Error setting config value for {key}: {e}")
def merge_configs(self, base_config: Dict[str, Any],
override_config: Dict[str, Any]) -> Dict[str, Any]:
"""
Merge two configuration dictionaries.
Args:
base_config: Base configuration
override_config: Configuration to merge in (takes precedence)
Returns:
Merged configuration dictionary (fully independent of both
inputs — a shallow copy would alias un-overridden nested dicts,
so mutating the result would mutate the caller's base config).
"""
merged = copy.deepcopy(base_config)
for key, value in override_config.items():
if key in merged and isinstance(merged[key], dict) and isinstance(value, dict):
# Recursively merge nested dictionaries
merged[key] = self.merge_configs(merged[key], value)
else:
# Override with new value — deep-copied so mutating the
# merged result can't reach back into override_config.
merged[key] = copy.deepcopy(value)
return merged
def validate_config(self, config: Dict[str, Any],
schema: Optional[Dict[str, Any]] = None) -> bool:
"""
Validate configuration against a schema.
Args:
config: Configuration to validate
schema: Validation schema (optional)
Returns:
True if valid, False otherwise
"""
if schema is None:
# Basic validation - just check if it's a dictionary
return isinstance(config, dict)
try:
return self._validate_against_schema(config, schema)
except Exception as e:
self.logger.error(f"Configuration validation error: {e}")
return False
def get_plugin_config(self, config: Dict[str, Any], plugin_id: str) -> Dict[str, Any]:
"""
Get plugin-specific configuration.
Args:
config: Full configuration dictionary
plugin_id: Plugin identifier
Returns:
Plugin-specific configuration
"""
plugin_key = f"{plugin_id}_config"
return config.get(plugin_key, {})
def create_default_config(self, plugin_id: str,
default_values: Dict[str, Any]) -> Dict[str, Any]:
"""
Create a default configuration for a plugin.
Args:
plugin_id: Plugin identifier
default_values: Default configuration values
Returns:
Default configuration dictionary
"""
return {
f"{plugin_id}_config": default_values
}
def validate_required_keys(self, config: Dict[str, Any],
required_keys: List[str]) -> List[str]:
"""
Validate that required keys are present in configuration.
Args:
config: Configuration to validate
required_keys: List of required keys
Returns:
List of missing keys
"""
missing_keys = []
for key in required_keys:
if not self._has_key(config, key):
missing_keys.append(key)
return missing_keys
def get_display_config(self, config: Dict[str, Any]) -> Dict[str, Any]:
"""
Get display-related configuration.
Args:
config: Full configuration dictionary
Returns:
Display configuration
"""
return config.get('display', {})
def get_sports_config(self, config: Dict[str, Any], sport: str) -> Dict[str, Any]:
"""
Get sport-specific configuration.
Args:
config: Full configuration dictionary
sport: Sport name (e.g., 'basketball', 'football')
Returns:
Sport-specific configuration
"""
return config.get(f"{sport}_scoreboard", {})
def is_plugin_enabled(self, config: Dict[str, Any], plugin_id: str) -> bool:
"""
Check if a plugin is enabled.
Args:
config: Full configuration dictionary
plugin_id: Plugin identifier
Returns:
True if plugin is enabled
"""
plugin_config = self.get_plugin_config(config, plugin_id)
return plugin_config.get('enabled', True)
def get_favorite_teams(self, config: Dict[str, Any], sport: str) -> List[str]:
"""
Get favorite teams for a sport.
Args:
config: Full configuration dictionary
sport: Sport name
Returns:
List of favorite team abbreviations
"""
sport_config = self.get_sports_config(config, sport)
return sport_config.get('favorite_teams', [])
def get_display_modes(self, config: Dict[str, Any], sport: str) -> Dict[str, bool]:
"""
Get display modes for a sport.
Args:
config: Full configuration dictionary
sport: Sport name
Returns:
Dictionary of display modes and their enabled status
"""
sport_config = self.get_sports_config(config, sport)
return sport_config.get('display_modes', {})
def _validate_against_schema(self, config: Dict[str, Any],
schema: Dict[str, Any]) -> bool:
"""Validate configuration against a schema."""
# This is a simplified schema validation
# In a real implementation, you might use a library like jsonschema
for key, schema_info in schema.items():
if key not in config:
if schema_info.get('required', False):
self.logger.error(f"Missing required configuration key: {key}")
return False
continue
value = config[key]
expected_type = schema_info.get('type')
if expected_type and not isinstance(value, expected_type):
self.logger.error(f"Configuration key {key} has wrong type. Expected {expected_type}, got {type(value)}")
return False
# Validate allowed values
allowed_values = schema_info.get('allowed_values')
if allowed_values and value not in allowed_values:
self.logger.error(f"Configuration key {key} has invalid value: {value}. Allowed: {allowed_values}")
return False
return True
def _has_key(self, config: Dict[str, Any], key: str) -> bool:
"""Check if a key exists in configuration (supports dot notation)."""
try:
keys = key.split('.')
current = config
for k in keys:
if not isinstance(current, dict) or k not in current:
return False
current = current[k]
return True
except Exception:
return False