Follow-ups from #441: secret-helper migration, ten more bug fixes, and coverage for every remaining untested module (#444)

* refactor(web): use canonical secret helpers in api_v3; make ConfigManager secret strip/merge array-aware

api_v3.py carried three inline nested copies of find_secret_fields/
separate_secrets (main-config save, plugin-config save, plugin-config
reset). They drifted from each other (one lacked isinstance guards) and
none supported the canonical module's array-item secrets
(accounts[].token). All three endpoints now import from
src/web_interface/secret_helpers.

Adopting the canonical behavior makes array-item secrets reachable, and
their parallel-placeholder shape ([{'token': ...}, {}] alongside the
regular list) was not survivable by ConfigManager's round-trip:
_strip_secrets_recursive dropped the whole key (losing the regular
fields from config.json) and _deep_merge replaced the regular list
wholesale on load. Both are now array-aware:

- strip removes the secret fields from each item and ALWAYS keeps the
  list so indices survive for merge-on-load; whole-key secrets (scalar
  lists, shape mismatches) still drop the key entirely — never leak.
- merge folds each secrets item into the config item at the same index,
  skipping {} placeholders. The regular list's length is authoritative
  in both directions: a user deleting an array item never has it
  resurrected from a stale secrets entry (extras warn and are ignored).

api_v3's own deep_merge intentionally still replaces lists wholesale —
form posts carry complete arrays and index-merging would resurrect
deleted items; a comment now documents that.

Tests: the parity guard flips from 'exactly 3 inline copies' to 'zero,
and the canonical import must exist'; TestArraySecretStripAndMerge
covers the new strip/merge semantics incl. length-mismatch contracts;
new test_api_v3_secret_roundtrip.py drives all three endpoints through
a Flask client with a REAL ConfigManager+SchemaManager over tmp_path,
proving secrets land in config_secrets.json, config.json stays clean,
and a fresh load merges them back into the right array items.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: repair broken helper paths across display, cache, odds, logging, resolver, repos, config, validator

Nine fixes for bugs surfaced while writing coverage for previously
untested modules (plus the bool-duration quirk pinned in PR #441):

- base_plugin.get_display_duration: exclude bools from both numeric
  branches — display_duration=True no longer reads as a 1-second slot;
  it falls through to config, then the 15.0 default.
- display_helper: draw_error_message/draw_no_data_message called
  _draw_centered_text with the wrong arguments and crashed with
  AttributeError — both now delegate to draw_centered_text.
  draw_scorebug_layout drew status and clock at the same y, overprinting
  each other — they now share one combined top line.
  draw_ticker_layout drew its text starting at x=display_width (fully
  off-canvas), returning a blank frame every time — now draws at x=0;
  scroll_speed stays accepted-but-unused and is documented as such.
- api_helper.clear_cache guarded on a nonexistent CacheManager.clear()
  method, silently never clearing anything; it now uses the real surface
  (clear_cache/delete/list_cache_files) and no-ops safely otherwise.
- base_odds_manager._extract_espn_data raised AttributeError when ESPN
  sent explicit JSON nulls ("homeTeamOdds": null) — every level now
  null-safes with 'or {}'. format_odds_summary gated on
  is_odds_available, which deliberately ignores money lines, so
  ML-only odds formatted as "No odds available" — it now gates only on
  empty/no_odds data and formats money lines.
- logging_config.ContextualFormatter mutated record.msg in place, so a
  second handler prepended the context prefix twice; it now formats a
  copy. log_error hardcoded exc_info=True and raised TypeError when the
  caller passed exc_info — now kwargs.setdefault.
- dynamic_team_resolver wrote its "shared" class cache through self,
  creating instance shadows — the cache was per-instance and every
  scoreboard refetched rankings. Writes now go through the class.
- saved_repositories cleaned URLs with an unanchored .replace('.git','')
  that mangled URLs merely containing '.git' (my.github.io -> myhub.io);
  now strips only a trailing suffix. add/remove also roll back the
  in-memory list when the save fails, so memory always matches disk.
- config_helper.merge_configs shallow-copied the base, aliasing every
  un-overridden nested dict into the result — now deep-copies.
- startup_validator.validate_all accumulated errors/warnings across
  calls — now resets both lists per run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: cover the previously untested modules

Nine new suites plus an extension, asserting the Phase-1b fixed behavior
and pinning the quirks deliberately left alone:

- test_logging_config.py: formatters (JSON shape, no record mutation,
  single prefix through two handlers), PluginLoggerAdapter precedence,
  setup_logging handler hygiene and LEDMATRIX_DEBUG, log_error exc_info.
- test_startup_validator.py: exact messages, error-vs-warning split,
  accessor split (load_config vs get_config), cache-dir branches with
  os.access monkeypatched (root can write anything in CI), idempotence,
  raise_on_errors classification precedence.
- test_config_helper.py (full): load/save round trips, dot-notation
  get/set incl. silent-failure contract, post-fix no-aliasing merge,
  schema validation branches, the '{id}_config' key pin, default-enabled
  pin.
- test_saved_repositories.py: three load shapes, bare-list rewrite pin,
  trailing-only .git strip (my.github.io regression), save-failure
  rollback, type-classification case-sensitivity pin.
- test_api_helper.py: rate-limit math, cache-hit short circuit, ESPN
  URL/key formats, exact User-Agent guard, retry adapter, post-fix
  clear_cache against the real CacheManager surface, ttl-dropped pin.
- test_base_odds_manager.py: cache-key/URL construction, no_odds
  sentinel round trip, stale-cache fallback, null-safe extraction,
  ML-only formatting, is_odds_available truth table (ML-blind by
  contract), config key/attr mismatch pin.
- test_dynamic_team_resolver.py: expansion/dedup/slicing, dropped
  unknown-dynamic names (TOP_ substring hazard pinned), genuinely
  shared class cache (second instance: zero HTTP), TTL expiry,
  failure degradation without raising.
- test_display_helper.py (full): the fixed error/no-data renders,
  combined scorebug top line, non-blank ticker with scroll_speed
  no-op pin, composite upconversion, logo bleed positions, square
  orientation pin.
- test_skin_runtime_cache.py: discovery-cache hit/invalidation
  semantics (manifest mtime, .py edits pinned as non-invalidating),
  sys.modules namespacing contract incl. bare-name restore and stdlib
  shadowing, entry-module execute-once, API minor-version tolerance,
  skin_matches_target table.
- test_sports_capabilities.py (extended): _draw_celebration_layout
  executed for real (flash window, matrix-dims fallback, highlight
  alternation, logo-failure isolation), _should_celebrate_for direct,
  strict duration boundary, score_to_int edges, both-teams-score
  precedence, expired-coalesce refire, disabled-win baseline
  preservation, id-less prune.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: real schedule/dim coverage for DisplayController; fix two vacuous schedule tests

New test_display_controller_schedule.py drives _check_schedule and
_check_dim_schedule on a bare controller stub: same-day and
midnight-crossing windows with inclusive boundaries, global vs per-day vs
legacy-inferred modes (and dim's global-only default — no legacy
inference), per-day disabled days, invalid %H:%M fallbacks, unknown
timezone -> UTC, dim_brightness default 30, inactive-display short
circuit, and the _was_display_active/_was_dimmed transition flags.

test_display_controller.py's test_schedule_disabled and
test_active_hours patched config_service.get_config — which
_check_schedule never reads — so both asserted the init-default value
and could not fail. Rewritten on the test_inactive_hours pattern
(inject controller.config['schedule'], reset the minute gate, flip the
flag to the opposite state first so the assertion has teeth).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* ci: raise coverage floor to 48%

Measured 50% with the new suites in place (was 47% baseline when the
gate was introduced at 45); floor stays two points under measured.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: address CodeQL alert and review findings

- config_manager: the "secrets list longer than config list" warning now
  interpolates only config-side data (no key name or secrets-derived
  values), resolving the CodeQL clear-text-logging alert.
- base_plugin: validate_config rejects bool display_duration, matching
  get_display_duration (bool is an int subclass and would otherwise pass
  as a positive number).
- config_helper: merge_configs deep-copies override values in the
  non-recursive branch so mutating the merged result cannot reach back
  into override_config.
- saved_repositories: saves are atomic (temp file + fsync + os.replace),
  so a failed write can no longer truncate saved_repositories.json.
- tests: regression cases for each fix, plus a pin that whole-item
  array secrets (key[] + key[].field both marked) strip to empty {}
  skeletons — no secret values can reach config.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-08-07 16:17:11 -04:00
committed by GitHub
co-authored by Claude Fable 5
parent fc25a70d75
commit ee59caa577
28 changed files with 3875 additions and 292 deletions
@@ -0,0 +1,259 @@
"""
End-to-end secret round-trips through the three api_v3 endpoints that
separate secrets from regular config (main-config save, plugin-config save,
plugin-config reset) — now backed by the canonical
src/web_interface/secret_helpers implementations.
Unlike test_web_api.py (which mocks the config manager), these tests run a
REAL ConfigManager and a REAL SchemaManager over tmp_path files, so they
prove the whole chain: endpoint separation -> config_secrets.json write ->
atomic config.json save (strip) -> load_config (merge back), including the
array-item secret shape (accounts[].token) the inline copies never
supported.
"""
import json
import sys
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from flask import Flask
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from src.config_manager import ConfigManager # noqa: E402
from src.plugin_system.schema_manager import SchemaManager # noqa: E402
from web_interface.blueprints.api_v3 import api_v3 # noqa: E402
PLUGIN_ID = "testplugin"
SCHEMA = {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"enabled": {"type": "boolean", "default": True},
"display_duration": {"type": "number", "default": 15},
"api_key": {"type": "string", "x-secret": True, "default": ""},
"city": {"type": "string", "default": "Austin"},
"accounts": {
"type": "array",
"default": [],
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"token": {"type": "string", "x-secret": True},
},
},
},
},
}
@pytest.fixture
def env(tmp_path):
"""Real ConfigManager + SchemaManager over tmp_path, wired onto the
api_v3 blueprint with the remaining managers mocked."""
config_file = tmp_path / "config.json"
config_file.write_text("{}")
plugins_dir = tmp_path / "plugins"
plugin_dir = plugins_dir / PLUGIN_ID
plugin_dir.mkdir(parents=True)
(plugin_dir / "config_schema.json").write_text(json.dumps(SCHEMA))
(plugin_dir / "manifest.json").write_text(json.dumps({
"id": PLUGIN_ID, "name": "Test Plugin", "version": "1.0.0",
}))
config_manager = ConfigManager(
config_path=str(config_file),
secrets_path=str(tmp_path / "config_secrets.json"))
config_manager.template_path = str(tmp_path / "no-template.json")
schema_manager = SchemaManager(plugins_dir=plugins_dir,
project_root=tmp_path)
plugin_manager = MagicMock()
plugin_manager.plugin_manifests = {PLUGIN_ID: {"id": PLUGIN_ID}}
plugin_manager.plugins_dir = plugins_dir
plugin_manager.get_plugin.return_value = None
api_v3.config_manager = config_manager
api_v3.schema_manager = schema_manager
api_v3.plugin_manager = plugin_manager
api_v3.plugin_store_manager = MagicMock()
api_v3.saved_repositories_manager = MagicMock()
api_v3.operation_queue = MagicMock()
api_v3.plugin_state_manager = MagicMock()
api_v3.operation_history = MagicMock()
api_v3.cache_manager = MagicMock()
app = Flask(__name__)
app.config["TESTING"] = True
app.register_blueprint(api_v3, url_prefix="/api/v3")
class Env:
pass
e = Env()
e.client = app.test_client()
e.config_manager = config_manager
e.config_file = config_file
e.secrets_file = tmp_path / "config_secrets.json"
e.tmp_path = tmp_path
def fresh_load():
"""Load via a NEW ConfigManager, as the next request/process would.
The endpoint's manager serves its post-save in-memory config via the
mtime fast path, and that copy predates the secrets it just
separated out — a pre-existing quirk that applies to scalar secrets
too. On-disk truth is what these tests care about.
"""
fresh = ConfigManager(config_path=str(config_file),
secrets_path=str(e.secrets_file))
fresh.template_path = str(tmp_path / "no-template.json")
return fresh.load_config()
e.fresh_load = fresh_load
return e
def _on_disk(path):
return json.loads(path.read_text())
class TestSaveMainConfig:
"""Site A: POST /config/main with a plugin-id key."""
def test_array_and_scalar_secrets_routed_to_secrets_file(self, env):
resp = env.client.post("/api/v3/config/main", json={
PLUGIN_ID: {
"city": "Dallas",
"api_key": "s3cret-key",
"accounts": [
{"name": "a", "token": "s3cret-a"},
{"name": "b"},
],
},
})
assert resp.status_code == 200, resp.get_json()
on_disk = _on_disk(env.config_file)
assert on_disk[PLUGIN_ID]["city"] == "Dallas"
assert "api_key" not in on_disk[PLUGIN_ID]
assert on_disk[PLUGIN_ID]["accounts"] == [{"name": "a"}, {"name": "b"}]
assert "s3cret" not in env.config_file.read_text()
secrets = _on_disk(env.secrets_file)
assert secrets[PLUGIN_ID]["api_key"] == "s3cret-key"
assert secrets[PLUGIN_ID]["accounts"] == [{"token": "s3cret-a"}, {}]
def test_load_config_merges_secrets_back(self, env):
env.client.post("/api/v3/config/main", json={
PLUGIN_ID: {"accounts": [{"name": "a", "token": "s3cret-a"}]},
})
merged = env.fresh_load()
assert merged[PLUGIN_ID]["accounts"] == [
{"name": "a", "token": "s3cret-a"}]
class TestSavePluginConfig:
"""Site B: POST /plugins/config (JSON body)."""
def _save(self, env, config):
return env.client.post("/api/v3/plugins/config", json={
"plugin_id": PLUGIN_ID, "config": config,
})
def test_round_trip_with_array_secrets(self, env):
resp = self._save(env, {
"enabled": True,
"city": "Houston",
"api_key": "s3cret-key",
"accounts": [
{"name": "a", "token": "s3cret-a"},
{"name": "b", "token": "s3cret-b"},
],
})
assert resp.status_code == 200, resp.get_json()
assert "s3cret" not in env.config_file.read_text()
on_disk = _on_disk(env.config_file)
assert on_disk[PLUGIN_ID]["accounts"] == [{"name": "a"}, {"name": "b"}]
secrets = _on_disk(env.secrets_file)
assert secrets[PLUGIN_ID]["accounts"] == [
{"token": "s3cret-a"}, {"token": "s3cret-b"}]
merged = env.fresh_load()
assert merged[PLUGIN_ID]["accounts"][1]["token"] == "s3cret-b"
def test_secret_count_message_counts_top_level_keys(self, env):
# Pinned: the "(N secret field(s))" message counts TOP-LEVEL keys of
# the separated secrets dict. Here that is 2: the posted accounts
# array (all its item tokens count as ONE key) plus the schema's
# api_key default ("") that merge_with_defaults adds before
# separation.
resp = self._save(env, {
"accounts": [{"name": "a", "token": "t"}],
})
message = resp.get_json()["message"]
assert "(2 secret field(s) saved to config_secrets.json)" in message
def test_resave_replaces_stored_secrets_list_wholesale(self, env):
# Characterized: api_v3's deep_merge intentionally replaces lists,
# so a re-save's parallel secrets list is authoritative.
self._save(env, {"accounts": [
{"name": "a", "token": "old-a"},
{"name": "b", "token": "old-b"},
]})
self._save(env, {"accounts": [{"name": "only", "token": "new-only"}]})
secrets = _on_disk(env.secrets_file)
assert secrets[PLUGIN_ID]["accounts"] == [{"token": "new-only"}]
merged = env.fresh_load()
assert merged[PLUGIN_ID]["accounts"] == [
{"name": "only", "token": "new-only"}]
class TestResetPluginConfig:
"""Site C: POST /plugins/config/reset."""
def _seed(self, env):
env.client.post("/api/v3/plugins/config", json={
"plugin_id": PLUGIN_ID,
"config": {"city": "Houston", "api_key": "s3cret-key",
"accounts": [{"name": "a", "token": "s3cret-a"}]},
})
def test_reset_preserving_secrets(self, env):
self._seed(env)
resp = env.client.post("/api/v3/plugins/config/reset", json={
"plugin_id": PLUGIN_ID, "preserve_secrets": True,
})
assert resp.status_code == 200, resp.get_json()
on_disk = _on_disk(env.config_file)
assert on_disk[PLUGIN_ID]["city"] == "Austin" # schema default
assert on_disk[PLUGIN_ID]["accounts"] == [] # schema default
# Existing secrets survive (top-level-only preserve merge, pinned).
secrets = _on_disk(env.secrets_file)
assert secrets[PLUGIN_ID]["api_key"] == "s3cret-key"
assert secrets[PLUGIN_ID]["accounts"] == [{"token": "s3cret-a"}]
def test_reset_without_preserving_secrets(self, env):
self._seed(env)
resp = env.client.post("/api/v3/plugins/config/reset", json={
"plugin_id": PLUGIN_ID, "preserve_secrets": False,
})
assert resp.status_code == 200, resp.get_json()
secrets = _on_disk(env.secrets_file)
# Replaced with schema-default secrets — the schema declares no
# secret defaults, so the plugin's secrets are emptied.
assert secrets[PLUGIN_ID] in ({}, {"api_key": ""})
@@ -1,18 +1,13 @@
"""
Drift guard for the duplicated secret-separation logic.
Drift guard: api_v3 must use the canonical secret helpers.
src/web_interface/secret_helpers.py is the canonical implementation of
find_secret_fields / separate_secrets, but web_interface/blueprints/api_v3.py
still carries THREE inline nested-function copies of each (in the plugin
config GET, POST, and reset endpoints). The copies lack the canonical
module's array-item support (`accounts[].token`), so migrating an endpoint
onto the module is a behavior change that must be made deliberately.
This file guards two things:
1. The copy count can only go DOWN. A fourth copy appearing means someone
re-implemented the logic again instead of importing secret_helpers.
2. The known behavioral gap is documented as an executable fact, so whoever
migrates the endpoints knows exactly what changes.
Historically web_interface/blueprints/api_v3.py carried THREE inline
nested-function copies of ``find_secret_fields``/``separate_secrets`` (in the
main-config save, plugin-config save, and plugin-config reset endpoints).
They lacked the canonical module's array-item secret support and drifted from
each other. They have been migrated onto
``src/web_interface/secret_helpers`` — this file now guards against copies
REAPPEARING, and keeps the canonical array-item behavior executable.
"""
import re
@@ -23,77 +18,44 @@ from src.web_interface.secret_helpers import find_secret_fields, separate_secret
API_V3_PATH = (Path(__file__).resolve().parents[2]
/ "web_interface" / "blueprints" / "api_v3.py")
# Update DOWNWARD as endpoints migrate onto src/web_interface/secret_helpers.
EXPECTED_INLINE_COPIES = 3
# The migration is complete: any inline reimplementation is a regression.
EXPECTED_INLINE_COPIES = 0
class TestInlineCopyCount:
class TestNoInlineCopies:
def _count(self, name: str) -> int:
source = API_V3_PATH.read_text(encoding="utf-8")
return len(re.findall(rf"^\s*def {name}\(", source, flags=re.MULTILINE))
def test_find_secret_fields_copy_count(self):
def test_no_inline_find_secret_fields(self):
count = self._count("find_secret_fields")
assert count == EXPECTED_INLINE_COPIES, (
f"api_v3.py has {count} inline find_secret_fields definitions, "
f"expected {EXPECTED_INLINE_COPIES}. New code must import it from "
f"src/web_interface/secret_helpers instead of re-implementing it; "
f"if you migrated an endpoint, lower EXPECTED_INLINE_COPIES."
f"expected {EXPECTED_INLINE_COPIES}. Import it from "
f"src/web_interface/secret_helpers instead of re-implementing it."
)
def test_separate_secrets_copy_count(self):
def test_no_inline_separate_secrets(self):
count = self._count("separate_secrets")
assert count == EXPECTED_INLINE_COPIES, (
f"api_v3.py has {count} inline separate_secrets definitions, "
f"expected {EXPECTED_INLINE_COPIES}. New code must import it from "
f"src/web_interface/secret_helpers instead of re-implementing it; "
f"if you migrated an endpoint, lower EXPECTED_INLINE_COPIES."
f"expected {EXPECTED_INLINE_COPIES}. Import it from "
f"src/web_interface/secret_helpers instead of re-implementing it."
)
def test_inline_copies_lack_array_item_support(self):
"""The documented gap: no inline copy recurses into array `items`
schemas, so array-item secrets (accounts[].token) are NOT routed to
config_secrets.json by these endpoints. The canonical module handles
them. When an endpoint migrates onto the module that behavior
changes (a fix, but a deliberate one).
If this fails, an inline copy has grown array support — duplicating
the canonical module even harder. Migrate the endpoint onto
src/web_interface/secret_helpers instead.
"""
for body in self._inline_bodies("find_secret_fields"):
# Array handling requires checking type == 'array'; no inline
# copy does. (Can't grep bare "items" — properties.items() the
# dict method appears legitimately.)
assert "'array'" not in body and '"array"' not in body
@staticmethod
def _inline_bodies(name: str):
"""Extract each inline def's body from api_v3.py by indentation."""
lines = API_V3_PATH.read_text(encoding="utf-8").splitlines()
bodies = []
i = 0
while i < len(lines):
match = re.match(rf"^(\s+)def {name}\(", lines[i])
if not match:
i += 1
continue
indent = len(match.group(1))
body = [lines[i]]
i += 1
while i < len(lines):
line = lines[i]
if line.strip() and (len(line) - len(line.lstrip())) <= indent:
break
body.append(line)
i += 1
bodies.append("\n".join(body))
assert bodies, f"no inline {name} definitions found"
return bodies
def test_canonical_import_present(self):
# Tripwire: the endpoints still need the helpers, so removing the
# import means either dead secret handling or a new local copy.
source = API_V3_PATH.read_text(encoding="utf-8")
assert re.search(
r"from src\.web_interface\.secret_helpers import .*find_secret_fields",
source,
), "api_v3.py no longer imports the canonical secret helpers"
class TestCanonicalArrayItemBehavior:
"""Executable documentation of what migrating endpoints will change."""
"""Executable documentation of the array-item secret contract the
endpoints now inherit from the canonical module."""
SCHEMA = {
"accounts": {