Files
LEDMatrix/test/test_config_manager_secrets.py
T
d6c5f97c13 Test suite overhaul + fixes for the three bugs it uncovered (#441)
* ci: run the whole test tree and make the plugin-safety job assert something real

The unit-tests CI job ran an explicit 24-file allowlist that had rotted:
63 of 90 test files (display, vegas, store manager, web API, web_interface)
never ran on a PR. The job now runs all of test/ (minus test/plugins, which
the plugin-safety job owns) so new test files are enrolled by default and
any exclusion needs a visible, commented --ignore.

The plugin-safety job was a green no-op: plugins/ is empty in CI, so every
test skipped with 'Manifest not found'. It now renders a bundled
deterministic fixture plugin (test/fixtures/plugins/ci-fixture-plugin,
golden images included for all 8 default sizes) via LEDMATRIX_PLUGINS_DIR,
and sets LEDMATRIX_REQUIRE_PLUGINS=1 so discovering zero plugins fails
loudly instead of skipping green. The per-plugin suites document that they
target dev machines with real plugins installed.

Coverage is now measured and enforced in exactly one place — the CI
unit-tests step (--cov=src --cov=web_interface --cov-fail-under=45, from a
measured 47% baseline). pytest.ini previously declared --cov-fail-under=30
but CI always passed --no-cov, so the gate had never run anywhere; local
pytest is now coverage-free and fast.

Enabling the 63 unenrolled files surfaced three cases of test rot, fixed
here: test_display_controller_vegas_tick.py could not collect without the
hardware rgbmatrix module (now uses the emulator convention), the
state-reconciliation unrecoverable-cache tests broke when production added
the is_plugin_uninstalled tombstone check (bare Mock returned truthy),
and test_get_system_status assumed the optional psutil dependency
(now installed via requirements-test.txt and guarded by importorskip).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: replace can't-fail tests with real assertions

test_font_manager.py was 5 of 6 tests shaped as 'try: call(); assert True /
except: assert True' — running in CI while unable to fail on any
regression. Rewritten against the real FontManager API and the bundled
assets/fonts: returned font types, cache-hit identity, distinct entries per
size, default-font fallback for unknown families and corrupt files
(recorded in failed_loads), BDF native-size reading, text measurement, and
cache lifecycle.

test_display_manager.py's test_draw_text ended in 'assert True'; it now
renders onto a known-black canvas and asserts pixels were actually lit —
which required un-breaking the fixture's freetype MagicMock so draw_text's
isinstance check doesn't silently swallow the draw.

test_display_controller.py carried a permanently-skipped test whose skip
reason already declared it redundant; deleted.

Both display test files now set EMULATOR=true before importing
display_manager (the same convention as test_display_dirty_tracking.py) so
they collect standalone instead of depending on which test module imports
display_manager first.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: cover the untested fragile logic (compatibility gate, secrets, config merges, durations, skin cards)

New unit tests for pure or filesystem-only logic that previously had zero
direct coverage:

- test_compatibility.py: the semver install gate (parse_semver suffix
  handling, every range operator, TRUSTWORTHY_FLOOR behavior for cores
  reporting untrustworthy versions, 'more restrictive wins', and the
  malformed-manifest shapes that used to raise).
- test/web_interface/test_secret_helpers.py: the canonical x-secret
  helpers — find/separate/mask/remove, array-item secrets, no input
  mutation, and a separate->recombine round-trip.
- test/web_interface/test_api_v3_helpers.py: the module-level helpers
  behind the plugin config save endpoint (_is_plugin_update_available,
  _coerce_to_bool including the int==1 quirk, deep_merge including its
  shared-subtree shallowness, _parse_form_value, dotted-key-aware
  _get_schema_property/_set_nested_value).
- test_base_plugin_duration.py: get_display_duration's full coercion
  ladder (instance attr -> config -> 15.0), including the bool-is-int
  quirk where display_duration=True means one second.
- test_config_manager_secrets.py: the secrets round-trip — deep-merge on
  load, strip on save, group pruning, the load fast path — and two
  characterized sharp edges marked SUSPECTED BUG: an unreadable secrets
  file at save time writes secrets into config.json in plaintext, and a
  same-mtime-same-size content swap is served stale.
- test_schema_manager_merge.py: merge_with_defaults branch behavior (None
  replacement vs falsey preservation, dict-vs-scalar mismatches, arrays
  replaced wholesale, defaults never mutated).
- test_skin_system.py (extended): render_skin_card shares _render_game's
  3-strike counter but never resets it on success — the asymmetry is
  pinned in both directions, along with card fallthrough and the disable
  interaction between the two paths.

Suspected bugs are characterized, not fixed — each carries a comment so a
future behavior change is deliberate rather than accidental.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: add drift guards for cross-file contracts

Three guard suites that pin contracts spanning multiple files, where one
side changing unilaterally breaks the other silently:

- test_version_comparison_consistency.py: the repo's four version
  comparators (compatibility.parse_semver, api_v3's packaging-based
  _is_plugin_update_available, store_manager update_plugin's raw string
  equality, skin_runtime._major) answer differently on the same inputs.
  A table pins each one's verdict; update_plugin is driven through its
  real code path to show the SUSPECTED BUGs: 'v1.2.0' vs '1.2.0'
  triggers a full reinstall the UI calls unnecessary, and a locally-ahead
  plugin gets downgraded. A pairwise-ordering check keeps parse_semver
  agreeing with packaging on plain X.Y.Z.
- test/web_interface/test_secret_separation_parity.py: api_v3.py carries
  three inline copies of find_secret_fields/separate_secrets that lack
  the canonical module's array-item support. The copy count is asserted
  exact (it may only go down; new copies must import
  src/web_interface/secret_helpers), the missing-array-support gap is
  asserted so it can't grow silently, and the canonical behavior that
  migration will adopt is documented executably.
- test_discovery_path_contract.py: the three 'where is plugin X'
  resolvers (PluginManager discovery, StoreManager._find_plugin_path,
  SchemaManager.get_schema_path) agree on the configured directory, and
  their divergent fallback chains are characterized. Also pins the
  .standalone-backup- naming contract shared by store rollback and
  discovery, and _resolve_skin_target's path-traversal rejection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* test: address review feedback — fixture lifecycle, test names, ClassVar

- ci-fixture-plugin: call display_manager.clear() before rendering (per
  plugin guidelines — the fixture should model a well-behaved plugin),
  add a class docstring, and document why Pillow is deliberately not
  pinned in its requirements.txt (core dependency; harness installs
  nothing).
- Rename two tests whose names contradicted their assertions:
  test_unparseable_core_version_is_compatible ->
  test_unparseable_core_with_high_floor_is_blocked, and
  test_unreadable_secrets_file... -> test_corrupt_secrets_file...
- Annotate TestGetSchemaProperty.SCHEMA as ClassVar (RUF012).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* ci: allow manual test.yml runs via workflow_dispatch

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: unify version comparison, refuse secret-leaking saves, reset skin strikes on card success

Fixes the three suspected bugs this PR's characterization tests pinned,
flipping those tests to assert the corrected behavior:

- plugins/store: ONE shared update comparator. New
  compatibility.is_update_available() (PEP 440 via packaging) is now used
  by both the web UI's update badge (api_v3._is_plugin_update_available
  is a thin alias) and store_manager.update_plugin's reinstall decision.
  Previously update_plugin used raw string equality: 'v1.2.0' vs '1.2.0'
  triggered a full reinstall the UI called unnecessary, and a locally-
  ahead plugin (2.0.0 installed, registry 1.9.0) was silently DOWNGRADED.
  Now equivalent spellings skip the reinstall and locally-ahead versions
  are never downgraded; unparseable versions still reconcile by
  reinstalling from the registry.

- config: save_config and save_config_atomic now refuse (ConfigError)
  when config_secrets.json exists but cannot be loaded. Both previously
  proceeded without stripping, writing the merged secrets into
  config.json in plaintext. The shared _load_secrets_for_save() helper
  raises with an actionable message instead; a missing secrets file is
  still fine (nothing to strip), and _migrate_config's catch-all keeps
  boot resilient.

- skins: render_skin_card resets _skin_failures on both success paths
  (vegas card returned, or mode renderer handled), mirroring
  _render_game. Transient card failures no longer accumulate across a
  session until they permanently disable a working skin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

* fix: harden shared comparator edges from review

- is_update_available: reject truthy non-string versions (a malformed
  manifest can carry a number; packaging raises TypeError on those) by
  surfacing the mismatch instead of raising.
- store_manager.update_plugin: drop the truthiness gate around the
  comparator so a missing version on either side follows the shared
  'no update' verdict, keeping the store consistent with the UI badge;
  a missing manifest still uses the reinstall recovery path.
- config_manager._load_secrets_for_save: catch only expected read/parse
  failures (OSError/ValueError/RecursionError) so implementation bugs
  propagate as themselves, and log with traceback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-07 10:17:30 -04:00

180 lines
7.5 KiB
Python

"""
Tests for the ConfigManager secrets round-trip and the load_config fast path.
The contract under test: config_secrets.json values are deep-merged INTO the
in-memory config at load time, and stripped back OUT before anything is
written to config.json — so secrets live in exactly one file on disk. This
suite pins that round-trip plus its sharp edges, including the guard that a
save REFUSES (ConfigError) when the secrets file exists but can't be loaded,
rather than leaking merged secrets into config.json in plaintext.
Complements test_config_manager.py, which covers loading/migration/validation.
"""
import json
import os
import pytest
from src.config_manager import ConfigManager
from src.exceptions import ConfigError
def make_manager(tmp_path, config=None, secrets=None):
"""A ConfigManager over tmp_path files, template migration neutralized."""
config_file = tmp_path / "config.json"
secrets_file = tmp_path / "config_secrets.json"
config_file.write_text(json.dumps(config if config is not None else {}))
if secrets is not None:
secrets_file.write_text(json.dumps(secrets))
manager = ConfigManager(config_path=str(config_file),
secrets_path=str(secrets_file))
# Point the (CWD-relative) template at nothing so migration never runs —
# these tests assert exact on-disk contents.
manager.template_path = str(tmp_path / "no-template.json")
return manager
class TestLoadMergesSecrets:
def test_secrets_deep_merged_into_config(self, tmp_path):
manager = make_manager(
tmp_path,
config={"weather": {"city": "Austin"}, "timezone": "UTC"},
secrets={"weather": {"api_key": "s3cret"}},
)
loaded = manager.load_config()
assert loaded["weather"] == {"city": "Austin", "api_key": "s3cret"}
assert loaded["timezone"] == "UTC"
def test_secret_scalar_overrides_config_value(self, tmp_path):
manager = make_manager(
tmp_path,
config={"weather": {"api_key": "YOUR_API_KEY"}},
secrets={"weather": {"api_key": "real-key"}},
)
assert manager.load_config()["weather"]["api_key"] == "real-key"
def test_missing_secrets_file_loads_config_fine(self, tmp_path):
manager = make_manager(tmp_path, config={"timezone": "UTC"})
assert manager.load_config() == {"timezone": "UTC"}
def test_corrupt_secrets_file_loads_config_without_secrets(self, tmp_path):
manager = make_manager(tmp_path, config={"timezone": "UTC"})
(tmp_path / "config_secrets.json").write_text("{not json")
loaded = manager.load_config()
assert loaded["timezone"] == "UTC"
class TestSaveStripsSecrets:
def test_round_trip_keeps_secrets_out_of_config_json(self, tmp_path):
manager = make_manager(
tmp_path,
config={"weather": {"city": "Austin"}},
secrets={"weather": {"api_key": "s3cret"}},
)
loaded = manager.load_config()
assert loaded["weather"]["api_key"] == "s3cret" # merged in memory
manager.save_config(loaded)
on_disk = json.loads((tmp_path / "config.json").read_text())
assert "api_key" not in on_disk.get("weather", {})
assert on_disk["weather"]["city"] == "Austin"
# In-memory config still carries the secret for runtime use.
assert manager.config["weather"]["api_key"] == "s3cret"
def test_group_dropped_when_only_secrets_remain(self, tmp_path):
# _strip_secrets_recursive drops a group entirely when nothing
# non-secret is left in it.
manager = make_manager(
tmp_path,
config={},
secrets={"weather": {"api_key": "s3cret"}},
)
manager.save_config({"weather": {"api_key": "s3cret"}, "timezone": "UTC"})
on_disk = json.loads((tmp_path / "config.json").read_text())
assert on_disk == {"timezone": "UTC"}
def test_scalar_secret_key_stripped_at_top_level(self, tmp_path):
manager = make_manager(tmp_path, config={}, secrets={"token": "t"})
manager.save_config({"token": "t", "timezone": "UTC"})
on_disk = json.loads((tmp_path / "config.json").read_text())
assert on_disk == {"timezone": "UTC"}
def test_corrupt_secrets_file_refuses_save_no_plaintext_leak(self, tmp_path):
# Regression guard: when the secrets file exists but is corrupt at
# save time, stripping is impossible — the save must raise instead of
# writing the merged secrets into config.json in plaintext (the
# historical behavior).
manager = make_manager(
tmp_path,
config={"weather": {"city": "Austin"}},
secrets={"weather": {"api_key": "s3cret"}},
)
loaded = manager.load_config()
(tmp_path / "config_secrets.json").write_text("{corrupt")
with pytest.raises(ConfigError):
manager.save_config(loaded)
# On-disk config untouched: no secret leaked.
on_disk = json.loads((tmp_path / "config.json").read_text())
assert "api_key" not in on_disk.get("weather", {})
def test_corrupt_secrets_file_refuses_atomic_save_too(self, tmp_path):
# Same refusal on the atomic save path, which shared the leak.
manager = make_manager(
tmp_path,
config={"weather": {"city": "Austin"}},
secrets={"weather": {"api_key": "s3cret"}},
)
loaded = manager.load_config()
(tmp_path / "config_secrets.json").write_text("{corrupt")
with pytest.raises(ConfigError):
manager.save_config_atomic(loaded)
on_disk = json.loads((tmp_path / "config.json").read_text())
assert "api_key" not in on_disk.get("weather", {})
class TestLoadFastPath:
def test_unchanged_files_return_cached_dict(self, tmp_path):
manager = make_manager(tmp_path, config={"timezone": "UTC"})
first = manager.load_config()
second = manager.load_config()
assert second is first # same aliased dict, no re-read
def test_touching_secrets_file_invalidates_cache(self, tmp_path):
manager = make_manager(
tmp_path,
config={"weather": {}},
secrets={"weather": {"api_key": "old"}},
)
assert manager.load_config()["weather"]["api_key"] == "old"
secrets_file = tmp_path / "config_secrets.json"
secrets_file.write_text(json.dumps({"weather": {"api_key": "new"}}))
# Force a different mtime_ns in case the write landed within the
# filesystem's timestamp granularity.
os.utime(secrets_file, ns=(1, 1))
assert manager.load_config()["weather"]["api_key"] == "new"
def test_same_mtime_same_size_change_served_stale(self, tmp_path):
# Characterized fast-path blind spot: the signature is (mtime_ns,
# size) only, so a same-length content swap with a forged identical
# mtime is not detected. Real writes bump mtime_ns, so this is
# acceptable — but it is a contract worth pinning.
manager = make_manager(tmp_path, config={"timezone": "AAA"})
config_file = tmp_path / "config.json"
os.utime(config_file, ns=(1_000_000_000, 1_000_000_000))
manager._loaded_sig = None
first = manager.load_config()
assert first["timezone"] == "AAA"
config_file.write_text(json.dumps({"timezone": "BBB"})) # same length
os.utime(config_file, ns=(1_000_000_000, 1_000_000_000))
assert manager.load_config()["timezone"] == "AAA" # stale, by design