ci: run the whole test tree and make the plugin-safety job assert something real

The unit-tests CI job ran an explicit 24-file allowlist that had rotted:
63 of 90 test files (display, vegas, store manager, web API, web_interface)
never ran on a PR. The job now runs all of test/ (minus test/plugins, which
the plugin-safety job owns) so new test files are enrolled by default and
any exclusion needs a visible, commented --ignore.

The plugin-safety job was a green no-op: plugins/ is empty in CI, so every
test skipped with 'Manifest not found'. It now renders a bundled
deterministic fixture plugin (test/fixtures/plugins/ci-fixture-plugin,
golden images included for all 8 default sizes) via LEDMATRIX_PLUGINS_DIR,
and sets LEDMATRIX_REQUIRE_PLUGINS=1 so discovering zero plugins fails
loudly instead of skipping green. The per-plugin suites document that they
target dev machines with real plugins installed.

Coverage is now measured and enforced in exactly one place — the CI
unit-tests step (--cov=src --cov=web_interface --cov-fail-under=45, from a
measured 47% baseline). pytest.ini previously declared --cov-fail-under=30
but CI always passed --no-cov, so the gate had never run anywhere; local
pytest is now coverage-free and fast.

Enabling the 63 unenrolled files surfaced three cases of test rot, fixed
here: test_display_controller_vegas_tick.py could not collect without the
hardware rgbmatrix module (now uses the emulator convention), the
state-reconciliation unrecoverable-cache tests broke when production added
the is_plugin_uninstalled tombstone check (bare Mock returned truthy),
and test_get_system_status assumed the optional psutil dependency
(now installed via requirements-test.txt and guarded by importorskip).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NohXi78cwsAKtN1sCfxjUh

.github/workflows/test.yml

@@ -13,6 +13,12 @@ jobs:
   plugin-safety:
     name: Plugin safety harness + unit tests
     runs-on: ubuntu-latest
+    env:
+      # The bundled fixture plugin gives the harness at least one real plugin
+      # to render, and REQUIRE_PLUGINS turns "discovered zero plugins" into a
+      # hard failure instead of a silent all-skip green run.
+      LEDMATRIX_PLUGINS_DIR: test/fixtures/plugins
+      LEDMATRIX_REQUIRE_PLUGINS: "1"
     steps:
       - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
         with:
@@ -29,12 +35,9 @@ jobs:
           pip install -r requirements.txt -r requirements-test.txt
           pip install RGBMatrixEmulator
 
-      - name: Run harness + visual rendering tests
+      - name: Run plugin safety harness
         run: |
-          pytest --no-cov \
+          pytest --no-cov test/plugins/
-            test/plugins/test_harness.py \
-            test/plugins/test_visual_rendering.py \
-            test/plugins/test_plugin_matrix.py
 
   unit-tests:
     name: Core unit tests
@@ -55,35 +58,15 @@ jobs:
           pip install -r requirements.txt -r requirements-test.txt
           pip install RGBMatrixEmulator
 
-      # Safety net for the shared sports/scroll/style infrastructure. These
+      # Run the ENTIRE test tree (except test/plugins, which the
-      # suites existed but were not enrolled in CI, so a refactor of
+      # plugin-safety job owns). New test files are enrolled automatically;
-      # src/base_classes or src/common could regress them silently. Enrolled
+      # excluding anything requires a visible, commented --ignore here.
-      # explicitly (not `pytest test/`) so known hardware-only suites don't
+      # Coverage is measured and enforced only in this step — pytest.ini
-      # break CI; grow this list as more suites are made headless.
+      # deliberately carries no coverage flags so local runs stay fast.
       - name: Run core unit suites
         run: |
-          pytest --no-cov \
+          pytest -m "not hardware" test/ \
-            test/test_skin_system.py \
+            --ignore=test/plugins \
-            test/test_font_manager.py \
+            --cov=src --cov=web_interface \
-            test/test_data_sources.py \
+            --cov-report=term \
-            test/test_api_extractors.py \
+            --cov-fail-under=45
-            test/test_scroll_helper.py \
-            test/test_scroll_helper_continuous.py \
-            test/test_adaptive_layout.py \
-            test/test_loader_compat_warning.py \
-            test/test_sports_base_characterization.py \
-            test/test_element_style.py \
-            test/test_sports_core_promotions.py \
-            test/test_sports_modes_promotions.py \
-            test/test_sports_capabilities.py \
-            test/test_sports_scroll.py \
-            test/test_version_consistency.py \
-            test/test_plugin_compatibility_gate.py \
-            test/test_install_preserves_existing.py \
-            test/test_core_owned_config_keys.py \
-            test/test_async_plugin_updates.py \
-            test/test_plugin_update_reservation.py \
-            test/test_template_targets.py \
-            test/test_widget_scripts.py \
-            test/test_doc_links.py \
-            test/web_interface/test_cache.py

pytest.ini

@@ -10,16 +10,13 @@ python_functions = test_*
 testpaths = test
 
 # Output options
-# Note: Coverage options require pytest-cov to be installed
+# Coverage is deliberately NOT configured here: a bare local `pytest` should
-# Run: pip install pytest-cov
+# be fast and dependency-light. Coverage is measured and enforced in exactly
+# one place — the unit-tests job in .github/workflows/test.yml.
 addopts =
     -v
     --strict-markers
     --tb=short
-    --cov=src
-    --cov-report=term-missing
-    --cov-report=html
-    --cov-fail-under=30
 
 # Markers
 markers =

requirements-test.txt

@@ -4,4 +4,6 @@ pytest>=9.0.3,<10.0.0
 pytest-cov>=4.1.0,<5.0.0
 pytest-mock>=3.11.0,<4.0.0
 freezegun>=1.2,<2        # deterministic time for golden-image tests
+psutil>=6.0.0,<8.0.0     # optional at runtime; installed for tests so the
+                         # /system/status endpoint's real path is exercised
 mypy>=1.5.0,<2.0.0       # static type checking (also pinned in .pre-commit-config.yaml)

test/fixtures/plugins/ci-fixture-plugin/config_schema.json

@@ -0,0 +1,31 @@
+{
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "title": "CI Fixture Plugin",
+    "type": "object",
+    "properties": {
+        "enabled": {
+            "type": "boolean",
+            "default": true
+        },
+        "display_duration": {
+            "type": "number",
+            "default": 5
+        },
+        "border_color": {
+            "type": "array",
+            "items": {"type": "integer", "minimum": 0, "maximum": 255},
+            "minItems": 3,
+            "maxItems": 3,
+            "default": [0, 255, 0],
+            "description": "RGB color of the border rectangle."
+        },
+        "diagonal_color": {
+            "type": "array",
+            "items": {"type": "integer", "minimum": 0, "maximum": 255},
+            "minItems": 3,
+            "maxItems": 3,
+            "default": [255, 0, 0],
+            "description": "RGB color of the diagonals."
+        }
+    }
+}

test/fixtures/plugins/ci-fixture-plugin/manager.py

@@ -0,0 +1,38 @@
+"""
+CI fixture plugin.
+
+Exists so the plugin safety harness (test/plugins/test_plugin_matrix.py and
+the plugin-safety CI job) always has at least one real plugin to load and
+render — without it, an empty plugins/ directory turns the whole job into a
+green no-op. The render is deliberately trivial and fully deterministic:
+a border rectangle plus both diagonals, sized from the display manager's
+declared dimensions. No fonts, no network, no time dependence, so golden
+images are stable across platforms.
+"""
+
+from PIL import ImageDraw
+
+from src.plugin_system.base_plugin import BasePlugin
+
+
+class CIFixturePlugin(BasePlugin):
+    def update(self) -> None:
+        """Nothing to fetch — the render is self-contained."""
+
+    def display(self, force_clear: bool = False) -> None:
+        width = self.display_manager.matrix.width
+        height = self.display_manager.matrix.height
+        border = tuple(self.config.get("border_color", [0, 255, 0]))
+        diagonal = tuple(self.config.get("diagonal_color", [255, 0, 0]))
+
+        image = self.display_manager.image
+        draw = ImageDraw.Draw(image)
+        # Blank only the declared panel area, then draw edge-to-edge content:
+        # the border proves the plugin reads dynamic dimensions (any overflow
+        # or underfill at any size is a harness bug or a dimensions bug), the
+        # diagonals make golden comparisons sensitive to size/offset drift.
+        draw.rectangle([0, 0, width - 1, height - 1], fill=(0, 0, 0))
+        draw.rectangle([0, 0, width - 1, height - 1], outline=border)
+        draw.line([0, 0, width - 1, height - 1], fill=diagonal)
+        draw.line([0, height - 1, width - 1, 0], fill=diagonal)
+        self.display_manager.update_display()

test/fixtures/plugins/ci-fixture-plugin/manifest.json

@@ -0,0 +1,13 @@
+{
+    "id": "ci-fixture-plugin",
+    "name": "CI Fixture Plugin",
+    "version": "1.0.0",
+    "description": "Bundled test fixture so the plugin safety harness always has at least one real plugin to render in CI. Draws a deterministic border + diagonals pattern at any panel size. Not installable from the store and never shipped to devices.",
+    "author": "LEDMatrix",
+    "entry_point": "manager.py",
+    "class_name": "CIFixturePlugin",
+    "display_modes": ["ci-fixture"],
+    "update_interval": 3600,
+    "min_ledmatrix_version": "2.0.0",
+    "compatible_versions": [">=2.0.0"]
+}

test/fixtures/plugins/ci-fixture-plugin/requirements.txt

@@ -0,0 +1 @@
+# No dependencies — the fixture must load in any environment.

test/plugins/conftest.py

@@ -23,9 +23,17 @@ os.environ['EMULATOR'] = 'true'
 def plugins_dir() -> Path:
     """Get the plugins directory path.
 
-    Checks plugins/ first, then falls back to plugin-repos/
+    Honors LEDMATRIX_PLUGINS_DIR (first entry) when set — the same override
-    for monorepo development environments.
+    test_plugin_matrix.py uses, so CI can point every plugin suite at the
+    bundled fixture plugins. Otherwise checks plugins/ first, then falls
+    back to plugin-repos/ for monorepo development environments.
     """
+    env = os.environ.get('LEDMATRIX_PLUGINS_DIR')
+    if env:
+        first = env.split(os.pathsep)[0]
+        if first:
+            return Path(first)
+
     plugins_path = project_root / 'plugins'
     plugin_repos_path = project_root / 'plugin-repos'
 

test/plugins/test_basketball_scoreboard.py

@@ -1,5 +1,10 @@
 """
 Integration tests for basketball-scoreboard plugin.
+
+Requires the real plugin to be installed (plugins/ or plugin-repos/,
+or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it,
+every test here skips by design. CI covers plugin safety with the
+bundled fixture plugin via test_plugin_matrix.py instead.
 """
 
 import pytest

test/plugins/test_calendar.py

@@ -1,5 +1,10 @@
 """
 Integration tests for calendar plugin.
+
+Requires the real plugin to be installed (plugins/ or plugin-repos/,
+or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it,
+every test here skips by design. CI covers plugin safety with the
+bundled fixture plugin via test_plugin_matrix.py instead.
 """
 
 import pytest

test/plugins/test_clock_simple.py

@@ -1,5 +1,10 @@
 """
 Integration tests for clock-simple plugin.
+
+Requires the real plugin to be installed (plugins/ or plugin-repos/,
+or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it,
+every test here skips by design. CI covers plugin safety with the
+bundled fixture plugin via test_plugin_matrix.py instead.
 """
 
 import pytest

test/plugins/test_odds_ticker.py

@@ -1,5 +1,10 @@
 """
 Integration tests for odds-ticker plugin.
+
+Requires the real plugin to be installed (plugins/ or plugin-repos/,
+or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it,
+every test here skips by design. CI covers plugin safety with the
+bundled fixture plugin via test_plugin_matrix.py instead.
 """
 
 import pytest

test/plugins/test_soccer_scoreboard.py

@@ -1,5 +1,10 @@
 """
 Integration tests for soccer-scoreboard plugin.
+
+Requires the real plugin to be installed (plugins/ or plugin-repos/,
+or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it,
+every test here skips by design. CI covers plugin safety with the
+bundled fixture plugin via test_plugin_matrix.py instead.
 """
 
 import pytest

test/plugins/test_text_display.py

@@ -1,5 +1,10 @@
 """
 Integration tests for text-display plugin.
+
+Requires the real plugin to be installed (plugins/ or plugin-repos/,
+or the dir named by LEDMATRIX_PLUGINS_DIR) — on machines without it,
+every test here skips by design. CI covers plugin safety with the
+bundled fixture plugin via test_plugin_matrix.py instead.
 """
 
 import pytest

test/test_display_controller_vegas_tick.py

@@ -11,9 +11,18 @@ orphaning VegasModeCoordinator.mark_plugin_updated() -- it has had zero
 callers since.
 """
 
+import os
 from typing import Dict, List, Optional
 from unittest.mock import MagicMock
 
+# display_controller imports display_manager, which imports the hardware
+# rgbmatrix module unless EMULATOR=true was set before import. Use the
+# emulator (same convention as test_display_dirty_tracking.py and
+# test/plugins/conftest.py) so this file collects on machines without the
+# hardware library — and so display_manager gets the emulator binding no
+# matter which test module imports it first.
+os.environ.setdefault("EMULATOR", "true")
+
 from src.display_controller import DisplayController
 
 

test/test_web_api.py

@@ -393,6 +393,9 @@ class TestSystemAPI:
     @patch('web_interface.blueprints.api_v3.subprocess')
     def test_get_system_status(self, mock_subprocess, client):
         """Test getting system status."""
+        # The endpoint returns 503 without psutil, which is an optional
+        # runtime dependency (requirements-test.txt installs it for CI).
+        pytest.importorskip("psutil")
         mock_result = MagicMock()
         mock_result.stdout = 'active\n'
         mock_result.returncode = 0

test/web_interface/test_state_reconciliation.py

@@ -367,6 +367,10 @@ class TestStateReconciliationUnrecoverable(unittest.TestCase):
         self.store_manager.fetch_registry.return_value = {"plugins": []}
         self.store_manager.install_plugin.return_value = False
         self.store_manager.was_recently_uninstalled.return_value = False
+        # A bare Mock() returns a truthy Mock for is_plugin_uninstalled(),
+        # which reads as "persistently uninstalled" and skips auto-repair
+        # entirely — these tests need the repair path to run.
+        self.store_manager.is_plugin_uninstalled.return_value = False
 
         self.reconciler = StateReconciliation(
             state_manager=self.state_manager,