3 Commits

Include renames

Author	SHA1	Message	Date
3 people	d9683e28be	Codebase audit: fix shipping bugs, remove verified-dead code, repair doc drift, add regression guards (#438) ... * fix(web): implement delete_cached so the font catalog cache actually invalidates api_v3.py's font upload/delete handlers import delete_cached from web_interface.cache, but the function was never defined. The surrounding except ImportError silently swallowed the failure, so the fonts_catalog cache entry survived uploads/deletes and newly uploaded fonts did not appear until the TTL expired or the service restarted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix(web): remove dead weather/stocks partial routes that returned 500 The partial dispatcher still routed 'weather' and 'stocks' to loaders rendering v3/partials/weather.html and stocks.html — templates that no longer exist since weather and stocks became store plugins. Requesting either partial raised TemplateNotFound, which the catch-all turned into a 500. No template or JS references these partials (the only 'weather' hit in the front end is a plugin-store category filter option), so the branches and both loader functions are removed; unknown partials now fall through to the existing 404 handler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix(deps): align contradictory psutil/Flask-Limiter/freetype-py pins requirements.txt's optional-install comment recommended psutil>=5.9,<6.0 while web_interface/requirements.txt hard-requires >=6.0,<7.0 — anyone following the comment ends up with an unsatisfiable pair. The comment now recommends the same range the web interface requires (all psutil APIs used — Process, boot_time, cpu_percent, disk_usage, virtual_memory — are stable in 6.x). Flask-Limiter gains the same <4.0 cap in both files and freetype-py the same >=2.5.1 floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix(config): add template keys the code already reads display.hardware gains pixel_mapper_config, row_address_type, multiplexing and panel_type (read at display_manager.py with these exact fallbacks — users on non-standard panels previously had no way to discover them from the template). vegas_scroll gains frame_based_scrolling and scroll_delay, the only two of its 27 keys the template omitted (read in src/vegas_mode/config.py). plugin_system gains development_mode, which the web UI reads and writes but the template never declared. Every added value is byte-identical to the code-side .get() fallback, so ConfigManager._migrate_config() merging these keys into existing user configs cannot change behavior on any installed device. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix(scripts): repair broken sys.path setup in utility scripts clear_cache.py and download_nba_logos.py pointed sys.path at a 'src' directory relative to the script's own folder (scripts/utils/src and scripts/src — neither exists), so both crashed on import; they now insert the project root and import via the src package like the other scripts. debug_web_manual.py resolved 'project root' to scripts/debug/ instead of two levels up. fix_nhl_cache.sh is removed: it used Python docstring syntax in a bash script and invoked clear_nhl_cache.py, which does not exist anywhere in the repo — it cannot ever have worked in its current location. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs: correct stale file:line references and the loader-fallback contradiction CLAUDE.md and .cursorrules disagreed about plugin-directory fallback behavior; the code (SchemaManager.get_schema_path) probes plugins/ BEFORE plugin-repos/, and the main discovery path has no fallback at all — both files now describe the real behavior, preferring symbol names over line numbers so the references rot slower. REST_API_REFERENCE.md pointed at app.py:144/:607 for mounts that live at :199/:799 and counted 92 routes where there are 94. PLUGIN_ARCHITECTURE_SPEC.md's historical banner gains a note that its example imports (src/plugin_system/base_classes/*_plugin.py) never shipped — the real base classes are src.base_classes.sports.SportsCore and src.base_classes.hockey.Hockey. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs: fix broken links, phantom script references, and stale CI description Repairs every broken relative link in active docs (targets renamed or archived long ago: PLUGIN_DEVELOPMENT.md -> PLUGIN_DEVELOPMENT_GUIDE.md, API_REFERENCE.md -> REST_API_REFERENCE.md, PLUGIN_STORE_USER_GUIDE.md -> PLUGIN_STORE_GUIDE.md, plugin_docs/ dir, TROUBLESHOOTING_QUICK_START.md, and MIGRATION_GUIDE's README link that silently resolved to the docs index instead of the project README). Replaces commands invoking scripts that do not exist (scripts/update_stats.py, validate_registry.py, check_updates.py, fix_permissions.sh) with the real tooling, and rewrites HOW_TO_RUN_TESTS.md's CI section, which described a security-audit workflow that was never committed and a pytest workflow 'queued to land' that landed long ago as test.yml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs: complete the docs index and refresh the web interface file tree docs/README.md's own policy says every page must be linked from the index, yet five weren't — including the entire skin system (SKIN_SYSTEM.md, CREATING_SKINS.md), ADAPTIVE_LAYOUT.md, plugin-safety-harness.md and SPORTS_UNIFICATION.md. Each is now listed in the section it belongs to, and PLUGIN_ARCHITECTURE_SPEC.md is marked historical in the index (the doc itself already carries the banner). web_interface/README.md's static/v3 tree showed only app.css/app.js; it now reflects the actual contents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore: remove dead modules confirmed unused in-repo and across all store plugins - src/common/cli.py: imports a 'ledmatrix_common' package that exists nowhere (not in this repo, any requirements file, or the plugin monorepo), so it cannot ever have run; its README section claimed scripts/dev/* used it, which was also untrue. - src/web_interface/logging_config.py: zero callers — the web app uses web_interface/logging_config.py (a different module), and nothing imports the src copy. - handle_errors decorator in src/web_interface/error_handler.py: zero call sites (the module's response helpers stay — they are used). - ConfigManager.get_clock_config(): reads a 'clock' config key that no longer exists anywhere; only caller was its own unit test. Deliberately kept despite zero in-repo callers: DisplayError, src/common/config_helper.py and display_helper.py — all documented as plugin-facing API (docs/PLUGIN_ERROR_HANDLING.md, src/common/README.md), and third-party plugins outside the official monorepo cannot be enumerated. Verified against a fresh clone of ledmatrix-plugins (43 plugins): zero references to any removed symbol. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore: remove manager-era NBA test files and one-off debug scripts The four test_nba_*.py files imported nba_managers, leaderboard_manager and odds_manager — top-level modules deleted when sports displays became plugins — inside try/except blocks that swallowed the ImportError, so they passed while exercising nothing. test_nba_data_structure.py and debug_nba_api.py (a diagnostic script living in test/) made live ESPN API calls rather than testing repo code. None were enrolled in CI. scripts/debug/direct_fix_imports.py and check_imports.py were one-shot artifacts that edited/inspected a hardcoded ~/LEDMatrix/web_interface/ app.py to fix an import problem solved long ago; nothing references them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore: remove generate_report.py, which aggregates artifacts of CI jobs that do not exist The script's only function is to merge JSON artifacts (bandit/semgrep/pip-audit/safety/gitleaks results) produced by a security-audit workflow that was never committed — .github/workflows/ has no such jobs, so there is nothing for it to aggregate and no way to run it usefully. Its siblings stay: prove_security.py and audit_plugins.py both run standalone (verified), and .codacy.yml stays because the Codacy service (README badge) reads it server-side without a workflow file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix(web): load the three widget scripts store plugins already declare time-picker.js, file-upload-single.js and plugin-file-manager.js register widgets that installed store plugins reference in their config schemas (countdown uses x-widget: time-picker and file-upload-single; of-the-day uses plugin-file-manager), but base.html never included the scripts. plugin_config.html renders such fields as an empty container that polls LEDMatrixWidgets.get(...) on a 50ms loop forever, so those plugin config fields appeared permanently blank. The audit initially flagged these files as dead code; the monorepo cross-check proved the opposite — they were unreachable, not unused. example-color-picker.js (the documented custom-widget example) gains an explicit warning that including it in base.html would shadow the built-in color-picker widget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore: drop the legacy youtube block from the secrets template No code reads a top-level youtube secrets key: the youtube-stats plugin receives its API key namespaced under its own plugin id (declared via x-secret in its config schema), like every other store plugin. The key survives only in state_reconciliation.py's non-plugin-key exclusion set, which stays — existing installs still carry the key in their generated config_secrets.json, and the exclusion prevents it from being misclassified as a plugin config. New installs simply stop being asked for a YouTube API key they have nowhere to use. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore(deps): remove packages nothing imports, declare direct imports, move mypy to test deps Removed from requirements.txt: python-socketio, python-engineio, websockets, websocket-client — zero imports anywhere in this repo, and the one store plugin that needs Socket.IO (ledmatrix-music) declares it in its own requirements.txt, which the plugin store installs. Removed the same quartet plus timezonefinder, geopy, google-auth-oauthlib, google-auth-httplib2, google-api-python-client, unidecode, icalevents, python-dateutil, flask-wtf and the werkzeug pin from web_interface/requirements.txt — all leftovers from the deleted built-in weather/calendar/music displays (flask-wtf was doubly dead: app.py explicitly disables CSRF and sets csrf=None). scripts/ install_dependencies_apt.py, which mirrors these lists for the first-time installer, drops the same packages. Added: urllib3 (imported directly in four core modules), jinja2 and markupsafe (imported directly in pages_v3.py) — previously reachable only as transitives. mypy moves from runtime requirements to requirements-test.txt. Verified in a fresh venv: all four requirements files co-install, pip check is clean, the full CI-enrolled suite (907 tests) and a Flask boot smoke pass with the trimmed dependency set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * refactor: single canonical DateTimeEncoder src/cache_manager.py and src/cache/disk_cache.py each defined an identical DateTimeEncoder (datetime -> ISO-8601). The disk_cache copy is the only one actually used for serialization; cache_manager now re-exports it instead of defining a twin, so the two can never silently diverge. Import compatibility is preserved — from src.cache_manager import DateTimeEncoder still works and is the same class object. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs(code): document deliberate duplicates instead of merging them The audit surfaced several near-duplicate implementations that turned out to be either deliberate forks or behaviorally different — merging any of them would risk changing behavior on installed devices, so each now carries an explicit comment stating the relationship: - VisualDisplayManager: headless fork of DisplayManager; header now lists the ~15 mirrored methods and warns that DisplayManager changes must be mirrored. - normalize_abbreviation: LogoDownloader's version (called directly by nine scoreboard plugins) replaces filesystem-unsafe characters; LogoHelper's strips spaces. Logo filenames on existing installs depend on both behaviors staying put. - The two PluginTestBase classes: the shipped one is plugin-author API, the repo's own richer harness lives in test/plugins/ — now cross-referenced. Also verified (no change needed): ConfigManager's backup/rollback methods genuinely delegate to AtomicConfigManager, and SportsCore already delegates _read_bdf_native_size to FontManager. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs: add a unified configuration reference There was no single place documenting what lives in config.json — display.* keys were scattered across README sections, vegas_scroll lived in ADVANCED_FEATURES.md, and dim_schedule, display.double_sided, sync.follower_position, plugin_system.development_mode and the four newly-templated hardware keys were documented nowhere. CONFIG_REFERENCE.md now lists every template key plus the code-read-only keys, each with type, default, and the code location that reads it, and explains the secrets file's plugin-id namespacing. Linked from the docs index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs: bring the README's feature tour into the plugin era The Core Features section still presented clock/weather/sports/stocks/ music displays as built into the project, when all of them are store plugins installed from the ledmatrix-plugins monorepo — only starlark-apps and web-ui-info ship in this repo. The intro now says so (the showcase itself is unchanged; those are real displays available in the store). The display_durations reference drops its built-in-calendar example in favor of plugin-id keys, and the Configuration section links the new CONFIG_REFERENCE.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs: archive the custom-icons status report, cross-link config docs, document assets/ PLUGIN_CUSTOM_ICONS_FEATURE.md was a 'What Was Implemented' status report duplicating the actual guide (PLUGIN_CUSTOM_ICONS.md) — moved to docs/archive/ per the docs index's own policy. The overlapping plugin-config docs keep their content but PLUGIN_CONFIG_ARCHITECTURE.md now states up front which doc is canonical for which purpose. assets/README.md is new and load-bearing: assets/stocks, weather, news_logos and broadcast_logos have zero references in this repo's code, which makes them look deletable — but store plugins (ledmatrix-stocks, ledmatrix-weather, news, odds-ticker) resolve those exact paths at runtime against the install directory. The README records that evidence so a future cleanup doesn't break installed plugins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * test: add regression guards for the bug classes fixed in this PR Three lightweight static checks, all enrolled in CI's unit-test allowlist along with the new web-cache test: - test_template_targets.py: every literal render_template() target must exist (would have caught the weather/stocks partial 500s at commit time). - test_widget_scripts.py: every widget JS file must be script-included in base.html or explicitly allowlisted with a reason (would have caught the unloaded time-picker/file-upload-single/plugin-file-manager widgets), and allowlisted files must NOT be included (prevents the example widget from shadowing the real color-picker). - test_doc_links.py: relative markdown links in active docs must resolve (docs/archive/ exempt). Each guard was verified to fail against the pre-PR tree and pass now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix(deps): restore the werkzeug version floor Commit 1ec22db removed the werkzeug>=3.1.6,<4.0.0 pin along with the genuinely-unused packages, but this one was a version floor on Flask's transitive dependency, not a phantom: Flask 3.1.3 itself only requires werkzeug>=3.1.0, so dropping the pin let fresh installs resolve 3.1.0-3.1.5. Restored with a comment explaining why it exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix(web): route pixel_mapper_config into display.hardware; guard time-picker registration pixel_mapper_config was the only display.hardware key absent from both the display_fields detection allowlist and the hardware write loop in the settings save path. No form posts it today, but if one ever did the key would fall through to the generic handler and land at the TOP level of config.json — where state_reconciliation would mistake it for a missing plugin id and loop auto-repair attempts (the failure class the 'github'/'youtube' exclusion comment documents). It now round-trips into display.hardware like its siblings. time-picker.js gains the same LEDMatrixWidgets-undefined guard its two sibling widgets already have; correct today only via defer ordering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore: align installer leftovers with the dependency cleanup first_time_install.sh's fallback secrets heredoc (used only when the template is missing) still wrote the legacy youtube block — now matches the template (github only). install_dependencies_apt.py drops the IMPORT_NAME_MAP entries for packages no longer in its install lists and a stale google-api reference in a docstring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix: address CodeRabbit review findings Verified each finding against the code; fixes for the valid ones: - install_dependencies_apt.py: the installer listed 'freetype', but the declared dependency is freetype-py — an apt miss would pip-install the wrong PyPI package. Now installs freetype-py with an import-name mapping (pre-existing bug, surfaced by the review). - api_v3.py: pixel_mapper_config is validated as a string before being saved to display.hardware (JSON callers could previously store an object/list the matrix library can't use). - .cursorrules: the Plugin Loading Process and File Organization sections still said discovery scans plugins/ — now consistent with the corrected overview (configured directory, default plugin-repos/). - README.md: removed the stale '(except the core calendar)' claim — no core calendar exists in src/ — and qualified the plugin inventory (official plugins in the monorepo; third-party from their own repos). - CONFIG_REFERENCE.md: hardware_mapping now shows the code fallback (adafruit-hat-pwm) alongside the template value. - PLUGIN_REGISTRY_SETUP_GUIDE.md: check_plugin.py takes --plugin, not a positional id. - scripts/fix_perms/fix_*.sh: exec bits set so the documented 'sudo ./...' invocations work. - Guard tests hardened: template guard now catches multi-line render_template() calls; widget guard parses actual <script> src values and fails if the widgets dir goes missing; type hints and docstrings added per repo coding guidelines. Skipped with reasons (noted on the PR): limit_refresh_rate_hz 100-vs-90 is documented as intentional in CONFIG_REFERENCE.md; the psutil comment already names the enforcing manifest; docs/archive/ findings are out of scope per the docs policy (archive may rot). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore: remove Cursor IDE tooling, consolidate its guidance into CLAUDE.md The maintainer no longer uses Cursor. .cursorrules, .cursorignore and the .cursor/ tree (rules, plugin templates, a parallel 751-line plugins guide) are removed; measurement showed near-zero literal overlap risk — the canonical content already lives in docs/. Unique guidance worth keeping moved before deletion: - CLAUDE.md gains the dev workflow (dev_plugin_setup.sh, dev_server.py, run.py -e, check_plugin.py), the plugin-secrets namespacing contract, and the no-draw_image()/paste-onto-PIL pitfall. - PLUGIN_DEVELOPMENT_GUIDE.md absorbs the plugin version-management rules (pre-push hook install, SKIP_TAG, version resolution order) that its own text previously linked out to .cursorrules for. - The one completed plan doc (.cursor/plans/) is archived to docs/archive/ per the docs policy rather than deleted. One of the deleted rule files (sports-managers.mdc) targeted src/*_managers.py globs that have matched nothing since the plugin migration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * chore: second-pass cleanup — dead installer branch, broken script, orphaned JS, misfiled test deps - first_time_install.sh: removed the pip fallback branch that installed from requirements_web_v2.txt — a file that has not existed since the v2 web interface was removed (the branch always printed its own 'not found; skipping' warning). - scripts/remove_plugin_backups.sh deleted: its PROJECT_ROOT resolved to the repo's PARENT directory, and its verify_submodules() checks for plugin submodules from an era before plugins moved to the store — it could never have worked from its current location. - plugins_manager.js: removed three functions with zero call sites anywhere (addKeyValuePair, formatCommit, togglePasswordVisibility) — verified against all templates, all JS, and the dynamic window[name] dispatch sites, which resolve widget-registry keys only. Also replaced base.html's misleading 'Legacy ... during migration' label: the file is deliberately loaded last and provides the LIVE implementations of seven window.* plugin actions that shadow same-named definitions in app.js/app-shell.js. - pytest/pytest-cov/pytest-mock moved from runtime requirements.txt to requirements-test.txt (CI already installs both files; the installer's line-by-line loop simply installs three fewer packages on devices; no store plugin declares pytest). HOW_TO_RUN_TESTS.md updated. - scripts/add_defaults_to_schemas.py and analyze_plugin_schemas.py scanned the empty legacy plugins/ dir — now scan plugin-repos/. Verified: fresh venv installs all four requirements files with pip check clean and pytest available; bash -n on the installer; node --check on the JS; widget/cache guard tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * docs: correct semantically stale content across the user and developer guides A second-pass content audit checked the guides' substantive claims against the code (the first pass only fixed mechanical drift). Fixes: - GETTING_STARTED: described booting a prebuilt SD image and seeing default clock/weather plugins — neither exists. Now documents the real install (Pi OS Lite + one-shot installer / first_time_install.sh) and that displays come from the Plugin Store. Duration and ordering instructions moved to the Rotation tab where the controls actually live. - WEB_INTERFACE_GUIDE: three whole tabs were undocumented (Rotation, Backup & Restore, Tools) and the Display tab's Vegas Scroll section was unmentioned. Fonts overrides are per display element (not per plugin); Logs has an Auto-scroll checkbox (not a Pause button); the aspirational keyboard-shortcut list and no-JS claim removed. - TROUBLESHOOTING: the hand-written service-file template (wrong user, wrong ExecStart, dropped the autostart gate) replaced with the real systemd/ units + install scripts; recovery steps no longer copy placeholder units verbatim; WiFi curl endpoint corrected to /api/v3/; cache-clearing advice now targets the real cache locations. - ADVANCED_FEATURES: removed a false claim that CacheManager has no delete(); fixed two example snippets that raise TypeError (BackgroundDataService and get_config_file_mode signatures); fixed cache paths, a 5-minute TTL that is actually 1 hour, and the vegas table now links the complete 26-key reference. - EMULATOR_SETUP_GUIDE: documented run.py flags that don't exist (--plugin/--test-plugins) removed in favor of dev_server.py and check_plugin.py; shipped emulator config values corrected (browser adapter default on :8888, not pygame). - PLUGIN_QUICK_REFERENCE: drag-and-drop reordering is shipped, not 'not yet supported'; discovery-fallback and registry-repo claims corrected. PLUGIN_API_REFERENCE: get_vegas_segment_width returns panels, not pixels. CONTRIBUTING: the repo uses flake8/mypy/bandit pre-commit hooks, not black/ruff, and tests need requirements-test.txt. - SKIN_SYSTEM/DEVELOPER_QUICK_REFERENCE: stale module paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr * fix: raise the two new dependency floors past their CVEs, silence a deliberate re-export All three of these were introduced by this PR, which is what makes them worth fixing here rather than deferring. `urllib3` and `jinja2` were added to the requirements so that direct imports stop relying on transitives — right call, but both floors were set to the version that introduced the API rather than a version that is safe to install. `urllib3>=1.26.0` sits below roughly ten CVEs including a decompression-bomb safeguard bypass, and `jinja2>=3.1.0` below five including two sandbox breakouts. Raised to 2.7.0 and 3.1.6, which is what a working device already runs, so no install is disturbed. The comments now say the floor is a security floor, since the next person to read "imported directly" would otherwise reasonably lower it again. This is the same reasoning the PR already applied to werkzeug; these two just missed it. The `DateTimeEncoder` import in cache_manager is unused on purpose — the canonical class moved to src.cache.disk_cache and this re-export keeps the documented import path working. flake8 cannot see intent, so it gets an explicit `# noqa: F401` rather than being removed and quietly breaking anything importing it from here. Verified the re-export still resolves to the same object and still serialises datetimes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Udr6MfaFLUPhX5Fgo67Jf5 * fix: address CodeRabbit re-review — installer robustness and doc lint - install_dependencies_apt.py: an import-only check let Debian Bookworm's python3-freetype 2.3.0 satisfy the freetype-py>=2.5.1 pin. check_package_installed() now verifies the installed freetype-py version, and an apt install that lands below the minimum falls through to pip instead of counting as success. - first_time_install.sh: the .web_deps_installed marker was created even when the smart installer failed, so re-runs skipped installation with dependencies missing. The marker is now created only on success. - CONTRIBUTING.md: document installing the pre-commit CLI before 'pre-commit install' (the requirements files don't provide it). - Doc lint: fence language on the on-demand cache example (MD040), blockquote continuation in GETTING_STARTED (MD028), and the suppress_adapter_load_errors key removed from the emulator debug example to match the options table. Skipped one finding with reason (noted on the PR): the per-plugin display_duration field in PLUGIN_QUICK_REFERENCE's example is not obsolete — BasePlugin.get_display_duration() reads it and PLUGIN_CONFIG_CORE_PROPERTIES.md documents it as a core property; display.display_durations is a per-mode override, not a replacement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SXb4mKcAkVaxkeTb3YnAdr --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-06 14:04:23 -04:00
4 people	05b3fa56cb	fix: Codacy security fixes, CVE dependency bumps, and code quality cleanup (#331) ... * fix(deps): bump minimum versions to address CVEs Pillow 10.4.0 → 12.2.0: CVE-2026-40192 (DoS via FITS decompression bomb), CVE-2026-25990 (OOB write via PSD image), CVE-2026-42311/42308/42310 requests 2.32.0 → 2.33.0: CVE-2026-25645 (temp file security bypass), CVE-2024-47081 (.netrc credentials leak) werkzeug 3.0.0 → 3.1.6: CVE-2023-46136, CVE-2024-49766/49767, CVE-2025-66221, CVE-2026-21860/27199 (DoS, path traversal, safe_join bypass) Flask 3.0.0 → 3.1.3: CVE-2026-27205 (session data caching info disclosure) spotipy 2.24.0 → 2.25.2: CVE-2025-27154, CVE-2025-66040 python-socketio 5.11.0 → 5.14.0: CVE-2025-61765 pytest 7.4.0 → 9.0.3: CVE-2025-71176 (insecure temp dir handling) Updated in requirements.txt, web_interface/requirements.txt, plugin-repos/starlark-apps/requirements.txt, and plugin-repos/march-madness/requirements.txt. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve Pylint errors in executor, data service, and odds call Rename TimeoutError to PluginTimeoutError in plugin_executor.py to avoid shadowing the built-in; no external callers affected. Remove dead try/except in BackgroundDataService.shutdown: executor.shutdown() never accepted a timeout kwarg so the try branch always raised TypeError. Simplify to a direct shutdown(wait=wait) call. Remove is_live kwarg from odds_manager.get_odds() call in sports.py; BaseOddsManager.get_odds() has no such parameter. The live update interval is already encoded in the update_interval_seconds argument passed alongside. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: MD5→SHA-256, shellcheck warnings, and broken doc links config_service.py: replace MD5 with SHA-256 for config change detection; same semantics (equality comparison), no stored hashes affected. Shell scripts — shellcheck warnings: - diagnose_web_interface.sh: remove useless cat (SC2002) - dev_plugin_setup.sh: restructure A&&B||C into if/then (SC2015) - fix_assets_permissions.sh: remove unused REAL_HOME block (SC2034) - install_web_service.sh: remove unused USER_HOME assignment (SC2034) - diagnose_web_ui.sh: remove unused SUDO assignments (SC2034) - diagnose_plugin_permissions.sh: remove unused BLUE color var (SC2034) - first_time_install.sh: remove unused CLEAR var, PACKAGE_NAME assignment, and replace loop variable with _ (SC2034) docs/PLUGIN_ARCHITECTURE_SPEC.md: fix 10 broken TOC anchor links to include section numbers matching the actual headings (MD051). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused imports and bare exception aliases (pyflakes F401/F841) Remove unused imports across 86 files in src/, web_interface/, test/, and scripts/ using autoflake. No logic changes — only dead import statements and unused names in from-imports are removed. Also remove bare exception aliases where the variable is never referenced in the handler body: - src/cache/disk_cache.py: except (IOError, OSError, PermissionError) as e - src/cache_manager.py: except (OSError, IOError, PermissionError) as perm_error - src/plugin_system/resource_monitor.py: except Exception as e - web_interface/app.py: except Exception as read_err 86 files changed, 205 lines removed, 18 pre-existing test failures unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused local variable assignments (pyflakes F841) Dead assignments removed across src/ and web_interface/: - background_data_service: drop future= on fire-and-forget executor.submit - base_classes/baseball: drop font= (all rendering uses self.fonts['time']) - base_classes/hockey: drop status_short= (never referenced after assignment) - common/cli: drop game_helper=/config_helper= bindings in import-test block; constructors called for instantiation-only validation - common/display_helper: drop text_width= (x_position uses display_width directly); drop draw= in create_error_image (uses _draw_centered_text) - config_manager: remove dead secrets_content loading block in migration path (comment already noted save_config_atomic handles secrets internally) - display_manager: drop setup_start= (timing was never completed or read) - font_manager: drop target_path= (catalog uses font_file_path directly); drop face=/font= bindings in validate_font (validation by construction — TypeError on failure is the signal, not the return value) - font_test_manager: drop width=/height= (draw_text uses display_manager directly) - plugin_system/state_reconciliation: drop manager= (only config/disk/state_mgr used) - plugin_system/store_manager: drop result= on pip install subprocess.run (check=True raises on failure; stdout unused) - web_interface/blueprints/pages_v3: drop main_config_path=""/secrets_config_path="" (render_template uses config_manager.get_*_path() inline) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(js): resolve ESLint no-undef warnings across 6 JS files Three distinct patterns: 1. Vendor library globals — htmx is injected by <script> before these extension files load; ESLint lints files in isolation and doesn't know. Fix: add /* global htmx */ to htmx-sse.js and htmx-json-enc.js. 2. Cross-file globals — showNotification is defined as window.showNotification in app.js/notification.js but called bare in app.js and error_handler.js. ESLint doesn't connect window.X = Y with a bare call to X. Fix: add /* global showNotification */ to app.js and error_handler.js. 3. Forward-reference window.* functions — in array-table.js, checkbox-group.js, and custom-feeds.js, functions like removeArrayTableRow are called early inside event-handler closures but assigned to window.* later in the file. At runtime this works (the handler fires after the assignment), but ESLint sees the bare name at the call site. Fix: change bare calls to window.removeArrayTableRow(this) etc. so the reference is explicit and ESLint-safe. Also guard the updateSystemStats call in app.js reconnectSSE: the function is called but defined nowhere in the codebase. Guard with typeof check so it won't throw ReferenceError if the reconnect path is hit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(js): resolve Biome lint warnings across 9 JS files noUnusedVariables (catch bindings → optional catch syntax): - app.js, file-upload.js, timezone-selector.js: } catch (e) { → } catch { ES2019 optional catch binding; e was unused in all three handlers noUnusedVariables (dead assignments): - app.js: remove const data= in display SSE stub (handler does nothing yet) - api_client.js: remove const timeoutId= (setTimeout ID never used to cancel) - custom-feeds.js: remove const oldIndex= (getAttribute result never read) - schedule-picker.js: remove const compactMode= (never used in HTML build) - select-dropdown.js: remove const icons= (icons not yet rendered in options) noPrototypeBuiltins: - day-selector.js: DAY_LABELS.hasOwnProperty(x) → Object.prototype.hasOwnProperty.call(DAY_LABELS, x) Safe form that works even on null-prototype objects useIterableCallbackReturn: - file-upload.js, notification.js: forEach(x => expr) → forEach(x => { expr; }) — forEach ignores return values; implicit return from arrow body was misleading htmx-sse.js is a vendor extension file with old-style var/== patterns that are correct for it; 18 Biome issues suppressed via Codacy API rather than modifying the vendor source. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(security): escape user input in raw HTML responses in pages_v3.py plugin_id comes directly from the URL path (/partials/plugin-config/<plugin_id>) and was interpolated into an HTML fragment without escaping. A crafted URL like /partials/plugin-config/<script>alert(1)</script> would inject that tag into the DOM via the HTMX partial response. Fix: wrap all user-controlled values in markupsafe.escape() before embedding in raw HTML strings. Affects the plugin-not-found 404 response and both error 500 responses in the plugin config partial. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address Bandit B108/B110 across production code B110 (try/except/pass): - display_controller.py: narrow 'except Exception' to 'except AttributeError' for get_offset_frame() — plugins not having this optional method is the expected case, not all exceptions - config_manager.py: B110 already resolved by the earlier removal of the dead secrets-loading block (the except/pass was inside it) - All other except/pass blocks in src/ and web_interface/ are intentional (last-resort recovery, best-effort fallbacks, non-critical startup probes). Annotated each with # nosec B110 and a brief inline reason so the decision is explicit for future reviewers. - Test files and plugin-repos B110 suppressed via Codacy API (not prod code). B108 (/tmp usage): - permission_utils.py: /tmp listed to PREVENT permission changes on it — not used as a temp path. Annotated # nosec B108. - display_manager.py: fixed snapshot path is intentional (web UI reads same path); path-check guard also annotated. - wifi_manager.py: named /tmp files match the sudoers allowlist installed with the system (the paths are hard-coded in both places by design). Annotated all six open/cp references # nosec B108. - scripts/render_plugin.py: dev script default overridable by user. Annotated. - web_interface/app.py: reads the same fixed path written by display_manager. Annotated # nosec B108. - Test files suppressed via Codacy API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address remaining Codacy security findings Flask debug=True (real fix): - web_interface/app.py: debug=True in __main__ block exposes the Werkzeug interactive debugger (arbitrary code execution). Changed to os.environ.get('FLASK_DEBUG', '0') == '1' — off by default, opt-in via environment variable for local development. nosec annotations (accepted risk with documented rationale): - disk_cache.py: os.chmod(0o660) is intentional — web UI and LED matrix service share a group, 660 gives group write while denying world access (B103 + Semgrep insecure-file-permissions suppressed in Codacy) - wifi_manager.py: urlopen to hardcoded connectivity-check.ubuntu.com URL (B310 — no user input involved) - font_manager.py: urlretrieve URL comes from user's own config file on their local device (B310) - start_web_conditionally.py: os.execvp with both sys.executable and a fixed PROJECT_DIR-relative constant (B606) Confirmed false positives suppressed via Codacy API (15 issues): - SSRF (3x): client-side JS fetch — SSRF is server-side; browser fetch is CORS-restricted to same origin - B105 (3x): test fixtures use dummy secrets by design; store_manager checks for the placeholder string, it is not itself a secret - PMD numeric literal (2x): 10000000 is within Number.MAX_SAFE_INTEGER - Prototype pollution (1x): read-only schema traversal, no writes - no-unsanitized_method (1x): dynamic import() is CORS-restricted - detect-unsafe-regex (1x): operates on server-controlled config values - plugin-repos B103 (1x): vendor code chmod on executable - Semgrep insecure-file-permissions (3x): same disk_cache 0o660 as above Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unnecessary f prefix from f-strings without placeholders (F541) Pyflakes F541 flags f-strings that contain no {} interpolation — they are identical to plain strings but trigger unnecessary string formatting overhead. Fixed in production code: - src/base_classes/data_sources.py (2 debug log calls) - src/logo_downloader.py (1 error log) - src/plugin_system/store_manager.py (5 strings across 3 log calls) - src/web_interface/validators.py (1 return value) - src/wifi_manager.py (4 log/message strings) - web_interface/start.py (1 print) F541 issues in test/, scripts/, and plugin-repos/ suppressed via Codacy API as non-production code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(dev): add Pillow compatibility smoke test script Covers all Pillow APIs used in LEDMatrix — image creation, drawing, font metrics, LANCZOS resampling, paste/alpha_composite, and PNG I/O. Run after any Pillow version bump to catch regressions before deploy. python3 scripts/dev/test_pillow_compat.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve 8 new Codacy issues introduced by PR changes shellcheck SC2034: - first_time_install.sh: 'type' loop variable also unused in the wifi status loop (we previously fixed 'device' → '_' but left 'type'). Changed to '_ _ state' since neither device nor type is referenced. ESLint no-undef: - app.js: typeof guards don't satisfy no-undef; added updateSystemStats to the /* global */ declaration alongside showNotification. nosec annotation: - web_interface/app.py: app.run(host='0.0.0.0') line changed when we fixed debug=True, giving it a new issue ID. Re-added # nosec B104. pyflakes F401: - scripts/dev/test_pillow_compat.py: ImageFilter was imported but never used in the smoke test. Removed from the import. Codacy API suppressions (false positives on changed lines): - disk_cache.py 0o660 chmod (2x): lines changed when # nosec B103 was added, producing new Semgrep issue IDs. Re-suppressed. - pages_v3.py raw-html-concat: Semgrep does not recognise escape() as a sanitizer; the escape() call IS the correct fix. - app.py flask 0.0.0.0: same line as B104 above; Semgrep rule also re-suppressed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address PR review findings Fix (10 of 15 findings): plugin-repos/march-madness/requirements.txt: Add urllib3>=1.26.0 — manager.py directly imports from urllib3; it was an undeclared transitive dependency via requests. scripts/dev/dev_plugin_setup.sh: Restore subshell form (cd "$target_dir" && git pull --rebase) || true so the shell's working directory is not permanently changed after the if-cd block. Previous fix for SC2015 leaked cwd into the remainder of the script. src/base_classes/sports.py: Narrow 'except Exception' to 'except RuntimeError as e' and log via self.logger.debug — Path.home() raises only RuntimeError for service users; other exceptions should not be silently swallowed. src/config_service.py: Fix stale "MD5 checksum" in ConfigVersion.__init__ docstring (line 40); the implementation uses SHA-256 since the Codacy fix. src/wifi_manager.py: Log the last-resort AP enable failure with exc_info=True instead of silently passing — failure here means the device may be unreachable. web_interface/blueprints/pages_v3.py: Log the outer metadata pre-load exception at debug level instead of swallowing it silently; schema still loads fully below. src/background_data_service.py: Remove unused 'timeout' parameter from shutdown() — executor.shutdown() does not accept timeout; update __del__ caller accordingly. src/font_manager.py: Validate URL scheme before urlretrieve — reject non-http/https schemes (e.g. file://) to prevent reading local files from config-supplied URLs. src/plugin_system/plugin_executor.py: Simplify redundant except tuple: (PluginTimeoutError, PluginError, Exception) → Exception, which already covers the others. test/test_display_controller.py: Mark empty test_plugin_discovery_and_loading as @pytest.mark.skip with reason. Move duplicate 'from datetime import datetime' to module header and remove the stray mid-module copy. Skip (5 of 15 findings, with reasons): - pytest 9.0.3 concerns: full suite already verified (467 pass, 18 pre-existing) - Pillow 12.2.0 API concerns: no deprecated APIs in codebase; tests + Pi smoke test pass - diagnose_web_ui.sh sudo validation: set -e already ensures fail-fast on any sudo failure - app.py request-logging except: must stay silent (recursive logging risk); annotated - app.py SSE file-read except: genuinely transient I/O; annotated Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Chuck <chuck@example.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-15 10:19:55 -04:00
3 people	7ec4323ff4	fix(config): Add defaults to schemas and fix config validation issues (#153) ... * fix(web): Resolve font display and config API error handling issues - Fix font catalog display error where path.startsWith fails (path is object, not string) - Update save_main_config to use error_response() helper - Improve save_raw_main_config error handling consistency - Add proper error codes and traceback details to API responses * fix(web): Prevent fontCatalog redeclaration error on HTMX reload - Use window object to store global font variables - Check if script has already loaded before declaring variables - Update both window properties and local references on assignment - Fixes 'Identifier fontCatalog has already been declared' error * fix(web): Wrap fonts script in IIFE to prevent all redeclaration errors - Wrap entire script in IIFE that only runs once - Check if script already loaded before declaring variables/functions - Expose initializeFontsTab to window for re-initialization - Prevents 'Identifier has already been declared' errors on HTMX reload * fix(web): Exempt config save API endpoints from CSRF protection - Exempt save_raw_main_config, save_raw_secrets_config, and save_main_config from CSRF - These endpoints are called via fetch from JavaScript and don't include CSRF tokens - Fixes 500 error when saving config via raw JSON editor * fix(web): Exempt system action endpoint from CSRF protection - Exempt execute_system_action from CSRF - Fixes 500 error when using system action buttons (restart display, restart Pi, etc.) - These endpoints are called via HTMX and don't include CSRF tokens * fix(web): Exempt all API v3 endpoints from CSRF protection - Add before_request handler to exempt all api_v3.* endpoints - All API endpoints are programmatic (HTMX/fetch) and don't include CSRF tokens - Prevents future CSRF errors on any API endpoint - Cleaner than exempting individual endpoints * refactor(web): Remove CSRF protection for local-only application - CSRF is designed for internet-facing apps to prevent cross-site attacks - For local-only Raspberry Pi app, threat model is different - All endpoints were exempted anyway, so it wasn't protecting anything - Forms use HTMX without CSRF tokens - If exposing to internet later, can re-enable with proper token implementation * fix(web): Fix font path double-prefixing in font catalog display - Only prefix with 'assets/fonts/' if path is a bare filename - If path starts with '/' (absolute) or 'assets/' (already prefixed), use as-is - Fixes double-prefixing when get_fonts_catalog returns relative paths like 'assets/fonts/press_start.ttf' * fix(web): Remove fontsTabInitialized guard to allow re-initialization on HTMX reload - Remove fontsTabInitialized check that prevented re-initialization on HTMX content swap - The window._fontsScriptLoaded guard is sufficient to prevent function redeclaration - Allow initializeFontsTab() to run on each HTMX swap to attach listeners to new DOM elements - Fixes fonts UI breaking after HTMX reload (buttons, upload dropzone, etc. not working) * fix(api): Preserve empty strings for optional string fields in plugin config - Add _is_field_required() helper to check if fields are required in schema - Update _parse_form_value_with_schema() to preserve empty strings for optional string fields - Fixes 400 error when saving MQTT plugin config with empty username/password - Resolves validation error: 'Expected type string, got NoneType' * fix(config): Add defaults to schemas and fix None value handling - Updated merge_with_defaults to replace None values with defaults - Fixed form processing to skip empty optional fields without defaults - Added script to automatically add defaults to all plugin config schemas - Added defaults to 89 fields across 10 plugin schemas - Prevents validation errors from None values in configs Changes: - schema_manager.py: Enhanced merge_with_defaults to replace None with defaults - api_v3.py: Added _SKIP_FIELD sentinel to skip optional fields without defaults - add_defaults_to_schemas.py: Script to add sensible defaults to schemas - Plugin schemas: Added defaults for number, boolean, and array fields * fix(config): Fix save button spinner by checking HTTP status code - Fixed handleConfigSave to check xhr.status instead of event.detail.successful - With hx-swap="none", HTMX doesn't set event.detail.successful - Now properly detects successful saves (status 200-299) and stops spinner - Improved error message extraction from API responses - Also fixed handleToggleResponse for consistency --------- Co-authored-by: Chuck <chuck@example.com>	2025-12-28 09:18:20 -05:00