5 Commits

Include renames

Author	SHA1	Message	Date
ChuckandClaude Opus 5.5	3967a6cffc	fix(security): re-harden root sudo helpers; installer fixes; ARCHITECTURE and PERMISSIONS docs (#640) ... * docs: add ARCHITECTURE and PERMISSIONS guides ARCHITECTURE.md maps the processes, the state the display and web services share through the cache, the display loop, the plugin system, the web UI and the update path, with links into the code and a where-to-start table. PERMISSIONS.md lists who owns what after install, both sudoers files (and why iptables is not granted), the polkit rule, and which scripts/fix_perms script to run as which user. Both are linked from the docs index, along with the MQTT bridge README and src/common/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: correct stale setup, service and troubleshooting claims - README: quick actions run systemctl on ledmatrix.service (run.py), not display_controller.py; use_short_date_format has no effect; the installer uses system pip with --break-system-packages, not a venv. - CONFIG_DEBUGGING: LEDMATRIX_DEBUG must be "true"; logs are in journald. - GETTING_STARTED, WEB_INTERFACE_GUIDE, TROUBLESHOOTING: enabling a plugin, plugin settings, brightness and Vegas settings apply without a restart; matrix hardware settings still need one. - TROUBLESHOOTING: install dependencies with sudo so the root service sees them; point permission problems at PERMISSIONS.md instead of a project-wide chown. - ADVANCED_FEATURES: real BackgroundDataService stats keys; Vegas hooks return VegasDisplayMode and None falls back to capture; cache files are 0660; fix_web_permissions.sh runs as the web user and does not touch sudoers. - STARLARK_APPS_GUIDE: only the linux-arm64 pixlet binary is downloaded. - HOW_TO_RUN_TESTS: test class examples that exist. - CLAUDE.md: PluginStoreManager, plugin_dirs.py, monorepo installs via the Trees API with ZIP fallback, requirements.txt is optional. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: mark deprecated plugin APIs and state manifest fields once Methods @deprecated("3.7.0") (the set pinned in test_deprecation.py) were shown as current API in the quick reference, API reference, advanced guide, development guide and FONT_MANAGER. Each is now marked deprecated with its replacement. FONT_MANAGER is rewritten around the current API; the override editor is gone and override methods are deprecated. Required manifest fields were stated three different ways. The API reference now has one section: the 7 schema-required fields, the 4 the store refuses without, class_name for the loader, and the 8 to set. The other guides link to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: document every src/common module and every widget - src/common/README.md covered 7 of 17 modules. It now has a table of all of them (purpose, whether plugins import it, release to floor on), a short entry each, and logging advice that matches the code. - SPORTS_UNIFICATION listed two shared modules and called sports_helpers the first; it now lists all six. - The widgets README lists all 28 registered widgets plus the support files, and absorbs the parts that only docs/widget-guide.md had (x-options.labels, x-advanced, x-display hidden, plugin-file-manager). docs/widget-guide.md is now a pointer to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(security): fix_web_permissions.sh re-hardens the root sudo helpers The script chowns the whole project to the web user. That included scripts/fix_perms/safe_plugin_rm.sh and safe_pip_install.sh -- the two helpers /etc/sudoers.d/ledmatrix_web lets the web user run as root -- so running it turned both into a root shell for whoever can edit them. It also re-grouped config_secrets.json away from ledmatrix. After the chown it now does what first_time_install.sh's Steps 11 and 11.1 do: helpers back to root:root 755, and config_secrets.json back to the web unit's User=:ledmatrix 640. Each step is non-fatal and prints the manual command if it fails. Also fixes what the script and its docs claimed: it never configured sudoers, its closing hint pointed at ./configure_web_sudo.sh (wrong path), and the README and ADVANCED_FEATURES.md said to run it with sudo, which it refuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(security): validate and harden every sudoers drop-in the scripts write configure_wifi_permissions.sh copied its rules into /etc/sudoers.d/ledmatrix_wifi without `visudo -c`. A malformed drop-in makes sudo refuse every command for every user, which on a headless Pi leaves no way back in. It now checks first and leaves the installed file alone when the rules do not parse, as the other two writers do. (It already used mktemp, so that part of the review did not apply.) It also grants the two literal commands wifi_manager.py runs for NetworkManager's shared-mode dnsmasq drop-in -- `cp /tmp/ledmatrix-nm-dnsmasq.conf .../dnsmasq-shared.d/ledmatrix-captive.conf` and `rm -f` of that file. The directory's mkdir was granted, the file was not. Both are pinned in test_sudo_allowlist_covers_calls.py. configure_web_sudo.sh wrote its rules to /tmp/ledmatrix_web_sudoers_$$, a predictable name in a world-writable directory; it now uses mktemp with an EXIT trap, as first_time_install.sh does. It sets mode 440 on the installed file instead of leaving the temp file's mode, and finds visudo in /usr/sbin when that is not on the user's PATH, which skipped the check silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(install): escape the project path in the DNS-fix and MQTT unit renderers install_dns_fix.sh and install_mqtt_bridge.sh substituted __PROJECT_ROOT_DIR__ with the raw path, while the other three renderers go through sed_escape_replacement from lib_systemd_render.sh. A checkout under a path containing `&`, `\` or `|` rendered a corrupted unit from these two only. Both now source the helper and use it, and a test checks that every placeholder substitution in scripts/install uses an escaped value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(install): stop the installer scripts reporting things that are not true - first_time_install.sh printed "Password: ledmatrix123" for the setup access point. wifi_manager creates it as an open network ("No password" on the panel), so it now says so. - Step 10.1 printed "✓ WiFi management permissions configured" straight after its own failure message; install_wifi_monitor.sh printed "✓ Package installation completed" after a failed apt install. The tick now only follows success. - Step 7 printed "Web dependencies already installed ... in Step 5" in the one branch that runs because Step 5 did not install them, then created .web_deps_installed on that basis. It now warns and leaves the marker off so the next run retries, as the comment below it intends. - check_system_compatibility.sh called Debian 12 Bookworm "full compatibility confirmed" while first_time_install.sh refuses anything but Debian 13. Bookworm, older Debian and non-Debian systems are now errors. Its counters used ((X++)), which under `set -e` exits the script at the first warning or error (the expression is 0), so the check never reached its summary on any system with one. - configure_web_sudo.sh and configure_wifi_permissions.sh finished by testing `sudo -n test -f ...` and `sudo -n nmcli device status`, neither of which is granted, so they always reported a failure. They now ask `sudo -n -l` about commands the new rules do grant, which checks the rule without running anything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(install): print the completion summary before rebooting With -y -- and so for every one-shot `curl | bash` install, which always passes -y -- first_time_install.sh ran `reboot` about 180 lines before its "Installation Complete / Web UI Access" summary. reboot returns at once, so the summary printed while the Pi was going down and the SSH session usually dropped before the web UI address could be read. The reboot block moves, unchanged, to the very end of the script. The interactive prompt now also follows the summary. Because the summary now runs before the -y reboot, its one command that could fail under `set -Eeuo pipefail` (the SSID lookup, when nmcli reports a connected device but no active network line) gets `|| true`; a missing SSID was already handled as "SSID unknown". one-shot-install.sh prints its "Next steps" after the installer returns, by which time the reboot is under way, so it now says so, and README's Quick Install mentions the automatic reboot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(scripts): correct wrong comments and messages, drop dead code No behaviour change except the output text noted below. - 2775 is setgid, not the sticky bit (first_time_install.sh Step 3.1, fix_plugin_permissions.sh), and root needs no "PWM hardware access" to plugin files. - The 777 comments in first_time_install.sh Step 3's fallback and fix_assets_permissions.sh said root needs it to write. Root ignores mode bits; the comments now say what 777 actually opens. The 777 itself is unchanged. - apt_remove ends in `|| true`, so Step 12's "Some packages could not be removed" branch could never run; it is gone and the helper stays non-fatal. - detect_web_service_user's comment named Step 8 for the web unit (install_service.sh installs it in Step 7.5) and now says which branch actually runs. - Step 5 described an "already installed" check that does not exist; the ACTUAL_USER comment described the re-exec backwards. - on_error printed a literal "\n" before "Common fixes:". - Dead code: one-shot-install.sh's uncalled fix_tmp_permissions, LEDMATRIX_ELEVATED=1 (never read) on the sudo re-exec, and configure_web_sudo.sh's unused PYTHON_PATH, which also made a missing python3 fatal for rules that never mention it. - start_display.sh / stop_display.sh said "for user: <you>"; the service runs as root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(fix_perms): fix_cache_permissions.sh uses setup_cache.sh's model There were two models for /var/cache/ledmatrix. setup_cache.sh (the installer's Step 2) and install_web_service.sh share it through the ledmatrix group: root:ledmatrix, 2775, files 660, which is also what DiskCache relies on to give files the directory's group. fix_cache_permissions.sh instead made it 777 and re-grouped it to the invoking user's group, undoing that. It now runs setup_cache.sh for /var/cache/ledmatrix and keeps its own handling of ~/.ledmatrix_cache. Dropped: /var/cache/ledmatrix/ placeholder_logos (nothing reads it) and the checks against the `daemon` user (no service runs as daemon). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: pin actions/checkout in the Claude workflows, drop template comments claude.yml and claude-code-review.yml used actions/checkout@v4 while test.yml and release-version-check.yml pin the v4.2.2 commit SHA; they now pin the same SHA. The commented-out starter-template settings (prompt, claude_args, paths, author filter) are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(scripts): index every script and list removal candidates New scripts/README.md gives one line per top-level script and scripts directory, marked keep, dev-only or diagnostic, and lists the eight scripts nothing in the repo refers to as candidates for removal (kept for now). The install, utils and dev READMEs now list the files they were missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: tighten two checks that mutation testing showed were too loose - The wifi sudoers check matched `visudo -c -f "$TEMP_SUDOERS"` in the error report too, so replacing the check with `if false` still passed. It now requires the command as the condition. - The summary test never had the setup access point up, so reinstating the bogus "Password: ledmatrix123" line went unnoticed. A case with hostapd active now checks the AP is described as open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(permissions): describe the repaired fix_perms scripts and new WiFi grants Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): docs-scripts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-24 17:31:41 -04:00
ChuckandClaude Opus 5.5	a231d4dbc7	chore: delete unreferenced scripts and archived docs; fix stale doc claims (#607) ... * chore(scripts): delete unreferenced helper scripts None of these is referenced by an installer, systemd unit, CI workflow, test, the web UI or src/: - utils/cleanup_venv.sh removes venv_web_v2, which nothing creates - utils/clear_python_cache.sh hardcodes ~/LEDMatrix and a .webassets-cache nothing uses - install/migrate_config.sh only copies the template, which the installer and ConfigManager already do - install/debug_install.sh, debug/debug_web_manual.py - diagnose_web_ui.sh and verify_web_ui.sh overlap diagnose_web_interface.sh, which the docs point to - fix_internet_connectivity.sh is iptables-only (stale on nftables) - diagnose_plugin_permissions.sh, dev/validate_python.py - download_nba_logos.py + README_NBA_LOGOS.md: logo_downloader fetches logos on demand - setup_plugin_repos.py linked into the production plugin-repos/ dir; the dev workflow is scripts/dev/dev_plugin_setup.sh, and MULTI_ROOT_WORKSPACE_SETUP.md now uses it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(config): drop unused plugin_system flags and a dead unit comment - config.template.json: remove plugin_system.auto_discover, auto_load_enabled and development_mode. Nothing reads them; the web UI only stores them when a client sends them. ConfigManager's migration only adds template keys, so existing configs keep theirs unchanged. - config.template.json: re-indent vegas_scroll's live_* keys. - systemd/ledmatrix.service: remove the comment documenting LEDMATRIX_ON_DEMAND_PLUGIN / on_demand_env.conf; nothing reads either. - CONFIG_REFERENCE.md: say the legacy keys are no longer in the template. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: delete docs/archive and PLUGIN_IMPLEMENTATION_SUMMARY.md - docs/archive/: superseded guides; the repository history keeps them and no live doc links into the directory. The one open document in it, WEB_UI_AUDIT_2026-09.md, moves to docs/audits/ and is linked from the docs index. - PLUGIN_IMPLEMENTATION_SUMMARY.md invented usage statistics, called v2.0.0 current, listed shipped auto-updates as future work and documented a BasePlugin.get_config() that does not exist. - docs/README.md: drop both, and stop telling contributors to archive obsolete pages instead of deleting them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(plugin-api): fix extra_small_font size, cache metric key and scroll pacing example - PLUGIN_API_REFERENCE: extra_small_font loads at 7, not 6 (crisp_size snaps it, src/display_manager.py); get_cache_metrics() returns cache_hit_rate, not hit_rate (src/cache/cache_metrics.py). - ADVANCED_PLUGIN_DEVELOPMENT: the basic scrolling example slept in a loop and never passed frame_hold; use ScrollHelper + scroll_config.configure() and set_scrolling_state(True, frame_hold=...) as PLUGIN_API_REFERENCE does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(plugin-config): match the config tab, icon and web-action docs to the code - PLUGIN_CONFIG_QUICK_START / PLUGIN_CONFIGURATION_TABS / PLUGIN_CONFIGURATION_GUIDE: there is no "Reset to Defaults" button (the tab has Refresh, Update, Uninstall, Save Configuration); plugin config hot-reloads (ConfigService + on_config_change), so no restart; the schema is found by the fixed name config_schema.json, not a manifest config_schema field; the tab row is "Plugin Manager", not "Plugins"; forms are server-rendered from /v3/partials/plugin-config/<id>; the duration hook is get_display_duration()/display_duration; a class_name mismatch raises PluginError; the store requires id, name, class_name and display_modes (not version); plugin_system.debug/log_level do not exist (use run.py -d / LEDMATRIX_DEBUG). Drop "future" features that shipped. - PLUGIN_CONFIG_CORE_PROPERTIES: list all of CORE_PLUGIN_PROPERTIES, including skin, skin_options and the vegas_* tuning keys. - PLUGIN_CUSTOM_ICONS: icon is only a Font Awesome class (fallback fa-puzzle-piece); emoji/URL icons and getPluginIcon() never existed in v3. Note that /api/v3/plugins/installed currently omits icon. - PLUGIN_WEB_UI_ACTIONS (+ example JSON): success_message, error_message and step1_message are never read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(store): describe the monorepo registry and the store UI as they are - PLUGIN_STORE_GUIDE: the Plugin Store is a section of the Plugin Manager tab; URL installs are "Install from GitHub" -> "Install Single Plugin"; bulk update exists (Check & Update All) plus opt-in weekly auto-update; PluginStoreManager() defaults to plugins/, so the Python examples pass plugin-repos; registry plugins are downloaded (GitHub API, ZIP fallback), not cloned; updates compare version with latest_version. - PLUGIN_REGISTRY_SETUP_GUIDE: replace the per-plugin-repo + tag walkthrough with a short page on the monorepo registry (plugin_path, latest_version, update_registry.py) that points at the monorepo's own SUBMISSION.md. Drops the reference to the deleted PLUGIN_IMPLEMENTATION_SUMMARY.md and setup_plugin_repos.py. - plugin_registry_template.json: use the real entry shape. - PLUGIN_QUICK_REFERENCE: automatic background updates exist (opt-in); registry example and publishing steps use the monorepo, not tags. - PLUGIN_DEVELOPMENT_GUIDE: tags/releases are not read by the store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(readme): fix the Triple Bonnet mapping, install prerequisites and backup names - README: the Adafruit Triple Bonnet uses `regular` (3 outputs), not `regular-pi1` (1 output) -- src/matrix_support.py MAPPING_OUTPUTS, and the README's own hardware_mapping section; the template default mapping is adafruit-hat, the PWM mod switches it to adafruit-hat-pwm; manual install only needs git up front (first_time_install.sh installs python-dev-is-python3, cmake, ninja-build etc.; cython3/scons are not used); the Pi Zero 2 W is a supported low-memory board, consistent with PRODUCT.md, LOW_MEMORY_BOARDS.md and the installer's low-memory build; fix the "First_time_install.sh" spelling, an orphan "2." list item and the hello-world starter link (it lives in the plugins monorepo). - CONFIG_DEBUGGING: automatic backups are config/backups/config.json.backup.<YYYYMMDD_HHMMSS_ffffff> (five kept), not config_YYYYMMDD_HHMMSS.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dev): correct the test-running and rgbmatrix build instructions - HOW_TO_RUN_TESTS: coverage is not collected by a plain pytest run and pytest.ini has no threshold; the only one is --cov-fail-under=52 in the core unit-test job of .github/workflows/test.yml, which runs the whole test/ tree (not an allowlist). Almost no tests carry markers, so -m integration / -m slow select nothing; drop them and -m unit as the quick check. Replace the hardcoded /home/chuck path. - DEVELOPMENT: the rgbmatrix package is built with pip install . from the submodule root (scikit-build-core + CMake + Ninja), as first_time_install.sh does; there is no make build-python / bindings/python step, and the build deps are python-dev-is-python3, cmake and ninja-build, not cython3/scons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(wifi): the setup AP is open; auto-enable can be turned off without code changes - WIFI_NETWORK_SETUP / SSH_UNAVAILABLE_AFTER_INSTALL: both AP paths in src/wifi_manager.py create an open network and nothing reads ap_password, so drop the "ledmatrix123" password and the ap_password key/advice. - SSH_UNAVAILABLE_AFTER_INSTALL: disabling automatic AP mode does not need code changes -- auto_enable_ap_mode is a WiFi-tab toggle and POST /api/v3/wifi/ap/auto-enable; note the monitor daemon reads wifi_config.json at start, so restart it after changing the setting. Use the ledpi username and a relative install path like the other docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(reference): add auto_update, drop drifted line numbers, fix UI and service details - CONFIG_REFERENCE: document the top-level auto_update.enabled key (read by web_interface/auto_update.py and src/auto_update_setup.py); replace drifted file:line references with function names; the template's dim_schedule mode is "global". - ADVANCED_FEATURES: core does not read a per-plugin background_service block (the sports plugins read their own), and priority is "higher number = higher priority" on FetchRequest but not used for ordering. - WEB_INTERFACE_GUIDE: the General tab toggle is "Web Display Autostart" (web interface service), brightness is 1-100, and config paths are relative to the LEDMatrix folder, not /config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: drop references to code removed in #608 get_installed_plugin_info, WiFiManager's saved_networks and the six always-skipping plugin test files are deleted there. NetworkManager already remembers joined networks; LEDMatrix no longer stores WiFi passwords. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: don't link SKIN_SYSTEM.md from the core-properties page #615 deletes SKIN_SYSTEM.md; with this link, whichever of the two merged second would break test_doc_links. The skin/skin_options entries go when #615 removes the keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-23 12:36:38 -04:00
ChuckandClaude Opus 5	116abb0daa	fix: September 16 core audit — partial saves, asset path safety, auto-update, display settings the library refuses, scroll speed (#595) ... * fix(sports): share the ESPN rejected-range memo with the background service BackgroundDataService always sent a season range first and, on a 400, fell back to chunks without recording the rejection, so every background season fetch spent a doomed request and live scoreboards learned nothing from it (or it from them). The worker now consults and sets the same 6-hour memo fetch_espn_scoreboard() uses: a known rejection goes straight to month/day chunks, and if every chunk fails the range is asked once for a real error without re-spending the chunks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): keep plugin asset and action routes inside their directories POST /plugins/assets/upload, GET /plugins/assets/list and POST /plugins/assets/delete joined the request's plugin_id onto assets/plugins unchecked, so '../../config' created, wrote, listed and deleted outside it. #561 guarded only the route that serves the files. All three now go through path_safety.resolve_under and answer 400 for anything but a plain name, and delete only unlinks a metadata path that resolves into that plugin's uploads directory. PluginManager.get_plugin_directory refuses ids that are not one plain path segment, so /plugins/action (which runs a manifest script from the returned directory) and every other caller get the guard; the action route also rejects such ids up front, covering its no-manager fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): report a no-op plugin update as already up to date update_plugin() returns True both for a real update and for "nothing to do" (a ZIP-installed monorepo plugin already at the registry version, a bundled plugin). With no git commit to compare, POST /plugins/update called every such success "updated successfully", so Check & Update All counted most official plugins as updated on every run. The route now reads what changed off the plugin itself (commit, else manifest version, else last_updated) and returns data.update_status (updated / up_to_date / local_only). The update-all toast is summarised by PluginInstallManager.summarizeUpdateResults from that status, falling back to the message for older servers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(sports): scoreboard scroll speed no longer follows target_fps sports_scroll computed the crisp speed ladder against the global target_fps whenever limit_refresh_rate_hz was the 100 Hz default. Since frame-locked presentation (#545) the helper steps a fixed number of whole pixels per presented frame and the panel presents at its real refresh, so the General tab's "Scroll Frame Rate" became a speed multiplier: 60 ran a 50 px/s scoreboard at 100 px/s, 200 ran it at 25 px/s. The ladder now uses the display manager's refresh_hz, then display.hardware.limit_refresh_rate_hz, then the default. target_fps is not consulted. Docstrings now say scroll_delay is ignored for pacing (no behaviour change there) and describe the fixed-step model. Tests: replace the tests that pinned target_fps as the ladder refresh and described time-based stepping; assert speed independence from target_fps (unit and end-to-end presented px/s against the real helper), that the fixed per-frame step is applied, and that scroll_delay does not change speed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): escape registry and upload values in plugin manager inline handlers The store, saved-repository and custom-registry buttons built onclick='...(${JSON.stringify(id)})...'. JSON.stringify leaves ' alone, so a custom registry entry whose id contained ' closed the attribute and added its own handler. One helper, jsStringAttr(), now HTML-escapes the JSON literal for every one of those handlers, and the store View button opens only http(s) repo links. The live window.updateImageList (plugins_manager.js loads last, so its copy wins over the file-upload widget's) wrote the uploaded file's original name, path and ids into markup raw; they are escaped now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): note plugin asset, action and inline handler guards Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): let the root pip wrapper install web_interface/requirements.txt Update Code, the automatic update's health check and Install Base Requirements install web_interface/requirements.txt through safe_pip_install.sh, which only allowed the root requirements.txt. The first commit changing that file would fail its dependency install, and the automatic updater rolls back any update whose dependencies did not install -- on every device, for every newer commit. The wrapper now lists both core requirement files. Only their folders are resolved, so a requirements.txt symlinked out of the project is compared by its target and refused (previously the root file's own symlink target was what got allowed). The updater's file list is a named constant, and a test runs the real wrapper (pip stubbed) on every file Update Code and the rollback install. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): do not retry plugin requests that got an HTTP answer PluginAPI.request wrapped everything that was not a structured error as NETWORK_ERROR: a proxy's 502 HTML page (response.json() throws) and a JSON error without error_code included. Check & Update All retries NETWORK_ERROR, so those updates were re-sent five more times with backoff, contrary to the #587 contract that an HTTP error response is the server's answer. NETWORK_ERROR now means only that fetch() rejected. Any HTTP response without an error_code, or with a body that is not JSON, is API_ERROR with the HTTP status attached. Tested against the shipped api_client.js. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scroll): restart the stats window when an idle gap is dropped by size #582 dropped an idle gap from the frame stats two ways: the reset_scroll() sentinel, which also restarts the 5s window timer, and a size guard for scrollers that never call reset_scroll(), which did not. On that path the first real frame after the gap found the boundary overdue and logged a stats line for a one-frame window. Both paths now share one seeding helper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): leave plugins alone when update_core's own rollback fails update_core returns rollback_failed directly when a partial pull or an update whose health check never started cannot be rolled back. run() only held plugins back for 'verifying', so those devices still got new plugin versions and a display restart on top of a core in an unknown state -- the opposite of what the health-check path does, and of the 3.4.0 changelog (plugins are left alone if the rollback fails). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(api): make the REST reference match the api_v3 package Every documented request body, query parameter and response shape was re-checked against the handlers in web_interface/blueprints/api_v3/. Fixes calls that failed as documented (repo_url, action_id/params, files/image_id, font_file+font_family, ?font=, cache key, auto_enable_ap_mode, plugin limit keys), removes the font-override endpoints dropped in #566, corrects response shapes (plugins/config, plugins/schema, health, metrics, operation history, github-status, fonts/catalog, cache/list, logs, wifi, on-demand, SSE streams), and adds the 26 routes it omitted (backup, system auto-update/git, wifi radio, starlark editor, MQTT bridge, status endpoints, skins). Documents the merge semantics of partial JSON saves to /config/main and /plugins/config and the dim-schedule POST accepting GET's days shape, which land in the same change set. Replaces app.py line numbers and the removed api_v3.py path with file and function names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): remove the General-tab plugin system toggles that did nothing plugin_system.auto_discover, auto_load_enabled and development_mode had General-tab toggles whose help tips promised dormant plugins and verbose logging, but nothing reads them: every enabled plugin is discovered and loaded regardless. Remove the three toggles. The keys stay tolerated in stored configs. The save handler now stores a flag only when a client sends it; treating a missing key as an unchecked box would otherwise rewrite all three to false on every General-tab save, which still posts plugins_directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(scroll): remove dead code left by #523/#570 - Drop the optional scipy.ndimage import and HAS_SCIPY; nothing read them since the numpy blend replaced the scipy path. - Drop ScrollHelper._last_integer_position and frame_time_target, which were written but never read. - Keep target_fps and set_target_fps() but document them as informational: nothing paces off them, yet ledmatrix-elections' test_scroll_pacing.py reads helper.target_fps back and third-party plugins may call the setter. - Fix stale comments: fixed_pixels_per_frame's "use scroll_delay to throttle", set_sub_pixel_scrolling's "default: True", and set_frame_based_scrolling's claim that it steps. The plugins monorepo was grepped for every removed name; none is used. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(fonts): point plugins at plugin_manager.font_manager; drop removed overrides UI FONT_MANAGER.md told plugins to read display_manager.font_manager, which does not exist, so a plugin following it failed to load with AttributeError. The shared FontManager lives on the PluginManager and BasePlugin._get_font_manager() returns it (with a fallback for harnesses). Also removes the Fonts-tab override workflow and element-override panels that #566 deleted, from FONT_MANAGER.md and WEB_INTERFACE_GUIDE.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(store): search via /plugins/store/list?query=; send Content-Type on registry curls /plugins/store/search does not exist (404) and the list endpoint reads query, not q. The registry guide's curl examples omitted the JSON Content-Type, so the handlers saw an empty body and answered 400. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(config): use the shared core-key list in the last three private copies StartupValidator warned "Plugin 'auto_update' is enabled but not found" on every display start with auto-update or a dim schedule on; the reserved plugin-id check missed auto_update, sync, location and the rest; and ConfigManager's (uncalled) orphan cleanup would have deleted display, schedule and auto_update. All three now read src/core_config_keys.py, which also gains CORE_SECRETS_KEYS for the github/youtube secrets sections. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): partial JSON saves to /config/main change only what they send A JSON body with one field reset every checkbox in the sections it touched: the MQTT bridge's brightness slider turned off disable_hardware_pulsing, inverse_colors, show_refresh_rate and use_short_date_format, and a timezone-only save turned off web-UI autostart and weekly auto-updates. Missing-means-unchecked now applies only to form posts: form-encoded bodies and the v3 forms, which mark themselves with a hidden __form_section input. Also on the config routes: - vegas_min/max_cycle_duration no longer match the generic *_duration rule, so they stop landing in display_durations and a blank one no longer rejects the whole Display save; - saving from the Raw JSON editor calls start_setup_if_needed like the General form, so enabling auto-update there finishes its setup; - the schedule and dim-schedule POSTs accept the per-day days.<day> shape their GETs return, as well as the flat form keys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): install plugin dependencies from the configured plugins directory install_plugin_dependencies.sh scanned only plugins/, but the Plugin Store installs into plugin_system.plugins_directory (default plugin-repos), so the documented "Recommended" fix found 0 plugins on every store install. It now reads plugins_directory from config/config.json (relative to the project root or absolute, default plugin-repos) and also scans plugins/ for dev symlinks, installing a plugin reached through both only once. With set -e alone, `pip ... | tee` took tee's exit status, so a failed pip install was reported as success; set -o pipefail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: replace stale API names, line numbers and the api_v3.py path - ADVANCED_FEATURES: StreamManager methods that exist (get_next_segment, take_next_group, refresh, advance_cycle, ...), and the real on-demand status envelope ({status, data: {state, service}}) - app.py:199 / :144 / :607-619 line citations and web_interface/blueprints/api_v3.py (now a package) replaced with file and function names in ADVANCED_FEATURES, CONFIG_DEBUGGING, PLUGIN_ARCHITECTURE_SPEC, PLUGIN_QUICK_REFERENCE, PLUGIN_CONFIGURATION_TABS, TROUBLESHOOTING and web_interface/README - CONFIG_DEBUGGING: partial /config/main saves change only sent keys; use /config/raw/main to replace the file; describe where validation runs - TROUBLESHOOTING: clear_cache.py needs --clear-all (no args only prints usage) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): verify the web interface that actually ships, on port 5000 verify_installation.sh failed every healthy install: it required the long-removed web_interface_v2.py and looked for a listener on port 5001, while the web interface binds 5000 (web_interface/start.py). It now checks the files ledmatrix-web.service runs (start_web_conditionally.py, web_interface/start.py, app.py) and port 5000. verify_web_ui.sh had the same 5001 port in its listen check, HTTP probe and printed URLs. Port matches are anchored so :50001 no longer counts as :5000. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(plugins): one display-size contract: display_manager.width/height CLAUDE.md (#580) says to read display_manager.width/height because matrix is None when hardware init fails; the development guide, the safety-harness doc and two DisplayManager docstrings still recommended matrix.width/height. The bundled starlark-apps plugin read matrix.width unguarded, so its magnify recommendation and frame scaling raised in fallback mode (e.g. after the Pi 5 hardware refusal). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(install): make install_service.sh --help print usage instead of installing install_service.sh parsed no arguments, so `sudo ./scripts/install/ install_service.sh --help` (presented as harmless in MIGRATION_GUIDE.md) rewrote ledmatrix.service, ledmatrix-web.service and both update-verify units and enabled/started them. It now handles -h/--help (usage, exit 0, no changes) and rejects any other argument with exit 2 before doing anything. Running it with no arguments, as first_time_install.sh does, is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(scroll): describe the fixed-step model and document frame_hold Since #545 a crisp speed from scroll_config.configure() makes the helper advance a fixed whole-pixel step per presented frame with no clock, and the display manager's frame hold is part of the speed. The docs still described the removed wall-clock model: - scroll_config's module and configure() docstrings said speed is applied in time-based mode and that omitting the hold "falls back to fractional pixels"; omitting it actually runs the scroll frame_hold times too fast. - SCROLL_PERFORMANCE.md said ScrollHelper accumulates elapsed time in both modes, and read a 20 ms stats median as missed refreshes although that is a healthy 50 px/s (hold 2) scroll. It now explains the fixed step, the hold-dependent healthy median, that target_fps plays no part, and that a hand-added scroll_pixels_per_second loses to a schema-default pair. - PLUGIN_API_REFERENCE.md documented set_scrolling_state(is_scrolling) without frame_hold; it now documents the parameter (core 3.4.0) with a configure() + set_scrolling_state example. - update_scroll_position/set_scroll_speed and set_scrolling_state docstrings say the same. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(config): mark target_fps legacy; describe what Vegas scroll_delay does - General tab "Scroll Frame Rate" (target_fps) is labelled legacy: after the sports_scroll fix nothing in core scrolling reads it. The field and its API validation stay so saved configs and plugins that read global_config['target_fps'] keep working. CONFIG_REFERENCE says the same. - Vegas frame_based_scrolling/scroll_delay were described as frame-count stepping at ~50 FPS. Neither steps nor sets a frame rate: frame-based mode converts the speed to px per scroll_delay, clamps it to 0.1-5, and still advances by elapsed time, so the applied speed is clamp(scroll_speed * scroll_delay, 0.1, 5) / scroll_delay px/s. The config comments, render_pipeline comment and CONFIG_REFERENCE rows now say so. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(deps): describe how plugin dependencies are really installed The guides said the web service runs as root, that installs pick --user from os.geteuid(), and quoted a warning and a PluginManager._install_plugin_dependencies() method that don't exist. The web unit runs as the installing user; store installs go through install_requirements_file() and sudo safe_pip_install.sh (root), with a user-level fallback that says so, and load-time installs run in the display service's own (root) interpreter. Manual paths now use the configured plugins directory (plugin-repos/ by default) instead of plugins/, which store installs no longer use, and install_plugin_dependencies.sh is described as scanning that directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): count local changes one way for the preflight and the pull The automatic update's preflight ignored mode-only changes and anything whose status line contained plugins/ or plugin-repos/, then promised "Automatic updates will not stash your changes". perform_core_update used plain git status (modes count) and ignored only 'plugins/', then ran 'git stash push -- :!plugins', which nothing ever pops. So an edit to a bundled plugin under plugin-repos/, or the installer's chmods on tracked scripts, passed the preflight and was stashed away for good. - auto_update.local_changes() is the one predicate both use: core.fileMode=false, porcelain -z, and plugins/ and plugin-repos/ excluded by leading folder rather than substring (a core file under web_interface/static/v3/js/plugins/ now counts). - Update Code's explicit stash leaves out both plugin folders; the pull's --autostash carries their edits and mode changes across and reapplies them. - The automatic updater calls perform_core_update(stash_local_changes= False), which refuses instead of stashing edits that appeared after the preflight; update_core reports that as 'blocked'. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): diagnostics follow the web autostart default and api_v3 package #556 made a missing web_display_autostart mean "start" (only an explicit false/off keeps the web interface down), but the diagnostics still said otherwise: diagnose_web_ui.sh reported a missing key as "defaults to false", diagnose_web_interface.sh said the web interface "will not start unless this is set to true" and recommended enabling it, and debug_web_manual.py printed False. Troubleshooting a down web UI pointed users at a non-cause. Both shell scripts now evaluate the setting with the launcher's own autostart_enabled() (inline fallback if it cannot be imported) and report on / off / not set (on) / unparseable config; debug_web_manual.py uses the same function. They also check web_interface/blueprints/api_v3/ __init__.py: api_v3.py became a package in #553, so every healthy checkout was reported as missing a file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(install): what install_service.sh installs; verify script port; no sudo for --help install_service.sh installs and starts ledmatrix, ledmatrix-web and the update-verify units, not only ledmatrix.service (systemd/README.md, README.md). MIGRATION_GUIDE presented 'sudo install_service.sh --help' as a harmless check; it now shows --help without sudo and warns what a real run does. SSH_UNAVAILABLE_AFTER_INSTALL: verify_installation.sh checks the web interface on port 5000. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): note update-all, plugin system settings and script fixes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(display): size the preview after orientation and pixel mappers display_geometry.physical_size claimed to give DisplayManager's answer but only computed cols*chain x rows*parallel. RGBMatrix.width/height are measured after the library's pixel mappers, so a Rotate:90 / orientation 90 chain previewed 128x32 for a 32x128 panel and a U-mapper chain of four 256x32 for 128x64. Model the built-in mappers' size effect as the pinned lib/pixel-mapper.cc does (Rotate, U-mapper, V-mapper, StackToRow, Remap; Mirror and unknown names leave it alone), and move the orientation composition here so DisplayManager and the preview share it. The module docstring no longer claims the sync handshake uses it; that imports only DEFAULT_CHAIN_LENGTH. Audit finding F18. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(display): refuse settings the rgbmatrix library aborts on, on every board The library answers several settings with a NULL matrix or abort() rather than an error, so the display service crash-looped (Restart=on-failure) instead of reaching fallback mode: rows above 64, chain_length above 255 (uint8_t binding setter, documented as "no upper limit"), a misspelled hardware_mapping, and parallel 2-3 on a single-output mapping, reachable from the Display form on the default adafruit-hat(-pwm) mapping. #586 only guarded the Pi 5 subset. - src/matrix_support.py holds the rules for every board (Options::Validate ranges, binding integer types, mapping names and outputs from lib/hardware-mapping.c) plus the Pi 5 ones, and is the one source of the API's numeric ranges. - DisplayManager checks them before building options and raises MatrixSettingsRefused, so a hand-edited config falls back with a logged, reported reason. Emulator mode only warns. - The config API refuses them with a 400 naming the setting; combinations are checked against stored values but reported only when the request sets a field involved. - The hardware status file gains "cause" (settings/library/forced). The fallback log and Display banner give the Pi 5 rebuild hint only for a library failure instead of rebuild + gpio_slowdown advice for every failure; one Pi 5 slowdown recommendation (1-3, start at 1). - The Display form offers classic/classic-pi1 and orientation 90/270 and renders any other stored mapping selected with a warning, so an unrelated save no longer rewrites them; the API accepts 90/270. Audit findings F03, F16, F19, F21. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(display): library limits, template defaults and Pi 5 slowdown - rows 8-64, chain_length 1-255, parallel limited by the mapping's outputs, classic/classic-pi1 mappings and orientation 90/270 documented. - Defaults are the config.template.json values: config migration adds missing keys from the template, so the listed "code defaults" never applied. - One Raspberry Pi 5 gpio_slowdown recommendation: 1-3 in PIO mode, starting at 1. - Troubleshooting describes the refused-settings fallback, and CHANGELOG corrects the Unreleased "no upper limit" entry. Audit findings F19, F20, F21. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): scroll_speeds.py opens the panel with the service's options --measure and --demo built RGBMatrixOptions from a private copy of the display service's builder that had drifted: gpio_slowdown came from display.hardware (default 2) instead of display.runtime (default 3), and rp1_rio, panel_type, disable_hardware_pulsing, inverse_colors, pixel_mapper_config and orientation were skipped, with different defaults (hardware_mapping "regular", pwm_bits 11). A panel needing a high slowdown was measured -- or garbled -- in a setup the service never drives. The option filling in DisplayManager._setup_matrix moves, unchanged, into DisplayManager.apply_matrix_options(options, config), which _setup_matrix calls and the script reuses (overriding only limit_refresh_rate_hz for --measure). The script now loads the whole config rather than the hardware block. Tests pin the script's options to the service's attribute for attribute. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): scroll_speeds.py recommends keys the resolver honours The ladder ended by telling users to set display_options.scroll_pixels_per_second. scroll_config ranks that key below the scroll_speed + scroll_delay pair, deliberately, and several plugin schemas default the pair into config, so the advised key was silently ignored (a schema-default 1/0.02 pair plus an advised 66 still resolved to 50 px/s). The advice is now the pair that selects the crisp speed exactly (pixels_per_frame every frame_hold/refresh seconds), explains that the pair outranks scroll_pixels_per_second, and gives the scoreboards' per-league scroll_settings.scroll_speed (px/s) form. Tests resolve the printed pair over a schema-default pair and check it lands on the advertised speed and hold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: withdraw the target_fps claim for sports_scroll; fix the Vegas speed formula - SPORTS_UNIFICATION.md still presented honouring global target_fps as sports_scroll's added behaviour and its one user-visible gain; note that it was withdrawn because it had become a speed multiplier. - ADVANCED_FEATURES.md gave Vegas scrolling as (scroll_speed / target_fps) * elapsed; the real rule is scroll_speed px/s by elapsed time, through a 0.1-5 px per scroll_delay clamp when frame_based_scrolling is on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): scroll model fixes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dev): link-github links plugins from the ledmatrix-plugins monorepo link-github <name> cloned https://github.com/ChuckBuilds/ledmatrix-<name>.git, and those per-plugin repositories no longer exist: official plugins are directories in the ledmatrix-plugins monorepo. It now clones (or pulls) the monorepo once into the dev directory, finds plugins/<name>, plugins/ledmatrix-<name> or the plugin whose manifest id is <name>, and links it under its manifest id. With an explicit repo URL it still links a single-repository plugin as before. dev_plugins.json: github_user is honoured again (monorepo owner, e.g. a fork), plus plugins_repo and plugins_branch; github_pattern, which was documented but never read, is dropped and warned about. Ships dev_plugins.json.example and git-ignores dev_plugins.json, both of which the guide promised. Reading JSON falls back to python3 when jq is missing (get_plugin_id silently returned nothing without jq). update/status/list find the git checkout above a monorepo plugin directory (its .git is not in the plugin dir), and update pulls a shared checkout once. status no longer exits 1 when nothing is broken. Docs: PLUGIN_DEVELOPMENT_GUIDE (quick start, link-github, configuration, workflow, store integration, hello-world link, submission), and the nonexistent scripts/git-hooks/pre-push-plugin-version and scripts/bump_plugin_version.py replaced with the real rule: bump the manifest version and run update_registry.py. scripts/dev/README.md and CLAUDE.md updated to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(scripts): monorepo workspace layout; fix_perms and install READMEs MULTI_ROOT_WORKSPACE_SETUP described one sibling repository per plugin; setup_plugin_repos.py links ../ledmatrix-plugins/plugins/* into plugin-repos/ and update_plugin_repos.py pulls only the monorepo, and the workspace file opens LEDMatrix plus ../ledmatrix-plugins. scripts/fix_perms/README.md listed cache directories fix_cache_permissions.sh never touches and a 'ledmatrix' service user that doesn't exist (also in scripts/install/README.md); adds safe_pip_install.sh. install/README: install_service.sh installs the web and update-verify units too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): keep the rollback's pip retries inside the unit time limit The health check reinstalled the previous requirements by trying the next bash path after any failure, including a 600 s pip timeout. Two files, two paths: up to 40 minutes of pip alone, while systemd stops ledmatrix-update-verify.service at TimeoutStartSec=30min -- killing the rollback half-way and leaving the update 'verifying' until the web UI calls it lost. - Like permission_utils.install_requirements_file, only a sudo refusal moves on to the next bash; a pip that ran and failed or timed out is not repeated. The refusal wording is one list (permission_utils.SUDO_REFUSAL_PHRASES), mirrored in the stdlib-only verifier and pinned equal by a test. - All reinstalls in one rollback share a 600 s budget. - WORST_CASE_SECONDS adds up every timeout on the longest path (27.5 min); a test holds it under the unit's TimeoutStartSec and that under the web UI's VERIFY_LOST_SECONDS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(plugins): prepare plugin configs one way for load, saves, GET, hot reload and dev tools Plugin config was prepared differently depending on how it arrived: - JSON POST /plugins/config built a partial body on schema defaults, so {"enabled": true} reset every other setting of the plugin. It now merges onto the stored section first, as the form path already did. - Legacy-boolean normalization (#588) ran only at load: GET /plugins/config returned the raw boolean, posting it back failed validation, and hot reload handed plugins the raw section (a legacy dynamic_duration: true came back as a boolean). schema_manager.prepare_plugin_config (normalize, then defaults) is now used by PluginManager.load_plugin, both save paths, GET, the save notifications and DisplayController's hot-reload callback. - The JSON save's filter kept only enabled/display_duration/live_priority and dropped a submitted skin, skin_options or vegas_* tuning key. There is now one core-owned per-plugin list, schema_manager.CORE_PLUGIN_PROPERTIES, used by validation and by the save filter; PluginManager's CORE_OWNED_CONFIG_KEYS is its vegas subset. - Plugin sections posted to /config/main were stored verbatim, including values /plugins/config rejects. They now go through the same preparation (_prepare_plugin_config_for_save, extracted from save_plugin_config), and a failing section rejects the whole save before anything is written. - dev_server read only top-level defaults and let a schema enabled:false win; build_full_config shallow-merged overrides, dropping sibling defaults; the harness extracted defaults differently from the device. loading.build_config now uses the device's extraction and preparation, and dev_server, check_plugin, render_plugin and the harness all use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(mqtt-bridge): brightness changes apply live and touch nothing else The display service's hot reload applies a saved brightness within a few seconds, and /config/main no longer resets other display settings on a brightness-only JSON body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): automatic update hardening Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(config): rewrite PLUGIN_CONFIG_ARCHITECTURE for the v3 web UI It described web_interface_v2.py and index_v2.html (both gone), client-side form generation, one POST per field with {key, value}, and 'no nested objects'. The v3 UI renders plugin forms server-side from the schema (pages_v3 partial + plugin_config.html macros, nested sections and x-widgets), posts the whole form once, and save_plugin_config() merges onto the stored section, validates, splits x-secret fields and notifies the plugin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(mqtt): brightness saves apply via hot reload and leave other settings alone The bridge README said brightness is applied on the display's next restart; the display controller's config hot reload applies it within seconds. It also now states that the bridge's partial JSON save changes only brightness (the /config/main merge fix in this change set). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): don't log pip's output from the health check's reinstall pip can echo a private index URL with embedded credentials; permission_utils redacts it, the stdlib-only verifier cannot, so it logs the exit code only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(config): mark the plugin_system toggles as unused legacy keys auto_discover, auto_load_enabled and development_mode are read by nothing and leave the General tab in this change set (F40). CONFIG_REFERENCE said they were read by the plugin loader; PLUGIN_CONFIGURATION_GUIDE and the REST reference listed them as live settings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): docs and developer tools group Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): legacy plugin-system toggles no longer count as a General save auto_discover, auto_load_enabled and development_mode have left the General form, so a post carrying only one of them is not a general-settings save and must not treat web_display_autostart and auto_update as unchecked. The plugin_system block itself is left as on main for the branch that reworks it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): config-save and plugin-config preparation fixes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(claude): re-check matrix_support.py rules when the library submodule is bumped Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: address Codacy findings on the core audit PR - plugin_manager.prepare_plugin_config: when the fallback legacy-boolean pass also fails, log a warning instead of a bare except/pass. - api_client.js: request() refuses any endpoint that is not a plain path under /api/v3 ("//host", backslashes, ".." or "." segments, whitespace, control characters) with INVALID_ENDPOINT before calling fetch(), and plugin ids are URL-encoded wherever they are put into a URL (also in the app-shell batch load). - test_update_all.js: pins both against the shipped client. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): check endpoint control characters without a control-character regex Codacy (ESLint no-control-regex, Biome noControlCharactersInRegex) flags the \x00-\x1f range in checkEndpoint's regex. Test the char codes instead; the endpoints refused are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(auto-update): make the seed script executable on disk, not only in the index On Linux Repo.publish() commits with -a, which recorded scripts/run.sh as 100644 upstream because the seed file was never chmod +x. The pull then brought in the same mode the installer chmod had made locally, so installer_chmod saw no mode change left to check. The updater was fine: with the upstream commit at 100755 the --autostash carries the device's chmod across. Verified under Linux (WSL, git 2.43): the old helper fails exactly as CI did, the fixed one passes all 63 tests in the file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-09-17 16:37:29 -04:00
3 people	6812dfe7a6	docs: refresh and correct stale documentation across repo (#306) ... * docs: refresh and correct stale documentation across repo Walked the README and docs/ tree against current code and fixed several real bugs and many stale references. Highlights: User-facing - README.md: web interface install instructions referenced install_web_service.sh at the repo root, but it actually lives at scripts/install/install_web_service.sh. - docs/GETTING_STARTED.md: every web UI port reference said 5050, but the real server in web_interface/start.py:123 binds 5000. Same bug was duplicated in docs/TROUBLESHOOTING.md (17 occurrences). Fixed both. - docs/GETTING_STARTED.md: rewrote tab-by-tab instructions. The doc referenced "Plugin Store", "Plugin Management", "Sports Configuration", "Durations", and "Font Management" tabs - none of which exist. Real tabs (verified in web_interface/templates/v3/base.html) are: Overview, General, WiFi, Schedule, Display, Config Editor, Fonts, Logs, Cache, Operation History, Plugin Manager (+ per-plugin tabs). - docs/GETTING_STARTED.md: removed references to a "Test Display" button (doesn't exist) and "Show Now" / "Stop" plugin buttons. Real controls are "Run On-Demand" / "Stop On-Demand" inside each plugin's tab (partials/plugin_config.html:792). - docs/TROUBLESHOOTING.md: removed dead reference to troubleshoot_weather.sh (doesn't exist anywhere in the repo); weather is now a plugin in ledmatrix-plugins. Developer-facing - docs/PLUGIN_API_REFERENCE.md: documented draw_image() doesn't exist on DisplayManager. Real plugins paste onto display_manager.image directly (verified in src/base_classes/{baseball,basketball,football, hockey}.py). Replaced with the canonical pattern. - docs/PLUGIN_API_REFERENCE.md: documented cache_manager.delete() doesn't exist. Real method is clear_cache(key=None). Updated the section. - docs/PLUGIN_API_REFERENCE.md: added 10 missing BasePlugin methods that the doc never mentioned: dynamic-duration hooks, live-priority hooks, and the full Vegas-mode interface. - docs/PLUGIN_DEVELOPMENT_GUIDE.md: same draw_image fix. - docs/DEVELOPMENT.md: corrected the "Plugin Submodules" section. Plugins are NOT git submodules - .gitmodules only contains rpi-rgb-led-matrix-master. Plugins are installed at runtime into the plugins directory configured by plugin_system.plugins_directory (default plugin-repos/). Both internal links in this doc were also broken (missing relative path adjustment). - docs/HOW_TO_RUN_TESTS.md: removed pytest-timeout from install line (not in requirements.txt) and corrected the test/integration/ path (real integration tests are at test/web_interface/integration/). Replaced the fictional file structure diagram with the real one. - docs/EMULATOR_SETUP_GUIDE.md: clone URL was a placeholder; default pixel_size was documented as 16 but emulator_config.json ships with 5. Index - docs/README.md: rewrote. Old index claimed "16-17 files after consolidation" but docs/ actually has 38 .md files. Four were missing from the index entirely (CONFIG_DEBUGGING, DEV_PREVIEW, PLUGIN_ERROR_HANDLING, STARLARK_APPS_GUIDE). Trimmed the navel-gazing consolidation/statistics sections. Out of scope but worth flagging: - src/plugin_system/resource_monitor.py:343 and src/common/api_helper.py:287 call cache_manager.delete(key) but no such method exists on CacheManager. Both call sites would AttributeError at runtime if hit. Not fixed in this docs PR - either add a delete() shim or convert callers to clear_cache(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix WEB_INTERFACE_GUIDE and WIFI_NETWORK_SETUP WEB_INTERFACE_GUIDE.md - Web UI port: 5050 -> 5000 (4 occurrences) - Tab list was almost entirely fictional. Documented tabs: General Settings, Display Settings, Durations, Sports Configuration, Plugin Management, Plugin Store, Font Management. None of these exist. Real tabs (verified in web_interface/templates/v3/base.html: 935-1000): Overview, General, WiFi, Schedule, Display, Config Editor, Fonts, Logs, Cache, Operation History, plus Plugin Manager and per-plugin tabs in the second nav row. Rewrote the navigation section, the General/Display/Plugin sections, and the Common Tasks walkthroughs to match. - Quick Actions list referenced "Test Display" button (doesn't exist). Replaced with the real button list verified in partials/overview.html:88-152: Start/Stop Display, Restart Display Service, Restart Web Service, Update Code, Reboot, Shutdown. - API endpoints used /api/* paths. The api_v3 blueprint mounts at /api/v3 (web_interface/app.py:144), so the real paths are /api/v3/config/main, /api/v3/system/status, etc. Fixed. - Removed bogus "Sports Configuration tab" walkthrough; sports favorites live inside each scoreboard plugin's own tab now. - Plugin directory listed as /plugins/. Real default is plugin-repos/ (verified in config/config.template.json:130 and display_controller.py:132); plugins/ is a fallback. - Removed "Swipe navigation between tabs" mobile claim (not implemented). WIFI_NETWORK_SETUP.md - 21 occurrences of port 5050 -> 5000. - All /api/wifi/* curl examples used the wrong path. The real wifi API routes are at /api/v3/wifi/* (api_v3.py:6367-6609). Fixed. - ap_password default was documented as "" (empty/open network) but config/wifi_config.json ships with "ledmatrix123". Updated the Quick Start, Configuration table, AP Mode Settings section, and Security Recommendations to match. Also clarified that setting ap_password to "" is the way to make it an open network. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix ADVANCED_FEATURES and REST_API_REFERENCE REST_API_REFERENCE.md - Wrong path: /fonts/delete/<font_family> -> /fonts/<font_family> (verified the real DELETE route in web_interface/blueprints/api_v3.py). - Diffed the documented routes against the real api_v3 blueprint (92 routes vs the 71 documented). Added missing sections: - Error tracking (/errors/summary, /errors/plugin/<id>, /errors/clear) - Health (/health) - Schedule dim/power (/config/dim-schedule GET/POST) - Plugin-specific endpoints (calendar/list-calendars, of-the-day/json/upload+delete, plugins/<id>/static/<path>) - Starlark Apps (12 endpoints: status, install-pixlet, apps CRUD, repository browse/install, upload) - Font preview (/fonts/preview) - Updated table of contents with the new sections. - Added a footer note that the API blueprint mounts at /api/v3 (app.py:144) and that SSE stream endpoints are defined directly on the Flask app at app.py:607-615. ADVANCED_FEATURES.md - Vegas Scroll Mode section was actually accurate (verified all config keys match src/vegas_mode/config.py:15-30). - On-Demand Display section had multiple bugs: - 5 occurrences of port 5050 -> 5000 - All API paths missing /v3 (e.g. /api/display/on-demand/start should be /api/v3/display/on-demand/start) - "Settings -> Plugin Management -> Show Now Button" UI flow doesn't exist. Real flow: open the plugin's tab in the second nav row, click Run On-Demand / Stop On-Demand. - "Python API Methods" section showed controller.show_on_demand() / clear_on_demand() / is_on_demand_active() / get_on_demand_info() — none of these methods exist on DisplayController. The on-demand machinery is all internal (_set_on_demand_*, _activate_on_demand, etc) and is driven through the cache_manager. Replaced the section with a note pointing to the REST API. - All Use Case Examples used the same fictional Python calls. Replaced with curl examples against the real API. - Cache Management section claimed "On-demand display uses Redis cache keys". LEDMatrix doesn't use Redis — verified with grep that src/cache_manager.py has no redis import. The cache is file-based, managed by CacheManager (file at /var/cache/ledmatrix/ or fallback paths). Rewrote the manual recovery section: - Removed redis-cli commands - Replaced cache.delete() Python calls with cache.clear_cache() (the real public method per the same bug already flagged in PLUGIN_API_REFERENCE.md) - Replaced "Settings -> Cache Management" with the real Cache tab - Documented the actual cache directory candidates - Background Data Service section: - Used "nfl_scoreboard" as the plugin id in the example. The real plugin is "football-scoreboard" (handles both NFL and NCAA). Fixed. - "Implementation Status: Phase 1 NFL only / Phase 2 planned" section was severely outdated. The background service is now used by all sports scoreboards (football, hockey, baseball, basketball, soccer, lacrosse, F1, UFC), the odds ticker, and the leaderboard plugin. Replaced with a current "Plugins using the background service" note. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix plugin config + store + dependency docs PLUGIN_STORE_GUIDE.md - 19 occurrences of port 5050 -> 5000 - All API paths missing /v3 (e.g. /api/plugins/install -> /api/v3/plugins/install). Bulk fix. PLUGIN_REGISTRY_SETUP_GUIDE.md - Same port + /api/v3 fixes (3 occurrences each) - "Go to Plugin Store tab" -> "Open the Plugin Manager tab and scroll to the Install from GitHub section" (the real flow for registry setup is the GitHub install section, not the Plugin Store search) PLUGIN_CONFIG_QUICK_START.md - Port 5001 -> 5000 (5001 is the dev_server.py default, not the web UI) - "Plugin Store tab" install flow -> real Plugin Manager + Plugin Store section + per-plugin tab in second nav row - Removed reference to PLUGIN_CONFIG_TABS_SUMMARY.md (archived doc) PLUGIN_CONFIGURATION_TABS.md - "Plugin Management vs Configuration" section confusingly described a "Plugins Tab" that doesn't exist as a single thing. Rewrote to describe the real two-piece structure: Plugin Manager tab (browse, install, toggle) vs per-plugin tabs (configure individual plugins). PLUGIN_DEPENDENCY_GUIDE.md - Port 5001 -> 5000 PLUGIN_DEPENDENCY_TROUBLESHOOTING.md - Wrong port (8080) and wrong UI nav ("Plugin Store or Plugin Management"). Fixed to the real flow. PLUGIN_QUICK_REFERENCE.md - "Plugin Location: ./plugins/ directory" -> default is plugin-repos/ (verified in config/config.template.json:130 and display_controller.py:132). plugins/ is a fallback. - File structure diagram showed plugins/ -> plugin-repos/. - Web UI install flow: "Plugin Store tab" -> "Plugin Manager tab -> Plugin Store section". Also fixed Configure ⚙️ button (doesn't exist) and "Drag and drop reorder" (not implemented). - API examples: replaced ad-hoc Python pseudocode with real curl examples against /api/v3/plugins/* endpoints. Pointed at REST_API_REFERENCE.md for the full list. - "Migration Path Phase 1-5" was a roadmap written before the plugin system shipped. The plugin system is now stable and live. Removed the migration phases as they're history, not a roadmap. - "Quick Migration" section called scripts/migrate_to_plugins.py which doesn't exist anywhere in the repo. Removed. - "Plugin Registry Structure" referenced ChuckBuilds/ledmatrix-plugin-registry which doesn't exist. The real registry is ChuckBuilds/ledmatrix-plugins. Fixed. - "Next Steps" / "Questions to Resolve" sections were pre-implementation planning notes. Replaced with a "Known Limitations" section that documents the actually-real gaps (sandboxing, resource limits, ratings, auto-updates). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix misc remaining docs (architecture, dev quickref, sub-dir READMEs) PLUGIN_ARCHITECTURE_SPEC.md - Added a banner at the top noting this is a historical design doc written before the plugin system shipped. The doc is ~1900 lines with 13 stale /api/plugins/* paths (real is /api/v3/plugins/*), references to web_interface_v2.py (current is app.py), and a Migration Strategy / Implementation Roadmap that's now history. Banner points readers at the current docs (PLUGIN_DEVELOPMENT_GUIDE, PLUGIN_API_REFERENCE, REST_API_REFERENCE) without needing to retrofit every section. PLUGIN_CONFIG_ARCHITECTURE.md - 10 occurrences of /api/plugins/* missing /v3 prefix. Bulk fixed. DEVELOPER_QUICK_REFERENCE.md - cache_manager.delete("key") -> cache_manager.clear_cache("key") with comment noting delete() doesn't exist. Same bug already documented in PLUGIN_API_REFERENCE.md. SSH_UNAVAILABLE_AFTER_INSTALL.md - 4 occurrences of port 5001 -> 5000 in AP-mode and Ethernet/WiFi recovery instructions. PLUGIN_CUSTOM_ICONS_FEATURE.md - Port 5001 -> 5000. CONFIG_DEBUGGING.md - Documented /api/v3/config/plugin/<id> and /api/v3/config/validate endpoints don't exist. Replaced with the real endpoints: /api/v3/config/main, /api/v3/plugins/schema?plugin_id=, /api/v3/plugins/config?plugin_id=. Added a note that validation runs server-side automatically on POST. STARLARK_APPS_GUIDE.md - "Plugins -> Starlark Apps" UI navigation path doesn't exist (5 occurrences). Replaced with the real path: Plugin Manager tab, then the per-plugin Starlark Apps tab in the second nav row. - "Navigate to Plugins" install step -> Plugin Manager tab. web_interface/README.md - Documented several endpoints that don't exist in the api_v3 blueprint: - GET /api/v3/plugins (list) -> /api/v3/plugins/installed - GET /api/v3/plugins/<id> -> doesn't exist - POST /api/v3/plugins/<id>/config -> POST /api/v3/plugins/config - GET /api/v3/plugins/<id>/enable + /disable -> POST /api/v3/plugins/toggle - GET /api/v3/store/plugins -> /api/v3/plugins/store/list - POST /api/v3/store/install/<id> -> POST /api/v3/plugins/install - POST /api/v3/store/uninstall/<id> -> POST /api/v3/plugins/uninstall - POST /api/v3/store/update/<id> -> POST /api/v3/plugins/update - POST /api/v3/display/start/stop/restart -> POST /api/v3/system/action - GET /api/v3/display/status -> GET /api/v3/system/status - Also fixed config/secrets.json -> config/config_secrets.json - Replaced the per-section endpoint duplication with a current real endpoint list and a pointer to docs/REST_API_REFERENCE.md. - Documented that SSE stream endpoints are defined directly on the Flask app at app.py:607-615, not in the api_v3 blueprint. scripts/install/README.md - Was missing 3 of the 9 install scripts in the directory: one-shot-install.sh, configure_wifi_permissions.sh, and debug_install.sh. Added them with brief descriptions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: clarify plugin paths and fix systemd manual install bug PLUGIN_DEVELOPMENT_GUIDE.md - Added a "Plugin directory note" callout near the top explaining the plugins/ vs plugin-repos/ split: - Dev workflow uses plugins/ (where dev_plugin_setup.sh creates symlinks) - Production / Plugin Store uses plugin-repos/ (the configurable default per config.template.json:130) - The plugin loader falls back to plugins/ so dev symlinks are picked up automatically (schema_manager.py:77) - User can set plugins_directory to "plugins" in the General tab if they want both to share a directory CLAUDE.md - The Project Structure section had plugins/ and plugin-repos/ exactly reversed: - Old: "plugins/ - Installed plugins directory (gitignored)" "plugin-repos/ - Development symlinks to monorepo plugin dirs" - Real: plugin-repos/ is the canonical Plugin Store install location and is not gitignored. plugins/* IS gitignored (verified in .gitignore) and is the legacy/dev location used by scripts/dev/dev_plugin_setup.sh. Reversed the descriptions and added line refs. systemd/README.md - "Manual Installation" section told users to copy the unit file directly to /etc/systemd/system/. Verified the unit file in systemd/ledmatrix.service contains __PROJECT_ROOT_DIR__ placeholders that the install scripts substitute at install time. A user following the manual steps would get a service that fails to start with "WorkingDirectory=__PROJECT_ROOT_DIR__" errors. Added a clear warning and a sed snippet that substitutes the placeholder before installing. src/common/README.md - Was missing 2 of the 11 utility modules in the directory (verified with ls): permission_utils.py and cli.py. Added brief descriptions for both. Out-of-scope code bug found while auditing (flagged but not fixed): - scripts/dev/dev_plugin_setup.sh:9 sets PROJECT_ROOT="$SCRIPT_DIR" which resolves to scripts/dev/, not the project root. This means the script's PLUGINS_DIR resolves to scripts/dev/plugins/ instead of the project's plugins/ — confirmed by the existence of scripts/dev/plugins/of-the-day/ from prior runs. Real fix is to set PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)". Not fixing in this docs PR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: flag aspirational/regressed features in plugin docs These docs describe features that exist as documented in the doc but either never wired up or regressed when v3 shipped. Each gets a clear status banner so plugin authors don't waste time chasing features that don't actually work. FONT_MANAGER.md - The "For Plugin Developers / Plugin Font Registration" section documents adding a "fonts" block to manifest.json that gets registered via FontManager.register_plugin_fonts(). The method exists at src/font_manager.py:150 but is **never called from anywhere** in the codebase (verified: zero callers). A plugin shipping a manifest "fonts" block has its fonts silently ignored. Added a status warning and a note about how to actually ship plugin fonts (regular files in the plugin dir, loaded directly). PLUGIN_IMPLEMENTATION_SUMMARY.md - Added a top-level status banner. - Architecture diagram referenced src/plugin_system/registry_manager.py (which doesn't exist) and listed plugins/ as the install location. Replaced with the real file list (plugin_loader, schema_manager, health_monitor, operation_queue, state_manager) and pointed at plugin-repos/ as the default install location. - "Dependency Management: Virtual Environments" — verified there's no per-plugin venv. Removed the bullet and added a note that plugin Python deps install into the system Python environment, with no conflict resolution. - "Permission System: File Access Control / Network Access / Resource Limits / CPU and memory constraints" — none of these exist. There's a resource_monitor.py and health_monitor.py for metrics/warnings, but no hard caps or sandboxing. Replaced the section with what's actually implemented and a clear note that plugins run in the same process with full file/network access. PLUGIN_CUSTOM_ICONS.md and PLUGIN_CUSTOM_ICONS_FEATURE.md - The custom-icon feature was implemented in the v2 web interface via a getPluginIcon() helper in templates/index_v2.html that read the manifest "icon" field. When the v3 web interface was built, that helper wasn't ported. Verified in web_interface/templates/v3/base.html:515 and :774, plugin tab icons are hardcoded to `fas fa-puzzle-piece`. The "icon" field in plugin manifests is currently silently ignored (verified with grep across web_interface/ and src/plugin_system/ — zero non-action- related reads of plugin.icon or manifest.icon). - Added a status banner to both docs noting the regression so plugin authors don't think their custom icons are broken in their own plugin code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix .cursor/ helper docs The .cursor/ directory holds the dev-side helper docs that Cursor and contributors using AI tooling rely on to bootstrap plugin development. Several of them had the same bug patterns as the user-facing docs. .cursor/plugin_templates/QUICK_START.md - "Adding Image Rendering" section showed display_manager.draw_image(image, x=0, y=0). That method doesn't exist on DisplayManager (same bug as PLUGIN_API_REFERENCE.md and PLUGIN_DEVELOPMENT_GUIDE.md). Replaced with the canonical display_manager.image.paste((x,y)) pattern, including the transparency-mask form. .cursor/plugins_guide.md - 10 occurrences of ./dev_plugin_setup.sh — the script lives at scripts/dev/dev_plugin_setup.sh, so anyone copy-pasting these examples gets "command not found". Bulk fixed via sed. - "Test with emulator: python run.py --emulator" — there's no --emulator flag. Replaced with the real options: EMULATOR=true python3 run.py for the full display, or scripts/dev_server.py for the dev preview. - Secrets management section showed a fictional "config_secrets": { "api_key": "my-plugin.api_key" } reference field. Verified in src/config_manager.py:162-172 that secrets are loaded by deep-merging config_secrets.json into the main config. There is no separate reference field — just put the secret under the same plugin namespace and read it from the merged config. Rewrote the section with the real pattern. - "ssh pi@raspberrypi" -> "ssh ledpi@your-pi-ip" (consistent with the rest of LEDMatrix docs which use ledpi as the default user) .cursor/README.md - Same ./dev_plugin_setup.sh -> ./scripts/dev/dev_plugin_setup.sh fix (×6 occurrences via replace_all). - Same "python run.py --emulator" -> "EMULATOR=true python3 run.py" fix. Also added a pointer to scripts/dev_server.py for previewing plugins without running the full display. - "Example Plugins: plugins/hockey-scoreboard/" — the canonical source is the ledmatrix-plugins repo. Installed copies land in plugin-repos/ or plugins/. Updated the line to point at the ledmatrix-plugins repo and explain both local locations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix .cursorrules — the file Cursor auto-loads to learn the API This is the file that Cursor reads to learn how plugin development works. Stale entries here directly mislead AI-assisted plugin authors on every new plugin. Several of the same bug patterns I've been fixing in the user-facing docs were here too. Display Manager section (highest impact) - "draw_image(image, x, y): Draw PIL Image" — that method doesn't exist on DisplayManager. Same bug already fixed in PLUGIN_API_REFERENCE.md, PLUGIN_DEVELOPMENT_GUIDE.md, ledmatrix-stocks/README.md, and .cursor/plugin_templates/QUICK_START.md. Removed the bullet and replaced it with a paragraph explaining the real pattern: paste onto display_manager.image directly, then update_display(). Includes the transparency-mask form. - Added the small_font/centered args to draw_text() since they're the ones that matter most for new plugin authors - Added draw_weather_icon since it's commonly used Cache Manager section - "delete(key): Remove cached value" — there's no delete() method on CacheManager. The real method is clear_cache(key=None) (also removes everything when called without args). Same bug as before. - Added get_cached_data_with_strategy and get_background_cached_data since contributors will hit these when working on sports plugins Plugin System Overview - "loaded from the plugins/ directory" — clarified that the default is plugin-repos/ (per config.template.json:130) with plugins/ as the dev fallback used by scripts/dev/dev_plugin_setup.sh Plugin Development Workflow - ./dev_plugin_setup.sh -> ./scripts/dev/dev_plugin_setup.sh (×2) - Manual setup step "Create directory in plugins/<plugin-id>/" -> plugin-repos/<plugin-id>/ as the canonical location - "Use emulator: python run.py --emulator or ./run_emulator.sh" — the --emulator flag doesn't exist; ./run_emulator.sh isn't at root (it lives at scripts/dev/run_emulator.sh). Replaced with the real options: scripts/dev_server.py for dev preview, or EMULATOR=true python3 run.py for the full emulator path. Configuration Management - "Reference secrets via config_secrets key in main config" — this is the same fictional reference syntax I just fixed in .cursor/plugins_guide.md. Verified in src/config_manager.py:162-172 that secrets are deep-merged into the main config; there's no separate reference field. Replaced with a clear explanation of the deep-merge approach. Code Organization - "plugins/<plugin-id>/" -> the canonical location is plugin-repos/<plugin-id>/ (or its dev-time symlink in plugins/) - "see plugins/hockey-scoreboard/ as reference" — the canonical source for example plugins is the ledmatrix-plugins repo. Updated the pointer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add LICENSE (GPL-3.0) and CONTRIBUTING.md LICENSE - The repository previously had no LICENSE file. The README and every downstream plugin README already reference GPL-3.0 ("same as LEDMatrix project"), but the canonical license text was missing — contributors had no formal record of what they were contributing under, and GitHub couldn't auto-detect the license for the repo banner. - Added the canonical GPL-3.0 text from https://www.gnu.org/licenses/gpl-3.0.txt (verbatim, 674 lines). - Compatibility verified: rpi-rgb-led-matrix is GPL-2.0-or-later (per its COPYING file and README; the "or any later version" clause in lib/*.h headers makes GPL-3.0 distribution legal). CONTRIBUTING.md - The repository had no CONTRIBUTING file. New contributors had to reconstruct the dev setup from DEVELOPMENT.md, PLUGIN_DEVELOPMENT_GUIDE.md, SUBMISSION.md, and the root README. - Added a single page covering: dev environment setup (preview server, emulator, hardware), running tests, PR submission flow, commit message convention, plugin contribution pointer, and the license terms contributors are agreeing to. > Note for the maintainer: I (the AI assistant doing this audit) am > selecting GPL-3.0 because every reference in the existing > documentation already says GPL-3.0 — this commit just makes that > declaration legally binding by adding the actual file. Please > confirm during PR review that GPL-3.0 is what you want; if you > prefer a different license, revert this commit before merging. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add CODE_OF_CONDUCT, SECURITY, PR template; link them from README Tier 1 organizational files that any open-source project at LEDMatrix's maturity is expected to have. None of these existed before. They're additive — no existing content was rewritten. CODE_OF_CONDUCT.md - Contributor Covenant 2.1 (the de facto standard for open-source projects). Mentions both the Discord and the GitHub Security Advisories channel for reporting violations. SECURITY.md - Private vulnerability disclosure flow with two channels: GitHub Security Advisories (preferred) and Discord DM. - Documents the project's known security model as intentional rather than vulnerabilities: no web UI auth, plugins run unsandboxed, display service runs as root for GPIO access, config_secrets.json is plaintext. These match the limitations already called out in PLUGIN_QUICK_REFERENCE.md and the audit flagging from earlier in this PR. - Out-of-scope section points users at upstream (rpi-rgb-led-matrix, third-party plugins) so reports land in the right place. .github/PULL_REQUEST_TEMPLATE.md - 10-line checklist that prompts for the things that would have caught the bugs in this very PR: did you load the changed plugin once, did you update docs alongside code, are there any plugin compatibility implications. - Linked from CONTRIBUTING.md for the full flow. README.md - Added a License section near the bottom (the README previously said nothing about the license despite the project being GPL-3.0). - Added a Contributing section pointing at CONTRIBUTING.md and SECURITY.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Customize bug report template for LEDMatrix hardware The bug_report.md template was the GitHub default and asked "Desktop (OS/Browser/Version)" and "Smartphone (Device/OS)" — neither of which is relevant for a project that runs on a Raspberry Pi with hardware LED panels. A user filing a bug under the old template was giving us none of the information we'd actually need to triage it. Replaced with a LEDMatrix-aware template that prompts for: - Pi model, OS/kernel, panel type, HAT/Bonnet, PWM jumper status, display chain dimensions - LEDMatrix git commit / release tag - Plugin id and version (if the bug is plugin-related) - Relevant config snippet (with redaction reminder for API keys) - journalctl log excerpt with the exact command to capture it - Optional photo of the actual display for visual issues Kept feature_request.md as-is — generic content there is fine. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix bare /api/plugins paths in PLUGIN_CONFIGURATION_TABS Found 5 more bare /api/plugins/* paths in PLUGIN_CONFIGURATION_TABS.md that I missed in the round 2 sweep — they're inside data flow diagrams and prose ("loaded via /api/plugins/installed", etc.) so the earlier grep over Markdown code blocks didn't catch them. Fixed all 5 to use /api/v3/plugins/* (the api_v3 blueprint mount path verified at web_interface/app.py:144). Also added a status banner noting that the "Implementation Details" section references the pre-v3 file layout (web_interface_v2.py, templates/index_v2.html) which no longer exists. The current implementation is in web_interface/app.py, blueprints/api_v3.py, and templates/v3/. Same kind of historical drift I flagged in PLUGIN_ARCHITECTURE_SPEC.md and the PLUGIN_CUSTOM_ICONS_FEATURE doc. The user-facing parts of the doc (Overview, Features, Form Generation Process) are still accurate. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(widgets): list the 20 undocumented built-in widgets The widget registry README documented 3 widgets (file-upload, checkbox-group, custom-feeds) but the directory contains 23 registered widgets total. A plugin author reading this doc would think those 3 were the only built-in options and either reach for a custom widget unnecessarily or settle for a generic text input. Verified the actual list with: grep -h "register('" web_interface/static/v3/js/widgets/*.js \ | sed -E "s|.*register\\('([^']+)'.*|\\1|" | sort -u Added an "Other Built-in Widgets" section after the 3 detailed sections, listing the remaining 20 with one-line descriptions organized by category: - Inputs (6): text-input, textarea, number-input, email-input, url-input, password-input - Selectors (7): select-dropdown, radio-group, toggle-switch, slider, color-picker, font-selector, timezone-selector - Date/time/scheduling (4): date-picker, day-selector, time-range, schedule-picker - Composite/data-source (2): array-table, google-calendar-picker - Internal (2): notification, base-widget Pointed at the .js source files as the canonical source for each widget's exact schema and options — keeps this list low-maintenance since I'm not duplicating each widget's full options table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix README_NBA_LOGOS and PLUGIN_CONFIGURATION_GUIDE scripts/README_NBA_LOGOS.md - "python download_nba_logos.py" — wrong on two counts. The script is at scripts/download_nba_logos.py (not the project root), and "python" is Python 2 on most systems. Replaced all 4 occurrences with "python3 scripts/download_nba_logos.py". - The doc framed itself as the way to set up "the NBA leaderboard". The basketball/leaderboard functionality is now in the basketball-scoreboard and ledmatrix-leaderboard plugins (in the ledmatrix-plugins repo), which auto-download logos on first run. Reframed the script as a pre-population utility for offline / dev use cases. - Bumped the documented Python minimum from 3.7 to 3.9 to match the rest of the project. docs/PLUGIN_CONFIGURATION_GUIDE.md - The "Plugin Manifest" example was missing 3 fields the plugin loader actually requires: id, entry_point, and class_name. A contributor copying this manifest verbatim would get PluginError("No class_name in manifest") at load time — the same loader bug already found in stock-news. Added all three. - The same example showed config_schema as an inline object. The loader expects config_schema to be a file path string (e.g. "config_schema.json") with the actual schema in a separate JSON file — verified earlier in this audit. Fixed. - Added a paragraph explaining the loader's required fields and the case-sensitivity rule on class_name (the bug that broke hello-world's manifest before this PR fixed it). - "Plugin Manager Class" example had the wrong constructor signature: (config, display_manager, cache_manager, font_manager). The real BasePlugin.__init__ at base_plugin.py:53-60 takes (plugin_id, config, display_manager, cache_manager, plugin_manager). A copy-pasted example would TypeError on instantiation. Fixed, including a comment noting which attributes BasePlugin sets up. - Renamed the example class from MyPluginManager to MyPlugin to match the project convention (XxxPlugin / XxxScoreboardPlugin in actual plugins). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs(requirements): document optional dependencies (scipy, psutil, Flask-Limiter) A doc-vs-code crosscheck of every Python import in src/ and web_interface/ against requirements.txt found 3 packages that the code uses but requirements.txt doesn't list. Verified with grep that all 3 are wrapped in try/except blocks with documented fallback paths, so they're optional features rather than missing required deps: - scipy src/common/scroll_helper.py:26 → from scipy.ndimage import shift; HAS_SCIPY flag. Used for sub-pixel interpolation in scrolling. Falls back to a simpler shift algorithm without it. - psutil src/plugin_system/resource_monitor.py:15 → import psutil; PSUTIL_AVAILABLE flag. Used for per-plugin CPU/memory monitoring. Silently no-ops without it. - flask-limiter web_interface/app.py:42-43 → from flask_limiter import Limiter; wrapped at the caller. Used for accidental-abuse rate limiting on the web interface (not security). Web interface starts without rate limiting when missing. These were latent in two ways: 1. A user reading requirements.txt thinks they have the full feature set after `pip install -r requirements.txt`, but they don't get smoother scrolling, plugin resource monitoring, or rate limiting. 2. A contributor who deletes one of the packages from their dev env wouldn't know which feature they just lost — the fallbacks are silent. Added an "Optional dependencies" section at the bottom of requirements.txt with the version constraint, the file:line where each is used, the feature it enables, and the install command. The comment-only format means `pip install -r requirements.txt` still gives the minimal-feature install (preserving current behavior), while users who want the full feature set can copy the explicit pip install commands. Other findings from the same scan that came back as false positives or known issues: - web_interface_v2: dead pattern flagged in earlier iteration (still no real implementation; affects 11+ plugins via the same try/except dead-fallback pattern) - urllib3: comes with `requests` transitively - All 'src.', 'web_interface.', 'rgbmatrix', 'RGBMatrixEmulator' imports: internal modules - base_plugin / plugin_manager / store_manager / mocks / visual_display_manager: relative imports to local modules - freetype: false positive (freetype-py is in requirements.txt under the package name) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix broken file references found by path-existence crosscheck Ran a doc-vs-filesystem crosscheck: extracted every backtick-quoted path with a file extension or known directory prefix from docs/*.md and verified each exists. After filtering false positives (placeholder paths, config keys mistaken for paths, paths inside docs that already have historical-status banners), found 4 real broken references — 3 fixed in docs, 1 fixed by creating the missing file: docs/HOW_TO_RUN_TESTS.md:339 - Claimed ".github/workflows/tests.yml" exists and runs pytest on multiple Python versions in CI. There is no such workflow. The only GitHub Actions file is security-audit.yml (bandit + semgrep). - Pytest runs locally but is NOT gated on PRs. - Replaced the fictional CI section with the actual state and a note explaining how someone could contribute a real test workflow. docs/MIGRATION_GUIDE.md:92 - Referenced scripts/fix_perms/README.md "(if exists)" — the hedge betrays that the writer wasn't sure. The README didn't exist. The 6 scripts in scripts/fix_perms/ were never documented. - Created the missing scripts/fix_perms/README.md from scratch with one-line descriptions of all 6 scripts (fix_assets, fix_cache, fix_plugin, fix_web, fix_nhl_cache, safe_plugin_rm) + when-to-use-each guidance + usage examples. - Updated MIGRATION_GUIDE link to drop the "(if exists)" hedge since the file now exists. docs/FONT_MANAGER.md:376 - "See test/font_manager_example.py for a complete working example" — that file does not exist. Verified by listing test/ directory. - Replaced with a pointer to src/font_manager.py itself and the existing scoreboard base classes in src/base_classes/ that actually use the font manager API in production. Path-existence check methodology: - Walked docs/ recursively, regex-extracted backtick-quoted paths matching either /\.(py|sh|json|yml|yaml|md|txt|service|html|js|css|ttf|bdf|png)/ or paths starting with known directory prefixes (scripts/, src/, config/, web_interface/, systemd/, assets/, docs/, test/, etc.) - Filtered out URLs, absolute paths (placeholders), and paths without slashes (likely not relative refs). - Checked existence relative to project root. - Out of 80 unique relative paths in docs/, 32 didn't exist on disk. Most were false positives (configkeys mistaken for paths, example placeholders like 'assets/myfont.ttf', historical references inside docs that already have status banners). The 4 above were genuine broken refs. This pattern is reusable for future iterations and worth wiring into CI (link checker like lychee, scoped to fenced code paths rather than just markdown links, would catch the same class). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: address CodeRabbit review comments on #306 Reviewed all 12 CodeRabbit comments on PR #306, verified each against the current code, and fixed the 11 valid ones. The 12th finding is a real code bug (cache_manager.delete() calls in api_helper.py and resource_monitor.py) that's already in the planned follow-up code-fix PR, so it stays out of this docs PR. Fixed: .cursor/plugins_guide.md, .cursor/README.md, .cursorrules - I claimed "there is no --emulator flag" in 3 places. Verified in run.py:19-20 that the -e/--emulator flag is defined and functional (it sets os.environ["EMULATOR"]="true" before the display imports). Other docs I didn't touch (.cursor/plugin_templates/QUICK_START.md, docs/PLUGIN_DEVELOPMENT_GUIDE.md) already use the flag correctly. Replaced all 3 wrong statements with accurate guidance that both forms work and explains the CLI flag's relationship to the env var. .cursorrules, docs/GETTING_STARTED.md, docs/WEB_INTERFACE_GUIDE.md, docs/PLUGIN_DEVELOPMENT_GUIDE.md - Four places claimed "the plugin loader also falls back to plugins/". Verified that PluginManager.discover_plugins() (src/plugin_system/plugin_manager.py:154) only scans the configured directory — no fallback. The fallback to plugins/ exists only in two narrower places: store_manager.py:1700-1718 (store install/update/uninstall operations) and schema_manager.py:70-80 (schema lookup for the web UI form generator). Rewrote all four mentions with the precise scope. Added a recommendation to set plugin_system.plugins_directory to "plugins" for the smoothest dev workflow with dev_plugin_setup.sh symlinks. docs/FONT_MANAGER.md - The "Status" warning told plugin authors to use display_manager.font_manager.resolve_font(...) as a workaround for loading plugin fonts. Verified in src/font_manager.py that resolve_font() takes a family name, not a file path — so the workaround as written doesn't actually work. Rewrote to tell authors to load the font directly with PIL or freetype-py in their plugin. - The same section said "the user-facing font override system in the Fonts tab still works for any element that's been registered via register_manager_font()". Verified in web_interface/blueprints/api_v3.py:5404-5428 that /api/v3/fonts/overrides is a placeholder implementation that returns empty arrays and contains "would integrate with the actual font system" comments — the Fonts tab does not have functional integration with register_manager_font() or the override system. Removed the false claim and added an explicit note that the tab is a placeholder. docs/ADVANCED_FEATURES.md:523 - The on-demand section said REST/UI calls write a request "into the cache manager (display_on_demand_config key)". Wrong — verified via grep that api_v3.py:1622 and :1687 write to display_on_demand_request, and display_on_demand_config is only written by the controller during activation (display_controller.py:1195, cleared at :1221). Corrected the key name and added controller file:line references so future readers can verify. docs/ADVANCED_FEATURES.md:803 - "Plugins using the background service" paragraph listed all scoreboard plugins but an orphaned "⏳ MLB (baseball)" bullet remained below from the old version of the section. Removed the orphan and added "baseball/MLB" to the inline list for clarity. web_interface/README.md - The POST /api/v3/system/action action list was incomplete. Verified in web_interface/app.py:1383,1386 that enable_autostart and disable_autostart are valid actions. Added both. - The Plugin Store section was missing GET /api/v3/plugins/store/github-status (verified at api_v3.py:3296). Added it. - The SSE line-range reference was app.py:607-615 but line 619 contains the "Exempt SSE streams from CSRF and add rate limiting" block that's semantically part of the same feature. Extended the range to 607-619. docs/GETTING_STARTED.md - Rows/Columns step said "Columns: 64 or 96 (match your hardware)". The web UI's validation accepts any integer in 16-128. Clarified that 64 and 96 are the common bundled-hardware values but the valid range is wider. Not addressed (out of scope for docs PR): - .cursorrules:184 CodeRabbit comment flagged the non-existent cache_manager.delete() calls in src/common/api_helper.py:287 and src/plugin_system/resource_monitor.py:343. These are real CODE bugs, not doc bugs, and they're the first item in the planned post-docs-refresh code-cleanup PR (see /home/chuck/.claude/plans/warm-imagining-river.md). The docs in this PR correctly state that delete() doesn't exist on CacheManager — the fix belongs in the follow-up code PR that either adds a delete() shim or updates the two callers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Chuck <chuck@example.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-07 21:04:25 -04:00
3 people	8fb2800495	feat: add error detection, monitoring, and code quality improvements (#223) ... * feat: add error detection, monitoring, and code quality improvements This comprehensive update addresses automatic error detection, code quality, and plugin development experience: ## Error Detection & Monitoring - Add ErrorAggregator service for centralized error tracking - Add pattern detection for recurring errors (5+ in 60 min) - Add error dashboard API endpoints (/api/v3/errors/*) - Integrate error recording into plugin executor ## Code Quality - Remove 10 silent `except: pass` blocks in sports.py and football.py - Remove hardcoded debug log paths - Add pre-commit hooks to prevent future bare except clauses ## Validation & Type Safety - Add warnings when plugins lack config_schema.json - Add config key collision detection for plugins - Improve type coercion logging in BasePlugin ## Testing - Add test_config_validation_edge_cases.py - Add test_plugin_loading_failures.py - Add test_error_aggregator.py ## Documentation - Add PLUGIN_ERROR_HANDLING.md guide - Add CONFIG_DEBUGGING.md guide Note: GitHub Actions CI workflow is available in the plan but requires workflow scope to push. Add .github/workflows/ci.yml manually. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address code review issues - Fix GitHub issues URL in CONFIG_DEBUGGING.md - Use RLock in error_aggregator.py to prevent deadlock in export_to_file - Distinguish missing vs invalid schema files in plugin_manager.py - Add assertions to test_null_value_for_required_field test - Remove unused initial_count variable in test_plugin_load_error_recorded - Add validation for max_age_hours in clear_old_errors API endpoint Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Chuck <chuck@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-01-30 10:05:09 -05:00