mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-04 06:15:09 +00:00
fix: September 16 core audit — partial saves, asset path safety, auto-update, display settings the library refuses, scroll speed (#595)
* fix(sports): share the ESPN rejected-range memo with the background service BackgroundDataService always sent a season range first and, on a 400, fell back to chunks without recording the rejection, so every background season fetch spent a doomed request and live scoreboards learned nothing from it (or it from them). The worker now consults and sets the same 6-hour memo fetch_espn_scoreboard() uses: a known rejection goes straight to month/day chunks, and if every chunk fails the range is asked once for a real error without re-spending the chunks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): keep plugin asset and action routes inside their directories POST /plugins/assets/upload, GET /plugins/assets/list and POST /plugins/assets/delete joined the request's plugin_id onto assets/plugins unchecked, so '../../config' created, wrote, listed and deleted outside it. #561 guarded only the route that serves the files. All three now go through path_safety.resolve_under and answer 400 for anything but a plain name, and delete only unlinks a metadata path that resolves into that plugin's uploads directory. PluginManager.get_plugin_directory refuses ids that are not one plain path segment, so /plugins/action (which runs a manifest script from the returned directory) and every other caller get the guard; the action route also rejects such ids up front, covering its no-manager fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): report a no-op plugin update as already up to date update_plugin() returns True both for a real update and for "nothing to do" (a ZIP-installed monorepo plugin already at the registry version, a bundled plugin). With no git commit to compare, POST /plugins/update called every such success "updated successfully", so Check & Update All counted most official plugins as updated on every run. The route now reads what changed off the plugin itself (commit, else manifest version, else last_updated) and returns data.update_status (updated / up_to_date / local_only). The update-all toast is summarised by PluginInstallManager.summarizeUpdateResults from that status, falling back to the message for older servers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(sports): scoreboard scroll speed no longer follows target_fps sports_scroll computed the crisp speed ladder against the global target_fps whenever limit_refresh_rate_hz was the 100 Hz default. Since frame-locked presentation (#545) the helper steps a fixed number of whole pixels per presented frame and the panel presents at its real refresh, so the General tab's "Scroll Frame Rate" became a speed multiplier: 60 ran a 50 px/s scoreboard at 100 px/s, 200 ran it at 25 px/s. The ladder now uses the display manager's refresh_hz, then display.hardware.limit_refresh_rate_hz, then the default. target_fps is not consulted. Docstrings now say scroll_delay is ignored for pacing (no behaviour change there) and describe the fixed-step model. Tests: replace the tests that pinned target_fps as the ladder refresh and described time-based stepping; assert speed independence from target_fps (unit and end-to-end presented px/s against the real helper), that the fixed per-frame step is applied, and that scroll_delay does not change speed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): escape registry and upload values in plugin manager inline handlers The store, saved-repository and custom-registry buttons built onclick='...(${JSON.stringify(id)})...'. JSON.stringify leaves ' alone, so a custom registry entry whose id contained ' closed the attribute and added its own handler. One helper, jsStringAttr(), now HTML-escapes the JSON literal for every one of those handlers, and the store View button opens only http(s) repo links. The live window.updateImageList (plugins_manager.js loads last, so its copy wins over the file-upload widget's) wrote the uploaded file's original name, path and ids into markup raw; they are escaped now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): note plugin asset, action and inline handler guards Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): let the root pip wrapper install web_interface/requirements.txt Update Code, the automatic update's health check and Install Base Requirements install web_interface/requirements.txt through safe_pip_install.sh, which only allowed the root requirements.txt. The first commit changing that file would fail its dependency install, and the automatic updater rolls back any update whose dependencies did not install -- on every device, for every newer commit. The wrapper now lists both core requirement files. Only their folders are resolved, so a requirements.txt symlinked out of the project is compared by its target and refused (previously the root file's own symlink target was what got allowed). The updater's file list is a named constant, and a test runs the real wrapper (pip stubbed) on every file Update Code and the rollback install. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): do not retry plugin requests that got an HTTP answer PluginAPI.request wrapped everything that was not a structured error as NETWORK_ERROR: a proxy's 502 HTML page (response.json() throws) and a JSON error without error_code included. Check & Update All retries NETWORK_ERROR, so those updates were re-sent five more times with backoff, contrary to the #587 contract that an HTTP error response is the server's answer. NETWORK_ERROR now means only that fetch() rejected. Any HTTP response without an error_code, or with a body that is not JSON, is API_ERROR with the HTTP status attached. Tested against the shipped api_client.js. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scroll): restart the stats window when an idle gap is dropped by size #582 dropped an idle gap from the frame stats two ways: the reset_scroll() sentinel, which also restarts the 5s window timer, and a size guard for scrollers that never call reset_scroll(), which did not. On that path the first real frame after the gap found the boundary overdue and logged a stats line for a one-frame window. Both paths now share one seeding helper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): leave plugins alone when update_core's own rollback fails update_core returns rollback_failed directly when a partial pull or an update whose health check never started cannot be rolled back. run() only held plugins back for 'verifying', so those devices still got new plugin versions and a display restart on top of a core in an unknown state -- the opposite of what the health-check path does, and of the 3.4.0 changelog (plugins are left alone if the rollback fails). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(api): make the REST reference match the api_v3 package Every documented request body, query parameter and response shape was re-checked against the handlers in web_interface/blueprints/api_v3/. Fixes calls that failed as documented (repo_url, action_id/params, files/image_id, font_file+font_family, ?font=, cache key, auto_enable_ap_mode, plugin limit keys), removes the font-override endpoints dropped in #566, corrects response shapes (plugins/config, plugins/schema, health, metrics, operation history, github-status, fonts/catalog, cache/list, logs, wifi, on-demand, SSE streams), and adds the 26 routes it omitted (backup, system auto-update/git, wifi radio, starlark editor, MQTT bridge, status endpoints, skins). Documents the merge semantics of partial JSON saves to /config/main and /plugins/config and the dim-schedule POST accepting GET's days shape, which land in the same change set. Replaces app.py line numbers and the removed api_v3.py path with file and function names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): remove the General-tab plugin system toggles that did nothing plugin_system.auto_discover, auto_load_enabled and development_mode had General-tab toggles whose help tips promised dormant plugins and verbose logging, but nothing reads them: every enabled plugin is discovered and loaded regardless. Remove the three toggles. The keys stay tolerated in stored configs. The save handler now stores a flag only when a client sends it; treating a missing key as an unchecked box would otherwise rewrite all three to false on every General-tab save, which still posts plugins_directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(scroll): remove dead code left by #523/#570 - Drop the optional scipy.ndimage import and HAS_SCIPY; nothing read them since the numpy blend replaced the scipy path. - Drop ScrollHelper._last_integer_position and frame_time_target, which were written but never read. - Keep target_fps and set_target_fps() but document them as informational: nothing paces off them, yet ledmatrix-elections' test_scroll_pacing.py reads helper.target_fps back and third-party plugins may call the setter. - Fix stale comments: fixed_pixels_per_frame's "use scroll_delay to throttle", set_sub_pixel_scrolling's "default: True", and set_frame_based_scrolling's claim that it steps. The plugins monorepo was grepped for every removed name; none is used. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(fonts): point plugins at plugin_manager.font_manager; drop removed overrides UI FONT_MANAGER.md told plugins to read display_manager.font_manager, which does not exist, so a plugin following it failed to load with AttributeError. The shared FontManager lives on the PluginManager and BasePlugin._get_font_manager() returns it (with a fallback for harnesses). Also removes the Fonts-tab override workflow and element-override panels that #566 deleted, from FONT_MANAGER.md and WEB_INTERFACE_GUIDE.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(store): search via /plugins/store/list?query=; send Content-Type on registry curls /plugins/store/search does not exist (404) and the list endpoint reads query, not q. The registry guide's curl examples omitted the JSON Content-Type, so the handlers saw an empty body and answered 400. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(config): use the shared core-key list in the last three private copies StartupValidator warned "Plugin 'auto_update' is enabled but not found" on every display start with auto-update or a dim schedule on; the reserved plugin-id check missed auto_update, sync, location and the rest; and ConfigManager's (uncalled) orphan cleanup would have deleted display, schedule and auto_update. All three now read src/core_config_keys.py, which also gains CORE_SECRETS_KEYS for the github/youtube secrets sections. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): partial JSON saves to /config/main change only what they send A JSON body with one field reset every checkbox in the sections it touched: the MQTT bridge's brightness slider turned off disable_hardware_pulsing, inverse_colors, show_refresh_rate and use_short_date_format, and a timezone-only save turned off web-UI autostart and weekly auto-updates. Missing-means-unchecked now applies only to form posts: form-encoded bodies and the v3 forms, which mark themselves with a hidden __form_section input. Also on the config routes: - vegas_min/max_cycle_duration no longer match the generic *_duration rule, so they stop landing in display_durations and a blank one no longer rejects the whole Display save; - saving from the Raw JSON editor calls start_setup_if_needed like the General form, so enabling auto-update there finishes its setup; - the schedule and dim-schedule POSTs accept the per-day days.<day> shape their GETs return, as well as the flat form keys. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): install plugin dependencies from the configured plugins directory install_plugin_dependencies.sh scanned only plugins/, but the Plugin Store installs into plugin_system.plugins_directory (default plugin-repos), so the documented "Recommended" fix found 0 plugins on every store install. It now reads plugins_directory from config/config.json (relative to the project root or absolute, default plugin-repos) and also scans plugins/ for dev symlinks, installing a plugin reached through both only once. With set -e alone, `pip ... | tee` took tee's exit status, so a failed pip install was reported as success; set -o pipefail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: replace stale API names, line numbers and the api_v3.py path - ADVANCED_FEATURES: StreamManager methods that exist (get_next_segment, take_next_group, refresh, advance_cycle, ...), and the real on-demand status envelope ({status, data: {state, service}}) - app.py:199 / :144 / :607-619 line citations and web_interface/blueprints/api_v3.py (now a package) replaced with file and function names in ADVANCED_FEATURES, CONFIG_DEBUGGING, PLUGIN_ARCHITECTURE_SPEC, PLUGIN_QUICK_REFERENCE, PLUGIN_CONFIGURATION_TABS, TROUBLESHOOTING and web_interface/README - CONFIG_DEBUGGING: partial /config/main saves change only sent keys; use /config/raw/main to replace the file; describe where validation runs - TROUBLESHOOTING: clear_cache.py needs --clear-all (no args only prints usage) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): verify the web interface that actually ships, on port 5000 verify_installation.sh failed every healthy install: it required the long-removed web_interface_v2.py and looked for a listener on port 5001, while the web interface binds 5000 (web_interface/start.py). It now checks the files ledmatrix-web.service runs (start_web_conditionally.py, web_interface/start.py, app.py) and port 5000. verify_web_ui.sh had the same 5001 port in its listen check, HTTP probe and printed URLs. Port matches are anchored so :50001 no longer counts as :5000. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(plugins): one display-size contract: display_manager.width/height CLAUDE.md (#580) says to read display_manager.width/height because matrix is None when hardware init fails; the development guide, the safety-harness doc and two DisplayManager docstrings still recommended matrix.width/height. The bundled starlark-apps plugin read matrix.width unguarded, so its magnify recommendation and frame scaling raised in fallback mode (e.g. after the Pi 5 hardware refusal). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(install): make install_service.sh --help print usage instead of installing install_service.sh parsed no arguments, so `sudo ./scripts/install/ install_service.sh --help` (presented as harmless in MIGRATION_GUIDE.md) rewrote ledmatrix.service, ledmatrix-web.service and both update-verify units and enabled/started them. It now handles -h/--help (usage, exit 0, no changes) and rejects any other argument with exit 2 before doing anything. Running it with no arguments, as first_time_install.sh does, is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(scroll): describe the fixed-step model and document frame_hold Since #545 a crisp speed from scroll_config.configure() makes the helper advance a fixed whole-pixel step per presented frame with no clock, and the display manager's frame hold is part of the speed. The docs still described the removed wall-clock model: - scroll_config's module and configure() docstrings said speed is applied in time-based mode and that omitting the hold "falls back to fractional pixels"; omitting it actually runs the scroll frame_hold times too fast. - SCROLL_PERFORMANCE.md said ScrollHelper accumulates elapsed time in both modes, and read a 20 ms stats median as missed refreshes although that is a healthy 50 px/s (hold 2) scroll. It now explains the fixed step, the hold-dependent healthy median, that target_fps plays no part, and that a hand-added scroll_pixels_per_second loses to a schema-default pair. - PLUGIN_API_REFERENCE.md documented set_scrolling_state(is_scrolling) without frame_hold; it now documents the parameter (core 3.4.0) with a configure() + set_scrolling_state example. - update_scroll_position/set_scroll_speed and set_scrolling_state docstrings say the same. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(config): mark target_fps legacy; describe what Vegas scroll_delay does - General tab "Scroll Frame Rate" (target_fps) is labelled legacy: after the sports_scroll fix nothing in core scrolling reads it. The field and its API validation stay so saved configs and plugins that read global_config['target_fps'] keep working. CONFIG_REFERENCE says the same. - Vegas frame_based_scrolling/scroll_delay were described as frame-count stepping at ~50 FPS. Neither steps nor sets a frame rate: frame-based mode converts the speed to px per scroll_delay, clamps it to 0.1-5, and still advances by elapsed time, so the applied speed is clamp(scroll_speed * scroll_delay, 0.1, 5) / scroll_delay px/s. The config comments, render_pipeline comment and CONFIG_REFERENCE rows now say so. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(deps): describe how plugin dependencies are really installed The guides said the web service runs as root, that installs pick --user from os.geteuid(), and quoted a warning and a PluginManager._install_plugin_dependencies() method that don't exist. The web unit runs as the installing user; store installs go through install_requirements_file() and sudo safe_pip_install.sh (root), with a user-level fallback that says so, and load-time installs run in the display service's own (root) interpreter. Manual paths now use the configured plugins directory (plugin-repos/ by default) instead of plugins/, which store installs no longer use, and install_plugin_dependencies.sh is described as scanning that directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): count local changes one way for the preflight and the pull The automatic update's preflight ignored mode-only changes and anything whose status line contained plugins/ or plugin-repos/, then promised "Automatic updates will not stash your changes". perform_core_update used plain git status (modes count) and ignored only 'plugins/', then ran 'git stash push -- :!plugins', which nothing ever pops. So an edit to a bundled plugin under plugin-repos/, or the installer's chmods on tracked scripts, passed the preflight and was stashed away for good. - auto_update.local_changes() is the one predicate both use: core.fileMode=false, porcelain -z, and plugins/ and plugin-repos/ excluded by leading folder rather than substring (a core file under web_interface/static/v3/js/plugins/ now counts). - Update Code's explicit stash leaves out both plugin folders; the pull's --autostash carries their edits and mode changes across and reapplies them. - The automatic updater calls perform_core_update(stash_local_changes= False), which refuses instead of stashing edits that appeared after the preflight; update_core reports that as 'blocked'. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): diagnostics follow the web autostart default and api_v3 package #556 made a missing web_display_autostart mean "start" (only an explicit false/off keeps the web interface down), but the diagnostics still said otherwise: diagnose_web_ui.sh reported a missing key as "defaults to false", diagnose_web_interface.sh said the web interface "will not start unless this is set to true" and recommended enabling it, and debug_web_manual.py printed False. Troubleshooting a down web UI pointed users at a non-cause. Both shell scripts now evaluate the setting with the launcher's own autostart_enabled() (inline fallback if it cannot be imported) and report on / off / not set (on) / unparseable config; debug_web_manual.py uses the same function. They also check web_interface/blueprints/api_v3/ __init__.py: api_v3.py became a package in #553, so every healthy checkout was reported as missing a file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(install): what install_service.sh installs; verify script port; no sudo for --help install_service.sh installs and starts ledmatrix, ledmatrix-web and the update-verify units, not only ledmatrix.service (systemd/README.md, README.md). MIGRATION_GUIDE presented 'sudo install_service.sh --help' as a harmless check; it now shows --help without sudo and warns what a real run does. SSH_UNAVAILABLE_AFTER_INSTALL: verify_installation.sh checks the web interface on port 5000. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): note update-all, plugin system settings and script fixes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(display): size the preview after orientation and pixel mappers display_geometry.physical_size claimed to give DisplayManager's answer but only computed cols*chain x rows*parallel. RGBMatrix.width/height are measured after the library's pixel mappers, so a Rotate:90 / orientation 90 chain previewed 128x32 for a 32x128 panel and a U-mapper chain of four 256x32 for 128x64. Model the built-in mappers' size effect as the pinned lib/pixel-mapper.cc does (Rotate, U-mapper, V-mapper, StackToRow, Remap; Mirror and unknown names leave it alone), and move the orientation composition here so DisplayManager and the preview share it. The module docstring no longer claims the sync handshake uses it; that imports only DEFAULT_CHAIN_LENGTH. Audit finding F18. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(display): refuse settings the rgbmatrix library aborts on, on every board The library answers several settings with a NULL matrix or abort() rather than an error, so the display service crash-looped (Restart=on-failure) instead of reaching fallback mode: rows above 64, chain_length above 255 (uint8_t binding setter, documented as "no upper limit"), a misspelled hardware_mapping, and parallel 2-3 on a single-output mapping, reachable from the Display form on the default adafruit-hat(-pwm) mapping. #586 only guarded the Pi 5 subset. - src/matrix_support.py holds the rules for every board (Options::Validate ranges, binding integer types, mapping names and outputs from lib/hardware-mapping.c) plus the Pi 5 ones, and is the one source of the API's numeric ranges. - DisplayManager checks them before building options and raises MatrixSettingsRefused, so a hand-edited config falls back with a logged, reported reason. Emulator mode only warns. - The config API refuses them with a 400 naming the setting; combinations are checked against stored values but reported only when the request sets a field involved. - The hardware status file gains "cause" (settings/library/forced). The fallback log and Display banner give the Pi 5 rebuild hint only for a library failure instead of rebuild + gpio_slowdown advice for every failure; one Pi 5 slowdown recommendation (1-3, start at 1). - The Display form offers classic/classic-pi1 and orientation 90/270 and renders any other stored mapping selected with a warning, so an unrelated save no longer rewrites them; the API accepts 90/270. Audit findings F03, F16, F19, F21. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(display): library limits, template defaults and Pi 5 slowdown - rows 8-64, chain_length 1-255, parallel limited by the mapping's outputs, classic/classic-pi1 mappings and orientation 90/270 documented. - Defaults are the config.template.json values: config migration adds missing keys from the template, so the listed "code defaults" never applied. - One Raspberry Pi 5 gpio_slowdown recommendation: 1-3 in PIO mode, starting at 1. - Troubleshooting describes the refused-settings fallback, and CHANGELOG corrects the Unreleased "no upper limit" entry. Audit findings F19, F20, F21. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): scroll_speeds.py opens the panel with the service's options --measure and --demo built RGBMatrixOptions from a private copy of the display service's builder that had drifted: gpio_slowdown came from display.hardware (default 2) instead of display.runtime (default 3), and rp1_rio, panel_type, disable_hardware_pulsing, inverse_colors, pixel_mapper_config and orientation were skipped, with different defaults (hardware_mapping "regular", pwm_bits 11). A panel needing a high slowdown was measured -- or garbled -- in a setup the service never drives. The option filling in DisplayManager._setup_matrix moves, unchanged, into DisplayManager.apply_matrix_options(options, config), which _setup_matrix calls and the script reuses (overriding only limit_refresh_rate_hz for --measure). The script now loads the whole config rather than the hardware block. Tests pin the script's options to the service's attribute for attribute. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(scripts): scroll_speeds.py recommends keys the resolver honours The ladder ended by telling users to set display_options.scroll_pixels_per_second. scroll_config ranks that key below the scroll_speed + scroll_delay pair, deliberately, and several plugin schemas default the pair into config, so the advised key was silently ignored (a schema-default 1/0.02 pair plus an advised 66 still resolved to 50 px/s). The advice is now the pair that selects the crisp speed exactly (pixels_per_frame every frame_hold/refresh seconds), explains that the pair outranks scroll_pixels_per_second, and gives the scoreboards' per-league scroll_settings.scroll_speed (px/s) form. Tests resolve the printed pair over a schema-default pair and check it lands on the advertised speed and hold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: withdraw the target_fps claim for sports_scroll; fix the Vegas speed formula - SPORTS_UNIFICATION.md still presented honouring global target_fps as sports_scroll's added behaviour and its one user-visible gain; note that it was withdrawn because it had become a speed multiplier. - ADVANCED_FEATURES.md gave Vegas scrolling as (scroll_speed / target_fps) * elapsed; the real rule is scroll_speed px/s by elapsed time, through a 0.1-5 px per scroll_delay clamp when frame_based_scrolling is on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): scroll model fixes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dev): link-github links plugins from the ledmatrix-plugins monorepo link-github <name> cloned https://github.com/ChuckBuilds/ledmatrix-<name>.git, and those per-plugin repositories no longer exist: official plugins are directories in the ledmatrix-plugins monorepo. It now clones (or pulls) the monorepo once into the dev directory, finds plugins/<name>, plugins/ledmatrix-<name> or the plugin whose manifest id is <name>, and links it under its manifest id. With an explicit repo URL it still links a single-repository plugin as before. dev_plugins.json: github_user is honoured again (monorepo owner, e.g. a fork), plus plugins_repo and plugins_branch; github_pattern, which was documented but never read, is dropped and warned about. Ships dev_plugins.json.example and git-ignores dev_plugins.json, both of which the guide promised. Reading JSON falls back to python3 when jq is missing (get_plugin_id silently returned nothing without jq). update/status/list find the git checkout above a monorepo plugin directory (its .git is not in the plugin dir), and update pulls a shared checkout once. status no longer exits 1 when nothing is broken. Docs: PLUGIN_DEVELOPMENT_GUIDE (quick start, link-github, configuration, workflow, store integration, hello-world link, submission), and the nonexistent scripts/git-hooks/pre-push-plugin-version and scripts/bump_plugin_version.py replaced with the real rule: bump the manifest version and run update_registry.py. scripts/dev/README.md and CLAUDE.md updated to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(scripts): monorepo workspace layout; fix_perms and install READMEs MULTI_ROOT_WORKSPACE_SETUP described one sibling repository per plugin; setup_plugin_repos.py links ../ledmatrix-plugins/plugins/* into plugin-repos/ and update_plugin_repos.py pulls only the monorepo, and the workspace file opens LEDMatrix plus ../ledmatrix-plugins. scripts/fix_perms/README.md listed cache directories fix_cache_permissions.sh never touches and a 'ledmatrix' service user that doesn't exist (also in scripts/install/README.md); adds safe_pip_install.sh. install/README: install_service.sh installs the web and update-verify units too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): keep the rollback's pip retries inside the unit time limit The health check reinstalled the previous requirements by trying the next bash path after any failure, including a 600 s pip timeout. Two files, two paths: up to 40 minutes of pip alone, while systemd stops ledmatrix-update-verify.service at TimeoutStartSec=30min -- killing the rollback half-way and leaving the update 'verifying' until the web UI calls it lost. - Like permission_utils.install_requirements_file, only a sudo refusal moves on to the next bash; a pip that ran and failed or timed out is not repeated. The refusal wording is one list (permission_utils.SUDO_REFUSAL_PHRASES), mirrored in the stdlib-only verifier and pinned equal by a test. - All reinstalls in one rollback share a 600 s budget. - WORST_CASE_SECONDS adds up every timeout on the longest path (27.5 min); a test holds it under the unit's TimeoutStartSec and that under the web UI's VERIFY_LOST_SECONDS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(plugins): prepare plugin configs one way for load, saves, GET, hot reload and dev tools Plugin config was prepared differently depending on how it arrived: - JSON POST /plugins/config built a partial body on schema defaults, so {"enabled": true} reset every other setting of the plugin. It now merges onto the stored section first, as the form path already did. - Legacy-boolean normalization (#588) ran only at load: GET /plugins/config returned the raw boolean, posting it back failed validation, and hot reload handed plugins the raw section (a legacy dynamic_duration: true came back as a boolean). schema_manager.prepare_plugin_config (normalize, then defaults) is now used by PluginManager.load_plugin, both save paths, GET, the save notifications and DisplayController's hot-reload callback. - The JSON save's filter kept only enabled/display_duration/live_priority and dropped a submitted skin, skin_options or vegas_* tuning key. There is now one core-owned per-plugin list, schema_manager.CORE_PLUGIN_PROPERTIES, used by validation and by the save filter; PluginManager's CORE_OWNED_CONFIG_KEYS is its vegas subset. - Plugin sections posted to /config/main were stored verbatim, including values /plugins/config rejects. They now go through the same preparation (_prepare_plugin_config_for_save, extracted from save_plugin_config), and a failing section rejects the whole save before anything is written. - dev_server read only top-level defaults and let a schema enabled:false win; build_full_config shallow-merged overrides, dropping sibling defaults; the harness extracted defaults differently from the device. loading.build_config now uses the device's extraction and preparation, and dev_server, check_plugin, render_plugin and the harness all use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(mqtt-bridge): brightness changes apply live and touch nothing else The display service's hot reload applies a saved brightness within a few seconds, and /config/main no longer resets other display settings on a brightness-only JSON body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): automatic update hardening Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(config): rewrite PLUGIN_CONFIG_ARCHITECTURE for the v3 web UI It described web_interface_v2.py and index_v2.html (both gone), client-side form generation, one POST per field with {key, value}, and 'no nested objects'. The v3 UI renders plugin forms server-side from the schema (pages_v3 partial + plugin_config.html macros, nested sections and x-widgets), posts the whole form once, and save_plugin_config() merges onto the stored section, validates, splits x-secret fields and notifies the plugin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(mqtt): brightness saves apply via hot reload and leave other settings alone The bridge README said brightness is applied on the display's next restart; the display controller's config hot reload applies it within seconds. It also now states that the bridge's partial JSON save changes only brightness (the /config/main merge fix in this change set). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(update): don't log pip's output from the health check's reinstall pip can echo a private index URL with embedded credentials; permission_utils redacts it, the stdlib-only verifier cannot, so it logs the exit code only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(config): mark the plugin_system toggles as unused legacy keys auto_discover, auto_load_enabled and development_mode are read by nothing and leave the General tab in this change set (F40). CONFIG_REFERENCE said they were read by the plugin loader; PLUGIN_CONFIGURATION_GUIDE and the REST reference listed them as live settings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): docs and developer tools group Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): legacy plugin-system toggles no longer count as a General save auto_discover, auto_load_enabled and development_mode have left the General form, so a post carrying only one of them is not a general-settings save and must not treat web_display_autostart and auto_update as unchecked. The plugin_system block itself is left as on main for the branch that reworks it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(changelog): config-save and plugin-config preparation fixes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(claude): re-check matrix_support.py rules when the library submodule is bumped Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: address Codacy findings on the core audit PR - plugin_manager.prepare_plugin_config: when the fallback legacy-boolean pass also fails, log a warning instead of a bare except/pass. - api_client.js: request() refuses any endpoint that is not a plain path under /api/v3 ("//host", backslashes, ".." or "." segments, whitespace, control characters) with INVALID_ENDPOINT before calling fetch(), and plugin ids are URL-encoded wherever they are put into a URL (also in the app-shell batch load). - test_update_all.js: pins both against the shipped client. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(web): check endpoint control characters without a control-character regex Codacy (ESLint no-control-regex, Biome noControlCharactersInRegex) flags the \x00-\x1f range in checkEndpoint's regex. Test the char codes instead; the endpoints refused are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(auto-update): make the seed script executable on disk, not only in the index On Linux Repo.publish() commits with -a, which recorded scripts/run.sh as 100644 upstream because the seed file was never chmod +x. The pull then brought in the same mode the installer chmod had made locally, so installer_chmod saw no mode change left to check. The updater was fine: with the upstream commit at 100755 the --autostash carries the device's chmod across. Verified under Linux (WSL, git 2.43): the old helper fails exactly as CI did, the fixed one passes all 63 tests in the file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Chuck
@@ -1274,8 +1274,13 @@ def _set_missing_booleans_to_false(plugin_config, schema_props, form_keys, prefi
|
||||
)
|
||||
def _enhance_schema_with_core_properties(schema):
|
||||
"""
|
||||
Enhance schema with core plugin properties (enabled, display_duration, live_priority).
|
||||
These properties are system-managed and should always be allowed even if not in the plugin's schema.
|
||||
Enhance schema with the core-owned per-plugin properties.
|
||||
|
||||
``enabled``, ``display_duration``, ``live_priority``, ``skin``,
|
||||
``skin_options`` and the ``vegas_*`` tuning keys are system-managed and
|
||||
always allowed, even when the plugin's schema doesn't declare them. The
|
||||
list is ``schema_manager.CORE_PLUGIN_PROPERTIES``, the one validation uses,
|
||||
so the save filter keeps exactly what validation accepts.
|
||||
|
||||
Args:
|
||||
schema: The original JSON schema dict
|
||||
@@ -1283,44 +1288,31 @@ def _enhance_schema_with_core_properties(schema):
|
||||
Returns:
|
||||
Enhanced schema dict with core properties injected
|
||||
"""
|
||||
import copy
|
||||
from src.plugin_system.schema_manager import with_core_plugin_properties
|
||||
|
||||
if not schema:
|
||||
return schema
|
||||
return with_core_plugin_properties(schema)
|
||||
|
||||
# Core plugin properties that should always be allowed
|
||||
# These match the definitions in SchemaManager.validate_config_against_schema()
|
||||
core_properties = {
|
||||
"enabled": {
|
||||
"type": "boolean",
|
||||
"default": True,
|
||||
"description": "Enable or disable this plugin"
|
||||
},
|
||||
"display_duration": {
|
||||
"type": "number",
|
||||
"default": 15,
|
||||
"minimum": 1,
|
||||
"maximum": 300,
|
||||
"description": "How long to display this plugin in seconds"
|
||||
},
|
||||
"live_priority": {
|
||||
"type": "boolean",
|
||||
"default": False,
|
||||
"description": "Enable live priority takeover when plugin has live content"
|
||||
}
|
||||
}
|
||||
|
||||
# Create a deep copy of the schema to modify (to avoid mutating the original)
|
||||
enhanced_schema = copy.deepcopy(schema)
|
||||
if "properties" not in enhanced_schema:
|
||||
enhanced_schema["properties"] = {}
|
||||
def _prepared_plugin_config(plugin_id, raw_config):
|
||||
"""A plugin's config section as the plugin runs with it, for on_config_change.
|
||||
|
||||
Loading a plugin reads legacy booleans as objects and fills in schema
|
||||
defaults (PluginManager.prepare_plugin_config); a save's notification must
|
||||
hand over the same shape. Falls back to the raw section.
|
||||
"""
|
||||
prepare = getattr(api_v3.plugin_manager, 'prepare_plugin_config', None)
|
||||
if callable(prepare):
|
||||
try:
|
||||
prepared = prepare(plugin_id, raw_config)
|
||||
if isinstance(prepared, dict):
|
||||
return prepared
|
||||
except Exception:
|
||||
logger.debug("Could not prepare config for %s", plugin_id, exc_info=True)
|
||||
return raw_config
|
||||
|
||||
# Inject core properties if they're not already defined in the schema
|
||||
for prop_name, prop_def in core_properties.items():
|
||||
if prop_name not in enhanced_schema["properties"]:
|
||||
enhanced_schema["properties"][prop_name] = copy.deepcopy(prop_def)
|
||||
|
||||
return enhanced_schema
|
||||
def _filter_config_by_schema(config, schema, prefix=''):
|
||||
"""
|
||||
Filter config to only include fields defined in the schema.
|
||||
|
||||
@@ -12,13 +12,37 @@ from web_interface.blueprints.api_v3 import (
|
||||
success_response,
|
||||
)
|
||||
from src.common.path_safety import resolve_under
|
||||
from src.pi5_matrix_support import is_raspberry_pi_5, pi5_unsupported_settings
|
||||
from src.display_geometry import ORIENTATION_ROTATE_DEGREES
|
||||
from src.matrix_support import INT_SETTING_LIMITS, describe_range, library_refusals, refusal_message
|
||||
from src.pi5_matrix_support import is_raspberry_pi_5
|
||||
import web_interface.blueprints.api_v3 as _pkg
|
||||
|
||||
# Read through the module rather than bound by value: tests patch these
|
||||
# as module attributes, and a value binding would not see the patch.
|
||||
# Several are also called from helpers that live in __init__, so the
|
||||
# package is the only patch point that covers every caller.
|
||||
|
||||
#: Hidden input the v3 settings forms (general.html, display.html,
|
||||
#: durations.html) post to /config/main. Its presence tells save_main_config
|
||||
#: that a missing checkbox was unchecked, not merely left out of an API call.
|
||||
FORM_SECTION_FIELD = '__form_section'
|
||||
|
||||
|
||||
def _day_setting(data, day, flat_key, nested_key):
|
||||
"""(present, value) of one per-day schedule setting in a POST body.
|
||||
|
||||
The schedule forms post flat keys (``monday_start``), while GET returns
|
||||
the stored shape, ``days.monday.start_time``. Accept both, so a client can
|
||||
post back what it read; a flat key wins when a body carries both.
|
||||
"""
|
||||
if flat_key in data:
|
||||
return True, data[flat_key]
|
||||
days = data.get('days')
|
||||
day_config = days.get(day) if isinstance(days, dict) else None
|
||||
if isinstance(day_config, dict) and nested_key in day_config:
|
||||
return True, day_config[nested_key]
|
||||
return False, None
|
||||
|
||||
|
||||
@api_v3.route('/config/main', methods=['GET'])
|
||||
def get_main_config():
|
||||
@@ -123,8 +147,8 @@ def save_schedule_config():
|
||||
end_key = f'{day}_end'
|
||||
|
||||
# Check if day is enabled
|
||||
if enabled_key in data:
|
||||
enabled_val = data[enabled_key]
|
||||
has_enabled, enabled_val = _day_setting(data, day, enabled_key, 'enabled')
|
||||
if has_enabled:
|
||||
# Handle checkbox values that may come as 'on', True, or False
|
||||
if isinstance(enabled_val, str):
|
||||
day_config['enabled'] = enabled_val.lower() in ('true', 'on', '1')
|
||||
@@ -140,15 +164,8 @@ def save_schedule_config():
|
||||
start_time = None
|
||||
end_time = None
|
||||
|
||||
if start_key in data and data[start_key]:
|
||||
start_time = data[start_key]
|
||||
else:
|
||||
start_time = '07:00'
|
||||
|
||||
if end_key in data and data[end_key]:
|
||||
end_time = data[end_key]
|
||||
else:
|
||||
end_time = '23:00'
|
||||
start_time = _day_setting(data, day, start_key, 'start_time')[1] or '07:00'
|
||||
end_time = _day_setting(data, day, end_key, 'end_time')[1] or '23:00'
|
||||
|
||||
# Validate time formats
|
||||
is_valid, error_msg = _validate_time_format(start_time)
|
||||
@@ -352,8 +369,8 @@ def save_dim_schedule_config():
|
||||
end_key = f'{day}_end'
|
||||
|
||||
# Check if day is enabled
|
||||
if enabled_key in data:
|
||||
enabled_val = data[enabled_key]
|
||||
has_enabled, enabled_val = _day_setting(data, day, enabled_key, 'enabled')
|
||||
if has_enabled:
|
||||
if isinstance(enabled_val, str):
|
||||
day_config['enabled'] = enabled_val.lower() in ('true', 'on', '1')
|
||||
else:
|
||||
@@ -364,8 +381,8 @@ def save_dim_schedule_config():
|
||||
# Only add times if day is enabled
|
||||
if day_config.get('enabled', True):
|
||||
enabled_days_count += 1
|
||||
start_time = data.get(start_key) or '20:00'
|
||||
end_time = data.get(end_key) or '07:00'
|
||||
start_time = _day_setting(data, day, start_key, 'start_time')[1] or '20:00'
|
||||
end_time = _day_setting(data, day, end_key, 'end_time')[1] or '07:00'
|
||||
|
||||
# Validate time formats
|
||||
is_valid, error_msg = _validate_time_format(start_time)
|
||||
@@ -433,8 +450,10 @@ def save_main_config():
|
||||
|
||||
# Try to get JSON data first, fallback to form data
|
||||
data = None
|
||||
if request.content_type == 'application/json':
|
||||
if request.is_json:
|
||||
data = request.get_json()
|
||||
if data is not None and not isinstance(data, dict):
|
||||
return jsonify({'status': 'error', 'message': 'Request body must be a JSON object'}), 400
|
||||
else:
|
||||
# Handle form data
|
||||
data = request.form.to_dict()
|
||||
@@ -446,6 +465,24 @@ def save_main_config():
|
||||
if not data:
|
||||
return jsonify({'status': 'error', 'message': 'No data provided'}), 400
|
||||
|
||||
# A missing checkbox means different things to the two kinds of caller.
|
||||
# The settings forms post every field, and a browser leaves an
|
||||
# unchecked box out entirely, so for them absent means False. A JSON
|
||||
# API client (the MQTT bridge's brightness slider, a curl call from the
|
||||
# REST docs) sends only what it is changing, and there absent means
|
||||
# "leave it alone": treating it as unchecked turned off
|
||||
# disable_hardware_pulsing and three other settings on every
|
||||
# brightness change, and weekly auto-updates on every timezone change.
|
||||
# The v3 forms post JSON too (htmx json-enc), so they identify
|
||||
# themselves with a hidden FORM_SECTION_FIELD input. A form-encoded
|
||||
# post is a form by definition.
|
||||
is_form_submission = bool(data.pop(FORM_SECTION_FIELD, None)) or not request.is_json
|
||||
|
||||
def _set_checkbox(section, key, field):
|
||||
"""Store checkbox ``field`` as ``section[key]``, if this request sets it."""
|
||||
if is_form_submission or field in data:
|
||||
section[key] = _coerce_to_bool(data.get(field))
|
||||
|
||||
# What arrives here is the config itself, and the headers carry the
|
||||
# session cookie -- neither belongs in the journal, least of all at
|
||||
# ERROR on every save. The shape of the request is the part with
|
||||
@@ -461,15 +498,16 @@ def save_main_config():
|
||||
# Note: Checkboxes don't send data when unchecked, so we need to check if we're updating general settings
|
||||
# If any general setting is present, we're updating the general tab
|
||||
is_general_update = any(k in data for k in ['timezone', 'city', 'state', 'country', 'web_display_autostart',
|
||||
'auto_discover', 'auto_load_enabled', 'development_mode', 'plugins_directory',
|
||||
'auto_update_enabled'])
|
||||
'plugins_directory', 'auto_update_enabled'])
|
||||
|
||||
if is_general_update:
|
||||
# For checkbox: if not present in data during general update, it means unchecked
|
||||
current_config['web_display_autostart'] = _coerce_to_bool(data.get('web_display_autostart'))
|
||||
if not isinstance(current_config.get('auto_update'), dict):
|
||||
current_config['auto_update'] = {}
|
||||
current_config['auto_update']['enabled'] = _coerce_to_bool(data.get('auto_update_enabled'))
|
||||
# For checkbox: if not present in data during a general *form*
|
||||
# update, it means unchecked (see _set_checkbox)
|
||||
_set_checkbox(current_config, 'web_display_autostart', 'web_display_autostart')
|
||||
if is_form_submission or 'auto_update_enabled' in data:
|
||||
if not isinstance(current_config.get('auto_update'), dict):
|
||||
current_config['auto_update'] = {}
|
||||
_set_checkbox(current_config['auto_update'], 'enabled', 'auto_update_enabled')
|
||||
|
||||
if 'timezone' in data:
|
||||
current_config['timezone'] = data['timezone']
|
||||
@@ -520,10 +558,14 @@ def save_main_config():
|
||||
if 'plugin_system' not in current_config:
|
||||
current_config['plugin_system'] = {}
|
||||
|
||||
# Handle plugin system checkboxes - always set to handle unchecked state
|
||||
# HTML checkboxes omit the key when unchecked, so missing key = unchecked = False
|
||||
for checkbox in ['auto_discover', 'auto_load_enabled', 'development_mode']:
|
||||
current_config['plugin_system'][checkbox] = _coerce_to_bool(data.get(checkbox))
|
||||
# auto_discover / auto_load_enabled / development_mode are read by
|
||||
# nothing and no longer have General-tab toggles. The form still
|
||||
# posts plugins_directory, so treating a missing key as an
|
||||
# unchecked box would rewrite stored values to false on every
|
||||
# save; only store what a client actually sends.
|
||||
for legacy_flag in ['auto_discover', 'auto_load_enabled', 'development_mode']:
|
||||
if legacy_flag in data:
|
||||
current_config['plugin_system'][legacy_flag] = _coerce_to_bool(data.get(legacy_flag))
|
||||
|
||||
# Handle plugins_directory
|
||||
if 'plugins_directory' in data:
|
||||
@@ -561,28 +603,20 @@ def save_main_config():
|
||||
return jsonify({'status': 'error', 'message': 'pixel_mapper_config must be a string (e.g. "U-mapper;Rotate:90" or empty)'}), 400
|
||||
|
||||
# Validate orientation (physical mounting rotation; composed onto pixel_mapper_config at runtime)
|
||||
ORIENTATION_ALLOWED = {'normal', '180'}
|
||||
ORIENTATION_ALLOWED = set(ORIENTATION_ROTATE_DEGREES)
|
||||
if 'orientation' in data and data['orientation'] not in ORIENTATION_ALLOWED:
|
||||
return jsonify({'status': 'error', 'message': f"Invalid orientation '{data['orientation']}'. Allowed values: {', '.join(sorted(ORIENTATION_ALLOWED))}"}), 400
|
||||
|
||||
# Panel geometry, PWM and GPIO timing, held to what the rgbmatrix library
|
||||
# accepts (RGBMatrix::Options::Validate in lib/options-initialize.cc,
|
||||
# the gpio_slowdown check in lib/led-matrix.cc). Outside those ranges
|
||||
# the config used to save, then the matrix refused to start and the
|
||||
# display dropped to fallback mode. cols, chain_length and
|
||||
# limit_refresh_rate_hz (0 = no cap) have no upper bound in the
|
||||
# library. rows has none here by choice: the library currently
|
||||
# rejects more than 64 per panel, and that limit is left to it so a
|
||||
# library that lifts it needs no change here.
|
||||
def _hardware_int_error(field, low, high=None, even=False):
|
||||
# Panel geometry, PWM and GPIO timing, held to what the rgbmatrix
|
||||
# library and its Python binding accept (src/matrix_support.py:
|
||||
# Options::Validate, the gpio_slowdown check, and the binding's
|
||||
# uint8_t setters, which cap chain_length at 255). Outside those
|
||||
# ranges the library returns no matrix and the display service
|
||||
# crash-loops rather than falling back, so they never save.
|
||||
def _hardware_int_error(field, low, high, even=False):
|
||||
"""A 400 response if data[field] is not an allowed integer, else None."""
|
||||
raw = data[field]
|
||||
kind = "an even integer" if even else "an integer"
|
||||
if high is None:
|
||||
allowed = f"{kind} of at least {low}"
|
||||
else:
|
||||
allowed = f"{kind} from {low} to {high}"
|
||||
rejection = (jsonify({'status': 'error', 'message': f"Invalid {field} '{raw}'. Must be {allowed}."}), 400)
|
||||
rejection = (jsonify({'status': 'error', 'message': f"Invalid {field} '{raw}'. Must be {describe_range(low, high, even)}."}), 400)
|
||||
# int() would quietly turn true into 1 and 48.5 into 48.
|
||||
if isinstance(raw, bool) or (isinstance(raw, float) and not raw.is_integer()):
|
||||
return rejection
|
||||
@@ -590,34 +624,31 @@ def save_main_config():
|
||||
value = int(raw)
|
||||
except (ValueError, TypeError, OverflowError):
|
||||
return rejection
|
||||
if value < low or (high is not None and value > high) or (even and value % 2):
|
||||
if value < low or value > high or (even and value % 2):
|
||||
return rejection
|
||||
return None
|
||||
|
||||
for field, low, high, even in (('rows', 8, None, True), ('cols', 16, None, False),
|
||||
('chain_length', 1, None, False), ('parallel', 1, 3, False),
|
||||
('brightness', 1, 100, False), ('scan_mode', 0, 1, False),
|
||||
('pwm_bits', 1, 11, False), ('pwm_dither_bits', 0, 2, False),
|
||||
('pwm_lsb_nanoseconds', 50, 3000, False),
|
||||
('limit_refresh_rate_hz', 0, None, False),
|
||||
('row_address_type', 0, 5, False), ('multiplexing', 0, 22, False),
|
||||
('gpio_slowdown', 0, 10, False)):
|
||||
if field in data:
|
||||
# rp1_rio has its own check below.
|
||||
for field, (_section, low, high, even) in INT_SETTING_LIMITS.items():
|
||||
if field in data and field != 'rp1_rio':
|
||||
error = _hardware_int_error(field, low, high, even)
|
||||
if error:
|
||||
return error
|
||||
|
||||
# A Pi 5 can't drive every combination (src/pi5_matrix_support.py),
|
||||
# and one it can't crashes the display service instead of falling
|
||||
# back. Checked only when this request sets one of those fields, so
|
||||
# a combination already stored doesn't block unrelated saves.
|
||||
pi5_fields = ('row_address_type', 'parallel', 'hardware_mapping')
|
||||
if any(k in data for k in pi5_fields) and is_raspberry_pi_5():
|
||||
# Combinations the library can't start with: a hardware mapping it
|
||||
# doesn't have, more parallel chains than the mapping has outputs,
|
||||
# and on a Pi 5 its narrower RP1 support. Reported only when this
|
||||
# request sets one of the settings involved, so a problem already
|
||||
# stored doesn't block unrelated saves.
|
||||
combination_fields = ('hardware_mapping', 'parallel', 'row_address_type')
|
||||
if any(k in data for k in combination_fields):
|
||||
effective = dict(current_config['display']['hardware'])
|
||||
effective.update({k: data[k] for k in pi5_fields if k in data})
|
||||
unsupported = pi5_unsupported_settings(effective)
|
||||
if unsupported:
|
||||
return jsonify({'status': 'error', 'message': unsupported}), 400
|
||||
effective.update({k: v for k, v in data.items()
|
||||
if k in combination_fields or k in INT_SETTING_LIMITS})
|
||||
refusals = [r for r in library_refusals(effective, pi5=is_raspberry_pi_5())
|
||||
if any(f in data for f in r.fields)]
|
||||
if refusals:
|
||||
return jsonify({'status': 'error', 'message': refusal_message(refusals)}), 400
|
||||
|
||||
# Handle hardware settings
|
||||
for field in ['rows', 'cols', 'chain_length', 'parallel', 'brightness', 'hardware_mapping', 'scan_mode',
|
||||
@@ -646,10 +677,10 @@ def save_main_config():
|
||||
|
||||
# Handle checkboxes - coerce to bool to ensure proper JSON types
|
||||
for checkbox in ['disable_hardware_pulsing', 'inverse_colors', 'show_refresh_rate']:
|
||||
current_config['display']['hardware'][checkbox] = _coerce_to_bool(data.get(checkbox))
|
||||
_set_checkbox(current_config['display']['hardware'], checkbox, checkbox)
|
||||
|
||||
# Handle display-level checkboxes (always set to handle unchecked state)
|
||||
current_config['display']['use_short_date_format'] = _coerce_to_bool(data.get('use_short_date_format'))
|
||||
# Handle display-level checkboxes (unchecked state on form saves)
|
||||
_set_checkbox(current_config['display'], 'use_short_date_format', 'use_short_date_format')
|
||||
|
||||
# Handle dynamic duration settings
|
||||
if 'max_dynamic_duration_seconds' in data:
|
||||
@@ -671,8 +702,11 @@ def save_main_config():
|
||||
# checkbox, so when the feature is off we accept the values without
|
||||
# rejecting the whole save — otherwise a stale copies/chain_length
|
||||
# mismatch locks the user out of every other display setting.
|
||||
enabled = _coerce_to_bool(data.get('double_sided_enabled'))
|
||||
ds_config['enabled'] = enabled
|
||||
if is_form_submission or 'double_sided_enabled' in data:
|
||||
enabled = _coerce_to_bool(data.get('double_sided_enabled'))
|
||||
ds_config['enabled'] = enabled
|
||||
else:
|
||||
enabled = _coerce_to_bool(ds_config.get('enabled'))
|
||||
|
||||
def _copies_fits_hardware(copies: int) -> Optional[str]:
|
||||
"""Error message if copies doesn't divide the panel evenly, else None."""
|
||||
@@ -742,15 +776,13 @@ def save_main_config():
|
||||
# Handle enabled checkbox
|
||||
# HTML checkboxes omit the key entirely when unchecked, so if the form
|
||||
# was submitted (any vegas field present) but enabled key is missing,
|
||||
# the checkbox was unchecked and we should set enabled=False
|
||||
vegas_config['enabled'] = _coerce_to_bool(data.get('vegas_scroll_enabled'))
|
||||
vegas_config['auto_trim'] = _coerce_to_bool(data.get('vegas_auto_trim'))
|
||||
vegas_config['dynamic_duration_enabled'] = _coerce_to_bool(
|
||||
data.get('vegas_dynamic_duration_enabled'))
|
||||
vegas_config['continuous_scroll'] = _coerce_to_bool(
|
||||
data.get('vegas_continuous_scroll'))
|
||||
vegas_config['smooth_scroll'] = _coerce_to_bool(
|
||||
data.get('vegas_smooth_scroll'))
|
||||
# the checkbox was unchecked and we should set enabled=False.
|
||||
# A JSON API call only changes the checkboxes it sends.
|
||||
_set_checkbox(vegas_config, 'enabled', 'vegas_scroll_enabled')
|
||||
_set_checkbox(vegas_config, 'auto_trim', 'vegas_auto_trim')
|
||||
_set_checkbox(vegas_config, 'dynamic_duration_enabled', 'vegas_dynamic_duration_enabled')
|
||||
_set_checkbox(vegas_config, 'continuous_scroll', 'vegas_continuous_scroll')
|
||||
_set_checkbox(vegas_config, 'smooth_scroll', 'vegas_smooth_scroll')
|
||||
|
||||
# max_plugin_width_ratio is the one fractional setting, so it is
|
||||
# handled outside the integer loop below.
|
||||
@@ -916,8 +948,12 @@ def save_main_config():
|
||||
# them AGAIN as bogus top-level config keys (e.g. "clock_duration": 30
|
||||
# sitting at config root alongside the correct
|
||||
# display.display_durations.clock_duration).
|
||||
# The Vegas cycle-time fields (vegas_min_cycle_duration, ...) share the
|
||||
# suffix but are Vegas settings, already handled above: counting them
|
||||
# here wrote junk mode durations, and a blank one 400'd the save.
|
||||
duration_fields = [k for k in list(data.keys())
|
||||
if k.endswith('_duration') or k in ('default_duration', 'transition_duration')]
|
||||
if (k.endswith('_duration') and k not in vegas_fields)
|
||||
or k in ('default_duration', 'transition_duration')]
|
||||
if duration_fields:
|
||||
if 'display' not in current_config:
|
||||
current_config['display'] = {}
|
||||
@@ -957,9 +993,14 @@ def save_main_config():
|
||||
current_config['display']['display_durations'][mode_key] = int_value
|
||||
|
||||
# Handle plugin configurations dynamically
|
||||
# Any key that matches a plugin ID should be saved as plugin config
|
||||
# This includes proper secret field handling from schema
|
||||
# Any key that matches a plugin ID is that plugin's settings. They go
|
||||
# through the same preparation as POST /plugins/config -- merged onto
|
||||
# the stored section, legacy booleans and schema defaults applied,
|
||||
# filtered, validated, secrets split out -- so this route can't store
|
||||
# a config that one rejects (stored verbatim, it left the plugin
|
||||
# flagged degraded at its next load).
|
||||
plugin_keys_to_remove = []
|
||||
plugin_secrets_updates = {}
|
||||
# Discovered first: a plugin key not recognised here skips secret
|
||||
# separation below and falls through to the generic merge, which wrote
|
||||
# the plugin's API key into config.json in plain text whenever nothing
|
||||
@@ -969,26 +1010,22 @@ def save_main_config():
|
||||
# Check if this key is a plugin ID
|
||||
if api_v3.plugin_manager and key in plugin_manifests:
|
||||
plugin_id = key
|
||||
plugin_config = data[key]
|
||||
submitted_config = data[key]
|
||||
if not isinstance(submitted_config, dict):
|
||||
return error_response(
|
||||
ErrorCode.VALIDATION_ERROR,
|
||||
f"Settings for plugin '{plugin_id}' must be a JSON object",
|
||||
status_code=400
|
||||
)
|
||||
|
||||
# Load plugin schema to identify secret fields (same logic as save_plugin_config)
|
||||
secret_fields = set()
|
||||
if api_v3.plugin_manager:
|
||||
plugins_dir = api_v3.plugin_manager.plugins_dir
|
||||
else:
|
||||
plugin_system_config = current_config.get('plugin_system', {})
|
||||
plugins_dir_name = plugin_system_config.get('plugins_directory', 'plugin-repos')
|
||||
if os.path.isabs(plugins_dir_name):
|
||||
plugins_dir = Path(plugins_dir_name)
|
||||
else:
|
||||
plugins_dir = PROJECT_ROOT / plugins_dir_name
|
||||
# plugin_id is already known to be a loaded plugin (the
|
||||
# membership test above), so this cannot currently traverse --
|
||||
# but the path is built from a request key, and the guard and
|
||||
# the join are far enough apart that a later edit could
|
||||
# separate them. Build it through the shared helper instead.
|
||||
schema_path = resolve_under(plugins_dir, plugin_id, 'config_schema.json')
|
||||
|
||||
# the schema load are far enough apart that a later edit could
|
||||
# separate them. Refuse rather than save without knowing which
|
||||
# fields are secrets.
|
||||
schema_path = resolve_under(api_v3.plugin_manager.plugins_dir,
|
||||
plugin_id, 'config_schema.json')
|
||||
if schema_path is None:
|
||||
return error_response(
|
||||
ErrorCode.VALIDATION_ERROR,
|
||||
@@ -996,77 +1033,48 @@ def save_main_config():
|
||||
status_code=400
|
||||
)
|
||||
|
||||
if schema_path.exists():
|
||||
try:
|
||||
with open(schema_path, 'r', encoding='utf-8') as f:
|
||||
schema = json.load(f)
|
||||
if 'properties' in schema:
|
||||
secret_fields = find_secret_fields(schema['properties'])
|
||||
except Exception as e:
|
||||
logger.debug("Error reading schema for secret detection: %s", e)
|
||||
schema_mgr = api_v3.schema_manager
|
||||
if not schema_mgr:
|
||||
return error_response(
|
||||
ErrorCode.SYSTEM_ERROR,
|
||||
'Schema manager not initialized',
|
||||
status_code=500
|
||||
)
|
||||
schema = schema_mgr.load_schema(plugin_id, use_cache=False)
|
||||
|
||||
# Separate secrets from regular config (same logic as save_plugin_config)
|
||||
regular_config, secrets_config = separate_secrets(plugin_config, secret_fields)
|
||||
# The config form renders secrets masked, so every save posts
|
||||
# them back blank. Without this the blank is merged over the
|
||||
# stored value and the credential is destroyed by the act of
|
||||
# changing an unrelated setting. A blank means "unchanged".
|
||||
secrets_config = remove_empty_secrets(secrets_config)
|
||||
|
||||
# PRE-PROCESSING: Preserve 'enabled' state if not in regular_config
|
||||
# This prevents overwriting the enabled state when saving config from a form that doesn't include the toggle
|
||||
if 'enabled' not in regular_config:
|
||||
try:
|
||||
if plugin_id in current_config and 'enabled' in current_config[plugin_id]:
|
||||
regular_config['enabled'] = current_config[plugin_id]['enabled']
|
||||
elif api_v3.plugin_manager:
|
||||
# Fallback to plugin instance if config doesn't have it
|
||||
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
|
||||
if plugin_instance:
|
||||
regular_config['enabled'] = plugin_instance.enabled
|
||||
# Final fallback: default to True if plugin is loaded (matches BasePlugin default)
|
||||
if 'enabled' not in regular_config:
|
||||
regular_config['enabled'] = True
|
||||
except Exception as e:
|
||||
logger.debug("Error preserving enabled state: %s", e)
|
||||
# Default to True on error to avoid disabling plugins
|
||||
regular_config['enabled'] = True
|
||||
|
||||
# Get current secrets config
|
||||
current_secrets = api_v3.config_manager.get_raw_file_content('secrets')
|
||||
from web_interface.blueprints.api_v3.plugins import (
|
||||
_merge_onto_stored_plugin_config, _prepare_plugin_config_for_save,
|
||||
)
|
||||
plugin_config = _merge_onto_stored_plugin_config(
|
||||
plugin_id, submitted_config, current_config)
|
||||
regular_config, secrets_config, error = _prepare_plugin_config_for_save(
|
||||
plugin_id, plugin_config, schema, schema_mgr, is_json=True)
|
||||
if error:
|
||||
return error
|
||||
|
||||
# Deep merge regular config into main config
|
||||
if plugin_id not in current_config:
|
||||
current_config[plugin_id] = {}
|
||||
current_config[plugin_id] = deep_merge(current_config[plugin_id], regular_config)
|
||||
|
||||
# Deep merge secrets into secrets config
|
||||
stored_section = current_config.get(plugin_id)
|
||||
current_config[plugin_id] = deep_merge(
|
||||
stored_section if isinstance(stored_section, dict) else {}, regular_config)
|
||||
if secrets_config:
|
||||
if plugin_id not in current_secrets:
|
||||
current_secrets[plugin_id] = {}
|
||||
# Lists merge by replacement, so deep_merge here wrote a
|
||||
# blanked array straight over the stored credentials.
|
||||
current_secrets[plugin_id] = merge_secrets(
|
||||
current_secrets[plugin_id], secrets_config)
|
||||
# Save secrets file
|
||||
api_v3.config_manager.save_raw_file_content('secrets', current_secrets)
|
||||
plugin_secrets_updates[plugin_id] = secrets_config
|
||||
|
||||
# Mark for removal from data dict (already processed)
|
||||
plugin_keys_to_remove.append(key)
|
||||
|
||||
# Notify plugin of config change if loaded (with merged config including secrets)
|
||||
try:
|
||||
if api_v3.plugin_manager:
|
||||
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
|
||||
if plugin_instance:
|
||||
# Reload merged config (includes secrets) and pass the plugin-specific section
|
||||
merged_config = api_v3.config_manager.load_config()
|
||||
plugin_full_config = merged_config.get(plugin_id, {})
|
||||
if hasattr(plugin_instance, 'on_config_change'):
|
||||
plugin_instance.on_config_change(plugin_full_config)
|
||||
except Exception as hook_err:
|
||||
# Don't fail the save if hook fails
|
||||
logger.warning("on_config_change failed: %s", hook_err)
|
||||
# Deep merge secrets into secrets config, once every plugin section
|
||||
# has validated
|
||||
if plugin_secrets_updates:
|
||||
current_secrets = api_v3.config_manager.get_raw_file_content('secrets')
|
||||
for plugin_id, secrets_config in plugin_secrets_updates.items():
|
||||
if plugin_id not in current_secrets:
|
||||
current_secrets[plugin_id] = {}
|
||||
# Lists merge by replacement, so deep_merge here wrote a
|
||||
# blanked array straight over the stored credentials.
|
||||
current_secrets[plugin_id] = merge_secrets(
|
||||
current_secrets[plugin_id], secrets_config)
|
||||
# Save secrets file
|
||||
api_v3.config_manager.save_raw_file_content('secrets', current_secrets)
|
||||
|
||||
# Remove processed plugin keys from data (they're already in current_config)
|
||||
for key in plugin_keys_to_remove:
|
||||
@@ -1115,6 +1123,19 @@ def save_main_config():
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Notify saved plugins of their new config (with secrets merged), now
|
||||
# that it is on disk.
|
||||
for plugin_id in plugin_keys_to_remove:
|
||||
try:
|
||||
plugin_instance = api_v3.plugin_manager.get_plugin(plugin_id)
|
||||
if plugin_instance and hasattr(plugin_instance, 'on_config_change'):
|
||||
merged_config = api_v3.config_manager.load_config()
|
||||
plugin_instance.on_config_change(_pkg._prepared_plugin_config(
|
||||
plugin_id, merged_config.get(plugin_id, {})))
|
||||
except Exception as hook_err:
|
||||
# Don't fail the save if hook fails
|
||||
logger.warning("on_config_change failed: %s", hook_err)
|
||||
|
||||
message = 'Configuration saved successfully'
|
||||
# Switching automatic updates on finishes their setup, which needs
|
||||
# the display service to restart (web_interface/auto_update.py).
|
||||
@@ -1167,10 +1188,27 @@ def save_raw_main_config():
|
||||
if not data:
|
||||
return jsonify({'status': 'error', 'message': 'No data provided'}), 400
|
||||
|
||||
was_auto_update_enabled = False
|
||||
try:
|
||||
previous = api_v3.config_manager.get_raw_file_content('main') or {}
|
||||
was_auto_update_enabled = bool((previous.get('auto_update') or {}).get('enabled'))
|
||||
except Exception:
|
||||
logger.debug("Could not read the previous auto_update setting", exc_info=True)
|
||||
|
||||
# Save the raw config file
|
||||
api_v3.config_manager.save_raw_file_content('main', data)
|
||||
|
||||
return jsonify({'status': 'success', 'message': 'Main configuration saved successfully'})
|
||||
message = 'Main configuration saved successfully'
|
||||
# Same hook as save_main_config: switching automatic updates on here
|
||||
# must finish their setup too, not wait for the next service restart.
|
||||
try:
|
||||
from web_interface import auto_update
|
||||
note = auto_update.start_setup_if_needed(was_auto_update_enabled, data)
|
||||
if note:
|
||||
message = f'{message}. {note}'
|
||||
except Exception:
|
||||
logger.warning("Automatic update setup could not be started", exc_info=True)
|
||||
return jsonify({'status': 'success', 'message': message})
|
||||
except Exception as e:
|
||||
from src.exceptions import ConfigError
|
||||
logger.error("Error saving raw main config", exc_info=True)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -228,14 +228,24 @@ def _sudo_hint_for(text):
|
||||
|
||||
_core_update_lock = threading.Lock()
|
||||
|
||||
#: The core's own requirement files, installed after a pull that changes them.
|
||||
#: scripts/fix_perms/safe_pip_install.sh must accept every one (it refuses
|
||||
#: anything it does not list), and scripts/utils/auto_update_verify.py
|
||||
#: reinstalls the same files when it rolls an update back.
|
||||
CORE_REQUIREMENT_FILES = ('requirements.txt', 'web_interface/requirements.txt')
|
||||
|
||||
def perform_core_update():
|
||||
|
||||
def perform_core_update(stash_local_changes=True):
|
||||
"""Pull the latest LEDMatrix code and sync its dependencies.
|
||||
|
||||
Shared by the Overview "Update Code" button and the weekly automatic
|
||||
updater (web_interface/auto_update.py), so both take exactly the same
|
||||
path. Returns the JSON-able payload the button has always received:
|
||||
``status``, ``message`` and ``restart_required``.
|
||||
|
||||
Update Code stashes local edits before pulling. The automatic updater
|
||||
passes ``stash_local_changes=False``: then local edits make this return
|
||||
an error carrying ``local_changes`` (the edited paths) without pulling.
|
||||
"""
|
||||
# The button and the scheduler can fire together; two pulls racing
|
||||
# over one checkout (and one stash) is how local changes get lost.
|
||||
@@ -243,12 +253,12 @@ def perform_core_update():
|
||||
return {'status': 'error', 'restart_required': False,
|
||||
'message': 'An update is already in progress; try again shortly.'}
|
||||
try:
|
||||
return _perform_core_update_locked()
|
||||
return _perform_core_update_locked(stash_local_changes)
|
||||
finally:
|
||||
_core_update_lock.release()
|
||||
|
||||
|
||||
def _perform_core_update_locked():
|
||||
def _perform_core_update_locked(stash_local_changes=True):
|
||||
project_dir = str(PROJECT_ROOT)
|
||||
|
||||
# Decide how to pull BEFORE stashing. If this checkout cannot be
|
||||
@@ -259,37 +269,35 @@ def _perform_core_update_locked():
|
||||
logger.warning("git pull not attempted: %s", pull_error)
|
||||
return {'status': 'error', 'message': pull_error, 'restart_required': False}
|
||||
|
||||
# Check if there are local changes that need to be stashed
|
||||
# Exclude plugins directory - plugins are separate repos and shouldn't be stashed with base project
|
||||
# Use --untracked-files=no to skip untracked files check (much faster with symlinked plugins)
|
||||
# Local changes, counted exactly as the automatic update's preflight
|
||||
# counts them (auto_update.local_changes): mode-only changes and the
|
||||
# plugin folders don't count, and the pull's --autostash carries those
|
||||
# across and reapplies them.
|
||||
from web_interface import auto_update
|
||||
try:
|
||||
status_result = subprocess.run(
|
||||
['git', 'status', '--porcelain', '--untracked-files=no'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
cwd=project_dir
|
||||
)
|
||||
# Filter out any changes in plugins directory - plugins are separate repositories
|
||||
# Git status format: XY filename (where X is status of index, Y is status of work tree)
|
||||
status_lines = [line for line in status_result.stdout.strip().split('\n')
|
||||
if line.strip() and 'plugins/' not in line]
|
||||
has_changes = bool('\n'.join(status_lines).strip())
|
||||
except subprocess.TimeoutExpired:
|
||||
# If status check times out, assume there might be changes and proceed
|
||||
# This is safer than failing the update
|
||||
has_changes = True
|
||||
status_result = type('obj', (object,), {'stdout': '', 'stderr': 'Status check timed out'})()
|
||||
changed = auto_update.local_changes(project_dir)
|
||||
except (subprocess.SubprocessError, OSError) as status_err:
|
||||
logger.warning("git status failed before pull: %s", status_err)
|
||||
changed = None
|
||||
# When git cannot say, assume there are changes rather than pull over them.
|
||||
has_changes = changed is None or bool(changed)
|
||||
|
||||
if has_changes and not stash_local_changes:
|
||||
# The automatic updater: it promised not to stash, and nothing would
|
||||
# ever restore a stash taken on its behalf.
|
||||
return {'status': 'error', 'restart_required': False, 'dependency_failures': [],
|
||||
'local_changes': list(changed or []),
|
||||
'message': auto_update.describe_local_changes(changed)}
|
||||
|
||||
stash_info = ""
|
||||
|
||||
# Stash local changes if they exist (excluding plugins)
|
||||
# Plugins are separate repositories and shouldn't be stashed with base project updates
|
||||
# Stash local changes if they exist. The plugin folders are left out:
|
||||
# plugins are separate installs, and --autostash carries their edits.
|
||||
if has_changes:
|
||||
try:
|
||||
# Use pathspec to exclude plugins directory from stash
|
||||
stash_result = subprocess.run(
|
||||
['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--', ':!plugins'],
|
||||
['git', 'stash', 'push', '-m', 'LEDMatrix auto-stash before update', '--',
|
||||
*(f':!{folder}' for folder in auto_update.SEPARATE_INSTALL_DIRS)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
@@ -369,7 +377,7 @@ def _perform_core_update_locked():
|
||||
['git', 'diff', '--name-only', f'{old_head}..{new_head}'],
|
||||
capture_output=True, text=True, timeout=15, cwd=project_dir)
|
||||
changed = set(diff.stdout.split()) if diff.returncode == 0 else set()
|
||||
for rel in ('requirements.txt', 'web_interface/requirements.txt'):
|
||||
for rel in CORE_REQUIREMENT_FILES:
|
||||
req_path = PROJECT_ROOT / rel
|
||||
if rel not in changed or not req_path.exists():
|
||||
continue
|
||||
|
||||
Reference in New Issue
Block a user