From 116abb0daab1c120e1d26d8ce34b33b40460d566 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 16:37:29 -0400 Subject: [PATCH] =?UTF-8?q?fix:=20September=2016=20core=20audit=20?= =?UTF-8?q?=E2=80=94=20partial=20saves,=20asset=20path=20safety,=20auto-up?= =?UTF-8?q?date,=20display=20settings=20the=20library=20refuses,=20scroll?= =?UTF-8?q?=20speed=20(#595)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(sports): share the ESPN rejected-range memo with the background service BackgroundDataService always sent a season range first and, on a 400, fell back to chunks without recording the rejection, so every background season fetch spent a doomed request and live scoreboards learned nothing from it (or it from them). The worker now consults and sets the same 6-hour memo fetch_espn_scoreboard() uses: a known rejection goes straight to month/day chunks, and if every chunk fails the range is asked once for a real error without re-spending the chunks. Co-Authored-By: Claude Opus 5 * fix(web): keep plugin asset and action routes inside their directories POST /plugins/assets/upload, GET /plugins/assets/list and POST /plugins/assets/delete joined the request's plugin_id onto assets/plugins unchecked, so '../../config' created, wrote, listed and deleted outside it. #561 guarded only the route that serves the files. All three now go through path_safety.resolve_under and answer 400 for anything but a plain name, and delete only unlinks a metadata path that resolves into that plugin's uploads directory. PluginManager.get_plugin_directory refuses ids that are not one plain path segment, so /plugins/action (which runs a manifest script from the returned directory) and every other caller get the guard; the action route also rejects such ids up front, covering its no-manager fallback. Co-Authored-By: Claude Opus 5 * fix(web): report a no-op plugin update as already up to date update_plugin() returns True both for a real update and for "nothing to do" (a ZIP-installed monorepo plugin already at the registry version, a bundled plugin). With no git commit to compare, POST /plugins/update called every such success "updated successfully", so Check & Update All counted most official plugins as updated on every run. The route now reads what changed off the plugin itself (commit, else manifest version, else last_updated) and returns data.update_status (updated / up_to_date / local_only). The update-all toast is summarised by PluginInstallManager.summarizeUpdateResults from that status, falling back to the message for older servers. Co-Authored-By: Claude Opus 5 * fix(sports): scoreboard scroll speed no longer follows target_fps sports_scroll computed the crisp speed ladder against the global target_fps whenever limit_refresh_rate_hz was the 100 Hz default. Since frame-locked presentation (#545) the helper steps a fixed number of whole pixels per presented frame and the panel presents at its real refresh, so the General tab's "Scroll Frame Rate" became a speed multiplier: 60 ran a 50 px/s scoreboard at 100 px/s, 200 ran it at 25 px/s. The ladder now uses the display manager's refresh_hz, then display.hardware.limit_refresh_rate_hz, then the default. target_fps is not consulted. Docstrings now say scroll_delay is ignored for pacing (no behaviour change there) and describe the fixed-step model. Tests: replace the tests that pinned target_fps as the ladder refresh and described time-based stepping; assert speed independence from target_fps (unit and end-to-end presented px/s against the real helper), that the fixed per-frame step is applied, and that scroll_delay does not change speed. Co-Authored-By: Claude Opus 5 * fix(web): escape registry and upload values in plugin manager inline handlers The store, saved-repository and custom-registry buttons built onclick='...(${JSON.stringify(id)})...'. JSON.stringify leaves ' alone, so a custom registry entry whose id contained ' closed the attribute and added its own handler. One helper, jsStringAttr(), now HTML-escapes the JSON literal for every one of those handlers, and the store View button opens only http(s) repo links. The live window.updateImageList (plugins_manager.js loads last, so its copy wins over the file-upload widget's) wrote the uploaded file's original name, path and ids into markup raw; they are escaped now. Co-Authored-By: Claude Opus 5 * docs(changelog): note plugin asset, action and inline handler guards Co-Authored-By: Claude Opus 5 * fix(update): let the root pip wrapper install web_interface/requirements.txt Update Code, the automatic update's health check and Install Base Requirements install web_interface/requirements.txt through safe_pip_install.sh, which only allowed the root requirements.txt. The first commit changing that file would fail its dependency install, and the automatic updater rolls back any update whose dependencies did not install -- on every device, for every newer commit. The wrapper now lists both core requirement files. Only their folders are resolved, so a requirements.txt symlinked out of the project is compared by its target and refused (previously the root file's own symlink target was what got allowed). The updater's file list is a named constant, and a test runs the real wrapper (pip stubbed) on every file Update Code and the rollback install. Co-Authored-By: Claude Opus 5 * fix(web): do not retry plugin requests that got an HTTP answer PluginAPI.request wrapped everything that was not a structured error as NETWORK_ERROR: a proxy's 502 HTML page (response.json() throws) and a JSON error without error_code included. Check & Update All retries NETWORK_ERROR, so those updates were re-sent five more times with backoff, contrary to the #587 contract that an HTTP error response is the server's answer. NETWORK_ERROR now means only that fetch() rejected. Any HTTP response without an error_code, or with a body that is not JSON, is API_ERROR with the HTTP status attached. Tested against the shipped api_client.js. Co-Authored-By: Claude Opus 5 * fix(scroll): restart the stats window when an idle gap is dropped by size #582 dropped an idle gap from the frame stats two ways: the reset_scroll() sentinel, which also restarts the 5s window timer, and a size guard for scrollers that never call reset_scroll(), which did not. On that path the first real frame after the gap found the boundary overdue and logged a stats line for a one-frame window. Both paths now share one seeding helper. Co-Authored-By: Claude Opus 5 * fix(update): leave plugins alone when update_core's own rollback fails update_core returns rollback_failed directly when a partial pull or an update whose health check never started cannot be rolled back. run() only held plugins back for 'verifying', so those devices still got new plugin versions and a display restart on top of a core in an unknown state -- the opposite of what the health-check path does, and of the 3.4.0 changelog (plugins are left alone if the rollback fails). Co-Authored-By: Claude Opus 5 * docs(api): make the REST reference match the api_v3 package Every documented request body, query parameter and response shape was re-checked against the handlers in web_interface/blueprints/api_v3/. Fixes calls that failed as documented (repo_url, action_id/params, files/image_id, font_file+font_family, ?font=, cache key, auto_enable_ap_mode, plugin limit keys), removes the font-override endpoints dropped in #566, corrects response shapes (plugins/config, plugins/schema, health, metrics, operation history, github-status, fonts/catalog, cache/list, logs, wifi, on-demand, SSE streams), and adds the 26 routes it omitted (backup, system auto-update/git, wifi radio, starlark editor, MQTT bridge, status endpoints, skins). Documents the merge semantics of partial JSON saves to /config/main and /plugins/config and the dim-schedule POST accepting GET's days shape, which land in the same change set. Replaces app.py line numbers and the removed api_v3.py path with file and function names. Co-Authored-By: Claude Opus 5 * fix(web): remove the General-tab plugin system toggles that did nothing plugin_system.auto_discover, auto_load_enabled and development_mode had General-tab toggles whose help tips promised dormant plugins and verbose logging, but nothing reads them: every enabled plugin is discovered and loaded regardless. Remove the three toggles. The keys stay tolerated in stored configs. The save handler now stores a flag only when a client sends it; treating a missing key as an unchecked box would otherwise rewrite all three to false on every General-tab save, which still posts plugins_directory. Co-Authored-By: Claude Opus 5 * refactor(scroll): remove dead code left by #523/#570 - Drop the optional scipy.ndimage import and HAS_SCIPY; nothing read them since the numpy blend replaced the scipy path. - Drop ScrollHelper._last_integer_position and frame_time_target, which were written but never read. - Keep target_fps and set_target_fps() but document them as informational: nothing paces off them, yet ledmatrix-elections' test_scroll_pacing.py reads helper.target_fps back and third-party plugins may call the setter. - Fix stale comments: fixed_pixels_per_frame's "use scroll_delay to throttle", set_sub_pixel_scrolling's "default: True", and set_frame_based_scrolling's claim that it steps. The plugins monorepo was grepped for every removed name; none is used. Co-Authored-By: Claude Opus 5 * docs(fonts): point plugins at plugin_manager.font_manager; drop removed overrides UI FONT_MANAGER.md told plugins to read display_manager.font_manager, which does not exist, so a plugin following it failed to load with AttributeError. The shared FontManager lives on the PluginManager and BasePlugin._get_font_manager() returns it (with a fallback for harnesses). Also removes the Fonts-tab override workflow and element-override panels that #566 deleted, from FONT_MANAGER.md and WEB_INTERFACE_GUIDE.md. Co-Authored-By: Claude Opus 5 * docs(store): search via /plugins/store/list?query=; send Content-Type on registry curls /plugins/store/search does not exist (404) and the list endpoint reads query, not q. The registry guide's curl examples omitted the JSON Content-Type, so the handlers saw an empty body and answered 400. Co-Authored-By: Claude Opus 5 * fix(config): use the shared core-key list in the last three private copies StartupValidator warned "Plugin 'auto_update' is enabled but not found" on every display start with auto-update or a dim schedule on; the reserved plugin-id check missed auto_update, sync, location and the rest; and ConfigManager's (uncalled) orphan cleanup would have deleted display, schedule and auto_update. All three now read src/core_config_keys.py, which also gains CORE_SECRETS_KEYS for the github/youtube secrets sections. Co-Authored-By: Claude Opus 5 * fix(web): partial JSON saves to /config/main change only what they send A JSON body with one field reset every checkbox in the sections it touched: the MQTT bridge's brightness slider turned off disable_hardware_pulsing, inverse_colors, show_refresh_rate and use_short_date_format, and a timezone-only save turned off web-UI autostart and weekly auto-updates. Missing-means-unchecked now applies only to form posts: form-encoded bodies and the v3 forms, which mark themselves with a hidden __form_section input. Also on the config routes: - vegas_min/max_cycle_duration no longer match the generic *_duration rule, so they stop landing in display_durations and a blank one no longer rejects the whole Display save; - saving from the Raw JSON editor calls start_setup_if_needed like the General form, so enabling auto-update there finishes its setup; - the schedule and dim-schedule POSTs accept the per-day days. shape their GETs return, as well as the flat form keys. Co-Authored-By: Claude Opus 5 * fix(scripts): install plugin dependencies from the configured plugins directory install_plugin_dependencies.sh scanned only plugins/, but the Plugin Store installs into plugin_system.plugins_directory (default plugin-repos), so the documented "Recommended" fix found 0 plugins on every store install. It now reads plugins_directory from config/config.json (relative to the project root or absolute, default plugin-repos) and also scans plugins/ for dev symlinks, installing a plugin reached through both only once. With set -e alone, `pip ... | tee` took tee's exit status, so a failed pip install was reported as success; set -o pipefail. Co-Authored-By: Claude Opus 5 * docs: replace stale API names, line numbers and the api_v3.py path - ADVANCED_FEATURES: StreamManager methods that exist (get_next_segment, take_next_group, refresh, advance_cycle, ...), and the real on-demand status envelope ({status, data: {state, service}}) - app.py:199 / :144 / :607-619 line citations and web_interface/blueprints/api_v3.py (now a package) replaced with file and function names in ADVANCED_FEATURES, CONFIG_DEBUGGING, PLUGIN_ARCHITECTURE_SPEC, PLUGIN_QUICK_REFERENCE, PLUGIN_CONFIGURATION_TABS, TROUBLESHOOTING and web_interface/README - CONFIG_DEBUGGING: partial /config/main saves change only sent keys; use /config/raw/main to replace the file; describe where validation runs - TROUBLESHOOTING: clear_cache.py needs --clear-all (no args only prints usage) Co-Authored-By: Claude Opus 5 * fix(scripts): verify the web interface that actually ships, on port 5000 verify_installation.sh failed every healthy install: it required the long-removed web_interface_v2.py and looked for a listener on port 5001, while the web interface binds 5000 (web_interface/start.py). It now checks the files ledmatrix-web.service runs (start_web_conditionally.py, web_interface/start.py, app.py) and port 5000. verify_web_ui.sh had the same 5001 port in its listen check, HTTP probe and printed URLs. Port matches are anchored so :50001 no longer counts as :5000. Co-Authored-By: Claude Opus 5 * docs(plugins): one display-size contract: display_manager.width/height CLAUDE.md (#580) says to read display_manager.width/height because matrix is None when hardware init fails; the development guide, the safety-harness doc and two DisplayManager docstrings still recommended matrix.width/height. The bundled starlark-apps plugin read matrix.width unguarded, so its magnify recommendation and frame scaling raised in fallback mode (e.g. after the Pi 5 hardware refusal). Co-Authored-By: Claude Opus 5 * fix(install): make install_service.sh --help print usage instead of installing install_service.sh parsed no arguments, so `sudo ./scripts/install/ install_service.sh --help` (presented as harmless in MIGRATION_GUIDE.md) rewrote ledmatrix.service, ledmatrix-web.service and both update-verify units and enabled/started them. It now handles -h/--help (usage, exit 0, no changes) and rejects any other argument with exit 2 before doing anything. Running it with no arguments, as first_time_install.sh does, is unchanged. Co-Authored-By: Claude Opus 5 * docs(scroll): describe the fixed-step model and document frame_hold Since #545 a crisp speed from scroll_config.configure() makes the helper advance a fixed whole-pixel step per presented frame with no clock, and the display manager's frame hold is part of the speed. The docs still described the removed wall-clock model: - scroll_config's module and configure() docstrings said speed is applied in time-based mode and that omitting the hold "falls back to fractional pixels"; omitting it actually runs the scroll frame_hold times too fast. - SCROLL_PERFORMANCE.md said ScrollHelper accumulates elapsed time in both modes, and read a 20 ms stats median as missed refreshes although that is a healthy 50 px/s (hold 2) scroll. It now explains the fixed step, the hold-dependent healthy median, that target_fps plays no part, and that a hand-added scroll_pixels_per_second loses to a schema-default pair. - PLUGIN_API_REFERENCE.md documented set_scrolling_state(is_scrolling) without frame_hold; it now documents the parameter (core 3.4.0) with a configure() + set_scrolling_state example. - update_scroll_position/set_scroll_speed and set_scrolling_state docstrings say the same. Co-Authored-By: Claude Opus 5 * docs(config): mark target_fps legacy; describe what Vegas scroll_delay does - General tab "Scroll Frame Rate" (target_fps) is labelled legacy: after the sports_scroll fix nothing in core scrolling reads it. The field and its API validation stay so saved configs and plugins that read global_config['target_fps'] keep working. CONFIG_REFERENCE says the same. - Vegas frame_based_scrolling/scroll_delay were described as frame-count stepping at ~50 FPS. Neither steps nor sets a frame rate: frame-based mode converts the speed to px per scroll_delay, clamps it to 0.1-5, and still advances by elapsed time, so the applied speed is clamp(scroll_speed * scroll_delay, 0.1, 5) / scroll_delay px/s. The config comments, render_pipeline comment and CONFIG_REFERENCE rows now say so. No behaviour change. Co-Authored-By: Claude Opus 5 * docs(deps): describe how plugin dependencies are really installed The guides said the web service runs as root, that installs pick --user from os.geteuid(), and quoted a warning and a PluginManager._install_plugin_dependencies() method that don't exist. The web unit runs as the installing user; store installs go through install_requirements_file() and sudo safe_pip_install.sh (root), with a user-level fallback that says so, and load-time installs run in the display service's own (root) interpreter. Manual paths now use the configured plugins directory (plugin-repos/ by default) instead of plugins/, which store installs no longer use, and install_plugin_dependencies.sh is described as scanning that directory. Co-Authored-By: Claude Opus 5 * fix(update): count local changes one way for the preflight and the pull The automatic update's preflight ignored mode-only changes and anything whose status line contained plugins/ or plugin-repos/, then promised "Automatic updates will not stash your changes". perform_core_update used plain git status (modes count) and ignored only 'plugins/', then ran 'git stash push -- :!plugins', which nothing ever pops. So an edit to a bundled plugin under plugin-repos/, or the installer's chmods on tracked scripts, passed the preflight and was stashed away for good. - auto_update.local_changes() is the one predicate both use: core.fileMode=false, porcelain -z, and plugins/ and plugin-repos/ excluded by leading folder rather than substring (a core file under web_interface/static/v3/js/plugins/ now counts). - Update Code's explicit stash leaves out both plugin folders; the pull's --autostash carries their edits and mode changes across and reapplies them. - The automatic updater calls perform_core_update(stash_local_changes= False), which refuses instead of stashing edits that appeared after the preflight; update_core reports that as 'blocked'. Co-Authored-By: Claude Opus 5 * fix(scripts): diagnostics follow the web autostart default and api_v3 package #556 made a missing web_display_autostart mean "start" (only an explicit false/off keeps the web interface down), but the diagnostics still said otherwise: diagnose_web_ui.sh reported a missing key as "defaults to false", diagnose_web_interface.sh said the web interface "will not start unless this is set to true" and recommended enabling it, and debug_web_manual.py printed False. Troubleshooting a down web UI pointed users at a non-cause. Both shell scripts now evaluate the setting with the launcher's own autostart_enabled() (inline fallback if it cannot be imported) and report on / off / not set (on) / unparseable config; debug_web_manual.py uses the same function. They also check web_interface/blueprints/api_v3/ __init__.py: api_v3.py became a package in #553, so every healthy checkout was reported as missing a file. Co-Authored-By: Claude Opus 5 * docs(install): what install_service.sh installs; verify script port; no sudo for --help install_service.sh installs and starts ledmatrix, ledmatrix-web and the update-verify units, not only ledmatrix.service (systemd/README.md, README.md). MIGRATION_GUIDE presented 'sudo install_service.sh --help' as a harmless check; it now shows --help without sudo and warns what a real run does. SSH_UNAVAILABLE_AFTER_INSTALL: verify_installation.sh checks the web interface on port 5000. Co-Authored-By: Claude Opus 5 * docs(changelog): note update-all, plugin system settings and script fixes Co-Authored-By: Claude Opus 5 * fix(display): size the preview after orientation and pixel mappers display_geometry.physical_size claimed to give DisplayManager's answer but only computed cols*chain x rows*parallel. RGBMatrix.width/height are measured after the library's pixel mappers, so a Rotate:90 / orientation 90 chain previewed 128x32 for a 32x128 panel and a U-mapper chain of four 256x32 for 128x64. Model the built-in mappers' size effect as the pinned lib/pixel-mapper.cc does (Rotate, U-mapper, V-mapper, StackToRow, Remap; Mirror and unknown names leave it alone), and move the orientation composition here so DisplayManager and the preview share it. The module docstring no longer claims the sync handshake uses it; that imports only DEFAULT_CHAIN_LENGTH. Audit finding F18. Co-Authored-By: Claude Opus 5 * fix(display): refuse settings the rgbmatrix library aborts on, on every board The library answers several settings with a NULL matrix or abort() rather than an error, so the display service crash-looped (Restart=on-failure) instead of reaching fallback mode: rows above 64, chain_length above 255 (uint8_t binding setter, documented as "no upper limit"), a misspelled hardware_mapping, and parallel 2-3 on a single-output mapping, reachable from the Display form on the default adafruit-hat(-pwm) mapping. #586 only guarded the Pi 5 subset. - src/matrix_support.py holds the rules for every board (Options::Validate ranges, binding integer types, mapping names and outputs from lib/hardware-mapping.c) plus the Pi 5 ones, and is the one source of the API's numeric ranges. - DisplayManager checks them before building options and raises MatrixSettingsRefused, so a hand-edited config falls back with a logged, reported reason. Emulator mode only warns. - The config API refuses them with a 400 naming the setting; combinations are checked against stored values but reported only when the request sets a field involved. - The hardware status file gains "cause" (settings/library/forced). The fallback log and Display banner give the Pi 5 rebuild hint only for a library failure instead of rebuild + gpio_slowdown advice for every failure; one Pi 5 slowdown recommendation (1-3, start at 1). - The Display form offers classic/classic-pi1 and orientation 90/270 and renders any other stored mapping selected with a warning, so an unrelated save no longer rewrites them; the API accepts 90/270. Audit findings F03, F16, F19, F21. Co-Authored-By: Claude Opus 5 * docs(display): library limits, template defaults and Pi 5 slowdown - rows 8-64, chain_length 1-255, parallel limited by the mapping's outputs, classic/classic-pi1 mappings and orientation 90/270 documented. - Defaults are the config.template.json values: config migration adds missing keys from the template, so the listed "code defaults" never applied. - One Raspberry Pi 5 gpio_slowdown recommendation: 1-3 in PIO mode, starting at 1. - Troubleshooting describes the refused-settings fallback, and CHANGELOG corrects the Unreleased "no upper limit" entry. Audit findings F19, F20, F21. Co-Authored-By: Claude Opus 5 * fix(scripts): scroll_speeds.py opens the panel with the service's options --measure and --demo built RGBMatrixOptions from a private copy of the display service's builder that had drifted: gpio_slowdown came from display.hardware (default 2) instead of display.runtime (default 3), and rp1_rio, panel_type, disable_hardware_pulsing, inverse_colors, pixel_mapper_config and orientation were skipped, with different defaults (hardware_mapping "regular", pwm_bits 11). A panel needing a high slowdown was measured -- or garbled -- in a setup the service never drives. The option filling in DisplayManager._setup_matrix moves, unchanged, into DisplayManager.apply_matrix_options(options, config), which _setup_matrix calls and the script reuses (overriding only limit_refresh_rate_hz for --measure). The script now loads the whole config rather than the hardware block. Tests pin the script's options to the service's attribute for attribute. Co-Authored-By: Claude Opus 5 * fix(scripts): scroll_speeds.py recommends keys the resolver honours The ladder ended by telling users to set display_options.scroll_pixels_per_second. scroll_config ranks that key below the scroll_speed + scroll_delay pair, deliberately, and several plugin schemas default the pair into config, so the advised key was silently ignored (a schema-default 1/0.02 pair plus an advised 66 still resolved to 50 px/s). The advice is now the pair that selects the crisp speed exactly (pixels_per_frame every frame_hold/refresh seconds), explains that the pair outranks scroll_pixels_per_second, and gives the scoreboards' per-league scroll_settings.scroll_speed (px/s) form. Tests resolve the printed pair over a schema-default pair and check it lands on the advertised speed and hold. Co-Authored-By: Claude Opus 5 * docs: withdraw the target_fps claim for sports_scroll; fix the Vegas speed formula - SPORTS_UNIFICATION.md still presented honouring global target_fps as sports_scroll's added behaviour and its one user-visible gain; note that it was withdrawn because it had become a speed multiplier. - ADVANCED_FEATURES.md gave Vegas scrolling as (scroll_speed / target_fps) * elapsed; the real rule is scroll_speed px/s by elapsed time, through a 0.1-5 px per scroll_delay clamp when frame_based_scrolling is on. Co-Authored-By: Claude Opus 5 * docs(changelog): scroll model fixes Co-Authored-By: Claude Opus 5 * fix(dev): link-github links plugins from the ledmatrix-plugins monorepo link-github cloned https://github.com/ChuckBuilds/ledmatrix-.git, and those per-plugin repositories no longer exist: official plugins are directories in the ledmatrix-plugins monorepo. It now clones (or pulls) the monorepo once into the dev directory, finds plugins/, plugins/ledmatrix- or the plugin whose manifest id is , and links it under its manifest id. With an explicit repo URL it still links a single-repository plugin as before. dev_plugins.json: github_user is honoured again (monorepo owner, e.g. a fork), plus plugins_repo and plugins_branch; github_pattern, which was documented but never read, is dropped and warned about. Ships dev_plugins.json.example and git-ignores dev_plugins.json, both of which the guide promised. Reading JSON falls back to python3 when jq is missing (get_plugin_id silently returned nothing without jq). update/status/list find the git checkout above a monorepo plugin directory (its .git is not in the plugin dir), and update pulls a shared checkout once. status no longer exits 1 when nothing is broken. Docs: PLUGIN_DEVELOPMENT_GUIDE (quick start, link-github, configuration, workflow, store integration, hello-world link, submission), and the nonexistent scripts/git-hooks/pre-push-plugin-version and scripts/bump_plugin_version.py replaced with the real rule: bump the manifest version and run update_registry.py. scripts/dev/README.md and CLAUDE.md updated to match. Co-Authored-By: Claude Opus 5 * docs(scripts): monorepo workspace layout; fix_perms and install READMEs MULTI_ROOT_WORKSPACE_SETUP described one sibling repository per plugin; setup_plugin_repos.py links ../ledmatrix-plugins/plugins/* into plugin-repos/ and update_plugin_repos.py pulls only the monorepo, and the workspace file opens LEDMatrix plus ../ledmatrix-plugins. scripts/fix_perms/README.md listed cache directories fix_cache_permissions.sh never touches and a 'ledmatrix' service user that doesn't exist (also in scripts/install/README.md); adds safe_pip_install.sh. install/README: install_service.sh installs the web and update-verify units too. Co-Authored-By: Claude Opus 5 * fix(update): keep the rollback's pip retries inside the unit time limit The health check reinstalled the previous requirements by trying the next bash path after any failure, including a 600 s pip timeout. Two files, two paths: up to 40 minutes of pip alone, while systemd stops ledmatrix-update-verify.service at TimeoutStartSec=30min -- killing the rollback half-way and leaving the update 'verifying' until the web UI calls it lost. - Like permission_utils.install_requirements_file, only a sudo refusal moves on to the next bash; a pip that ran and failed or timed out is not repeated. The refusal wording is one list (permission_utils.SUDO_REFUSAL_PHRASES), mirrored in the stdlib-only verifier and pinned equal by a test. - All reinstalls in one rollback share a 600 s budget. - WORST_CASE_SECONDS adds up every timeout on the longest path (27.5 min); a test holds it under the unit's TimeoutStartSec and that under the web UI's VERIFY_LOST_SECONDS. Co-Authored-By: Claude Opus 5 * fix(plugins): prepare plugin configs one way for load, saves, GET, hot reload and dev tools Plugin config was prepared differently depending on how it arrived: - JSON POST /plugins/config built a partial body on schema defaults, so {"enabled": true} reset every other setting of the plugin. It now merges onto the stored section first, as the form path already did. - Legacy-boolean normalization (#588) ran only at load: GET /plugins/config returned the raw boolean, posting it back failed validation, and hot reload handed plugins the raw section (a legacy dynamic_duration: true came back as a boolean). schema_manager.prepare_plugin_config (normalize, then defaults) is now used by PluginManager.load_plugin, both save paths, GET, the save notifications and DisplayController's hot-reload callback. - The JSON save's filter kept only enabled/display_duration/live_priority and dropped a submitted skin, skin_options or vegas_* tuning key. There is now one core-owned per-plugin list, schema_manager.CORE_PLUGIN_PROPERTIES, used by validation and by the save filter; PluginManager's CORE_OWNED_CONFIG_KEYS is its vegas subset. - Plugin sections posted to /config/main were stored verbatim, including values /plugins/config rejects. They now go through the same preparation (_prepare_plugin_config_for_save, extracted from save_plugin_config), and a failing section rejects the whole save before anything is written. - dev_server read only top-level defaults and let a schema enabled:false win; build_full_config shallow-merged overrides, dropping sibling defaults; the harness extracted defaults differently from the device. loading.build_config now uses the device's extraction and preparation, and dev_server, check_plugin, render_plugin and the harness all use it. Co-Authored-By: Claude Opus 5 * docs(mqtt-bridge): brightness changes apply live and touch nothing else The display service's hot reload applies a saved brightness within a few seconds, and /config/main no longer resets other display settings on a brightness-only JSON body. Co-Authored-By: Claude Opus 5 * docs(changelog): automatic update hardening Co-Authored-By: Claude Opus 5 * docs(config): rewrite PLUGIN_CONFIG_ARCHITECTURE for the v3 web UI It described web_interface_v2.py and index_v2.html (both gone), client-side form generation, one POST per field with {key, value}, and 'no nested objects'. The v3 UI renders plugin forms server-side from the schema (pages_v3 partial + plugin_config.html macros, nested sections and x-widgets), posts the whole form once, and save_plugin_config() merges onto the stored section, validates, splits x-secret fields and notifies the plugin. Co-Authored-By: Claude Opus 5 * docs(mqtt): brightness saves apply via hot reload and leave other settings alone The bridge README said brightness is applied on the display's next restart; the display controller's config hot reload applies it within seconds. It also now states that the bridge's partial JSON save changes only brightness (the /config/main merge fix in this change set). Co-Authored-By: Claude Opus 5 * fix(update): don't log pip's output from the health check's reinstall pip can echo a private index URL with embedded credentials; permission_utils redacts it, the stdlib-only verifier cannot, so it logs the exit code only. Co-Authored-By: Claude Opus 5 * docs(config): mark the plugin_system toggles as unused legacy keys auto_discover, auto_load_enabled and development_mode are read by nothing and leave the General tab in this change set (F40). CONFIG_REFERENCE said they were read by the plugin loader; PLUGIN_CONFIGURATION_GUIDE and the REST reference listed them as live settings. Co-Authored-By: Claude Opus 5 * docs(changelog): docs and developer tools group Co-Authored-By: Claude Opus 5 * fix(web): legacy plugin-system toggles no longer count as a General save auto_discover, auto_load_enabled and development_mode have left the General form, so a post carrying only one of them is not a general-settings save and must not treat web_display_autostart and auto_update as unchecked. The plugin_system block itself is left as on main for the branch that reworks it. Co-Authored-By: Claude Opus 5 * docs(changelog): config-save and plugin-config preparation fixes Co-Authored-By: Claude Opus 5 * docs(claude): re-check matrix_support.py rules when the library submodule is bumped Co-Authored-By: Claude Opus 5 * fix: address Codacy findings on the core audit PR - plugin_manager.prepare_plugin_config: when the fallback legacy-boolean pass also fails, log a warning instead of a bare except/pass. - api_client.js: request() refuses any endpoint that is not a plain path under /api/v3 ("//host", backslashes, ".." or "." segments, whitespace, control characters) with INVALID_ENDPOINT before calling fetch(), and plugin ids are URL-encoded wherever they are put into a URL (also in the app-shell batch load). - test_update_all.js: pins both against the shipped client. Co-Authored-By: Claude Opus 5 * fix(web): check endpoint control characters without a control-character regex Codacy (ESLint no-control-regex, Biome noControlCharactersInRegex) flags the \x00-\x1f range in checkEndpoint's regex. Test the char codes instead; the endpoints refused are unchanged. Co-Authored-By: Claude Opus 5 * test(auto-update): make the seed script executable on disk, not only in the index On Linux Repo.publish() commits with -a, which recorded scripts/run.sh as 100644 upstream because the seed file was never chmod +x. The pull then brought in the same mode the installer chmod had made locally, so installer_chmod saw no mode change left to check. The updater was fine: with the upstream commit at 100755 the --autostash carries the device's chmod across. Verified under Linux (WSL, git 2.43): the old helper fails exactly as CI did, the fixed one passes all 63 tests in the file. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 --- .gitignore | 7 +- CHANGELOG.md | 215 ++- CLAUDE.md | 4 +- README.md | 35 +- dev_plugins.json.example | 6 + docs/ADVANCED_FEATURES.md | 50 +- docs/CONFIG_DEBUGGING.md | 19 +- docs/CONFIG_REFERENCE.md | 49 +- docs/FONT_MANAGER.md | 71 +- docs/MIGRATION_GUIDE.md | 7 +- docs/MULTI_ROOT_WORKSPACE_SETUP.md | 177 ++- docs/PLUGIN_API_REFERENCE.md | 53 +- docs/PLUGIN_ARCHITECTURE_SPEC.md | 3 +- docs/PLUGIN_CONFIGURATION_GUIDE.md | 10 +- docs/PLUGIN_CONFIGURATION_TABS.md | 3 +- docs/PLUGIN_CONFIG_ARCHITECTURE.md | 516 ++----- docs/PLUGIN_DEPENDENCY_GUIDE.md | 294 ++-- docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md | 159 ++- docs/PLUGIN_DEVELOPMENT_GUIDE.md | 197 +-- docs/PLUGIN_QUICK_REFERENCE.md | 3 +- docs/PLUGIN_REGISTRY_SETUP_GUIDE.md | 3 + docs/PLUGIN_STORE_GUIDE.md | 9 +- docs/REST_API_REFERENCE.md | 1212 +++++++++++------ docs/SCROLL_PERFORMANCE.md | 58 +- docs/SPORTS_UNIFICATION.md | 13 +- docs/SSH_UNAVAILABLE_AFTER_INSTALL.md | 4 +- docs/TROUBLESHOOTING.md | 4 +- docs/WEB_INTERFACE_GUIDE.md | 19 +- docs/plugin-safety-harness.md | 3 +- integrations/mqtt_bridge/README.md | 9 +- plugin-repos/starlark-apps/manager.py | 14 +- scripts/debug/debug_web_manual.py | 9 +- scripts/dev/README.md | 10 + scripts/dev/dev_plugin_setup.sh | 268 +++- scripts/dev_server.py | 28 +- scripts/diagnose_web_interface.sh | 66 +- scripts/diagnose_web_ui.sh | 65 +- scripts/fix_perms/README.md | 22 +- scripts/fix_perms/safe_pip_install.sh | 27 +- scripts/install/README.md | 14 +- scripts/install/install_service.sh | 35 + scripts/install_plugin_dependencies.sh | 59 +- scripts/scroll_speeds.py | 114 +- scripts/utils/auto_update_verify.py | 64 +- scripts/verify_installation.sh | 23 +- scripts/verify_web_ui.sh | 46 +- src/background_data_service.py | 44 +- src/common/permission_utils.py | 13 +- src/common/scroll_config.py | 46 +- src/common/scroll_helper.py | 104 +- src/common/sports_scroll.py | 88 +- src/config_manager.py | 12 +- src/core_config_keys.py | 10 + src/display_controller.py | 8 + src/display_geometry.py | 150 +- src/display_manager.py | 219 +-- src/matrix_support.py | 205 +++ src/plugin_system/plugin_manager.py | 103 +- src/plugin_system/schema_manager.py | 456 ++++--- src/plugin_system/testing/harness.py | 6 +- src/plugin_system/testing/loading.py | 84 +- src/startup_validator.py | 6 +- src/vegas_mode/config.py | 9 +- src/vegas_mode/render_pipeline.py | 12 +- systemd/README.md | 8 +- test/js/README.md | 3 +- test/js/run_all.js | 2 +- test/js/unit/test_inline_handler_escaping.js | 215 +++ test/js/unit/test_update_all.js | 83 ++ test/test_api_v3_display_hardware.py | 128 +- test/test_api_v3_lazy_plugin_discovery.py | 6 +- test/test_api_v3_partial_main_save.py | 288 ++++ test/test_auto_update.py | 179 ++- test/test_auto_update_verify.py | 76 ++ ...est_background_data_service_espn_ranges.py | 52 +- test/test_core_config_key_adopters.py | 108 ++ test/test_display_geometry.py | 51 + test/test_display_manager.py | 104 +- test/test_matrix_support.py | 125 ++ test/test_path_traversal_guards.py | 199 +++ test/test_plugin_config_preparation.py | 205 +++ test/test_plugin_system_legacy_flags.py | 70 + test/test_safe_pip_install_allowlist.py | 128 ++ test/test_scroll_helper.py | 13 + test/test_scroll_speeds_script.py | 144 ++ test/test_sports_scroll.py | 168 ++- .../test_plugin_config_json_saves.py | 258 ++++ test/web_interface/test_update_all_plugins.py | 61 + web_interface/README.md | 13 +- web_interface/auto_update.py | 81 +- web_interface/blueprints/api_v3/__init__.py | 58 +- web_interface/blueprints/api_v3/config.py | 362 ++--- web_interface/blueprints/api_v3/plugins.py | 903 ++++++------ web_interface/blueprints/api_v3/system.py | 64 +- web_interface/static/v3/js/app-shell.js | 4 +- .../static/v3/js/plugins/api_client.js | 96 +- .../static/v3/js/plugins/install_manager.js | 52 + web_interface/static/v3/plugins_manager.js | 70 +- .../templates/v3/partials/display.html | 68 +- .../templates/v3/partials/durations.html | 2 + .../templates/v3/partials/general.html | 50 +- 101 files changed, 7244 insertions(+), 2904 deletions(-) create mode 100644 dev_plugins.json.example create mode 100644 src/matrix_support.py create mode 100644 test/js/unit/test_inline_handler_escaping.js create mode 100644 test/test_api_v3_partial_main_save.py create mode 100644 test/test_core_config_key_adopters.py create mode 100644 test/test_matrix_support.py create mode 100644 test/test_plugin_config_preparation.py create mode 100644 test/test_plugin_system_legacy_flags.py create mode 100644 test/test_safe_pip_install_allowlist.py create mode 100644 test/test_scroll_speeds_script.py create mode 100644 test/web_interface/test_plugin_config_json_saves.py diff --git a/.gitignore b/.gitignore index afb9b3df..3b59b144 100644 --- a/.gitignore +++ b/.gitignore @@ -39,11 +39,12 @@ htmlcov/ # Cache directory (root level only, not src/cache which is source code) /cache/ -# Development plugins directory -# Plugins are managed as separate repositories via multi-root workspace -# See docs/MULTI_ROOT_WORKSPACE_SETUP.md for details +# Development plugins directory: symlinks into a ledmatrix-plugins checkout +# See docs/PLUGIN_DEVELOPMENT_GUIDE.md and docs/MULTI_ROOT_WORKSPACE_SETUP.md plugins/* !plugins/.gitkeep +# Local settings for scripts/dev/dev_plugin_setup.sh (template: dev_plugins.json.example) +/dev_plugins.json # Binary files and backups bin/pixlet/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 51752b5f..a90c9f6c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,44 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +Config saves and plugin config preparation: + +- A JSON `POST /api/v3/config/main` changes only the keys it sends. The MQTT + bridge's brightness slider used to turn off `disable_hardware_pulsing`, + `inverse_colors`, `show_refresh_rate` and `use_short_date_format`, and a + timezone- or location-only save turned off web-UI autostart and weekly + automatic updates. Missing checkboxes still save as unchecked for the + settings forms (they now send a hidden `__form_section` field) and for + form-encoded posts. +- A partial JSON `POST /api/v3/plugins/config` merges onto the plugin's stored + settings instead of resetting everything it didn't send to the schema + defaults, and keeps a submitted `skin`, `skin_options`, `vegas_width_pct`, + `vegas_overflow` or `vegas_max_width_screens` (they were silently dropped). +- Plugin sections posted to `/config/main` are validated and prepared exactly + like `/plugins/config`; a value that endpoint rejects is rejected here too, + and nothing is saved. +- Legacy boolean settings (#588) are read as `{"enabled": ...}` objects + everywhere, not just when the plugin loads: `GET /plugins/config` returns + the object, posting it back saves, and hot reload hands plugins the same + shape (schema defaults included) they were constructed with. + `schema_manager.prepare_plugin_config` is the one implementation. +- `scripts/dev_server.py`, `check_plugin.py`, `render_plugin.py` and the plugin + harness build configs the way a device does: nested defaults are included, + a schema `enabled: false` no longer beats the forced `enabled: true` in the + dev server, and nested overrides such as `{"nhl": {"enabled": true}}` keep + the other defaults of that section. +- Clearing Vegas "Min/Max Cycle Time" no longer rejects the whole Display save, + and those fields no longer add junk entries to `display.display_durations`. +- Turning automatic updates on from the Raw JSON editor finishes their setup + like the General tab does, instead of waiting for the next display restart. +- `POST /config/schedule` and `/config/dim-schedule` accept the per-day + `days..{enabled,start_time,end_time}` shape their GETs return, as well + as the flat form keys. +- The startup check no longer warns that `auto_update` or `dim_schedule` is + "enabled but not found in plugins directory", and plugin ids that collide + with any core config section are flagged: the last private copies of the + core-key list now use `src/core_config_keys.py`. + New module a plugin may import via `src.*` (floor on the release that ships this): @@ -59,6 +97,36 @@ Sports data: to decide whether to submit a season range to the service or fetch it themselves on an older core. +Scrolling: + +- **Scoreboard scroll speed no longer changes with the General tab's "Scroll + Frame Rate" (`target_fps`).** Scoreboards on `src.common.sports_scroll` + computed their speed for that rate while the panel kept presenting at its + real refresh, so on a 100 Hz panel 60 ran a 50 px/s scoreboard at 100 px/s + and 200 ran it at 25 px/s. Speed now comes from `scroll_speed` and the panel + refresh only. The field is labelled legacy: nothing in core scrolling reads + it. Anyone who lowered it will see scoreboards scroll slower than before -- + at the speed they configured. +- `scripts/scroll_speeds.py --measure` / `--demo` open the panel with the + display service's own options (`DisplayManager.apply_matrix_options`), so + `display.runtime.gpio_slowdown`, `rp1_rio`, `panel_type` and orientation are + honoured; the script used to read `gpio_slowdown` from `display.hardware`. + Its closing advice now gives the `scroll_speed` + `scroll_delay` pair + instead of `scroll_pixels_per_second`, which the resolver ignores whenever + the pair is present. +- The frame-stats log no longer opens a scroll with a one-frame window for + scrollers that never call `reset_scroll()`. +- Removed dead scroll code: the optional scipy import (`HAS_SCIPY`), + `ScrollHelper._last_integer_position` and `frame_time_target`. + `ScrollHelper.target_fps` / `set_target_fps()` remain, documented as + informational. +- Docs describe the fixed-step scroll model: `PLUGIN_API_REFERENCE.md` + documents `set_scrolling_state(..., frame_hold)` (omitting the hold runs a + scroll `frame_hold` times too fast), `SCROLL_PERFORMANCE.md` no longer reads a + held 20 ms frame as missed refreshes, and Vegas `frame_based_scrolling` / + `scroll_delay` are described as the speed clamp they are rather than frame + stepping. Scoreboard `scroll_delay` is documented as ignored for pacing. + Web interface: - The plugin settings form honours `"x-display": "hidden"` in config schemas: @@ -68,9 +136,9 @@ Web interface: declared, e.g. countdown's row `id` and weather's `api_key` / `radar_zoom`. See `docs/widget-guide.md`. - Display settings no longer silently cut values on save: columns were capped - at 128, chain length at 24 and PWM LSB nanoseconds at 500. Rows, columns and - chain length now have no upper limit (the current rgbmatrix library still - rejects more than 64 rows per panel); rows must be even and at least 8, + at 128, chain length at 24 and PWM LSB nanoseconds at 500. Columns have no + upper limit, chain length is 1–255 and rows must be even and 8–64 (see + "Display hardware settings the library refuses" below); parallel is 1–3 and PWM dither bits 0–2, matching the library. A stored GPIO slowdown, PWM dither bits or refresh-rate cap of 0 no longer shows (and re-saves) as 3, 1 or 120, and the refresh cap accepts 0 (no cap). The config @@ -112,6 +180,57 @@ Web interface: re-sent with backoff instead of being counted as failed and skipped — that is how a disabled plugin with an update waiting was silently left out. +Security (request paths and inline handlers, siblings of #561): + +- `POST /api/v3/plugins/assets/upload`, `GET .../assets/list` and + `POST .../assets/delete` validate `plugin_id` with `src/common/path_safety` + and answer 400 otherwise. A `plugin_id` of `../../config` used to create an + `uploads/` directory outside `assets/plugins`, write images and + `.metadata.json` there, list it, and delete whatever file a metadata entry + named. Delete now unlinks only a path that resolves inside that plugin's + uploads directory (any other entry is dropped without touching a file). +- `PluginManager.get_plugin_directory()` returns `None` for anything but a + plain name, so `POST /api/v3/plugins/action` can no longer run a manifest + script from a directory outside the plugins directory (`../elsewhere`); the + route also rejects such ids with 400. +- Plugin Store, saved-repository and custom-registry buttons escape registry + values for their inline `onclick` handlers (`jsStringAttr` in + `plugins_manager.js`). An entry id containing `'` used to close the attribute + and add its own script. The store's View button opens only `http(s)` links. +- The uploaded-images list escapes each file's original name, path and ids; a + name like `.png` was inserted as markup. + +Display hardware settings the library refuses: + +- The rgbmatrix library answers several settings with no matrix or `abort()` + rather than an error, on every board, so the display service crash-looped + instead of falling back: rows above 64, `chain_length` above 255 (the Python + binding stores it in one byte; this was documented as "no upper limit"), a + misspelled `hardware_mapping`, and `parallel` 2–3 on a mapping with one output + (`adafruit-hat`, `adafruit-hat-pwm`, `regular-pi1`, `classic-pi1`) — the last + one reachable from the Display form on the default mapping. The config API + now refuses them with a 400 naming the setting, and `DisplayManager` refuses + a hand-edited one before creating the matrix: logged, fallback mode, reported + by `/api/v3/hardware/status`. The rules, including the Pi 5 ones, live in + `src/matrix_support.py` and must be re-checked when the submodule is bumped. +- `/api/v3/hardware/status` adds `cause`: `"settings"` when LEDMatrix refused + the config, `"library"` when the library failed. The Display tab banner and + the fallback log line give the Pi 5 rebuild hint only for a library failure; + they used to follow every failure with it and with GPIO slowdown advice. +- The Display form offers the `classic` and `classic-pi1` mappings and the + `90` / `270` orientations, and renders any other stored mapping selected with + a warning. With no option selected the browser posted the first one, so one + unrelated save rewrote those settings. The API accepts orientation `90` and + `270`, which `DisplayManager` already applied. +- The display size the web preview, Starlark magnify default and + `scripts/dev/vegas_audit.py` compute (`src/display_geometry.py`) now applies + `orientation` and `pixel_mapper_config` as the library does: `Rotate:90` + swaps width and height, `U-mapper` folds the chain. +- One Raspberry Pi 5 GPIO slowdown recommendation everywhere: 1–3 in PIO mode, + starting at 1. README and the config reference now describe the template + values as the defaults; the "code default" values they listed never apply, + because config migration fills missing keys from the template. + Plugin system: - A plugin no longer starts with a schema warning and a degraded flag because @@ -141,6 +260,96 @@ Core: ownership step is now skipped where `os.chown` is missing. No behaviour change on the Pi. +Automatic updates and Update Code: + +- An update that changes `web_interface/requirements.txt` is no longer rolled + back on every auto-updating device. `safe_pip_install.sh` allowed only the + root `requirements.txt`, so the install Update Code and the health check run + for the web requirements was refused, and the health check rolls back any + update whose dependencies failed (Install Base Requirements failed the same + way). The wrapper now allows both core requirement files; a core requirement + file symlinked out of the project is refused. +- The automatic update's local-change check and Update Code now count changes + the same way (`auto_update.local_changes`): permission-only changes and + anything under `plugins/` or `plugin-repos/` don't count, and a core path + that merely contains `plugins/` does. Such edits used to pass the check and + then be stashed by the pull and never restored, despite "will not stash your + changes". The pull's `--autostash` now carries them across. Update Code + still stashes other edits; the automatic update refuses instead. +- When the automatic update's own rollback fails (a partial pull, or a health + check that never started), plugins are no longer updated and the display is + not restarted, as the 3.4.0 notes promised. +- The health check's dependency reinstall no longer retries pip failures or + timeouts with a second bash path, and all reinstalls share a 10-minute + budget, so a rollback finishes inside the unit's 30-minute limit instead of + being killed mid-way. + +Small fixes (update-all, plugin system settings, scripts): + +- **Check & Update All** counts a plugin that had nothing to update as + "already up to date" instead of "updated". ZIP-installed monorepo plugins + (most official ones) already at the registry version were called "updated + successfully" on every run. `POST /plugins/update` now returns + `data.update_status` (`updated`, `up_to_date`, `local_only`). +- An update request that got an HTTP error answer without an `error_code`, or + a body that is not JSON (e.g. a reverse proxy's 502 page), is no longer + classified as `NETWORK_ERROR` and re-sent five times. Only a request that got + no HTTP answer is retried; the rest are `API_ERROR` with the HTTP status. +- The General tab no longer shows Auto Discover Plugins, Auto Load Enabled + Plugins or Development Mode. Nothing read `plugin_system.auto_discover`, + `auto_load_enabled` or `development_mode`: every enabled plugin was always + discovered and loaded. Stored values are kept, and saving the General tab no + longer rewrites them to `false`. +- `BackgroundDataService` shares the 6-hour "ESPN rejects date ranges" memo + with `fetch_espn_scoreboard`, so a background season fetch no longer spends a + doomed range request first once either path has seen a rejection. +- `scripts/install_plugin_dependencies.sh` installs from the configured + `plugin_system.plugins_directory` (default `plugin-repos`, where the Plugin + Store installs) and also scans `plugins/` for dev symlinks. It used to scan + only `plugins/` and find nothing. A failed `pip install` is now reported as a + failure instead of being hidden by `tee`. +- `scripts/verify_installation.sh` no longer fails a healthy install: it + checked for the removed `web_interface_v2.py` and port 5001. It and + `scripts/verify_web_ui.sh` now check port 5000, where the web interface + listens. +- `scripts/install/install_service.sh --help` prints usage and exits without + changes. It used to ignore the flag and reinstall and restart every service. + Unknown arguments are rejected before anything runs. +- `scripts/diagnose_web_ui.sh`, `scripts/diagnose_web_interface.sh` and + `scripts/debug/debug_web_manual.py` apply the launcher's own autostart rule + (only an explicit `web_display_autostart: false` keeps the web interface + down), so a missing key no longer shows as disabled. The shell scripts also + check `web_interface/blueprints/api_v3/`, which became a package, instead of + reporting `api_v3.py` as missing. + +Docs and developer tools: + +- `docs/REST_API_REFERENCE.md` rechecked against every handler: request + fields that made documented calls fail (`repo_url`, `action_id`/`params`, + `files`/`image_id`, `font_file`+`font_family`, `?font=`, cache `key`, + `auto_enable_ap_mode`, plugin limit keys) and response shapes are fixed, the + removed font-override endpoints are gone, and the 26 undocumented routes + (backup, git/auto-update, WiFi radio, Starlark editor, MQTT bridge, status + endpoints, skins) are listed. Store search is `/plugins/store/list?query=`. +- `FONT_MANAGER.md` no longer tells plugins to read + `display_manager.font_manager`, which does not exist; use + `plugin_manager.font_manager` / `BasePlugin._get_font_manager()`. +- Plugin docs, `DisplayManager` docstrings and the bundled `starlark-apps` + plugin now all read the display size from `display_manager.width/height`, + which works in fallback mode where `matrix` is `None`. +- `scripts/dev/dev_plugin_setup.sh link-github ` links the plugin from a + clone of the `ledmatrix-plugins` monorepo (per-plugin `ledmatrix-` + repositories no longer exist). `dev_plugins.json` honours `github_user`, + `plugins_repo` and `plugins_branch`; `dev_plugins.json.example` ships and + `dev_plugins.json` is git-ignored. `update`/`status` handle monorepo links, + and `status` no longer exits 1 when nothing is broken. +- Rewritten for current behaviour: plugin dependency installation (web service + runs as the installing user and installs through `safe_pip_install.sh`), + `PLUGIN_CONFIG_ARCHITECTURE.md`, `MULTI_ROOT_WORKSPACE_SETUP.md`; stale + `app.py` line numbers, `api_v3.py` paths, StreamManager method names, + nonexistent version-bump scripts and `ledmatrix` service user references + removed. + ## 3.4.0 Plugin-facing changes since 3.3.0 (tag `v3.3.1`) not covered further down: diff --git a/CLAUDE.md b/CLAUDE.md index 764b637a..08942c40 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,7 @@ `config.get(...)`, never a separate accessor ## Dev Workflow -- Link a plugin for development: `./scripts/dev/dev_plugin_setup.sh link-github ` (or `link `); symlinks land in `plugins/` — set `plugin_system.plugins_directory` to `plugins` so discovery picks them up +- Link a plugin for development: `./scripts/dev/dev_plugin_setup.sh link-github ` clones the `ledmatrix-plugins` monorepo into `~/.ledmatrix-dev-plugins/` and links its `plugins/` under the manifest id (add a repo URL for a plugin with its own repo; or `link `); symlinks land in `plugins/` — set `plugin_system.plugins_directory` to `plugins` so discovery picks them up. Fork/location overrides: `dev_plugins.json` (from `dev_plugins.json.example`) - Browser preview without the display loop: `python3 scripts/dev_server.py` → http://localhost:5001 - Full display in emulator mode: `python3 run.py -e` (or `EMULATOR=true python3 run.py`) - Validate one plugin headlessly: `python3 scripts/check_plugin.py --plugin ` @@ -63,4 +63,4 @@ `self.display_manager.image.paste(img, (x, y))` then `update_display()` (use a mask for transparency: `image.paste(rgba, (x, y), rgba)`) - When modifying a plugin in the monorepo, you MUST bump `version` in its `manifest.json` and run `python update_registry.py` — otherwise users won't receive the update -- `src/pi5_matrix_support.py` hardcodes what the pinned `rpi-rgb-led-matrix-master` can drive on a Raspberry Pi 5 (`Rp1PioConfigSupported()` in `lib/rp1/rp1_pio_backend.cc`). Re-check it whenever the submodule is bumped: a stale rule blocks Pi 5 settings the new library supports, and a missing one lets the display service crash-loop +- `src/pi5_matrix_support.py` hardcodes what the pinned `rpi-rgb-led-matrix-master` can drive on a Raspberry Pi 5 (`Rp1PioConfigSupported()` in `lib/rp1/rp1_pio_backend.cc`). Re-check it whenever the submodule is bumped: a stale rule blocks Pi 5 settings the new library supports, and a missing one lets the display service crash-loop. `src/matrix_support.py` holds the same kind of rules for every board (rows, chain length, mapping names, parallel per mapping) and needs the same re-check diff --git a/README.md b/README.md index 0a77ce1f..96bdb1c9 100644 --- a/README.md +++ b/README.md @@ -148,7 +148,7 @@ The system supports live, recent, and upcoming game information for multiple spo ```bash sudo RPI_RGB_FORCE_REBUILD=1 ./first_time_install.sh ``` - - Pi 5 config: leave `rp1_rio` at `0` (PIO mode, default) and set `gpio_slowdown` to `1` or `2`. + - Pi 5 config: leave `rp1_rio` at `0` (PIO mode, default) and start `gpio_slowdown` at `1`, raising it a step at a time if the image flickers or shows garbage (see `gpio_slowdown` under Display Settings). - **1GB models (Pi 3B / 3B+) and other low-memory boards**: supported, but the `rpi-rgb-led-matrix` C++ build needs more memory than the Pi has. The installer detects this automatically, compiles with fewer parallel jobs, and adds a temporary swapfile for the build which it removes afterwards. Expect that step to take 15-25 minutes instead of 2-5, and leave at least **3GB free** on the SD card. If you manage swap yourself, opt out with `--skip-swap`. To pin the compiler down further, use `--build-jobs 1`. @@ -485,6 +485,10 @@ If you are copying my exact setup, you can likely leave the defaults alone. Howe The display settings are located in `config/config.json` under the `"display"` key and are organized into three main sections: `hardware`, `runtime`, and `display_durations`. +The defaults below are the values in `config/config.template.json`. They are what applies when you haven't set a key: on every load, LEDMatrix adds any key your `config.json` lacks from the template, so `DisplayManager`'s own fallbacks are never reached on a normal install. + +The web UI and the config API refuse values the rgbmatrix library can't start with. If one is written into `config.json` by hand anyway, the display logs which setting it is (`Failed to initialize RGB Matrix` in `sudo journalctl -u ledmatrix`), runs in fallback mode, and the Display tab shows the message. + ### Hardware Configuration (`display.hardware`) These settings control the physical hardware configuration and how the matrix is driven. @@ -494,9 +498,7 @@ These settings control the physical hardware configuration and how the matrix is - **`rows`** (integer, default: 32) - Number of LED rows (vertical pixels) in each panel - Common values: 16, 32, 48, 64 - - Must be an even number, at least 8. LEDMatrix sets no upper limit, but the - current rgbmatrix library rejects more than 64 rows per panel — the display - then won't start (see Troubleshooting Display Settings below) + - An even number from 8 to 64, the most the rgbmatrix library drives per panel - Must match your physical panel configuration - **`cols`** (integer, default: 64) @@ -507,7 +509,7 @@ These settings control the physical hardware configuration and how the matrix is - **`chain_length`** (integer, default: 2) - Number of LED panels chained together horizontally - - At least 1, with no upper limit; longer chains lower the refresh rate + - 1 to 255 (the library's Python binding stores it in one byte); longer chains lower the refresh rate - If you have 2 panels side-by-side, set to 2 - If you have 4 panels in a row, set to 4 - Total display width = `cols × chain_length` @@ -516,7 +518,7 @@ These settings control the physical hardware configuration and how the matrix is - Number of parallel chains (panels stacked vertically) - Use 1 for a single row of panels - Use 2 if you have panels stacked in two rows - - 1–3 on a Raspberry Pi, and the HAT needs that many outputs + - 1–3, and no more than your `hardware_mapping` has outputs: `regular` and `classic` have 3 (e.g. the Adafruit Triple LED Matrix Bonnet); `adafruit-hat`, `adafruit-hat-pwm`, `regular-pi1` and `classic-pi1` have 1. The library stops the display service outright on a mismatch, so it is refused - Total display height = `rows × parallel` #### Brightness and Visual Settings @@ -529,12 +531,14 @@ These settings control the physical hardware configuration and how the matrix is #### Hardware Mapping -- **`hardware_mapping`** (string, default: "adafruit-hat-pwm") +- **`hardware_mapping`** (string, default: "adafruit-hat") - Specifies which GPIO pin mapping to use for your hardware - **`"adafruit-hat-pwm"`**: Use this for Adafruit RGB Matrix Bonnet/HAT WITH the jumper mod (PWM enabled). This is the recommended setting for Adafruit hardware with the PWM jumper soldered. - **`"adafruit-hat"`**: Use this for Adafruit RGB Matrix Bonnet/HAT WITHOUT the jumper mod (no PWM). Remove `-pwm` from the value if you did not solder the jumper. - **`"regular"`**: Standard GPIO pin mapping for direct GPIO connections (Generic). Also the right choice for the Adafruit Triple LED Matrix Bonnet - **`"regular-pi1"`**: Standard GPIO pin mapping for Raspberry Pi 1 (older hardware or non-standard hat mapping) + - **`"classic"`** / **`"classic-pi1"`**: the library's original pin-outs, for old adapter boards wired to them. Not used by current HATs + - Any other name is refused. `compute-module` is only compiled in when the library is built with `ENABLE_WIDE_GPIO_COMPUTE_MODULE`, which the installer doesn't do. On a Raspberry Pi 5, `classic-pi1` isn't supported - Choose the option that matches your specific hardware setup, if aren't sure try them all. - Hardware pulsing (see `disable_hardware_pulsing`) needs the panel's OE line on GPIO 18, which `adafruit-hat-pwm` and `regular` provide and `adafruit-hat` does not @@ -571,7 +575,6 @@ These settings affect color fidelity and smoothness of color transitions: - Caps the panel refresh rate in Hz; `0` = no cap - A steady cap reduces flicker caused by other activity on the Pi, and in camera recordings - Scroll speeds are worked out against this value (against 100 Hz when it is `0`), so a cap the panel can actually hold keeps scrolling even - - If the key is missing from the config, `DisplayManager` uses 90 - Recommended: 80-120. `sudo python3 scripts/scroll_speeds.py --measure` reports the rate your panel really achieves - **`disable_hardware_pulsing`** (boolean, default: false) @@ -613,8 +616,10 @@ These settings are typically only needed for non-standard panels or custom confi - Set to `"180"` (or use the "Upside Down" option in the web UI's Display settings) if the panel is mounted upside down — useful for optimizing where the Raspberry Pi and wiring sit relative to the mounting location + - `"90"` and `"270"` are for a panel mounted on its side; they swap the + display's width and height - Applied independently of `pixel_mapper_config` (appended as a trailing - `Rotate:180` mapper), so custom mapper configs keep working alongside it + `Rotate:` mapper), so custom mapper configs keep working alongside it - **`row_address_type`** (integer, default: 0) - How rows are addressed on the panel @@ -663,7 +668,7 @@ These settings control runtime behavior and GPIO timing: - **Raspberry Pi Zero/1**: 0-1 - **Raspberry Pi 2/3**: 1-3 - **Raspberry Pi 4**: 2-4 (the config template ships 3) - - **Raspberry Pi 5**: 1–3 in PIO mode (`rp1_rio: 0`, the default); start with `1` and increase if you see flickering + - **Raspberry Pi 5**: 1–3 in PIO mode (`rp1_rio: 0`, the default). Start at `1` (the library treats `0` as `1` there) and raise it a step at a time if the image flickers or shows garbage — chained panels are the likeliest to need it - Panels on `row_address_type` 5 (SM5368 row drivers) can need 6-8 on a Pi 4 - Too low: garbage, flicker or rows jumping. Too high: a lower refresh rate - If you experience issues, try adjusting this value up or down by 1 @@ -752,7 +757,7 @@ Controls how long each installed plugin stays visible in seconds before switchin - Verify `hardware_mapping` matches your HAT/connection type - Try adjusting `gpio_slowdown` - Ensure your display doesn't need the E-Addressable line -- If it went blank right after a settings change, check `sudo journalctl -u ledmatrix` for `Failed to initialize RGB Matrix`: the rgbmatrix library refused a value (for example more than 64 `rows`, or `pwm_dither_bits` above 2) and the display fell back to no output. The library's own message nearby names the setting; on a Raspberry Pi 5, LEDMatrix's message names any unsupported `row_address_type`, `parallel` or `hardware_mapping` +- If it went blank right after a settings change, the Display tab shows a "simulation mode" banner, and `sudo journalctl -u ledmatrix` shows `Failed to initialize RGB Matrix` followed by the reason. When LEDMatrix refused the settings (for example more than 64 `rows`, `parallel` 2 on an `adafruit-hat` mapping, a misspelled `hardware_mapping`, or on a Raspberry Pi 5 a `row_address_type` other than 0 or 2), the message names each one: change them, save, and restart the display service. Otherwise the library itself failed, and its own message just before names the problem - A repeating scramble points at `row_address_type` or `multiplexing`; a panel that stays dark, at `panel_type` **Rows jump up and down, or the bottom row repeats other rows:** @@ -834,9 +839,11 @@ sudo ./scripts/install/install_service.sh The script will: - Detect your user account and home directory -- Install the service file with the correct paths -- Enable the service to start on boot -- Start the service immediately +- Install `ledmatrix.service` (display, runs as root), `ledmatrix-web.service` + (web interface, runs as your user) and the `ledmatrix-update-verify` units, + with the correct paths +- Enable them to start on boot +- Start them immediately ### Managing the Service diff --git a/dev_plugins.json.example b/dev_plugins.json.example new file mode 100644 index 00000000..64f26abb --- /dev/null +++ b/dev_plugins.json.example @@ -0,0 +1,6 @@ +{ + "dev_plugins_dir": "~/.ledmatrix-dev-plugins", + "github_user": "ChuckBuilds", + "plugins_repo": "ledmatrix-plugins", + "plugins_branch": "main" +} diff --git a/docs/ADVANCED_FEATURES.md b/docs/ADVANCED_FEATURES.md index d620fef7..7506bc8b 100644 --- a/docs/ADVANCED_FEATURES.md +++ b/docs/ADVANCED_FEATURES.md @@ -377,9 +377,16 @@ Vegas mode consists of four core components working together to provide smooth 1 5. Compose into continuous stream with separators **Key Methods:** -- `get_stream_content()` - Returns current stream content as PIL Image -- `advance_stream(pixels)` - Advances stream by N pixels -- `refresh_stream()` - Regenerates stream from current plugins +- `get_next_segment()` - Returns the next buffered `ContentSegment` (or `None`) +- `take_next_group(count=None, offscreen_only=False)` - Hands over the next + slice of the rotation as `(plugin_id, images)` groups +- `get_grouped_content_for_composition()` - Buffered images grouped by plugin +- `mark_plugin_updated(plugin_id)` / `process_updates()` - Refresh one + plugin's segment in place when its data changes +- `refresh()` - Re-read the plugin list and config +- `advance_cycle()` - Clear the active buffer when a scroll cycle completes + +(`src/vegas_mode/stream_manager.py`) #### 3. PluginAdapter @@ -433,10 +440,14 @@ Vegas mode consists of four core components working together to provide smooth 1 - **Frame Rate Control:** Precise timing to maintain 125 FPS - **Pre-rendered Content:** Plugins pre-render during update() -**Scroll Speed Calculation:** +**Scroll Speed Calculation:** motion is by elapsed time; `target_fps` paces +the render loop, not the speed. ```python -pixels_per_frame = (scroll_speed / target_fps) -scroll_position += pixels_per_frame * elapsed_time +# frame_based_scrolling: false +scroll_position += scroll_speed * elapsed_time # scroll_speed in px/s +# frame_based_scrolling: true (the default) -- not stepping, just a clamp +applied = clamp(scroll_speed * scroll_delay, 0.1, 5) / scroll_delay +scroll_position += applied * elapsed_time ``` #### Component Interactions @@ -552,7 +563,8 @@ time when something is active. ### REST API Reference -The API is mounted at `/api/v3` (`web_interface/app.py:199`). +The API is mounted at `/api/v3` (the `api_v3` blueprint, registered in +`web_interface/app.py`). Full details: [REST_API_REFERENCE.md](REST_API_REFERENCE.md#display-control). #### Start On-Demand Display @@ -608,20 +620,30 @@ curl http://localhost:5000/api/v3/display/on-demand/status # Response: { - "active": true, - "plugin_id": "weather", - "mode": "weather", - "remaining": 25.5, - "pinned": false, - "status": "active" + "status": "success", + "data": { + "state": { + "active": true, + "plugin_id": "weather", + "mode": "weather", + "duration": 30, + "pinned": false, + "status": "running", + "last_updated": 1234567890.1 + }, + "service": {"active": true, "returncode": 0, "stdout": "active", "stderr": ""} + } } ``` +When nothing is running on demand, `data.state` is +`{"active": false, "status": "idle", "last_updated": null}`. + > There is no public Python on-demand API. The display controller's > on-demand machinery is internal — drive it through the REST endpoints > above (or the web UI buttons). The API handlers > (`start_on_demand_display()` / `stop_on_demand_display()` in -> `web_interface/blueprints/api_v3.py`) write a request into the cache +> `web_interface/blueprints/api_v3/display.py`) write a request into the cache > manager under the `display_on_demand_request` key, which > `DisplayController._poll_on_demand_requests()` > (`src/display_controller.py`) picks up. A separate diff --git a/docs/CONFIG_DEBUGGING.md b/docs/CONFIG_DEBUGGING.md index 2454f755..4adae1fb 100644 --- a/docs/CONFIG_DEBUGGING.md +++ b/docs/CONFIG_DEBUGGING.md @@ -250,14 +250,21 @@ WARNING - Plugin ID 'Football-Scoreboard' may conflict with 'football-scoreboard ## Checking Configuration via API -The API blueprint mounts at `/api/v3` (`web_interface/app.py:144`). +The API blueprint (`web_interface/blueprints/api_v3/`) is registered at +`/api/v3` in `web_interface/app.py`. ```bash -# Get full main config (includes all plugin sections) +# Get full main config (includes all plugin sections; credential-named +# fields are blanked in the response) curl http://localhost:5000/api/v3/config/main -# Save updated main config +# Change some settings: only the keys you send are changed curl -X POST http://localhost:5000/api/v3/config/main \ + -H "Content-Type: application/json" \ + -d '{"timezone": "America/Chicago", "brightness": 80}' + +# Replace config.json wholesale (advanced) +curl -X POST http://localhost:5000/api/v3/config/raw/main \ -H "Content-Type: application/json" \ -d @new-config.json @@ -269,8 +276,10 @@ curl "http://localhost:5000/api/v3/plugins/config?plugin_id=football-scoreboard" ``` > There is no dedicated `/config/plugin/` or `/config/validate` -> endpoint — config validation runs server-side automatically when you -> POST to `/config/main` or `/plugins/config`. See +> endpoint. `POST /plugins/config` validates against the plugin's schema +> and rejects an invalid config with `400`; `POST /config/main` checks the +> individual fields it knows (display hardware values, durations, Vegas +> and sync settings). See > [REST_API_REFERENCE.md](REST_API_REFERENCE.md) for the full list. ## Backup and Recovery diff --git a/docs/CONFIG_REFERENCE.md b/docs/CONFIG_REFERENCE.md index f4d991ca..cdc7b786 100644 --- a/docs/CONFIG_REFERENCE.md +++ b/docs/CONFIG_REFERENCE.md @@ -17,7 +17,7 @@ tooling against it. |---|---|---|---| | `web_display_autostart` | bool, `true` | Whether the web interface service starts with the system | `scripts/utils/start_web_conditionally.py` | | `timezone` | string, `"America/New_York"` | IANA timezone for schedules and displays | `ConfigManager.get_timezone()` | -| `target_fps` | int, `100` | Frame-rate ceiling for plugin rendering | `src/plugin_system/base_plugin.py`, `src/common/sports_scroll.py` | +| `target_fps` | int, `100` | Legacy "Scroll Frame Rate". Core scrolling no longer reads it: scroll frames are presented at `display.hardware.limit_refresh_rate_hz` divided by each scroll's frame hold, and speed comes from each plugin's scroll settings. Still exposed to plugins via `BasePlugin.global_config` | `src/plugin_system/base_plugin.py` | | `location` | object | `city` / `state` / `country`. Supplies the **default** for a plugin's own `location_city` / `location_state` / `location_country` setting, so weather, radar and friends follow this device without being configured twice. A value saved on the plugin itself still overrides it. | `SchemaManager.apply_device_location()`, then plugins via merged config | ## `schedule` — display on/off hours @@ -47,38 +47,43 @@ saved via `POST /api/v3/config/dim-schedule`). The display returns to ## `display.hardware` — matrix panel hardware All keys map to the corresponding `rpi-rgb-led-matrix` options and are read -in `DisplayManager` (`src/display_manager.py`, ~lines 270–295). +in `DisplayManager._setup_matrix` (`src/display_manager.py`). Defaults are the +`config/config.template.json` values: `ConfigManager` adds any key missing from +`config.json` from the template on load, so `DisplayManager`'s own fallbacks +don't apply on a normal install. + +The ranges are what the pinned rgbmatrix library and its Python binding accept +(`src/matrix_support.py`). The config API refuses anything else; a value +hand-edited into `config.json` makes the display log the setting and run in +fallback mode instead of starting the matrix. | Key | Type / default | |---|---| -| `rows` / `cols` | int, `32` / `64` — rows: even, at least 8, no upper limit here (the current rgbmatrix library rejects more than 64); cols: at least 16, no upper limit | -| `chain_length` | int, `2` — at least 1, no upper limit | -| `parallel` | int, `1` — 1–3 | +| `rows` / `cols` | int, `32` / `64` — rows: even, 8–64; cols: at least 16 | +| `chain_length` | int, `2` — 1–255 (the Python binding stores it in one byte) | +| `parallel` | int, `1` — 1–3, and no more than `hardware_mapping` has outputs (`regular`, `classic`: 3; the others: 1) | | `brightness` | int, `90` — 1–100 | -| `hardware_mapping` | string, `"adafruit-hat"` (code default `"adafruit-hat-pwm"`) | +| `hardware_mapping` | string, `"adafruit-hat"` — `"adafruit-hat-pwm"`, `"adafruit-hat"`, `"regular"`, `"regular-pi1"`, `"classic"` or `"classic-pi1"` (case-insensitive; `compute-module` isn't in the installed build). A Pi 5 doesn't support `"classic-pi1"` | | `scan_mode` | int, `0` — `0` progressive, `1` interlaced | -| `pwm_bits` | int, `9` (code default 10) — 1–11 | +| `pwm_bits` | int, `9` — 1–11 | | `pwm_dither_bits` | int, `1` — 0–2 | -| `pwm_lsb_nanoseconds` | int, `130` (code default 150) — 50–3000 | +| `pwm_lsb_nanoseconds` | int, `130` — 50–3000 | | `disable_hardware_pulsing` | bool, `false` — `true` times brightness pulses in software (less exact); hardware pulsing needs the OE line on GPIO 18 and the Pi's onboard sound driver off | | `inverse_colors` | bool, `false` | | `show_refresh_rate` | bool, `false` — prints the refresh rate to stdout; draws nothing on the panel | | `led_rgb_sequence` | string, `"RGB"` — `"RGB"`, `"RBG"`, `"GRB"`, `"GBR"`, `"BRG"` or `"BGR"` | -| `limit_refresh_rate_hz` | int, `100` (code default 90) — `0` = no cap; scroll timing assumes 100 Hz when `0` | -| `pixel_mapper_config` | string, `""` — e.g. `"U-mapper"` / `"Rotate:90"` | -| `orientation` | string, `"normal"` — `"180"` rotates the rendered image 180° for panels physically mounted upside down (e.g. to move the Pi/wiring to a more convenient side); composed onto `pixel_mapper_config` as a trailing `Rotate:180` mapper, so it stays independent of any custom `pixel_mapper_config` value | +| `limit_refresh_rate_hz` | int, `100` — `0` = no cap; scroll timing assumes 100 Hz when `0` | +| `pixel_mapper_config` | string, `""` — e.g. `"U-mapper"` / `"Rotate:90"`; mappers that rotate or fold the chain change the display size plugins and the web preview see | +| `orientation` | string, `"normal"` — `"180"` rotates the rendered image 180° for panels physically mounted upside down (e.g. to move the Pi/wiring to a more convenient side); `"90"` / `"270"` for a panel on its side, swapping width and height; composed onto `pixel_mapper_config` as a trailing `Rotate:` mapper, so it stays independent of any custom `pixel_mapper_config` value | | `row_address_type` | int, `0` — non-standard panel row addressing: `1` AB, `2` direct row select, `3` ABC, `4` ABC shift + DE direct, `5` SM5368 / B707 row shift register (e.g. Waveshare 96x48 V2, with `led_rgb_sequence` `"BGR"`). On a Pi 5 the library supports only `0` and `2`, and LEDMatrix enforces that (`src/pi5_matrix_support.py`) | | `multiplexing` | int, `0` — 0–22, pixel wiring scheme for outdoor/specialty panels (names listed in the README) | | `panel_type` | string, `""` — set to `"FM6126A"` or `"FM6127"` for panels needing init; FM6124 / FM6124D / FM6124DJ panels need none, so leave it `""` | -Where "code default" differs from the template value, the code default only -applies if the key is missing entirely from your config. - ## `display.runtime` | Key | Type / default | Meaning | |---|---|---| -| `gpio_slowdown` | int, `3` | GPIO timing slowdown for faster Pis (0–10). Panels on `row_address_type` `5` (SM5368 row drivers) can need 6–8 on a Pi 4 — lower values make rows jump | +| `gpio_slowdown` | int, `3` | GPIO timing slowdown for faster Pis (0–10). On a Pi 5 in PIO mode start at `1` (`0` acts as `1`) and raise it if the image flickers or shows garbage. Panels on `row_address_type` `5` (SM5368 row drivers) can need 6–8 on a Pi 4 — lower values make rows jump | | `rp1_rio` | int, `0` | Pi 5 only: `0` = PIO (less CPU), `1` = RIO (higher refresh; `gpio_slowdown` effect inverted). Applied only if the installed matrix library supports it | ## `display.double_sided` @@ -134,8 +139,8 @@ Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See | `dynamic_duration_enabled` | bool, `true` | | `min_cycle_duration` | int, `60` | | `max_cycle_duration` | int, `240` | -| `frame_based_scrolling` | bool, `true` — frame-count-based scroll stepping | -| `scroll_delay` | float, `0.02` — seconds between scroll updates (~50 FPS) | +| `frame_based_scrolling` | bool, `true` — does not step or set a frame rate; motion is by elapsed time either way. When `true`, `scroll_speed` passes through a clamp of 0.1–5 px per `scroll_delay` (see next row) | +| `scroll_delay` | float, `0.02` — not a frame period. Only used with `frame_based_scrolling`: the applied speed is `clamp(scroll_speed × scroll_delay, 0.1, 5) / scroll_delay` px/s, so at `0.02` speeds under 5 px/s run at 5, and at `0.001` nothing runs slower than 100 px/s | | `live_in_ticker` | bool, `false` — keep scrolling during live games instead of handing the display to a full-screen scoreboard | | `live_weight` | int, `3` (1–10) — slots per cycle for a plugin with live content | | `favorite_live_weight` | int, `5` (1–10) — slots per cycle when a plugin reports a favorite team is live | @@ -152,14 +157,12 @@ Read by `src/common/sync_manager.py` and `src/display_controller.py`. ## `plugin_system` -Read by the plugin loader/manager (`src/plugin_system/`). - | Key | Type / default | Meaning | |---|---|---| -| `plugins_directory` | string, `"plugin-repos"` | Where the Plugin Store installs plugins | -| `auto_discover` | bool, `true` | Scan the plugins directory at startup | -| `auto_load_enabled` | bool, `true` | Load discovered plugins automatically | -| `development_mode` | bool, `false` | Development conveniences in the web UI (editable under General settings) | +| `plugins_directory` | string, `"plugin-repos"` | Where the Plugin Store installs plugins and the only directory the plugin loader scans. Read by `PluginManager` and `PluginStoreManager` (`src/plugin_system/`); editable under General settings | +| `auto_discover` | bool, `true` | **Unused.** Legacy key, read by nothing. Plugins are always discovered, and every plugin with `enabled: true` is loaded. Not shown in the web UI; may be left in or removed from config.json | +| `auto_load_enabled` | bool, `true` | **Unused.** Legacy key, read by nothing (see `auto_discover`). To keep a plugin installed but dormant, set its own `enabled` to `false` | +| `development_mode` | bool, `false` | **Unused.** Legacy key, read by nothing | ## Plugin config blocks diff --git a/docs/FONT_MANAGER.md b/docs/FONT_MANAGER.md index e6691dd7..3678e955 100644 --- a/docs/FONT_MANAGER.md +++ b/docs/FONT_MANAGER.md @@ -12,10 +12,28 @@ The enhanced FontManager provides comprehensive font management for the LEDMatrix application with support for: - Manager font registration and detection - Plugin font management -- Manual font overrides via web interface +- Programmatic per-element font overrides - Performance monitoring and caching - Dynamic font discovery +## Getting the FontManager + +There is one shared FontManager per display process. The display controller +creates it and hands it to the `PluginManager`, so a plugin reaches it +through its `plugin_manager`: + +```python +class MyPlugin(BasePlugin): + def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager): + super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) + self.font_manager = self._get_font_manager() +``` + +`BasePlugin._get_font_manager()` returns `plugin_manager.font_manager`, or a +standalone FontManager when none is available (test harnesses, mocks). +`DisplayManager` has **no** `font_manager` attribute — +`display_manager.font_manager` raises `AttributeError`. + ## Architecture ### Manager-Centric Design @@ -40,8 +58,9 @@ Manager requests font → Check manual overrides → Apply manager choice → Ca from src.font_manager import FontManager class MyManager: - def __init__(self, config, display_manager, cache_manager): - self.font_manager = display_manager.font_manager # Access shared FontManager + def __init__(self, config, display_manager, cache_manager, plugin_manager): + self.display_manager = display_manager + self.font_manager = plugin_manager.font_manager # Shared FontManager self.manager_id = "my_manager" def display(self): @@ -80,8 +99,9 @@ class MyManager: ```python class AdvancedManager: - def __init__(self, config, display_manager, cache_manager): - self.font_manager = display_manager.font_manager + def __init__(self, config, display_manager, cache_manager, plugin_manager): + self.display_manager = display_manager + self.font_manager = plugin_manager.font_manager self.manager_id = "advanced_manager" # Define your font specifications @@ -152,19 +172,13 @@ font = self.font_manager.resolve_font( > URIs documented below are resolved relative to the plugin's > install directory. > -> The **Fonts** tab in the web UI that lists detected -> manager-registered fonts is still a **placeholder -> implementation** — fonts that managers register through -> `register_manager_font()` do not yet appear there. The -> programmatic per-element override workflow described in -> [Manual Font Overrides](#manual-font-overrides) below -> (`set_override()` / `remove_override()` / the -> `config/font_overrides.json` store) **does** work today and is -> the supported way to override a font for an element until the -> Fonts tab is wired up. If you can't wait and need a workaround -> right now, you can also just load the font directly with PIL -> (or `freetype-py` for BDF) inside your plugin's `manager.py` -> and skip the override system entirely. +> The web UI's **Fonts** tab lists, uploads, previews and deletes the +> font files in `assets/fonts/`. It does not show fonts registered +> through `register_manager_font()` and has no override editor (the +> override panels and `/api/v3/fonts/overrides` endpoints were removed). +> The programmatic override workflow in +> [Manual Font Overrides](#manual-font-overrides) below still works. +> Let users pick fonts through your plugin's own config schema. ### Plugin Font Registration @@ -200,10 +214,10 @@ In your plugin's `manifest.json`: ### Using Plugin Fonts ```python -class PluginManager: - def __init__(self, config, display_manager, cache_manager, plugin_id): - self.font_manager = display_manager.font_manager - self.plugin_id = plugin_id +class MyPlugin(BasePlugin): + def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager): + super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) + self.font_manager = self._get_font_manager() def display(self): # Use plugin font (automatically namespaced) @@ -219,17 +233,8 @@ class PluginManager: ## Manual Font Overrides -Users can override any font through the web interface: - -1. Navigate to **Fonts** tab -2. View **Detected Manager Fonts** to see what's currently in use -3. In **Element Overrides** section: - - Select the element (e.g., "nfl.live.score") - - Choose a different font family - - Choose a different size - - Click **Add Override** - -Overrides are stored in `config/font_overrides.json` and persist across restarts. +Overrides are set in code (there is no web UI or REST endpoint for them). +They are stored in `config/font_overrides.json` and persist across restarts. ### Programmatic Overrides diff --git a/docs/MIGRATION_GUIDE.md b/docs/MIGRATION_GUIDE.md index b64269b9..c6b17c94 100644 --- a/docs/MIGRATION_GUIDE.md +++ b/docs/MIGRATION_GUIDE.md @@ -59,9 +59,12 @@ sudo ./scripts/install/install_service.sh After updating your scripts, verify they still work: ```bash -# Test installation scripts (if needed) +# Check the installation scripts are at their new paths ls scripts/install/*.sh -sudo ./scripts/install/install_service.sh --help +./scripts/install/install_service.sh --help # prints usage only +# Note: running install_service.sh for real (with sudo, no --help) +# reinstalls, enables and restarts ledmatrix.service, ledmatrix-web.service +# and the update-verify units. # Test permission scripts ls scripts/fix_perms/*.sh diff --git a/docs/MULTI_ROOT_WORKSPACE_SETUP.md b/docs/MULTI_ROOT_WORKSPACE_SETUP.md index 2fc0a833..56bbab2f 100644 --- a/docs/MULTI_ROOT_WORKSPACE_SETUP.md +++ b/docs/MULTI_ROOT_WORKSPACE_SETUP.md @@ -1,169 +1,154 @@ # Multi-Root Workspace Setup Guide -This document explains how the LEDMatrix project uses a multi-root workspace to manage plugins as separate Git repositories. +This document explains how to work on LEDMatrix and the official plugins side +by side, with one editor workspace and the plugins loaded straight from your +plugin checkout. ## Overview -The LEDMatrix project has been migrated from a git submodule implementation to a **multi-root workspace** implementation for managing plugins. This allows: +Official plugins live in a single repository, +[ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins), with one +directory per plugin under `plugins/`. There are no separate per-plugin +repositories. For development you clone that monorepo **next to** LEDMatrix +and symlink its plugin directories into LEDMatrix's `plugin-repos/`, which is +where the plugin loader looks by default. -- ✅ Plugins to exist as independent Git repositories -- ✅ Updates to plugins without modifying the LEDMatrix project -- ✅ Easy development workflow with all repos in one workspace -- ✅ Plugin system discovers plugins via symlinks in `plugin-repos/` +- ✅ Plugin code stays in the monorepo checkout, with its own git history +- ✅ LEDMatrix discovers the plugins through symlinks in `plugin-repos/` +- ✅ `LEDMatrix.code-workspace` opens both repositories in VS Code/Cursor ## Directory Structure ```text -/home/chuck/Github/ -├── LEDMatrix/ # Main project -│ ├── plugin-repos/ # Symlinks to actual repos (managed automatically) -│ │ ├── ledmatrix-clock-simple -> ../../ledmatrix-clock-simple -│ │ ├── ledmatrix-weather -> ../../ledmatrix-weather +~/Github/ +├── LEDMatrix/ # Main project +│ ├── plugin-repos/ # Plugin directory the loader scans +│ │ ├── starlark-apps/ # Bundled with LEDMatrix (tracked in git) +│ │ ├── web-ui-info/ # Bundled with LEDMatrix (tracked in git) +│ │ ├── clock-simple -> ../../ledmatrix-plugins/plugins/clock-simple +│ │ ├── ledmatrix-weather -> ../../ledmatrix-plugins/plugins/ledmatrix-weather │ │ └── ... -│ ├── LEDMatrix.code-workspace # Multi-root workspace configuration +│ ├── LEDMatrix.code-workspace # Opens LEDMatrix and ../ledmatrix-plugins │ └── ... -├── ledmatrix-clock-simple/ # Plugin repository (actual git repo) -├── ledmatrix-weather/ # Plugin repository (actual git repo) -├── ledmatrix-football-scoreboard/ # Plugin repository (actual git repo) -└── ... # Other plugin repos +└── ledmatrix-plugins/ # Plugin monorepo (git repo) + ├── plugins/ + │ ├── clock-simple/ + │ ├── ledmatrix-weather/ + │ └── ... + ├── plugins.json # Store registry + └── update_registry.py ``` ## How It Works -### 1. Plugin Repositories +### 1. The plugin monorepo -All plugin repositories are cloned to `/home/chuck/Github/` (parent directory of LEDMatrix) as regular Git repositories: +Clone ledmatrix-plugins into the same parent directory as LEDMatrix (the +scripts below look for `../ledmatrix-plugins` relative to the LEDMatrix +root): -- `ledmatrix-clock-simple/` -- `ledmatrix-weather/` -- `ledmatrix-football-scoreboard/` -- etc. +```bash +cd ~/Github +git clone https://github.com/ChuckBuilds/ledmatrix-plugins.git +``` ### 2. Symlinks in plugin-repos/ -The `LEDMatrix/plugin-repos/` directory contains symlinks pointing to the actual repositories in the parent directory. This allows the plugin system to discover plugins without modifying the project structure. +`scripts/setup_plugin_repos.py` creates one symlink per plugin in +`LEDMatrix/plugin-repos/`, named after the plugin's manifest `id` and pointing +at `../ledmatrix-plugins/plugins/`. -### 3. Multi-Root Workspace +### 3. Multi-root workspace -The `LEDMatrix.code-workspace` file configures VS Code/Cursor to open all plugin repositories as separate workspace roots, allowing easy development across all repos. +`LEDMatrix.code-workspace` has two roots: LEDMatrix itself and +`../ledmatrix-plugins`. ## Setup Scripts ### Initial Setup -If you already have plugin repositories cloned, use the setup script: - ```bash -cd /home/chuck/Github/LEDMatrix +cd ~/Github/LEDMatrix python3 scripts/setup_plugin_repos.py ``` This script: -- Reads the workspace configuration -- Creates symlinks in `plugin-repos/` pointing to actual repos -- Verifies all links are created correctly +- Reads each `manifest.json` under `../ledmatrix-plugins/plugins/` +- Creates `plugin-repos/` symlinks (relative) to those directories +- Leaves correct links alone, replaces links that point elsewhere, and skips + (does not overwrite) a real directory of the same name — for example a + plugin you installed from the Plugin Store. Remove that directory first if + you want the linked copy. ### Updating Plugins -To update all plugin repositories: - ```bash -cd /home/chuck/Github/LEDMatrix +cd ~/Github/LEDMatrix python3 scripts/update_plugin_repos.py ``` -This script: -- Finds all plugins in the workspace -- Runs `git pull` on each repository -- Reports which plugins were updated +This runs `git pull` in `../ledmatrix-plugins` and prints the result. The +symlinks pick up the new code; restart the display to load it. ## Configuration -The plugin system is configured in `config/config.json`: +The loader reads plugins from `plugin_system.plugins_directory` in +`config/config.json`. The default is already right for this setup: ```json { "plugin_system": { - "plugins_directory": "plugin-repos", - "auto_discover": true, - "auto_load_enabled": true + "plugins_directory": "plugin-repos" } } ``` -The `plugins_directory` points to `plugin-repos/`, which contains symlinks to the actual repositories. - ## Workflow ### Daily Development 1. **Open Workspace**: Open `LEDMatrix.code-workspace` in VS Code/Cursor -2. **All Repos Available**: All plugin repos appear as separate folders in the workspace -3. **Edit Plugins**: Edit plugin code directly in their repositories -4. **Update Plugins**: Run `update_plugin_repos.py` to pull latest changes +2. **Edit Plugins**: Edit code under `ledmatrix-plugins/plugins//` +3. **Test**: `python3 run.py -e` (emulator) or + `python3 scripts/check_plugin.py --plugin ` from LEDMatrix +4. **Ship**: Bump `version` in the plugin's `manifest.json`, run + `python update_registry.py` in ledmatrix-plugins, commit there ### Adding New Plugins -1. **Clone Repository**: Clone the new plugin repo to `/home/chuck/Github/` -2. **Add to Workspace**: Add the plugin folder to `LEDMatrix.code-workspace` -3. **Create Symlink**: Run `setup_plugin_repos.py` to create the symlink - -### Updating Individual Plugins - -Since plugins are regular Git repositories, you can update them individually: - -```bash -cd /home/chuck/Github/ledmatrix-weather -git pull origin master -``` - -Or update all at once: - -```bash -cd /home/chuck/Github/LEDMatrix -python3 scripts/update_plugin_repos.py -``` - -## Benefits - -1. **No Submodule Hassle**: No need to update `.gitmodules` or run `git submodule update` -2. **Independent Updates**: Update plugins independently without touching LEDMatrix -3. **Clean Separation**: Each plugin is a separate repository with its own history -4. **Easy Development**: Multi-root workspace makes it easy to work across repos -5. **Automatic Discovery**: Plugin system automatically discovers plugins via symlinks +1. Create `plugins//` in the monorepo checkout +2. Run `python3 scripts/setup_plugin_repos.py` in LEDMatrix to link it ## Troubleshooting -### Symlinks Not Working - -If plugins aren't being discovered: +### Plugins not discovered ```bash -cd /home/chuck/Github/LEDMatrix -python3 scripts/setup_plugin_repos.py +cd ~/Github/LEDMatrix +ls -la plugin-repos/ # links present and not broken? +python3 scripts/setup_plugin_repos.py # recreate them ``` -This will recreate all symlinks. +Also check that `plugin_system.plugins_directory` is `plugin-repos`. -### Missing Plugins +### "Monorepo plugins directory not found" -If a plugin is in the workspace but not found: +`setup_plugin_repos.py` expects the monorepo at `../ledmatrix-plugins`. Clone +it there (or symlink it there). -1. Check if the repo exists in `/home/chuck/Github/` -2. Check if the symlink exists in `plugin-repos/` -3. Run `setup_plugin_repos.py` to recreate symlinks +### Plugin updates not showing -### Plugin Updates Not Showing - -If changes to plugins aren't appearing: - -1. Verify the symlink points to the correct directory: `ls -la plugin-repos/ledmatrix-weather` -2. Check that you're editing in the actual repo, not a copy -3. Restart the LEDMatrix service if running +1. Verify the link target: `ls -la plugin-repos/` +2. Check that you're editing the monorepo checkout, not a store-installed copy +3. Restart the LEDMatrix service (or `run.py`) ## Notes -- The `plugin-repos/` directory is tracked in git, but only contains symlinks -- Actual plugin code lives in `/home/chuck/Github/ledmatrix-*/` -- Each plugin repo can be updated independently via `git pull` -- The LEDMatrix project doesn't need to be updated when plugins change +- `plugin-repos/` is tracked in git only for the bundled plugins + (`starlark-apps`, `web-ui-info`). The symlinks you create are untracked + files; don't commit them. +- For linking a single plugin into `plugins/` instead (without a sibling + checkout), see `scripts/dev/dev_plugin_setup.sh` in the + [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md). +- When changing a plugin in the monorepo, bump its manifest `version` and run + `python update_registry.py`, or users won't receive the update. diff --git a/docs/PLUGIN_API_REFERENCE.md b/docs/PLUGIN_API_REFERENCE.md index 0a98d0f1..d96d8259 100644 --- a/docs/PLUGIN_API_REFERENCE.md +++ b/docs/PLUGIN_API_REFERENCE.md @@ -514,21 +514,59 @@ self.display_manager.draw_text_with_icons( For plugins that implement scrolling content, use these methods to coordinate with the display system. -#### `set_scrolling_state(is_scrolling: bool) -> None` +#### `set_scrolling_state(is_scrolling: bool, frame_hold: int = 1) -> None` -Mark the display as scrolling or not scrolling. Call when scrolling starts/stops. +Mark the display as scrolling or not scrolling, and set this scroll's frame +pacing. Call it when a scroll starts (calling it on every scroll frame is fine) +and with `False` when it stops. **Parameters**: - `is_scrolling` (bool): True if currently scrolling, False otherwise +- `frame_hold` (int, default 1): how many panel refreshes each pushed frame is + held for (clamped to 1-255; ignored when `is_scrolling` is False, which + resets it to 1). Pass the `frame_hold` of the settings + `src.common.scroll_config.configure()` returned. Added in core 3.4.0. + +**Why `frame_hold` matters**: `scroll_config.configure()` snaps the speed to +one the panel can show in whole pixels and sets the `ScrollHelper` to advance a +fixed number of pixels on every presented frame -- no clock is consulted. The +panel presents frames at its refresh rate divided by the hold, so the hold is +part of the speed. Omit it and a 50 px/s scroll (1px every 2nd refresh on a +100 Hz panel) runs at 100 px/s. The hold is not applied by `configure()` +because it must not outlive the scroll: plugins share one display manager. **Example**: ```python +from src.common import scroll_config +from src.common.scroll_helper import ScrollHelper + +def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.scroll_helper = ScrollHelper( + self.display_manager.width, self.display_manager.height, self.logger) + # ...later, hand it content with self.scroll_helper.set_scrolling_image(img) + self.scroll_settings = scroll_config.configure( + self.scroll_helper, + plugin_config=self.config, + global_config=self.global_config, + display_manager=self.display_manager, + plugin_logger=self.logger, + ) + def display(self, force_clear=False): - self.display_manager.set_scrolling_state(True) - # Scroll content... - self.display_manager.set_scrolling_state(False) + self.display_manager.set_scrolling_state( + True, frame_hold=self.scroll_settings.frame_hold) + self.scroll_helper.update_scroll_position() + self.display_manager.image = self.scroll_helper.get_visible_portion() + self.display_manager.update_display() + if self.scroll_helper.is_scroll_complete(): + self.display_manager.set_scrolling_state(False) ``` +Don't pace the loop with `time.sleep()`: `update_display()` blocks on the +panel's vsync, which is what paces a scroll. See `docs/SCROLL_PERFORMANCE.md` +for choosing a speed. + #### `is_currently_scrolling() -> bool` Check if the display is currently in a scrolling state. @@ -998,9 +1036,10 @@ if "weather" in enabled_plugins: self.display_manager.update_display() ``` -3. **Handle scrolling state**: If your plugin scrolls, use scrolling state methods +3. **Handle scrolling state**: If your plugin scrolls, use scrolling state methods, + passing the frame hold `scroll_config.configure()` returned ```python - self.display_manager.set_scrolling_state(True) + self.display_manager.set_scrolling_state(True, frame_hold=settings.frame_hold) # Scroll content... self.display_manager.set_scrolling_state(False) ``` diff --git a/docs/PLUGIN_ARCHITECTURE_SPEC.md b/docs/PLUGIN_ARCHITECTURE_SPEC.md index 3c8ddb6f..8df78fd9 100644 --- a/docs/PLUGIN_ARCHITECTURE_SPEC.md +++ b/docs/PLUGIN_ARCHITECTURE_SPEC.md @@ -8,7 +8,8 @@ > - Code paths reference `web_interface_v2.py`; the current web UI is > `web_interface/app.py` with v3 Blueprint-based templates. > - The example Flask routes use `/api/plugins/*`; the real API -> blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). +> blueprint (`web_interface/blueprints/api_v3/`) is mounted at `/api/v3` +> in `web_interface/app.py`. > - The default plugin location is `plugin-repos/` (configurable via > `plugin_system.plugins_directory`), not `./plugins/`. > - Example imports use `src/plugin_system/base_classes/*_plugin.py`; diff --git a/docs/PLUGIN_CONFIGURATION_GUIDE.md b/docs/PLUGIN_CONFIGURATION_GUIDE.md index 7471130a..5c4956fa 100644 --- a/docs/PLUGIN_CONFIGURATION_GUIDE.md +++ b/docs/PLUGIN_CONFIGURATION_GUIDE.md @@ -67,9 +67,7 @@ The main configuration file (`config/config.json`) now contains only essential s "time_format": "%I:%M %p" }, "plugin_system": { - "plugins_directory": "plugin-repos", - "auto_discover": true, - "auto_load_enabled": true + "plugins_directory": "plugin-repos" } } ``` @@ -93,9 +91,9 @@ The main configuration file (`config/config.json`) now contains only essential s #### 4. Plugin System - **plugin_system**: Plugin system configuration - - **plugins_directory**: Directory where plugins are stored - - **auto_discover**: Automatically discover plugins - - **auto_load_enabled**: Automatically load enabled plugins + - **plugins_directory**: Directory where plugins are stored (the only one the loader scans) + - `auto_discover`, `auto_load_enabled`, `development_mode` may still appear in + older configs; nothing reads them (see [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md#plugin_system)) ## Plugin Configuration diff --git a/docs/PLUGIN_CONFIGURATION_TABS.md b/docs/PLUGIN_CONFIGURATION_TABS.md index ed5de33d..9fcecef4 100644 --- a/docs/PLUGIN_CONFIGURATION_TABS.md +++ b/docs/PLUGIN_CONFIGURATION_TABS.md @@ -6,7 +6,8 @@ > in the "Implementation Details" section below still reference the > pre-v3 file layout (`web_interface_v2.py`, `templates/index_v2.html`). > The current implementation lives in `web_interface/app.py`, -> `web_interface/blueprints/api_v3.py`, and `web_interface/templates/v3/`. +> `web_interface/blueprints/api_v3/` (plugin config handlers in +> `plugins.py`), and `web_interface/templates/v3/`. > The user-facing description (Overview, Features, Form Generation > Process) is still accurate. diff --git a/docs/PLUGIN_CONFIG_ARCHITECTURE.md b/docs/PLUGIN_CONFIG_ARCHITECTURE.md index 823bb4db..b34ffc89 100644 --- a/docs/PLUGIN_CONFIG_ARCHITECTURE.md +++ b/docs/PLUGIN_CONFIG_ARCHITECTURE.md @@ -11,427 +11,179 @@ ### Component Overview ``` -┌─────────────────────────────────────────────────────────────────┐ -│ Web Browser │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Tab Navigation Bar │ │ -│ │ [Overview] [General] ... [Plugins] [Plugin X] [Plugin Y]│ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────┐ ┌──────────────────────────────────┐ │ -│ │ Plugins Tab │ │ Plugin X Configuration Tab │ │ -│ │ │ │ │ │ -│ │ • Install │ │ Form Generated from Schema: │ │ -│ │ • Update │ │ • Boolean → Toggle │ │ -│ │ • Uninstall │ │ • Number → Number Input │ │ -│ │ • Enable │ │ • String → Text Input │ │ -│ │ • [Configure]──────→ • Array → Comma Input │ │ -│ │ │ │ • Enum → Dropdown │ │ -│ └─────────────────┘ │ │ │ -│ │ [Save] [Back] [Reset] │ │ -│ └──────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Web browser (templates/v3/base.html, Alpine.js + HTMX) │ +│ │ +│ Second nav row: one tab per installed plugin │ +│ Clicking a tab: GET /v3/partials/plugin-config/ │ +│ → server-rendered form swapped into the tab │ +│ │ +│ Save: hx-post="/api/v3/plugins/config?plugin_id=" (form data) │ +└──────────────────────────────────────────────────────────────────┘ │ - │ HTTP API ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Flask Backend │ -│ ┌───────────────────────────────────────────────────────┐ │ -│ │ /api/v3/plugins/installed │ │ -│ │ • Discover plugins in plugins/ directory │ │ -│ │ • Load manifest.json for each plugin │ │ -│ │ • Load config_schema.json if exists │ │ -│ │ • Load current config from config.json │ │ -│ │ • Return combined data to frontend │ │ -│ └───────────────────────────────────────────────────────┘ │ -│ │ -│ ┌───────────────────────────────────────────────────────┐ │ -│ │ /api/v3/plugins/config │ │ -│ │ • Receive key-value pair │ │ -│ │ • Update config.json │ │ -│ │ • Return success/error │ │ -│ └───────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Flask (web_interface/app.py) │ +│ │ +│ pages_v3 blueprint (blueprints/pages_v3.py) │ +│ _load_plugin_config_partial(plugin_id) │ +│ • SchemaManager.load_schema() → config_schema.json │ +│ • config.json section for the plugin │ +│ • masks x-secret fields │ +│ • renders partials/plugin_config.html (render_field macros) │ +│ │ +│ api_v3 blueprint (blueprints/api_v3/plugins.py) │ +│ save_plugin_config() POST /api/v3/plugins/config │ +│ get_plugin_config() GET /api/v3/plugins/config │ +│ get_plugin_schema() GET /api/v3/plugins/schema │ +│ reset_plugin_config() POST /api/v3/plugins/config/reset │ +└──────────────────────────────────────────────────────────────────┘ │ - │ File System ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ File System │ -│ │ -│ plugins/ │ -│ ├── hello-world/ │ -│ │ ├── manifest.json ───┐ │ -│ │ ├── config_schema.json ─┼─→ Defines UI structure │ -│ │ ├── manager.py │ │ -│ │ └── requirements.txt │ │ -│ └── clock-simple/ │ │ -│ ├── manifest.json │ │ -│ └── config_schema.json ──┘ │ -│ │ -│ config/ │ -│ └── config.json ────────────→ Stores configuration values │ -│ { │ -│ "hello-world": { │ -│ "enabled": true, │ -│ "message": "Hello!", │ -│ ... │ -│ } │ -│ } │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Files │ +│ plugin-repos//config_schema.json JSON Schema (Draft-7) │ +│ config/config.json { "": { ... } } │ +│ config/config_secrets.json { "": { secrets } } │ +└──────────────────────────────────────────────────────────────────┘ ``` +The plugins directory is `plugin_system.plugins_directory` in +`config/config.json` (default `plugin-repos/`). Plugin configuration lives in +`config/config.json`, not in the plugin directory, so it survives reinstalls. + ## Data Flow -### 1. Page Load Sequence +### 1. Rendering a plugin's tab ``` -User Opens Web Interface - │ - ▼ -DOMContentLoaded Event - │ - ▼ -refreshPlugins() - │ - ▼ -GET /api/v3/plugins/installed - │ - ├─→ For each plugin directory: - │ ├─→ Read manifest.json - │ ├─→ Read config_schema.json (if exists) - │ └─→ Read config from config.json - │ - ▼ -Return JSON Array: -[{ - id: "hello-world", - name: "Hello World", - config: { enabled: true, message: "Hello!" }, - config_schema_data: { - properties: { - enabled: { type: "boolean", ... }, - message: { type: "string", ... } - } - } -}, ...] - │ - ▼ -generatePluginTabs(plugins) - │ - ├─→ For each plugin: - │ ├─→ Create tab button - │ ├─→ Create tab content div - │ └─→ generatePluginConfigForm(plugin) - │ │ - │ ├─→ Read schema properties - │ ├─→ Get current config values - │ └─→ Generate HTML form inputs - │ - ▼ -Tabs Rendered in UI +User opens the plugin's tab + │ + ▼ +GET /v3/partials/plugin-config/ (pages_v3) + │ + ├─→ Load schema (SchemaManager, no cache) + ├─→ Load config.json[] + ├─→ Mask "x-secret" values (fails closed if the schema is unusable) + └─→ render partials/plugin_config.html + │ + └─→ render_field() per property, recursively: + boolean → toggle, number/integer → input or slider, + string → input / textarea / select (enum), + array → list or table widget, + object → collapsible nested section, + "x-widget" → a registered widget + (static/v3/js/widgets/, or one the plugin ships) ``` -### 2. Configuration Save Sequence +Nested objects are supported: a nested field is posted with a dotted name +(e.g. `transition.type`). + +### 2. Saving ``` -User Modifies Form - │ - ▼ -User Clicks "Save" - │ - ▼ -savePluginConfiguration(pluginId) - │ - ├─→ Get form data - ├─→ For each field: - │ ├─→ Get schema type - │ ├─→ Convert value to correct type - │ │ • boolean: checkbox.checked - │ │ • integer: parseInt() - │ │ • number: parseFloat() - │ │ • array: split(',') - │ │ • string: as-is - │ │ - │ └─→ POST /api/v3/plugins/config - │ { - │ plugin_id: "hello-world", - │ key: "message", - │ value: "Hello, World!" - │ } - │ - ▼ -Backend Updates config.json - │ - ▼ -Return Success - │ - ▼ -Show Notification - │ - ▼ -Refresh Plugins +User clicks Save + │ + ▼ +validatePluginConfigForm() (client-side checks) + │ + ▼ +POST /api/v3/plugins/config?plugin_id= (form data, all fields of the form) + │ + ▼ +save_plugin_config() (api_v3/plugins.py) + ├─→ Start from the stored config.json[] + ├─→ Apply form fields: dotted names → nested keys, "[]" checkbox + │ groups → lists, values coerced to the schema's types + ├─→ Merge schema defaults for keys that are still missing + ├─→ Validate against the schema (plus core per-plugin properties); + │ invalid → 400 with the validation errors, nothing saved + ├─→ Split "x-secret" fields out; masked/blank secrets are dropped so + │ an untouched secret keeps its stored value + ├─→ Deep-merge regular fields into config.json[] (atomic save) + ├─→ Merge secrets into config_secrets.json[] + └─→ Call the loaded plugin's on_config_change() (and + on_enable/on_disable if "enabled" changed) + │ + ▼ +One response for the whole form → notification in the UI ``` -## Class and Function Hierarchy +The display service picks up the new config through its config hot reload +(ConfigService) without a restart. -### Frontend (JavaScript) +JSON clients can post `{"plugin_id": ..., "config": {...}}` instead; the keys +sent are merged onto the stored config the same way. See +[REST_API_REFERENCE.md](REST_API_REFERENCE.md#save-plugin-configuration). -``` -Window Load - └── DOMContentLoaded - └── refreshPlugins() - ├── fetch('/api/v3/plugins/installed') - ├── renderInstalledPlugins(plugins) - └── generatePluginTabs(plugins) - └── For each plugin: - ├── Create tab button - ├── Create tab content - └── generatePluginConfigForm(plugin) - ├── Read config_schema_data - ├── Read current config - └── Generate form HTML - ├── Boolean → Toggle switch - ├── Number → Number input - ├── String → Text input - ├── Array → Comma-separated input - └── Enum → Select dropdown +### 3. Reset -User Interactions - ├── configurePlugin(pluginId) - │ └── showTab(`plugin-${pluginId}`) - │ - ├── savePluginConfiguration(pluginId) - │ ├── Process form data - │ ├── Convert types per schema - │ └── For each field: - │ └── POST /api/v3/plugins/config - │ - └── resetPluginConfig(pluginId) - ├── Get schema defaults - └── For each field: - └── POST /api/v3/plugins/config -``` - -### Backend (Python) - -``` -Flask Routes - ├── /api/v3/plugins/installed (GET) - │ └── api_plugins_installed() - │ ├── PluginManager.discover_plugins() - │ ├── For each plugin: - │ │ ├── PluginManager.get_plugin_info() - │ │ ├── Load config_schema.json - │ │ └── Load config from config.json - │ └── Return JSON response - │ - └── /api/v3/plugins/config (POST) - └── api_plugin_config() - ├── Parse request JSON - ├── Load current config - ├── Update config[plugin_id][key] = value - └── Save config.json -``` - -## File Structure - -``` -LEDMatrix/ -│ -├── web_interface_v2.py -│ └── Flask backend with plugin API endpoints -│ -├── templates/ -│ └── index_v2.html -│ └── Frontend with dynamic tab generation -│ -├── config/ -│ └── config.json -│ └── Stores all plugin configurations -│ -├── plugins/ -│ ├── hello-world/ -│ │ ├── manifest.json ← Plugin metadata -│ │ ├── config_schema.json ← UI schema definition -│ │ ├── manager.py ← Plugin logic -│ │ └── requirements.txt -│ │ -│ └── clock-simple/ -│ ├── manifest.json -│ ├── config_schema.json -│ └── manager.py -│ -└── docs/ - ├── PLUGIN_CONFIGURATION_TABS.md ← Full documentation - ├── PLUGIN_CONFIG_TABS_SUMMARY.md ← Implementation summary - ├── PLUGIN_CONFIG_QUICK_START.md ← Quick start guide - └── PLUGIN_CONFIG_ARCHITECTURE.md ← This file -``` +`POST /api/v3/plugins/config/reset` replaces the plugin's section with the +schema defaults (keeping secrets unless `preserve_secrets` is false). ## Key Design Decisions -### 1. Dynamic Tab Generation +### 1. Server-side rendered forms -**Why**: Plugins are installed/uninstalled dynamically -**How**: JavaScript creates/removes tab elements on plugin list refresh -**Benefit**: No server-side template rendering needed +**Why**: One renderer for every plugin, no per-plugin frontend code +**How**: Jinja macros in `partials/plugin_config.html` walk the schema +**Benefit**: The settings search index is built from the same rendered HTML +(`/v3/settings/search-index`) -### 2. JSON Schema as Source of Truth +### 2. JSON Schema as source of truth -**Why**: Standard, well-documented, validation-ready -**How**: Frontend interprets schema to generate forms -**Benefit**: Plugin developers use familiar format +**Why**: Standard, well-documented, validation-ready +**How**: The same schema drives the form, the defaults and server-side validation +**Benefit**: Plugin developers use a familiar format -### 3. Individual Config Updates +### 3. Whole-form saves that merge -**Why**: Simplifies backend API -**How**: Each field saved separately via `/api/v3/plugins/config` -**Benefit**: Atomic updates, easier error handling +**Why**: A partial form (or a field the form doesn't show) must not wipe +stored values +**How**: The handler starts from the stored section and merges what was posted +**Benefit**: One request per save, atomic write -### 4. Type Conversion in Frontend +### 4. Secrets kept out of config.json -**Why**: HTML forms only return strings -**How**: JavaScript converts based on schema type before sending -**Benefit**: Backend receives correctly-typed values - -### 5. No Nested Objects - -**Why**: Keeps UI simple -**How**: Only flat property structures supported -**Benefit**: Easy form generation, clear to users +**Why**: `config.json` is shown in the raw editor and returned by the API +**How**: `"x-secret": true` fields go to `config_secrets.json`, which is +deep-merged back into the plugin's config at load time +**Benefit**: Plugins read secrets with plain `config.get(...)` ## Extension Points -### Adding New Input Types +### Custom input widgets -Location: `generatePluginConfigForm()` in `index_v2.html` +Set `"x-widget": ""` on a property. Core widgets are in +`web_interface/static/v3/js/widgets/` (see its README); a plugin can ship its +own widget script, served from `/static/plugin-widgets//.js`. +See [widget-guide.md](widget-guide.md). -```javascript -if (type === 'your-new-type') { - formHTML += ` - - `; -} -``` +### Custom actions -### Custom Validation +Buttons that run plugin scripts are declared in the manifest's +`web_ui_actions`. See [PLUGIN_WEB_UI_ACTIONS.md](PLUGIN_WEB_UI_ACTIONS.md). -Location: `savePluginConfiguration()` in `index_v2.html` +### Reacting to changes -```javascript -// Add validation before sending -if (!validateCustomConstraint(value, propSchema)) { - throw new Error('Validation failed'); -} -``` +Implement `on_config_change(new_config)` in the plugin (see +[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md)). -### Backend Hook +## Where to Look -Location: `api_plugin_config()` in `web_interface_v2.py` - -```python -# Add custom logic before saving -if plugin_id == 'special-plugin': - value = transform_value(value) -``` - -## Performance Considerations - -### Frontend - -- **Tab Generation**: O(n) where n = number of plugins (typically < 20) -- **Form Generation**: O(m) where m = number of config properties (typically < 10) -- **Memory**: Each plugin tab ~5KB HTML -- **Total Impact**: Negligible for typical use cases - -### Backend - -- **Schema Loading**: Cached after first load -- **Config Updates**: Single file write (atomic) -- **API Calls**: One per config field on save (sequential) -- **Optimization**: Could batch updates in single API call - -## Security Considerations - -1. **Input Validation**: Schema constraints enforced client-side (UX) and should be enforced server-side -2. **Path Traversal**: Plugin paths validated against known plugin directory -3. **XSS**: All user inputs escaped before rendering in HTML -4. **CSRF**: Flask CSRF tokens should be used in production -5. **File Permissions**: config.json requires write access +| Concern | File | +|---------|------| +| Tab partial loader | `web_interface/blueprints/pages_v3.py` (`_load_plugin_config_partial`) | +| Form template and field macros | `web_interface/templates/v3/partials/plugin_config.html` | +| Save / get / schema / reset handlers | `web_interface/blueprints/api_v3/plugins.py` | +| Schema loading, defaults, validation | `src/plugin_system/schema_manager.py` | +| Secret masking and splitting | `src/web_interface/secret_helpers.py` | +| Widgets | `web_interface/static/v3/js/widgets/` | ## Error Handling -### Frontend - -- Network errors: Show notification, don't crash -- Schema errors: Graceful fallback to no config tab -- Type errors: Log to console, continue processing other fields - -### Backend - -- Invalid plugin_id: 400 Bad Request -- Schema not found: Return null, frontend handles gracefully -- Config save error: 500 Internal Server Error with message - -## Testing Strategy - -### Unit Tests - -- `generatePluginConfigForm()` for each schema type -- Type conversion logic in `savePluginConfiguration()` -- Backend schema loading logic - -### Integration Tests - -- Full save flow: form → API → config.json -- Tab generation from API response -- Reset to defaults - -### E2E Tests - -- Install plugin → verify tab appears -- Configure plugin → verify config saved -- Uninstall plugin → verify tab removed - -## Monitoring - -### Frontend Metrics - -- Time to generate tabs -- Form submission success rate -- User interactions (configure, save, reset) - -### Backend Metrics - -- API response times -- Config update success rate -- Schema loading errors - -### User Feedback - -- Are users finding the configuration interface? -- Are validation errors clear? -- Are default values sensible? - -## Future Roadmap - -### Phase 2: Enhanced Validation -- Real-time validation feedback -- Custom error messages -- Dependent field validation - -### Phase 3: Advanced Inputs -- Color pickers for RGB arrays -- File upload for assets -- Rich text editor for descriptions - -### Phase 4: Configuration Management -- Export/import configurations -- Configuration presets -- Version history/rollback - -### Phase 5: Developer Tools -- Schema editor in web UI -- Live preview while editing schema -- Validation tester - +- Unknown plugin or unreadable schema: the partial renders an error message +- Validation failure: `400` with `details` and `context.validation_errors`; + the form shows them and nothing is saved +- Save failure: `500` with an error message; config.json is written + atomically, so a failed save leaves the previous file intact diff --git a/docs/PLUGIN_DEPENDENCY_GUIDE.md b/docs/PLUGIN_DEPENDENCY_GUIDE.md index 1388a6fc..5396b0c6 100644 --- a/docs/PLUGIN_DEPENDENCY_GUIDE.md +++ b/docs/PLUGIN_DEPENDENCY_GUIDE.md @@ -2,234 +2,160 @@ ## Overview -The LEDMatrix system has smart dependency installation that adapts based on who is running it. This guide explains how it works and potential pitfalls. +A plugin lists its Python packages in its `requirements.txt`. LEDMatrix +installs them for you when a plugin is installed, updated or loaded. This +guide explains where they end up and what to do when a plugin can't import a +package. -## How It Works +The rule to remember: **packages must be importable by `ledmatrix.service`, +which runs as root.** Anything installed only into another user's +`~/.local/` is invisible to it. -### Execution Context Detection +## Who Runs What -The plugin manager checks if it's running as root: -```python -running_as_root = os.geteuid() == 0 +| Service | Runs as | Set by | +|---------|---------|--------| +| `ledmatrix.service` (display) | `root` | `systemd/ledmatrix.service` | +| `ledmatrix-web.service` (web UI) | the user who ran the installer (e.g. `ledpi`) | `User=__USER__` in `systemd/ledmatrix-web.service`, filled in by `scripts/install/install_service.sh` | + +## How Dependencies Get Installed + +### 1. Installing or updating a plugin from the web UI + +The web interface is not root, so it installs through a narrow sudo helper: + +1. `PluginStoreManager._install_dependencies()` + (`src/plugin_system/store_manager.py`) calls + `install_requirements_file()` (`src/common/permission_utils.py`). +2. That runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh /requirements.txt`. + The helper checks the path is the project's own `requirements.txt` or a + `requirements.txt` under `plugin-repos/` or `plugins/`, then runs + `python3 -m pip install --break-system-packages --ignore-installed -r ...` + **as root**, so the display service can import the packages. +3. The sudoers rule that allows this is written by the installer + (`first_time_install.sh`) or by `scripts/install/configure_web_sudo.sh`. + +If sudo refuses (the rule isn't installed), `install_requirements_file()` +falls back to installing with the web process's own interpreter, as the web +user, and prefixes the pip output with a note like: + +``` +[Root install unavailable (...); installed for the current process's user only. +Packages may not be visible to ledmatrix.service if it runs as a different +user — run scripts/install/configure_web_sudo.sh to fix this.] ``` -Based on this, it chooses the appropriate installation method: +Fix it by running `./scripts/install/configure_web_sudo.sh` as the web +user (not with `sudo`; it asks for your password itself), then +reinstall the plugin (or use the manual install below). -| Running As | Installation Method | Location | Accessible To | -|------------|-------------------|----------|---------------| -| **root** (systemd service) | System-wide (`--break-system-packages`) | `/usr/local/lib/python3.X/dist-packages/` | All users | -| **ledpi** or other user | User-specific (`--user`) | `~/.local/lib/python3.X/site-packages/` | Only that user | +The **Reinstall Plugin Deps** button on the web UI's Tools tab goes +through the same helper for every installed plugin. + +### 2. Loading a plugin + +When a plugin loads, `PluginLoader.install_dependencies()` +(`src/plugin_system/plugin_loader.py`) checks its `requirements.txt`. If the +requirements are already satisfied it does nothing; otherwise it runs +`python3 -m pip install --break-system-packages -r requirements.txt` with the +interpreter of the process doing the loading (retrying with +`--ignore-installed` when a system package without a pip RECORD file is in +the way). + +In `ledmatrix.service` that process is root, so restarting the display +service installs anything missing system-wide: + +```bash +sudo systemctl restart ledmatrix +``` + +If you run `python3 run.py` by hand as a normal user instead, pip cannot +write to the system site-packages and installs into your `~/.local/`. That +works for your manual run but not for the service. ## Common Scenarios -### ✅ Scenario 1: Normal Production Use (Recommended) +### Installing plugins from the web UI (recommended) -**What:** Services running via systemd +Use the **Plugin Manager** tab. Dependencies are installed as root through +the sudo helper and the display service can use them. + +### Running the display manually for debugging ```bash -sudo systemctl start ledmatrix -sudo systemctl start ledmatrix-web +cd ~/LEDMatrix +sudo python3 run.py # same user as the service ``` -- **Runs as:** root (configured in .service files) -- **Installs to:** System-wide -- **Result:** ✅ Works perfectly, all dependencies accessible +Running as your own user works for plugins whose packages are already +installed system-wide, but any *missing* package lands in `~/.local/`. -### ✅ Scenario 2: Web Interface Plugin Installation +### A plugin works when run manually but fails in the service -**What:** Installing/enabling plugins via web interface at `http://pi-ip:5000` +Its packages were installed for your user only. Install them as root (see +below) and restart the service. -- **Web service runs as:** root (ledmatrix-web.service) -- **Installs to:** System-wide -- **Result:** ✅ Works perfectly, systemd service can access them +## Manual Installation -### ✅ Scenario 3: Manual Testing as ledpi (Read-only) - -**What:** Running display manually as ledpi to test/debug +### All plugins ```bash -# As ledpi user -cd /home/ledpi/LEDMatrix -python3 run.py -``` - -- **Runs as:** ledpi -- **Can import:** ✅ System-wide packages (installed by root) -- **Result:** ✅ Works! Can use existing plugins with root-installed dependencies - -### ⚠️ Scenario 4: Manual Plugin Installation as ledpi (Problematic) - -**What:** Enabling a NEW plugin and running manually as ledpi - -```bash -# As ledpi user -cd /home/ledpi/LEDMatrix -# Edit config to enable new plugin -nano config/config.json -# Run display - will try to install new plugin dependencies -python3 run.py -``` - -**What Happens:** -1. Plugin manager runs as `ledpi` -2. Installs dependencies with `--user` flag -3. Dependencies go to `~/.local/lib/python3.X/site-packages/` -4. ⚠️ **Warning logged:** "Installing plugin dependencies for current user (not root)" - -**Problem:** -- When systemd service restarts (as root), it **can't see** `~/.local/` packages -- Plugin will fail to load for the systemd service - -**Solution:** -After testing, restart the service to install dependencies system-wide: -```bash +sudo ~/LEDMatrix/scripts/install_plugin_dependencies.sh sudo systemctl restart ledmatrix ``` -## Best Practices +The script installs every `requirements.txt` found in the plugins directory +configured by `plugin_system.plugins_directory` in `config/config.json` +(default `plugin-repos/`). Run it with `sudo` so the packages are installed +system-wide. -### For Production/Normal Use +### One plugin -1. **Always use the web interface** to install/enable plugins -2. **Or restart the systemd service** after config changes: - ```bash - sudo systemctl restart ledmatrix - ``` - -### For Development/Testing - -1. **Read existing plugins:** Safe to run as `ledpi` - can import system packages -2. **Test new plugins:** Use sudo or restart service to install dependencies: - ```bash - # Option 1: Run as root - sudo python3 run.py - - # Option 2: Install deps manually - sudo pip3 install --break-system-packages -r plugins/my-plugin/requirements.txt - python3 run.py - - # Option 3: Let service install them - sudo systemctl restart ledmatrix - ``` - -## Warning Messages - -### If you see this warning: -``` -Installing plugin dependencies for current user (not root). -These will NOT be accessible to the systemd service. -For production use, install plugins via the web interface or restart the ledmatrix service. -``` - -**What it means:** -- You're running as a non-root user -- Dependencies were installed to your user directory only -- The systemd service won't be able to use this plugin - -**What to do:** ```bash -# Restart the service to install dependencies system-wide +cd ~/LEDMatrix/plugin-repos/PLUGIN-NAME # or your configured plugins directory +sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt sudo systemctl restart ledmatrix ``` +`--no-cache-dir` avoids errors about `/root/.cache/pip` not being writable. + ## Troubleshooting -### Plugin works when I run manually but fails in systemd service - -**Cause:** Dependencies installed to user directory (`~/.local/`) instead of system-wide - -**Fix:** -```bash -# Check where package is installed -pip3 list -v | grep - -# If it shows ~/.local/, reinstall system-wide: -sudo pip3 install --break-system-packages - -# Or just restart the service: -sudo systemctl restart ledmatrix -``` - ### Permission denied when installing dependencies -**If you see errors like:** ``` ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/root/.local' WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable ``` -**Quick Fix - Use the Helper Script:** -```bash -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh -sudo systemctl restart ledmatrix -``` +Use one of the manual installs above (they pass `--no-cache-dir`). -**Manual Fix:** -```bash -# Install dependencies with --no-cache-dir to avoid cache permission issues -cd /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME -sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt -sudo systemctl restart ledmatrix -``` - -**For more detailed troubleshooting, see:** [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md) - -## Architecture Summary - -``` -┌─────────────────────────────────────────────────────────────┐ -│ LEDMatrix Services │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ ledmatrix.service (User=root) │ -│ ledmatrix-web.service (User=root) │ -│ ├── Install dependencies system-wide │ -│ └── Accessible to all users │ -│ │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ Manual execution as ledpi │ -│ ├── Can READ system-wide packages ✅ │ -│ ├── WRITES go to ~/.local/ ⚠️ │ -│ └── Not accessible to root service │ -│ │ -└─────────────────────────────────────────────────────────────┘ -``` - -## Recommendations - -1. **For end users:** Always use the web interface for plugin management -2. **For developers:** Be aware of the user context when testing -3. **For plugin authors:** Test with `sudo systemctl restart ledmatrix` to ensure dependencies install correctly -4. **For CI/CD:** Always run installation as root or use the service - -## Helper Scripts - -### Install Plugin Dependencies Script - -Located at: `scripts/install_plugin_dependencies.sh` - -This script automatically finds and installs dependencies for all plugins: +### Checking where a package is installed ```bash -# Run as root (recommended for production) -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +# How the service sees it +sudo python3 -c "import package_name; print(package_name.__file__)" -# Make executable if needed -chmod +x /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +# A path under /home//.local/ means it was installed for that user only +python3 -m pip show -f package_name ``` -Features: -- Auto-detects all plugins with requirements.txt -- Uses correct installation method (system-wide vs user) -- Bypasses pip cache to avoid permission issues -- Provides detailed logging and error messages +For more, see the [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md). + +## For Plugin Authors + +1. Keep `requirements.txt` minimal and pin only what you need. +2. Test that it installs the way the Pi will install it: + ```bash + sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt + ``` +3. Note any `apt` packages your plugin needs in its README. ## Files to Reference -- Service configs: `ledmatrix.service`, `ledmatrix-web.service` -- Plugin manager: `src/plugin_system/plugin_manager.py` -- Installation script: `first_time_install.sh` -- Dependency installer: `scripts/install_plugin_dependencies.sh` -- Troubleshooting guide: `PLUGIN_DEPENDENCY_TROUBLESHOOTING.md` - +- Service units: `systemd/ledmatrix.service`, `systemd/ledmatrix-web.service` +- Store installs: `src/plugin_system/store_manager.py` (`_install_dependencies`) +- Root install helper: `src/common/permission_utils.py` (`install_requirements_file`), `scripts/fix_perms/safe_pip_install.sh` +- Load-time installs: `src/plugin_system/plugin_loader.py` (`install_dependencies`) +- Sudo rules: `scripts/install/configure_web_sudo.sh` +- Manual installer: `scripts/install_plugin_dependencies.sh` diff --git a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md index 1f33cb88..a62b9644 100644 --- a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md +++ b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md @@ -1,6 +1,7 @@ # Plugin Dependency Installation Troubleshooting -This guide helps resolve issues with automatic plugin dependency installation in the LEDMatrix system. +This guide helps resolve problems installing a plugin's Python packages. For +how installation works, see the [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md). ## Common Error Symptoms @@ -10,109 +11,118 @@ ERROR: Could not install packages due to an OSError: [Errno 13] Permission denie WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable ``` -### Context Mismatch +### Installed for the wrong user +The pip output shown after a web-UI install starts with: ``` -WARNING: Installing plugin dependencies for current user (not root). -These will NOT be accessible to the systemd service. +[Root install unavailable (...); installed for the current process's user only. +Packages may not be visible to ledmatrix.service if it runs as a different +user — run scripts/install/configure_web_sudo.sh to fix this.] ``` +### Plugin fails to load with `ModuleNotFoundError` +The display service can't see a package the plugin needs. + ## Root Cause -Plugin dependencies must be installed in a context accessible to the LEDMatrix systemd service, which runs as root. Permission errors typically occur when: +Plugin packages must be importable by `ledmatrix.service`, which runs as +root. The web interface (`ledmatrix-web.service`) runs as the user who +installed LEDMatrix, so it installs through a sudo helper +(`scripts/fix_perms/safe_pip_install.sh`). Problems usually come from: -1. The pip cache directory has incorrect permissions -2. The process tries to install to user directories without proper permissions -3. Environment variables (like HOME) are not set correctly for the service context +1. The sudoers rule for that helper missing, so the web UI installed the + packages for its own user only +2. Running `python3 run.py` by hand as a normal user, which installs missing + packages into `~/.local/` +3. pip's cache directory not being writable for root ## Solutions -### Solution 1: Use the Manual Installation Script (Recommended) - -We provide a helper script that handles dependency installation correctly: +### Solution 1: Restore the sudo rule, then reinstall ```bash -# Run as root to install system-wide (for production) -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +cd ~/LEDMatrix +./scripts/install/configure_web_sudo.sh # as the web user, not with sudo +``` -# After installation, restart the service +Then reinstall the plugin from the **Plugin Manager** tab, or click +**Reinstall Plugin Deps** on the **Tools** tab. + +### Solution 2: Install every plugin's dependencies from the terminal + +```bash +sudo ~/LEDMatrix/scripts/install_plugin_dependencies.sh sudo systemctl restart ledmatrix ``` -This script: -- Detects all plugins with requirements.txt files -- Installs dependencies with correct permissions -- Uses `--no-cache-dir` to avoid cache permission issues -- Provides detailed logging for troubleshooting +The script finds each `requirements.txt` in the plugins directory set by +`plugin_system.plugins_directory` in `config/config.json` (default +`plugin-repos/`), installs with `--no-cache-dir`, and reports what it found. -### Solution 2: Manual Installation per Plugin - -If you need to install dependencies for a specific plugin: +### Solution 3: Install one plugin's dependencies ```bash -# Navigate to the plugin directory -cd /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME +# Your configured plugins directory; plugin-repos/ by default +cd ~/LEDMatrix/plugin-repos/PLUGIN-NAME -# Install as root (system-wide) -sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt +sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt -# Restart the service sudo systemctl restart ledmatrix ``` -### Solution 3: Fix Cache Directory Permissions +### Solution 4: Let the display service install them -If you specifically have cache permission issues: +When a plugin loads, the display service installs any missing requirements +itself, as root: + +```bash +sudo systemctl restart ledmatrix +sudo journalctl -u ledmatrix -f # watch for "Installing dependencies for plugin ..." +``` + +### Solution 5: Fix pip cache permissions ```bash # Option A: Skip the cache (recommended) -sudo pip3 install --no-cache-dir --break-system-packages -r requirements.txt +sudo python3 -m pip install --no-cache-dir --break-system-packages -r requirements.txt -# Option B: Fix cache permissions (if needed) +# Option B: Fix cache permissions sudo mkdir -p /root/.cache/pip sudo chown -R root:root /root/.cache sudo chmod -R 755 /root/.cache ``` -### Solution 4: Install via Web Interface - -The web interface handles dependency installation correctly in the service context: - -1. Access the web interface (`http://ledpi:5000` or `http://your-pi-ip:5000`) -2. Open the **Plugin Manager** tab (use the **Plugin Store** section to - find the plugin, or **Install from GitHub**) -3. Install the plugin through the web UI -4. The system automatically handles dependency installation in the - service context (which has the right permissions) - ## Prevention ### For Plugin Developers -When creating plugins with dependencies: - 1. **Keep requirements minimal**: Only include essential packages -2. **Test installation**: Verify your requirements.txt works with: +2. **Test installation** the way the Pi does it: ```bash - sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt + sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt ``` 3. **Document dependencies**: Note any system packages needed (via apt) ### For Users -1. **Use web interface**: Install plugins via the web UI when possible -2. **Install as root**: When using SSH/terminal, use sudo for plugin installations -3. **Restart service**: After manual installations, restart the ledmatrix service +1. **Use the web interface** to install plugins +2. **Use sudo** for installs from SSH/terminal +3. **Restart the service** after manual installations ## Technical Details -### How Dependency Installation Works +### Where installs happen -The `PluginManager._install_plugin_dependencies()` method: - -1. Detects if running as root using `os.geteuid() == 0` -2. If root: Uses system-wide installation with `--break-system-packages --no-cache-dir` -3. If not root: Uses user installation with `--user --break-system-packages --no-cache-dir` -4. The `--no-cache-dir` flag prevents cache-related permission issues +- **Web UI install/update:** `PluginStoreManager._install_dependencies()` + → `install_requirements_file()` in `src/common/permission_utils.py`, which + runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh `. + The helper only accepts the project's `requirements.txt` or one under + `plugin-repos/` or `plugins/`, and runs + `pip install --break-system-packages --ignore-installed` as root. If sudo + refuses, it falls back to a pip install as the web user and says so. +- **Plugin load:** `PluginLoader.install_dependencies()` in + `src/plugin_system/plugin_loader.py` skips satisfied requirements and + otherwise runs `pip install --break-system-packages` with the loading + process's interpreter — root in `ledmatrix.service`. ### Why `--break-system-packages`? @@ -120,26 +130,20 @@ Debian 12+ (Bookworm) and Raspberry Pi OS based on it implement PEP 668, which p ### Service Context -The ledmatrix.service runs as: -- **User**: root -- **WorkingDirectory**: /home/ledpi/LEDMatrix -- **Python**: /usr/bin/python3 +- `ledmatrix.service` runs as **root** with `/usr/bin/python3` +- `ledmatrix-web.service` runs as **the installing user** -Dependencies must be installed in root's Python environment or system-wide to be accessible. +Dependencies must be installed system-wide (as root) to be visible to the +display service. ## Checking Installation -Verify dependencies are installed correctly: - ```bash # Check as root (how the service sees it) -sudo python3 -c "import package_name" +sudo python3 -c "import package_name; print(package_name.__file__)" -# List installed packages -pip3 list - -# Check specific package -pip3 show package_name +# A path under /home//.local/ means a user-only install +python3 -m pip show -f package_name ``` ## Getting Help @@ -151,19 +155,11 @@ If you continue to experience issues: sudo journalctl -u ledmatrix -f ``` -2. Check pip logs (created by manual script): +2. Verify the plugin manifest and requirements (default plugins directory + shown): ```bash - cat /tmp/pip_install_*.log - ``` - -3. Verify plugin manifest is correct: - ```bash - cat /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME/manifest.json - ``` - -4. Check plugin requirements: - ```bash - cat /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME/requirements.txt + cat ~/LEDMatrix/plugin-repos/PLUGIN-NAME/manifest.json + cat ~/LEDMatrix/plugin-repos/PLUGIN-NAME/requirements.txt ``` ## Related Documentation @@ -171,4 +167,3 @@ If you continue to experience issues: - [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md) - [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - [Troubleshooting](TROUBLESHOOTING.md) - diff --git a/docs/PLUGIN_DEVELOPMENT_GUIDE.md b/docs/PLUGIN_DEVELOPMENT_GUIDE.md index a9db48c4..491bd8ad 100644 --- a/docs/PLUGIN_DEVELOPMENT_GUIDE.md +++ b/docs/PLUGIN_DEVELOPMENT_GUIDE.md @@ -3,8 +3,10 @@ This guide explains how to set up a development workflow for plugins that are maintained in separate Git repositories while still being able to test them within the LEDMatrix project. > **Rendering guidance:** plugins should read the display size dynamically -> (`self.display_manager.matrix.width/height`) rather than hardcoding one -> panel. For plugins that want to *scale* their layout to any panel, the +> (`self.display_manager.width/height`) rather than hardcoding one +> panel. Don't read `display_manager.matrix.width/height`: `matrix` is +> `None` when hardware init fails, while the `width`/`height` properties +> fall back to the canvas size. For plugins that want to *scale* their layout to any panel, the > opt-in adaptive layout system ([ADAPTIVE_LAYOUT.md](ADAPTIVE_LAYOUT.md)) > provides the shared helpers — fonts, images, and composite layouts that > scale. Existing plugins keep their classic rendering unless they adopt @@ -43,28 +45,45 @@ The solution uses **symbolic links** to connect plugin repositories to the `plug ## Quick Start -### 1. Link a Plugin from GitHub +Official plugins all live in one repository, +[ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins), with +one directory per plugin under `plugins/` (there are no per-plugin +`ledmatrix-` repositories). The helper script links a plugin directory +from a checkout of that monorepo into LEDMatrix's `plugins/` directory. -The easiest way to link a plugin that's already on GitHub: +### 1. Link an Official Plugin ```bash -./scripts/dev/dev_plugin_setup.sh link-github music +./scripts/dev/dev_plugin_setup.sh link-github football-scoreboard ``` This will: -- Clone `https://github.com/ChuckBuilds/ledmatrix-music.git` to `~/.ledmatrix-dev-plugins/ledmatrix-music` -- Create a symbolic link from `plugins/music` to the cloned repository -- Validate that the plugin has a proper `manifest.json` +- Clone `https://github.com/ChuckBuilds/ledmatrix-plugins.git` to + `~/.ledmatrix-dev-plugins/ledmatrix-plugins` (or `git pull` it if it is + already there) +- Find `plugins/football-scoreboard` in it (also accepted: + `plugins/ledmatrix-`, or a plugin whose manifest `id` is the name) +- Validate that it has a `manifest.json` +- Create a symbolic link named after the plugin's manifest id, e.g. + `plugins/football-scoreboard` → `~/.ledmatrix-dev-plugins/ledmatrix-plugins/plugins/football-scoreboard` -### 2. Link a Local Plugin Repository +`link-github music` finds the monorepo's `plugins/ledmatrix-music` directory +and links it into LEDMatrix as `plugins/ledmatrix-music`, because +`ledmatrix-music` is that plugin's manifest id. -If you already have a plugin repository cloned locally: +To work from your fork of the monorepo, set `github_user` in +`dev_plugins.json` (see [Configuration](#configuration)). + +### 2. Link a Local Plugin Directory + +If you already have the monorepo (or a third-party plugin repository) cloned +locally: ```bash -./scripts/dev/dev_plugin_setup.sh link music ../ledmatrix-music +./scripts/dev/dev_plugin_setup.sh link hello-world ../ledmatrix-plugins/plugins/hello-world ``` -This creates a symlink from `plugins/music` to your local repository path. +This creates a symlink from `plugins/hello-world` to that directory. ### 3. Check Status @@ -77,13 +96,17 @@ See which plugins are linked and their git status: ### 4. Work on Your Plugin ```bash -cd plugins/music # Actually editing the linked repository -# Make your changes +cd plugins/football-scoreboard # Actually editing the monorepo checkout +# Make your changes, then bump "version" in manifest.json git add . -git commit -m "feat: add new feature" -git push origin main +git commit -m "feat(football-scoreboard): add new feature" +git push # to your fork, then open a PR against ledmatrix-plugins ``` +In the monorepo, every plugin change must bump `version` in the plugin's +`manifest.json` and run `python update_registry.py`, or users won't receive +the update. + ### 5. Update Plugins Pull latest changes from remote: @@ -116,7 +139,7 @@ Links a local plugin repository to the plugins directory. **Example:** ```bash -./scripts/dev/dev_plugin_setup.sh link football-scoreboard ../ledmatrix-football-scoreboard +./scripts/dev/dev_plugin_setup.sh link football-scoreboard ../ledmatrix-plugins/plugins/football-scoreboard ``` **Notes:** @@ -129,23 +152,25 @@ Links a local plugin repository to the plugins directory. Clones a plugin from GitHub and links it. **Arguments:** -- `plugin-name`: The name of the plugin (will be the directory name in `plugins/`) -- `repo-url`: (Optional) Full GitHub repository URL. If omitted, constructs from pattern: `https://github.com/ChuckBuilds/ledmatrix-.git` +- `plugin-name`: Without `repo-url`, the plugin to link from the monorepo: a + directory under `plugins/` (`` or `ledmatrix-`) or a manifest + id. The link is named after the plugin's manifest id. With `repo-url`, the + name of the link in `plugins/`. +- `repo-url`: (Optional) A plugin that has its own repository (e.g. a + third-party plugin). The repository root is linked. **Examples:** ```bash -# Auto-construct URL from plugin name -./scripts/dev/dev_plugin_setup.sh link-github music +# Official plugin, from the ledmatrix-plugins monorepo +./scripts/dev/dev_plugin_setup.sh link-github stocks -# Use explicit URL -./scripts/dev/dev_plugin_setup.sh link-github stocks https://github.com/ChuckBuilds/ledmatrix-stocks.git - -# Link from a different GitHub user +# Third-party plugin with its own repository ./scripts/dev/dev_plugin_setup.sh link-github custom-plugin https://github.com/OtherUser/custom-plugin.git ``` **Notes:** - Repositories are cloned to `~/.ledmatrix-dev-plugins/` by default (configurable) +- The monorepo is cloned once and shared by every plugin you link from it - If the repository already exists, it will be updated with `git pull` instead of re-cloning - The cloned repository is preserved when you unlink the plugin @@ -217,30 +242,28 @@ Updates plugin(s) by running `git pull` in their repositories. ### Custom Development Directory -By default, GitHub repositories are cloned to `~/.ledmatrix-dev-plugins/`. You can customize this by creating a `dev_plugins.json` file: +By default, GitHub repositories are cloned to `~/.ledmatrix-dev-plugins/` +and official plugins come from `ChuckBuilds/ledmatrix-plugins`. To change +either, copy `dev_plugins.json.example` (in the LEDMatrix root) to +`dev_plugins.json` and edit it. `dev_plugins.json` is git-ignored. ```json { - "dev_plugins_dir": "/path/to/your/dev/plugins", - "github_user": "ChuckBuilds", - "github_pattern": "ledmatrix-", - "plugins": { - "music": { - "source": "github", - "url": "https://github.com/ChuckBuilds/ledmatrix-music.git", - "branch": "main" - } - } + "dev_plugins_dir": "~/.ledmatrix-dev-plugins", + "github_user": "your-github-user", + "plugins_repo": "ledmatrix-plugins", + "plugins_branch": "main" } ``` -**Configuration options:** +**Configuration options** (all optional): - `dev_plugins_dir`: Where to clone GitHub repositories (default: `~/.ledmatrix-dev-plugins`) -- `github_user`: Default GitHub username for auto-constructing URLs -- `github_pattern`: Pattern for repository names (default: `ledmatrix-`) -- `plugins`: Plugin definitions (optional, for future auto-discovery features) +- `github_user`: Owner of the plugin monorepo that `link-github ` clones — set it to use your fork (default: `ChuckBuilds`) +- `plugins_repo`: Name of that monorepo (default: `ledmatrix-plugins`) +- `plugins_branch`: Branch to clone it at (default: the repository's default branch). Only applies when the clone is first made. -**Note:** Copy `dev_plugins.json.example` to `dev_plugins.json` and customize it. The `dev_plugins.json` file is git-ignored. +`github_pattern` from older versions of this guide is no longer used (the +script warns if it is set). ## Development Workflow @@ -248,43 +271,46 @@ By default, GitHub repositories are cloned to `~/.ledmatrix-dev-plugins/`. You c 1. **Link your plugin for development:** ```bash - ./scripts/dev/dev_plugin_setup.sh link-github music + ./scripts/dev/dev_plugin_setup.sh link-github clock-simple ``` 2. **Test in LEDMatrix:** ```bash - # Run LEDMatrix with your plugin - python run.py + # Run LEDMatrix with your plugin (emulator shown) + python3 run.py -e ``` 3. **Make changes:** ```bash - cd plugins/music + cd plugins/clock-simple # Edit files... # Test changes... ``` -4. **Commit to plugin repository:** +4. **Commit to the plugin repository:** ```bash - cd plugins/music # This is actually your repo + cd plugins/clock-simple # This is inside your monorepo checkout + # bump "version" in manifest.json, then from the monorepo root: + # python update_registry.py git add . - git commit -m "feat: add new feature" - git push origin main + git commit -m "feat(clock-simple): add new feature" + git push ``` 5. **Update from remote (if needed):** ```bash - ./scripts/dev/dev_plugin_setup.sh update music + ./scripts/dev/dev_plugin_setup.sh update clock-simple ``` 6. **When done developing:** ```bash - ./scripts/dev/dev_plugin_setup.sh unlink music + ./scripts/dev/dev_plugin_setup.sh unlink clock-simple ``` ### Working with Multiple Plugins -You can have multiple plugins linked simultaneously: +You can have multiple plugins linked simultaneously. Plugins linked from the +monorepo share one checkout: ```bash ./scripts/dev/dev_plugin_setup.sh link-github music @@ -294,7 +320,7 @@ You can have multiple plugins linked simultaneously: # Check status of all ./scripts/dev/dev_plugin_setup.sh status -# Update all at once +# Update all at once (the shared monorepo checkout is pulled once) ./scripts/dev/dev_plugin_setup.sh update ``` @@ -409,7 +435,7 @@ If you have conflicts when updating: 1. **Manually resolve in the plugin repository:** ```bash - cd ~/.ledmatrix-dev-plugins/ledmatrix-music + cd ~/.ledmatrix-dev-plugins/ledmatrix-plugins git pull # Resolve conflicts... git add . @@ -470,18 +496,19 @@ You can mix local and GitHub plugins: The development workflow is separate from the plugin store installation: -- **Plugin Store:** Installs plugins to `plugins/` as regular directories -- **Development Setup:** Links plugin repositories as symlinks +- **Plugin Store:** Installs plugins as regular directories in the configured + plugins directory (`plugin-repos/` by default) +- **Development Setup:** Links plugin directories as symlinks in `plugins/` -If you install a plugin via the store, you can still link it for development: +The plugin loader scans only one directory, so while developing set +`plugin_system.plugins_directory` to `plugins` (see the note at the top of +this guide). If `plugins/` already holds a regular directory of the same +name, `link`/`link-github` offers to rename it to +`.backup.` before linking. -```bash -# Store installs to plugins/music (regular directory) -# Link for development (will prompt to replace) -./scripts/dev/dev_plugin_setup.sh link-github music -``` - -When you unlink, the directory is removed. If you want to switch back to the store version, re-install it via the plugin store. +`unlink` removes only the symlink. To switch back to the store version, set +`plugins_directory` back to `plugin-repos` (or reinstall the plugin from the +store). ## API Reference @@ -525,7 +552,7 @@ Want to create and share your own plugin? Here's everything you need to know. - [Advanced Plugin Development](ADVANCED_PLUGIN_DEVELOPMENT.md) - Patterns and examples 2. **Start with a template**: - - Use the [Hello World plugin](https://github.com/ChuckBuilds/ledmatrix-hello-world) as a starting point + - Use the [Hello World plugin](https://github.com/ChuckBuilds/ledmatrix-plugins/tree/main/plugins/hello-world) as a starting point - Or fork an existing plugin and modify it 3. **Follow the plugin structure**: @@ -589,24 +616,16 @@ Your plugin must: ### Versioning Best Practices - **Use semantic versioning**: `MAJOR.MINOR.PATCH` (e.g., `1.2.3`) -- **GitHub as source of truth**: the plugin store resolves versions in this - order: GitHub Releases → GitHub Tags → manifest from branch → git commit hash -- **Automatic version bumping**: install the self-contained pre-push hook in - your plugin repo and patch versions bump themselves on push (a git tag - `v{version}` is created and `manifest.json` staged automatically): - - ```bash - # From your plugin repository directory - cp /path/to/LEDMatrix/scripts/git-hooks/pre-push-plugin-version .git/hooks/pre-push - chmod +x .git/hooks/pre-push - ``` - - Set `SKIP_TAG=1` in the environment to skip auto-tagging for one push. -- **Manual versioning**: only needed for major/minor bumps, CI pipelines that - bypass hooks, or forks without the hook — use - `scripts/bump_plugin_version.py`. -- **Registry stores no versions**: `plugins.json` holds only metadata (name, - description, repo URL). +- **Bump `version` in `manifest.json` by hand** for every change you ship. + There is no automatic version-bump hook or bump script. +- **Official (monorepo) plugins**: after bumping the manifest, run + `python update_registry.py` in the `ledmatrix-plugins` checkout. It copies + each manifest's version into `plugins.json` as `latest_version`, which is + what the store compares installed versions against. Without it, users + won't be offered the update. +- **Plugins in their own repository**: still bump the manifest `version`, + so users can see which version they run; tagging releases (`v1.2.3`) to + match is a good habit. ### Submitting to Official Registry @@ -618,12 +637,14 @@ To have your plugin added to the official plugin store: - Follows best practices - Tested on Raspberry Pi hardware -2. **Create GitHub repository**: - - Repository name: `ledmatrix-` - - Public repository - - Proper README.md with installation instructions +2. **Choose where it lives** (see `SUBMISSION.md` in + [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins)): + - **In the monorepo (preferred):** fork ledmatrix-plugins, add + `plugins//`, and open a pull request + - **In your own public repository** (conventionally + `ledmatrix-`), with a README that covers installation -3. **Contact maintainers**: +3. **Contact maintainers** (own-repository plugins): - Open a GitHub issue in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) repository - Or reach out on Discord: https://discord.gg/uW36dVAtcT - Include: Repository URL, plugin description, why it's useful diff --git a/docs/PLUGIN_QUICK_REFERENCE.md b/docs/PLUGIN_QUICK_REFERENCE.md index 850fbb22..09c54b76 100644 --- a/docs/PLUGIN_QUICK_REFERENCE.md +++ b/docs/PLUGIN_QUICK_REFERENCE.md @@ -127,7 +127,8 @@ git push origin v1.0.0 ### REST API -The API is mounted at `/api/v3` (`web_interface/app.py:199`). +The API is mounted at `/api/v3` (the `api_v3` blueprint in +`web_interface/blueprints/api_v3/`, registered in `web_interface/app.py`). ```bash # Install plugin from the registry diff --git a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md index d1426979..87457c14 100644 --- a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md +++ b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md @@ -103,6 +103,7 @@ All plugins can be installed through the LEDMatrix web interface: Or via API: ```bash curl -X POST http://your-pi-ip:5000/api/v3/plugins/install \ + -H "Content-Type: application/json" \ -d '{"plugin_id": "clock-simple"}' ``` @@ -153,6 +154,7 @@ Before submitting, ensure your plugin: ```bash # Install via URL on your Pi curl -X POST http://your-pi:5000/api/v3/plugins/install-from-url \ + -H "Content-Type: application/json" \ -d '{"repo_url": "https://github.com/you/ledmatrix-your-plugin"}' ``` @@ -312,6 +314,7 @@ git push # 2. Review using VERIFICATION.md checklist # 3. Test installation: curl -X POST http://pi:5000/api/v3/plugins/install-from-url \ + -H "Content-Type: application/json" \ -d '{"repo_url": "https://github.com/contributor/plugin"}' # 4. If approved, merge PR diff --git a/docs/PLUGIN_STORE_GUIDE.md b/docs/PLUGIN_STORE_GUIDE.md index 421c99de..fbec2a36 100644 --- a/docs/PLUGIN_STORE_GUIDE.md +++ b/docs/PLUGIN_STORE_GUIDE.md @@ -131,13 +131,13 @@ else: **Via REST API:** ```bash # Search by query -curl "http://your-pi-ip:5000/api/v3/plugins/store/search?q=hockey" +curl "http://your-pi-ip:5000/api/v3/plugins/store/list?query=hockey" # Filter by category -curl "http://your-pi-ip:5000/api/v3/plugins/store/search?category=sports" +curl "http://your-pi-ip:5000/api/v3/plugins/store/list?category=sports" # Filter by tags -curl "http://your-pi-ip:5000/api/v3/plugins/store/search?tags=nhl&tags=hockey" +curl "http://your-pi-ip:5000/api/v3/plugins/store/list?tags=nhl&tags=hockey" ``` **Via Python:** @@ -351,8 +351,7 @@ All API endpoints return JSON with this structure: | Method | Endpoint | Description | |--------|----------|-------------| -| GET | `/api/v3/plugins/store/list` | List all plugins in store | -| GET | `/api/v3/plugins/store/search` | Search for plugins | +| GET | `/api/v3/plugins/store/list` | List plugins in store; `?query=`, `?category=`, `?tags=` search and filter | | GET | `/api/v3/plugins/installed` | List installed plugins | | POST | `/api/v3/plugins/install` | Install from registry | | POST | `/api/v3/plugins/install-from-url` | Install from GitHub URL | diff --git a/docs/REST_API_REFERENCE.md b/docs/REST_API_REFERENCE.md index 2f312151..4f36386a 100644 --- a/docs/REST_API_REFERENCE.md +++ b/docs/REST_API_REFERENCE.md @@ -1,10 +1,10 @@ # LEDMatrix REST API Reference -Complete reference for all REST API endpoints available in the LEDMatrix web interface. +Reference for the REST API served by the LEDMatrix web interface. **Base URL**: `http://your-pi-ip:5000/api/v3` -All endpoints return JSON responses with a standard format: +Most endpoints answer JSON in this envelope: ```json { "status": "success" | "error", @@ -13,6 +13,11 @@ All endpoints return JSON responses with a standard format: } ``` +Not every endpoint follows it exactly. Where a response puts fields at the +top level instead of under `data` (install-from-url, registry-from-url, the +auth endpoints, upload endpoints, `system/git-info`, `system/check-update`), +the entry below says so. + ## Table of Contents - [Configuration](#configuration) @@ -20,21 +25,29 @@ All endpoints return JSON responses with a standard format: - [Plugins](#plugins) - [Plugin Store](#plugin-store) - [System](#system) +- [Backup and Restore](#backup-and-restore) - [Fonts](#fonts) - [Cache](#cache) - [WiFi](#wifi) - [Streams](#streams) - [Logs](#logs) - [Error tracking](#error-tracking) -- [Health](#health) +- [Health and Status](#health-and-status) - [Schedule (dim/power)](#schedule-dimpower) +- [Integrations](#integrations) - [Plugin-specific endpoints](#plugin-specific-endpoints) - [Starlark Apps](#starlark-apps) +- [Skins](#skins) -> The API blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). -> SSE stream endpoints (`/api/v3/stream/*`) are defined directly on the -> Flask app at `app.py:799-809`. There are 111 routes total — see -> `web_interface/blueprints/api_v3.py` for the canonical list. +> The API blueprint is the `api_v3` package in +> `web_interface/blueprints/api_v3/` (one module per area: `config.py`, +> `display.py`, `plugins.py`, `system.py`, `backup.py`, `fonts.py`, +> `misc.py`, `wifi.py`, `starlark.py`). `web_interface/app.py` registers it +> at `/api/v3` (`app.register_blueprint(api_v3, url_prefix='/api/v3')`). +> The three SSE endpoints (`/api/v3/stream/*`) are defined directly on the +> Flask app in `app.py` (`stream_stats`, `stream_display`, `stream_logs`). +> `test/fixtures/api_v3_url_map.json` is the canonical list of blueprint +> routes (116 URL rules); a test fails if the code and that fixture differ. --- @@ -44,7 +57,9 @@ All endpoints return JSON responses with a standard format: **GET** `/api/v3/config/main` -Retrieve the complete main configuration file. +Return `config/config.json`. Fields whose names look like credentials +(`api_key`, `token`, `password`, `secret`, ...) are blanked to `""` in the +response; `config/config_secrets.json` values are never included. **Response**: ```json @@ -67,19 +82,36 @@ Retrieve the complete main configuration file. **POST** `/api/v3/config/main` -Update the main configuration. Accepts both JSON and form data. +Update the main configuration. Accepts JSON (`Content-Type: application/json`) +or form data. The body uses the web UI's flat field names, which the handler +maps into the nested config: + +| Fields | Stored at | +|--------|-----------| +| `timezone`, `city`, `state`, `country` | `timezone`, `location.*` | +| `web_display_autostart`, `auto_update_enabled` | `web_display_autostart`, `auto_update.enabled` | +| `plugins_directory` (and the unused legacy flags `auto_discover`, `auto_load_enabled`, `development_mode`, stored only when sent) | `plugin_system.*` | +| `target_fps` (30-200) | `target_fps` | +| `rows`, `cols`, `chain_length`, `parallel`, `brightness`, `hardware_mapping`, `pwm_bits`, `led_rgb_sequence`, `panel_type`, `pixel_mapper_config`, `disable_hardware_pulsing`, `inverse_colors`, `show_refresh_rate`, ... | `display.hardware.*` | +| `gpio_slowdown`, `rp1_rio` | `display.runtime.*` | +| `use_short_date_format` | `display.use_short_date_format` | +| `max_dynamic_duration_seconds` | `display.dynamic_duration.max_duration_seconds` | +| `double_sided_*`, `vegas_*`, `sync_*` | `display.double_sided`, `display.vegas_scroll`, `sync` | +| `_duration`, `default_duration`, `duration__` | `display.display_durations.*` | +| `plugin_rotation_order` (list of plugin ids) | `display.plugin_rotation_order` | + +Any other top-level key is deep-merged into the config as given. + +A JSON body changes only the keys it contains; everything else keeps its +stored value. (Form posts from the web UI send every field of a tab, and +there an unchecked checkbox — which the browser omits — is saved as +`false`.) **Request Body** (JSON): ```json { "timezone": "America/New_York", "city": "New York", - "state": "NY", - "country": "US", - "web_display_autostart": true, - "rows": 32, - "cols": 64, - "chain_length": 2, "brightness": 90 } ``` @@ -92,11 +124,15 @@ Update the main configuration. Accepts both JSON and form data. } ``` +Invalid values (e.g. an out-of-range `target_fps`, a hardware option the +Raspberry Pi 5 driver cannot use) are rejected with `400` and nothing is +saved. + ### Get Schedule Configuration **GET** `/api/v3/config/schedule` -Retrieve the current schedule configuration. +Retrieve the current on/off schedule. **Response**: ```json @@ -134,7 +170,7 @@ Retrieve the current schedule configuration. **POST** `/api/v3/config/schedule` -Update the schedule configuration. +Replace the schedule configuration. **Request Body** (Global mode): ```json @@ -146,7 +182,7 @@ Update the schedule configuration. } ``` -**Request Body** (Per-day mode): +**Request Body** (Per-day mode, flat form-field names): ```json { "enabled": true, @@ -160,6 +196,9 @@ Update the schedule configuration. } ``` +A day whose `_enabled` key is absent counts as enabled, with default +times `07:00`-`23:00`. At least one day must be enabled. + **Response**: ```json { @@ -172,19 +211,17 @@ Update the schedule configuration. **GET** `/api/v3/config/secrets` -Retrieve the secrets configuration (API keys, tokens, etc.). Secret values are masked for security. +Retrieve `config/config_secrets.json` with every set value replaced by eight +bullet characters (`"••••••••"`). Empty values and `YOUR_*` placeholders +are returned as-is, so a client can tell "set" from "not set". **Response**: ```json { "status": "success", "data": { - "weather": { - "api_key": "***" - }, - "spotify": { - "client_id": "***", - "client_secret": "***" + "ledmatrix-weather": { + "api_key": "••••••••" } } } @@ -194,11 +231,14 @@ Retrieve the secrets configuration (API keys, tokens, etc.). Secret values are m **POST** `/api/v3/config/raw/main` -Save raw JSON configuration (advanced use only). +Replace `config/config.json` with the JSON body (advanced use only). **POST** `/api/v3/config/raw/secrets` -Save raw secrets configuration (advanced use only). +Save the secrets file (advanced use only). Masked values (`"••••••••"`) +and blank strings in the body are dropped, and the rest is merged onto the +stored secrets, so posting back the GET response unchanged changes nothing. +A secret cannot be cleared by blanking it here. --- @@ -208,7 +248,8 @@ Save raw secrets configuration (advanced use only). **GET** `/api/v3/display/current` -Get the current display state and preview image. +Get the latest display snapshot as a base64 PNG (`image` is `null` when no +snapshot is available). **Response**: ```json @@ -223,6 +264,27 @@ Get the current display state and preview image. } ``` +### Get Current Display Status + +**GET** `/api/v3/display/current-status` + +The mode and plugin the display service is currently showing, as published +by the display process (stale after 120 seconds). + +**Response**: +```json +{ + "status": "success", + "data": { + "mode": "nfl_live", + "plugin_id": "football-scoreboard", + "last_updated": 1234567890.123 + } +} +``` + +When nothing has been published, every field is `null`. + ### List Display Modes **GET** `/api/v3/display/modes` @@ -295,12 +357,17 @@ Get the current on-demand display state. }, "service": { "active": true, - "returncode": 0 + "returncode": 0, + "stdout": "active", + "stderr": "" } } } ``` +With no on-demand request, `state` is +`{"active": false, "status": "idle", "last_updated": null}`. + ### Start On-Demand Display **POST** `/api/v3/display/on-demand/start` @@ -318,12 +385,12 @@ Request a specific plugin to display on-demand. } ``` -**Parameters**: +**Parameters** (at least one of `plugin_id` and `mode` is required): - `plugin_id` (string, optional): Plugin identifier - `mode` (string, optional): Display mode name (plugin_id inferred if not provided) - `duration` (number, optional): Duration in seconds (0 = until stopped) - `pinned` (boolean, optional): Pin display (pause rotation) -- `start_service` (boolean, optional): Auto-start display service if not running (default: true) +- `start_service` (boolean, optional): (Re)start the display service so it picks the request up (default: true) **Response**: ```json @@ -333,11 +400,15 @@ Request a specific plugin to display on-demand. "request_id": "uuid-here", "plugin_id": "football-scoreboard", "mode": "nfl_live", - "active": true + "duration": 45, + "pinned": true, + "service": { "active": true, "returncode": 0, "stdout": "", "stderr": "" } } } ``` +`service` is `null` when `start_service` is false. + ### Stop On-Demand Display **POST** `/api/v3/display/on-demand/stop` @@ -358,7 +429,10 @@ Stop the current on-demand display. ```json { "status": "success", - "message": "On-demand display stopped" + "data": { + "request_id": "uuid-here", + "service": null + } } ``` @@ -381,6 +455,9 @@ List all installed plugins with their status and metadata. { "id": "football-scoreboard", "name": "Football Scoreboard", + "version": "1.2.3", + "latest_version": "1.2.4", + "update_available": true, "author": "ChuckBuilds", "category": "Sports", "description": "NFL and NCAA Football scores", @@ -388,11 +465,15 @@ List all installed plugins with their status and metadata. "enabled": true, "verified": true, "loaded": true, + "state": "loaded", + "error_info": null, "last_updated": "2025-01-15T10:30:00Z", "last_commit": "abc1234", "last_commit_message": "feat: Add live game updates", "branch": "main", - "web_ui_actions": [] + "web_ui_actions": [], + "vegas_mode": null, + "vegas_content_type": null } ] } @@ -403,7 +484,8 @@ List all installed plugins with their status and metadata. **GET** `/api/v3/plugins/config?plugin_id=` -Get configuration for a specific plugin. +Get a plugin's configuration, with schema defaults filled in for keys that +are not stored. `data` is the configuration object itself. **Query Parameters**: - `plugin_id` (required): Plugin identifier @@ -413,12 +495,9 @@ Get configuration for a specific plugin. { "status": "success", "data": { - "plugin_id": "football-scoreboard", - "config": { - "enabled": true, - "display_duration": 30, - "favorite_teams": ["TB", "DAL"] - } + "enabled": true, + "display_duration": 30, + "favorite_teams": ["TB", "DAL"] } } ``` @@ -427,14 +506,17 @@ Get configuration for a specific plugin. **POST** `/api/v3/plugins/config` -Update plugin configuration. +Update a plugin's configuration. With a JSON body, the keys in `config` are +merged onto the plugin's stored configuration: keys you do not send keep +their stored values. Fields the schema marks `"x-secret": true` are written +to `config/config_secrets.json` instead of `config.json`. The web UI posts +form data instead (`?plugin_id=` in the query string, fields as form fields). **Request Body**: ```json { "plugin_id": "football-scoreboard", "config": { - "enabled": true, "display_duration": 30, "favorite_teams": ["TB", "DAL"] } @@ -445,15 +527,19 @@ Update plugin configuration. ```json { "status": "success", - "message": "Plugin configuration saved successfully" + "message": "Plugin football-scoreboard configuration saved successfully" } ``` +A config that fails schema validation is rejected with `400` and nothing is +saved. + ### Get Plugin Schema **GET** `/api/v3/plugins/schema?plugin_id=` -Get the JSON schema for a plugin's configuration. +Get the JSON schema for a plugin's configuration. A plugin without a +`config_schema.json` gets a minimal default schema. **Query Parameters**: - `plugin_id` (required): Plugin identifier @@ -463,27 +549,53 @@ Get the JSON schema for a plugin's configuration. { "status": "success", "data": { - "type": "object", - "properties": { - "enabled": { - "type": "boolean", - "default": true - }, - "display_duration": { - "type": "number", - "minimum": 1, - "maximum": 300 + "schema": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "display_duration": { + "type": "number", + "minimum": 1, + "maximum": 300 + } } } } } ``` +### Reset Plugin Configuration + +**POST** `/api/v3/plugins/config/reset` + +Reset a plugin's configuration to its schema defaults. + +**Request Body**: +```json +{ + "plugin_id": "football-scoreboard", + "preserve_secrets": true +} +``` + +**Response**: +```json +{ + "status": "success", + "message": "Plugin football-scoreboard configuration reset to defaults", + "data": { "config": { ... } } +} +``` + ### Toggle Plugin **POST** `/api/v3/plugins/toggle` -Enable or disable a plugin. +Enable or disable a plugin. A `plugin_id` of the form `starlark:` +toggles a Starlark app. **Request Body**: ```json @@ -497,7 +609,7 @@ Enable or disable a plugin. ```json { "status": "success", - "message": "Plugin football-scoreboard enabled" + "message": "Plugin football-scoreboard enabled successfully" } ``` @@ -510,22 +622,25 @@ Install a plugin from the plugin store. **Request Body**: ```json { - "plugin_id": "football-scoreboard" + "plugin_id": "football-scoreboard", + "branch": "main" } ``` -**Response**: +`branch` is optional. + +**Response** (queued; poll `/plugins/operation/`): ```json { "status": "success", - "data": { - "operation_id": "uuid-here", - "plugin_id": "football-scoreboard", - "status": "installing" - } + "data": { "operation_id": "uuid-here" }, + "message": "Plugin football-scoreboard installation queued" } ``` +When the operation queue is unavailable the install runs synchronously and +the response has only a `message`. + ### Uninstall Plugin **POST** `/api/v3/plugins/uninstall` @@ -535,15 +650,17 @@ Remove an installed plugin. **Request Body**: ```json { - "plugin_id": "football-scoreboard" + "plugin_id": "football-scoreboard", + "preserve_config": false } ``` -**Response**: +**Response** (queued): ```json { "status": "success", - "message": "Plugin football-scoreboard uninstalled" + "data": { "operation_id": "uuid-here" }, + "message": "Plugin uninstallation queued" } ``` @@ -551,7 +668,7 @@ Remove an installed plugin. **POST** `/api/v3/plugins/update` -Update a plugin to the latest version. +Update a plugin to the latest version. Runs synchronously. **Request Body**: ```json @@ -564,10 +681,10 @@ Update a plugin to the latest version. ```json { "status": "success", + "message": "Plugin football-scoreboard updated ...", "data": { - "operation_id": "uuid-here", - "plugin_id": "football-scoreboard", - "status": "updating" + "last_updated": "2025-01-15T10:30:00Z", + "commit": "abc1234..." } } ``` @@ -576,31 +693,31 @@ Update a plugin to the latest version. **POST** `/api/v3/plugins/install-from-url` -Install a plugin directly from a GitHub repository URL. +Install a plugin directly from a GitHub repository URL. Runs synchronously. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-my-plugin", + "repo_url": "https://github.com/user/ledmatrix-my-plugin", "branch": "main", "plugin_path": null } ``` **Parameters**: -- `url` (required): GitHub repository URL -- `branch` (optional): Branch name (default: "main") -- `plugin_path` (optional): Path within repository for monorepo plugins +- `repo_url` (required): GitHub repository URL +- `branch` (optional): Branch name (default: `main`, then `master`) +- `plugin_path` (optional): Path within the repository, for monorepo plugins +- `plugin_id` (optional): Plugin id, for monorepo installations -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "operation_id": "uuid-here", - "plugin_id": "my-plugin", - "status": "installing" - } + "message": "Plugin my-plugin installed successfully", + "plugin_id": "my-plugin", + "name": "My Plugin", + "branch": "main" } ``` @@ -608,28 +725,27 @@ Install a plugin directly from a GitHub repository URL. **POST** `/api/v3/plugins/registry-from-url` -Load a plugin registry from a GitHub repository URL. +Load a `plugins.json` registry from a GitHub repository URL. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-plugins" + "repo_url": "https://github.com/user/ledmatrix-plugins" } ``` -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "plugins": [ - { - "id": "plugin-1", - "name": "Plugin One", - "description": "..." - } - ] - } + "plugins": [ + { + "id": "plugin-1", + "name": "Plugin One", + "description": "..." + } + ], + "registry_url": "https://github.com/user/ledmatrix-plugins" } ``` @@ -637,7 +753,7 @@ Load a plugin registry from a GitHub repository URL. **GET** `/api/v3/plugins/health` -Get health metrics for all plugins. +Get health state for all installed plugins, keyed by plugin id. **Response**: ```json @@ -645,10 +761,20 @@ Get health metrics for all plugins. "status": "success", "data": { "football-scoreboard": { - "status": "healthy", - "last_update": 1234567890.123, - "error_count": 0, - "last_error": null + "plugin_id": "football-scoreboard", + "circuit_state": "closed", + "consecutive_failures": 0, + "total_failures": 2, + "total_successes": 1500, + "success_rate": 99.87, + "last_success_time": 1234567890.123, + "last_failure_time": 1234560000.0, + "last_error": null, + "is_healthy": true, + "degraded": false, + "degraded_reason": null, + "circuit_opened_time": null, + "half_open_start_time": null } } } @@ -658,20 +784,8 @@ Get health metrics for all plugins. **GET** `/api/v3/plugins/health/` -Get health metrics for a specific plugin. - -**Response**: -```json -{ - "status": "success", - "data": { - "status": "healthy", - "last_update": 1234567890.123, - "error_count": 0, - "last_error": null - } -} -``` +Health state for one plugin; `data` has the same fields as one entry above. +Answers `503` when health tracking is unavailable. ### Reset Plugin Health @@ -691,7 +805,7 @@ Reset health state for a plugin (manual recovery). **GET** `/api/v3/plugins/metrics` -Get resource usage metrics for all plugins. +Get resource usage metrics for all installed plugins, keyed by plugin id. **Response**: ```json @@ -699,21 +813,34 @@ Get resource usage metrics for all plugins. "status": "success", "data": { "football-scoreboard": { - "update_count": 150, - "display_count": 500, - "avg_update_time": 0.5, - "avg_display_time": 0.1, - "memory_usage": 1024000 + "plugin_id": "football-scoreboard", + "memory_mb": 24.5, + "cpu_percent": 3.2, + "execution_time": 0.12, + "avg_execution_time": 0.1, + "min_execution_time": 0.05, + "max_execution_time": 0.9, + "call_count": 500, + "last_update_time": 1234567890.123, + "limits": { + "max_memory_mb": 50, + "max_cpu_percent": 50, + "max_execution_time": 5.0, + "warning_threshold": 0.8 + } } } } ``` +`limits` (and usage percentages derived from it) appear only when limits +are configured for the plugin. + ### Get Plugin Metrics (Single) **GET** `/api/v3/plugins/metrics/` -Get resource usage metrics for a specific plugin. +Metrics for one plugin; `data` has the same fields as one entry above. ### Reset Plugin Metrics @@ -725,18 +852,33 @@ Reset metrics for a plugin. **GET** `/api/v3/plugins/limits/` -Get rate limits and resource limits for a plugin. +Get a plugin's resource limits. `data` is `null` when none are configured. + +**Response**: +```json +{ + "status": "success", + "data": { + "max_memory_mb": 50, + "max_cpu_percent": 50, + "max_execution_time": 5.0, + "warning_threshold": 0.8 + } +} +``` **POST** `/api/v3/plugins/limits/` -Update rate limits and resource limits for a plugin. +Set a plugin's resource limits. The body replaces all four limits: a key you +omit is stored as no limit (`warning_threshold` defaults to `0.8`). **Request Body**: ```json { - "max_update_interval": 60, - "max_display_time": 5.0, - "max_memory_mb": 50 + "max_memory_mb": 50, + "max_cpu_percent": 50, + "max_execution_time": 5.0, + "warning_threshold": 0.8 } ``` @@ -744,7 +886,8 @@ Update rate limits and resource limits for a plugin. **GET** `/api/v3/plugins/state` -Get the current state of all plugins. +Get the state manager's record for every plugin, keyed by plugin id. Pass +`?plugin_id=` for one plugin (`data` is then that record). **Response**: ```json @@ -752,9 +895,14 @@ Get the current state of all plugins. "status": "success", "data": { "football-scoreboard": { - "state": "loaded", + "plugin_id": "football-scoreboard", + "status": "loaded", "enabled": true, - "last_update": 1234567890.123 + "version": "1.2.3", + "installed_at": "2025-01-15T10:30:00", + "last_updated": "2025-01-15T10:30:00", + "config_version": 1, + "metadata": {} } } } @@ -764,21 +912,57 @@ Get the current state of all plugins. **POST** `/api/v3/plugins/state/reconcile` -Reconcile plugin state with configuration (fix inconsistencies). +Reconcile plugin state across config, disk and the state manager. + +**Request Body** (optional): +```json +{ + "force": false +} +``` **Response**: ```json { "status": "success", - "message": "Plugin state reconciled" + "message": "...", + "data": { + "inconsistencies_found": 1, + "inconsistencies_fixed": 1, + "inconsistencies_manual": 0, + "inconsistencies": [ + {"plugin_id": "...", "type": "...", "description": "...", "fix_action": "..."} + ], + "fixed": [ ... ], + "manual_fix_required": [ ... ] + } } ``` +### Get Reconciliation Status + +**GET** `/api/v3/plugins/reconciliation-status` + +Result of the last startup reconciliation, as written by the display service. + +**Response**: +```json +{ + "status": "success", + "data": { + "done": true, + "unresolved": [] + } +} +``` + +Before a run has finished, `data` is `{"done": false, "unresolved": []}`. + ### Get Plugin Operation **GET** `/api/v3/plugins/operation/` -Get status of an async plugin operation (install, update, etc.). +Get status of a queued plugin operation (install, uninstall). **Response**: ```json @@ -786,67 +970,80 @@ Get status of an async plugin operation (install, update, etc.). "status": "success", "data": { "operation_id": "uuid-here", - "type": "install", + "operation_type": "install", "plugin_id": "football-scoreboard", + "parameters": {}, "status": "completed", "progress": 100, - "message": "Installation completed successfully" + "message": "Installation completed successfully", + "error": null, + "result": { ... }, + "created_at": "2025-01-15T10:30:00", + "started_at": "2025-01-15T10:30:01", + "completed_at": "2025-01-15T10:30:20" } } ``` ### Get Operation History -**GET** `/api/v3/plugins/operation/history?limit=100` +**GET** `/api/v3/plugins/operation/history?limit=50` -Get history of plugin operations. +Get the plugin operation audit log. `data` is a list. **Query Parameters**: -- `limit` (optional): Maximum number of operations to return (default: 100) +- `limit` (optional): Maximum number of records (default: 50) +- `plugin_id` (optional): Only records for this plugin +- `operation_type` (optional): Only records of this type (`install`, `update`, `enable`, ...) **Response**: ```json { "status": "success", - "data": { - "operations": [ - { - "operation_id": "uuid-here", - "type": "install", - "plugin_id": "football-scoreboard", - "status": "completed", - "timestamp": 1234567890.123 - } - ] - } + "data": [ + { + "operation_id": "uuid-here", + "operation_type": "install", + "plugin_id": "football-scoreboard", + "timestamp": "2025-01-15T10:30:00", + "status": "success", + "user": null, + "details": null, + "error": null + } + ] } ``` +### Clear Operation History + +**DELETE** `/api/v3/plugins/operation/history` + +Clear the operation audit log. + ### Execute Plugin Action **POST** `/api/v3/plugins/action` -Execute a custom plugin action (defined in plugin's web_ui_actions). +Execute an action declared in the plugin manifest's `web_ui_actions`. See +[PLUGIN_WEB_UI_ACTIONS.md](PLUGIN_WEB_UI_ACTIONS.md). **Request Body**: ```json { "plugin_id": "football-scoreboard", - "action": "refresh_games", - "parameters": {} + "action_id": "refresh_games", + "params": {} } ``` -### Reset Plugin Configuration - -**POST** `/api/v3/plugins/config/reset` - -Reset a plugin's configuration to defaults. - -**Request Body**: +**Response** (fields at the top level; a script that prints JSON can return +its own object instead): ```json { - "plugin_id": "football-scoreboard" + "status": "success", + "message": "Action completed successfully", + "output": "script stdout" } ``` @@ -854,21 +1051,27 @@ Reset a plugin's configuration to defaults. **POST** `/api/v3/plugins/assets/upload` -Upload assets (images, files) for a plugin. +Upload images for a plugin. Stored under +`assets/plugins//uploads/`. **Request**: Multipart form data - `plugin_id` (required): Plugin identifier -- `file` (required): File to upload -- `asset_type` (optional): Type of asset (logo, image, etc.) +- `files` (required, repeatable, up to 10): PNG, JPEG, BMP or GIF images, 5 MB each, 50 MB total per plugin -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "filename": "logo.png", - "path": "plugins/football-scoreboard/assets/logo.png" - } + "uploaded_files": [ + { + "id": "uuid-here", + "filename": "image_1700000000_abcd1234.png", + "path": "assets/plugins/football-scoreboard/uploads/image_1700000000_abcd1234.png", + "size": 1024, + "uploaded_at": "2025-01-15T10:30:00Z" + } + ], + "total_files": 3 } ``` @@ -876,13 +1079,13 @@ Upload assets (images, files) for a plugin. **POST** `/api/v3/plugins/assets/delete` -Delete a plugin asset. +Delete an uploaded plugin image by its id (the `id` from upload or list). **Request Body**: ```json { "plugin_id": "football-scoreboard", - "filename": "logo.png" + "image_id": "uuid-here" } ``` @@ -890,7 +1093,7 @@ Delete a plugin asset. **GET** `/api/v3/plugins/assets/list?plugin_id=` -List all assets for a plugin. +List uploaded images for a plugin. **Query Parameters**: - `plugin_id` (required): Plugin identifier @@ -902,9 +1105,12 @@ List all assets for a plugin. "data": { "assets": [ { - "filename": "logo.png", - "path": "plugins/football-scoreboard/assets/logo.png", - "size": 1024 + "id": "uuid-here", + "filename": "image_1700000000_abcd1234.png", + "path": "assets/plugins/football-scoreboard/uploads/image_1700000000_abcd1234.png", + "size": 1024, + "uploaded_at": "2025-01-15T10:30:00Z", + "original_filename": "logo.png" } ] } @@ -915,59 +1121,76 @@ List all assets for a plugin. **POST** `/api/v3/plugins/authenticate/spotify` -Initiate Spotify authentication flow for music plugin. +Spotify OAuth for the music plugin (`ledmatrix-music`; the plugin is fixed, +not taken from the body). Two steps: call with an empty body to get the +authorization URL, then call again with the URL Spotify redirected to. -**Request Body**: +**Request Body** (step 2): ```json { - "plugin_id": "music" + "redirect_url": "http://127.0.0.1:8888/callback?code=..." } ``` -**Response**: +**Response** (step 1, fields at the top level): ```json { "status": "success", - "data": { - "auth_url": "https://accounts.spotify.com/authorize?..." - } + "message": "Authorization URL generated", + "auth_url": "https://accounts.spotify.com/authorize?..." } ``` +Step 2 returns `status`, `message` and the script's `output`. + ### Authenticate YouTube Music **POST** `/api/v3/plugins/authenticate/ytm` -Initiate YouTube Music authentication flow. - -**Request Body**: -```json -{ - "plugin_id": "music" -} -``` +Run the music plugin's YouTube Music authentication script. No body. Returns +`status`, `message` and the script's `output`. ### Upload Calendar Credentials **POST** `/api/v3/plugins/calendar/upload-credentials` -Upload Google Calendar credentials file. +Upload the Google OAuth client file for the calendar plugin. **Request**: Multipart form data -- `file` (required): credentials.json file +- `file` (required): `credentials.json` (JSON, max 1 MB) + +### Authenticate Calendar + +**POST** `/api/v3/plugins/calendar/authenticate` + +Google OAuth for the calendar plugin, in two steps. Step 1 (no body) returns +the consent URL. Step 2 posts the URL Google redirected to (it fails to load +in the browser, but its address carries the authorization code): + +```json +{ + "redirect_url": "http://localhost/?code=..." +} +``` + +Requires `credentials.json` to have been uploaded first (`400` otherwise). --- ## Plugin Store -### List Store Plugins +### List / Search Store Plugins -**GET** `/api/v3/plugins/store/list?fetch_commit_info=true` +**GET** `/api/v3/plugins/store/list` -Get list of available plugins from the plugin store. +List plugins from the registry and saved repositories. The same endpoint +searches. **Query Parameters**: -- `fetch_commit_info` (optional): Include commit information (default: false) +- `query` (optional): Text search over name, description, id +- `category` (optional): Category filter +- `tags` (optional, repeatable): Tag filter +- `fetch_commit_info` (optional): `false` to skip fetching commit metadata from GitHub (default: fetched) **Response**: ```json @@ -978,13 +1201,22 @@ Get list of available plugins from the plugin store. { "id": "football-scoreboard", "name": "Football Scoreboard", - "description": "NFL and NCAA Football scores", "author": "ChuckBuilds", "category": "Sports", + "description": "NFL and NCAA Football scores", + "tags": ["sports"], + "stars": 0, + "verified": true, + "repo": "https://github.com/ChuckBuilds/ledmatrix-plugins", + "last_updated": "2025-01-15", + "last_updated_iso": "2025-01-15T10:30:00Z", + "last_commit": "abc1234", + "last_commit_message": "...", + "last_commit_author": "...", "version": "1.2.3", - "repository_url": "https://github.com/ChuckBuilds/ledmatrix-football-scoreboard", - "installed": true, - "update_available": false + "branch": "main", + "default_branch": "main", + "plugin_path": "plugins/football-scoreboard" } ] } @@ -995,31 +1227,37 @@ Get list of available plugins from the plugin store. **GET** `/api/v3/plugins/store/github-status` -Get GitHub API rate limit status. +Whether a GitHub token is configured and valid. **Response**: ```json { "status": "success", "data": { + "token_status": "valid", + "authenticated": true, "rate_limit": 5000, - "rate_remaining": 4500, - "rate_reset": 1234567890 + "message": "GitHub API authenticated", + "error": null } } ``` +`token_status` is `none`, `valid` or `invalid`; `rate_limit` is the nominal +hourly limit (60 unauthenticated), not a live count. + ### Refresh Plugin Store **POST** `/api/v3/plugins/store/refresh` -Force refresh of the plugin store cache. +Force refresh of the registry cache. **Response**: ```json { "status": "success", - "message": "Plugin store refreshed" + "message": "Plugin store refreshed", + "plugin_count": 42 } ``` @@ -1027,7 +1265,7 @@ Force refresh of the plugin store cache. **GET** `/api/v3/plugins/saved-repositories` -Get list of saved custom plugin repositories. +Get the list of saved custom plugin repositories. **Response**: ```json @@ -1037,8 +1275,8 @@ Get list of saved custom plugin repositories. "repositories": [ { "url": "https://github.com/user/ledmatrix-plugins", - "name": "Custom Plugins", - "auto_load": true + "name": "ledmatrix-plugins", + "type": "registry" } ] } @@ -1049,14 +1287,14 @@ Get list of saved custom plugin repositories. **POST** `/api/v3/plugins/saved-repositories` -Save a custom plugin repository for easy access. +Save a custom plugin repository. Returns the updated list in +`data.repositories`. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-plugins", - "name": "Custom Plugins", - "auto_load": true + "repo_url": "https://github.com/user/ledmatrix-plugins", + "name": "Custom Plugins" } ``` @@ -1064,12 +1302,12 @@ Save a custom plugin repository for easy access. **DELETE** `/api/v3/plugins/saved-repositories` -Remove a saved repository. +Remove a saved repository. Returns the updated list in `data.repositories`. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-plugins" + "repo_url": "https://github.com/user/ledmatrix-plugins" } ``` @@ -1081,7 +1319,7 @@ Remove a saved repository. **GET** `/api/v3/system/status` -Get system status and metrics. +Get system status and metrics (cached for 10 seconds). **Response**: ```json @@ -1089,12 +1327,18 @@ Get system status and metrics. "status": "success", "data": { "timestamp": 1234567890.123, - "uptime": "Running", + "uptime": "3d 4h", + "uptime_seconds": 273600, "service_active": true, "cpu_percent": 25.5, "memory_used_percent": 45.2, + "memory_total_mb": 3794.0, + "memory_used_mb": 1715.0, + "memory_available_mb": 1900.0, "cpu_temp": 45.0, - "disk_used_percent": 60.0 + "disk_used_percent": 60.0, + "disk_total_gb": 29.0, + "disk_used_gb": 17.4 } } ``` @@ -1115,17 +1359,68 @@ Get LEDMatrix repository version. } ``` +### Check for Update + +**GET** `/api/v3/system/check-update` + +Whether `origin/main` has commits the checkout lacks. Cached briefly. +Fields at the top level (no envelope): + +```json +{ + "update_available": true, + "remote_sha": "abc123...", + "commits_behind": 3 +} +``` + +When git cannot run the check, the response also carries +`"check_failed": true` and an `error` explaining why. + +### Automatic Update Status + +**GET** `/api/v3/system/auto-update` + +Weekly automatic-update status for the General tab and the Overview banner: +`last_run`, `summary`, `status`, `next_due`, `alert`, `alert_id`, +`verifier_installed`, `setup_status`, `setup_message`, `verifying` (in +`data`). + +**POST** `/api/v3/system/auto-update/dismiss` + +Hide the current automatic-update alert until a new one replaces it. + +```json +{ + "alert_id": "..." +} +``` + +### Git Info + +**GET** `/api/v3/system/git-info` + +Branch, dirty state, recent commits and remote for the Tools tab. Fields at +the top level: `branch`, `dirty`, `status`, `recent_commits`, `remote_url` +(credentials scrubbed), `upstream`, `can_pull`. + +### Git Branches + +**GET** `/api/v3/system/git-branches` + +Fetches `origin` and lists branches to switch to: `current`, `upstream`, +`local` (list), `remote_only` (list), at the top level. + ### Execute System Action **POST** `/api/v3/system/action` -Execute system-level actions. +Execute system-level actions. JSON or form data. **Request Body**: ```json { - "action": "start_display", - "mode": "nfl_live" + "action": "restart_display_service" } ``` @@ -1137,19 +1432,87 @@ Execute system-level actions. - `enable_autostart`: Enable display service autostart - `disable_autostart`: Disable display service autostart - `reboot_system`: Reboot the Raspberry Pi -- `git_pull`: Update code from git repository +- `shutdown_system`: Power off the Raspberry Pi +- `git_pull`: Update LEDMatrix from git (the Update button) +- `checkout_branch`: Switch branch; takes `branch` and optional `stash` +- `force_git_reset`: `git reset --hard origin/main` +- `install_base_requirements`: pip install `requirements.txt` and `web_interface/requirements.txt` +- `install_plugin_requirements`: pip install every plugin's `requirements.txt` +- `clear_pycache`: Delete `__pycache__` directories -**Response**: +**Response** (service actions): ```json { "status": "success", - "message": "Action start_display completed", - "returncode": 0, - "stdout": "...", - "stderr": "" + "message": "Action completed" } ``` +A failed service action returns `"status": "error"` with `returncode` and +`stderr`. `git_pull` returns `message`, `restart_required` and +`dependency_failures`; the install actions return `output` or `details`. + +--- + +## Backup and Restore + +Backups are ZIP files kept in the backup export directory. + +### Preview + +**GET** `/api/v3/backup/preview` + +Summary of what a new backup would include. + +### List + +**GET** `/api/v3/backup/list` + +Stored backups, newest first. `data` is a list of +`{"filename", "size", "created_at"}`. + +### Export + +**POST** `/api/v3/backup/export` + +Create a backup. Returns `{"status": "success", "filename": "..."}`. + +### Validate + +**POST** `/api/v3/backup/validate` + +Check an uploaded backup and return its manifest in `data`. + +**Request**: Multipart form data +- `backup_file` (required): the ZIP + +### Restore + +**POST** `/api/v3/backup/restore` + +Restore an uploaded backup. + +**Request**: Multipart form data +- `backup_file` (required): the ZIP +- `options` (optional): JSON object; keys `restore_config`, `restore_secrets`, + `restore_wifi`, `restore_fonts`, `restore_plugin_uploads`, + `reinstall_plugins` (each defaults to `true`; unknown keys are rejected) + +A partial restore answers `500` with `"status": "error"`, a message listing +what did and didn't restore, and the result in `data`. + +### Download + +**GET** `/api/v3/backup/download/` + +Download a stored backup. + +### Delete + +**DELETE** `/api/v3/backup/` + +Delete a stored backup. + --- ## Fonts @@ -1158,20 +1521,26 @@ Execute system-level actions. **GET** `/api/v3/fonts/catalog` -Get list of available fonts. +Fonts in `assets/fonts/`, keyed by file name without extension. **Response**: ```json { "status": "success", "data": { - "fonts": [ - { - "family": "Press Start 2P", - "files": ["PressStart2P-Regular.ttf"], - "sizes": [8, 10, 12] + "catalog": { + "press_start": { + "filename": "press_start.ttf", + "family_name": "Press Start 2P", + "display_name": "Press Start 2P", + "path": "assets/fonts/press_start.ttf", + "type": "ttf", + "is_system": true, + "scalable": true, + "native_size": null, + "metadata": { ... } } - ] + } } } ``` @@ -1192,71 +1561,31 @@ Get font size token definitions. "sm": 8, "md": 10, "lg": 12, - "xl": 16 + "xl": 14, + "xxl": 16 } } } ``` -### Get Font Overrides - -**GET** `/api/v3/fonts/overrides` - -Get current font overrides. - -**Response**: -```json -{ - "status": "success", - "data": { - "overrides": { - "plugin.football-scoreboard.title": { - "family": "Arial", - "size_px": 12 - } - } - } -} -``` - -### Set Font Override - -**POST** `/api/v3/fonts/overrides` - -Set a font override for a specific element. - -**Request Body**: -```json -{ - "element_key": "plugin.football-scoreboard.title", - "family": "Arial", - "size_px": 12 -} -``` - -### Delete Font Override - -**DELETE** `/api/v3/fonts/overrides/` - -Remove a font override. - ### Upload Font **POST** `/api/v3/fonts/upload` -Upload a custom font file. +Upload a custom font file. It is saved as `assets/fonts/`. **Request**: Multipart form data -- `file` (required): Font file (.ttf, .otf, etc.) +- `font_file` (required): `.ttf`, `.otf` or `.bdf`, max 10 MB +- `font_family` (required): name for the font (letters, numbers, `_`, `-`) -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "family": "Custom Font", - "filename": "custom-font.ttf" - } + "message": "Font custom_font uploaded successfully", + "font_family": "custom_font", + "filename": "custom_font.ttf", + "path": "assets/fonts/custom_font.ttf" } ``` @@ -1264,13 +1593,36 @@ Upload a custom font file. **DELETE** `/api/v3/fonts/` -Delete an uploaded font. +Delete an uploaded font (`` is the file name without +extension). System fonts answer `403`. ### Font Preview -**GET** `/api/v3/fonts/preview?family=&text=` +**GET** `/api/v3/fonts/preview?font=&text=&size=12` -Render a small preview image of a font for use in the web UI font picker. +Render text in a font, for the web UI font picker. BDF fonts are not +previewed (`400`). + +**Query Parameters**: +- `font` (required): font file name in `assets/fonts/` (e.g. `press_start.ttf`) +- `text` (optional): up to 100 characters (default `Sample Text 123`) +- `size` (optional): 4-72 (default 12) +- `bg`, `fg` (optional): hex colours without `#` (default `000000` / `ffffff`) + +**Response**: +```json +{ + "status": "success", + "data": { + "image": "data:image/png;base64,...", + "width": 140, + "height": 32 + } +} +``` + +> Font overrides (`/api/v3/fonts/overrides`) were removed. Per-plugin font +> choices are made in each plugin's own settings. --- @@ -1280,20 +1632,16 @@ Render a small preview image of a font for use in the web UI font picker. **GET** `/api/v3/cache/list` -List all cache entries. +List cache files. **Response**: ```json { "status": "success", "data": { - "entries": [ - { - "key": "weather_current_12345", - "age": 300, - "size": 1024 - } - ] + "cache_files": [ ... ], + "cache_dir": "/var/cache/ledmatrix", + "total_files": 12 } } ``` @@ -1302,7 +1650,8 @@ List all cache entries. **POST** `/api/v3/cache/delete` -Delete a cache entry or clear all cache. +Delete one cache entry by key. There is no clear-all option here; use +`scripts/utils/clear_cache.py --clear-all` on the Pi for that. **Request Body**: ```json @@ -1311,13 +1660,6 @@ Delete a cache entry or clear all cache. } ``` -**Or clear all**: -```json -{ - "clear_all": true -} -``` - --- ## WiFi @@ -1336,7 +1678,10 @@ Get current WiFi connection status. "connected": true, "ssid": "MyNetwork", "ip_address": "192.168.1.100", - "signal_strength": -50 + "signal": 70, + "ap_mode_active": false, + "auto_enable_ap_mode": true, + "last_connect_attempt": null } } ``` @@ -1345,22 +1690,21 @@ Get current WiFi connection status. **GET** `/api/v3/wifi/scan` -Scan for available WiFi networks. +Scan for available WiFi networks. `data` is a list. If AP mode is active it is +turned off for the scan and back on afterwards, and `message` says so. **Response**: ```json { "status": "success", - "data": { - "networks": [ - { - "ssid": "MyNetwork", - "signal_strength": -50, - "encryption": "WPA2", - "connected": true - } - ] - } + "data": [ + { + "ssid": "MyNetwork", + "signal": 70, + "security": "WPA2", + "frequency": 2437 + } + ] } ``` @@ -1378,13 +1722,10 @@ Connect to a WiFi network. } ``` -**Response**: -```json -{ - "status": "success", - "message": "Connecting to MyNetwork..." -} -``` +**Response**: `"status": "success"` when connected; `"status": "pending"` +(with `data.ssid`) when the connection continues in the background — poll +`/wifi/status` and read `last_connect_attempt`. A wrong password answers +`400` with `"error_type": "wrong_password"`. ### Disconnect from WiFi @@ -1396,7 +1737,7 @@ Disconnect from current WiFi network. **POST** `/api/v3/wifi/ap/enable` -Enable WiFi access point mode. +Enable WiFi access point mode. Optional body `{"force": true}`. ### Disable Access Point Mode @@ -1404,19 +1745,16 @@ Enable WiFi access point mode. Disable WiFi access point mode. -### Get Auto-Enable AP Status +### Get Auto-Enable AP Setting **GET** `/api/v3/wifi/ap/auto-enable` -Get access point auto-enable configuration. - **Response**: ```json { "status": "success", "data": { - "auto_enable": true, - "timeout_seconds": 300 + "auto_enable_ap_mode": true } } ``` @@ -1425,13 +1763,29 @@ Get access point auto-enable configuration. **POST** `/api/v3/wifi/ap/auto-enable` -Configure access point auto-enable settings. - **Request Body**: ```json { - "auto_enable": true, - "timeout_seconds": 300 + "auto_enable_ap_mode": true +} +``` + +### WiFi Radio + +**GET** `/api/v3/wifi/radio` + +Radio state: `data.enabled` (`null` if unknown), `data.ethernet_connected`, +`data.available`. + +**POST** `/api/v3/wifi/radio` + +Turn the WiFi radio on or off. Turning it off is refused unless Ethernet is +connected or `force` is true, so you don't cut off your own connection. + +```json +{ + "enabled": false, + "force": false } ``` @@ -1439,39 +1793,34 @@ Configure access point auto-enable settings. ## Streams +Server-Sent Events, defined in `web_interface/app.py`. Each event is one +`data: ` line; idle connections get `: heartbeat` comments. + ### System Statistics Stream **GET** `/api/v3/stream/stats` -Server-Sent Events (SSE) stream for real-time system statistics. - -**Response**: SSE stream ``` -data: {"cpu_percent": 25.5, "memory_used_percent": 45.2, ...} - -data: {"cpu_percent": 26.0, "memory_used_percent": 45.3, ...} +data: {"timestamp": 1234567890.1, "uptime": "Running", "service_active": true, "cpu_percent": 25.5, "memory_used_percent": 45.2, "memory_available_mb": 1900.0, "cpu_temp": 45.0, "disk_used_percent": 60.0, "power": {...}} ``` ### Display Preview Stream **GET** `/api/v3/stream/display` -Server-Sent Events (SSE) stream for real-time display preview images. - -**Response**: SSE stream with base64-encoded images ``` -data: {"image": "base64_data_here", "timestamp": 1234567890.123} +data: {"timestamp": 1234567890.123, "width": 128, "height": 32, "image": "base64_data_here"} ``` ### Service Logs Stream **GET** `/api/v3/stream/logs` -Server-Sent Events (SSE) stream for real-time service logs. +Each event carries the latest journal lines for `ledmatrix` and +`ledmatrix-web` as one text block: -**Response**: SSE stream ``` -data: {"level": "INFO", "message": "Plugin loaded", "timestamp": 1234567890.123} +data: {"timestamp": 1234567890.123, "logs": "2025-01-15T10:30:00+0000 host python[123]: ..."} ``` --- @@ -1480,26 +1829,17 @@ data: {"level": "INFO", "message": "Plugin loaded", "timestamp": 1234567890.123} ### Get Logs -**GET** `/api/v3/logs?limit=100&level=INFO` +**GET** `/api/v3/logs` -Get recent log entries. - -**Query Parameters**: -- `limit` (optional): Maximum number of log entries (default: 100) -- `level` (optional): Filter by log level (DEBUG, INFO, WARNING, ERROR) +The last 100 journal lines for `ledmatrix.service` and +`ledmatrix-web.service`, as one text block. Takes no parameters. **Response**: ```json { "status": "success", "data": { - "logs": [ - { - "level": "INFO", - "message": "Plugin loaded: football-scoreboard", - "timestamp": 1234567890.123 - } - ] + "logs": "2025-01-15T10:30:00+0000 host python[123]: Plugin loaded: football-scoreboard\n..." } } ``` @@ -1512,31 +1852,54 @@ Get recent log entries. **GET** `/api/v3/errors/summary` -Aggregated counts of recent errors across all plugins and core -components, used by the web UI's error indicator. +Aggregated counts, detected patterns and recent errors across plugins and +core components. ### Get Plugin Errors **GET** `/api/v3/errors/plugin/` -Recent errors for a specific plugin. +Error health and statistics for one plugin. ### Clear Errors **POST** `/api/v3/errors/clear` -Clear the in-memory error aggregator. +Clear error records older than `max_age_hours` (default 24, 1-8760). +Returns `data.cleared_count`. + +```json +{ + "max_age_hours": 24 +} +``` --- -## Health +## Health and Status ### Health Check **GET** `/api/v3/health` -Lightweight liveness check used by the WiFi monitor and external -monitoring tools. +Health of the web interface, display service, config file, plugin system and +display snapshot. `data.status` is `healthy` or `degraded`, with +`data.services` and `data.checks`. + +### Hardware Status + +**GET** `/api/v3/hardware/status` + +LED matrix initialization result written by the display service at startup. +Before the service has written it, `data` is +`{"ok": null, "error": "Display service not yet started"}`. + +### Sync Status + +**GET** `/api/v3/sync/status` + +Live multi-display sync status from the display process; before it has +written one, `data` is `{"role", "port", "state": "starting"}` from config. --- @@ -1546,44 +1909,97 @@ monitoring tools. **GET** `/api/v3/config/dim-schedule` -Read the dim/power schedule that automatically reduces brightness or -turns the display off at configured times. +Read the schedule that lowers brightness at configured times. + +**Response**: +```json +{ + "status": "success", + "data": { + "enabled": true, + "dim_brightness": 30, + "mode": "per-day", + "days": { + "monday": { "enabled": true, "start_time": "20:00", "end_time": "07:00" }, + "tuesday": { ... } + } + } +} +``` + +In `global` mode, `start_time` and `end_time` sit at the top level instead +of `days`. ### Update Dim Schedule **POST** `/api/v3/config/dim-schedule` -Update the dim schedule. Body matches the structure returned by GET. +Replace the dim schedule. `dim_brightness` is 0-100 (default 30). In +`per-day` mode the days can be sent either as the `days` object that GET +returns, or as the web form's flat fields (`monday_enabled`, +`monday_start`, `monday_end`, ...). A day that is not sent counts as +enabled with default times `20:00`-`07:00`; at least one day must be +enabled. + +--- + +## Integrations + +### MQTT Bridge + +**GET** `/api/v3/integrations/mqtt-bridge` + +Home Assistant MQTT bridge service state and settings: `data.service`, +`data.config_exists`, `data.config_path`, `data.config` (password +omitted), `data.password_set`, `data.env_override_prefix`. + +**PUT** `/api/v3/integrations/mqtt-bridge/config` + +Write `integrations/mqtt_bridge/bridge_config.json`. Only the keys you send +change. The password is write-only: omit `mqtt_password` to keep it, send a +value to replace it, or send `"clear_password": true`. A password with +`mqtt_tls` off is refused unless `allow_insecure_mqtt` is true. Returns +`data.password_set` and `data.restart_required` (the bridge must be +restarted to pick up changes). See +[integrations/mqtt_bridge/README.md](../integrations/mqtt_bridge/README.md). --- ## Plugin-specific endpoints -A handful of endpoints belong to individual built-in or shipped plugins. +A handful of endpoints belong to individual plugins. ### Calendar **GET** `/api/v3/plugins/calendar/list-calendars` -List the calendars available on the authenticated Google account. -Used by the calendar plugin's config UI. +List the calendars on the authenticated Google account. Used by the calendar +plugin's config UI. Returns `calendars` at the top level. The upload and +authenticate endpoints are under [Plugins](#upload-calendar-credentials). ### Of The Day **POST** `/api/v3/plugins/of-the-day/json/upload` -Upload a JSON data file for the Of-The-Day plugin's category data. +Upload JSON data files (multipart field `files`) as Of-The-Day categories. +Returns `uploaded_files` and `total_files` at the top level. **POST** `/api/v3/plugins/of-the-day/json/delete` -Delete a previously uploaded Of-The-Day data file. +Delete an uploaded data file. + +```json +{ + "file_id": "category_name" +} +``` ### Plugin Static Assets **GET** `/api/v3/plugins//static/` -Serve a static asset (image, font, etc.) from a plugin's directory. -Used internally by the web UI to render plugin previews and icons. +Serve a static file from a plugin's directory. Used internally by the web UI +to render plugin previews and icons. --- @@ -1613,40 +2029,77 @@ Download and install the Pixlet binary on the Pi. **GET** `/api/v3/starlark/apps//config` — get app config schema **PUT** `/api/v3/starlark/apps//config` — update app config **POST** `/api/v3/starlark/apps//render` — render app to a frame -**POST** `/api/v3/starlark/apps//toggle` — enable/disable app +**POST** `/api/v3/starlark/apps//toggle` — enable/disable app (`{"enabled": bool}`; omit to flip) ### Repository (Tronbyt community apps) -**GET** `/api/v3/starlark/repository/categories` — browse categories -**GET** `/api/v3/starlark/repository/browse?category=` — browse apps -**POST** `/api/v3/starlark/repository/install` — install an app from the -community repository +**GET** `/api/v3/starlark/repository/categories` — list categories +**GET** `/api/v3/starlark/repository/browse` — every app with metadata (filtering happens client-side; cached for 2 hours) +**POST** `/api/v3/starlark/repository/install` — install an app: `{"app_id": "...", "render_interval": 300, "display_duration": 15}` ### Upload custom app **POST** `/api/v3/starlark/upload` -Upload a custom Starlark `.star` file as a new app. +Upload a custom Starlark `.star` file as a new app. Multipart fields: `file` +(required, max 5 MB), `name`, `app_id`, `render_interval`, +`display_duration`. + +### Editor + +A Pixlet editing session for one app. The display is stopped while a session +runs. + +**GET** `/api/v3/starlark/editor/apps` — apps the editor can open (`data.apps`, `data.apps_dir`, `data.pixlet_available`) +**GET** `/api/v3/starlark/editor/status` — `data.running`, plus `app_id`, `port`, `pid`, `started_at`, `timeout`, `seconds_remaining`, `host_bound` while running +**POST** `/api/v3/starlark/editor/start` — `{"app_id": "...", "timeout": 1800, "port": 8080}` (`timeout` and `port` optional) +**POST** `/api/v3/starlark/editor/stop` — end the session and restart the display + +--- + +## Skins + +**GET** `/api/v3/skins` + +Installed scoreboard skins (optional `?plugin_id=` filter). Skins are not +supported by the current scoreboard plugins, so the response carries +`data.supported: false` and a `data.message`; clients must not offer these +as selectable. See [SKIN_SYSTEM.md](SKIN_SYSTEM.md). --- ## Error Responses -All endpoints may return error responses in the following format: +Errors use one of two shapes. Most endpoints answer: ```json { "status": "error", "message": "Error description", - "error_code": "ERROR_CODE", "details": "Additional error details (optional)" } ``` +Endpoints built on the structured error helper add a code and category: + +```json +{ + "status": "error", + "error_code": "CONFIG_SAVE_FAILED", + "error_category": "configuration", + "message": "Error description", + "details": "optional", + "context": { }, + "suggested_fixes": [ ] +} +``` + **Common HTTP Status Codes**: - `200`: Success - `400`: Bad Request (invalid parameters) +- `403`: Forbidden (e.g. deleting a system font) - `404`: Not Found (resource doesn't exist) +- `408`: Timed out (plugin actions, auth scripts) - `500`: Internal Server Error - `503`: Service Unavailable (feature not available) @@ -1657,4 +2110,3 @@ All endpoints may return error responses in the following format: - [Plugin API Reference](PLUGIN_API_REFERENCE.md) - API for plugin developers - [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - Complete plugin development guide - [Web Interface README](../web_interface/README.md) - Web interface documentation - diff --git a/docs/SCROLL_PERFORMANCE.md b/docs/SCROLL_PERFORMANCE.md index ba5600d9..82e9f912 100644 --- a/docs/SCROLL_PERFORMANCE.md +++ b/docs/SCROLL_PERFORMANCE.md @@ -83,8 +83,9 @@ second refresh, instead of half a pixel every refresh (which has no good rendering, only a choice between blur and judder). `scroll_config.configure()` snaps the requested speed to the nearest entry on -the ladder and reports the hold that speed needs. It does **not** apply the -hold: the hold belongs to a scroll, not to a plugin's lifetime, and plugins +the ladder, sets the helper to advance that entry's whole-pixel step on every +presented frame (`ScrollHelper.set_pixels_per_frame`), and reports the hold +that speed needs. It does **not** apply the hold: the hold belongs to a scroll, not to a plugin's lifetime, and plugins share one display manager -- one set at construction is reset the moment any other plugin finishes scrolling. Apply it yourself when the scroll starts: @@ -102,10 +103,18 @@ self.display_manager.set_scrolling_state(True, frame_hold=settings.frame_hold) Passing `display_manager` only lets `configure` read the true refresh rate from `display.hardware`, which a plugin config cannot see. Skipping the -`set_scrolling_state` call is the mistake that matters: the speed still -resolves, but the panel keeps presenting a new frame every refresh, so a slow -snapped speed falls back to fractional pixels. Pass `snap_to_crisp=False` to -keep an exact requested speed and accept the artefacts. +`set_scrolling_state(True, frame_hold=...)` call is the mistake that matters. +The helper consults no clock in this mode -- it moves the fixed step once per +`update_scroll_position()` call, and `SwapOnVSync` is what paces those calls -- +so without the hold the panel presents a new frame every refresh and the scroll +runs `frame_hold` times too fast: 50 px/s (hold 2) plays at 100 px/s. + +Pass `snap_to_crisp=False` to keep an exact requested speed and accept the +artefacts. The helper then paces off elapsed time instead of stepping, and the +hold is 1. + +The General tab's `target_fps` ("Scroll Frame Rate") plays no part in any of +this: frames are presented at the panel refresh divided by the hold. Speeds slower than about 20 px/s are stepped no matter what, because a 1-pixel advance at 20 fps is simply a coarse increment. That is the pixel pitch, not a @@ -123,9 +132,12 @@ settings = scroll_config.configure( self.scroll_helper, plugin_config=self.config, global_config=self.global_config, - refresh_hz=scroll_config.refresh_hz_from_config(self.global_config), + display_manager=self.display_manager, plugin_logger=self.logger, ) + +# each frame of a scroll (or at least when it starts): +self.display_manager.set_scrolling_state(True, frame_hold=settings.frame_hold) ``` It resolves every config shape in one place, applies the speed, and returns @@ -154,6 +166,14 @@ it is always present and always wins, so the documented settings become unreachable. That is a real, shipped bug — see [ledmatrix-plugins#408](https://github.com/ChuckBuilds/ledmatrix-plugins/issues/408). +The flip side: a `scroll_pixels_per_second` you add by hand is ignored whenever +the plugin's config also carries the pair, which it does whenever the pair has +a schema default. Set the speed through the pair instead. + +The sports scoreboards (`src.common.sports_scroll`) are the exception to all of +the above: they read `scroll_settings.scroll_speed` per league as px/s directly, +and their `scroll_delay` is kept for compatibility but ignored for pacing. + If you are writing a plugin: do not give a deprecated key a schema default. ## What was actually wrong @@ -198,8 +218,16 @@ zero pixels and rendered an identical frame, which dirty-tracking skipped, so it returned in ~2 ms and the beat repeated. No `scroll_delay` value tunes this out — a shorter delay just trades stalled frames for periodic double-steps. -`ScrollHelper` now accumulates elapsed time in both modes at the same -configured speed, so position stays proportional to real time. +A crisp speed configured through `scroll_config` no longer consults a clock at +all. Once `SwapOnVSync` blocks until the panel has taken the frame, the frame +count is a truer clock than `time.time()`, so the helper advances a fixed whole +number of pixels per presented frame (`set_pixels_per_frame`) and the display +manager holds each frame for `frame_hold` refreshes. Every frame moves the eye +by the same amount. + +The time-based path remains only for callers that set a speed directly or pass +`snap_to_crisp=False`. There, frame-based mode no longer steps either: it +advances by elapsed time at `scroll_speed / scroll_delay` px/s. ## Diagnosing a juddery scroller @@ -222,11 +250,17 @@ p95 10.11ms max 12.03ms min 7.98ms | stalls 0 (0.0%) skips 0 (0.0%) Reading it, on a 100 Hz panel: +A healthy median is the refresh period times the scroll's frame hold: 10 ms +for a hold of 1 (100 px/s), **20 ms for 50 px/s** (hold 2), 30 ms for 33.3 px/s. +A 20 ms median on a 50 px/s scroll is the hold doing its job, not missed +refreshes. The `Scroll configured:` log line gives the hold (`1px every 2 +refreshes`). + | you see | it means | |---|---| -| median 10 ms, p95 within ~0.5 ms of it | healthy — locked to the panel | -| p95 or max at 20/30/50 ms | frames missing refreshes — per-frame work is overrunning, or a background thread is holding the GIL | -| non-zero **skips**, or a median *below* 10 ms | **duplicate frames** — the swap was skipped because the image did not change, so the frame never waited on vsync. The scroller is advancing less than one pixel per frame. | +| median = refresh period × hold, p95 within ~0.5 ms of it | healthy — locked to the panel | +| p95 or max a whole refresh period or more above that median | frames missing refreshes — per-frame work is overrunning, or a background thread is holding the GIL | +| non-zero **skips**, or a median *below* the expected one | **duplicate frames** — the swap was skipped because the image did not change, so the frame never waited on vsync. The scroller is advancing less than one pixel per frame, which a crisp fixed-step scroll never does; look for a plugin pacing off time or not passing the hold. | | non-zero **stalls** | frames past 1.5× the median, which is the measure of judder that survives averaging | `stalls` and `skips` are both counted against that window's own median, so they diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index 89592f80..c752337f 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -218,6 +218,14 @@ The one behavior the upstreamed version adds is native Part A threaded it through each copy by hand, and this makes that threading legacy compatibility rather than the mechanism. +> **Superseded.** Once presentation became frame-locked (#545) the helper +> steps a fixed whole-pixel amount per presented frame and the panel presents +> at its own refresh, so honouring `target_fps` only turned it into a speed +> multiplier (60 doubled a scoreboard's speed, 200 halved it). `sports_scroll` +> no longer reads it: the crisp-speed ladder uses the panel refresh +> (`display_manager.refresh_hz`), and speed comes from +> `scroll_settings.scroll_speed` alone. See `docs/SCROLL_PERFORMANCE.md`. + ## Phases B0–B3 are merged and shipping in core 3.2.0. Everything that remains is @@ -464,8 +472,9 @@ After adoption plus the frozen legacy copies it was 10,610; removing the dead inline duplication (plugins #252) brought it to roughly 8,620. B6 would take it to about 3,300 including the shared core module — some 2,400 fewer than before this project started. **Until B6 runs, the adoption is net negative on disk**, -and its one delivered user-visible gain is that adopted plugins honour the -global `target_fps` instead of hardcoding ~100 FPS. +and its one delivered user-visible gain was that adopted plugins honoured the +global `target_fps` instead of hardcoding ~100 FPS (since withdrawn: see the +note under the B3 design above). ### Decision: stop adopting further modules until B6 closes diff --git a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md index bc065eb1..4978b4c6 100644 --- a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md +++ b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md @@ -158,9 +158,11 @@ This script will check: - Python dependencies - Configuration files - File permissions -- Web interface availability +- Web interface availability (`ledmatrix-web` listening on port 5000) - Network connectivity +Once it passes, the web interface is at `http://:5000`. + ## Quick Reference Commands ```bash diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index fe3a1a74..db5b3cac 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -273,7 +273,7 @@ sudo systemctl cat ledmatrix-web | grep User 1. **Verify file structure:** ```bash ls -l web_interface/app.py - ls -l web_interface/blueprints/api_v3.py + ls -ld web_interface/blueprints/api_v3/ ls -l web_interface/blueprints/pages_v3.py ``` @@ -531,7 +531,7 @@ sudo systemctl cat ledmatrix-web | grep User ```bash # Clear the cache with the helper script - sudo python3 scripts/utils/clear_cache.py + sudo python3 scripts/utils/clear_cache.py --clear-all # Or remove files manually from the cache dir in use, e.g.: sudo rm -rf /var/cache/ledmatrix/* diff --git a/docs/WEB_INTERFACE_GUIDE.md b/docs/WEB_INTERFACE_GUIDE.md index 10ef082b..ebe6a1c7 100644 --- a/docs/WEB_INTERFACE_GUIDE.md +++ b/docs/WEB_INTERFACE_GUIDE.md @@ -102,7 +102,9 @@ Configure basic system settings: check are shown under the toggle, and anything other than success raises a banner on **Overview**. - *Checks first:* the code update is skipped, with the reason shown, if - tracked files were edited locally, the checkout has local commits, a + tracked files were edited locally (permission-only changes and edits under + `plugins/` or `plugin-repos/` don't count; the pull carries those across + and puts them back), the checkout has local commits, a rebase/merge is in progress, the branch has no upstream, less than 300 MB is free, or the newest version already failed once. A failed fetch is retried the next day. @@ -206,11 +208,11 @@ Manage fonts for your display: - See font previews - Check font sizes and styles -**Font Overrides:** -- Overrides are set per display *element* (e.g. a specific score or - clock text element), not per plugin -- Override default font choices for individual elements -- Preview font changes +**Font Preview:** +- Render sample text in any TTF/OTF font at a chosen size + +Fonts used by a plugin are chosen in that plugin's own settings tab; the +Fonts tab has no per-element override editor. **Delete Fonts:** - Remove unused fonts @@ -327,9 +329,8 @@ The web interface is built on a REST API that you can access programmatically: http://your-pi-ip:5000/api/v3 ``` -The API blueprint mounts at `/api/v3` (see -`web_interface/app.py:199`). All endpoints below are relative to that -base. +The API blueprint (`web_interface/blueprints/api_v3/`) is registered at +`/api/v3` in `web_interface/app.py`. **Common Endpoints:** - `GET /api/v3/config/main` — Get main configuration diff --git a/docs/plugin-safety-harness.md b/docs/plugin-safety-harness.md index eb127f18..5a869ad2 100644 --- a/docs/plugin-safety-harness.md +++ b/docs/plugin-safety-harness.md @@ -10,7 +10,8 @@ plugin without breaking a size or screen you didn't think to test. There is **no fixed set of supported panel sizes** — an RGB matrix build can be any width/height and configuration (square, rectangle, 2×2, 4×4, 8×2, long strips, tall stacks). Plugins are expected to read dimensions dynamically -(`self.display_manager.matrix.width/height`) and lay themselves out +(`self.display_manager.width/height` — not `matrix.width/height`, since +`matrix` is `None` when hardware init fails) and lay themselves out accordingly, so a hardcoded coordinate or unscaled font shows up as a failure here. diff --git a/integrations/mqtt_bridge/README.md b/integrations/mqtt_bridge/README.md index 29d31b04..555c6364 100644 --- a/integrations/mqtt_bridge/README.md +++ b/integrations/mqtt_bridge/README.md @@ -114,5 +114,10 @@ python3 integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py --config integrations/ discovery itself. Discovery is lazy and normally happens because somebody opened the dashboard; without that endpoint a bridge that never does would see an empty list. -- Brightness writes `display.hardware.brightness` through `/api/v3/config/main`. - The display service picks it up on its next restart, not instantly. +- Brightness writes `display.hardware.brightness` by posting + `{"brightness": N}` to `/api/v3/config/main`. A JSON save changes only the + keys it sends, so the other display settings are left as they were. The + display service's config hot reload notices the change within a few seconds + and applies it without a restart (unless `LEDMATRIX_HOT_RELOAD=false`; while + a dim schedule is dimming the panel, the dim level wins until the dim period + ends). diff --git a/plugin-repos/starlark-apps/manager.py b/plugin-repos/starlark-apps/manager.py index b3f26118..5a810079 100644 --- a/plugin-repos/starlark-apps/manager.py +++ b/plugin-repos/starlark-apps/manager.py @@ -239,7 +239,7 @@ class StarlarkAppsPlugin(BasePlugin): # Calculate optimal magnification based on display size self.calculated_magnify = self._calculate_optimal_magnify() if self.calculated_magnify > 1: - self.logger.info(f"Display size: {self.display_manager.matrix.width}x{self.display_manager.matrix.height}, " + self.logger.info(f"Display size: {self.display_manager.width}x{self.display_manager.height}, " f"recommended magnify: {self.calculated_magnify}") # Load installed apps @@ -323,8 +323,8 @@ class StarlarkAppsPlugin(BasePlugin): Recommended magnify value (1-8) """ try: - display_width = self.display_manager.matrix.width - display_height = self.display_manager.matrix.height + display_width = self.display_manager.width + display_height = self.display_manager.height # Tronbyte native resolution NATIVE_WIDTH = 64 @@ -362,8 +362,8 @@ class StarlarkAppsPlugin(BasePlugin): Dictionary with recommendation details """ try: - display_width = self.display_manager.matrix.width - display_height = self.display_manager.matrix.height + display_width = self.display_manager.width + display_height = self.display_manager.height NATIVE_WIDTH = 64 NATIVE_HEIGHT = 32 @@ -852,8 +852,8 @@ class StarlarkAppsPlugin(BasePlugin): # Scale frames if needed if self.config.get("scale_output", True): - width = self.display_manager.matrix.width - height = self.display_manager.matrix.height + width = self.display_manager.width + height = self.display_manager.height # Get scaling method from config scale_method_str = self.config.get("scale_method", "nearest") diff --git a/scripts/debug/debug_web_manual.py b/scripts/debug/debug_web_manual.py index 62620946..387a4300 100644 --- a/scripts/debug/debug_web_manual.py +++ b/scripts/debug/debug_web_manual.py @@ -54,8 +54,13 @@ def main(): config = config_manager.load_config() print(" ✅ Config loaded") - autostart = config.get('web_display_autostart', False) - print(f" 🔧 web_display_autostart: {autostart}") + # Same rule ledmatrix-web.service applies: only an explicit false/off + # keeps the web interface down; a missing key means on. + sys.path.insert(0, str(project_root / 'scripts' / 'utils')) + from start_web_conditionally import autostart_enabled + raw = config.get('web_display_autostart', '(not set, defaults to on)') + state = 'starts' if autostart_enabled(config) else 'will NOT start' + print(f" 🔧 web_display_autostart: {raw} (web interface {state})") except Exception as e: print(f" ❌ Config check failed: {e}") traceback.print_exc() diff --git a/scripts/dev/README.md b/scripts/dev/README.md index 3768f17e..cec5a631 100644 --- a/scripts/dev/README.md +++ b/scripts/dev/README.md @@ -12,9 +12,19 @@ This directory contains scripts and utilities for development and testing. ### Plugin Development Setup ```bash +# Official plugin: clones ChuckBuilds/ledmatrix-plugins (once) and links +# its plugins/ into plugins/ ./scripts/dev/dev_plugin_setup.sh link-github + +# Plugin with its own repository +./scripts/dev/dev_plugin_setup.sh link-github ``` +Set `plugin_system.plugins_directory` to `plugins` so the loader finds the +links. To use a fork or another clone location, copy +`dev_plugins.json.example` to `dev_plugins.json`. Details: +[docs/PLUGIN_DEVELOPMENT_GUIDE.md](../../docs/PLUGIN_DEVELOPMENT_GUIDE.md). + ### Running Emulator ```bash ./scripts/dev/run_emulator.sh diff --git a/scripts/dev/dev_plugin_setup.sh b/scripts/dev/dev_plugin_setup.sh index f9f6363b..1d4d1d39 100755 --- a/scripts/dev/dev_plugin_setup.sh +++ b/scripts/dev/dev_plugin_setup.sh @@ -10,8 +10,14 @@ PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" PLUGINS_DIR="$PROJECT_ROOT/plugins" CONFIG_FILE="$PROJECT_ROOT/dev_plugins.json" DEFAULT_DEV_DIR="$HOME/.ledmatrix-dev-plugins" -GITHUB_USER="ChuckBuilds" -GITHUB_PATTERN="ledmatrix-" +# Official plugins live in one monorepo: /, one +# directory per plugin under plugins/. Both can be overridden in +# dev_plugins.json (e.g. to work from a fork). +DEFAULT_GITHUB_USER="ChuckBuilds" +DEFAULT_PLUGINS_REPO="ledmatrix-plugins" +GITHUB_USER="$DEFAULT_GITHUB_USER" +PLUGINS_REPO="$DEFAULT_PLUGINS_REPO" +PLUGINS_BRANCH="" # Colors for output RED='\033[0;31m' @@ -37,18 +43,55 @@ log_error() { echo -e "${RED}[ERROR]${NC} $1" } +# Print a top-level string field of a JSON file, or nothing if it is absent. +# Uses jq when installed, else python3. +json_field() { + local file="$1" + local key="$2" + if command -v jq >/dev/null 2>&1; then + jq -r --arg k "$key" '.[$k] // empty | select(type == "string")' "$file" 2>/dev/null || true + elif command -v python3 >/dev/null 2>&1; then + python3 - "$file" "$key" <<'PY' 2>/dev/null || true +import json, sys +try: + with open(sys.argv[1], encoding="utf-8") as f: + value = json.load(f).get(sys.argv[2]) +except Exception: + value = None +if isinstance(value, str): + print(value) +PY + fi +} + # Load configuration file load_config() { + DEV_PLUGINS_DIR="$DEFAULT_DEV_DIR" if [[ -f "$CONFIG_FILE" ]]; then - DEV_PLUGINS_DIR=$(jq -r '.dev_plugins_dir // "'"$DEFAULT_DEV_DIR"'"' "$CONFIG_FILE" 2>/dev/null || echo "$DEFAULT_DEV_DIR") - # Expand ~ in path - DEV_PLUGINS_DIR="${DEV_PLUGINS_DIR/#\~/$HOME}" - else - DEV_PLUGINS_DIR="$DEFAULT_DEV_DIR" + local value + value=$(json_field "$CONFIG_FILE" dev_plugins_dir) + [[ -n "$value" ]] && DEV_PLUGINS_DIR="$value" + value=$(json_field "$CONFIG_FILE" github_user) + [[ -n "$value" ]] && GITHUB_USER="$value" + value=$(json_field "$CONFIG_FILE" plugins_repo) + [[ -n "$value" ]] && PLUGINS_REPO="$value" + value=$(json_field "$CONFIG_FILE" plugins_branch) + [[ -n "$value" ]] && PLUGINS_BRANCH="$value" + if [[ -n "$(json_field "$CONFIG_FILE" github_pattern)" ]]; then + log_warn "dev_plugins.json: github_pattern is no longer used (official plugins are in the $PLUGINS_REPO monorepo)" + fi fi + # Expand ~ in path + DEV_PLUGINS_DIR="${DEV_PLUGINS_DIR/#\~/$HOME}" mkdir -p "$DEV_PLUGINS_DIR" } +# Top level of the git checkout containing a path, or nothing. +# A monorepo plugin is a subdirectory, so its .git is not in the plugin dir. +git_root_of() { + git -C "$1" rev-parse --show-toplevel 2>/dev/null || true +} + # Validate plugin structure validate_plugin() { local plugin_path="$1" @@ -63,7 +106,7 @@ validate_plugin() { get_plugin_id() { local plugin_path="$1" if [[ -f "$plugin_path/manifest.json" ]]; then - jq -r '.id // empty' "$plugin_path/manifest.json" 2>/dev/null || echo "" + json_field "$plugin_path/manifest.json" id fi } @@ -176,50 +219,104 @@ clone_from_github() { return 0 } +# Clone a repository into DEV_PLUGINS_DIR, or update the existing clone. +# Prints the clone's path on stdout (log output goes to stderr). +ensure_clone() { + local repo_url="$1" + local branch="${2:-}" + local repo_name + repo_name=$(basename "$repo_url" .git) + local target_dir="$DEV_PLUGINS_DIR/$repo_name" + + if [[ -d "$target_dir" ]]; then + log_info "Repository already exists at $target_dir" >&2 + if [[ -d "$target_dir/.git" ]]; then + log_info "Updating repository..." >&2 + (cd "$target_dir" && git pull --rebase) >&2 || true + fi + else + if ! clone_from_github "$repo_url" "$target_dir" "$branch" >&2; then + return 1 + fi + fi + echo "$target_dir" +} + +# Find a plugin's directory inside a monorepo clone: plugins/, +# plugins/ledmatrix-, or the directory whose manifest id is . +find_monorepo_plugin() { + local repo_dir="$1" + local name="$2" + local candidate + for candidate in "$repo_dir/plugins/$name" "$repo_dir/plugins/ledmatrix-$name"; do + if [[ -f "$candidate/manifest.json" ]]; then + echo "$candidate" + return 0 + fi + done + for candidate in "$repo_dir"/plugins/*/; do + candidate="${candidate%/}" + [[ -f "$candidate/manifest.json" ]] || continue + if [[ "$(get_plugin_id "$candidate")" == "$name" ]]; then + echo "$candidate" + return 0 + fi + done + return 1 +} + # Link plugin from GitHub link_github_plugin() { - local plugin_name="$1" + local plugin_name="${1:-}" local repo_url="${2:-}" - + if [[ -z "$plugin_name" ]]; then log_error "Usage: $0 link-github [repo-url]" exit 1 fi - + load_config - - # Construct repo URL if not provided - if [[ -z "$repo_url" ]]; then - repo_url="https://github.com/${GITHUB_USER}/${GITHUB_PATTERN}${plugin_name}.git" - log_info "Using default GitHub URL: $repo_url" - fi - - # Determine target directory name from URL - local repo_name=$(basename "$repo_url" .git) - local target_dir="$DEV_PLUGINS_DIR/$repo_name" - - # Check if already cloned - if [[ -d "$target_dir" ]]; then - log_info "Repository already exists at $target_dir" - if [[ -d "$target_dir/.git" ]]; then - log_info "Updating repository..." - (cd "$target_dir" && git pull --rebase) || true - fi - else - # Clone the repository - if ! clone_from_github "$repo_url" "$target_dir"; then + + if [[ -n "$repo_url" ]]; then + # A plugin with its own repository (e.g. a third-party plugin): the + # repository root is the plugin. + local target_dir + if ! target_dir=$(ensure_clone "$repo_url"); then exit 1 fi + if ! validate_plugin "$target_dir"; then + log_error "Cloned repository does not appear to be a valid plugin" + exit 1 + fi + link_plugin "$plugin_name" "$target_dir" + return fi - - # Validate plugin structure - if ! validate_plugin "$target_dir"; then - log_error "Cloned repository does not appear to be a valid plugin" + + # Official plugins: clone the monorepo once, link plugins/ from it. + repo_url="https://github.com/${GITHUB_USER}/${PLUGINS_REPO}.git" + log_info "Using plugin monorepo: $repo_url" + local repo_dir + if ! repo_dir=$(ensure_clone "$repo_url" "$PLUGINS_BRANCH"); then exit 1 fi - - # Link the plugin - link_plugin "$plugin_name" "$target_dir" + + local plugin_dir + if ! plugin_dir=$(find_monorepo_plugin "$repo_dir" "$plugin_name"); then + log_error "No plugin named '$plugin_name' in $repo_dir/plugins" + log_info "Plugins are the directory names under $repo_dir/plugins, or their manifest ids" + exit 1 + fi + + # Link under the manifest id: that is the name the plugin loader and + # config.json use, and it can differ from the directory name + # (plugins/ledmatrix-music has id ledmatrix-music, not music). + local link_name + link_name=$(get_plugin_id "$plugin_dir") + [[ -n "$link_name" ]] || link_name=$(basename "$plugin_dir") + if [[ "$link_name" != "$plugin_name" ]]; then + log_info "Linking as '$link_name' (the plugin's manifest id)" + fi + link_plugin "$link_name" "$plugin_dir" } # Unlink a plugin @@ -274,7 +371,7 @@ list_plugins() { echo " → $target" # Check git status if it's a git repo - if [[ -d "$target/.git" ]]; then + if [[ -n "$(git_root_of "$target")" ]]; then local branch=$(cd "$target" && git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown") local status=$(cd "$target" && git status --porcelain 2>/dev/null | head -1) if [[ -n "$status" ]]; then @@ -327,7 +424,7 @@ check_status() { echo -e "${GREEN}✓${NC} ${BLUE}$plugin_name${NC}" echo " Path: $target" - if [[ -d "$target/.git" ]]; then + if [[ -n "$(git_root_of "$target")" ]]; then local branch=$(cd "$target" && git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown") local remote=$(cd "$target" && git remote get-url origin 2>/dev/null || echo "no remote") local commits_behind=$(cd "$target" && git rev-list --count HEAD..@{upstream} 2>/dev/null || echo "0") @@ -360,9 +457,13 @@ check_status() { done echo "Summary:" - echo " ${GREEN}Clean: $clean_count${NC}" - echo " ${YELLOW}Needs attention: $dirty_count${NC}" - [[ $broken_count -gt 0 ]] && echo -e " ${RED}Broken: $broken_count${NC}" + echo -e " ${GREEN}Clean: $clean_count${NC}" + echo -e " ${YELLOW}Needs attention: $dirty_count${NC}" + # An if, not `[[ ]] &&`: as the function's last command a false test made + # `status` exit 1 whenever nothing was broken. + if [[ $broken_count -gt 0 ]]; then + echo -e " ${RED}Broken: $broken_count${NC}" + fi } # Update plugin(s) @@ -384,39 +485,46 @@ update_plugins() { fi local target=$(get_symlink_target "$plugin_name") - - if [[ ! -d "$target/.git" ]]; then + local root + root=$(git_root_of "$target") + + if [[ -z "$root" ]]; then log_error "Plugin repository is not a git repository: $target" exit 1 fi - - log_info "Updating $plugin_name from $target" - (cd "$target" && git pull --rebase) + + log_info "Updating $plugin_name from $root" + (cd "$root" && git pull --rebase) log_success "Updated $plugin_name" else - # Update all linked plugins + # Update all linked plugins. Plugins linked from the monorepo share one + # checkout, which is pulled once. log_info "Updating all linked plugins..." local updated=0 local failed=0 - + local pulled_roots=" " + for item in "$PLUGINS_DIR"/*; do [[ -e "$item" ]] || continue [[ -d "$item" ]] || continue - + local name=$(basename "$item") [[ "$name" =~ ^\.|^_ ]] && continue - + if is_symlink "$item"; then local target=$(get_symlink_target "$name") - if [[ -d "$target/.git" ]]; then - log_info "Updating $name..." - if (cd "$target" && git pull --rebase); then - log_success "Updated $name" - updated=$((updated + 1)) - else - log_error "Failed to update $name" - failed=$((failed + 1)) - fi + local root + root=$(git_root_of "$target") + [[ -n "$root" ]] || continue + [[ "$pulled_roots" == *" $root "* ]] && continue + pulled_roots="$pulled_roots$root " + log_info "Updating $root (for $name)..." + if (cd "$root" && git pull --rebase); then + log_success "Updated $root" + updated=$((updated + 1)) + else + log_error "Failed to update $root" + failed=$((failed + 1)) fi fi done @@ -439,7 +547,12 @@ Commands: link-github [repo-url] Clone and link a plugin from GitHub - If repo-url is not provided, uses: https://github.com/${GITHUB_USER}/${GITHUB_PATTERN}.git + Without repo-url: clones (or updates) the official plugin monorepo, + https://github.com/${DEFAULT_GITHUB_USER}/${DEFAULT_PLUGINS_REPO}.git, and links its + plugins/ (or plugins/ledmatrix-) under the + plugin's manifest id + With repo-url: clones a plugin that has its own repository and links + the repository root unlink Remove symlink for a plugin (preserves repository) @@ -458,25 +571,30 @@ Commands: Show this help message Examples: - # Link a local plugin - $0 link music ../ledmatrix-music - - # Link from GitHub (auto-detects URL) - $0 link-github music - - # Link from GitHub with custom URL - $0 link-github stocks https://github.com/ChuckBuilds/ledmatrix-stocks.git - + # Link an official plugin from the monorepo + $0 link-github football-scoreboard + + # Link a plugin from a local monorepo checkout + $0 link hello-world ../ledmatrix-plugins/plugins/hello-world + + # Link a third-party plugin from its own repository + $0 link-github my-plugin https://github.com/OtherUser/ledmatrix-my-plugin.git + # Check status $0 status - + # Update all plugins $0 update Configuration: - Create dev_plugins.json in project root to customize: + Copy dev_plugins.json.example to dev_plugins.json (git-ignored) to customize: - dev_plugins_dir: Where to clone GitHub repos (default: ~/.ledmatrix-dev-plugins) - - plugins: Plugin definitions (optional, for auto-discovery) + - github_user: Owner of the plugin monorepo, e.g. your fork (default: ${DEFAULT_GITHUB_USER}) + - plugins_repo: Name of the plugin monorepo (default: ${DEFAULT_PLUGINS_REPO}) + - plugins_branch: Branch to clone the monorepo at (default: its default branch) + + Symlinks are created in plugins/. Set plugin_system.plugins_directory to + "plugins" in config/config.json so the plugin loader discovers them. EOF } diff --git a/scripts/dev_server.py b/scripts/dev_server.py index 7a7dd945..4372f14b 100644 --- a/scripts/dev_server.py +++ b/scripts/dev_server.py @@ -142,17 +142,18 @@ def find_plugin_dir(plugin_id: str) -> Optional[Path]: def load_config_defaults(plugin_dir: 'str | Path') -> Dict[str, Any]: - """Extract default values from config_schema.json.""" + """Extract default values from config_schema.json. + + The same extraction a device and the plugin harness use + (src/plugin_system/testing/loading.py), nested defaults included. + """ + from src.plugin_system.testing.loading import ( + load_config_defaults as _load_config_defaults, + ) schema_path = resolve_under(plugin_dir, 'config_schema.json') if schema_path is None or not schema_path.exists(): return {} - with open(schema_path, 'r') as f: - schema = json.load(f) - defaults: Dict[str, Any] = {} - for key, prop in schema.get('properties', {}).items(): - if 'default' in prop: - defaults[key] = prop['default'] - return defaults + return _load_config_defaults(schema_path.parent) # -------------------------------------------------------------------------- @@ -303,10 +304,13 @@ def _parse_render_request(data): with open(manifest_path, 'r') as f: manifest = json.load(f) - # Build config: schema defaults + user overrides - config = {'enabled': True} - config.update(load_config_defaults(trusted_dir)) - config.update(data.get('config', {})) + # Build config the way a device would: schema defaults under a forced + # enabled, with the user's overrides deep-merged on top + from src.plugin_system.testing.loading import build_config + overrides = data.get('config') or {} + if not isinstance(overrides, dict): + raise ValueError('config must be a JSON object') + config = build_config(trusted_dir, overrides) return trusted_dir, manifest, config, data.get('mock_data', {}), data.get('skip_update', False) diff --git a/scripts/diagnose_web_interface.sh b/scripts/diagnose_web_interface.sh index b9b304c3..4bc3dd70 100755 --- a/scripts/diagnose_web_interface.sh +++ b/scripts/diagnose_web_interface.sh @@ -20,6 +20,38 @@ PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" cd "$PROJECT_DIR" +# Report web_display_autostart the way scripts/utils/start_web_conditionally.py +# (what ledmatrix-web.service runs) decides it: only an explicit false/off keeps +# the web interface down; a missing key or an unreadable config starts it. +# Prints "on ", "off ", "default" (key not set) or "unreadable". +web_autostart_state() { + (cd "$1" && python3 - 2>/dev/null <<'PY' +import json, os, sys +sys.path.insert(0, os.path.join(os.getcwd(), "scripts", "utils")) +try: + from start_web_conditionally import autostart_enabled +except Exception: + def autostart_enabled(config): + value = config.get("web_display_autostart", True) + if isinstance(value, str): + return value.strip().lower() not in ("off", "false", "no", "0") + return bool(value) +try: + with open(os.path.join("config", "config.json"), encoding="utf-8") as f: + config = json.load(f) +except Exception: + config = None +if not isinstance(config, dict): + print("unreadable") +elif "web_display_autostart" not in config: + print("default") +else: + raw = json.dumps(config["web_display_autostart"]) + print(("on " if autostart_enabled(config) else "off ") + raw) +PY + ) || echo "unknown" +} + echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" echo -e "${BLUE}1. SERVICE STATUS${NC}" echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" @@ -41,14 +73,26 @@ if [ -f "$PROJECT_DIR/config/config.json" ]; then echo -e "${GREEN}✓ Config file found${NC}" # Check web_display_autostart setting - AUTOSTART=$(grep -o '"web_display_autostart"[[:space:]]*:[[:space:]]*[a-z]*' "$PROJECT_DIR/config/config.json" | grep -o '[a-z]*$') - - if [ "$AUTOSTART" == "true" ]; then - echo -e "${GREEN}✓ web_display_autostart: true${NC}" - else - echo -e "${YELLOW}⚠ web_display_autostart: ${AUTOSTART:-not set}${NC}" - echo -e "${YELLOW} Web interface will not start unless this is set to true${NC}" - fi + AUTOSTART=$(web_autostart_state "$PROJECT_DIR") + + case "$AUTOSTART" in + on\ *) + echo -e "${GREEN}✓ web_display_autostart: ${AUTOSTART#on }${NC}" + ;; + off\ *) + echo -e "${YELLOW}⚠ web_display_autostart: ${AUTOSTART#off }${NC}" + echo -e "${YELLOW} Web interface will not start with this value${NC}" + ;; + default) + echo -e "${GREEN}✓ web_display_autostart: not set (defaults to on)${NC}" + ;; + unreadable) + echo -e "${YELLOW}⚠ config.json could not be parsed (the web interface still starts so it can be repaired)${NC}" + ;; + *) + echo -e "${YELLOW}⚠ web_display_autostart: could not be evaluated (python3 unavailable?)${NC}" + ;; + esac else echo -e "${RED}✗ Config file not found at: $PROJECT_DIR/config/config.json${NC}" fi @@ -63,7 +107,7 @@ declare -a REQUIRED_FILES=( "web_interface/app.py" "web_interface/start.py" "web_interface/requirements.txt" - "web_interface/blueprints/api_v3.py" + "web_interface/blueprints/api_v3/__init__.py" "web_interface/blueprints/pages_v3.py" "scripts/utils/start_web_conditionally.py" ) @@ -134,8 +178,8 @@ if ! sudo systemctl is-active --quiet ledmatrix-web; then echo " sudo systemctl start ledmatrix-web" fi -if [ "$AUTOSTART" != "true" ]; then - echo -e "${YELLOW}→ Enable web_display_autostart in config/config.json${NC}" +if [ "${AUTOSTART%% *}" = "off" ]; then + echo -e "${YELLOW}→ Set web_display_autostart to true in config/config.json (or remove it; missing means on)${NC}" fi if [ "$ALL_FILES_OK" = false ]; then diff --git a/scripts/diagnose_web_ui.sh b/scripts/diagnose_web_ui.sh index e2226f1a..1bc26306 100755 --- a/scripts/diagnose_web_ui.sh +++ b/scripts/diagnose_web_ui.sh @@ -22,6 +22,38 @@ fi PROJECT_DIR="${HOME}/LEDMatrix" +# Report web_display_autostart the way scripts/utils/start_web_conditionally.py +# (what ledmatrix-web.service runs) decides it: only an explicit false/off keeps +# the web interface down; a missing key or an unreadable config starts it. +# Prints "on ", "off ", "default" (key not set) or "unreadable". +web_autostart_state() { + (cd "$1" && python3 - 2>/dev/null <<'PY' +import json, os, sys +sys.path.insert(0, os.path.join(os.getcwd(), "scripts", "utils")) +try: + from start_web_conditionally import autostart_enabled +except Exception: + def autostart_enabled(config): + value = config.get("web_display_autostart", True) + if isinstance(value, str): + return value.strip().lower() not in ("off", "false", "no", "0") + return bool(value) +try: + with open(os.path.join("config", "config.json"), encoding="utf-8") as f: + config = json.load(f) +except Exception: + config = None +if not isinstance(config, dict): + print("unreadable") +elif "web_display_autostart" not in config: + print("default") +else: + raw = json.dumps(config["web_display_autostart"]) + print(("on " if autostart_enabled(config) else "off ") + raw) +PY + ) || echo "unknown" +} + echo "1. Checking service status..." echo "------------------------------" if systemctl is-active --quiet ledmatrix-web 2>/dev/null || sudo systemctl is-active --quiet ledmatrix-web 2>/dev/null; then @@ -47,16 +79,25 @@ echo "3. Checking configuration file..." echo "------------------------------" if [ -f "${PROJECT_DIR}/config/config.json" ]; then echo -e "${GREEN}✓ Config file exists${NC}" - AUTOSTART=$(grep -o '"web_display_autostart":\s*\(true\|false\)' "${PROJECT_DIR}/config/config.json" | grep -o '\(true\|false\)' || echo "not found") - if [ "$AUTOSTART" = "true" ]; then - echo -e "${GREEN}✓ web_display_autostart is set to TRUE${NC}" - elif [ "$AUTOSTART" = "false" ]; then - echo -e "${RED}✗ web_display_autostart is set to FALSE (web UI won't start!)${NC}" - echo " Fix: Edit config.json and set 'web_display_autostart': true" - else - echo -e "${YELLOW}⚠ web_display_autostart setting not found (defaults to false)${NC}" - echo " Fix: Add 'web_display_autostart': true to config.json" - fi + AUTOSTART=$(web_autostart_state "$PROJECT_DIR") + case "$AUTOSTART" in + on\ *) + echo -e "${GREEN}✓ web_display_autostart is ${AUTOSTART#on } (web UI starts)${NC}" + ;; + off\ *) + echo -e "${RED}✗ web_display_autostart is ${AUTOSTART#off } (web UI won't start!)${NC}" + echo " Fix: Edit config.json and set 'web_display_autostart': true" + ;; + default) + echo -e "${GREEN}✓ web_display_autostart is not set (defaults to on; web UI starts)${NC}" + ;; + unreadable) + echo -e "${YELLOW}⚠ config.json could not be parsed (the web UI still starts so it can be repaired)${NC}" + ;; + *) + echo -e "${YELLOW}⚠ Could not evaluate web_display_autostart (python3 unavailable?)${NC}" + ;; + esac else echo -e "${RED}✗ Config file NOT FOUND at ${PROJECT_DIR}/config/config.json${NC}" fi @@ -82,7 +123,7 @@ FILES_TO_CHECK=( "web_interface/start.py" "web_interface/app.py" "web_interface/requirements.txt" - "web_interface/blueprints/api_v3.py" + "web_interface/blueprints/api_v3/__init__.py" "web_interface/blueprints/pages_v3.py" ) @@ -175,7 +216,7 @@ echo "Diagnostic Summary" echo "==========================================" echo "" echo "Most common issues:" -echo " 1. web_display_autostart is false or missing in config.json" +echo " 1. web_display_autostart is set to false in config.json (a missing key means on)" echo " 2. Service not enabled or not started" echo " 3. Missing dependencies (Flask, etc.)" echo " 4. Import errors in web_interface/app.py" diff --git a/scripts/fix_perms/README.md b/scripts/fix_perms/README.md index 5616e51a..77575a71 100644 --- a/scripts/fix_perms/README.md +++ b/scripts/fix_perms/README.md @@ -7,8 +7,10 @@ install as the wrong user, after a manual file copy that didn't preserve ownership, or after a permissions-related error from the display or web service. -Most of these scripts require `sudo` since they touch directories -owned by the `ledmatrix` service user or by `root`. +Most of these scripts require `sudo` since they touch directories owned +by `root` (the display service's user) or by the user you installed +LEDMatrix as (the web service's user). There is no dedicated `ledmatrix` +system user. ## Scripts @@ -16,11 +18,12 @@ owned by the `ledmatrix` service user or by `root`. permissions on the `assets/` tree so plugins can download and cache team logos, fonts, and other static content. -- **`fix_cache_permissions.sh`** — Fixes permissions on every cache - directory the project may use (`/var/cache/ledmatrix/`, - `~/.cache/ledmatrix/`, `/opt/ledmatrix/cache/`, project-local - `cache/`). Also creates placeholder logo subdirectories used by the - sports plugins. +- **`fix_cache_permissions.sh`** — Creates (if missing) and fixes + permissions on `/var/cache/ledmatrix/` and `~/.ledmatrix_cache/` of the + user running `sudo`, and creates + `/var/cache/ledmatrix/placeholder_logos/` for the sports plugins. It does + not touch the cache manager's other fallbacks (`/opt/ledmatrix/cache`, + `$TMPDIR/ledmatrix_cache`). - **`fix_plugin_permissions.sh`** — Fixes ownership on the plugins directory so both the root display service and the web service user @@ -31,6 +34,11 @@ owned by the `ledmatrix` service user or by `root`. systemd journal access, and the sudoers entries the web interface needs to control the display service. +- **`safe_pip_install.sh`** — Installs a `requirements.txt` as root + after checking it is the project's own or one under `plugin-repos/` or + `plugins/`. Used by the web interface (via sudo) to install plugin + dependencies where `ledmatrix.service` can import them. + - **`safe_plugin_rm.sh`** — Validates that a plugin removal path is inside an allowed base directory before deleting it. Used by the web interface (via sudo) when a user clicks **Uninstall** on a plugin — diff --git a/scripts/fix_perms/safe_pip_install.sh b/scripts/fix_perms/safe_pip_install.sh index ae7d266c..06557e6f 100755 --- a/scripts/fix_perms/safe_pip_install.sh +++ b/scripts/fix_perms/safe_pip_install.sh @@ -1,6 +1,7 @@ #!/bin/bash # safe_pip_install.sh — Install a requirements.txt as root after validating -# that the resolved path is the project's own requirements.txt or a plugin's +# that the resolved path is one of the project's own requirements files +# (requirements.txt, web_interface/requirements.txt) or a plugin's # requirements.txt under plugin-repos/ or plugins/. # # This script is intended to be called via sudo from the web interface, so @@ -25,9 +26,17 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" # Allowed locations (resolved, no trailing slash): -# - the project's own requirements.txt +# - the project's own requirements files. Update Code, the automatic +# update's health check and Install Base Requirements install both, so +# one missing here is refused on every device -- and the automatic +# updater rolls back any update that changes it. +# Only their folders are resolved: resolving the files too would follow +# a requirements.txt symlinked out of the project and allow its target. # - any requirements.txt under plugin-repos/ or plugins/ -ALLOWED_EXACT="$(realpath --canonicalize-missing "$PROJECT_ROOT/requirements.txt")" +ALLOWED_EXACT=( + "$(realpath --canonicalize-missing "$PROJECT_ROOT")/requirements.txt" + "$(realpath --canonicalize-missing "$PROJECT_ROOT/web_interface")/requirements.txt" +) ALLOWED_BASES=( "$(realpath --canonicalize-missing "$PROJECT_ROOT/plugin-repos")" "$(realpath --canonicalize-missing "$PROJECT_ROOT/plugins")" @@ -43,9 +52,13 @@ if [ "$(basename "$RESOLVED_TARGET")" != "requirements.txt" ]; then fi ALLOWED=false -if [ "$RESOLVED_TARGET" = "$ALLOWED_EXACT" ]; then - ALLOWED=true -else +for EXACT in "${ALLOWED_EXACT[@]}"; do + if [ "$RESOLVED_TARGET" = "$EXACT" ]; then + ALLOWED=true + break + fi +done +if [ "$ALLOWED" = false ]; then for BASE in "${ALLOWED_BASES[@]}"; do if [[ "$RESOLVED_TARGET" == "$BASE/"* ]]; then ALLOWED=true @@ -56,7 +69,7 @@ fi if [ "$ALLOWED" = false ]; then echo "DENIED: $RESOLVED_TARGET is not an allowed requirements.txt location" >&2 - echo "Allowed: $ALLOWED_EXACT, or any requirements.txt under: ${ALLOWED_BASES[*]}" >&2 + echo "Allowed: ${ALLOWED_EXACT[*]}, or any requirements.txt under: ${ALLOWED_BASES[*]}" >&2 exit 2 fi diff --git a/scripts/install/README.md b/scripts/install/README.md index 0813f286..dc092d9d 100644 --- a/scripts/install/README.md +++ b/scripts/install/README.md @@ -7,14 +7,18 @@ This directory contains scripts for installing and configuring the LEDMatrix sys - **`one-shot-install.sh`** - Single-command installer; clones the repo, checks prerequisites, then runs `first_time_install.sh`. Invoked via `curl ... | bash` from the project root README. -- **`install_service.sh`** - Installs the main LED Matrix display service (systemd) -- **`install_web_service.sh`** - Installs the web interface service (systemd) +- **`install_service.sh`** - Installs, enables and starts the display + service (`ledmatrix.service`), the web interface service + (`ledmatrix-web.service`) and the update-verify units (systemd) +- **`install_web_service.sh`** - Installs only the web interface service + and the update-verify units (systemd) - **`install_wifi_monitor.sh`** - Installs the WiFi monitor daemon service - **`setup_cache.sh`** - Sets up persistent cache directory with proper permissions - **`configure_web_sudo.sh`** - Configures passwordless sudo access for web interface actions -- **`configure_wifi_permissions.sh`** - Grants the `ledmatrix` user - the WiFi management permissions needed by the web interface and - the WiFi monitor service +- **`configure_wifi_permissions.sh`** - Grants the web interface's user + (the user who runs the script, i.e. the one you installed LEDMatrix as; + there is no `ledmatrix` system user) the passwordless `nmcli` and related + WiFi permissions the web interface needs - **`migrate_config.sh`** - Migrates configuration files to new formats (if needed) - **`debug_install.sh`** - Diagnostic helper used when an install fails; collects environment info and recent logs diff --git a/scripts/install/install_service.sh b/scripts/install/install_service.sh index 0d88891e..f0d9de0a 100755 --- a/scripts/install/install_service.sh +++ b/scripts/install/install_service.sh @@ -3,6 +3,41 @@ # Exit on error set -e +usage() { + cat <<'USAGE' +Usage: sudo ./scripts/install/install_service.sh [-h|--help] + +Installs (or reinstalls) the LEDMatrix systemd units from the templates in +systemd/, then enables and starts them: + - ledmatrix.service main display (runs as root) + - ledmatrix-web.service web interface (runs as the invoking user) + - ledmatrix-update-verify.service automatic-update health check + - ledmatrix-update-verify.path + +Existing unit files in /etc/systemd/system are overwritten. The script takes +no other options; run it with no arguments to install. + +Options: + -h, --help Show this help and exit without changing anything. +USAGE +} + +# Parse arguments before touching anything: this script rewrites and restarts +# services, so an unrecognised option must not fall through to a full install. +for arg in "$@"; do + case "$arg" in + -h|--help) + usage + exit 0 + ;; + *) + echo "ERROR: unknown option: $arg" >&2 + usage >&2 + exit 2 + ;; + esac +done + # Get the actual user who invoked sudo if [ -n "$SUDO_USER" ]; then ACTUAL_USER="$SUDO_USER" diff --git a/scripts/install_plugin_dependencies.sh b/scripts/install_plugin_dependencies.sh index fc0cf503..6e4eca09 100755 --- a/scripts/install_plugin_dependencies.sh +++ b/scripts/install_plugin_dependencies.sh @@ -3,6 +3,8 @@ # Use this if automatic dependency installation fails set -e +# A failed pip must fail the `pip ... | tee` pipeline below, not be hidden by tee. +set -o pipefail # Colors for output RED='\033[0;31m' @@ -28,15 +30,50 @@ echo "" # Get the directory where this script is located SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" LEDMATRIX_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" -PLUGINS_DIR="$LEDMATRIX_DIR/plugins" +CONFIG_FILE="$LEDMATRIX_DIR/config/config.json" + +# The Plugin Store installs into plugin_system.plugins_directory from +# config.json (default plugin-repos), resolved against the project root like +# the display and web services do. plugins/ is also scanned: it holds the +# symlinks scripts/dev/dev_plugin_setup.sh creates. +CONFIGURED_DIR="plugin-repos" +if [ -f "$CONFIG_FILE" ] && command -v python3 >/dev/null 2>&1; then + CONFIGURED_DIR="$(LEDMATRIX_CONFIG_FILE="$CONFIG_FILE" python3 -c ' +import json, os +try: + with open(os.environ["LEDMATRIX_CONFIG_FILE"], encoding="utf-8") as f: + value = (json.load(f).get("plugin_system") or {}).get("plugins_directory") +except Exception: + value = None +print(value if isinstance(value, str) and value.strip() else "plugin-repos") +' 2>/dev/null)" || CONFIGURED_DIR="plugin-repos" + [ -n "$CONFIGURED_DIR" ] || CONFIGURED_DIR="plugin-repos" +fi +case "$CONFIGURED_DIR" in + /*) PLUGINS_DIR="$CONFIGURED_DIR" ;; + *) PLUGINS_DIR="$LEDMATRIX_DIR/$CONFIGURED_DIR" ;; +esac +DEV_PLUGINS_DIR="$LEDMATRIX_DIR/plugins" echo "LEDMatrix directory: $LEDMATRIX_DIR" -echo "Plugins directory: $PLUGINS_DIR" +echo "Plugins directory: $PLUGINS_DIR (plugin_system.plugins_directory)" + +SCAN_DIRS=() +PLUGINS_DIR_REAL="" +if [ -d "$PLUGINS_DIR" ]; then + SCAN_DIRS+=("$PLUGINS_DIR") + PLUGINS_DIR_REAL="$(cd "$PLUGINS_DIR" && pwd -P)" +fi +if [ -d "$DEV_PLUGINS_DIR" ] && [ "$(cd "$DEV_PLUGINS_DIR" && pwd -P)" != "$PLUGINS_DIR_REAL" ]; then + echo "Also scanning dev plugins: $DEV_PLUGINS_DIR" + SCAN_DIRS+=("$DEV_PLUGINS_DIR") +fi echo "" -# Check if plugins directory exists -if [ ! -d "$PLUGINS_DIR" ]; then +# Check if a plugins directory exists +if [ ${#SCAN_DIRS[@]} -eq 0 ]; then echo -e "${RED}Error: Plugins directory not found at $PLUGINS_DIR${NC}" + echo "Install a plugin from the Plugin Store first, or check plugin_system.plugins_directory in $CONFIG_FILE" exit 1 fi @@ -47,12 +84,21 @@ echo "" PLUGINS_FOUND=0 PLUGINS_INSTALLED=0 PLUGINS_FAILED=0 +SEEN_PLUGIN_PATHS=" " -for plugin_dir in "$PLUGINS_DIR"/*/ ; do +for scan_dir in "${SCAN_DIRS[@]}"; do +for plugin_dir in "$scan_dir"/*/ ; do if [ -d "$plugin_dir" ]; then plugin_name=$(basename "$plugin_dir") requirements_file="$plugin_dir/requirements.txt" - + + # A dev symlink can point at a plugin already scanned; install it once. + real_plugin_dir="$(cd "$plugin_dir" && pwd -P)" + case "$SEEN_PLUGIN_PATHS" in + *" $real_plugin_dir "*) continue ;; + esac + SEEN_PLUGIN_PATHS="$SEEN_PLUGIN_PATHS$real_plugin_dir " + if [ -f "$requirements_file" ]; then PLUGINS_FOUND=$((PLUGINS_FOUND + 1)) echo -e "${GREEN}Found plugin: ${plugin_name}${NC}" @@ -79,6 +125,7 @@ for plugin_dir in "$PLUGINS_DIR"/*/ ; do fi fi done +done # Summary echo "" diff --git a/scripts/scroll_speeds.py b/scripts/scroll_speeds.py index e824b19f..bdabf023 100644 --- a/scripts/scroll_speeds.py +++ b/scripts/scroll_speeds.py @@ -47,52 +47,47 @@ from src.common import scroll_config # noqa: E402 CONFIG = Path(__file__).resolve().parent.parent / "config" / "config.json" -def load_hardware(): +def load_config(): + """The whole config.json, or {} when it is missing or unreadable.""" try: with open(CONFIG, encoding="utf-8") as handle: - return (json.load(handle).get("display") or {}).get("hardware") or {} + config = json.load(handle) except (OSError, ValueError): return {} + return config if isinstance(config, dict) else {} -def build_options(hardware, refresh_override=None): - from rgbmatrix import RGBMatrixOptions - - o = RGBMatrixOptions() - from src.display_geometry import ( - DEFAULT_CHAIN_LENGTH, DEFAULT_COLS, DEFAULT_PARALLEL, DEFAULT_ROWS, - ) - o.rows = int(hardware.get("rows", DEFAULT_ROWS)) - o.cols = int(hardware.get("cols", DEFAULT_COLS)) - o.chain_length = int(hardware.get("chain_length", DEFAULT_CHAIN_LENGTH)) - o.parallel = int(hardware.get("parallel", DEFAULT_PARALLEL)) - o.brightness = int(hardware.get("brightness", 80)) - o.hardware_mapping = hardware.get("hardware_mapping", "regular") - o.pwm_bits = int(hardware.get("pwm_bits", 11)) - o.pwm_dither_bits = int(hardware.get("pwm_dither_bits", 0)) - o.pwm_lsb_nanoseconds = int(hardware.get("pwm_lsb_nanoseconds", 130)) - o.led_rgb_sequence = hardware.get("led_rgb_sequence", "RGB") - o.scan_mode = int(hardware.get("scan_mode", 0)) - o.row_address_type = int(hardware.get("row_address_type", 0)) - o.multiplexing = int(hardware.get("multiplexing", 0)) - o.gpio_slowdown = int(hardware.get("gpio_slowdown", 2)) - o.limit_refresh_rate_hz = ( - int(refresh_override) if refresh_override is not None - else int(hardware.get("limit_refresh_rate_hz", 0)) - ) - return o +def hardware_of(config): + return (config.get("display") or {}).get("hardware") or {} -def open_matrix(hardware, refresh_override=None): +def build_options(config, refresh_override=None): + """The matrix options the display service would use for this config. + + Built by DisplayManager.apply_matrix_options, not a copy of it, so the + measurement and the demo drive the panel exactly as the service does + (runtime gpio_slowdown, rp1_rio, panel_type, orientation, defaults). + ``refresh_override`` replaces limit_refresh_rate_hz; 0 means uncapped. + """ + from src.display_manager import DisplayManager, RGBMatrixOptions + + options = DisplayManager.apply_matrix_options(RGBMatrixOptions(), config) + if refresh_override is not None: + options.limit_refresh_rate_hz = int(refresh_override) + return options + + +def open_matrix(config, refresh_override=None): """Construct the matrix, or explain why it will not open.""" if os.geteuid() != 0: sys.exit("this needs root for GPIO access - rerun with sudo") try: - from rgbmatrix import RGBMatrix - except ImportError: - sys.exit("rgbmatrix is not installed on this machine") + from src.display_manager import RGBMatrix + except ImportError as exc: + sys.exit("could not load the display stack ({}); is rgbmatrix " + "installed on this machine?".format(exc)) try: - return RGBMatrix(options=build_options(hardware, refresh_override)) + return RGBMatrix(options=build_options(config, refresh_override)) except Exception as exc: # pragma: no cover - hardware dependent sys.exit( "could not open the panel ({}).\n" @@ -101,7 +96,7 @@ def open_matrix(hardware, refresh_override=None): ) -def measure_refresh(hardware, seconds=6.0): +def measure_refresh(config, seconds=6.0): """Actual refresh rate, by running uncapped and timing the swaps. SwapOnVSync blocks until the panel's next refresh, so an unthrottled loop @@ -109,7 +104,7 @@ def measure_refresh(hardware, seconds=6.0): chain will really give you, as opposed to whatever limit_refresh_rate_hz optimistically asks for. """ - matrix = open_matrix(hardware, refresh_override=0) + matrix = open_matrix(config, refresh_override=0) canvas = matrix.CreateFrameCanvas() canvas = matrix.SwapOnVSync(canvas) # discard the first, it includes setup frames = 0 @@ -122,17 +117,17 @@ def measure_refresh(hardware, seconds=6.0): return measured -def demo(hardware, target, seconds): +def demo(config, target, seconds): """Scroll text at the crisp speed nearest `target`.""" from PIL import Image, ImageDraw, ImageFont from src.common.font_layout import load_truetype - hz = float(hardware.get("limit_refresh_rate_hz") or scroll_config.DEFAULT_REFRESH_HZ) + hz = scroll_config.refresh_hz_from_config(config) choice = scroll_config.solve_crisp(target, hz) print("asked for {:.0f} px/s -> {}".format(target, choice.describe())) - matrix = open_matrix(hardware) + matrix = open_matrix(config) canvas = matrix.CreateFrameCanvas() W, H = canvas.width, canvas.height @@ -195,9 +190,38 @@ def print_ladder(hz, highlight=None): ) else "" print(" " + entry.describe() + mark) print("") - print("Set one in config.json as pixels per second, e.g.") - print(' "display_options": {{"scroll_pixels_per_second": {:.0f}}}'.format( - scroll_config.solve_crisp(highlight if highlight else hz / 2, hz).pixels_per_second)) + print_config_advice(scroll_config.solve_crisp(highlight if highlight else hz / 2, hz)) + + +def config_advice(choice): + """The config that selects ``choice``, in the keys the resolver honours. + + Tickers take a ``scroll_speed`` (px per step) + ``scroll_delay`` (seconds) + pair, and scroll_config ranks that pair ABOVE ``scroll_pixels_per_second`` + -- deliberately, because some plugins give the flat key a schema default. + Many schemas default the pair too, so a flat key added by hand is usually + ignored. Advise the pair: pixels_per_frame every frame_hold/refresh + seconds is exactly the crisp speed. + """ + pair = { + "scroll_speed": choice.pixels_per_frame, + "scroll_delay": round(choice.frame_hold / choice.refresh_hz, 6), + } + scoreboard = {"scroll_speed": round(choice.pixels_per_second, 2)} + return pair, scoreboard + + +def print_config_advice(choice): + pair, scoreboard = config_advice(choice) + print("To use {:.1f} px/s, set it where the plugin keeps its scroll speed.".format( + choice.pixels_per_second)) + print("Tickers take a scroll_speed (px per step) + scroll_delay (seconds) pair:") + print(' "display_options": {}'.format(json.dumps(pair))) + print("(some plugins keep the pair at the top level or under \"display\").") + print("The pair outranks scroll_pixels_per_second, which is ignored whenever the") + print("pair is present -- and schema defaults usually put it there.") + print("Sports scoreboards take pixels per second per league instead:") + print(' "scroll_settings": {}'.format(json.dumps(scoreboard))) def main(): @@ -214,15 +238,15 @@ def main(): help="highlight the entry nearest this speed") args = ap.parse_args() - hardware = load_hardware() - configured = float(hardware.get("limit_refresh_rate_hz") or 0) + config = load_config() + configured = float(hardware_of(config).get("limit_refresh_rate_hz") or 0) if args.demo is not None: - demo(hardware, args.demo, args.seconds) + demo(config, args.demo, args.seconds) return if args.measure: - measured = measure_refresh(hardware) + measured = measure_refresh(config) print("measured panel refresh: {:.1f}Hz".format(measured)) if configured: print("configured limit_refresh_rate_hz: {:.0f}".format(configured)) diff --git a/scripts/utils/auto_update_verify.py b/scripts/utils/auto_update_verify.py index 048721a2..d75fe276 100644 --- a/scripts/utils/auto_update_verify.py +++ b/scripts/utils/auto_update_verify.py @@ -42,10 +42,30 @@ HEALTH_TIMEOUT_SECONDS = 180 #: loop look healthy between attempts, so a single "is-active" proves nothing. STABLE_SECONDS = 45 POLL_SECONDS = 5 +WEB_CHECK_TIMEOUT_SECONDS = 5 +SYSTEMCTL_QUERY_TIMEOUT_SECONDS = 10 +RESTART_TIMEOUT_SECONDS = 90 +GIT_TIMEOUT_SECONDS = 60 +GIT_RESET_TIMEOUT_SECONDS = 120 PIP_TIMEOUT_SECONDS = 600 +#: All of a rollback's dependency reinstalls together. A pip that times out +#: or fails is not retried: systemd stops this unit at TimeoutStartSec, and a +#: rollback killed half-way leaves the update reported as still verifying. +PIP_BUDGET_SECONDS = 600 #: sudoers matches the exact command line, so bash is named by path, the same -#: candidates src/common/permission_utils.install_requirements_file tries. +#: candidates src/common/permission_utils.install_requirements_file tries... BASH_CANDIDATES = ('/usr/bin/bash', '/bin/bash') +#: ...and, like it, moves to the next one only when sudo refused the command +#: line (permission_utils.SUDO_REFUSAL_PHRASES), never after pip itself ran. +SUDO_REFUSAL_PHRASES = ('a password is required', 'is not allowed to run', 'no tty present') + +#: The longest one health check can take: restart and wait, roll back +#: (diff, reset, reinstalls), restart and wait again. A wait's last poll can +#: start just before its deadline and run every query to its timeout. +_WAIT_WORST_SECONDS = (HEALTH_TIMEOUT_SECONDS + STABLE_SECONDS + WEB_CHECK_TIMEOUT_SECONDS + + 2 * SYSTEMCTL_QUERY_TIMEOUT_SECONDS + POLL_SECONDS) +WORST_CASE_SECONDS = (2 * (2 * RESTART_TIMEOUT_SECONDS + _WAIT_WORST_SECONDS) + + GIT_TIMEOUT_SECONDS + GIT_RESET_TIMEOUT_SECONDS + PIP_BUDGET_SECONDS) #: What a command that could not run at all reports: its callers only read #: these three fields, the same ones a completed subprocess has. @@ -83,7 +103,7 @@ def write_pending(path, data): def _web_responds(url=WEB_HEALTH_URL): try: - with urllib.request.urlopen(url, timeout=5) as resp: # nosec B310 - fixed loopback URL + with urllib.request.urlopen(url, timeout=WEB_CHECK_TIMEOUT_SECONDS) as resp: # nosec B310 - fixed loopback URL return resp.status == 200 except (urllib.error.URLError, OSError, ValueError): return False @@ -104,7 +124,7 @@ class Verifier: self.web_responds = web_responds self.log = log or (lambda msg: print(f'[auto-update-verify] {msg}', flush=True)) - def _run(self, args, timeout=60): + def _run(self, args, timeout=GIT_TIMEOUT_SECONDS): try: return self.run(args, cwd=str(self.project_root), capture_output=True, text=True, timeout=timeout) @@ -114,15 +134,17 @@ class Verifier: # -- services --------------------------------------------------------- def service_active(self, unit): - return self._run(['systemctl', 'is-active', unit], timeout=10).stdout.strip() == 'active' + return self._run(['systemctl', 'is-active', unit], + timeout=SYSTEMCTL_QUERY_TIMEOUT_SECONDS).stdout.strip() == 'active' def restart_count(self, unit): out = self._run(['systemctl', 'show', '-p', 'NRestarts', '--value', unit], - timeout=10).stdout.strip() + timeout=SYSTEMCTL_QUERY_TIMEOUT_SECONDS).stdout.strip() return int(out) if out.isdigit() else None def restart(self, unit): - result = self._run(['sudo', '-n', 'systemctl', 'restart', f'{unit}.service'], timeout=90) + result = self._run(['sudo', '-n', 'systemctl', 'restart', f'{unit}.service'], + timeout=RESTART_TIMEOUT_SECONDS) if result.returncode != 0: self.log(f'restarting {unit} failed: {(result.stderr or "").strip()}') return result.returncode == 0 @@ -173,16 +195,28 @@ class Verifier: changed = set(result.stdout.split()) if result.returncode == 0 else set(REQUIREMENT_FILES) return [rel for rel in REQUIREMENT_FILES if rel in changed] - def install_requirements(self, rel): + def install_requirements(self, rel, deadline=None): + """Install one requirements file through the root wrapper, by ``deadline``.""" wrapper = self.project_root / 'scripts' / 'fix_perms' / 'safe_pip_install.sh' req = self.project_root / rel if not req.exists(): return True for bash in BASH_CANDIDATES: - result = self._run(['sudo', '-n', bash, str(wrapper), str(req)], - timeout=PIP_TIMEOUT_SECONDS) + timeout = PIP_TIMEOUT_SECONDS + if deadline is not None: + timeout = min(timeout, deadline - self.clock()) + if timeout <= 0: + self.log(f'no time left to reinstall {rel}') + return False + result = self._run(['sudo', '-n', bash, str(wrapper), str(req)], timeout=timeout) if result.returncode == 0: return True + # Only a refused command line is worth the next candidate. A pip + # that ran and failed, or timed out, would just do it again. + if not any(phrase in (result.stderr or '') for phrase in SUDO_REFUSAL_PHRASES): + # Not pip's output: it can echo an index URL's credentials. + self.log(f'reinstalling {rel} failed (exit {result.returncode})') + return False return False def rollback(self, pending): @@ -191,13 +225,17 @@ class Verifier: if not old: return False, 'the commit to roll back to is unknown' requirements = self.changed_requirements(old, new) if new else list(REQUIREMENT_FILES) - # Safe to --hard: the updater refuses to run with local edits to - # tracked files, so the only thing this discards is the update. - result = self._run(['git', 'reset', '--hard', old], timeout=120) + # --hard: the updater refuses to run with local edits to tracked core + # files (web_interface/auto_update.local_changes), so outside the + # plugin folders the only thing this discards is the update. Edits + # under plugins/ and plugin-repos/, which that check leaves to the + # pull's --autostash, are reset along with it. + result = self._run(['git', 'reset', '--hard', old], timeout=GIT_RESET_TIMEOUT_SECONDS) if result.returncode != 0: return False, (f'"git reset --hard {old}" failed: ' f'{(result.stderr or result.stdout or "").strip()}') - failed = [rel for rel in requirements if not self.install_requirements(rel)] + deadline = self.clock() + PIP_BUDGET_SECONDS + failed = [rel for rel in requirements if not self.install_requirements(rel, deadline)] if failed: return True, ('reinstalling the previous dependencies from ' + ', '.join(failed) + ' failed; run Install Base Requirements from the Tools tab') diff --git a/scripts/verify_installation.sh b/scripts/verify_installation.sh index 9de30bfc..ea7b34ad 100755 --- a/scripts/verify_installation.sh +++ b/scripts/verify_installation.sh @@ -121,19 +121,24 @@ fi echo "" # 5. Check web interface +# ledmatrix-web.service runs scripts/utils/start_web_conditionally.py, which +# starts web_interface/start.py; the app binds port 5000 (web_interface/start.py). echo "=== Web Interface ===" -if [ -f "$PROJECT_ROOT/web_interface_v2.py" ]; then - check_pass "web_interface_v2.py exists" -else - check_fail "web_interface_v2.py is missing" -fi +WEB_PORT=5000 +for web_file in scripts/utils/start_web_conditionally.py web_interface/start.py web_interface/app.py; do + if [ -f "$PROJECT_ROOT/$web_file" ]; then + check_pass "$web_file exists" + else + check_fail "$web_file is missing" + fi +done # Check if web service is listening if systemctl is-active --quiet ledmatrix-web.service 2>/dev/null; then - if netstat -tuln 2>/dev/null | grep -q ":5001" || ss -tuln 2>/dev/null | grep -q ":5001"; then - check_pass "Web interface is listening on port 5001" + if netstat -tuln 2>/dev/null | grep -qE ":${WEB_PORT}([^0-9]|$)" || ss -tuln 2>/dev/null | grep -qE ":${WEB_PORT}([^0-9]|$)"; then + check_pass "Web interface is listening on port $WEB_PORT" else - check_warn "Web service is running but port 5001 may not be listening" + check_warn "Web service is running but port $WEB_PORT may not be listening" fi else check_warn "Web service is not running (cannot check port)" @@ -204,7 +209,7 @@ if [ "$ALL_PASSED" = true ]; then echo -e "${GREEN}Installation verification PASSED${NC}" echo "" echo "Next steps:" - echo "1. Access the web interface at: http://$(hostname -I | awk '{print $1}'):5001" + echo "1. Access the web interface at: http://$(hostname -I | awk '{print $1}'):$WEB_PORT" echo "2. Check service status: sudo systemctl status ledmatrix.service" echo "3. View logs: journalctl -u ledmatrix.service -f" exit 0 diff --git a/scripts/verify_web_ui.sh b/scripts/verify_web_ui.sh index 53927734..2dfc7628 100755 --- a/scripts/verify_web_ui.sh +++ b/scripts/verify_web_ui.sh @@ -8,6 +8,10 @@ echo "Web UI Verification" echo "==========================================" echo "" +# The web interface binds port 5000 (web_interface/start.py). +WEB_PORT=5000 +PORT_PATTERN=":${WEB_PORT}([^0-9]|$)" + # Colors GREEN='\033[0;32m' RED='\033[0;31m' @@ -32,25 +36,25 @@ else fi echo "" -# 2. Check if port 5001 is listening -echo "2. Checking if port 5001 is listening..." +# 2. Check if port $WEB_PORT is listening +echo "2. Checking if port $WEB_PORT is listening..." if command -v ss >/dev/null 2>&1; then - if ss -tuln 2>/dev/null | grep -q ":5001"; then - echo -e "${GREEN}✓${NC} Port 5001 is listening" + if ss -tuln 2>/dev/null | grep -qE "$PORT_PATTERN"; then + echo -e "${GREEN}✓${NC} Port $WEB_PORT is listening" echo "" - echo "Active connections on port 5001:" - ss -tuln | grep ":5001" + echo "Active connections on port $WEB_PORT:" + ss -tuln | grep -E "$PORT_PATTERN" else - echo -e "${RED}✗${NC} Port 5001 is NOT listening" + echo -e "${RED}✗${NC} Port $WEB_PORT is NOT listening" fi elif command -v netstat >/dev/null 2>&1; then - if netstat -tuln 2>/dev/null | grep -q ":5001"; then - echo -e "${GREEN}✓${NC} Port 5001 is listening" + if netstat -tuln 2>/dev/null | grep -qE "$PORT_PATTERN"; then + echo -e "${GREEN}✓${NC} Port $WEB_PORT is listening" echo "" - echo "Active connections on port 5001:" - netstat -tuln | grep ":5001" + echo "Active connections on port $WEB_PORT:" + netstat -tuln | grep -E "$PORT_PATTERN" else - echo -e "${RED}✗${NC} Port 5001 is NOT listening" + echo -e "${RED}✗${NC} Port $WEB_PORT is NOT listening" fi else echo -e "${YELLOW}⚠${NC} Cannot check port (ss/netstat not available)" @@ -59,15 +63,15 @@ echo "" # 3. Test HTTP connection echo "3. Testing HTTP connection..." -if curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:5001 > /dev/null 2>&1; then - HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:5001 2>/dev/null) +if curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:$WEB_PORT > /dev/null 2>&1; then + HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:$WEB_PORT 2>/dev/null) if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "302" ] || [ "$HTTP_CODE" = "301" ]; then echo -e "${GREEN}✓${NC} Web interface is responding (HTTP $HTTP_CODE)" else echo -e "${YELLOW}⚠${NC} Web interface responded with HTTP $HTTP_CODE" fi else - echo -e "${RED}✗${NC} Cannot connect to web interface on port 5001" + echo -e "${RED}✗${NC} Cannot connect to web interface on port $WEB_PORT" fi echo "" @@ -79,7 +83,7 @@ if [ -n "$IP_ADDRESSES" ]; then echo "" echo "Access web interface at:" for ip in $IP_ADDRESSES; do - echo " http://$ip:5001" + echo " http://$ip:$WEB_PORT" done else echo -e "${YELLOW}⚠${NC} Could not determine IP address" @@ -118,12 +122,12 @@ if systemctl is-active --quiet ledmatrix-web.service 2>/dev/null; then SERVICE_RUNNING=true fi -if (ss -tuln 2>/dev/null | grep -q ":5001") || (netstat -tuln 2>/dev/null | grep -q ":5001"); then +if (ss -tuln 2>/dev/null | grep -qE "$PORT_PATTERN") || (netstat -tuln 2>/dev/null | grep -qE "$PORT_PATTERN"); then PORT_LISTENING=true fi -if curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:5001 > /dev/null 2>&1; then - HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:5001 2>/dev/null) +if curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:$WEB_PORT > /dev/null 2>&1; then + HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://localhost:$WEB_PORT 2>/dev/null) if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "302" ] || [ "$HTTP_CODE" = "301" ]; then HTTP_RESPONDING=true fi @@ -134,7 +138,7 @@ if [ "$SERVICE_RUNNING" = true ] && [ "$PORT_LISTENING" = true ] && [ "$HTTP_RES echo "" echo "You can access it at:" for ip in $IP_ADDRESSES; do - echo " http://$ip:5001" + echo " http://$ip:$WEB_PORT" done exit 0 elif [ "$SERVICE_RUNNING" = false ]; then @@ -145,7 +149,7 @@ elif [ "$SERVICE_RUNNING" = false ]; then echo " sudo systemctl enable ledmatrix-web.service # to start on boot" exit 1 elif [ "$PORT_LISTENING" = false ]; then - echo -e "${RED}✗ Service is running but port 5001 is not listening${NC}" + echo -e "${RED}✗ Service is running but port $WEB_PORT is not listening${NC}" echo "" echo "Check logs for errors:" echo " sudo journalctl -u ledmatrix-web.service -f" diff --git a/src/background_data_service.py b/src/background_data_service.py index 148232cb..de435147 100644 --- a/src/background_data_service.py +++ b/src/background_data_service.py @@ -25,7 +25,14 @@ from enum import Enum import queue from concurrent.futures import ThreadPoolExecutor from src.cache_manager import CacheManager -from src.common.espn_dates import clamp_espn_limit, fetch_espn_date_chunks +from src.common.espn_dates import ( + RANGE_RETRY_SECONDS, + _note_range_rejected, + _ranges_known_rejected, + clamp_espn_limit, + fetch_espn_date_chunks, + parse_espn_date_range, +) # Configure logging logger = logging.getLogger(__name__) @@ -350,19 +357,38 @@ class BackgroundDataService: logger.info(f"Starting background fetch for {request.sport} {request.year}") - # Perform HTTP request with retry logic - response = self._make_request_with_retry(request) - # ESPN stopped accepting dates=YYYYMMDD-YYYYMMDD on 2026-09-15 and # answers 400 for every sport. Re-ask in months and days rather # than let a whole season fail. See src/common/espn_dates.py. - if response.status_code == 400: + # The "ranges are rejected" memo is shared with + # fetch_espn_scoreboard(): once either path has seen a range + # rejected, the other skips the doomed range request too. + is_range = parse_espn_date_range(request.params.get("dates")) is not None + data = None + chunks_tried = False + if is_range and _ranges_known_rejected(): data = self._fetch_in_date_chunks(request) - if data is None: + # Every chunk failed: ask for the range itself below so the + # failure carries a real HTTP error, without re-spending chunks. + chunks_tried = data is None + + if data is None: + # Perform HTTP request with retry logic + response = self._make_request_with_retry(request) + if is_range and response.status_code == 400 and not chunks_tried: + _note_range_rejected() + logger.warning( + "ESPN rejected the date range %s (400); fetching it as " + "month/day chunks, and fetching ranges that way for the " + "next %d hours", + request.params.get("dates"), RANGE_RETRY_SECONDS // 3600, + ) + data = self._fetch_in_date_chunks(request) + if data is None: + response.raise_for_status() + else: response.raise_for_status() - else: - response.raise_for_status() - data = response.json() + data = response.json() # Validate data structure if not isinstance(data, dict): diff --git a/src/common/permission_utils.py b/src/common/permission_utils.py index 29be2f04..f59b31cc 100644 --- a/src/common/permission_utils.py +++ b/src/common/permission_utils.py @@ -373,6 +373,14 @@ def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> b return False +#: What sudo prints when it refuses a command line outright (not in sudoers for +#: that exact argv, or it wants a password). Only then is another bash path +#: worth trying: after pip itself ran, a retry just repeats the failure. The +#: automatic update's health check keeps its own copy of this list +#: (scripts/utils/auto_update_verify.py runs without importing src/). +SUDO_REFUSAL_PHRASES = ("a password is required", "is not allowed to run", "no tty present") + + def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess.CompletedProcess: """ Install a requirements.txt file for a plugin (or the project itself). @@ -434,10 +442,7 @@ def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess. # Distinguish "sudo rejected this exact command line" (worth # trying the next bash candidate) from "sudo ran it but pip # itself failed" (a real error — stop and surface it). - denied = any( - phrase in result.stderr - for phrase in ("a password is required", "is not allowed to run", "no tty present") - ) + denied = any(phrase in result.stderr for phrase in SUDO_REFUSAL_PHRASES) if not denied: # Deliberately don't interpolate req_file or the pip output here: # this log line is scanner-visible, and a static analyzer can't diff --git a/src/common/scroll_config.py b/src/common/scroll_config.py index 4dd7f532..5719f84c 100644 --- a/src/common/scroll_config.py +++ b/src/common/scroll_config.py @@ -22,12 +22,27 @@ shimmer) or repeat frames (which reads as judder). :func:`resolve` warns when the requested speed will not divide evenly, because that is a real display artefact and not a rounding detail. -Speed is always expressed to the helper as pixels per second and applied in -time-based mode. Frame-based stepping gates motion on a wall clock at -``1/scroll_delay`` steps per second; plugins set ``scroll_delay`` to the frame -period, which puts that comparison exactly on its own threshold and makes the -step count flip on sub-millisecond jitter. Accumulating elapsed time keeps -position proportional to real time instead. +How the speed is applied +------------------------ +:func:`configure` snaps the requested speed to a :class:`CrispSpeed` -- a whole +number of pixels per presented frame, with each frame held for ``frame_hold`` +panel refreshes -- and puts the helper in fixed-step mode +(``ScrollHelper.set_pixels_per_frame``). In that mode the helper consults no +clock: every ``update_scroll_position()`` call moves exactly that many pixels. +``SwapOnVSync`` blocks until the panel has taken each frame, so the frame count +is the clock, and the speed on the panel is:: + + px/s = refresh_hz / frame_hold * pixels_per_frame + +That makes the hold part of the speed. The caller must pass +``settings.frame_hold`` to ``display_manager.set_scrolling_state(True, ...)``; +a caller that omits it is presented every refresh and scrolls ``frame_hold`` +times too fast. The refresh is the panel's (``display_manager.refresh_hz``, +i.e. ``display.hardware.limit_refresh_rate_hz``) -- never the global +``target_fps``, which no longer paces anything. + +With ``snap_to_crisp=False`` there is no fixed step: the helper is set to +px/s and advances by elapsed time, and the hold is 1. """ from __future__ import annotations @@ -330,17 +345,18 @@ def configure( ) -> ScrollSettings: """Resolve the config and apply it to ``scroll_helper``. - Applied in time-based mode: see the module docstring for why frame-based - stepping is not used. ``hasattr`` guards keep this usable against older - ScrollHelper builds that a plugin may be running on. + Frame-based mode is switched off, the (snapped) speed is set, and with + ``snap_to_crisp`` the helper steps a fixed whole-pixel amount per presented + frame -- see the module docstring. ``hasattr`` guards keep this usable + against older ScrollHelper builds that a plugin may be running on. :param display_manager: consulted for the panel's refresh rate only (it can see display.hardware; a plugin cannot). The frame hold is NOT applied here -- see the note in the body. The caller must pass ``settings.frame_hold`` to ``display_manager.set_scrolling_state(True, - ...)`` when it starts scrolling, or a sub-refresh speed still presents - a new frame every refresh and the motion falls back to fractional - pixels. + ...)`` when it scrolls. The helper moves its fixed step on every call, + so without the hold each step is presented every refresh and the + scroll runs ``frame_hold`` times faster than the resolved speed. :param snap_to_crisp: move the requested speed to the nearest speed the panel can show in whole pixels. On by default because a speed that does not divide evenly has no good rendering, only a choice of artefacts. @@ -357,7 +373,8 @@ def configure( # refresh to fill in target_fps, pixels_per_frame and the judder warning, # so deriving it afterwards described a 100Hz panel to everyone running at # 60 -- and with snap_to_crisp=False nothing downstream corrected it, so - # set_target_fps() paced the helper to 100 FPS on a 60Hz panel. + # the settings and the helper's (informational) target_fps said 100 FPS on + # a 60Hz panel. hz = _coerce(refresh_hz) if hz is None and display_manager is not None: hz = _coerce(getattr(display_manager, "refresh_hz", None)) @@ -399,6 +416,9 @@ def configure( if hasattr(scroll_helper, "set_pixels_per_frame"): scroll_helper.set_pixels_per_frame( choice.pixels_per_frame if choice else None) + # Informational only: nothing in the helper paces off target_fps. It is + # still recorded because plugins read it back (ledmatrix-elections' + # test_scroll_pacing.py asserts it equals the crisp presentation rate). if choice and hasattr(scroll_helper, "set_target_fps"): scroll_helper.set_target_fps(choice.frames_per_second) elif settings.target_fps and hasattr(scroll_helper, "set_target_fps"): diff --git a/src/common/scroll_helper.py b/src/common/scroll_helper.py index 71b9080a..73c6e1e8 100644 --- a/src/common/scroll_helper.py +++ b/src/common/scroll_helper.py @@ -22,13 +22,6 @@ from typing import Optional, Dict, Any from PIL import Image import numpy as np -# Try to import scipy for sub-pixel interpolation, fallback to simpler method if not available -try: - from scipy.ndimage import shift - HAS_SCIPY = True -except ImportError: - HAS_SCIPY = False - # How often the frame-stats line is emitted, and therefore also the ceiling # on a believable frame time: a scroll that renders at all cannot take this @@ -138,21 +131,22 @@ class ScrollHelper: # blend (blur) or repeat frames (judder); blending is the worse of the # two here. Vegas mode still opts in via set_sub_pixel_scrolling(). self.sub_pixel_scrolling = False - self._last_integer_position = 0 # Cache for integer position to avoid repeated calculations - + # Frame-based scrolling settings self.frame_based_scrolling = False #: Whole pixels to advance per presented frame, or None to pace #: off elapsed time. See set_pixels_per_frame. - self.fixed_pixels_per_frame = None # If True, use scroll_delay to throttle and move scroll_speed pixels - self.last_step_time = 0.0 # Track last step time for frame-based throttling - + self.fixed_pixels_per_frame = None + self.last_step_time = 0.0 # Time of the last position update + # Time tracking for scroll updates self.last_update_time: Optional[float] = None - - # High FPS settings - self.target_fps = 120 # Target 120 FPS for smooth scrolling - self.frame_time_target = 1.0 / self.target_fps + + #: Informational only: the presentation rate scroll_config chose + #: (panel refresh / frame hold). Nothing paces off it -- the helper + #: steps per call and SwapOnVSync paces the calls. Kept because + #: plugins and their tests read it back. + self.target_fps = 120 # Dynamic duration settings self.dynamic_duration_enabled = True @@ -288,7 +282,14 @@ class ScrollHelper: def update_scroll_position(self) -> None: """ - Update scroll position with high FPS control and handle wrap-around. + Advance the scroll by one presented frame and handle wrap-around. + + With a fixed per-frame step (set_pixels_per_frame, which + scroll_config.configure sets for a crisp speed) every call moves + exactly that many pixels and no clock is read; the caller's + vsync-blocking swap, held for ``frame_hold`` refreshes, sets the rate. + Otherwise the position advances by elapsed time at the configured + speed. """ if not self.cached_image: return @@ -461,10 +462,9 @@ class ScrollHelper: Linear blend between the frames at ``start_x`` and ``start_x + 1``. Implemented with numpy rather than scipy.ndimage.shift: scipy is not - installed on the target devices (HAS_SCIPY is False there), which is why - the pre-existing sub-pixel path was dead code — get_visible_portion never - consulted the flag, and the scipy fallback would not have interpolated - anyway. + installed on the target devices, and the old scipy-based sub-pixel path + was dead code -- get_visible_portion never consulted the flag. The scipy + import was removed with it; installing scipy has no effect. Args: start_x: Left column of the earlier of the two frames @@ -830,11 +830,13 @@ class ScrollHelper: def set_scroll_speed(self, speed: float) -> None: """ - Set the scroll speed. - - In time-based mode: pixels per second (typically 10-200) - In frame-based mode: pixels per frame (typically 0.5-5 for smooth scrolling) - + Set the scroll speed, and leave fixed-step mode. + + In time-based mode: pixels per second (clamped to 1-500). + In frame-based mode: pixels per ``scroll_delay`` seconds (clamped to + 0.1-5), still applied by elapsed time as scroll_speed / scroll_delay + px/s. + Args: speed: Scroll speed (interpretation depends on frame_based_scrolling mode) """ @@ -896,14 +898,19 @@ class ScrollHelper: def set_target_fps(self, fps: float) -> None: """ - Set the target frames per second for scrolling. - + Record the presentation rate, for diagnostics only. + + Nothing paces off this value: with a fixed per-frame step the helper + advances once per call, and without one it advances by elapsed time. + The rate frames are shown at is the panel refresh divided by the frame + hold passed to ``display_manager.set_scrolling_state``. scroll_config + sets it to that rate so it can be read back. + Args: - fps: Target FPS (typically 30-200, default 120) + fps: Frames per second (clamped to 30-200) """ self.target_fps = max(30.0, min(200.0, fps)) - self.frame_time_target = 1.0 / self.target_fps - self.logger.debug(f"Target FPS set to: {self.target_fps} FPS (frame_time_target: {self.frame_time_target:.4f}s)") + self.logger.debug("Target FPS recorded: %s FPS (informational)", self.target_fps) def set_sub_pixel_scrolling(self, enabled: bool) -> None: """ @@ -914,7 +921,7 @@ class ScrollHelper: When disabled, uses integer pixel positioning (faster but may skip pixels). Args: - enabled: True to enable sub-pixel scrolling (default: True) + enabled: True to enable sub-pixel scrolling (default: False) """ self.sub_pixel_scrolling = enabled self.logger.debug(f"Sub-pixel scrolling {'enabled' if enabled else 'disabled'}") @@ -923,10 +930,12 @@ class ScrollHelper: """ Enable or disable frame-based scrolling. - When enabled, update_scroll_position() respects scroll_delay and moves - scroll_speed pixels per step. This provides a "stepped" look similar to - traditional tickers and can be visually smoother on LED matrices. - + This does not step. When enabled, ``scroll_speed`` is read as pixels + per ``scroll_delay`` seconds (set_scroll_speed clamps it to 0.1-5), and + update_scroll_position() still advances by elapsed time at + ``scroll_speed / scroll_delay`` px/s. A fixed per-frame step set by + set_pixels_per_frame() takes precedence over both modes. + Args: enabled: True to enable frame-based scrolling (default: False) """ @@ -985,11 +994,7 @@ class ScrollHelper: # the real stall rates being measured, so the number could not be # trusted at all. Seed the clock and take no sample. if self.last_frame_time is None: - self.last_frame_time = current_time - # Restart the window with the scroll. Otherwise the boundary is - # already long overdue when the second frame arrives, and the new - # scroll opens by reporting a window of exactly one frame. - self.last_fps_log_time = current_time + self._restart_stats_window(current_time) return # Calculate instantaneous frame time @@ -998,9 +1003,10 @@ class ScrollHelper: # A caller that scrolls without ever calling reset_scroll() never arms # the sentinel above, so catch the same gap by its size. Nothing that # renders a scroll produces a frame longer than the log interval; a - # sample that large is an idle period, not a frame. + # sample that large is an idle period, not a frame. It starts a new + # scroll exactly as the sentinel does, window timer included. if frame_time >= FPS_LOG_INTERVAL: - self.last_frame_time = current_time + self._restart_stats_window(current_time) return self.frame_times.append(frame_time) @@ -1034,7 +1040,17 @@ class ScrollHelper: self.last_frame_time = current_time self.frame_count += 1 - + + def _restart_stats_window(self, current_time: float) -> None: + """Seed the frame clock at the start of a scroll, taking no sample. + + The window timer restarts with it. Otherwise the 5s boundary is + already long overdue when the next frame arrives, and the new scroll + opens by reporting a window of exactly one frame. + """ + self.last_frame_time = current_time + self.last_fps_log_time = current_time + def clear_cache(self) -> None: """ Clear the cached scrolling image. diff --git a/src/common/sports_scroll.py b/src/common/sports_scroll.py index 0278a1d3..d32fa76a 100644 --- a/src/common/sports_scroll.py +++ b/src/common/sports_scroll.py @@ -18,10 +18,15 @@ Promoting the content layer would be exactly the mistake ``docs/SPORTS_UNIFICATION.md`` warns against — merging on the intuition that same-named methods are the same method. Same name, different job. -The one behavior this module adds over the plugin copies is native support for -``global_config['target_fps']``: the bundled copies hardcode ~100 FPS via -``scroll_delay=0.01`` and never consult the global smooth-scrolling target. A -plugin inheriting from here gets it for free. +What this module adds over the plugin copies is pacing through +:mod:`src.common.scroll_config`, the resolver every other scroller uses: the +configured px/s is snapped to a whole-pixel speed for the panel's refresh, the +helper steps a fixed number of pixels per presented frame, and each drawn frame +publishes the frame hold to the display manager. Speed depends only on +``scroll_speed`` and the panel refresh +(``display.hardware.limit_refresh_rate_hz``) -- not on the global +``target_fps``, and not on ``scroll_delay``, which is kept in the settings for +compatibility only. Usage:: @@ -94,9 +99,9 @@ class SportsScrollDisplay: :param display_manager: the core display manager :param config: the plugin's configuration :param custom_logger: the plugin's logger, so scroll lines are attributed - :param global_config: the LEDMatrix global config — the source of - ``target_fps``. Optional so an older caller that does not pass it - keeps working at the config-derived pacing. + :param global_config: the LEDMatrix global config, consulted for the + panel refresh when the display manager cannot report one. + Optional so an older caller that does not pass it keeps working. """ self.display_manager = display_manager self.config = config @@ -196,21 +201,6 @@ class SportsScrollDisplay: return {**settings, **override} return settings - def _resolve_target_fps(self) -> Optional[float]: - """The global smooth-scrolling FPS target, or None to keep config pacing. - - Coerced before use: a malformed value in the global config must degrade - to the existing ``scroll_delay`` pacing, never raise on a display path. - """ - raw = self.global_config.get("target_fps") or self.global_config.get( - "scroll_target_fps" - ) - try: - return float(raw) if raw is not None else None - except (TypeError, ValueError): - self.logger.debug("Ignoring unusable target_fps: %r", raw) - return None - def _coerce_float(self, value: Any, default: float) -> float: """A usable float from config, or ``default``. @@ -246,8 +236,8 @@ class SportsScrollDisplay: # settings dict: the two read the same key names with different # meanings, and the collision is a factor of 1/scroll_delay. # - # sports_scroll: scroll_speed is px/SECOND; scroll_delay is only the - # frame period used to reach px/frame. + # sports_scroll: scroll_speed is px/SECOND; scroll_delay is ignored + # for pacing (see _resolve_pixels_per_second). # scroll_config: scroll_speed is px per STEP, so px/s = speed/delay. # # Passing {"scroll_speed": 50.0, "scroll_delay": 0.01} straight through @@ -276,8 +266,13 @@ class SportsScrollDisplay: def _resolve_pixels_per_second(self, settings: Dict[str, Any]) -> float: """This module's config shape, expressed as plain pixels per second. - ``scroll_speed`` is already px/s here. ``scroll_delay`` only matters - when a caller supplied px/frame instead, which the 0 case covers. + ``scroll_speed`` is already px/s here, and it is the whole answer. + ``scroll_delay`` is kept in the settings for compatibility but is + ignored for pacing: the frame rate is the panel refresh divided by the + frame hold scroll_config picks, so no positive delay changes the speed. + The one exception is a delay of exactly 0 (below every scoreboard + schema's minimum), which is read as ``scroll_speed`` being px per + frame at ``ASSUMED_FPS_WHEN_UNPACED``. """ scroll_speed = self._coerce_float(settings.get("scroll_speed"), 50.0) scroll_delay = self._coerce_float(settings.get("scroll_delay"), 0.01) @@ -285,19 +280,29 @@ class SportsScrollDisplay: return scroll_speed * ASSUMED_FPS_WHEN_UNPACED return scroll_speed - def _resolve_refresh_hz(self) -> Optional[float]: + def _resolve_refresh_hz(self) -> float: """The panel refresh the crisp ladder should be computed against. - Prefers the configured hardware refresh. Falls back to the global - ``target_fps``/``scroll_target_fps`` this module has always honoured: - under the old model that key *was* the rate frames were presented at, - so it is the faithful translation for anyone who set it. Returning - None lets scroll_config apply its own default. + This has to be the rate frames are actually presented at, because the + helper advances a fixed number of whole pixels per presented frame and + the display manager holds each frame for ``frame_hold`` refreshes. A + ladder built for any other rate plays back at the wrong speed: built + for 60 Hz and shown on a 100 Hz panel, it runs 100/60 too fast. + + So it is the display manager's ``refresh_hz`` (what the panel is + driven at), then ``display.hardware.limit_refresh_rate_hz`` from the + global config, then scroll_config's default. The global ``target_fps`` + is deliberately NOT consulted: before frame-locked presentation it was + the rate frames were shown at, but now honouring it only turned the + General tab's "Scroll Frame Rate" into a scoreboard speed multiplier. """ - hardware = scroll_config.refresh_hz_from_config(self.global_config) - if hardware and hardware != scroll_config.DEFAULT_REFRESH_HZ: - return hardware - return self._resolve_target_fps() or hardware or None + reported = getattr(self.display_manager, "refresh_hz", None) + # A real number only: a stub or a mock answering for anything must not + # become the refresh rate. + if (isinstance(reported, (int, float)) and not isinstance(reported, bool) + and reported > 0): + return float(reported) + return scroll_config.refresh_hz_from_config(self.global_config) def _scroll_frame_hold(self) -> int: """Refreshes to hold each frame for, from the resolved settings.""" @@ -327,11 +332,12 @@ class SportsScrollDisplay: return False # Tell the core the panel is scrolling, and for how many - # refreshes to hold each frame. Without this the frame hold is - # never applied -- so a speed the ladder made crisp still presents - # a new frame every refresh and judders -- and, because deferred - # updates only run while nothing is scrolling, core would run - # blocking work in the middle of this scroll. + # refreshes to hold each frame. The helper advances a fixed number + # of whole pixels per presented frame, so without the hold a new + # frame is presented every refresh and the scroll runs frame_hold + # times too fast. And because deferred updates only run while + # nothing is scrolling, core would otherwise run blocking work in + # the middle of this scroll. if hasattr(self.display_manager, "set_scrolling_state"): self.display_manager.set_scrolling_state( True, frame_hold=self._scroll_frame_hold()) diff --git a/src/config_manager.py b/src/config_manager.py index 453e73b1..f544f35c 100644 --- a/src/config_manager.py +++ b/src/config_manager.py @@ -29,6 +29,7 @@ import os import logging from pathlib import Path from typing import Dict, Any, Optional, List +from src.core_config_keys import CORE_CONFIG_KEYS, CORE_SECRETS_KEYS from src.exceptions import ConfigError from src.logging_config import get_logger from src.config_manager_atomic import ( @@ -756,12 +757,13 @@ class ConfigManager: valid_set = set(valid_plugin_ids) - # Find orphaned plugins in main config - main_plugins = set(main_config.keys()) + # Find orphaned plugins in main config. Core sections (display, + # schedule, auto_update, ...) are not plugins and never orphans. + main_plugins = set(main_config.keys()) - CORE_CONFIG_KEYS orphaned_main = main_plugins - valid_set # Find orphaned plugins in secrets config - secrets_plugins = set(secrets_config.keys()) + secrets_plugins = set(secrets_config.keys()) - CORE_CONFIG_KEYS - CORE_SECRETS_KEYS orphaned_secrets = secrets_plugins - valid_set all_orphaned = orphaned_main | orphaned_secrets @@ -815,8 +817,8 @@ class ConfigManager: if not isinstance(plugin_config, dict): continue - # Skip non-plugin config sections - if plugin_id in ['display', 'schedule', 'timezone', 'plugin_system']: + # Skip core config sections + if plugin_id in CORE_CONFIG_KEYS: continue schema = plugin_schema_manager.load_schema(plugin_id, use_cache=True) diff --git a/src/core_config_keys.py b/src/core_config_keys.py index d68e785f..8d8a07fa 100644 --- a/src/core_config_keys.py +++ b/src/core_config_keys.py @@ -41,3 +41,13 @@ CORE_CONFIG_KEYS = frozenset({ 'vegas_excluded_plugins', 'vegas_scroll_enabled', }) + +#: Top-level keys of ``config_secrets.json`` that belong to the core rather than +#: to a plugin: the GitHub token the Plugin Store reads, and the historical +#: ``youtube`` section. Plugin secrets are namespaced by plugin id, so anything +#: deciding whether a secrets section is a plugin's needs this as well as +#: ``CORE_CONFIG_KEYS``. +CORE_SECRETS_KEYS = frozenset({ + 'github', + 'youtube', +}) diff --git a/src/display_controller.py b/src/display_controller.py index 0434528e..7f5d7a66 100644 --- a/src/display_controller.py +++ b/src/display_controller.py @@ -2976,6 +2976,14 @@ class DisplayController: _pid: str = plugin_id, _plugin: Any = plugin_instance) -> None: """Callback for plugin config changes.""" try: + # ConfigService hands over the raw config.json section. + # Prepare it as loading did (legacy booleans read as + # objects, schema defaults filled in), so the plugin gets + # the same shape it was constructed with. + prepare = getattr(self.plugin_manager, 'prepare_plugin_config', None) + prepared = prepare(_pid, new_config) if callable(prepare) else None + if isinstance(prepared, dict): + new_config = prepared _plugin.on_config_change(new_config) logger.debug("Plugin %s notified of config change", _pid) except Exception as e: diff --git a/src/display_geometry.py b/src/display_geometry.py index b74cb361..b5655c12 100644 --- a/src/display_geometry.py +++ b/src/display_geometry.py @@ -1,17 +1,25 @@ -"""Display size from config: the one computation every caller shares. +"""Display size from config: the one computation the size readers share. ``DisplayManager`` sizes its canvas from ``display.hardware`` plus -``display.double_sided``. The web preview, the Starlark magnify default and -the multi-display sync handshake used to re-derive that size themselves, -each with its own defaults (``chain_length`` fell back to 2 in one place and -1 in three others) and none of them applying double-sided mode. They now all -call this module. +``display.double_sided``. The web preview endpoints (``/display/current``, the +SSE fallback), the Starlark magnify default and ``scripts/dev/vegas_audit.py`` +used to re-derive that size themselves, each with its own defaults +(``chain_length`` fell back to 2 in one place and 1 in others) and none of +them applying double-sided mode. They now call this module. (The multi-display +sync handshake doesn't compute a size; it shares only +``DEFAULT_CHAIN_LENGTH``.) + +The size includes what the rgbmatrix library's pixel mappers do to it -- +``orientation`` and ``pixel_mapper_config`` (``Rotate:90`` swaps the axes, +``U-mapper`` folds the chain) -- because ``RGBMatrix.width``/``height``, which +``DisplayManager`` reports on hardware, are measured after them. Kept free of hardware imports on purpose: the web interface imports it, and ``display_manager`` pulls in ``rgbmatrix``. """ import logging +import re from typing import Any, Dict, Mapping, Optional, Tuple logger = logging.getLogger(__name__) @@ -22,6 +30,10 @@ DEFAULT_COLS = 64 DEFAULT_CHAIN_LENGTH = 2 DEFAULT_PARALLEL = 1 +#: ``display.hardware.orientation`` -> degrees of the ``Rotate`` mapper +#: DisplayManager appends to ``pixel_mapper_config`` (None: no mapper). +ORIENTATION_ROTATE_DEGREES = {'normal': None, '90': 90, '180': 180, '270': 270} + def _display(config: Optional[Mapping[str, Any]]) -> Mapping[str, Any]: # A hand-edited config.json can hold anything here; treat a non-mapping @@ -35,10 +47,128 @@ def _hardware(config: Optional[Mapping[str, Any]]) -> Mapping[str, Any]: return hw if isinstance(hw, Mapping) else {} +def compose_pixel_mapper_config(hardware: Mapping[str, Any]) -> str: + """The ``pixel_mapper_config`` DisplayManager hands the library. + + ``pixel_mapper_config`` stays a free-form advanced field (e.g. "U-mapper" + for chain layouts); ``orientation`` is the user-facing mounting rotation, + appended as a trailing ``Rotate:`` mapper rather than overwriting it. + """ + base = hardware.get('pixel_mapper_config') or '' + base = base.strip() if isinstance(base, str) else '' + degrees = ORIENTATION_ROTATE_DEGREES.get(hardware.get('orientation', 'normal')) + if degrees is None: + return base + rotate = f'Rotate:{degrees}' + return f'{base};{rotate}' if base else rotate + + +def _c_div(a: int, b: int) -> int: + """C integer division (truncates toward zero).""" + q = abs(a) // abs(b) + return q if (a >= 0) == (b >= 0) else -q + + +def _c_strtol(text: str) -> Tuple[int, str]: + """strtol(text, &end, 10): the parsed value (0 if none) and the rest.""" + match = re.match(r'\s*([+-]?\d+)', text) + if not match: + return 0, text + return int(match.group(1)), text[match.end():] + + +def _remap_size(param: Optional[str], width: int, height: int, + chain: int, parallel: int) -> Optional[Tuple[int, int]]: + """RemapMapper::SetParameters and GetSizeMapping (lib/pixel-mapper.cc).""" + if not param: + return None + new_w, rest = _c_strtol(param) + if not rest.startswith(','): + return None + new_h, rest = _c_strtol(rest[1:]) + if not rest.startswith('|'): + return None + rest = rest[1:] + tiles = [] + while rest: + x, rest = _c_strtol(rest) + if not rest.startswith(','): + return None + y, rest = _c_strtol(rest[1:]) + if not rest or rest[0].lower() not in 'neswx': + return None + tiles.append((x, y, rest[0].lower())) + rest = rest[1:] + if rest.startswith('|'): + rest = rest[1:] + elif rest: + return None + if len(tiles) != chain * parallel: + return None + panel_w, panel_h = _c_div(width, chain), _c_div(height, parallel) + for x, y, kind in tiles: + if kind == 'x': + continue + # MapTile::MapToVisible of the panel's (0, 0) and far corner. + x0, y0, x1, y1 = { + 'n': (x, y, x + panel_w - 1, y + panel_h - 1), + 'w': (x, y + panel_w - 1, x + panel_h - 1, y), + 's': (x + panel_w - 1, y + panel_h - 1, x, y), + 'e': (x + panel_h - 1, y, x, y + panel_w - 1), + }[kind] + if x1 < 0 or x0 >= new_w or y1 < 0 or y0 >= new_h: + return None + return new_w, new_h + + +def apply_pixel_mappers(width: int, height: int, mapper_config: str, + chain: int, parallel: int) -> Tuple[int, int]: + """The canvas size after the library applies ``mapper_config``. + + Mirrors ``RGBMatrix::Impl::ApplyNamedPixelMappers`` and each built-in + mapper's ``SetParameters``/``GetSizeMapping`` in the pinned + ``lib/pixel-mapper.cc``: mappers apply left to right, and one the library + doesn't know or can't configure is skipped and leaves the size alone. + ``multiplexing`` isn't modelled: its mappers give back the configured + size for the panel sizes they are made for. + """ + for entry in (mapper_config or '').split(';'): + name, colon, param = entry.partition(':') + name = name.lower() + param = param if colon else None + if name == 'rotate': + if not param: + continue + angle, rest = _c_strtol(param) + if rest or angle % 90: + continue + if angle % 180: + width, height = height, width + elif name == 'u-mapper': + if chain < 2 or chain % 2 or height % parallel: + continue + width, height = _c_div(width, 64) * 32, 2 * height + elif name == 'v-mapper': + width, height = (_c_div(width * parallel, chain), + _c_div(height * chain, parallel)) + elif name == 'stacktorow': + if param and any(c not in 'ZzFf, ' for c in param): + continue + width, height = width * parallel, _c_div(height, parallel) + elif name == 'remap': + size = _remap_size(param, width, height, chain, parallel) + if size is not None: + width, height = size + # "mirror" keeps the size; the library skips names it doesn't know. + return width, height + + def physical_size(config: Optional[Mapping[str, Any]]) -> Tuple[int, int]: """Width and height of the whole panel chain, in pixels. - ``cols * chain_length`` by ``rows * parallel``. Raises ``ValueError`` or + ``cols * chain_length`` by ``rows * parallel``, then through the pixel + mappers ``orientation`` and ``pixel_mapper_config`` set up -- what + ``RGBMatrix.width``/``height`` report. Raises ``ValueError`` or ``TypeError`` on a non-numeric value, as ``DisplayManager`` does; callers decide their own fallback. @@ -54,7 +184,11 @@ def physical_size(config: Optional[Mapping[str, Any]]) -> Tuple[int, int]: parallel = int(hw.get('parallel', DEFAULT_PARALLEL)) except OverflowError as e: raise ValueError(f"display.hardware size is not finite: {e}") from e - return max(1, cols * chain_length), max(1, rows * parallel) + width, height = max(1, cols * chain_length), max(1, rows * parallel) + if chain_length >= 1 and parallel >= 1: + width, height = apply_pixel_mappers( + width, height, compose_pixel_mapper_config(hw), chain_length, parallel) + return max(1, width), max(1, height) def resolve_double_sided(physical_width: int, physical_height: int, diff --git a/src/display_manager.py b/src/display_manager.py index a0b105fa..6c01dc55 100644 --- a/src/display_manager.py +++ b/src/display_manager.py @@ -37,9 +37,11 @@ from PIL import Image, ImageDraw, ImageFont from src.common.font_layout import crisp_size, load_truetype, resolve_asset_path from src.display_geometry import ( DEFAULT_CHAIN_LENGTH, DEFAULT_COLS, DEFAULT_PARALLEL, DEFAULT_ROWS, - physical_size, resolve_double_sided, + ORIENTATION_ROTATE_DEGREES, compose_pixel_mapper_config, physical_size, + resolve_double_sided, ) -from src.pi5_matrix_support import is_raspberry_pi_5, pi5_unsupported_settings +from src.matrix_support import MatrixSettingsRefused, library_refusals, refusal_message +from src.pi5_matrix_support import is_raspberry_pi_5 import threading import time from collections import OrderedDict @@ -91,9 +93,11 @@ class _LogicalMatrix: """Proxy that reports a logical (per-screen) size for a physical matrix. In double-sided mode the physical panel chain shows N identical copies of a - smaller logical screen. Plugins size themselves from ``matrix.width`` / - ``matrix.height`` (the documented convention, used at 30+ call sites), so - this proxy reports the logical dimensions while delegating every real + smaller logical screen. Plugins size themselves from + ``display_manager.width`` / ``height`` (the documented convention), which + defer to ``matrix.width`` / ``matrix.height`` -- and many older plugins read + ``matrix.width`` directly -- so this proxy reports the logical dimensions + while delegating every real operation — ``CreateFrameCanvas``, ``SwapOnVSync``, ``brightness``, ``Clear`` and so on — to the underlying physical matrix. The duplication itself happens once per frame in :meth:`DisplayManager.update_display`. @@ -247,28 +251,41 @@ class DisplayManager: # Calendar manager is now initialized by DisplayController # Orientation setting -> rpi-rgb-led-matrix "Rotate:" pixel-mapper suffix. - # "normal" needs no suffix since 0 degrees is the identity transform. - _ORIENTATION_ROTATE_DEGREES = {'normal': None, '90': 90, '180': 180, '270': 270} + _ORIENTATION_ROTATE_DEGREES = ORIENTATION_ROTATE_DEGREES def _build_pixel_mapper_config(self, hardware_config: dict) -> str: - """Compose the raw pixel_mapper_config string with the orientation setting. + """Compose pixel_mapper_config with the orientation setting. - `pixel_mapper_config` stays available as a free-form advanced field (e.g. - for "U-mapper" chain layouts); `orientation` is the user-facing dropdown - for physical mounting (e.g. panels mounted upside down) and is appended as - a "Rotate:" mapper rather than overwriting any existing config. + See :func:`src.display_geometry.compose_pixel_mapper_config`, which the + web preview shares so it sizes the canvas the same way. """ - base_mapper = (hardware_config.get('pixel_mapper_config') or '').strip() - orientation = hardware_config.get('orientation', 'normal') - degrees = self._ORIENTATION_ROTATE_DEGREES.get(orientation) - if degrees is None: - return base_mapper - rotate_mapper = f'Rotate:{degrees}' - return f'{base_mapper};{rotate_mapper}' if base_mapper else rotate_mapper + return compose_pixel_mapper_config(hardware_config) + + @staticmethod + def _fallback_advice(cause: str, error: Exception) -> str: + """What to do about a failed matrix init, for the log. + + Only a library failure gets the rebuild hint: advice about the build + or GPIO timing sends someone whose settings were refused the wrong way. + """ + if cause == "settings": + return (f"{error} Change these in the web interface's Display tab " + "(or display.hardware / display.runtime in config.json) " + "and restart the display service.") + if cause == "forced": + return f"Error: {error}." + advice = (f"Error: {error}. If the rgbmatrix library printed a message " + "just before this, it names the problem.") + if is_raspberry_pi_5(): + advice += (" On a Raspberry Pi 5, an mmap error means the library was " + "built without Pi 5 support: sudo RPI_RGB_FORCE_REBUILD=1 " + "./first_time_install.sh") + return advice def _setup_matrix(self): """Initialize the RGB matrix with configuration settings.""" _init_error_str = None + _init_cause = None try: # Allow callers (e.g., web UI) to force non-hardware fallback mode if getattr(self, '_force_fallback', False): @@ -278,63 +295,25 @@ class DisplayManager: # Hardware configuration hardware_config = self.config.get('display', {}).get('hardware', {}) runtime_config = self.config.get('display', {}).get('runtime', {}) + + # The library has no error path for many settings it can't use: + # it returns no matrix (which the binding doesn't check, so the + # process crashes on its next call) or calls abort(), and systemd + # restarts the service into the same crash. Refuse those first so + # they become a logged, reported fallback (src/matrix_support.py). + refused = refusal_message(library_refusals( + hardware_config, runtime_config, pi5=is_raspberry_pi_5())) + if refused: + if os.getenv("EMULATOR", "false") != "true": + raise MatrixSettingsRefused(refused) + logger.warning("Emulator mode: continuing, but on a real panel the display would not start. %s", refused) - # Basic hardware settings - options.rows = hardware_config.get('rows', DEFAULT_ROWS) - options.cols = hardware_config.get('cols', DEFAULT_COLS) - options.chain_length = hardware_config.get('chain_length', DEFAULT_CHAIN_LENGTH) - options.parallel = hardware_config.get('parallel', DEFAULT_PARALLEL) - options.hardware_mapping = hardware_config.get('hardware_mapping', 'adafruit-hat-pwm') - - # Performance and stability settings - options.brightness = hardware_config.get('brightness', 90) - options.pwm_bits = hardware_config.get('pwm_bits', 10) - options.pwm_lsb_nanoseconds = hardware_config.get('pwm_lsb_nanoseconds', 150) - options.led_rgb_sequence = hardware_config.get('led_rgb_sequence', 'RGB') - options.pixel_mapper_config = self._build_pixel_mapper_config(hardware_config) - options.row_address_type = hardware_config.get('row_address_type', 0) - options.multiplexing = hardware_config.get('multiplexing', 0) - options.panel_type = hardware_config.get('panel_type', '') - options.disable_hardware_pulsing = hardware_config.get('disable_hardware_pulsing', False) - options.show_refresh_rate = hardware_config.get('show_refresh_rate', False) - options.limit_refresh_rate_hz = hardware_config.get('limit_refresh_rate_hz', 90) - options.gpio_slowdown = runtime_config.get('gpio_slowdown', 3) - - # Disable internal privilege dropping - we manage this via systemd or remain root - # This prevents the library from dropping to 'daemon' user which breaks file permissions - options.drop_privileges = False - - # Additional settings from config - if 'scan_mode' in hardware_config: - options.scan_mode = hardware_config.get('scan_mode') - if 'pwm_dither_bits' in hardware_config: - options.pwm_dither_bits = hardware_config.get('pwm_dither_bits') - if 'inverse_colors' in hardware_config: - options.inverse_colors = hardware_config.get('inverse_colors') - # Pi 5 only: 0=PIO/RP1 coprocessor (default, less CPU), - # 1=RIO/Registered IO (faster; gpio_slowdown effect is inverted in this mode) - if 'rp1_rio' in runtime_config: - if hasattr(options, 'rp1_rio'): - options.rp1_rio = runtime_config.get('rp1_rio') - else: - logger.warning( - "rp1_rio is set in config but the installed rgbmatrix library does " - "not support it — the library was likely built without Pi 5 RP1 " - "support (mmap to 0x3f000000 instead of RP1 chip). " - "Fix: sudo RPI_RGB_FORCE_REBUILD=1 ./first_time_install.sh" - ) + # Every option comes from display.hardware / display.runtime, in + # one place that scripts/scroll_speeds.py shares. + self.apply_matrix_options(options, self.config) logger.info(f"Initializing RGB Matrix with settings: rows={options.rows}, cols={options.cols}, chain_length={options.chain_length}, parallel={options.parallel}, hardware_mapping={options.hardware_mapping}") - # On a Pi 5 the library hands back no matrix for settings its RP1 - # path can't drive, and the binding doesn't check -- the process - # would crash on its next call instead of reaching the fallback - # below. Raise first so it is a logged, reported init failure. - if os.getenv("EMULATOR", "false") != "true" and is_raspberry_pi_5(): - unsupported = pi5_unsupported_settings(hardware_config) - if unsupported: - raise RuntimeError(unsupported) - # Initialize the matrix self.matrix = RGBMatrix(options=options) logger.info("RGB Matrix initialized successfully") @@ -379,7 +358,12 @@ class DisplayManager: except Exception as e: _init_error_str = str(e) - logger.error(f"Failed to initialize RGB Matrix: {e}", exc_info=True) + if isinstance(e, MatrixSettingsRefused): + _init_cause = "settings" + logger.error("Failed to initialize RGB Matrix: %s", e) + else: + _init_cause = "forced" if getattr(self, '_force_fallback', False) else "library" + logger.error(f"Failed to initialize RGB Matrix: {e}", exc_info=True) # Create a fallback image for web preview using configured dimensions when available self.matrix = None try: @@ -406,15 +390,18 @@ class DisplayManager: # Best-effort; ignore drawing errors in fallback pass logger.error( - f"Matrix initialization failed — running in fallback/simulation mode " - f"(size {fallback_width}x{fallback_height}). Error: {e}. " - "On Raspberry Pi 5: ensure rpi-rgb-led-matrix was built from the latest " - "submodule (re-run first_time_install.sh). gpio_slowdown of 2–3 is typical for Pi 5 PIO mode." - ) + "Matrix initialization failed — running in fallback/simulation mode " + "(size %dx%d). %s", + fallback_width, fallback_height, self._fallback_advice(_init_cause, e)) # Do not raise here; allow fallback mode so web preview and non-hardware environments work # Write hardware status file so the web UI can surface init failures - _hw_status = {"ok": self.matrix is not None, "error": _init_error_str} + # cause: None when ok; "settings" when LEDMatrix refused the config + # (fix the named settings), "library" when the library itself failed, + # "forced" for a caller-requested fallback. The Display tab keys its + # advice on it. + _hw_status = {"ok": self.matrix is not None, "error": _init_error_str, + "cause": None if self.matrix is not None else _init_cause} _status_path = "/tmp/led_matrix_hw_status.json" # nosec B108 try: if os.path.islink(_status_path): @@ -682,8 +669,10 @@ class DisplayManager: def render_size(self, width: int, height: Optional[int] = None): """Temporarily present a smaller logical canvas to plugins. - Plugins lay out against ``display_manager.matrix.width`` (and the - ``width``/``height`` properties, which defer to it), so the only way to + Plugins lay out against the ``display_manager.width``/``height`` + properties (which defer to ``matrix.width`` when hardware is present, + and to the canvas when it is not; some older plugins read + ``matrix.width`` directly), so the only way to get a *narrower layout* rather than a cropped one is to tell the plugin the screen is narrower while it renders. Trimming after the fact cannot fix a forecast spread across five columns or a progress bar drawn at @@ -1370,6 +1359,68 @@ class DisplayManager: return dt.strftime(f"%b %-d{suffix}") + @classmethod + def apply_matrix_options(cls, options, config: Dict[str, Any]): + """Fill ``options`` (an ``RGBMatrixOptions``) from the LEDMatrix config. + + This is exactly what the display service drives the panel with, so a + tool that opens the matrix itself (``scripts/scroll_speeds.py``) gets + the same panel -- same runtime ``gpio_slowdown``, ``rp1_rio``, + ``panel_type``, orientation and defaults -- rather than a private copy + that drifts. Does not open the matrix. Returns ``options``. + """ + display = config.get('display', {}) if isinstance(config, dict) else {} + hardware_config = display.get('hardware', {}) + runtime_config = display.get('runtime', {}) + + # Basic hardware settings + options.rows = hardware_config.get('rows', DEFAULT_ROWS) + options.cols = hardware_config.get('cols', DEFAULT_COLS) + options.chain_length = hardware_config.get('chain_length', DEFAULT_CHAIN_LENGTH) + options.parallel = hardware_config.get('parallel', DEFAULT_PARALLEL) + options.hardware_mapping = hardware_config.get('hardware_mapping', 'adafruit-hat-pwm') + + # Performance and stability settings + options.brightness = hardware_config.get('brightness', 90) + options.pwm_bits = hardware_config.get('pwm_bits', 10) + options.pwm_lsb_nanoseconds = hardware_config.get('pwm_lsb_nanoseconds', 150) + options.led_rgb_sequence = hardware_config.get('led_rgb_sequence', 'RGB') + # _build_pixel_mapper_config reads only class attributes, so the class + # stands in for an instance here. + options.pixel_mapper_config = cls._build_pixel_mapper_config(cls, hardware_config) + options.row_address_type = hardware_config.get('row_address_type', 0) + options.multiplexing = hardware_config.get('multiplexing', 0) + options.panel_type = hardware_config.get('panel_type', '') + options.disable_hardware_pulsing = hardware_config.get('disable_hardware_pulsing', False) + options.show_refresh_rate = hardware_config.get('show_refresh_rate', False) + options.limit_refresh_rate_hz = hardware_config.get('limit_refresh_rate_hz', 90) + options.gpio_slowdown = runtime_config.get('gpio_slowdown', 3) + + # Disable internal privilege dropping - we manage this via systemd or remain root + # This prevents the library from dropping to 'daemon' user which breaks file permissions + options.drop_privileges = False + + # Additional settings from config + if 'scan_mode' in hardware_config: + options.scan_mode = hardware_config.get('scan_mode') + if 'pwm_dither_bits' in hardware_config: + options.pwm_dither_bits = hardware_config.get('pwm_dither_bits') + if 'inverse_colors' in hardware_config: + options.inverse_colors = hardware_config.get('inverse_colors') + # Pi 5 only: 0=PIO/RP1 coprocessor (default, less CPU), + # 1=RIO/Registered IO (faster; gpio_slowdown effect is inverted in this mode) + if 'rp1_rio' in runtime_config: + if hasattr(options, 'rp1_rio'): + options.rp1_rio = runtime_config.get('rp1_rio') + else: + logger.warning( + "rp1_rio is set in config but the installed rgbmatrix library does " + "not support it — the library was likely built without Pi 5 RP1 " + "support (mmap to 0x3f000000 instead of RP1 chip). " + "Fix: sudo RPI_RGB_FORCE_REBUILD=1 ./first_time_install.sh" + ) + return options + @property def refresh_hz(self) -> float: """The panel's refresh rate in Hz, from the hardware config. @@ -1415,6 +1466,12 @@ class DisplayManager: pixel every second refresh, which is how a scroll runs at half the refresh rate without fractional pixel positions. + The hold is part of the scroll's speed. A ScrollHelper configured by + ``scroll_config.configure()`` advances a fixed whole-pixel step per + presented frame and reads no clock, so pass the returned + ``settings.frame_hold`` here: a scroll that leaves it at 1 is + presented every refresh and runs ``frame_hold`` times too fast. + The hold is set here rather than once at plugin construction because it must not outlive the scroll that asked for it: plugins share one display manager, so a hold left set by whoever scrolled last would diff --git a/src/matrix_support.py b/src/matrix_support.py new file mode 100644 index 00000000..b99d4e08 --- /dev/null +++ b/src/matrix_support.py @@ -0,0 +1,205 @@ +"""Display settings the pinned rgbmatrix library refuses, on every board. + +The library does not raise on a setting it can't use. For most it returns no +matrix -- ``RGBMatrix::CreateFromOptions()`` gives back NULL when +``Options::Validate()`` (``lib/options-initialize.cc``) or its GPIO slowdown +check (``lib/led-matrix.cc``) fails -- and the Python binding stores that +pointer without checking, so the display process crashes on its first call +into the matrix. For two it calls ``abort()``: an unknown hardware mapping +name, and more parallel chains than the mapping has outputs +(``Framebuffer::InitHardwareMapping()`` and the ``Framebuffer`` constructor in +``lib/framebuffer.cc``). Either way systemd restarts the service into the same +crash, and no fallback or hardware status is ever reported. + +``DisplayManager`` runs :func:`library_refusals` before creating the matrix, +turning those crashes into a logged error and the usual fallback mode. The +config API uses the same rules to refuse the settings up front, and +:data:`INT_SETTING_LIMITS` is the one place the numeric ranges live. + +Pi 5-only limits come from :mod:`src.pi5_matrix_support` and are added when +``pi5=True``. + +**Re-check this when the submodule is bumped** (pinned at 1ee4f76): the +ranges in ``Options::Validate()``, the mapping table in +``lib/hardware-mapping.c`` and the setter types in +``bindings/python/rgbmatrix/core.pyx``. A stale rule here blocks settings a +new library accepts; a missing one lets the display service crash-loop. +""" + +from typing import Any, Dict, List, Mapping, NamedTuple, Optional, Tuple + +from src.pi5_matrix_support import pi5_unsupported_settings + +#: The binding's setters for rows, chain_length, parallel and the other small +#: integers are declared ``uint8_t`` (``core.pyx``), cols ``uint32_t``, and +#: limit_refresh_rate_hz lands in a C ``int``; a larger value raises +#: ``OverflowError`` before the library sees it. +UINT8_MAX = 255 +UINT32_MAX = 2 ** 32 - 1 +INT32_MAX = 2 ** 31 - 1 + +#: field -> (config section, lowest, highest, must be even). The library's own +#: ranges where it has one, otherwise the binding's integer type. +INT_SETTING_LIMITS: Dict[str, Tuple[str, int, int, bool]] = { + 'rows': ('hardware', 8, 64, True), + 'cols': ('hardware', 16, UINT32_MAX, False), + 'chain_length': ('hardware', 1, UINT8_MAX, False), + # 3 is the most any mapping in the default build has (see MAPPING_OUTPUTS). + 'parallel': ('hardware', 1, 3, False), + 'brightness': ('hardware', 1, 100, False), + 'scan_mode': ('hardware', 0, 1, False), + 'pwm_bits': ('hardware', 1, 11, False), + 'pwm_dither_bits': ('hardware', 0, 2, False), + 'pwm_lsb_nanoseconds': ('hardware', 50, 3000, False), + # 0 = no cap; the library has no upper bound. + 'limit_refresh_rate_hz': ('hardware', 0, INT32_MAX, False), + 'row_address_type': ('hardware', 0, 5, False), + # 22 registered multiplexers (lib/multiplex-mappers.cc). + 'multiplexing': ('hardware', 0, 22, False), + 'gpio_slowdown': ('runtime', 0, 10, False), + 'rp1_rio': ('runtime', 0, 1, False), +} + +#: Hardware mapping name -> parallel chains it has outputs for, as +#: ``lib/hardware-mapping.c`` defines them. ``compute-module`` exists only when +#: the library is built with ENABLE_WIDE_GPIO_COMPUTE_MODULE, which the pinned +#: Makefile leaves commented out and the installer does not set, so the library +#: LEDMatrix installs aborts on it like any other unknown name. +MAPPING_OUTPUTS: Dict[str, int] = { + 'regular': 3, + 'adafruit-hat': 1, + 'adafruit-hat-pwm': 1, + 'regular-pi1': 1, + 'classic': 3, + 'classic-pi1': 1, +} + +#: What DisplayManager passes when a key is missing from display.hardware / +#: display.runtime. Config migration normally fills these from +#: config/config.template.json first, so they rarely apply. +DISPLAY_MANAGER_DEFAULTS: Dict[str, Any] = { + 'rows': 32, 'cols': 64, 'chain_length': 2, 'parallel': 1, + 'hardware_mapping': 'adafruit-hat-pwm', 'brightness': 90, 'pwm_bits': 10, + 'pwm_lsb_nanoseconds': 150, 'led_rgb_sequence': 'RGB', + 'row_address_type': 0, 'multiplexing': 0, 'limit_refresh_rate_hz': 90, + 'gpio_slowdown': 3, +} + + +class Refusal(NamedTuple): + """One reason the library can't start with a config. + + ``fields`` are the settings involved; the config API reports a refusal + only when the request sets one of them, so a problem already stored + doesn't block an unrelated save. + """ + fields: Tuple[str, ...] + message: str + #: A Raspberry Pi 5-only limit, whose message is already a full sentence. + pi5: bool = False + + +class MatrixSettingsRefused(RuntimeError): + """Raised by DisplayManager instead of handing the library such a config.""" + + +def _as_int(value: Any) -> Optional[int]: + """The integer the binding would receive, or None if it isn't one. + + A value that isn't a number is left to the binding, which raises a + ``TypeError`` DisplayManager already turns into fallback mode. + """ + try: + return int(value) + except (TypeError, ValueError, OverflowError): + return None + + +def _setting(hardware: Mapping[str, Any], runtime: Mapping[str, Any], field: str) -> Any: + section = runtime if INT_SETTING_LIMITS.get(field, ('hardware',))[0] == 'runtime' else hardware + if field in section: + return section[field] + return DISPLAY_MANAGER_DEFAULTS.get(field) + + +def describe_range(low: int, high: int, even: bool = False) -> str: + kind = "an even integer" if even else "an integer" + if high >= INT32_MAX: + return f"{kind} of at least {low}" + return f"{kind} from {low} to {high}" + + +def library_refusals(hardware: Optional[Mapping[str, Any]], + runtime: Optional[Mapping[str, Any]] = None, + pi5: bool = False) -> List[Refusal]: + """Every reason the pinned library would refuse (NULL, abort or overflow) + to start with this ``display.hardware`` / ``display.runtime`` config. + + Missing keys take DisplayManager's defaults. ``pi5`` adds the Raspberry + Pi 5 limits from :func:`pi5_unsupported_settings`. + """ + hardware = hardware if isinstance(hardware, Mapping) else {} + runtime = runtime if isinstance(runtime, Mapping) else {} + refusals: List[Refusal] = [] + + for field, (section, low, high, even) in INT_SETTING_LIMITS.items(): + # DisplayManager sets these only when present, and scan_mode / + # pwm_dither_bits / rp1_rio have no default of its own. + source = runtime if section == 'runtime' else hardware + if field not in source and field not in DISPLAY_MANAGER_DEFAULTS: + continue + value = _as_int(_setting(hardware, runtime, field)) + if value is None: + continue + if not low <= value <= high or (even and value % 2): + refusals.append(Refusal( + (field,), f"{field} {value} (must be {describe_range(low, high, even)})")) + + mapping = _setting(hardware, runtime, 'hardware_mapping') + outputs = None + if not isinstance(mapping, str): + refusals.append(Refusal( + ('hardware_mapping',), f"hardware mapping {mapping!r} (must be a mapping name)")) + else: + # The library matches names case-insensitively and reads an empty + # name as "regular". + outputs = MAPPING_OUTPUTS.get((mapping or 'regular').lower()) + if outputs is None: + refusals.append(Refusal( + ('hardware_mapping',), + f'hardware mapping "{mapping}" (the installed library has ' + + ", ".join(MAPPING_OUTPUTS) + ")")) + + parallel = _as_int(_setting(hardware, runtime, 'parallel')) + if outputs is not None and parallel is not None and 1 <= parallel <= 3 and parallel > outputs: + refusals.append(Refusal( + ('parallel', 'hardware_mapping'), + f'parallel {parallel} with hardware mapping "{mapping}", which has ' + f'{outputs} output{"s" if outputs != 1 else ""}')) + + sequence = _setting(hardware, runtime, 'led_rgb_sequence') + if isinstance(sequence, str) and not ( + len(sequence) == 3 and set(sequence.upper()) == {'R', 'G', 'B'}): + refusals.append(Refusal( + ('led_rgb_sequence',), + f'LED RGB sequence "{sequence}" (must be R, G and B in some order)')) + + if pi5: + unsupported = pi5_unsupported_settings(hardware) + if unsupported: + refusals.append(Refusal( + ('row_address_type', 'parallel', 'hardware_mapping'), unsupported, pi5=True)) + + return refusals + + +def refusal_message(refusals: List[Refusal]) -> Optional[str]: + """One sentence naming every refusal, or None when there are none.""" + general = [r.message for r in refusals if not r.pi5] + pi5 = [r.message for r in refusals if r.pi5] + parts = [] + if general: + parts.append("The installed rgbmatrix library can't start with these " + "display settings: " + "; ".join(general) + ".") + parts.extend(pi5) + return " ".join(parts) or None diff --git a/src/plugin_system/plugin_manager.py b/src/plugin_system/plugin_manager.py index c587acc8..2af661a5 100644 --- a/src/plugin_system/plugin_manager.py +++ b/src/plugin_system/plugin_manager.py @@ -22,7 +22,10 @@ from src.logging_config import get_logger from src.plugin_system.plugin_loader import PluginLoader from src.plugin_system.plugin_executor import PluginExecutor from src.plugin_system.plugin_state import PluginStateManager, PluginState -from src.plugin_system.schema_manager import SchemaManager, normalize_legacy_booleans +from src.plugin_system.schema_manager import ( + CORE_VEGAS_TUNING_KEYS, SchemaManager, normalize_legacy_booleans, +) +from src.common.path_safety import safe_path_component from src.common.permission_utils import ( ensure_directory_permissions, get_plugin_dir_mode @@ -369,32 +372,11 @@ class PluginManager: f"The schema may be invalid. Please verify the schema file at: {schema_path}" ) - # A plugin that turned an on/off boolean into an {enabled, ...} - # object still finds the boolean in config.json until the user saves - # its settings form, which carries it over (plugin_config.html). - # Read it the same way here, before the defaults fill in the rest - # of the object and before schema validation, so the plugin doesn't - # start with a schema warning and a degraded flag. In memory only: - # config.json is written by saves, never by loading a plugin. - if schema: - upgraded: List[str] = [] - config = normalize_legacy_booleans(config, schema, upgraded) - if upgraded: - self.logger.info( - "Plugin %s: reading legacy boolean setting %s as " - "{\"enabled\": ...}; saving the plugin's settings " - "stores the new shape", - plugin_id, ", ".join(upgraded), - ) - - # Merge config with schema defaults to ensure all defaults are applied - try: - defaults = self.schema_manager.generate_default_config(plugin_id, use_cache=True) - config = self.schema_manager.merge_with_defaults(config, defaults) - self.logger.debug(f"Merged config with schema defaults for {plugin_id}") - except Exception as e: - self.logger.warning(f"Could not apply schema defaults for {plugin_id}: {e}") - # Continue with original config if defaults can't be applied + # Legacy booleans read as objects, then schema defaults: the same + # preparation saves, GET /plugins/config and hot reload apply + # (prepare_plugin_config). In memory only: config.json is written + # by saves, never by loading a plugin. + config = self.prepare_plugin_config(plugin_id, config, schema=schema) # Use PluginLoader to load plugin plugin_instance, module = self.plugin_loader.load_plugin( @@ -489,11 +471,57 @@ class PluginManager: #: #: Read by: ``vegas_mode/plugin_adapter.py`` (``vegas_width_pct``, #: ``vegas_overflow``) and ``base_plugin.py`` (``vegas_max_width_screens``). - CORE_OWNED_CONFIG_KEYS = frozenset({ - 'vegas_width_pct', - 'vegas_overflow', - 'vegas_max_width_screens', - }) + #: + #: The list itself lives with the other core-owned per-plugin properties in + #: ``schema_manager.CORE_PLUGIN_PROPERTIES``, which the web save path also + #: uses to keep these keys. + CORE_OWNED_CONFIG_KEYS = CORE_VEGAS_TUNING_KEYS + + def prepare_plugin_config(self, plugin_id: str, config: Any, + schema: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: + """The config a plugin runs with, built from its raw config.json section. + + A plugin that turned an on/off boolean into an ``{enabled, ...}`` + object still finds the boolean in config.json until its settings are + next saved; it is read as the object, and schema defaults fill in the + rest (``SchemaManager.prepare_plugin_config``). Used when loading a + plugin and on hot reload (DisplayController), so ``on_config_change`` + receives the same shape the plugin was constructed with. + + Never raises: on failure the legacy-boolean pass alone is applied, or + failing that the section is returned as it was. + """ + if schema is None: + try: + schema = self.schema_manager.load_schema(plugin_id) + except Exception as e: + self.logger.debug("Could not load schema for %s: %s", plugin_id, e) + schema = None + upgraded: List[str] = [] + try: + prepared = self.schema_manager.prepare_plugin_config( + plugin_id, config, schema=schema, changed_paths=upgraded) + self.logger.debug("Merged config with schema defaults for %s", plugin_id) + except Exception as e: + self.logger.warning("Could not apply schema defaults for %s: %s", plugin_id, e) + # Continue without defaults if they can't be applied + upgraded = [] + prepared = config if isinstance(config, dict) else {} + if schema: + try: + prepared = normalize_legacy_booleans(prepared, schema, upgraded) + except Exception as legacy_error: + self.logger.warning( + "Could not read legacy boolean settings for %s: %s", + plugin_id, legacy_error) + if upgraded: + self.logger.info( + "Plugin %s: reading legacy boolean setting %s as " + "{\"enabled\": ...}; saving the plugin's settings " + "stores the new shape", + plugin_id, ", ".join(upgraded), + ) + return prepared def _strip_core_owned_keys(self, config: Dict[str, Any]) -> Dict[str, Any]: """A shallow copy of ``config`` without the core's own tuning keys. @@ -743,11 +771,20 @@ class PluginManager: Returns: Directory path as string or None if not found + + ``plugin_id`` often comes straight from a request, so anything that is + not one plain path segment (``..``, ``a/b``, an absolute path) is + refused instead of being joined onto ``plugins_dir``. The join is not + resolved further: dev plugins are symlinks into ``plugins_dir``. """ with self._discovery_lock: if hasattr(self, 'plugin_directories') and plugin_id in self.plugin_directories: return str(self.plugin_directories[plugin_id]) - + + plugin_id = safe_path_component(plugin_id) + if plugin_id is None: + return None + plugin_dir = self.plugins_dir / plugin_id if plugin_dir.exists(): return str(plugin_dir) diff --git a/src/plugin_system/schema_manager.py b/src/plugin_system/schema_manager.py index b56e6126..d4eb70b8 100644 --- a/src/plugin_system/schema_manager.py +++ b/src/plugin_system/schema_manager.py @@ -13,6 +13,8 @@ from typing import Any, Dict, List, Optional, Tuple import jsonschema from jsonschema import Draft7Validator, ValidationError +from src.core_config_keys import CORE_CONFIG_KEYS + def _renders_as_object(prop: Dict[str, Any]) -> bool: """``field_type == 'object'`` as ``plugin_config.html`` computes it. @@ -93,6 +95,222 @@ def normalize_legacy_booleans(config: Any, schema: Any, return result +#: Per-plugin settings the **core** owns: it reads them out of each plugin's +#: config section, so they are allowed in every plugin's config whether or not +#: the plugin's schema declares them. The one list for validation, for the web +#: save filter and for the load-time checks -- a private copy is how JSON saves +#: came to drop ``skin`` and the ``vegas_*`` keys while the validator accepted +#: them. +#: +#: Values are the schema used when the plugin does not declare the property. +CORE_PLUGIN_PROPERTIES: Dict[str, Dict[str, Any]] = { + # Defaults match BasePlugin behavior: enabled=True, display_duration=15, + # live_priority=False. + "enabled": { + "type": "boolean", + "default": True, + "description": "Enable or disable this plugin" + }, + "display_duration": { + "type": "number", + "default": 15, + "minimum": 1, + "maximum": 300, + "description": "How long to display this plugin in seconds" + }, + "live_priority": { + "type": "boolean", + "default": False, + "description": "Enable live priority takeover when plugin has live content" + }, + # Skin selection (docs/SKIN_SYSTEM.md). Deliberately NOT an enum here: + # validation must keep passing when a configured skin gets uninstalled + # (rendering falls back to built-in). The install-dependent enum is + # injected only at serve time (inject_skin_selector) for the web UI + # dropdown. + "skin": { + "type": ["string", "object", "null"], + "description": "Visual skin id, or a per-mode mapping like {\"live\": \"my-skin\"}" + }, + "skin_options": { + "type": "object", + "description": "Options passed through to the selected skin" + }, + # Vegas tuning read by vegas_mode/plugin_adapter.py and base_plugin.py. + # Left untyped: the adapter validates them itself and ignores a bad + # value with a log line, so a stored one must never block a save. + "vegas_width_pct": { + "description": "Vegas mode: width of this plugin's card, as a percentage of the panel" + }, + "vegas_overflow": { + "description": "Vegas mode: 'rotate' or 'truncate' when this plugin's content overflows" + }, + "vegas_max_width_screens": { + "description": "Vegas mode: widest this plugin's card may be, in screens" + }, +} + +#: The keys of CORE_PLUGIN_PROPERTIES that are Vegas tuning rather than plugin +#: state. PluginManager strips these before its soft validation (see +#: PluginManager.CORE_OWNED_CONFIG_KEYS). +CORE_VEGAS_TUNING_KEYS = frozenset({ + 'vegas_width_pct', 'vegas_overflow', 'vegas_max_width_screens', +}) + + +def with_core_plugin_properties(schema: Dict[str, Any]) -> Dict[str, Any]: + """A deep copy of a plugin schema with CORE_PLUGIN_PROPERTIES allowed. + + Properties the plugin declares itself are left as declared. Core + properties are removed from ``required``: they are system-managed. + """ + enhanced = copy.deepcopy(schema) if isinstance(schema, dict) else {} + properties = enhanced.setdefault("properties", {}) + for name, definition in CORE_PLUGIN_PROPERTIES.items(): + if name not in properties: + properties[name] = copy.deepcopy(definition) + if "required" in enhanced: + enhanced["required"] = [field for field in enhanced["required"] + if field not in CORE_PLUGIN_PROPERTIES] + return enhanced + + +def extract_schema_defaults(schema: Dict[str, Any]) -> Dict[str, Any]: + """Default values of a JSON Schema's properties, recursively. + + A property's own ``default`` wins; otherwise a nested object contributes + its children's defaults, and an array contributes ``[]`` (or a one-item + list of its ``items`` default). This is what a device runs with, so the + dev tools use it too (src/plugin_system/testing/loading.py). + """ + defaults: Dict[str, Any] = {} + properties = schema.get('properties', {}) if isinstance(schema, dict) else {} + if not isinstance(properties, dict): + return defaults + + for key, prop_schema in properties.items(): + if not isinstance(prop_schema, dict): + continue + # If property has a default, use it + if 'default' in prop_schema: + defaults[key] = prop_schema['default'] + continue + + # Handle nested objects + if prop_schema.get('type') == 'object' and 'properties' in prop_schema: + nested_defaults = extract_schema_defaults(prop_schema) + if nested_defaults: + defaults[key] = nested_defaults + + # Handle arrays with object items + elif prop_schema.get('type') == 'array' and 'items' in prop_schema: + items_schema = prop_schema['items'] + if items_schema.get('type') == 'object' and 'properties' in items_schema: + # For arrays of objects, use empty array as default + # Individual objects will use their defaults when created + defaults[key] = [] + elif 'default' in items_schema: + # Array with default item value + defaults[key] = [items_schema['default']] + else: + # Empty array as default + defaults[key] = [] + + # For other types without defaults, don't add to defaults dict + # This allows plugins to handle missing values as needed + + return defaults + + +def plugin_config_defaults(schema: Optional[Dict[str, Any]]) -> Dict[str, Any]: + """Every default a plugin's config gets: the schema's plus the core ones. + + A plugin with no schema gets the minimal ``enabled: False, + display_duration: 15``. Device location is not applied here; that needs a + config manager (SchemaManager.generate_default_config). + """ + if not schema: + return { + 'enabled': False, + 'display_duration': 15 + } + + defaults = extract_schema_defaults(schema) + + # Ensure core properties have defaults (they may not be in the schema) + # These match BasePlugin behavior + for name in ('enabled', 'display_duration', 'live_priority'): + if name not in defaults: + defaults[name] = CORE_PLUGIN_PROPERTIES[name]['default'] + return defaults + + +def merge_config_defaults(config: Dict[str, Any], defaults: Dict[str, Any]) -> Dict[str, Any]: + """Merge configuration with defaults, preserving user values. + + Also replaces None values with defaults so a config never starts with + None where a default exists. Neither argument is mutated. + """ + merged = copy.deepcopy(defaults) + + def deep_merge(target: Dict[str, Any], source: Dict[str, Any], default_dict: Dict[str, Any]) -> None: + """Recursively merge source into target, replacing None with defaults.""" + for key, value in source.items(): + default_value = default_dict.get(key) + + if key in target and isinstance(target[key], dict) and isinstance(value, dict): + # Both are dicts, recursively merge + if isinstance(default_value, dict): + deep_merge(target[key], value, default_value) + else: + deep_merge(target[key], value, {}) + elif value is None and default_value is not None: + # Value is None and we have a default, use the default + target[key] = copy.deepcopy(default_value) if isinstance(default_value, (dict, list)) else default_value + else: + # Normal merge: user value takes precedence (copy if dict/list) + if isinstance(value, (dict, list)): + target[key] = copy.deepcopy(value) + else: + target[key] = value + + deep_merge(merged, config, defaults) + + # Final pass: replace any remaining None values at any level with defaults + def replace_none_with_defaults(target: Dict[str, Any], default_dict: Dict[str, Any]) -> None: + """Recursively replace None values with defaults.""" + for key in list(target.keys()): + value = target[key] + default_value = default_dict.get(key) + + if value is None and default_value is not None: + # Replace None with default + target[key] = copy.deepcopy(default_value) if isinstance(default_value, (dict, list)) else default_value + elif isinstance(value, dict) and isinstance(default_value, dict): + # Recursively process nested dicts + replace_none_with_defaults(value, default_value) + + replace_none_with_defaults(merged, defaults) + return merged + + +def prepare_plugin_config(config: Any, schema: Optional[Dict[str, Any]], + defaults: Dict[str, Any], + changed_paths: Optional[List[str]] = None) -> Dict[str, Any]: + """The config a plugin runs with, from its stored (or submitted) section. + + Legacy booleans are read as ``{"enabled": ...}`` objects + (normalize_legacy_booleans), then schema defaults fill in whatever is + missing. Loading a plugin, both config saves, GET /plugins/config, hot + reload and the dev tools all go through this, so a plugin sees the same + shape however its config reached it. + """ + config = config if isinstance(config, dict) else {} + if schema: + config = normalize_legacy_booleans(config, schema, changed_paths) + return merge_config_defaults(config, defaults) + + class SchemaManager: """ Manages plugin configuration schemas with caching and validation. @@ -262,57 +480,12 @@ class SchemaManager: def extract_defaults_from_schema(self, schema: Dict[str, Any], prefix: str = '') -> Dict[str, Any]: """ Recursively extract default values from a JSON Schema. - - Handles nested objects, arrays, and all schema types. - - Args: - schema: JSON Schema dictionary - prefix: Optional prefix for logging/debugging - - Returns: - Dictionary of default values + + See :func:`extract_schema_defaults`; ``prefix`` is accepted for + compatibility and unused. """ - defaults = {} - - # Handle schema with properties - properties = schema.get('properties', {}) - if not properties: - return defaults - - for key, prop_schema in properties.items(): - field_path = f"{prefix}.{key}" if prefix else key - - # If property has a default, use it - if 'default' in prop_schema: - defaults[key] = prop_schema['default'] - self.logger.debug(f"Found default for {field_path}: {prop_schema['default']}") - continue - - # Handle nested objects - if prop_schema.get('type') == 'object' and 'properties' in prop_schema: - nested_defaults = self.extract_defaults_from_schema(prop_schema, field_path) - if nested_defaults: - defaults[key] = nested_defaults - - # Handle arrays with object items - elif prop_schema.get('type') == 'array' and 'items' in prop_schema: - items_schema = prop_schema['items'] - if items_schema.get('type') == 'object' and 'properties' in items_schema: - # For arrays of objects, use empty array as default - # Individual objects will use their defaults when created - defaults[key] = [] - elif 'default' in items_schema: - # Array with default item value - defaults[key] = [items_schema['default']] - else: - # Empty array as default - defaults[key] = [] - - # For other types without defaults, don't add to defaults dict - # This allows plugins to handle missing values as needed - - return defaults - + return extract_schema_defaults(schema) + def get_device_location(self) -> Optional[Dict[str, Any]]: """ Return the device-wide ``location`` block from config.json, or None. @@ -391,31 +564,39 @@ class SchemaManager: schema = self.load_schema(plugin_id, use_cache=use_cache) if not schema: # Return minimal defaults if no schema - return { - 'enabled': False, - 'display_duration': 15 - } - - # Extract defaults from schema - defaults = self.extract_defaults_from_schema(schema) - - # Ensure core properties have defaults (they may not be in the schema) - # These match BasePlugin behavior - if 'enabled' not in defaults: - defaults['enabled'] = schema.get('properties', {}).get('enabled', {}).get('default', True) - - if 'display_duration' not in defaults: - defaults['display_duration'] = schema.get('properties', {}).get('display_duration', {}).get('default', 15) - - if 'live_priority' not in defaults: - defaults['live_priority'] = schema.get('properties', {}).get('live_priority', {}).get('default', False) - + return plugin_config_defaults(None) + + # Schema defaults plus the core properties' (they may not be in the + # schema) + defaults = plugin_config_defaults(schema) + # Cache the defaults *before* the device location is layered on, so a # later change to the device location is picked up by the next call. self._defaults_cache[plugin_id] = defaults.copy() return self.apply_device_location(defaults) + def prepare_plugin_config(self, plugin_id: str, config: Any, + schema: Optional[Dict[str, Any]] = None, + changed_paths: Optional[List[str]] = None) -> Dict[str, Any]: + """ + The config a plugin runs with: see :func:`prepare_plugin_config`. + + Args: + plugin_id: Plugin identifier + config: The plugin's stored or submitted config section + schema: The plugin's schema, when the caller already has it + changed_paths: Receives the dotted path of each legacy boolean + read as an object + + Returns: + A new dict; ``config`` is not mutated + """ + if schema is None: + schema = self.load_schema(plugin_id, use_cache=True) + defaults = self.generate_default_config(plugin_id, use_cache=True) + return prepare_plugin_config(config, schema, defaults, changed_paths) + def validate_config_against_schema(self, config: Dict[str, Any], schema: Dict[str, Any], plugin_id: Optional[str] = None) -> Tuple[bool, List[str]]: """ @@ -436,80 +617,18 @@ class SchemaManager: errors = [] try: - # Core plugin properties that should always be allowed - # These are handled by the base plugin system and should not cause validation failures - # Defaults match BasePlugin behavior: enabled=True, display_duration=15, live_priority=False - core_properties = { - "enabled": { - "type": "boolean", - "default": True, - "description": "Enable or disable this plugin" - }, - "display_duration": { - "type": "number", - "default": 15, - "minimum": 1, - "maximum": 300, - "description": "How long to display this plugin in seconds" - }, - "live_priority": { - "type": "boolean", - "default": False, - "description": "Enable live priority takeover when plugin has live content" - }, - # Skin selection (docs/SKIN_SYSTEM.md). Deliberately NOT an - # enum here: validation must keep passing when a configured - # skin gets uninstalled (rendering falls back to built-in). - # The install-dependent enum is injected only at serve time - # (inject_skin_selector) for the web UI dropdown. - "skin": { - "type": ["string", "object", "null"], - "description": "Visual skin id, or a per-mode mapping like {\"live\": \"my-skin\"}" - }, - "skin_options": { - "type": "object", - "description": "Options passed through to the selected skin" - } - } - - # Create a deep copy of the schema to modify (to avoid mutating the original) - enhanced_schema = copy.deepcopy(schema) - if "properties" not in enhanced_schema: - enhanced_schema["properties"] = {} - - # Inject core properties if they're not already defined in the schema - # This ensures core properties are always allowed even if not in the plugin's schema - properties_added = [] - for prop_name, prop_def in core_properties.items(): - if prop_name not in enhanced_schema["properties"]: - enhanced_schema["properties"][prop_name] = copy.deepcopy(prop_def) - properties_added.append(prop_name) - - # Log if we added any core properties (for debugging) - if properties_added and plugin_id: + # Core plugin properties (CORE_PLUGIN_PROPERTIES) are handled by + # the base plugin system and should not cause validation failures: + # they are allowed even when the plugin's schema doesn't declare + # them, and never required. + enhanced_schema = with_core_plugin_properties(schema) + if plugin_id: + declared = schema.get("properties", {}) if isinstance(schema, dict) else {} self.logger.debug( - f"Injected core properties into schema for {plugin_id}: {properties_added}" + "Injected core properties into schema for %s: %s", plugin_id, + [name for name in CORE_PLUGIN_PROPERTIES if name not in declared] ) - - # Remove core properties from required array (they're system-managed) - # Core properties should be allowed but not required for validation - if "required" in enhanced_schema: - core_prop_names = list(core_properties.keys()) - removed_from_required = [ - field for field in enhanced_schema["required"] - if field in core_prop_names - ] - enhanced_schema["required"] = [ - field for field in enhanced_schema["required"] - if field not in core_prop_names - ] - - # Log if we removed any core properties from required (for debugging) - if removed_from_required and plugin_id: - self.logger.debug( - f"Removed core properties from required array for {plugin_id}: {removed_from_required}" - ) - + # Create validator with enhanced schema validator = Draft7Validator(enhanced_schema) @@ -626,55 +745,15 @@ class SchemaManager: """ Merge configuration with defaults, preserving user values. Also replaces None values with defaults to ensure config never has None from the start. - + Args: config: User configuration defaults: Default values from schema - + Returns: Merged configuration with defaults applied where missing or None """ - merged = copy.deepcopy(defaults) - - def deep_merge(target: Dict[str, Any], source: Dict[str, Any], default_dict: Dict[str, Any]) -> None: - """Recursively merge source into target, replacing None with defaults.""" - for key, value in source.items(): - default_value = default_dict.get(key) - - if key in target and isinstance(target[key], dict) and isinstance(value, dict): - # Both are dicts, recursively merge - if isinstance(default_value, dict): - deep_merge(target[key], value, default_value) - else: - deep_merge(target[key], value, {}) - elif value is None and default_value is not None: - # Value is None and we have a default, use the default - target[key] = copy.deepcopy(default_value) if isinstance(default_value, (dict, list)) else default_value - else: - # Normal merge: user value takes precedence (copy if dict/list) - if isinstance(value, (dict, list)): - target[key] = copy.deepcopy(value) - else: - target[key] = value - - deep_merge(merged, config, defaults) - - # Final pass: replace any remaining None values at any level with defaults - def replace_none_with_defaults(target: Dict[str, Any], default_dict: Dict[str, Any]) -> None: - """Recursively replace None values with defaults.""" - for key in list(target.keys()): - value = target[key] - default_value = default_dict.get(key) - - if value is None and default_value is not None: - # Replace None with default - target[key] = copy.deepcopy(default_value) if isinstance(default_value, (dict, list)) else default_value - elif isinstance(value, dict) and isinstance(default_value, dict): - # Recursively process nested dicts - replace_none_with_defaults(value, default_value) - - replace_none_with_defaults(merged, defaults) - return merged + return merge_config_defaults(config, defaults) def detect_config_key_collisions( self, @@ -698,10 +777,11 @@ class SchemaManager: """ collisions = [] - # Reserved top-level config keys that plugins should not use as IDs - reserved_keys = { - 'display', 'schedule', 'timezone', 'plugin_system', - 'display_modes', 'system', 'hardware', 'debug', + # Reserved top-level config keys that plugins should not use as IDs: + # every core section (src/core_config_keys.py), plus a few names that + # read as core even though no current section uses them. + reserved_keys = set(CORE_CONFIG_KEYS) | { + 'display_modes', 'hardware', 'debug', 'log_level', 'emulator', 'web_interface' } diff --git a/src/plugin_system/testing/harness.py b/src/plugin_system/testing/harness.py index 267774c2..90351542 100644 --- a/src/plugin_system/testing/harness.py +++ b/src/plugin_system/testing/harness.py @@ -27,7 +27,7 @@ from PIL import Image, ImageChops from src.logging_config import get_logger from .bounds_display_manager import BoundsCheckingDisplayManager -from .loading import load_config_defaults, load_manifest, merge_config +from .loading import build_config, load_manifest from .sizes import DEFAULT_TEST_SIZES, safe_mode_filename, size_label logger = get_logger("[Plugin Harness]") @@ -305,9 +305,7 @@ def render_plugin_matrix( manifest = load_manifest(plugin_dir) # Start from config_schema.json defaults so the plugin behaves like a real # install; explicit caller config still wins over a schema default. - config = merge_config( - merge_config({"enabled": True}, load_config_defaults(plugin_dir)), - config or {}) + config = build_config(plugin_dir, config) sizes = sizes or DEFAULT_TEST_SIZES results: List[RenderResult] = [] diff --git a/src/plugin_system/testing/loading.py b/src/plugin_system/testing/loading.py index a7992231..2a0dd8e0 100644 --- a/src/plugin_system/testing/loading.py +++ b/src/plugin_system/testing/loading.py @@ -42,29 +42,6 @@ def load_manifest(plugin_dir: Union[str, Path]) -> Dict[str, Any]: return json.load(f) -def _defaults_from_properties(properties: Dict[str, Any]) -> Dict[str, Any]: - """Defaults for one `properties` block, recursing into nested objects. - - An object property carries its defaults on its children, not on itself, so - reading only the top level dropped everything nested. That is most of the - fleet: config organised by league, or under customization/display_options, - lost 2,386 defaults across 37 of 44 plugins -- soccer-scoreboard alone lost - 539 of 565 -- and the harness rendered them with a config no install would - ever have. - """ - defaults: Dict[str, Any] = {} - for key, prop in (properties or {}).items(): - if not isinstance(prop, dict): - continue - if prop.get('type') == 'object' and isinstance(prop.get('properties'), dict): - nested = _defaults_from_properties(prop['properties']) - if nested: - defaults[key] = nested - elif 'default' in prop: - defaults[key] = prop['default'] - return defaults - - def merge_config(base: Dict[str, Any], override: Dict[str, Any]) -> Dict[str, Any]: """Deep-merge override onto base, without dropping sibling defaults. @@ -81,14 +58,45 @@ def merge_config(base: Dict[str, Any], override: Dict[str, Any]) -> Dict[str, An return merged -def load_config_defaults(plugin_dir: Union[str, Path]) -> Dict[str, Any]: - """Extract default values from a plugin's config_schema.json (empty if none).""" +def load_schema(plugin_dir: Union[str, Path]) -> Optional[Dict[str, Any]]: + """A plugin's config_schema.json, or None when it has none.""" schema_path = Path(plugin_dir) / 'config_schema.json' if not schema_path.exists(): - return {} + return None with open(schema_path, 'r', encoding='utf-8') as f: - schema = json.load(f) - return _defaults_from_properties(schema.get('properties', {})) + return json.load(f) + + +def load_config_defaults(plugin_dir: Union[str, Path]) -> Dict[str, Any]: + """Default values from a plugin's config_schema.json (empty if none). + + The device's own extraction (schema_manager.extract_schema_defaults), so a + harness run starts from the config an install would have: nested objects + contribute their children's defaults (organised-by-league configs lost + thousands of them when only the top level was read), and arrays without a + default start as []. + """ + from src.plugin_system.schema_manager import extract_schema_defaults + schema = load_schema(plugin_dir) + return extract_schema_defaults(schema) if schema else {} + + +def build_config(plugin_dir: Union[str, Path], + overrides: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: + """The config a device would give this plugin, with ``overrides`` applied. + + Starts from a forced ``enabled: True``, deep-merges ``overrides`` onto it + (merge_config), then prepares the result exactly as the device does when it + loads a plugin: legacy booleans read as objects, and schema plus core + defaults filled in (schema_manager.prepare_plugin_config). Used by the + harness, check_plugin, render_plugin and the dev preview server. + """ + from src.plugin_system.schema_manager import ( + plugin_config_defaults, prepare_plugin_config, + ) + schema = load_schema(plugin_dir) + requested = merge_config({"enabled": True}, overrides or {}) + return prepare_plugin_config(requested, schema, plugin_config_defaults(schema)) def load_harness_spec(plugin_dir: Union[str, Path]) -> Dict[str, Any]: @@ -143,16 +151,14 @@ def build_full_config( Merge order: config_schema.json defaults, then a forced ``enabled: True``, then harness.json's config overlay, then the caller's explicit config -- - most specific wins. `enabled` is re-asserted *after* the schema defaults - so a plugin that reasonably ships `enabled: false` (e.g. a seasonal or - opt-in plugin) can't silently make every harness run test "disabled, do - nothing" by accident -- callers that genuinely want to test the disabled - path can still do so via `cli_config={"enabled": False}`. + most specific wins, and each layer deep-merges (a nested override such as + ``{"nhl": {"enabled": true}}`` keeps the other nhl defaults). `enabled` is + asserted over the schema defaults so a plugin that reasonably ships + `enabled: false` (e.g. a seasonal or opt-in plugin) can't silently make + every harness run test "disabled, do nothing" by accident -- callers that + genuinely want to test the disabled path can still do so via + `cli_config={"enabled": False}`. See build_config. """ spec = spec or {} - config: Dict[str, Any] = {} - config.update(load_config_defaults(plugin_dir)) - config["enabled"] = True - config.update(spec.get("config", {})) - config.update(cli_config or {}) - return config + overrides = merge_config(spec.get("config", {}) or {}, cli_config or {}) + return build_config(plugin_dir, overrides) diff --git a/src/startup_validator.py b/src/startup_validator.py index 7bf15a11..954c751f 100644 --- a/src/startup_validator.py +++ b/src/startup_validator.py @@ -8,6 +8,7 @@ Fails fast with clear error messages to prevent runtime issues. import os from typing import Any, List, Optional, Tuple from pathlib import Path +from src.core_config_keys import CORE_CONFIG_KEYS from src.exceptions import ConfigError, PluginError, CacheError from src.logging_config import get_logger @@ -275,8 +276,9 @@ class StartupValidator: # Check for enabled plugins that don't exist for plugin_id, plugin_config in config.items(): - # Skip non-plugin config sections - if plugin_id in ['display', 'schedule', 'timezone', 'plugin_system']: + # Skip core sections: auto_update and dim_schedule have an + # 'enabled' key too, and are not plugins that went missing. + if plugin_id in CORE_CONFIG_KEYS: continue if not isinstance(plugin_config, dict): diff --git a/src/vegas_mode/config.py b/src/vegas_mode/config.py index f1039f5b..a84b3321 100644 --- a/src/vegas_mode/config.py +++ b/src/vegas_mode/config.py @@ -155,9 +155,14 @@ class VegasModeConfig: target_fps: int = 125 # Target frame rate buffer_ahead: int = 2 # Number of plugins to buffer ahead - # Scroll behavior + # Scroll behavior. Neither key steps the scroll or sets a frame rate: + # motion is always by elapsed time at scroll_speed px/s. With + # frame_based_scrolling the speed is first converted to px per + # scroll_delay and clamped to 0.1-5 (ScrollHelper.set_scroll_speed), so + # the speed actually applied is clamp(scroll_speed * scroll_delay, 0.1, 5) + # / scroll_delay -- at the 0.02 default, speeds under 5 px/s run at 5. frame_based_scrolling: bool = True - scroll_delay: float = 0.02 # 50 FPS effective scroll updates + scroll_delay: float = 0.02 # only feeds the clamp above; not a frame period # Dynamic duration dynamic_duration_enabled: bool = True diff --git a/src/vegas_mode/render_pipeline.py b/src/vegas_mode/render_pipeline.py index 8aceabe1..d9b713f5 100644 --- a/src/vegas_mode/render_pipeline.py +++ b/src/vegas_mode/render_pipeline.py @@ -127,12 +127,14 @@ class RenderPipeline: self.scroll_helper.set_sub_pixel_scrolling(self.config.smooth_scroll) # Config scroll_speed is always pixels per second, but ScrollHelper - # interprets it differently based on frame_based_scrolling mode: - # - Frame-based: pixels per frame step - # - Time-based: pixels per second + # takes it in different units depending on frame_based_scrolling: + # - Frame-based: pixels per scroll_delay seconds (clamped to 0.1-5) + # - Time-based: pixels per second (clamped to 1-500) + # Both modes then advance by elapsed time; frame-based mode does not + # step. So frame-based with scroll_delay only adds the clamp: the + # applied speed is clamp(scroll_speed * scroll_delay, 0.1, 5) / + # scroll_delay px/s. if self.config.frame_based_scrolling: - # Convert pixels/second to pixels/frame - # pixels_per_frame = pixels_per_second * seconds_per_frame pixels_per_frame = self.config.scroll_speed * self.config.scroll_delay self.scroll_helper.set_scroll_speed(pixels_per_frame) else: diff --git a/systemd/README.md b/systemd/README.md index a3a8cd5d..cdc65ec6 100644 --- a/systemd/README.md +++ b/systemd/README.md @@ -52,9 +52,11 @@ This directory contains systemd service unit files for LEDMatrix services. ## Installation These service files are installed by the installation scripts in `scripts/install/`: -- `install_service.sh` installs `ledmatrix.service` -- `install_web_service.sh` installs `ledmatrix-web.service` and the - `ledmatrix-update-verify` service and path units +- `install_service.sh` installs `ledmatrix.service`, `ledmatrix-web.service` + and the `ledmatrix-update-verify` service and path units, then enables and + starts them +- `install_web_service.sh` installs only `ledmatrix-web.service` and the + `ledmatrix-update-verify` units - `install_wifi_monitor.sh` installs `ledmatrix-wifi-monitor.service` - `install_dns_fix.sh` installs `ledmatrix-dns-fix.service` (opt-in, not run by the normal installer) diff --git a/test/js/README.md b/test/js/README.md index 9d7c3b95..691b5d08 100644 --- a/test/js/README.md +++ b/test/js/README.md @@ -35,11 +35,12 @@ nothing is listening, so it stays useful in a bare checkout. | Suite | Needs a server | Covers | |---|---|---| | `unit/test_list_filter.js` | no | `ListFilter` search/filter/sort/count/sticky, and the installed-plugins config **extracted verbatim** from `plugins_manager.js` so the test can't drift from it | -| `unit/test_update_all.js` | no | `PluginInstallManager.updateAll` from `plugins/install_manager.js`: Check & Update All sends only plugin ids (never `starlark:` app entries), and re-sends a request that got no HTTP answer (web service restarting) instead of skipping that plugin. Also run by `test/web_interface/test_update_all_plugins.py` so CI covers it | +| `unit/test_update_all.js` | no | `PluginInstallManager.updateAll` from `plugins/install_manager.js`: Check & Update All sends only plugin ids (never `starlark:` app entries), re-sends a request that got no HTTP answer (web service restarting) instead of skipping that plugin, never re-sends one that got any HTTP answer (the real `api_client.js` classifies a proxy 502 or a JSON error without `error_code` as `API_ERROR`), and counts a no-op update as already up to date in the summary. Also run by `test/web_interface/test_update_all_plugins.py` so CI covers it | | `unit/test_render_cards.js` | no | `renderInstalledCards` markup, both empty states, and HTML-escaping of hostile plugin metadata | | `unit/test_style_editor_element_keys.js` | no | `elementKeys()`/`styleRows()`/`positionRows()` from `widgets/style-editor.js`: every `customization.layout` entry gets exactly one row -- paired with its style element through core's `x-layout-key` (so `score` belongs to `score_text`, not a second row), or a position row of its own, leaves included -- since the widget claims the whole `layout` block from the generic fallback renderer | | `unit/test_style_editor_layout_leaf_columns.js` | no | `columnsFor()` from `widgets/style-editor.js`: a layout-only key whose own value is a leaf (no x/y sub-object, e.g. a `show_logo` toggle) gets a self-keyed column instead of a blank, uneditable row | | `unit/test_style_editor_layout_leaf_collision.js` | no | `columnsFor()` from `widgets/style-editor.js`: a layout-only leaf key still gets its own column even when its name collides with an unrelated element's style sub-field or another layout axis's sub-field | +| `unit/test_inline_handler_escaping.js` | no | The store, saved-repository and custom-registry inline `onclick` handlers and the live `window.updateImageList` from `plugins_manager.js`: a registry id, URL or uploaded file name carrying `'`, `"` or entities adds no attributes and reaches the handler intact, and the store's View button opens only http(s) links | | `dom/test_installed_dom.js` | yes | The toolbar in a real DOM: pill/search/sort interaction, the HTMX partial re-swap, and a `getComputedStyle` check that `.filter-pill[data-active]` really matches the emitted markup | | `dom/test_store_dom.js` | yes | Store pagination, per-page, category, tri-state Installed button, and persistence across a re-boot, against the live registry | | `dom/test_no_double_fetch.js` | yes | Loads the **whole** `plugins_manager.js` and counts requests: typing in the store search must filter the cached list, not refetch `/api/v3/plugins/store/list` | diff --git a/test/js/run_all.js b/test/js/run_all.js index bf9ca625..abf0fff7 100755 --- a/test/js/run_all.js +++ b/test/js/run_all.js @@ -18,7 +18,7 @@ const UNIT = ['unit/test_list_filter.js', 'unit/test_render_cards.js', 'unit/test_html_escaping.js', 'unit/test_style_editor_element_keys.js', 'unit/test_style_editor_layout_leaf_columns.js', 'unit/test_style_editor_layout_leaf_collision.js', - 'unit/test_update_all.js']; + 'unit/test_update_all.js', 'unit/test_inline_handler_escaping.js']; const DOM = ['dom/test_installed_dom.js', 'dom/test_store_dom.js', 'dom/test_no_double_fetch.js', 'dom/test_tools_sections.js']; diff --git a/test/js/unit/test_inline_handler_escaping.js b/test/js/unit/test_inline_handler_escaping.js new file mode 100644 index 00000000..984d4299 --- /dev/null +++ b/test/js/unit/test_inline_handler_escaping.js @@ -0,0 +1,215 @@ +// Registry- and upload-supplied strings must not escape the inline handlers +// plugins_manager.js builds for them. +// +// The store, saved-repository and custom-registry renderers wrote +// +//