From 3967a6cffc2b6425c06a85c704bcb93826fe5a6c Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 24 Sep 2026 17:31:41 -0400 Subject: [PATCH] fix(security): re-harden root sudo helpers; installer fixes; ARCHITECTURE and PERMISSIONS docs (#640) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: add ARCHITECTURE and PERMISSIONS guides ARCHITECTURE.md maps the processes, the state the display and web services share through the cache, the display loop, the plugin system, the web UI and the update path, with links into the code and a where-to-start table. PERMISSIONS.md lists who owns what after install, both sudoers files (and why iptables is not granted), the polkit rule, and which scripts/fix_perms script to run as which user. Both are linked from the docs index, along with the MQTT bridge README and src/common/README.md. Co-Authored-By: Claude Opus 5.5 * docs: correct stale setup, service and troubleshooting claims - README: quick actions run systemctl on ledmatrix.service (run.py), not display_controller.py; use_short_date_format has no effect; the installer uses system pip with --break-system-packages, not a venv. - CONFIG_DEBUGGING: LEDMATRIX_DEBUG must be "true"; logs are in journald. - GETTING_STARTED, WEB_INTERFACE_GUIDE, TROUBLESHOOTING: enabling a plugin, plugin settings, brightness and Vegas settings apply without a restart; matrix hardware settings still need one. - TROUBLESHOOTING: install dependencies with sudo so the root service sees them; point permission problems at PERMISSIONS.md instead of a project-wide chown. - ADVANCED_FEATURES: real BackgroundDataService stats keys; Vegas hooks return VegasDisplayMode and None falls back to capture; cache files are 0660; fix_web_permissions.sh runs as the web user and does not touch sudoers. - STARLARK_APPS_GUIDE: only the linux-arm64 pixlet binary is downloaded. - HOW_TO_RUN_TESTS: test class examples that exist. - CLAUDE.md: PluginStoreManager, plugin_dirs.py, monorepo installs via the Trees API with ZIP fallback, requirements.txt is optional. Co-Authored-By: Claude Opus 5.5 * docs: mark deprecated plugin APIs and state manifest fields once Methods @deprecated("3.7.0") (the set pinned in test_deprecation.py) were shown as current API in the quick reference, API reference, advanced guide, development guide and FONT_MANAGER. Each is now marked deprecated with its replacement. FONT_MANAGER is rewritten around the current API; the override editor is gone and override methods are deprecated. Required manifest fields were stated three different ways. The API reference now has one section: the 7 schema-required fields, the 4 the store refuses without, class_name for the loader, and the 8 to set. The other guides link to it. Co-Authored-By: Claude Opus 5.5 * docs: document every src/common module and every widget - src/common/README.md covered 7 of 17 modules. It now has a table of all of them (purpose, whether plugins import it, release to floor on), a short entry each, and logging advice that matches the code. - SPORTS_UNIFICATION listed two shared modules and called sports_helpers the first; it now lists all six. - The widgets README lists all 28 registered widgets plus the support files, and absorbs the parts that only docs/widget-guide.md had (x-options.labels, x-advanced, x-display hidden, plugin-file-manager). docs/widget-guide.md is now a pointer to it. Co-Authored-By: Claude Opus 5.5 * fix(security): fix_web_permissions.sh re-hardens the root sudo helpers The script chowns the whole project to the web user. That included scripts/fix_perms/safe_plugin_rm.sh and safe_pip_install.sh -- the two helpers /etc/sudoers.d/ledmatrix_web lets the web user run as root -- so running it turned both into a root shell for whoever can edit them. It also re-grouped config_secrets.json away from ledmatrix. After the chown it now does what first_time_install.sh's Steps 11 and 11.1 do: helpers back to root:root 755, and config_secrets.json back to the web unit's User=:ledmatrix 640. Each step is non-fatal and prints the manual command if it fails. Also fixes what the script and its docs claimed: it never configured sudoers, its closing hint pointed at ./configure_web_sudo.sh (wrong path), and the README and ADVANCED_FEATURES.md said to run it with sudo, which it refuses. Co-Authored-By: Claude Opus 5.5 * fix(security): validate and harden every sudoers drop-in the scripts write configure_wifi_permissions.sh copied its rules into /etc/sudoers.d/ledmatrix_wifi without `visudo -c`. A malformed drop-in makes sudo refuse every command for every user, which on a headless Pi leaves no way back in. It now checks first and leaves the installed file alone when the rules do not parse, as the other two writers do. (It already used mktemp, so that part of the review did not apply.) It also grants the two literal commands wifi_manager.py runs for NetworkManager's shared-mode dnsmasq drop-in -- `cp /tmp/ledmatrix-nm-dnsmasq.conf .../dnsmasq-shared.d/ledmatrix-captive.conf` and `rm -f` of that file. The directory's mkdir was granted, the file was not. Both are pinned in test_sudo_allowlist_covers_calls.py. configure_web_sudo.sh wrote its rules to /tmp/ledmatrix_web_sudoers_$$, a predictable name in a world-writable directory; it now uses mktemp with an EXIT trap, as first_time_install.sh does. It sets mode 440 on the installed file instead of leaving the temp file's mode, and finds visudo in /usr/sbin when that is not on the user's PATH, which skipped the check silently. Co-Authored-By: Claude Opus 5.5 * fix(install): escape the project path in the DNS-fix and MQTT unit renderers install_dns_fix.sh and install_mqtt_bridge.sh substituted __PROJECT_ROOT_DIR__ with the raw path, while the other three renderers go through sed_escape_replacement from lib_systemd_render.sh. A checkout under a path containing `&`, `\` or `|` rendered a corrupted unit from these two only. Both now source the helper and use it, and a test checks that every placeholder substitution in scripts/install uses an escaped value. Co-Authored-By: Claude Opus 5.5 * fix(install): stop the installer scripts reporting things that are not true - first_time_install.sh printed "Password: ledmatrix123" for the setup access point. wifi_manager creates it as an open network ("No password" on the panel), so it now says so. - Step 10.1 printed "✓ WiFi management permissions configured" straight after its own failure message; install_wifi_monitor.sh printed "✓ Package installation completed" after a failed apt install. The tick now only follows success. - Step 7 printed "Web dependencies already installed ... in Step 5" in the one branch that runs because Step 5 did not install them, then created .web_deps_installed on that basis. It now warns and leaves the marker off so the next run retries, as the comment below it intends. - check_system_compatibility.sh called Debian 12 Bookworm "full compatibility confirmed" while first_time_install.sh refuses anything but Debian 13. Bookworm, older Debian and non-Debian systems are now errors. Its counters used ((X++)), which under `set -e` exits the script at the first warning or error (the expression is 0), so the check never reached its summary on any system with one. - configure_web_sudo.sh and configure_wifi_permissions.sh finished by testing `sudo -n test -f ...` and `sudo -n nmcli device status`, neither of which is granted, so they always reported a failure. They now ask `sudo -n -l` about commands the new rules do grant, which checks the rule without running anything. Co-Authored-By: Claude Opus 5.5 * fix(install): print the completion summary before rebooting With -y -- and so for every one-shot `curl | bash` install, which always passes -y -- first_time_install.sh ran `reboot` about 180 lines before its "Installation Complete / Web UI Access" summary. reboot returns at once, so the summary printed while the Pi was going down and the SSH session usually dropped before the web UI address could be read. The reboot block moves, unchanged, to the very end of the script. The interactive prompt now also follows the summary. Because the summary now runs before the -y reboot, its one command that could fail under `set -Eeuo pipefail` (the SSID lookup, when nmcli reports a connected device but no active network line) gets `|| true`; a missing SSID was already handled as "SSID unknown". one-shot-install.sh prints its "Next steps" after the installer returns, by which time the reboot is under way, so it now says so, and README's Quick Install mentions the automatic reboot. Co-Authored-By: Claude Opus 5.5 * chore(scripts): correct wrong comments and messages, drop dead code No behaviour change except the output text noted below. - 2775 is setgid, not the sticky bit (first_time_install.sh Step 3.1, fix_plugin_permissions.sh), and root needs no "PWM hardware access" to plugin files. - The 777 comments in first_time_install.sh Step 3's fallback and fix_assets_permissions.sh said root needs it to write. Root ignores mode bits; the comments now say what 777 actually opens. The 777 itself is unchanged. - apt_remove ends in `|| true`, so Step 12's "Some packages could not be removed" branch could never run; it is gone and the helper stays non-fatal. - detect_web_service_user's comment named Step 8 for the web unit (install_service.sh installs it in Step 7.5) and now says which branch actually runs. - Step 5 described an "already installed" check that does not exist; the ACTUAL_USER comment described the re-exec backwards. - on_error printed a literal "\n" before "Common fixes:". - Dead code: one-shot-install.sh's uncalled fix_tmp_permissions, LEDMATRIX_ELEVATED=1 (never read) on the sudo re-exec, and configure_web_sudo.sh's unused PYTHON_PATH, which also made a missing python3 fatal for rules that never mention it. - start_display.sh / stop_display.sh said "for user: "; the service runs as root. Co-Authored-By: Claude Opus 5.5 * refactor(fix_perms): fix_cache_permissions.sh uses setup_cache.sh's model There were two models for /var/cache/ledmatrix. setup_cache.sh (the installer's Step 2) and install_web_service.sh share it through the ledmatrix group: root:ledmatrix, 2775, files 660, which is also what DiskCache relies on to give files the directory's group. fix_cache_permissions.sh instead made it 777 and re-grouped it to the invoking user's group, undoing that. It now runs setup_cache.sh for /var/cache/ledmatrix and keeps its own handling of ~/.ledmatrix_cache. Dropped: /var/cache/ledmatrix/ placeholder_logos (nothing reads it) and the checks against the `daemon` user (no service runs as daemon). Co-Authored-By: Claude Opus 5.5 * ci: pin actions/checkout in the Claude workflows, drop template comments claude.yml and claude-code-review.yml used actions/checkout@v4 while test.yml and release-version-check.yml pin the v4.2.2 commit SHA; they now pin the same SHA. The commented-out starter-template settings (prompt, claude_args, paths, author filter) are removed. Co-Authored-By: Claude Opus 5.5 * docs(scripts): index every script and list removal candidates New scripts/README.md gives one line per top-level script and scripts directory, marked keep, dev-only or diagnostic, and lists the eight scripts nothing in the repo refers to as candidates for removal (kept for now). The install, utils and dev READMEs now list the files they were missing. Co-Authored-By: Claude Opus 5.5 * test: tighten two checks that mutation testing showed were too loose - The wifi sudoers check matched `visudo -c -f "$TEMP_SUDOERS"` in the error report too, so replacing the check with `if false` still passed. It now requires the command as the condition. - The summary test never had the setup access point up, so reinstating the bogus "Password: ledmatrix123" line went unnoticed. A case with hostapd active now checks the AP is described as open. Co-Authored-By: Claude Opus 5.5 * docs(permissions): describe the repaired fix_perms scripts and new WiFi grants Co-Authored-By: Claude Opus 5.5 * docs(changelog): docs-scripts Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- .github/workflows/claude-code-review.yml | 16 +- .github/workflows/claude.yml | 10 +- CHANGELOG.md | 14 + CLAUDE.md | 17 +- README.md | 29 +- docs/ADVANCED_FEATURES.md | 129 ++- docs/ADVANCED_PLUGIN_DEVELOPMENT.md | 96 +-- docs/ARCHITECTURE.md | 215 +++++ docs/CONFIG_DEBUGGING.md | 16 +- docs/DEVELOPER_QUICK_REFERENCE.md | 15 +- docs/FONT_MANAGER.md | 449 +++------- docs/GETTING_STARTED.md | 21 +- docs/HOW_TO_RUN_TESTS.md | 10 +- docs/PERMISSIONS.md | 154 ++++ docs/PLUGIN_API_REFERENCE.md | 149 ++-- docs/PLUGIN_CONFIG_ARCHITECTURE.md | 6 +- docs/PLUGIN_DEVELOPMENT_GUIDE.md | 13 +- docs/PLUGIN_QUICK_REFERENCE.md | 5 +- docs/PLUGIN_REGISTRY_SETUP_GUIDE.md | 6 +- docs/README.md | 10 +- docs/SPORTS_UNIFICATION.md | 20 +- docs/STARLARK_APPS_GUIDE.md | 17 +- docs/TROUBLESHOOTING.md | 59 +- docs/WEB_INTERFACE_GUIDE.md | 14 +- docs/widget-guide.md | 593 +------------- first_time_install.sh | 88 +- scripts/README.md | 59 ++ scripts/check_system_compatibility.sh | 22 +- scripts/dev/README.md | 2 + scripts/fix_perms/README.md | 27 +- scripts/fix_perms/fix_assets_permissions.sh | 10 +- scripts/fix_perms/fix_cache_permissions.sh | 94 +-- scripts/fix_perms/fix_plugin_permissions.sh | 11 +- scripts/fix_perms/fix_web_permissions.sh | 56 +- scripts/install/README.md | 15 + scripts/install/configure_web_sudo.sh | 46 +- scripts/install/configure_wifi_permissions.sh | 31 +- scripts/install/install_dns_fix.sh | 7 +- scripts/install/install_mqtt_bridge.sh | 7 +- scripts/install/install_wifi_monitor.sh | 7 +- scripts/install/lib_systemd_render.sh | 7 +- scripts/install/one-shot-install.sh | 30 +- scripts/utils/README.md | 1 + src/common/README.md | 254 +++++- start_display.sh | 5 +- stop_display.sh | 5 +- test/test_fix_web_permissions_rehardens.py | 109 +++ test/test_install_reboot_is_last.py | 134 +++ test/test_sudo_allowlist_covers_calls.py | 11 +- test/test_sudoers_is_validated.py | 49 ++ test/test_systemd_unit_drift.py | 19 + web_interface/static/v3/js/widgets/README.md | 772 ++++++------------ 52 files changed, 1957 insertions(+), 2004 deletions(-) create mode 100644 docs/ARCHITECTURE.md create mode 100644 docs/PERMISSIONS.md create mode 100644 scripts/README.md create mode 100644 test/test_fix_web_permissions_rehardens.py create mode 100644 test/test_install_reboot_is_last.py diff --git a/.github/workflows/claude-code-review.yml b/.github/workflows/claude-code-review.yml index 4d3c8dad..7c432109 100644 --- a/.github/workflows/claude-code-review.yml +++ b/.github/workflows/claude-code-review.yml @@ -3,21 +3,9 @@ name: Claude Code Review on: pull_request: types: [opened, synchronize, ready_for_review, reopened] - # Optional: Only run on specific file changes - # paths: - # - "src/**/*.ts" - # - "src/**/*.tsx" - # - "src/**/*.js" - # - "src/**/*.jsx" jobs: claude-review: - # Optional: Filter by PR author - # if: | - # github.event.pull_request.user.login == 'external-contributor' || - # github.event.pull_request.user.login == 'new-developer' || - # github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR' - runs-on: ubuntu-latest permissions: contents: read @@ -27,7 +15,7 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with: fetch-depth: 1 @@ -45,6 +33,4 @@ jobs: plugin_marketplaces: 'https://github.com/anthropics/claude-code.git' plugins: 'code-review@claude-code-plugins' prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}' - # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md - # or https://code.claude.com/docs/en/cli-reference for available options diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index 6b15fac7..cd9e45a9 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -26,7 +26,7 @@ jobs: actions: read # Required for Claude to read CI results on PRs steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 with: fetch-depth: 1 @@ -40,11 +40,3 @@ jobs: additional_permissions: | actions: read - # Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it. - # prompt: 'Update the pull request description to include a summary of changes.' - - # Optional: Add claude_args to customize behavior and configuration - # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md - # or https://code.claude.com/docs/en/cli-reference for available options - # claude_args: '--allowed-tools Bash(gh pr *)' - diff --git a/CHANGELOG.md b/CHANGELOG.md index d4620e98..81da2d21 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,20 @@ accepts both, but the store flags the old spelling as deprecated ## Unreleased +- Scripts and installer: + - `fix_web_permissions.sh` makes `safe_plugin_rm.sh` and `safe_pip_install.sh` root-owned again after resetting ownership. A web-user-owned copy of either is a root shell, since sudo lets the web user run them as root. It also restores `config_secrets.json` to mode 640. + - `configure_wifi_permissions.sh` checks its rules with `visudo -c` before installing them, and grants the NetworkManager captive-portal `cp` and `rm` commands `wifi_manager` runs. + - `configure_web_sudo.sh` uses a random temp file and installs its rules with mode 440. + - The installer prints its completion summary before the `-y` reboot, and describes the setup access point as an open network (it was shown with a password it doesn't have). + - `fix_cache_permissions.sh` applies `setup_cache.sh`'s `ledmatrix`-group model instead of setting 777. + - `check_system_compatibility.sh` reports anything but Debian 13 (Trixie) as unsupported, and reaches its summary. + - New `scripts/README.md` lists every script. +- Docs: + - New `docs/ARCHITECTURE.md` (processes, shared state, display loop, plugin system, web UI) and `docs/PERMISSIONS.md` (owners, modes, both sudoers files, repair scripts). + - Deprecated plugin APIs are marked in the plugin docs. + - `src/common/README.md` covers every module. + - Stale setup, service and troubleshooting claims are corrected. + - The web service (`ledmatrix-web`) logs through `src.logging_config` like the display service, so `journalctl -p err -u ledmatrix-web` works. Successful GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of diff --git a/CLAUDE.md b/CLAUDE.md index 714ff7b0..e5bb33b0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,14 +13,17 @@ loader does NOT fall back to it — `PluginManager.discover_plugins()` (`src/plugin_system/plugin_manager.py`) scans only the configured directory. Fallbacks exist in two narrower places: store operations - (`StoreManager._find_plugin_path()` in `store_manager.py`) and schema - lookup (`SchemaManager.get_schema_path()` in `schema_manager.py`, - which probes `plugins/` *before* `plugin-repos/`). + (`PluginStoreManager._find_plugin_path()` in `store_manager.py`, which + searches `store_search_dirs()` from `plugin_dirs.py`) and schema lookup + (`SchemaManager.get_schema_path()` in `schema_manager.py`, which probes + `plugins/` *before* `plugin-repos/`). +- `src/plugin_system/plugin_dirs.py` — the one resolver for "which directory + holds plugin X" (manifest `id` first, then `` / `ledmatrix-`) ## Plugin System - Plugins inherit from `BasePlugin` in `src/plugin_system/base_plugin.py` - Required abstract methods: `update()`, `display(force_clear=False)` -- Each plugin needs: `manifest.json`, `config_schema.json`, `manager.py`, `requirements.txt` +- Each plugin needs: `manifest.json`, `config_schema.json`, and the entry point (`manager.py` by default); `requirements.txt` if it has dependencies. Required manifest fields: `docs/PLUGIN_API_REFERENCE.md#manifest-required-fields` - Plugin instantiation args: `plugin_id, config, display_manager, cache_manager, plugin_manager` - Config schemas use JSON Schema Draft-7 - Display dimensions: always read dynamically from `self.display_manager.width/height` — not `display_manager.matrix.width/height`, because `matrix` is `None` when hardware init fails (the properties fall back to the canvas size) @@ -39,14 +42,14 @@ - Official plugins live in the `ledmatrix-plugins` monorepo (not individual repos) - Plugin repo naming convention: `ledmatrix-` (e.g., `ledmatrix-football-scoreboard`) - `plugins.json` registry at `https://raw.githubusercontent.com/ChuckBuilds/ledmatrix-plugins/main/plugins.json` -- Store manager (`src/plugin_system/store_manager.py`) handles install/update/uninstall -- Monorepo plugins are installed via ZIP extraction (no `.git` directory) +- Store manager (`PluginStoreManager` in `src/plugin_system/store_manager.py`) handles install/update/uninstall +- Monorepo plugins are installed without a `.git` directory: GitHub Trees API + raw downloads, falling back to ZIP extraction - Update detection for monorepo plugins uses version comparison (manifest version vs registry latest_version) - Plugin configs stored in `config/config.json`, NOT in plugin directories — safe across reinstalls - Third-party plugins can use their own repo URL with empty `plugin_path` ## Common Pitfalls -- paho-mqtt 2.x needs `callback_api_version=mqtt.CallbackAPIVersion.VERSION1` for v1 compat +- paho-mqtt 2.x requires a `CallbackAPIVersion` argument: `VERSION1` for code written against v1 callback signatures (the MQTT bridge uses `VERSION2`) - BasePlugin uses `get_logger()` from `src.logging_config`, not standard `logging.getLogger()` - `DisplayManager` has no `draw_image()` — paste onto the PIL image directly: `self.display_manager.image.paste(img, (x, y))` then `update_display()` diff --git a/README.md b/README.md index 7ae97dd0..16e66fe7 100644 --- a/README.md +++ b/README.md @@ -328,6 +328,7 @@ This one-shot installer will automatically: - Install required system packages (git, python3, build tools, etc.) - Clone or update the LEDMatrix repository - Run the complete first-time installation script +- Print the web interface address, then **reboot the Pi automatically** (your SSH session will disconnect; give it a few minutes to come back) The installation process typically takes 10-30 minutes depending on your internet connection and Pi model. Pi 3B/3B+ and other 1GB boards land at the top of that range, because the C++ library is compiled serially to stay within available memory. All errors are reported explicitly with actionable fixes. @@ -689,9 +690,10 @@ Controls how long each installed plugin stays visible in seconds before switchin ### Display Format Settings - **`use_short_date_format`** (boolean, default: true) - - Use short date format (e.g., "Jan 15") instead of long format (e.g., "January 15th") - - Set to `false` for longer, more readable dates - - Set to `true` to save space and show more information + - Currently has no effect. The web UI still saves it, but no core code + reads it. Scoreboard plugins that offer a short date format read the + setting from their own plugin config instead. See + [CONFIG_REFERENCE.md](docs/CONFIG_REFERENCE.md#display--other-keys). ### Dynamic Duration Settings (`display.dynamic_duration`) @@ -779,15 +781,21 @@ Controls how long each installed plugin stays visible in seconds before switchin
Manual SSH Commands (for reference) -The quick actions essentially just execute the following commands on the Pi. +The web interface's quick actions (Start/Stop/Restart Display) call +`sudo systemctl start|stop|restart ledmatrix.service` — see +`execute_system_action()` in +[`web_interface/blueprints/api_v3/system.py`](web_interface/blueprints/api_v3/system.py). +The service runs [`run.py`](run.py) as root. -From the project root directory (ex: /home/ledpi/LEDMatrix): +To run the display in the foreground instead (for debugging), stop the service +first, then from the project root (e.g. `/home/ledpi/LEDMatrix`): ```bash -sudo python3 display_controller.py +sudo systemctl stop ledmatrix.service +sudo python3 run.py # add -d for debug logging ``` -This will start the display cycle but only stays active as long as your ssh session is active. +This only runs as long as your SSH session stays open. ### Convenience Scripts @@ -957,9 +965,10 @@ sudo systemctl enable ledmatrix-web.service 3. Check if another service is using port 5000 **Service Fails to Start:** -1. Check Python dependencies are installed -2. Verify the virtual environment is set up correctly -3. Check file permissions and ownership +1. Check Python dependencies are installed. The installer puts them in the + system Python with `pip install --break-system-packages` (there is no + virtual environment), so `python3 -c "import flask"` should succeed. +2. Check file permissions and ownership
diff --git a/docs/ADVANCED_FEATURES.md b/docs/ADVANCED_FEATURES.md index 5a9deb07..6c79cc40 100644 --- a/docs/ADVANCED_FEATURES.md +++ b/docs/ADVANCED_FEATURES.md @@ -185,63 +185,53 @@ their config section to control how oversized content is handled (see ### Plugin Integration (Developer Guide) +All of these have defaults in +[`BasePlugin`](../src/plugin_system/base_plugin.py); override only what you +need. + **1. Implement Content Method:** ```python def get_vegas_content(self): - """ - Return PIL Image or list of Images for Vegas mode. - - Returns: - PIL.Image or list[PIL.Image]: Content to display - - Single image: fixed-width content - - List of images: multiple segments - - None: skip this cycle - """ - # Example: Return single wide image - img = Image.new('RGB', (256, 32)) - # ... render your content ... - return img - - # Example: Return multiple segments - return [image1, image2, image3] + # Return a PIL Image, a list of Images, or None. + # A single image is one block; a list becomes one item per image. + return [self._render_game(game) for game in self.games] ``` +If it returns `None` (the default), Vegas falls back to the plugin's +`scroll_helper` image, then to capturing `display()` output +(`PluginAdapter.get_content()` in +[`src/vegas_mode/plugin_adapter.py`](../src/vegas_mode/plugin_adapter.py)). + **2. Specify Content Type:** ```python def get_vegas_content_type(self): - """ - Specify how content should be handled. - - Returns: - str: 'multi' | 'static' | 'none' - """ - return 'multi' # Default for most plugins + # 'multi' | 'static' | 'none' -- default is 'static' + return 'multi' ``` +`'none'` excludes the plugin from Vegas mode. + **3. Optionally Specify Display Mode:** -```python -def get_vegas_display_mode(self): - """ - Preferred display mode for this plugin. +These return `VegasDisplayMode` members, not strings: - Returns: - str: 'scroll' | 'fixed' | 'static' - """ - return 'scroll' +```python +from src.plugin_system.base_plugin import VegasDisplayMode + +def get_vegas_display_mode(self): + return VegasDisplayMode.SCROLL def get_supported_vegas_modes(self): - """ - List of supported modes. - - Returns: - list: ['scroll', 'fixed', 'static'] - """ - return ['scroll', 'static'] + return [VegasDisplayMode.SCROLL, VegasDisplayMode.STATIC] ``` +`VegasDisplayMode` has `SCROLL` (`"scroll"`), `FIXED_SEGMENT` (`"fixed"`) and +`STATIC` (`"static"`). The default `get_vegas_display_mode()` uses the +plugin's `vegas_mode` config value if set, otherwise maps the content type +(`multi` to `SCROLL`, anything else to `FIXED_SEGMENT`). + ### Content Rendering Guidelines **Image Dimensions:** @@ -966,11 +956,16 @@ from src.cache_manager import CacheManager service = get_background_service(CacheManager()) stats = service.get_statistics() -print(f"Active tasks: {stats['active_tasks']}") -print(f"Completed: {stats['completed']}") -print(f"Failed: {stats['failed']}") +print(f"Active: {stats['active_requests']}") +print(f"Completed: {stats['completed_requests']}") +print(f"Failed: {stats['failed_requests']}") ``` +Other keys: `total_requests`, `cached_hits`, `cache_misses`, +`average_fetch_time`, `completed_requests_count` (results currently held in +memory) — see `BackgroundDataService.get_statistics()` in +[`src/background_data_service.py`](../src/background_data_service.py). + **Enable Debug Logging:** ```python import logging @@ -981,6 +976,10 @@ logging.getLogger('src.background_data_service').setLevel(logging.DEBUG) ## 5. Permission Management +Ownership, modes, sudo rules and the repair scripts are listed in +[PERMISSIONS.md](PERMISSIONS.md). This section covers the helpers code uses +to keep files shareable. + ### Overview LEDMatrix uses a dual-user architecture: the display service runs as root (hardware access), while the web interface runs as a non-privileged user. Centralized permission management ensures both can access necessary files. @@ -1044,7 +1043,7 @@ ensure_file_permissions(config_path, get_config_file_mode(config_path)) | Config (secrets) | `rw-r-----` | `0o640` | Owner write, group read | | Assets | `rw-rw-r--` | `0o664` | Owner/group write, all read | | Plugins | `rw-rw-r--` | `0o664` | Owner/group write, all read | -| Cache files | `rw-rw-r--` | `0o664` | Owner/group write, all read | +| Cache files | `rw-rw----` | `0o660` | Owner/group write, no world access (`_CACHE_FILE_MODE` in `src/cache/disk_cache.py`) | **Directory Permissions:** @@ -1115,40 +1114,22 @@ These core utilities **already handle permissions** - you don't need to call per ### Manual Fixes -If you encounter permission issues: +[PERMISSIONS.md](PERMISSIONS.md) lists who owns what on an installed system, +the expected modes, and which `scripts/fix_perms/` script to run as which +user. In short: -```bash -# Targeted permission fixes (see scripts/fix_perms/README.md) -sudo ./scripts/fix_perms/fix_assets_permissions.sh # assets/ tree (logos, fonts) -sudo ./scripts/fix_perms/fix_cache_permissions.sh # all cache directories -sudo ./scripts/fix_perms/fix_plugin_permissions.sh # plugin directories -sudo ./scripts/fix_perms/fix_web_permissions.sh # web interface files +- `fix_assets_permissions.sh`, `fix_cache_permissions.sh` and + `fix_plugin_permissions.sh` are run with `sudo`. +- `fix_web_permissions.sh` is run as the web interface user, without + `sudo` (it refuses to run as root and calls `sudo` itself where needed). + It resets project file ownership for that user, then makes the two + helper scripts the web user may run as root (`safe_plugin_rm.sh`, + `safe_pip_install.sh`) root-owned again and restores `config_secrets.json` + to its owner, the `ledmatrix` group and mode `640`. It does not write + sudoers rules; `scripts/install/configure_web_sudo.sh` does that. -# Fix specific directory -sudo chown -R ledpi:ledpi /home/ledpi/LEDMatrix/config -sudo chmod -R 2775 /home/ledpi/LEDMatrix/config -sudo find /home/ledpi/LEDMatrix/config -type f -exec chmod 664 {} \; - -# Verify permissions -ls -la config/ -ls -la assets/ -``` - -### Verification - -```bash -# Check directory has setgid bit -ls -ld assets/ -# Should show: drwxrwsr-x (note the 's') - -# Check file has correct group -ls -l assets/logo.png -# Should show group 'ledpi' - -# Check file permissions -stat -c "%a %n" config/config.json -# Should show: 644 config/config.json -``` +Do not `chmod` the whole `config/` directory: `config_secrets.json` must stay +`640`. --- diff --git a/docs/ADVANCED_PLUGIN_DEVELOPMENT.md b/docs/ADVANCED_PLUGIN_DEVELOPMENT.md index b92e1649..20d5a792 100644 --- a/docs/ADVANCED_PLUGIN_DEVELOPMENT.md +++ b/docs/ADVANCED_PLUGIN_DEVELOPMENT.md @@ -13,7 +13,7 @@ Advanced patterns, examples, and best practices for developing LEDMatrix plugins - [Using Weather Icons](#using-weather-icons) - [Implementing Scrolling with Deferred Updates](#implementing-scrolling-with-deferred-updates) - [Cache Strategy Patterns](#cache-strategy-patterns) -- [Font Management and Overrides](#font-management-and-overrides) +- [Font Management](#font-management) - [Error Handling Best Practices](#error-handling-best-practices) - [Performance Optimization](#performance-optimization) - [Testing Plugins with Mocks](#testing-plugins-with-mocks) @@ -25,69 +25,12 @@ Advanced patterns, examples, and best practices for developing LEDMatrix plugins ## Using Weather Icons -The Display Manager provides built-in weather icon drawing methods for easy visual representation of weather conditions. - -### Basic Weather Icon Usage - -```python -def display(self, force_clear=False): - if force_clear: - self.display_manager.clear() - - # Draw weather icon based on condition - condition = self.data.get('condition', 'clear') - self.display_manager.draw_weather_icon(condition, x=5, y=5, size=16) - - # Draw temperature next to icon - temp = self.data.get('temp', 72) - self.display_manager.draw_text( - f"{temp}°F", - x=25, y=10, - color=(255, 255, 255) - ) - - self.display_manager.update_display() -``` - -### Supported Weather Conditions - -The `draw_weather_icon()` method automatically maps condition strings to appropriate icons: - -- `"clear"`, `"sunny"` → Sun icon -- `"clouds"`, `"cloudy"`, `"partly cloudy"` → Cloud icon -- `"rain"`, `"drizzle"`, `"shower"` → Rain icon -- `"snow"`, `"sleet"`, `"hail"` → Snow icon -- `"thunderstorm"`, `"storm"` → Storm icon - -### Custom Weather Icons - -For more control, use individual icon methods: - -```python -# Draw specific icons -self.display_manager.draw_sun(x=10, y=10, size=16) -self.display_manager.draw_cloud(x=10, y=10, size=16, color=(150, 150, 150)) -self.display_manager.draw_rain(x=10, y=10, size=16) -self.display_manager.draw_snow(x=10, y=10, size=16) -``` - -### Text with Weather Icons - -Use `draw_text_with_icons()` to combine text and icons: - -```python -icons = [ - ("sun", 5, 5), # Sun icon at (5, 5) - ("cloud", 100, 5) # Cloud icon at (100, 5) -] - -self.display_manager.draw_text_with_icons( - "Weather: Sunny, Cloudy", - icons=icons, - x=10, y=20, - color=(255, 255, 255) -) -``` +The Display Manager's icon methods — `draw_weather_icon()`, `draw_sun()`, +`draw_cloud()`, `draw_rain()`, `draw_snow()` and `draw_text_with_icons()` — +are deprecated, removed in 3.7.0. Draw your own icons instead: render them +onto a PIL image and paste it onto `self.display_manager.image`, or ship +icon images with the plugin. The weather plugin's `WeatherIcons` class is an +example. See [Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis). --- @@ -251,11 +194,8 @@ def update(self): sport_key = "nhl" cache_key = f"{self.plugin_id}_{sport_key}_games" - # Uses sport-specific live_update_interval from config - cached = self.cache_manager.get_background_cached_data( - cache_key, - sport_key=sport_key - ) + # get_background_cached_data() is deprecated, removed in 3.7.0 — use get() + cached = self.cache_manager.get(cache_key, max_age=60) if cached: self.games = cached @@ -282,9 +222,9 @@ def on_config_change(self, new_config): --- -## Font Management and Overrides +## Font Management -Use the Font Manager for advanced font handling and user customization. +The display manager's built-in fonts and text measurement. For fonts shipped with a plugin, see [FONT_MANAGER.md](FONT_MANAGER.md). ### Using Different Fonts @@ -656,14 +596,12 @@ def update(self): ```python def update(self): - # Check if another plugin is enabled - enabled_plugins = self.plugin_manager.get_enabled_plugins() - if "weather" in enabled_plugins: - # Weather plugin is available - weather_plugin = self.plugin_manager.get_plugin("weather") - if weather_plugin: - # Use weather data - pass + # get_enabled_plugins() is deprecated, removed in 3.7.0 — check the + # instance's `enabled` flag instead + weather_plugin = self.plugin_manager.get_plugin("weather") + if weather_plugin is not None and weather_plugin.enabled: + # Use weather data + pass ``` ### Sharing Data Between Plugins diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..6a8918b1 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,215 @@ +# Architecture + +A map of the codebase for a new contributor: which process does what, how +they talk to each other, and where to start reading for common changes. + +## Processes + +| systemd unit | Runs as | Runs | Installed by | +|---|---|---|---| +| `ledmatrix.service` | root | [`run.py`](../run.py) → `DisplayController` | [`install_service.sh`](../scripts/install/install_service.sh) | +| `ledmatrix-web.service` | the installing user | [`start_web_conditionally.py`](../scripts/utils/start_web_conditionally.py) → [`web_interface/start.py`](../web_interface/start.py) (Flask, port 5000) | `install_service.sh`, [`install_web_service.sh`](../scripts/install/install_web_service.sh) | +| `ledmatrix-update-verify.path` / `.service` | the web user | Health check after an automatic update | the same installers, or [`src/auto_update_setup.py`](../src/auto_update_setup.py) at runtime | +| `ledmatrix-wifi-monitor.service` | root | [`wifi_monitor_daemon.py`](../scripts/utils/wifi_monitor_daemon.py) | [`install_wifi_monitor.sh`](../scripts/install/install_wifi_monitor.sh) | +| `ledmatrix-mqtt-bridge.service` | root | [MQTT bridge](../integrations/mqtt_bridge/README.md) (optional) | [`install_mqtt_bridge.sh`](../scripts/install/install_mqtt_bridge.sh) | +| `ledmatrix-dns-fix.service` | root | DNS workaround (optional) | [`install_dns_fix.sh`](../scripts/install/install_dns_fix.sh) | + +Unit templates are in [`systemd/`](../systemd/README.md). The display runs as +root because the LED matrix library needs direct GPIO access. The web +interface runs unprivileged and uses a fixed list of `sudo` rules for the +few privileged things it does; see [PERMISSIONS.md](PERMISSIONS.md). + +`start_web_conditionally.py` exits without starting Flask when +`web_display_autostart` is explicitly false in `config.json`. + +## How the two main processes share state + +The display and the web interface are separate processes that never call +each other. They share three things: + +1. **`config/config.json` and `config/config_secrets.json`.** The web + interface writes them through `ConfigManager` + ([`src/config_manager.py`](../src/config_manager.py)); the display + notices through `ConfigService` (below). +2. **The disk cache**, `/var/cache/ledmatrix` (owned `root:ledmatrix`, + setgid, files `0660`), read and written through `CacheManager` + ([`src/cache_manager.py`](../src/cache_manager.py), + [`src/cache/disk_cache.py`](../src/cache/disk_cache.py)). Readers in the + other process pass `memory_ttl=0` so they do not serve a stale in-memory + copy. +3. **A few files in `/tmp`.** + +| State | Where | Written by | Read by | +|---|---|---|---| +| On-demand request | cache `display_on_demand_request` | web: `start_on_demand_display()` / `stop_on_demand_display()` in [`api_v3/display.py`](../web_interface/blueprints/api_v3/display.py) | display: `_poll_on_demand_requests()` | +| On-demand state | cache `display_on_demand_state` | display: `_publish_on_demand_state()` | web: `/api/v3/display/on-demand/status` | +| Current screen | cache `display_current_state` | display | web: `/api/v3/display/current-status` | +| Plugin errors | cache `plugin_error_snapshot` | display: `ErrorSnapshotPublisher` ([`src/error_aggregator.py`](../src/error_aggregator.py)) | web: `read_error_report()` for `/api/v3/errors/*` | +| Error clear | cache `plugin_error_clear_request` | web | display | +| Font usage | cache `font_usage_snapshot` | display: `FontUsagePublisher` ([`src/font_usage.py`](../src/font_usage.py)) | web: Fonts tab | +| Plugin health | cache `plugin_health:` | display (web writes on reset) | web: `/api/v3/plugins/health` | +| Preview frame | `/tmp/led_matrix_preview.png` | display: `DisplayManager`, gated by [`snapshot_policy`](../src/common/snapshot_policy.py) | web: display SSE stream, `/api/v3/health` (file age) | +| Preview viewer marker | `/tmp/led_matrix_preview_viewer` | web, while a preview is open | display: writes full-rate snapshots only while it is fresh | +| Hardware init status | `/tmp/led_matrix_hw_status.json` | display | web: `/api/v3/hardware/status` | + +The on-demand start route also restarts `ledmatrix.service` by default so the +request takes effect straight away. + +## Display loop + +[`src/display_controller.py`](../src/display_controller.py), class +`DisplayController`. `__init__` loads config, starts the cache and the +error-snapshot publisher, runs the startup validator, creates the +`DisplayManager` ([`src/display_manager.py`](../src/display_manager.py)), +`FontManager` and `PluginManager`, loads the enabled plugins in parallel, +runs an initial `update()` pass within a 20-second budget +(`_INITIAL_UPDATE_BUDGET_SECONDS`; a plugin that misses it is deferred to +the scheduler), and sets up Vegas mode. + +`run()` is the main loop. Each pass, in order: apply a pending plugin +enable/disable, poll on-demand requests, run scheduled plugin updates, check +the on/off schedule and brightness, then show one screen. Priority is +on-demand, then WiFi status messages, then live priority, then Vegas mode, +then normal rotation. + +- **Rotation.** `available_modes` is the ordered list of display modes; + `current_mode_index` advances after each screen. + `_apply_plugin_rotation_order()` applies `display.plugin_rotation_order`. +- **Durations.** `_get_display_duration()`: `display.display_durations[mode]`, + else the plugin's `get_display_duration()`, else 30 s. Plugins that + support dynamic duration run until `is_cycle_complete()`, capped by + `display.dynamic_duration.max_duration_seconds` (default 180 s). +- **On-demand.** A request from the web interface pins one plugin (or mode) + for a duration. `_activate_on_demand()` / `_clear_on_demand()`; the + session is saved under `display_on_demand_config` so it survives a + restart. It also keeps the display on during scheduled off hours. +- **Live priority.** `_check_live_priority()` looks for a plugin whose + `has_live_priority()` and `has_live_content()` are both true and switches + to it, rotating between several live games. +- **Schedule and dim schedule.** `_check_schedule()` reads `schedule`; + `_check_dim_schedule()` reads `dim_schedule` and + `display.hardware.brightness`. Both are re-evaluated once a minute. +- **Long screens.** While a screen is showing (a dwell, a scroll, a Vegas + iteration), `_service_pending_changes()` repeats the on-demand, schedule + and brightness checks every 0.25 s, so a change does not wait for the + screen to end. +- **Config hot reload.** `ConfigService` + ([`src/config_service.py`](../src/config_service.py)) polls the config and + secrets files' mtimes every 2 s and notifies subscribers when the content + changes. The controller refreshes its cached settings; enabling or + disabling a plugin queues `_reconcile_enabled_plugins()`, which loads or + unloads it on the display thread; each plugin gets `on_config_change()` + for its own section. Set `LEDMATRIX_HOT_RELOAD=false` to turn this off. + Matrix hardware settings are only read at start-up. +- **Vegas mode.** [`src/vegas_mode/`](../src/vegas_mode/): the display loop + calls `VegasModeCoordinator.run_iteration()` + ([`coordinator.py`](../src/vegas_mode/coordinator.py)) when + `display.vegas_scroll.enabled` is set. `PluginAdapter` gets each plugin's + content (`get_vegas_content()`, else its `scroll_helper` image, else a + capture of `display()`), `StreamManager` orders it and `RenderPipeline` + scrolls it. See [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md). +- **Multi-display sync.** `DisplaySyncManager` + ([`src/common/sync_manager.py`](../src/common/sync_manager.py)), enabled by + `sync.role`: a leader sends a follower its share of each frame over UDP + (port 5765). + +## Plugin system + +[`src/plugin_system/`](../src/plugin_system/): + +| Area | Where | +|---|---| +| Base class plugins implement | [`base_plugin.py`](../src/plugin_system/base_plugin.py) (`BasePlugin`, `VegasDisplayMode`) | +| Finding a plugin's directory | [`plugin_dirs.py`](../src/plugin_system/plugin_dirs.py): manifest `id` first, then directory `` or `ledmatrix-` | +| Discovery, load, unload, scheduled updates | [`plugin_manager.py`](../src/plugin_system/plugin_manager.py) (`PluginManager`) | +| Import and instantiate | [`plugin_loader.py`](../src/plugin_system/plugin_loader.py) (`PluginLoader.load_plugin()`: dependencies, module, class) | +| Timeouts | [`plugin_executor.py`](../src/plugin_system/plugin_executor.py) (`PluginExecutor`, 30 s default; a timed-out thread is abandoned, not killed) | +| Circuit breaker | [`plugin_health.py`](../src/plugin_system/plugin_health.py) (`PluginHealthTracker`: 3 consecutive failures open the circuit for 300 s) | +| Resource metrics | [`resource_monitor.py`](../src/plugin_system/resource_monitor.py) | +| Config schemas and defaults | [`schema_manager.py`](../src/plugin_system/schema_manager.py) | +| Install, update, uninstall | [`store_manager.py`](../src/plugin_system/store_manager.py) (`PluginStoreManager`) | +| Core-version gate | [`compatibility.py`](../src/plugin_system/compatibility.py) | + +Discovery scans only `plugin_system.plugins_directory` (default +`plugin-repos/`). Scheduled `update()` calls run on one background worker +thread; a per-plugin lock keeps `display()` from running during an update. + +**Store flow.** `install_plugin()` renames any existing copy aside +(`.standalone-backup-preinstall`), installs the new one, and puts the old +copy back if the install fails. Monorepo plugins come from the GitHub Trees +API, falling back to the repository ZIP; other plugins by `git clone` or +download. The manifest is checked (see +[required fields](PLUGIN_API_REFERENCE.md#manifest-required-fields)), the core +version gate runs, then dependencies are installed as root through +`scripts/fix_perms/safe_pip_install.sh`. `update_plugin()` pulls git +installs, undoing a pull whose new version is incompatible, and reinstalls +everything else through `_reinstall_with_rollback()`. + +## Web interface + +- **App.** [`web_interface/app.py`](../web_interface/app.py) builds the + Flask `app` at import time, creates the managers, and registers two + blueprints. `web_interface/start.py` runs it on port 5000. +- **Pages.** [`blueprints/pages_v3.py`](../web_interface/blueprints/pages_v3.py) + serves the shell `templates/v3/base.html` at `/` and each tab as a + partial at `/partials/` (templates in + `web_interface/templates/v3/partials/`). Plugin configuration tabs are + rendered from the plugin's schema by `plugin_config.html`. +- **API.** [`blueprints/api_v3/`](../web_interface/blueprints/api_v3/) is one + blueprint at `/api/v3`, split by area: `backup.py`, `config.py`, + `display.py`, `fonts.py`, `misc.py` (health, logs, errors, cache, sync), + `plugins.py`, `starlark.py`, `system.py` (service actions, updates, git), + `wifi.py`. `__init__.py` defines the blueprint and shared helpers and + imports the modules so their routes register. Endpoints are listed in + [REST_API_REFERENCE.md](REST_API_REFERENCE.md). +- **Front end.** HTMX loads each tab's partial on first open + (`hx-trigger="loadtab"`); Alpine.js holds page state. Scripts are in + `web_interface/static/v3/js/`; form widgets are bundled from + [`js/widgets/`](../web_interface/static/v3/js/widgets/README.md). +- **Server-sent events** (`app.py`): `/api/v3/stream/stats` (CPU, memory, + temperature, service state, every 10 s), `/api/v3/stream/display` (preview + frames when the PNG changes) and `/api/v3/stream/logs` (journal of both + services). One generator thread per stream is shared by all clients. + +## Updates + +- **Update Code** on the Overview tab and the automatic updater both call + `perform_core_update()` in + [`api_v3/system.py`](../web_interface/blueprints/api_v3/system.py): + `git pull --rebase`, reinstall changed requirement files, report whether a + restart is needed. +- **Automatic updates** (`auto_update.enabled`, off by default): + `AutoUpdater` in [`web_interface/auto_update.py`](../web_interface/auto_update.py) + runs in the web process, checks every 30 minutes, and updates at most + weekly between 02:00 and 05:00. Before pulling it copies + [`scripts/utils/auto_update_verify.py`](../scripts/utils/auto_update_verify.py) + to `data/auto_update_verifier.py`, then writes + `data/auto_update_verify.request`. That file triggers + `ledmatrix-update-verify.path`, which runs the verifier as a separate unit + (so restarting the web service does not kill it). The verifier restarts + both services, waits for the web API to answer and the display service to + stay up, and on failure resets to the previous commit and restarts again. + Plugin updates run only after a verified core update. State is in + `data/auto_update_state.json` and `data/auto_update_pending.json`. +- **Startup validator.** `StartupValidator` + ([`src/startup_validator.py`](../src/startup_validator.py)) runs twice in + `DisplayController.__init__`: config and cache directory first, then + enabled plugins once the plugin manager exists. It also warns when an + installed systemd unit differs from its template in `systemd/`. Results + are logged; startup continues either way. + +## Where to start reading + +| Task | Start with | +|---|---| +| Change rotation, durations or priorities | `DisplayController.run()` and `_get_display_duration()` in [`display_controller.py`](../src/display_controller.py) | +| Add a config key | [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md), [`config/config.template.json`](../config/config.template.json), the tab's partial and `api_v3/config.py` | +| Change drawing or fonts | [`display_manager.py`](../src/display_manager.py), [`font_manager.py`](../src/font_manager.py), [`src/common/bdf_font.py`](../src/common/bdf_font.py) | +| Add a plugin-facing API | [`base_plugin.py`](../src/plugin_system/base_plugin.py) or [`src/common/`](../src/common/README.md); document it in [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) | +| Plugin install/update bugs | `PluginStoreManager` in [`store_manager.py`](../src/plugin_system/store_manager.py) | +| A plugin that won't load | `PluginManager.load_plugin()` and `PluginLoader.load_plugin()`; `python3 scripts/check_plugin.py --plugin ` | +| Add an API endpoint | the matching module in [`api_v3/`](../web_interface/blueprints/api_v3/) | +| Add a web UI tab or control | `templates/v3/base.html`, the tab's partial, `pages_v3.py` | +| Vegas scroll | [`src/vegas_mode/coordinator.py`](../src/vegas_mode/coordinator.py) | +| Installer or permissions | [`first_time_install.sh`](../first_time_install.sh), [`scripts/install/`](../scripts/install/), [PERMISSIONS.md](PERMISSIONS.md) | +| Work without a Pi | [DEV_PREVIEW.md](DEV_PREVIEW.md), [EMULATOR_SETUP_GUIDE.md](EMULATOR_SETUP_GUIDE.md), [HOW_TO_RUN_TESTS.md](HOW_TO_RUN_TESTS.md) | diff --git a/docs/CONFIG_DEBUGGING.md b/docs/CONFIG_DEBUGGING.md index 01b41224..4db3d051 100644 --- a/docs/CONFIG_DEBUGGING.md +++ b/docs/CONFIG_DEBUGGING.md @@ -172,10 +172,14 @@ ERROR - Plugin football-scoreboard configuration validation failed: 'api_key' is ### Enable Debug Logging -Set environment variable: +Run the display in the foreground with `-d`, or set `LEDMATRIX_DEBUG=true` +(the value must be `true`; `1` is ignored — see `setup_logging()` in +[`src/logging_config.py`](../src/logging_config.py)): ```bash -export LEDMATRIX_DEBUG=1 -python run.py +sudo systemctl stop ledmatrix.service +sudo python3 run.py -d +# or +sudo LEDMATRIX_DEBUG=true python3 run.py ``` ### Check Merged Configuration @@ -321,8 +325,10 @@ cp config/backups/config.json.backup.20240115_120000_000000 config/config.json ## Getting Help -1. Check logs: `tail -f logs/ledmatrix.log` -2. Enable debug: `LEDMATRIX_DEBUG=1` +1. Check logs. Both services log to journald, not to a file: + `sudo journalctl -u ledmatrix.service -f` (display) and + `sudo journalctl -u ledmatrix-web.service -f` (web interface) +2. Enable debug: `LEDMATRIX_DEBUG=true` or `python3 run.py -d` 3. Check error dashboard: `/api/v3/errors/summary` 4. Validate JSON: https://jsonlint.com/ 5. File an issue: https://github.com/ChuckBuilds/LEDMatrix/issues diff --git a/docs/DEVELOPER_QUICK_REFERENCE.md b/docs/DEVELOPER_QUICK_REFERENCE.md index 7eefb0c9..ed2c92e9 100644 --- a/docs/DEVELOPER_QUICK_REFERENCE.md +++ b/docs/DEVELOPER_QUICK_REFERENCE.md @@ -54,8 +54,8 @@ rows = self.layout.bounds.inset(1).split_v(3, 1, gap=1) self.draw_fit("12:34", rows[0]) # largest crisp font that fits self.draw_image(logo, rows[1], mode="fill_height", crop_to_ink=True) -# Weather icons -display_manager.draw_weather_icon("rain", x=10, y=10, size=16) +# Weather icons: draw_weather_icon() is deprecated, removed in 3.7.0 — +# draw your own icons (the weather plugin ships WeatherIcons) # Scrolling state display_manager.set_scrolling_state(True) @@ -72,20 +72,23 @@ cache_manager.delete("key") # alias for clear_cache(key) # Advanced caching data = cache_manager.get_cached_data_with_strategy("key", data_type="weather") -data = cache_manager.get_background_cached_data("key", sport_key="nhl") # Strategy strategy = cache_manager.get_cache_strategy("weather") -interval = cache_manager.get_sport_live_interval("nhl") ``` +`get_background_cached_data()` (use `get()`) and `get_sport_live_interval()` +are deprecated, removed in 3.7.0. See +[Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis). + ## Plugin Manager Quick Methods ```python # Get plugins plugin = plugin_manager.get_plugin("plugin-id") all_plugins = plugin_manager.get_all_plugins() -enabled = plugin_manager.get_enabled_plugins() +# get_enabled_plugins() is deprecated, removed in 3.7.0 — check `enabled` +# on the entries in plugin_manager.plugins # Get info info = plugin_manager.get_plugin_info("plugin-id") @@ -168,7 +171,7 @@ def display(self, force_clear=False): - [ ] Plugin inherits from `BasePlugin` - [ ] Implements `update()` and `display()` methods -- [ ] `manifest.json` with required fields +- [ ] `manifest.json` with the [required fields](PLUGIN_API_REFERENCE.md#manifest-required-fields) - [ ] `config_schema.json` for web UI (recommended) - [ ] `README.md` with documentation - [ ] Error handling implemented diff --git a/docs/FONT_MANAGER.md b/docs/FONT_MANAGER.md index 0d552687..3390b1ac 100644 --- a/docs/FONT_MANAGER.md +++ b/docs/FONT_MANAGER.md @@ -9,12 +9,14 @@ ## Overview -The enhanced FontManager provides comprehensive font management for the LEDMatrix application with support for: -- Manager font registration and detection -- Plugin font management -- Programmatic per-element font overrides -- Performance monitoring and caching -- Dynamic font discovery +[`src/font_manager.py`](../src/font_manager.py) loads and caches the TTF and +BDF fonts in `assets/fonts/`, registers fonts that plugins ship, and records +which plugin uses which font so the web UI can show it. + +Several methods are deprecated and will be removed in LEDMatrix 3.7.0; they +log a warning on first call. They are listed in +[Deprecated methods](#deprecated-methods) below, and the full set is pinned in +[`test/test_deprecation.py`](../test/test_deprecation.py). ## Getting the FontManager @@ -34,157 +36,60 @@ standalone FontManager when none is available (test harnesses, mocks). `DisplayManager` has **no** `font_manager` attribute — `display_manager.font_manager` raises `AttributeError`. -## Architecture - -### Manager-Centric Design - -Managers define their own fonts, but the FontManager: -1. **Loads and caches fonts** for performance -2. **Detects font usage** for visibility -3. **Allows manual overrides** when needed -4. **Supports plugin fonts** with namespacing - -### Font Resolution Flow - -``` -Manager requests font → Check manual overrides → Apply manager choice → Cache & return -``` - -## For Manager Developers - -### Basic Font Usage +## Resolving a font ```python -from src.font_manager import FontManager +element_key = f"{self.plugin_id}.title" -class MyManager: - def __init__(self, config, display_manager, cache_manager, plugin_manager): - self.display_manager = display_manager - self.font_manager = plugin_manager.font_manager # Shared FontManager - self.manager_id = "my_manager" - - def display(self): - # Define your font choices - element_key = "my_manager.title" - font_family = "press_start" - font_size_px = 10 - color = (255, 255, 255) # RGB white - - # Register your font choice (for detection and future overrides) - self.font_manager.register_manager_font( - manager_id=self.manager_id, - element_key=element_key, - family=font_family, - size_px=font_size_px, - color=color - ) - - # Get the font (checks for manual overrides automatically) - font = self.font_manager.resolve_font( - element_key=element_key, - family=font_family, - size_px=font_size_px - ) - - # Use the font for rendering - self.display_manager.draw_text( - "Hello World", - x=10, y=10, - color=color, - font=font - ) -``` - -### Advanced Font Usage - -```python -class AdvancedManager: - def __init__(self, config, display_manager, cache_manager, plugin_manager): - self.display_manager = display_manager - self.font_manager = plugin_manager.font_manager - self.manager_id = "advanced_manager" - - # Define your font specifications - self.font_specs = { - "title": {"family": "press_start", "size_px": 12, "color": (255, 255, 0)}, - "body": {"family": "four_by_six", "size_px": 8, "color": (255, 255, 255)}, - "footer": {"family": "five_by_seven", "size_px": 7, "color": (128, 128, 128)} - } - - # Register all font specs - for element_type, spec in self.font_specs.items(): - element_key = f"{self.manager_id}.{element_type}" - self.font_manager.register_manager_font( - manager_id=self.manager_id, - element_key=element_key, - family=spec["family"], - size_px=spec["size_px"], - color=spec["color"] - ) - - def get_font(self, element_type: str): - """Helper method to get fonts with override support.""" - spec = self.font_specs[element_type] - element_key = f"{self.manager_id}.{element_type}" - - return self.font_manager.resolve_font( - element_key=element_key, - family=spec["family"], - size_px=spec["size_px"] - ) - - def display(self): - # Get fonts (automatically checks for overrides) - title_font = self.get_font("title") - body_font = self.get_font("body") - footer_font = self.get_font("footer") - - # Render with fonts - self.display_manager.draw_text("Title", font=title_font, color=self.font_specs["title"]["color"]) - self.display_manager.draw_text("Body Text", font=body_font, color=self.font_specs["body"]["color"]) - self.display_manager.draw_text("Footer", font=footer_font, color=self.font_specs["footer"]["color"]) -``` - -### Using Size Tokens - -```python -# Get available size tokens -tokens = self.font_manager.get_size_tokens() -# Returns: {'xs': 6, 'sm': 8, 'md': 10, 'lg': 12, 'xl': 14, 'xxl': 16} - -# Use token to get size -size_px = tokens.get('md', 10) # 10px - -# Then use in font resolution -font = self.font_manager.resolve_font( - element_key="my_manager.text", +# Register the choice so the web UI's Fonts tab can list it. +self.font_manager.register_manager_font( + manager_id=self.plugin_id, + element_key=element_key, family="press_start", - size_px=size_px + size_px=10, + color=(255, 255, 255), ) + +font = self.font_manager.resolve_font( + element_key=element_key, + family="press_start", + size_px=10, +) +self.display_manager.draw_text("Hello", x=10, y=10, font=font) ``` -## For Plugin Developers +`resolve_font()` applies any entry for `element_key` in +`config/font_overrides.json`, maps a plugin-local family to its namespaced +name when `plugin_id` is passed, and then calls `get_font(family, size_px)`. +On error it returns a fallback font rather than raising. -> **Note**: plugins that ship their own fonts via a `"fonts"` block -> in `manifest.json` are registered automatically during plugin load -> (`src/plugin_system/plugin_manager.py` calls -> `FontManager.register_plugin_fonts()`). The `plugin://…` source -> URIs documented below are resolved relative to the plugin's -> install directory. -> -> The web UI's **Fonts** tab lists, uploads, previews and deletes the -> font files in `assets/fonts/`. Its **Used by** column shows which -> loaded plugins registered each file through `register_manager_font()` -> (see [Font usage in the web UI](#font-usage-in-the-web-ui)), and it -> warns before deleting one of them. It has no override editor (the -> override panels and `/api/v3/fonts/overrides` endpoints were removed). -> The programmatic override workflow in -> [Manual Font Overrides](#manual-font-overrides) below still works. -> Let users pick fonts through your plugin's own config schema. +`get_font(family, size_px)` looks the family up in `font_catalog` and loads +it (cached per family and size). -### Plugin Font Registration +## Font families -In your plugin's `manifest.json`: +At start-up the FontManager scans `assets/fonts/` for `.ttf` and `.bdf` +files. Each becomes a family named after the file, lower-cased and without +the extension (`PressStart2P-Regular.ttf` → `pressstart2p-regular`). Four +aliases are added on top: + +| Alias | File | +|---|---| +| `press_start` | `assets/fonts/PressStart2P-Regular.ttf` | +| `four_by_six` | `assets/fonts/4x6-font.ttf` | +| `five_by_seven` | `assets/fonts/5x7.bdf` | +| `tom_thumb` | `assets/fonts/tom-thumb.bdf` | + +Read the catalog directly: `font_manager.font_catalog` is a dict of family +name to file path. Files added later are picked up on the next start of the +display service. + +## Plugin fonts + +Plugins that ship their own fonts declare them in a `"fonts"` block in +`manifest.json`. The plugin manager calls +`FontManager.register_plugin_fonts()` during plugin load. `plugin://…` +sources are resolved relative to the plugin's install directory. ```json { @@ -195,231 +100,123 @@ In your plugin's `manifest.json`: { "family": "custom_font", "source": "plugin://fonts/custom.ttf", - "metadata": { - "description": "Custom plugin font", - "license": "MIT" - } + "metadata": {"description": "Custom plugin font", "license": "MIT"} }, { "family": "web_font", "source": "https://example.com/fonts/font.ttf", - "metadata": { - "description": "Downloaded font", - "checksum": "sha256:abc123..." - } + "metadata": {"checksum": "sha256:abc123..."} } ] } } ``` -### Using Plugin Fonts +Registered families are namespaced as `::`. Pass +`plugin_id` to `resolve_font()` to use the short name: ```python -class MyPlugin(BasePlugin): - def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager): - super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) - self.font_manager = self._get_font_manager() - - def display(self): - # Use plugin font (automatically namespaced) - font = self.font_manager.resolve_font( - element_key=f"{self.plugin_id}.text", - family="custom_font", # Will be resolved as "my-plugin::custom_font" - size_px=10, - plugin_id=self.plugin_id - ) - - self.display_manager.draw_text("Plugin Text", font=font) -``` - -## Manual Font Overrides - -Overrides are set in code (there is no web UI or REST endpoint for them). -They are stored in `config/font_overrides.json` and persist across restarts. - -### Programmatic Overrides - -```python -# Set override -font_manager.set_override( - element_key="nfl.live.score", - family="four_by_six", - size_px=8 +font = self.font_manager.resolve_font( + element_key=f"{self.plugin_id}.text", + family="custom_font", # resolved as "my-plugin::custom_font" + size_px=10, + plugin_id=self.plugin_id, ) - -# Remove override -font_manager.remove_override("nfl.live.score") - -# Get all overrides -overrides = font_manager.get_overrides() ``` -## Font Discovery +## Overrides -### Available Fonts - -The FontManager automatically scans `assets/fonts/` for TTF and BDF fonts: - -```python -# Get all available fonts -fonts = font_manager.get_available_fonts() -# Returns: {'press_start': 'assets/fonts/PressStart2P-Regular.ttf', ...} - -# Check if font exists -if "my_font" in fonts: - font = font_manager.get_font("my_font", 10) -``` - -### Adding Custom Fonts - -Place font files in `assets/fonts/` directory: -- Supported formats: `.ttf`, `.bdf` -- Font family name is derived from filename (without extension) -- Will be automatically discovered on next initialization +`resolve_font()` still honours `config/font_overrides.json` (a map of +element key to `family` and/or `size_px`), which is read once at start-up. +The methods that edit it — `set_override()`, `remove_override()`, +`get_overrides()` — are deprecated, and there is no web UI or REST endpoint +for overrides (the override editor and `/api/v3/fonts/overrides` were +removed). To let users choose a font, add a field to your plugin's config +schema. ## Font usage in the web UI -The web interface runs in its own process and has no FontManager, so the -display service publishes which plugin uses which font -(`src/font_usage.py`), and the Fonts tab's **Used by** column reads it: +The web UI's **Fonts** tab lists, uploads, previews and deletes the font +files in `assets/fonts/`. The web interface runs in its own process and has +no FontManager, so the display service publishes which plugin uses which +font ([`src/font_usage.py`](../src/font_usage.py)), and the tab's **Used by** +column reads it: - **Source**: `register_manager_font()` registrations of the loaded plugins. `get_font()` and `resolve_font()` do not know the calling plugin and are not counted, and neither is a plugin that opens a font file directly with PIL — register the fonts your plugin draws with if you want them listed. -- **Names**: a family, alias (`press_start`, `four_by_six`, - `five_by_seven`, `tom_thumb`) or path is resolved through - `font_catalog` to the file it loads and reported under that file's name - without extension (`PressStart2P-Regular`, `4x6-font`, `5x7`, - `tom-thumb`), which is how the Fonts tab keys its rows. Fonts outside - `assets/fonts/` (a plugin's own `plugin_id::family` fonts) and families - that resolve to nothing are left out. +- **Names**: a family, alias or path is resolved through `font_catalog` to + the file it loads and reported under that file's name without extension + (`PressStart2P-Regular`, `4x6-font`, `5x7`, `tom-thumb`), which is how the + Fonts tab keys its rows. Fonts outside `assets/fonts/` (a plugin's own + `plugin_id::family` fonts) and families that resolve to nothing are left + out. - **When**: a daemon thread started once plugins have loaded checks every 10 seconds and writes the `font_usage_snapshot` cache key only when the usage changed (and once a day, so the cache's cleanup never expires it). Unloading a plugin drops its registrations (`forget_manager_fonts`). - **Unknown**: until the display service has published, the column reads "unknown" and `GET /api/v3/fonts/catalog` returns `used_by: null`. +- The tab warns before deleting a font that a loaded plugin registered. -## Performance Monitoring +## Text measurement ```python -# Get performance stats -stats = font_manager.get_performance_stats() - -print(f"Cache hit rate: {stats['cache_hit_rate']*100:.1f}%") -print(f"Total fonts cached: {stats['total_fonts_cached']}") -print(f"Failed loads: {stats['failed_loads']}") -print(f"Manager fonts: {stats['manager_fonts']}") -print(f"Plugin fonts: {stats['plugin_fonts']}") -``` - -## Text Measurement - -```python -# Measure text dimensions width, height, baseline = font_manager.measure_text("Hello", font) - -# Get font height font_height = font_manager.get_font_height(font) ``` -## Best Practices +## Tips -### For Managers - -1. **Register all fonts** you use for visibility -2. **Use consistent element keys** (e.g., `{manager_id}.{element_type}`) -3. **Cache font references** if using same font multiple times -4. **Use `resolve_font()`** not `get_font()` directly to support overrides -5. **Define sensible defaults** that work well on LED matrix - -### For Plugins - -1. **Use plugin-relative paths** (`plugin://fonts/...`) -2. **Include font metadata** (license, description) -3. **Provide fallback** fonts if custom fonts fail to load -4. **Test with different display sizes** - -### General - -1. **BDF fonts** are often better for small sizes on LED matrices -2. **TTF fonts** work well for larger sizes -3. **Monospace fonts** are easier to align -4. **Test on actual hardware** - what looks good on screen may not work on LED matrix - -## Migration from Old System - -### Old Way (Direct Font Loading) -```python -self.font = ImageFont.truetype("assets/fonts/PressStart2P-Regular.ttf", 8) -``` - -### New Way (FontManager) -```python -element_key = f"{self.manager_id}.text" -self.font_manager.register_manager_font( - manager_id=self.manager_id, - element_key=element_key, - family="pressstart2p-regular", - size_px=8 -) -self.font = self.font_manager.resolve_font( - element_key=element_key, - family="pressstart2p-regular", - size_px=8 -) -``` +- BDF fonts usually look better than TTF at small sizes on LED panels. +- Use `{plugin_id}.{element}` element keys. +- Register the fonts you draw with, so the Fonts tab can warn before one is + deleted. +- Replace direct `ImageFont.truetype("assets/fonts/...", 8)` calls with + `resolve_font()`: it caches, resolves paths against the install directory, + and handles BDF files. ## Troubleshooting -### Font Not Found -- Check font file exists in `assets/fonts/` -- Verify font family name matches filename (without extension, lowercase) -- Check logs for font discovery errors +**Font not found** +- Check the file exists in `assets/fonts/`. +- The family name is the filename without extension, lower-cased. +- Check the display service log for font discovery errors. -### Override Not Working -- Verify element key matches exactly what manager registered -- Check `config/font_overrides.json` for correct syntax -- Restart application to ensure overrides are loaded +**Plugin fonts not loading** +- Check the manifest's `"fonts"` block. +- Check the log for download or registration errors, and that font URLs are + reachable. -### Performance Issues -- Check cache hit rate in performance stats -- Reduce number of unique font/size combinations -- Clear cache if it grows too large: `font_manager.clear_cache()` +## API reference -### Plugin Fonts Not Loading -- Verify plugin manifest syntax -- Check plugin directory structure -- Review logs for download/registration errors -- Ensure font URLs are accessible +Current methods: -## API Reference +| Method | Purpose | +|---|---| +| `register_manager_font(manager_id, element_key, family, size_px, color=None)` | Record a font choice (feeds the Fonts tab) | +| `forget_manager_fonts(manager_id)` | Drop a manager's registrations (core calls it when a plugin unloads) | +| `resolve_font(element_key, family, size_px, plugin_id=None)` | Get a font, applying overrides and plugin namespacing | +| `get_font(family, size_px)` | Get a font directly | +| `get_native_bdf_size(family)` | Native pixel size of a BDF family, or `None` | +| `measure_text(text, font)` | `(width, height, baseline)` | +| `get_font_height(font)` | Line height | +| `register_plugin_fonts(plugin_id, font_manifest)` | Register a plugin's fonts (core calls it at load) | +| `clear_cache()` | Drop cached fonts and metrics | +| `font_catalog` (attribute) | Family name → file path | -### FontManager Methods +### Deprecated methods -- `register_manager_font(manager_id, element_key, family, size_px, color=None)` - Register font usage -- `forget_manager_fonts(manager_id)` - Drop a manager's registrations (core calls it when a plugin unloads) -- `resolve_font(element_key, family, size_px, plugin_id=None)` - Get font with override support -- `get_font(family, size_px)` - Get font directly (bypasses overrides) -- `measure_text(text, font)` - Measure text dimensions -- `get_font_height(font)` - Get font height -- `set_override(element_key, family=None, size_px=None)` - Set manual override -- `remove_override(element_key)` - Remove override -- `get_overrides()` - Get all overrides -- `get_detected_fonts()` - Get all detected font usage -- `get_manager_fonts(manager_id=None)` - Get fonts by manager -- `get_available_fonts()` - Get font catalog -- `get_size_tokens()` - Get size token definitions -- `get_performance_stats()` - Get performance metrics -- `clear_cache()` - Clear font cache -- `register_plugin_fonts(plugin_id, font_manifest)` - Register plugin fonts -- `unregister_plugin_fonts(plugin_id)` - Unregister plugin fonts - -## Example: Complete Manager Implementation - -For a working example of the font manager API in use, see -`src/font_manager.py` itself. +Removed in 3.7.0. Each logs a warning on first call. +| Method | Use instead | +|---|---| +| `get_available_fonts()`, `get_font_catalog()` | read `font_catalog` | +| `get_size_tokens()` | pass a pixel size | +| `get_performance_stats()` | — | +| `set_override()`, `remove_override()`, `get_overrides()` | a font field in your plugin's config schema | +| `get_manager_fonts()`, `get_detected_fonts()` | — | +| `get_plugin_fonts()`, `unregister_plugin_fonts()` | — | +| `add_font()`, `remove_font()`, `validate_font()` | the web UI's Fonts tab | diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index a7dceae0..d4c38915 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -116,8 +116,8 @@ weather and other location-aware plugins. 4. Wait for installation to finish — installed plugins appear in the **Installed Plugins** section above and get their own tab in the second nav row -5. Toggle the plugin to enabled -6. From **Overview**, click **Restart Display Service** +5. Toggle the plugin to enabled. The running display loads it within a + few seconds; no restart is needed You can also install community plugins straight from a GitHub URL using the **Install from GitHub** section further down the same tab — see @@ -128,9 +128,9 @@ You can also install community plugins straight from a GitHub URL using the 1. Each installed plugin gets its own tab in the second navigation row 2. Open that plugin's tab to edit its settings (favorite teams, API keys, update intervals, etc.) -3. Click **Save** -4. Restart the display service from **Overview** so the new settings take - effect +3. Click **Save**. The display service watches `config.json` and hands the + new settings to the running plugin, so no restart is needed. If a plugin + still shows old settings, restart the display service from **Overview** **Note:** how long each plugin stays on screen is not set in the plugin's own tab — use the **Rotation** tab's **Screen Durations** @@ -197,14 +197,15 @@ The fastest way to verify a plugin works without waiting for the rotation: **Check:** 1. Plugin is enabled (toggle on the **Plugin Manager** tab) -2. Display service was restarted after enabling -3. Plugin's display duration is non-zero -4. No errors in the **Logs** tab for that plugin +2. Plugin's display duration is non-zero +3. No errors in the **Logs** tab for that plugin. A plugin whose + `validate_config()` fails is not loaded until its settings are fixed **Fix:** 1. Enable the plugin from **Plugin Manager** -2. Click **Restart Display Service** on **Overview** -3. Check the **Logs** tab for plugin-specific errors +2. Check the **Logs** tab for plugin-specific errors +3. If it still does not appear, click **Restart Display Service** on + **Overview** ### Weather Plugin Shows "No Data" diff --git a/docs/HOW_TO_RUN_TESTS.md b/docs/HOW_TO_RUN_TESTS.md index 636a53b0..7ee9a097 100644 --- a/docs/HOW_TO_RUN_TESTS.md +++ b/docs/HOW_TO_RUN_TESTS.md @@ -52,10 +52,10 @@ pytest test/test_display_controller.py test/test_plugin_system.py ```bash # Run a specific test class -pytest test/test_display_controller.py::TestDisplayControllerModeRotation +pytest test/test_display_controller.py::TestDisplayControllerLivePriority # Run a specific test function -pytest test/test_display_controller.py::TestDisplayControllerModeRotation::test_basic_rotation +pytest test/test_display_controller.py::TestDisplayControllerSchedule::test_active_hours ``` ### Run Tests by Marker @@ -98,7 +98,7 @@ When you run `pytest`, you'll see: ``` test/test_display_controller.py::TestDisplayControllerInitialization::test_init_success PASSED -test/test_display_controller.py::TestDisplayControllerModeRotation::test_basic_rotation PASSED +test/test_display_controller.py::TestDisplayControllerOnDemand::test_activate_on_demand PASSED ... ``` @@ -174,10 +174,10 @@ pytest ```bash # Run with maximum verbosity and show print statements -pytest -vv -s test/test_display_controller.py::TestDisplayControllerModeRotation::test_basic_rotation +pytest -vv -s test/test_display_controller.py::TestDisplayControllerSchedule::test_active_hours # Run with Python debugger (pdb) -pytest --pdb test/test_display_controller.py::TestDisplayControllerModeRotation::test_basic_rotation +pytest --pdb test/test_display_controller.py::TestDisplayControllerSchedule::test_active_hours ``` ### Run Tests in Parallel (Faster) diff --git a/docs/PERMISSIONS.md b/docs/PERMISSIONS.md new file mode 100644 index 00000000..2d3d7125 --- /dev/null +++ b/docs/PERMISSIONS.md @@ -0,0 +1,154 @@ +# Permissions + +Who owns what on an installed system, which privileged commands the web +interface may run, and how to repair ownership when it goes wrong. The +installer, [`first_time_install.sh`](../first_time_install.sh), sets all of +this up; this page describes the result. + +## Users and groups + +| Account | Used by | Why | +|---|---|---| +| `root` | `ledmatrix.service` (the display) | The LED matrix library needs direct GPIO access | +| The installing user (e.g. `ledpi`) | `ledmatrix-web.service`, `ledmatrix-update-verify.service` | A web server should not run as root | +| `ledmatrix` group | shared files | Members: the installing user, `root`, and `daemon` if it exists. Created by [`setup_cache.sh`](../scripts/install/setup_cache.sh) and the installer | + +The installer also adds the web user to `systemd-journal` and `adm` so the +**Logs** tab can read the journal. Group changes apply after the user logs +in again (services pick them up on restart). + +## Files and directories + +| Path | Owner | Mode | Notes | +|---|---|---|---| +| Project directory | web user | dirs `755`, files `644`, `*.sh` `755` | Set in the installer's "Normalize project file permissions" step | +| `config/` | web user | `2775` | | +| `config/config.json` | web user | `644` | Written by the web interface | +| `config/config_secrets.json` | web user : `ledmatrix` | `640` | Owned by the web user because the web interface writes it; root reads it regardless of mode | +| `plugin-repos/`, `plugins/` | web user | dirs `2775`, files `664` | The web interface installs and removes plugins | +| `assets/` | web user | dirs `755`, files `644` | Root writes downloaded logos regardless | +| `/var/cache/ledmatrix/` | `root:ledmatrix` | `2775` (setgid) | Shared cache: see below | +| Cache files | creator : `ledmatrix` | `660` | | +| `scripts/fix_perms/safe_plugin_rm.sh`, `safe_pip_install.sh` | `root:root` | `755` | Run as root through sudo, so the web user must not be able to edit them | +| `/etc/sudoers.d/ledmatrix_web`, `ledmatrix_wifi` | `root` | `440` | | + +What keeps it that way at runtime: + +- **Config files.** Saves go through + [`src/config_manager_atomic.py`](../src/config_manager_atomic.py), which + applies `get_config_file_mode()` (`640` for secrets, `644` otherwise) and, + when running as root, moves the file's group to the project directory's + group (`ensure_shared_group_ownership()` in + [`src/common/permission_utils.py`](../src/common/permission_utils.py)). +- **Cache files.** [`src/cache/disk_cache.py`](../src/cache/disk_cache.py) + sets every file it writes to `0660` and gives it the cache directory's + group, without relying on the setgid bit. So a file root writes stays + readable by the web user. +- **Plugin directories.** [`run.py`](../run.py) sets + `sys.dont_write_bytecode`, because root-owned `__pycache__` directories + inside a plugin stop the web user updating or removing it. + +`ledmatrix-web.service` deliberately has no `CacheDirectory=`: systemd would +re-own `/var/cache/ledmatrix` to the web user and its primary group, and the +web interface could no longer read what the display writes (see the comment +in [`systemd/ledmatrix-web.service`](../systemd/ledmatrix-web.service)). + +If `/var/cache/ledmatrix` is not usable, `CacheManager` falls back to +`~/.ledmatrix_cache`, `/opt/ledmatrix/cache` or a temp directory +([`src/cache_manager.py`](../src/cache_manager.py)). The two services then +may not share a cache, and the web UI shows stale or empty display status, +on-demand state and plugin health. Fix the directory rather than living +with the fallback. + +## sudo rules + +### `/etc/sudoers.d/ledmatrix_web` + +Generated by `web_sudoers_rules()` in +[`scripts/install/lib_sudoers.sh`](../scripts/install/lib_sudoers.sh), the +only place these rules are defined. Installed by the installer and by +[`configure_web_sudo.sh`](../scripts/install/configure_web_sudo.sh), both of +which check them with `visudo -c` first. The web user may run, without a +password: + +- `reboot`, `poweroff` +- `systemctl start|stop|restart|enable|disable|status ledmatrix.service`, + `systemctl is-active ledmatrix[.service]` +- `systemctl start|stop|restart ledmatrix-web.service` +- `bash /scripts/fix_perms/safe_plugin_rm.sh *` — removes a + directory only if it resolves to a child of `plugin-repos/` or `plugins/` +- `bash /scripts/fix_perms/safe_pip_install.sh *` — installs a + `requirements.txt` only if it is the project's own or one under + `plugin-repos/` or `plugins/`, so the root display service can import the + packages +- `journalctl -u ledmatrix.service *`, `-u ledmatrix *`, `-t ledmatrix *`, + tagged `NOEXEC`: journalctl opens a pager on a terminal, and a shell + escape from that pager would be a root shell + +### `/etc/sudoers.d/ledmatrix_wifi` + +Written by +[`scripts/install/configure_wifi_permissions.sh`](../scripts/install/configure_wifi_permissions.sh) +(run as the web user; the installer calls it). It refuses to grant a binary +that is not root-owned or is group/world-writable. The rules cover: + +- `nmcli device wifi connect|disconnect *`, `nmcli device connect|disconnect *`, + `nmcli radio wifi on|off` +- `systemctl start|stop|restart hostapd`, `... dnsmasq`, + `systemctl restart NetworkManager` +- `sysctl -w net.ipv4.ip_forward=0|1` +- `nft add|delete table ip ledmatrix` +- `rfkill unblock wifi` +- `mkdir -p /etc/NetworkManager/dnsmasq-shared.d` +- `cp` of `/tmp/hostapd.conf` and `/tmp/dnsmasq.conf` to their fixed + destinations, and `rm -f /etc/dnsmasq.d/ledmatrix-captive.conf` +- `cp /tmp/ledmatrix-nm-dnsmasq.conf` to + `/etc/NetworkManager/dnsmasq-shared.d/ledmatrix-captive.conf`, and + `rm -f` of that file + +**`iptables` is deliberately not granted.** The captive portal's rules are +built from the interface name and port, so a rule covering them would need a +trailing wildcard, and `iptables --modprobe=` runs `` as root: a +wildcard grant is a root shell for the web user. Doing it safely needs a +wrapper script that builds the rules itself, like `safe_plugin_rm.sh`. On a +stock Raspberry Pi OS image the default user's blanket `NOPASSWD` rule +(`/etc/sudoers.d/010_pi-nopasswd`) hides this gap. + +### polkit + +The same script installs `/etc/polkit-1/rules.d/10-ledmatrix-wifi.rules`, +which lets the web user perform any `org.freedesktop.NetworkManager.*` +action without authentication. + +## Repair scripts + +In [`scripts/fix_perms/`](../scripts/fix_perms/). Run them from the project +directory. + +| Script | Run as | What it does | Notes | +|---|---|---|---| +| `fix_plugin_permissions.sh` | `sudo` | `plugins/` and `plugin-repos/` to `root:`, dirs `2775`, files `664`; makes a `700` home directory `755` so root can traverse it | Safe. Group-writable, so the web user keeps write access | +| `fix_assets_permissions.sh` | `sudo` | `assets/` to `:`, mode `777` recursively | Works, but looser than the installer's `755`/`644` | +| `fix_cache_permissions.sh` | `sudo` | Runs [`setup_cache.sh`](../scripts/install/setup_cache.sh) for `/var/cache/ledmatrix` (`root:ledmatrix`, `2775`, files `660`), then makes `~/.ledmatrix_cache` (the fallback cache) `:` mode `777` | Safe. The `~/.ledmatrix_cache` mode is still `777` | +| `fix_web_permissions.sh` | the web user, **without** `sudo` | Resets project file ownership for the web user (it calls `sudo` itself), then makes `safe_plugin_rm.sh` and `safe_pip_install.sh` `root:root` `755` again and restores `config_secrets.json` to its owner, group `ledmatrix`, mode `640` | Refuses to run as root. It does not write sudoers rules | +| `safe_plugin_rm.sh`, `safe_pip_install.sh` | — | Called by the web interface through sudo | Not for manual use | + +To reinstall the sudoers rules, run +`./scripts/install/configure_web_sudo.sh` (web rules) or +`./scripts/install/configure_wifi_permissions.sh` (WiFi rules and polkit) as +the web user, not with `sudo`. + +After any of these, restart both services: + +```bash +sudo systemctl restart ledmatrix.service ledmatrix-web.service +``` + +## Checking + +```bash +ls -ld /var/cache/ledmatrix # drwxrwsr-x root ledmatrix +stat -c '%U:%G %a %n' config/config.json config/config_secrets.json +id # web user should list ledmatrix +sudo -l # lists the NOPASSWD rules +``` diff --git a/docs/PLUGIN_API_REFERENCE.md b/docs/PLUGIN_API_REFERENCE.md index 5aadaadc..1620c61e 100644 --- a/docs/PLUGIN_API_REFERENCE.md +++ b/docs/PLUGIN_API_REFERENCE.md @@ -9,6 +9,7 @@ Complete API reference for plugin developers. This document describes all method ## Table of Contents +- [Manifest Required Fields](#manifest-required-fields) - [BasePlugin](#baseplugin) - [Display Manager](#display-manager) - [Cache Manager](#cache-manager) @@ -17,6 +18,50 @@ Complete API reference for plugin developers. This document describes all method --- +## Manifest Required Fields + +Three parts of core check `manifest.json`, each for a different set of +fields: + +| Check | Fields | What happens when one is missing | +|---|---|---| +| JSON schema, [`schema/manifest_schema.json`](../schema/manifest_schema.json) | `id`, `name`, `version`, `author`, `entry_point`, `class_name`, `compatible_versions` | Install from URL logs a warning (`PluginStoreManager._validate_manifest_schema()`); nothing is refused | +| Plugin Store install, [`src/plugin_system/store_manager.py`](../src/plugin_system/store_manager.py) | `id`, `name`, `class_name`, `display_modes` | Install is refused. A registry install first tries to detect a missing `class_name` from the entry-point file | +| Plugin loader, [`src/plugin_system/plugin_loader.py`](../src/plugin_system/plugin_loader.py) | `class_name` | The plugin fails to load | + +Defaults and other uses: + +- `entry_point` defaults to `manager.py`; the store writes the default back + into the manifest on install. +- `compatible_versions` (a list of semver ranges such as `">=2.0.0"`) is how + the store decides whether a plugin can run on this core. An install is + refused only when the field excludes the running version + (`compatibility.check()` in + [`src/plugin_system/compatibility.py`](../src/plugin_system/compatibility.py)). +- `version` is compared with the registry's `latest_version` to decide + whether an update is available. +- If `display_modes` is empty at load time, the display controller uses the + plugin id as the only mode. + +**Set all eight:** `id`, `name`, `version`, `author`, `entry_point`, +`class_name`, `display_modes`, `compatible_versions`. That satisfies every +check. The schema lists the optional fields. + +```json +{ + "id": "my-plugin", + "name": "My Plugin", + "version": "1.0.0", + "author": "YourName", + "entry_point": "manager.py", + "class_name": "MyPlugin", + "display_modes": ["my-plugin"], + "compatible_versions": [">=2.0.0"] +} +``` + +--- + ## BasePlugin All plugins must inherit from `BasePlugin` and implement the required methods. The base class provides access to managers and common functionality. @@ -432,82 +477,17 @@ self.display_manager.update_display() This is the canonical way to render arbitrary images. -### Weather Icons +### Weather Icons (deprecated) -#### `draw_weather_icon(condition: str, x: int, y: int, size: int = 16) -> None` +> Deprecated, removed in 3.7.0 — draw your own icons (the weather plugin +> ships `WeatherIcons`). See [Deprecated APIs](#deprecated-apis). -Draw a weather icon based on the condition string. - -**Parameters**: -- `condition` (str): Weather condition (e.g., "clear", "cloudy", "rain", "snow", "storm") -- `x` (int): X position -- `y` (int): Y position -- `size` (int): Icon size in pixels (default: 16) - -**Supported Conditions**: -- `"clear"`, `"sunny"` → Sun icon -- `"clouds"`, `"cloudy"`, `"partly cloudy"` → Cloud icon -- `"rain"`, `"drizzle"`, `"shower"` → Rain icon -- `"snow"`, `"sleet"`, `"hail"` → Snow icon -- `"thunderstorm"`, `"storm"` → Storm icon - -**Example**: -```python -self.display_manager.draw_weather_icon("rain", x=10, y=10, size=16) -``` - -#### `draw_sun(x: int, y: int, size: int = 16) -> None` - -Draw a sun icon with rays. - -**Parameters**: -- `x` (int): X position -- `y` (int): Y position -- `size` (int): Icon size (default: 16) - -#### `draw_cloud(x: int, y: int, size: int = 16, color: tuple = (200, 200, 200)) -> None` - -Draw a cloud icon. - -**Parameters**: -- `x` (int): X position -- `y` (int): Y position -- `size` (int): Icon size (default: 16) -- `color` (tuple): RGB color (default: light gray) - -#### `draw_rain(x: int, y: int, size: int = 16) -> None` - -Draw rain icon with cloud and droplets. - -#### `draw_snow(x: int, y: int, size: int = 16) -> None` - -Draw snow icon with cloud and snowflakes. - -#### `draw_text_with_icons(text: str, icons: List[tuple] = None, x: int = None, y: int = None, color: tuple = (255, 255, 255)) -> None` - -Draw text with weather icons at specified positions. - -**Parameters**: -- `text` (str): Text to display -- `icons` (List[tuple], optional): List of (icon_type, x, y) tuples -- `x` (int, optional): X position for text -- `y` (int, optional): Y position for text -- `color` (tuple): Text color - -**Note**: Automatically calls `update_display()` after drawing. - -**Example**: -```python -icons = [ - ("sun", 5, 5), - ("cloud", 100, 5) -] -self.display_manager.draw_text_with_icons( - "Weather: Sunny, Cloudy", - icons=icons, - x=10, y=20 -) -``` +- `draw_weather_icon(condition, x, y, size=16)` — icon for a condition + string such as `"clear"`, `"clouds"`, `"rain"`, `"snow"`, `"storm"` +- `draw_sun(x, y, size=16)`, `draw_cloud(x, y, size=16, color=(200, 200, 200))`, + `draw_rain(x, y, size=16)`, `draw_snow(x, y, size=16)` +- `draw_text_with_icons(text, icons=None, x=None, y=None, color=(255, 255, 255))` + — text plus a list of `(icon_type, x, y)` icons; calls `update_display()` ### Scrolling State Management @@ -601,6 +581,8 @@ Process any deferred updates if not currently scrolling. Called automatically by #### `get_scrolling_stats() -> dict` +> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis). + Get current scrolling statistics for debugging. **Returns**: Dictionary with scrolling state information @@ -742,6 +724,8 @@ data = self.cache_manager.get_with_auto_strategy("nhl_live_scores") #### `get_background_cached_data(key: str, sport_key: Optional[str] = None) -> Optional[Dict[str, Any]]` +> Deprecated, removed in 3.7.0 — use `get()`. See [Deprecated APIs](#deprecated-apis). + Get background service cached data with sport-specific intervals. **Parameters**: @@ -779,6 +763,8 @@ max_age = strategy['max_age'] # Get configured max age #### `get_sport_live_interval(sport_key: str) -> int` +> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis). + Get the live_update_interval for a specific sport from config. **Parameters**: @@ -803,6 +789,8 @@ Extract data type from cache key to determine appropriate cache strategy. #### `get_sport_key_from_cache_key(key: str) -> Optional[str]` +> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis). + Extract sport key from cache key for sport-specific strategies. **Parameters**: @@ -847,10 +835,12 @@ for file_info in files: self.logger.info(f"Cache: {file_info['key']}, Age: {file_info['age_display']}") ``` -### Metrics Methods +### Metrics Methods (deprecated) #### `get_cache_metrics() -> Dict[str, Any]` +> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis). + Get cache performance metrics. **Returns**: Dictionary with cache statistics (`total_requests`, `cache_hit_rate`, `background_hit_rate`, `api_calls_saved`, `average_fetch_time`, etc.) @@ -863,6 +853,8 @@ self.logger.info(f"Cache hit rate: {metrics['cache_hit_rate']:.2%}") #### `get_memory_cache_stats() -> Dict[str, Any]` +> Deprecated, removed in 3.7.0. See [Deprecated APIs](#deprecated-apis). + Get memory cache statistics. **Returns**: Dictionary with memory cache stats (size, max_size, etc.) @@ -907,6 +899,8 @@ for plugin_id, plugin in all_plugins.items(): #### `get_enabled_plugins() -> List[str]` +> Deprecated, removed in 3.7.0 — check `enabled` on the instances in `plugin_manager.plugins`. See [Deprecated APIs](#deprecated-apis). + Get list of enabled plugin IDs. **Returns**: List of plugin identifier strings @@ -985,9 +979,8 @@ def update(self): **Example - Checking if another plugin is enabled**: ```python -enabled_plugins = self.plugin_manager.get_enabled_plugins() -if "weather" in enabled_plugins: - # Weather plugin is enabled +weather = self.plugin_manager.plugins.get("weather") +if weather is not None and weather.enabled: pass ``` diff --git a/docs/PLUGIN_CONFIG_ARCHITECTURE.md b/docs/PLUGIN_CONFIG_ARCHITECTURE.md index b34ffc89..1ab47b9a 100644 --- a/docs/PLUGIN_CONFIG_ARCHITECTURE.md +++ b/docs/PLUGIN_CONFIG_ARCHITECTURE.md @@ -155,9 +155,9 @@ deep-merged back into the plugin's config at load time ### Custom input widgets Set `"x-widget": ""` on a property. Core widgets are in -`web_interface/static/v3/js/widgets/` (see its README); a plugin can ship its -own widget script, served from `/static/plugin-widgets//.js`. -See [widget-guide.md](widget-guide.md). +`web_interface/static/v3/js/widgets/`; a plugin can ship its own widget +script, served from `/static/plugin-widgets//.js`. See the +[widget guide](../web_interface/static/v3/js/widgets/README.md). ### Custom actions diff --git a/docs/PLUGIN_DEVELOPMENT_GUIDE.md b/docs/PLUGIN_DEVELOPMENT_GUIDE.md index 95857464..da844d57 100644 --- a/docs/PLUGIN_DEVELOPMENT_GUIDE.md +++ b/docs/PLUGIN_DEVELOPMENT_GUIDE.md @@ -520,19 +520,22 @@ When developing plugins, you'll need to use the APIs provided by the LEDMatrix s `display_manager.image` (a PIL Image) and call `update_display()`; there is no `draw_image()` helper method. - `draw_weather_icon()`, `draw_sun()`, `draw_cloud()` - Weather icons + (deprecated, removed in 3.7.0 — draw your own icons) - `get_text_width()`, `get_font_height()` - Text utilities - `set_scrolling_state()`, `defer_update()` - Scrolling state management **Cache Manager** (`self.cache_manager`): - `get()`, `set()`, `delete()` - Basic caching - `get_cached_data_with_strategy()` - Advanced caching with strategies -- `get_background_cached_data()` - Background service caching +- `get_background_cached_data()` - deprecated, removed in 3.7.0 — use `get()` **Plugin Manager** (`self.plugin_manager`): - `get_plugin()`, `get_all_plugins()` - Access other plugins - `get_plugin_info()` - Get plugin information -See [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) for complete documentation. +See [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) for complete +documentation, and its [Deprecated APIs](PLUGIN_API_REFERENCE.md#deprecated-apis) +table for everything removed in 3.7.0. ## 3rd Party Plugin Development @@ -577,12 +580,14 @@ Your plugin must: pass ``` -2. **Include manifest.json** with required fields: +2. **Include manifest.json** with the required fields listed in + [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#manifest-required-fields): ```json { "id": "my-plugin", "name": "My Plugin", "version": "1.0.0", + "author": "YourName", "class_name": "MyPlugin", "entry_point": "manager.py", "display_modes": ["my_plugin"], @@ -658,7 +663,7 @@ For your plugin to work well in the plugin store: with the registry's `latest_version`; releases and tags are not read - **README.md**: Clear installation and configuration instructions - **config_schema.json**: Recommended for web UI configuration -- **manifest.json**: Required with all required fields +- **manifest.json**: Required, with the [required fields](PLUGIN_API_REFERENCE.md#manifest-required-fields) - **requirements.txt**: If your plugin has Python dependencies ### Distribution Options diff --git a/docs/PLUGIN_QUICK_REFERENCE.md b/docs/PLUGIN_QUICK_REFERENCE.md index 8eb7f793..16bc0886 100644 --- a/docs/PLUGIN_QUICK_REFERENCE.md +++ b/docs/PLUGIN_QUICK_REFERENCE.md @@ -45,7 +45,8 @@ LEDMatrix/ ### 1. Minimal Plugin Structure -**manifest.json**: +**manifest.json** (the required fields are explained in +[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#manifest-required-fields)): ```json { "id": "my-plugin", @@ -54,6 +55,8 @@ LEDMatrix/ "author": "YourName", "entry_point": "manager.py", "class_name": "MyPlugin", + "display_modes": ["my-plugin"], + "compatible_versions": [">=2.0.0"], "category": "custom" } ``` diff --git a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md index 0fcf7db8..f12a7a97 100644 --- a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md +++ b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md @@ -67,9 +67,9 @@ Don't edit `latest_version` or `last_updated` by hand for monorepo plugins: ## Adding or changing an official plugin -1. Add or edit `plugins//` in the monorepo. The store refuses - a manifest without `id`, `name`, `class_name` and `display_modes`; also - set `version`. +1. Add or edit `plugins//` in the monorepo, with the + manifest fields listed in + [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#manifest-required-fields). 2. Bump `version` in the plugin's `manifest.json` for every change, or users won't be offered the update. 3. Run `python update_registry.py` in ledmatrix-plugins and commit the diff --git a/docs/README.md b/docs/README.md index 475cb92e..bd88bdc6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,7 +11,7 @@ the one-shot installer. The pages here go deeper. 2. [WEB_INTERFACE_GUIDE.md](WEB_INTERFACE_GUIDE.md) — using the web UI 3. [PLUGIN_STORE_GUIDE.md](PLUGIN_STORE_GUIDE.md) — installing and managing plugins 4. [WIFI_NETWORK_SETUP.md](WIFI_NETWORK_SETUP.md) — WiFi and AP-mode setup -5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes +5. [TROUBLESHOOTING.md](TROUBLESHOOTING.md) — common issues and fixes ([PERMISSIONS.md](PERMISSIONS.md) for "Permission denied") 6. [SSH_UNAVAILABLE_AFTER_INSTALL.md](SSH_UNAVAILABLE_AFTER_INSTALL.md) — recovering SSH after install 7. [CONFIG_DEBUGGING.md](CONFIG_DEBUGGING.md) — diagnosing config problems 8. [LOW_MEMORY_BOARDS.md](LOW_MEMORY_BOARDS.md) — Pi Zero 2 W / 3B+ / 1GB Pi 4 memory limits @@ -37,7 +37,7 @@ Going deeper: - [PLUGIN_CUSTOM_ICONS.md](PLUGIN_CUSTOM_ICONS.md) - [PLUGIN_REGISTRY_SETUP_GUIDE.md](PLUGIN_REGISTRY_SETUP_GUIDE.md) (+ [registry template](plugin_registry_template.json)) - [STARLARK_APPS_GUIDE.md](STARLARK_APPS_GUIDE.md) — Starlark-based mini-apps -- [widget-guide.md](widget-guide.md) — widget development +- [Widget guide](../web_interface/static/v3/js/widgets/README.md) — built-in `x-widget`s and custom widgets - [ADAPTIVE_LAYOUT.md](ADAPTIVE_LAYOUT.md) — render legibly on any panel size (opt-in font/layout scaling) - [plugin-safety-harness.md](plugin-safety-harness.md) — test a plugin across every screen and matrix size @@ -56,21 +56,25 @@ Going deeper: - [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) — Vegas scroll, on-demand display, cache management, background services, permissions - [FONT_MANAGER.md](FONT_MANAGER.md) — font system +- [PERMISSIONS.md](PERMISSIONS.md) — file ownership, sudo rules, repair scripts +- [MQTT bridge](../integrations/mqtt_bridge/README.md) — control the display from Home Assistant over MQTT ## Reference - [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md) — every key in config.json and config_secrets.json - [REST_API_REFERENCE.md](REST_API_REFERENCE.md) — all web-interface HTTP endpoints - [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md) — Python APIs available to plugins +- [src/common/README.md](../src/common/README.md) — shared helper modules plugins can import - [DEVELOPER_QUICK_REFERENCE.md](DEVELOPER_QUICK_REFERENCE.md) — common dev tasks ## Contributing to LEDMatrix itself +- [ARCHITECTURE.md](ARCHITECTURE.md) — processes, display loop, plugin system, web UI; where to start reading - [DEVELOPMENT.md](DEVELOPMENT.md) — environment setup - [HOW_TO_RUN_TESTS.md](HOW_TO_RUN_TESTS.md) — running the test suite - [MULTI_ROOT_WORKSPACE_SETUP.md](MULTI_ROOT_WORKSPACE_SETUP.md) — multi-repo workspace - [MIGRATION_GUIDE.md](MIGRATION_GUIDE.md) — breaking changes between releases -- [SPORTS_UNIFICATION.md](SPORTS_UNIFICATION.md) — how the sports scoreboard base classes are organized +- [SPORTS_UNIFICATION.md](SPORTS_UNIFICATION.md) — how shared sports scoreboard code moves into `src/common` ## Audits diff --git a/docs/SPORTS_UNIFICATION.md b/docs/SPORTS_UNIFICATION.md index 015c6e2d..b03e5767 100644 --- a/docs/SPORTS_UNIFICATION.md +++ b/docs/SPORTS_UNIFICATION.md @@ -73,14 +73,16 @@ B2 below promoted code into it (`SportsCore`, the mode classes, capabilities sections record that design, but none of it ships in core any more. Shared sports code lives in `src/common`: -``` -src/common/ - sports_scroll.py SportsScrollDisplay / …Manager — scroll orchestration - (content building stays in the plugins) - sports_helpers.py clamp/logo/rotation free functions + SportsHelpersMixin - (3.5.0) — the helpers byte-identical in the - plugins' sports.py, and the _favorite_key seam -``` +| Module | Since | Holds | +|---|---|---| +| `sports_scroll.py` | 3.2.0 | `SportsScrollDisplay` / `SportsScrollDisplayManager` — scroll orchestration (content building stays in the plugins) | +| `sports_card.py` | 3.3.0 | Free functions for card settings, colours, favourite-team rules, dates and font sizes | +| `sports_game_renderer.py` | 3.3.0 | `SportsGameRendererMixin` — scroll/Vegas card geometry | +| `sports_shared.py` | 3.3.0 | `SportsCoreSharedMixin`, `SportsLiveSharedMixin`, `SportsRecentSharedMixin` — the sport-independent `sports.py` methods | +| `sports_helpers.py` | 3.5.0 | clamp/logo/rotation free functions and `SportsHelpersMixin`, plus the `_favorite_key` seam | +| `espn_dates.py` | 3.5.0 | ESPN date-range and `limit` workarounds | + +Each is described in [src/common/README.md](../src/common/README.md). ### Converging on `src/common` @@ -90,7 +92,7 @@ modules taken from the plugin copies, each a **new module** rather than growth on an existing one: a plugin that deletes a method copy and relies on an older module having gained it fails at runtime with an `AttributeError`, while a missing module fails at load, where the version checks can see it. -`sports_helpers.py` is the first (it holds `_favorite_key`, the override point +`sports_helpers.py` is the newest (it holds `_favorite_key`, the override point listed below, for later phases); its parity test compares every body against the plugin copies when `LEDMATRIX_PLUGINS` points at a checkout, and `test/test_common_is_hardware_free.py` keeps `src/common` free of diff --git a/docs/STARLARK_APPS_GUIDE.md b/docs/STARLARK_APPS_GUIDE.md index efd5b77d..6fbd9454 100644 --- a/docs/STARLARK_APPS_GUIDE.md +++ b/docs/STARLARK_APPS_GUIDE.md @@ -102,9 +102,16 @@ cd /path/to/LEDMatrix bash scripts/download_pixlet.sh ``` +The script downloads only the Linux ARM64 build (Raspberry Pi OS 64-bit), +from the `tronbyt/pixlet` releases, to `bin/pixlet/pixlet-linux-arm64`. On +any other platform (32-bit Pi OS, x86_64, macOS), put a `pixlet` binary on +your `PATH`, or set the plugin's `pixlet_path`, or place it in `bin/pixlet/` +under the name `_find_pixlet_binary()` looks for +([`web_interface/blueprints/api_v3/__init__.py`](../web_interface/blueprints/api_v3/__init__.py)). + Verify installation: ```bash -./bin/pixlet/pixlet-linux-amd64 version +./bin/pixlet/pixlet-linux-arm64 version # Pixlet 0.50.2 (or later) ``` @@ -276,10 +283,8 @@ LEDMatrix/ │ ├── hour_hand.png │ └── minute_hand.png │ -├── bin/pixlet/ # Pixlet binaries -│ ├── pixlet-linux-amd64 -│ ├── pixlet-linux-arm64 -│ └── pixlet-darwin-arm64 +├── bin/pixlet/ # Pixlet binary +│ └── pixlet-linux-arm64 # the only one download_pixlet.sh fetches │ └── scripts/ └── download_pixlet.sh # Pixlet installer @@ -324,7 +329,7 @@ Many apps require API keys for external services: **Solutions**: 1. Check logs: `journalctl -u ledmatrix | grep -i pixlet` 2. Verify config: Ensure all required fields are filled -3. Test manually: `./bin/pixlet/pixlet-linux-amd64 render starlark-apps/{app-id}/{app-id}.star` +3. Test manually: `./bin/pixlet/pixlet-linux-arm64 render starlark-apps/{app-id}/{app-id}.star` 4. Missing assets: Some apps need images/fonts that may fail to download 5. API issues: Check API keys and rate limits diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index db5b3cac..fede8dbb 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -201,10 +201,11 @@ sudo systemctl restart ledmatrix-web **Solutions:** -1. **Install dependencies:** +1. **Install dependencies** as root, so the root display service can import + them: ```bash - pip3 install --break-system-packages -r requirements.txt - pip3 install --break-system-packages -r web_interface/requirements.txt + sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt + sudo python3 -m pip install --break-system-packages --no-cache-dir -r web_interface/requirements.txt ``` 2. **Test imports step-by-step:** @@ -250,15 +251,18 @@ sudo systemctl restart ledmatrix-web **Solutions:** -```bash -# Fix ownership of LEDMatrix directory -sudo chown -R ledpi:ledpi /home/ledpi/LEDMatrix +[PERMISSIONS.md](PERMISSIONS.md) lists the expected owner and mode of every +file and directory, and which `scripts/fix_perms/` script to run as which +user. Don't `chown -R` the whole project: the two sudo helper scripts in +`scripts/fix_perms/` must stay owned by root. -# Fix config file permissions +```bash +# Config files: web user owns both; secrets must stay 640 +stat -c '%U:%G %a %n' config/config.json config/config_secrets.json sudo chmod 644 config/config.json sudo chmod 640 config/config_secrets.json -# Verify service runs as correct user +# Which user the web interface runs as sudo systemctl cat ledmatrix-web | grep User ``` @@ -462,15 +466,17 @@ sudo systemctl cat ledmatrix-web | grep User } ``` -2. **Restart display:** - ```bash - sudo systemctl restart ledmatrix - ``` + Or toggle the plugin on in the **Plugin Manager** tab, which writes the + same flag. -3. **Verify in web interface:** - - Open the **Plugin Manager** tab - - Toggle the plugin switch to enable - - From **Overview**, click **Restart Display Service** +2. **Wait a few seconds.** The display service watches `config.json` and + loads a newly enabled plugin without a restart + (`DisplayController._reconcile_enabled_plugins()` in + [`src/display_controller.py`](../src/display_controller.py)). This + needs hot reload, which is on unless `LEDMATRIX_HOT_RELOAD=false` is set. + +3. **If it still does not appear**, check the logs for a config validation + error, then restart: `sudo systemctl restart ledmatrix` #### Plugin Not Loading @@ -491,10 +497,12 @@ sudo systemctl cat ledmatrix-web | grep User # Verify all required fields present ``` -3. **Check dependencies installed:** +3. **Check dependencies installed.** Install them with `sudo`: the display + service runs as root and does not see packages pip put in your user's + `~/.local` (see [PLUGIN_DEPENDENCY_GUIDE.md](PLUGIN_DEPENDENCY_GUIDE.md)): ```bash if [ -f plugin-repos/plugin-id/requirements.txt ]; then - pip3 install --break-system-packages -r plugin-repos/plugin-id/requirements.txt + sudo python3 -m pip install --break-system-packages --no-cache-dir -r plugin-repos/plugin-id/requirements.txt fi ``` @@ -503,14 +511,9 @@ sudo systemctl cat ledmatrix-web | grep User sudo journalctl -u ledmatrix -f | grep plugin-id ``` -5. **Test plugin import:** +5. **Load and render the plugin headlessly:** ```bash - python3 -c " - import sys - sys.path.insert(0, 'plugin-repos/plugin-id') - from manager import PluginClass - print('Plugin imports successfully') - " + python3 scripts/check_plugin.py --plugin plugin-id ``` #### Stale Cache Data @@ -540,10 +543,12 @@ sudo systemctl cat ledmatrix-web | grep User sudo systemctl restart ledmatrix ``` -2. **Check cache permissions:** +2. **Check cache permissions.** Expected: `root:ledmatrix`, `drwxrwsr-x`. + `setup_cache.sh` restores that layout (see + [PERMISSIONS.md](PERMISSIONS.md#repair-scripts)): ```bash ls -ld /var/cache/ledmatrix - sudo ./scripts/fix_perms/fix_cache_permissions.sh + sudo bash scripts/install/setup_cache.sh ``` --- diff --git a/docs/WEB_INTERFACE_GUIDE.md b/docs/WEB_INTERFACE_GUIDE.md index 7b442c6d..1d354bf5 100644 --- a/docs/WEB_INTERFACE_GUIDE.md +++ b/docs/WEB_INTERFACE_GUIDE.md @@ -161,7 +161,11 @@ duration, and related settings — so you can configure Vegas mode entirely from the web UI without hand-editing JSON. See [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) for what the options do. -Changes require **Restart Display Service** from the Overview tab. +Brightness and the Vegas Scroll settings apply to the running display +within a few seconds. Matrix hardware settings (rows, columns, chain length, +mapping, GPIO slowdown, PWM and refresh settings) are only read when the +display starts, so those need **Restart Display Service** from the Overview +tab. ### Plugin Manager Tab @@ -248,16 +252,16 @@ View real-time system logs: 1. Open the **Display** tab 2. Adjust the **Brightness** slider (1–100) -3. Click **Save** -4. Click **Restart Display Service** on the **Overview** tab +3. Click **Save**. The panel picks up the new brightness within a few + seconds; no restart is needed ### Installing a New Plugin 1. Open the **Plugin Manager** tab 2. Scroll to the **Plugin Store** section and browse or search 3. Click **Install** next to the plugin -4. Toggle the plugin on in **Installed Plugins** -5. Click **Restart Display Service** on **Overview** +4. Toggle the plugin on in **Installed Plugins**. The running display + loads it within a few seconds; no restart is needed ### Configuring a Plugin diff --git a/docs/widget-guide.md b/docs/widget-guide.md index d4aa0f51..e433e85b 100644 --- a/docs/widget-guide.md +++ b/docs/widget-guide.md @@ -1,590 +1,7 @@ # Widget Development Guide -## Overview - -The LEDMatrix Widget Registry system allows plugins to use reusable UI components for configuration forms. This enables: - -- **Reusable Components**: Use existing widgets (file upload, checkboxes, etc.) without custom code -- **Custom Widgets**: Create plugin-specific widgets without modifying the LEDMatrix codebase -- **Backwards Compatibility**: Existing plugins continue to work without changes - -## Available Core Widgets - -### Plugin File Manager Widget (`plugin-file-manager`) - -Full inline file management UI for plugins that manage files via the `web_ui_actions` system. Renders a card grid, upload zone, create/delete modals, and an entry table editor — entirely inline, no iframe. - -`plugin_id` is **automatically injected** from template context. File operations call `/api/v3/plugins/action` immediately on user action; no Save Configuration needed. - -**Schema Configuration:** -```json -{ - "file_manager": { - "type": "null", - "title": "Data Files", - "x-widget": "plugin-file-manager", - "x-widget-config": { - "actions": { - "list": "list-files", - "get": "get-file", - "save": "save-file", - "upload": "upload-file", - "delete": "delete-file", - "create": "create-file", - "toggle": "toggle-category" - }, - "upload_hint": "JSON files with day numbers 1–365 as keys", - "directory_label": "my_data/", - "create_fields": [ - { "key": "category_name", "label": "Category Name", - "placeholder": "e.g., my_words", "pattern": "^[a-z0-9_]+$", - "hint": "Lowercase letters, numbers, underscores" }, - { "key": "display_name", "label": "Display Name", - "placeholder": "e.g., My Words", "hint": "Optional" } - ] - } - } -} -``` - -**`list` is required** — the widget calls it on render to populate the file grid; omitting it leaves the widget stuck in a loading state. All other actions are optional — omit any key to hide its UI element (e.g., no `create` = no New File button, no `toggle` = no enable/disable switch). - -The edit view auto-detects whether file content is tabular (object-of-objects with uniform keys) and shows a paginated table editor with inline cells. Otherwise falls back to a JSON textarea. - -**Used by:** of-the-day - ---- - -### Time Picker Widget (`time-picker`) - -Single time selection using the browser's native time input. Returns a string in `HH:MM` (24-hour) format. Generic — works in any plugin without configuration. - -**Schema Configuration:** -```json -{ - "target_time": { - "type": "string", - "x-widget": "time-picker", - "default": "00:00", - "x-options": { - "placeholder": "Select time", - "clearable": true - } - } -} -``` - -**Used by:** countdown - ---- - -### File Upload Single Widget (`file-upload-single`) - -Single-image upload for string fields. Uploads to the plugin's asset folder (`assets/plugins//uploads/`) and sets the string field value to the returned relative path. Shows a thumbnail preview and a clear button. The `plugin_id` is **automatically injected** from the template context — no need to specify it in the schema. - -**Schema Configuration:** -```json -{ - "image_path": { - "type": "string", - "x-widget": "file-upload-single", - "x-upload-config": { - "allowed_types": ["image/png", "image/jpeg", "image/bmp", "image/gif"], - "max_size_mb": 5 - } - } -} -``` - -Note: Unlike `file-upload` (array-level), this widget is for a single `string` field. It is ideal for per-item images inside `array-table` rows. - -**Used by:** countdown - ---- - -### File Upload Widget (`file-upload`) - -Upload and manage image files with drag-and-drop support, preview, delete, and scheduling. - -**Schema Configuration:** -```json -{ - "type": "array", - "x-widget": "file-upload", - "x-upload-config": { - "plugin_id": "my-plugin", - "max_files": 10, - "max_size_mb": 5, - "allowed_types": ["image/png", "image/jpeg", "image/bmp", "image/gif"] - } -} -``` - -**Used by:** static-image, news plugins - -### Checkbox Group Widget (`checkbox-group`) - -Multi-select checkboxes for array fields with enum items. - -**Schema Configuration:** -```json -{ - "type": "array", - "x-widget": "checkbox-group", - "items": { - "type": "string", - "enum": ["option1", "option2", "option3"] - }, - "x-options": { - "labels": { - "option1": "Option 1 Label", - "option2": "Option 2 Label" - } - } -} -``` - -**Used by:** odds-ticker, news plugins - -### Custom Feeds Widget (`custom-feeds`) - -Table-based RSS feed editor with logo uploads. - -**Schema Configuration:** -```json -{ - "type": "array", - "x-widget": "custom-feeds", - "items": { - "type": "object", - "properties": { - "name": { "type": "string" }, - "url": { "type": "string", "format": "uri" }, - "enabled": { "type": "boolean" }, - "logo": { "type": "object" } - } - }, - "maxItems": 50 -} -``` - -**Used by:** news plugin (for custom RSS feeds) - -## Using Existing Widgets - -To use an existing widget in your plugin's `config_schema.json`, simply add the `x-widget` property: - -```json -{ - "properties": { - "my_images": { - "type": "array", - "x-widget": "file-upload", - "x-upload-config": { - "plugin_id": "my-plugin", - "max_files": 5 - } - }, - "enabled_leagues": { - "type": "array", - "x-widget": "checkbox-group", - "items": { - "type": "string", - "enum": ["nfl", "nba", "mlb"] - }, - "x-options": { - "labels": { - "nfl": "NFL", - "nba": "NBA", - "mlb": "MLB" - } - } - } - } -} -``` - -The widget will be automatically rendered when the plugin configuration form is loaded. - -## Labelling Enum Options (`x-options.labels`) - -A plain `enum` renders as a dropdown whose option text is the value with -underscores replaced and title case applied — `day_first` becomes "Day First". -That is fine for values that read as their own label, and wrong for values that -do not: `vs` becomes "Vs", and `abbrev` says nothing about the `Sep 19` it -actually produces. - -Supply `x-options.labels` to set the visible text. This is the same convention -the `checkbox-group` widget uses: - -```json -{ - "date_format": { - "type": "string", - "enum": ["abbrev", "numeric", "day_first"], - "default": "abbrev", - "x-options": { - "labels": { - "abbrev": "Sep 19", - "numeric": "9/19", - "day_first": "19 Sep" - } - } - } -} -``` - -Labels are **display only** — the stored value is still the enum value, so -adding them never changes a saved config. The map may be partial: any value -without a label keeps the humanised fallback. Older cores that predate this -support ignore `x-options` and render the fallback for every option, so a -plugin can ship labels without requiring a core upgrade. - -Array-table columns (`x-widget: array-table`) accept the same -`x-options.labels` on a column definition, but their fallback is the **raw -value** rather than the humanised one, because those columns hold values such -as ticker symbols where `aapl` → "Aapl" would be wrong. Rows added in the -browser use the labels too (`array-table.js`), so a column reads the same -before and after a page reload. - -## Marking Fields as Advanced (`x-advanced`) - -Add `"x-advanced": true` to any top-level, non-object property to move it out -of the main form and into a single collapsed **Advanced Settings** section at -the bottom of the plugin's configuration page: - -```json -{ - "properties": { - "city": { - "type": "string", - "title": "City" - }, - "request_timeout": { - "type": "integer", - "default": 10, - "description": "HTTP timeout in seconds", - "x-advanced": true - } - } -} -``` - -Guidelines: - -- Use it for fine-tuning knobs most users never touch (timeouts, retry - behavior, cache TTLs, styling overrides). Anything a first-time user must - set to get the plugin working should stay basic. -- Nothing is hidden permanently — the section expands on click, and the - settings search finds and auto-expands advanced fields like any others. -- The flag is ignored on `object`-type properties (they already render as - their own collapsible sections) and is safely ignored by older cores, so - adding it never breaks compatibility. - -## Hiding Fields From the Form (`x-display: "hidden"`) - -Add `"x-display": "hidden"` to a property that must stay in the schema but -should not appear as a control: a deprecated key kept so existing configs keep -validating, or an internal value such as an auto-generated row id. - -```json -{ - "properties": { - "radar_zoom": { - "type": "integer", - "default": 6, - "title": "Radar Zoom Level (deprecated)", - "x-display": "hidden" - } - } -} -``` - -What the core does with it: - -- **Not rendered** at any depth: top-level fields, children of an object - section, and properties of array-of-object items (never a table column, even - if `x-columns` names it, and never in the row editor). A hidden field flagged - `x-advanced` is not listed or counted in Advanced Settings, and an object - whose children are all hidden draws no empty section. Hidden fields don't - show up in the settings search either, since it indexes the rendered form. -- **Stored value preserved on save.** Saving the form never changes a hidden - value. The unchecked-checkbox rule ignores a hidden boolean. Array rows carry - a hidden property's stored value through the form, so the value survives the - row being posted back; a new row gets no value (the plugin fills it in). -- **The API is unaffected.** A JSON save to `POST /api/v3/plugins/config` can - still set a hidden field. - -Older cores ignore the flag and render the field as a normal control. - -## Creating Custom Widgets - -### Step 1: Create Widget File - -Create a JavaScript file in your plugin's `widgets/` directory, named -`widgets/[widget-name].js`. The directory is not optional: it is the only -place the core will serve a widget from. - -```javascript -// Ensure LEDMatrixWidgets registry is available -if (typeof window.LEDMatrixWidgets === 'undefined') { - console.error('LEDMatrixWidgets registry not found'); - return; -} - -// Register your widget -window.LEDMatrixWidgets.register('my-custom-widget', { - name: 'My Custom Widget', - version: '1.0.0', - - /** - * Render the widget HTML - * @param {HTMLElement} container - Container element to render into - * @param {Object} config - Widget configuration from schema - * @param {*} value - Current value - * @param {Object} options - Additional options (fieldId, pluginId, etc.) - */ - render: function(container, config, value, options) { - const fieldId = options.fieldId || container.id; - - // Always escape HTML to prevent XSS - const escapeHtml = (text) => { - const div = document.createElement('div'); - div.textContent = text; - return div.innerHTML; - }; - - container.innerHTML = ` -
- -
- `; - - // Attach event listeners - const input = container.querySelector('input'); - input.addEventListener('change', (e) => { - this.handlers.onChange(fieldId, e.target.value); - }); - }, - - /** - * Get current value from widget - */ - getValue: function(fieldId) { - const input = document.querySelector(`#${fieldId}_input`); - return input ? input.value : null; - }, - - /** - * Set value programmatically - */ - setValue: function(fieldId, value) { - const input = document.querySelector(`#${fieldId}_input`); - if (input) { - input.value = value || ''; - } - }, - - /** - * Event handlers - */ - handlers: { - onChange: function(fieldId, value) { - // Trigger form change event - const event = new CustomEvent('widget-change', { - detail: { fieldId, value }, - bubbles: true - }); - document.dispatchEvent(event); - } - } -}); -``` - -### Step 2: Declare the Widget in `manifest.json` - -The manifest is the allowlist. A widget is served only if the plugin declares -it, so shipping a file under `widgets/` does not by itself publish it: - -```json -{ - "widgets": [ - { - "name": "my-custom-widget", - "script": "my-custom-widget.js", - "description": "What this widget is for" - } - ] -} -``` - -`name` is what you use in `x-widget` and in the URL. `script` is optional and -defaults to `[name].js`; it must be a plain filename directly inside -`widgets/` (no paths). Both are validated against -`schema/manifest_schema.json`. - -### Step 3: Reference Widget in Schema - -In your plugin's `config_schema.json`: - -```json -{ - "properties": { - "my_field": { - "type": "string", - "description": "My custom field", - "x-widget": "my-custom-widget", - "default": "" - } - } -} -``` - -### Step 4: Widget Loading - -The widget is loaded on demand when the plugin's configuration form renders a -field that references it. The system will: - -1. Check whether the widget is already registered in the core registry. -2. If not, fetch it from `/static/plugin-widgets/[plugin-id]/[widget-name].js`. - That route serves the declared `script` from your plugin's `widgets/` - directory, as `text/javascript`. -3. Render it by calling the `render` function your script registered. - -The fetch uses a dynamic `import()`, so the file must parse as an ES module. -A plain IIFE does — modules are strict mode, so avoid sloppy-mode constructs. - -**If the widget fails to load** (not declared, file missing, script throws, or -it never calls `register`), the field falls back to a plain text input holding -the current value. This is deliberate: a broken widget costs the user an -editor, not their configured value. - -**Limitation:** the on-demand path applies to `string`-typed fields (the -default branch of the config-form renderer). Fields typed `object`, `array`, -`boolean`, `integer` or `number`, and fields whose `enum` is set, are -dispatched by the server-side template to its own built-in renderers, so a -plugin-supplied `x-widget` on one of those is ignored today. - -## Widget API Reference - -### Widget Definition Object - -```javascript -{ - name: string, // Human-readable widget name - version: string, // Widget version - render: function, // Required: Render function - getValue: function, // Optional: Get current value - setValue: function, // Optional: Set value programmatically - handlers: object // Optional: Event handlers -} -``` - -### Render Function - -```javascript -render(container, config, value, options) -``` - -**Parameters:** -- `container` (HTMLElement): Container element to render into -- `config` (Object): Widget configuration from schema -- `value` (*): Current field value -- `options` (Object): Additional options - - `fieldId` (string): Field ID - - `pluginId` (string): Plugin ID - - `fullKey` (string): Full field key path - -### Get Value Function - -```javascript -getValue(fieldId) -``` - -**Returns:** Current widget value - -### Set Value Function - -```javascript -setValue(fieldId, value) -``` - -**Parameters:** -- `fieldId` (string): Field ID -- `value` (*): Value to set - -## Examples - -See [`web_interface/static/v3/js/widgets/example-color-picker.js`](../web_interface/static/v3/js/widgets/example-color-picker.js) for a complete example of a custom color picker widget. - -## Best Practices - -### Security - -1. **Always escape HTML**: Use `escapeHtml()` or `textContent` to prevent XSS -2. **Validate inputs**: Validate user input before processing -3. **Sanitize values**: Clean values before storing - -### Performance - -1. **Lazy loading**: Load widget scripts only when needed -2. **Event delegation**: Use event delegation for dynamic content -3. **Debounce**: Debounce frequent events (e.g., input changes) - -### Accessibility - -1. **Labels**: Always associate labels with inputs -2. **ARIA attributes**: Use appropriate ARIA attributes -3. **Keyboard navigation**: Ensure keyboard accessibility - -## Troubleshooting - -### Widget Not Loading - -1. Check browser console for errors -2. Verify widget file path is correct -3. Ensure `LEDMatrixWidgets.register()` is called -4. Check that widget name matches schema `x-widget` value - -### Widget Not Rendering - -1. Verify `render` function is defined -2. Check container element exists -3. Ensure widget is registered before form loads -4. Check for JavaScript errors in console - -### Value Not Saving - -1. Ensure widget triggers `widget-change` event -2. Verify form submission includes widget value -3. Check `getValue` function returns correct type -4. Verify field name matches schema property - -## Current Implementation Status - -**Phase 1 Complete:** -- ✅ Widget registry system created -- ✅ Core widgets extracted to separate files -- ✅ Widget handlers available globally (backwards compatible) -- ✅ Plugin widget loading system implemented - -**Current Behavior:** -- Core widgets are server-side rendered via Jinja2 templates (existing behavior preserved) -- Widget handlers are registered and available globally -- Custom widgets can be created, declared in `manifest.json`, and are served - and rendered on demand for `string`-typed fields -- Plugin widgets on non-string fields are not dispatched yet (see Step 4) - -**Backwards Compatibility:** -- All existing plugins using widgets continue to work without changes -- Server-side rendering remains the primary method -- Widget registry provides foundation for future enhancements - -## See Also - -- [Widget README](../web_interface/static/v3/js/widgets/README.md) - Complete widget development guide with examples -- [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - General plugin development -- [Plugin Configuration Guide](PLUGIN_CONFIGURATION_GUIDE.md) - Configuration setup +The widget guide lives next to the widgets, in +[web_interface/static/v3/js/widgets/README.md](../web_interface/static/v3/js/widgets/README.md). +It lists every built-in `x-widget`, the schema keywords the config form +understands (`x-options.labels`, `x-advanced`, `x-display: "hidden"`), and how +to ship a custom widget with a plugin. diff --git a/first_time_install.sh b/first_time_install.sh index 8b8ca6dc..8d67175a 100755 --- a/first_time_install.sh +++ b/first_time_install.sh @@ -18,7 +18,7 @@ on_error() { echo "-- Last 100 lines from log --" >&2 tail -n 100 "$LOG_FILE" >&2 || true fi - echo "\nCommon fixes:" >&2 + printf '\nCommon fixes:\n' >&2 echo "- Ensure the Pi is online (try: ping -c1 8.8.8.8)." >&2 echo "- If you saw an APT lock error: wait a minute, close other installers, then run: sudo dpkg --configure -a" >&2 echo "- Re-run this script. It is safe to run multiple times." >&2 @@ -115,7 +115,8 @@ fi echo "✓ OS requirements met" echo "" -# Get the actual user who invoked sudo (set after we ensure sudo below) +# The user who ran the installer: SUDO_USER once we are running under sudo +# (the re-exec below guarantees that), otherwise whoever we are now. if [ -n "${SUDO_USER:-}" ]; then ACTUAL_USER="$SUDO_USER" else @@ -202,7 +203,7 @@ echo "" # Check if running as root; if not, try to elevate automatically for novices if [ "$EUID" -ne 0 ]; then echo "This script needs administrator privileges. Attempting to re-run with sudo..." - exec sudo -E env LEDMATRIX_ELEVATED=1 bash "$0" "$@" + exec sudo -E bash "$0" "$@" fi echo "✓ Running as root (required for installation)" @@ -507,8 +508,11 @@ print_rgbmatrix_build_failure() { # it. The logic was pasted three times, identically, and is kept verbatim here. # Note: install_web_service.sh and install_service.sh no longer contain the # "User=root" / "User=${ACTUAL_USER}" strings grepped for below (the units come -# from systemd/*.service templates with User=__USER__), so until Step 8 has -# installed the unit this yields "root". +# from systemd/*.service templates with User=__USER__). So once the unit is +# installed (Step 7.5, by install_service.sh) the first branch reads its real +# User=; before that the second branch is taken whenever +# install_web_service.sh exists, matches neither string, and yields "root" -- +# the later branches are reached only if that script is missing. detect_web_service_user() { WEB_SERVICE_USER="root" if [ -f "/etc/systemd/system/ledmatrix-web.service" ]; then @@ -669,8 +673,9 @@ else echo "Setting ownership of assets directory..." chown -R "$ACTUAL_USER:$ACTUAL_USER" "$PROJECT_ROOT_DIR/assets" - # Set permissions to allow read/write for owner, group, and others (for root service user) - # Note: 777 allows root (service user) to write, which is necessary when service runs as root + # 777: read/write for owner, group and every other account. Root (the + # display service) does not need it -- root ignores mode bits -- so the + # "other" bits only matter to accounts that are neither the owner nor root. echo "Setting permissions for assets directory..." chmod -R 777 "$PROJECT_ROOT_DIR/assets" @@ -782,8 +787,8 @@ else chown -R root:"$ACTUAL_USER" "$PLUGIN_REPOS_DIR" fi -# Set directory permissions (775: rwxrwxr-x) -echo "Setting plugin-repos directory permissions to 2775 (sticky bit)..." +# Set directory permissions (2775: rwxrwsr-x, setgid so new entries inherit the group) +echo "Setting plugin-repos directory permissions to 2775 (setgid)..." find "$PLUGIN_REPOS_DIR" -type d -exec chmod 2775 {} \; # Set file permissions (664: rw-rw-r--) @@ -990,9 +995,9 @@ if [ -f "$PROJECT_ROOT_DIR/requirements.txt" ]; then PACKAGE_NUM=$((PACKAGE_NUM + 1)) echo "[$PACKAGE_NUM/$TOTAL_PACKAGES] Installing: $line" - # Check if package is already installed (basic check - may not catch all cases) - # Try installing with verbose output and timeout (if available) - # Use --no-cache-dir to avoid cache issues, --verbose for diagnostics + # Install with a timeout where available. --verbose output goes to + # $INSTALL_OUTPUT (filtered below, full copy in the log); --no-cache-dir + # avoids pip cache issues. INSTALL_OUTPUT=$(mktemp) INSTALL_SUCCESS=false @@ -1297,7 +1302,11 @@ else WEB_DEPS_OK=false fi else - echo "Web dependencies already installed from web_interface/requirements.txt in Step 5" + # No marker means Step 5 did not install web_interface/requirements.txt, + # and without the smart installer there is nothing else to try here. + echo "⚠ scripts/install_dependencies_apt.py not found, and Step 5 did not install" + echo " web_interface/requirements.txt, so web interface dependencies may be missing." + WEB_DEPS_OK=false fi # Create the marker only when installation actually succeeded, so a @@ -1562,11 +1571,12 @@ echo "-----------------------------------------------------" if [ -f "$PROJECT_ROOT_DIR/scripts/install/configure_wifi_permissions.sh" ]; then echo "Configuring WiFi management permissions..." # Run as the actual user (not root) since the script checks for that - sudo -u "$ACTUAL_USER" bash "$PROJECT_ROOT_DIR/scripts/install/configure_wifi_permissions.sh" || { + if sudo -u "$ACTUAL_USER" bash "$PROJECT_ROOT_DIR/scripts/install/configure_wifi_permissions.sh"; then + echo "✓ WiFi management permissions configured" + else echo "⚠ WiFi permissions configuration failed, but continuing installation" echo " You can run it manually later: ./scripts/install/configure_wifi_permissions.sh" - } - echo "✓ WiFi management permissions configured" + fi else echo "⚠ configure_wifi_permissions.sh not found; skipping WiFi permissions configuration" echo " You can configure WiFi permissions later by running:" @@ -1736,10 +1746,11 @@ echo "-------------------------------------" echo "Removing potential conflicting services (bluetooth and others)..." if [ "$SKIP_SOUND" = "1" ]; then echo "Skipping sound module configuration as requested (--skip-sound)." -elif apt_remove bluez bluez-firmware pi-bluetooth triggerhappy pigpio; then - echo "✓ Unnecessary services removed (or not present)" else - echo "⚠ Some packages could not be removed; continuing" + # apt_remove never fails (it ends in `|| true`); apt itself reports any + # package it could not remove. + apt_remove bluez bluez-firmware pi-bluetooth triggerhappy pigpio + echo "✓ Unnecessary services removed (or not present)" fi # Blacklist onboard sound module (idempotent) @@ -1954,20 +1965,6 @@ if systemctl list-unit-files | grep -q "ledmatrix-wifi-monitor.service"; then fi echo "" -if [ "$SKIP_REBOOT_PROMPT" = "1" ]; then - echo "Skipping reboot prompt as requested (--no-reboot-prompt)." -elif [ "$ASSUME_YES" = "1" ]; then - echo "Non-interactive mode: rebooting now to apply changes..." - reboot -else - read -p "A reboot is recommended to apply kernel and audio changes. Reboot now? (y/N): " -n 1 -r - echo - if [[ $REPLY =~ ^[Yy]$ ]]; then - echo "Rebooting now..." - reboot - fi -fi - echo "==========================================" echo "Installation Complete!" echo "==========================================" @@ -2013,7 +2010,7 @@ if command -v nmcli >/dev/null 2>&1; then if [ -n "$WIFI_STATUS" ]; then echo "$WIFI_STATUS" | while IFS=':' read -r _ _ state; do if [ "$state" = "connected" ]; then - SSID=$(nmcli -t -f active,ssid device wifi 2>/dev/null | grep "^yes:" | cut -d: -f2 | head -1) + SSID=$(nmcli -t -f active,ssid device wifi 2>/dev/null | grep "^yes:" | cut -d: -f2 | head -1 || true) if [ -n "$SSID" ]; then echo " ✓ Connected to: $SSID" else @@ -2037,7 +2034,7 @@ echo "AP Mode Status:" if systemctl is-active --quiet hostapd 2>/dev/null; then echo " ✓ AP Mode is ACTIVE" echo " → Connect to WiFi network: LEDMatrix-Setup" - echo " → Password: ledmatrix123" + echo " → Open network, no password" echo " → Access web UI at: http://192.168.4.1:5000" AP_MODE_ACTIVE=true else @@ -2045,7 +2042,7 @@ else if ip addr show wlan0 2>/dev/null | grep -q "192.168.4.1"; then echo " ✓ AP Mode is ACTIVE (IP detected)" echo " → Connect to WiFi network: LEDMatrix-Setup" - echo " → Password: ledmatrix123" + echo " → Open network, no password" echo " → Access web UI at: http://192.168.4.1:5000" AP_MODE_ACTIVE=true else @@ -2147,3 +2144,22 @@ echo " - Main config: $PROJECT_ROOT_DIR/config/config.json" echo " - Secrets: $PROJECT_ROOT_DIR/config/config_secrets.json" echo "" echo "Enjoy your LED Matrix display!" + +# Reboot last. It used to come before the summary above, so with -y (and +# the one-shot installer, which always passes -y) the reboot was already +# under way while the summary printed, and the SSH session usually dropped +# before any of it -- the web UI address included -- could be read. +echo "" +if [ "$SKIP_REBOOT_PROMPT" = "1" ]; then + echo "Skipping reboot prompt as requested (--no-reboot-prompt)." +elif [ "$ASSUME_YES" = "1" ]; then + echo "Non-interactive mode: rebooting now to apply changes..." + reboot +else + read -p "A reboot is recommended to apply kernel and audio changes. Reboot now? (y/N): " -n 1 -r + echo + if [[ $REPLY =~ ^[Yy]$ ]]; then + echo "Rebooting now..." + reboot + fi +fi diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 00000000..d4c4a5be --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,59 @@ +# Scripts + +Helper scripts for installing, repairing, diagnosing and developing +LEDMatrix. Most users only ever run the one-shot installer (see the project +README); everything else here is for troubleshooting or development. + +Status key: **keep** — part of install/runtime or referenced by docs, CI, +tests or code; **dev-only** — for plugin/core development, not needed on a +display; **diagnostic** — run by hand on a Pi when something is wrong. + +## Directories + +| Directory | Status | What it holds | +|---|---|---| +| [`install/`](install/README.md) | keep | The installers: one-shot, services, sudoers/WiFi permissions, cache setup, and the shared `lib_*.sh` helpers `first_time_install.sh` sources | +| [`fix_perms/`](fix_perms/README.md) | keep | Permission repair scripts, plus the two root helpers the web interface runs through sudo (`safe_plugin_rm.sh`, `safe_pip_install.sh`) | +| [`utils/`](utils/README.md) | keep | Scripts run by systemd units or the web interface (conditional web start, WiFi monitor, update verify, DNS fix, Pixlet config editor, cache clearing) | +| [`dev/`](dev/README.md) | dev-only | Plugin linking, emulator runner, Vegas density audit, Pillow smoke test | +| `templates/` | dev-only | `dev_preview.html`, the page `dev_server.py` serves | + +## Top-level scripts + +| Script | Status | What it does | +|---|---|---| +| `build_rgbmatrix_nogil.sh` | keep | Rebuilds the rgbmatrix Python binding so `SwapOnVSync` releases the GIL (docs/SCROLL_PERFORMANCE.md) | +| `check_plugin.py` | dev-only | Renders a plugin across every mode and matrix size and fails on crashes, overflow or golden-image drift | +| `check_release_version.py` | keep | Checks a release tag, CHANGELOG and `src.__version__` agree (release-version-check workflow) | +| `check_system_compatibility.sh` | diagnostic | Pre-install check of hardware, OS (Trixie only), kernel, Python, packages, disk and network | +| `dev_server.py` | dev-only | Browser preview server for plugins without the display loop (http://localhost:5001) | +| `diagnose_dependencies.sh` | diagnostic | Investigates pip installs stuck on "Preparing metadata" | +| `diagnose_web_interface.sh` | diagnostic | Checks why the web interface is not reachable | +| `download_pixlet.sh` | keep | Downloads the bundled Pixlet binaries for Starlark apps (also run from the web UI) | +| `emergency_reconnect.sh` | diagnostic | Reconnects to your WiFi network if captive-portal testing leaves the Pi offline | +| `install_dependencies_apt.py` | keep | Dependency installer that tries apt packages first, then pip (installer Step 7, plugin loader) | +| `install_plugin_dependencies.sh` | diagnostic | Installs plugin requirements by hand when the automatic install fails | +| `prove_security.py` | keep | Security property checks run by pre-commit | +| `render_plugin.py` | dev-only | Runs a plugin's `update()` + `display()` and saves the frame as a PNG | +| `run_plugin_tests.py` | dev-only | Discovers and runs plugin test suites | +| `scroll_speeds.py` | keep | Shows and tries the scroll speeds your panel can display cleanly | +| `troubleshoot_captive_portal.sh` | diagnostic | Troubleshoots captive-portal WiFi setup after you can SSH back in | +| `update_plugin_repos.py` | dev-only | Pulls the latest `ledmatrix-plugins` monorepo | +| `verify_installation.sh` | diagnostic | Checks that an installation completed correctly | +| `verify_wifi_setup.sh` | diagnostic | Health check of the WiFi management setup | + +## Candidates for removal + +Nothing in the repo (docs, CI, tests, other scripts or code) refers to these. +They are kept for now; each one needs an owner decision before it goes. + +| Script | What it does | +|---|---| +| `add_defaults_to_schemas.py` | One-off: adds missing `default` values to plugin config schemas | +| `analyze_plugin_schemas.py` | One-off: reports duplicate/inconsistent fields across plugin schemas | +| `audit_plugins.py` | AST security audit of plugin code; says it is "designed to run in CI" but no workflow runs it | +| `audit_render_path.py` | Finds blocking calls reachable from a plugin's `display()` | +| `sports_scroll_check.py` | Drives a sports scoreboard scroll on the panel and reports its pacing | +| `test_captive_portal.sh` | Tests the captive portal from a device connected to the AP | +| `verify_wifi_before_testing.sh` | Pre-flight check before unplugging Ethernet to test WiFi | +| `dev/test_pillow_compat.py` | Pillow API smoke test to run after upgrading Pillow | diff --git a/scripts/check_system_compatibility.sh b/scripts/check_system_compatibility.sh index e897277a..22cc5050 100755 --- a/scripts/check_system_compatibility.sh +++ b/scripts/check_system_compatibility.sh @@ -28,12 +28,12 @@ print_success() { print_warning() { echo -e "${YELLOW}⚠${NC} $1" - ((WARNINGS++)) + WARNINGS=$((WARNINGS + 1)) } print_error() { echo -e "${RED}✗${NC} $1" - ((COMPATIBILITY_ISSUES++)) + COMPATIBILITY_ISSUES=$((COMPATIBILITY_ISSUES + 1)) } # Check if running on Raspberry Pi @@ -61,20 +61,18 @@ if [ -f /etc/os-release ]; then echo "OS: $PRETTY_NAME" echo "Version ID: ${VERSION_ID:-unknown}" + # first_time_install.sh refuses anything but Raspberry Pi OS / Debian 13 + # (Trixie), so anything else is an error here too, not a warning. if [[ "$ID" == "raspbian" ]] || [[ "$ID" == "debian" ]]; then - if [ "${VERSION_ID:-0}" -ge "12" ]; then - print_success "Running compatible Debian/Raspbian version (${VERSION_ID})" - - if [ "${VERSION_ID:-0}" -eq "13" ]; then - print_success "Detected Debian 13 Trixie - full compatibility expected" - elif [ "${VERSION_ID:-0}" -eq "12" ]; then - print_success "Detected Debian 12 Bookworm - full compatibility confirmed" - fi + if [ "${VERSION_ID:-0}" = "13" ]; then + print_success "Detected Debian 13 Trixie - supported" + elif [ "${VERSION_ID:-0}" = "12" ]; then + print_error "Debian 12 Bookworm is not supported - the installer requires Raspberry Pi OS Lite (Trixie), Debian 13" else - print_warning "Old Debian/Raspbian version (${VERSION_ID}) - upgrade recommended" + print_error "Debian/Raspbian ${VERSION_ID:-unknown} is not supported - the installer requires Raspberry Pi OS Lite (Trixie), Debian 13" fi else - print_warning "Not running Debian/Raspbian - compatibility not guaranteed" + print_error "${ID:-unknown} is not supported - the installer requires Raspberry Pi OS Lite (Trixie), Debian 13" fi else print_error "Could not detect OS version" diff --git a/scripts/dev/README.md b/scripts/dev/README.md index 01b1ebc9..6dfdc676 100644 --- a/scripts/dev/README.md +++ b/scripts/dev/README.md @@ -6,6 +6,8 @@ This directory contains scripts and utilities for development and testing. - **`dev_plugin_setup.sh`** - Sets up plugin development environment by linking plugin repositories - **`run_emulator.sh`** - Runs the LED Matrix display in emulator mode (for development without hardware) +- **`vegas_audit.py`** - Measures how much of the Vegas ticker strip actually shows content (dead-frame ratio) +- **`test_pillow_compat.py`** - Pillow API smoke test to run after upgrading Pillow (`python3 scripts/dev/test_pillow_compat.py`) ## Usage diff --git a/scripts/fix_perms/README.md b/scripts/fix_perms/README.md index 77575a71..d3843ee4 100644 --- a/scripts/fix_perms/README.md +++ b/scripts/fix_perms/README.md @@ -18,21 +18,26 @@ system user. permissions on the `assets/` tree so plugins can download and cache team logos, fonts, and other static content. -- **`fix_cache_permissions.sh`** — Creates (if missing) and fixes - permissions on `/var/cache/ledmatrix/` and `~/.ledmatrix_cache/` of the - user running `sudo`, and creates - `/var/cache/ledmatrix/placeholder_logos/` for the sports plugins. It does - not touch the cache manager's other fallbacks (`/opt/ledmatrix/cache`, - `$TMPDIR/ledmatrix_cache`). +- **`fix_cache_permissions.sh`** — Restores `/var/cache/ledmatrix/` to the + shared `ledmatrix`-group setup by running + `scripts/install/setup_cache.sh` (the same script the installer uses), + and creates/fixes `~/.ledmatrix_cache/` of the user running `sudo`. It + does not touch the cache manager's other fallbacks + (`/opt/ledmatrix/cache`, `$TMPDIR/ledmatrix_cache`). - **`fix_plugin_permissions.sh`** — Fixes ownership on the plugins directory so both the root display service and the web service user can read and write plugin files (manifests, configs, requirements installs). -- **`fix_web_permissions.sh`** — Fixes permissions on log files, - systemd journal access, and the sudoers entries the web interface - needs to control the display service. +- **`fix_web_permissions.sh`** — Adds you to the `systemd-journal` and + `adm` groups so the web UI can read logs, and makes the project + directory yours again, keeping the root-owned sudo helpers + (`safe_plugin_rm.sh`, `safe_pip_install.sh`) and `config_secrets.json` + the way the installer leaves them. Run it as the web interface's user, + **without** `sudo` (it refuses to run as root and calls `sudo` itself). + It does not write sudoers rules; that is + `scripts/install/configure_web_sudo.sh`. - **`safe_pip_install.sh`** — Installs a `requirements.txt` as root after checking it is the project's own or one under `plugin-repos/` or @@ -67,7 +72,9 @@ Run these scripts only when: sudo ./scripts/fix_perms/fix_cache_permissions.sh sudo ./scripts/fix_perms/fix_assets_permissions.sh sudo ./scripts/fix_perms/fix_plugin_permissions.sh -sudo ./scripts/fix_perms/fix_web_permissions.sh + +# Run as the web interface's user, without sudo (it asks for sudo itself) +./scripts/fix_perms/fix_web_permissions.sh ``` If you're not sure which one you need, run `fix_cache_permissions.sh` diff --git a/scripts/fix_perms/fix_assets_permissions.sh b/scripts/fix_perms/fix_assets_permissions.sh index 9000a338..bd28ae45 100755 --- a/scripts/fix_perms/fix_assets_permissions.sh +++ b/scripts/fix_perms/fix_assets_permissions.sh @@ -36,11 +36,12 @@ else exit 1 fi -# Set permissions to allow read/write for owner, group, and others (for root service user) -# Note: 777 allows root (service user) to write, which is necessary when service runs as root +# 777: read/write for owner, group and every other account. Root (the display +# service) does not need it -- root ignores mode bits -- so the "other" bits +# only matter to accounts that are neither $REAL_USER nor root. echo "Setting permissions for assets directory..." if sudo chmod -R 777 "$ASSETS_DIR"; then - echo "✓ Set assets directory permissions to 777 (writable by root service user)" + echo "✓ Set assets directory permissions to 777" else echo "✗ Failed to set assets directory permissions" exit 1 @@ -70,8 +71,7 @@ for SPORTS_DIR in "${SPORTS_DIRS[@]}"; do echo " - Current permissions:" ls -ld "$FULL_PATH" - # Ensure the directory is writable by both the real user and root (service user) - # Use 777 permissions to allow root (service) to write, or set group ownership + # Owned by the real user; 777 as above (root can write here regardless) sudo chmod 777 "$FULL_PATH" sudo chown "$REAL_USER:$REAL_GROUP" "$FULL_PATH" diff --git a/scripts/fix_perms/fix_cache_permissions.sh b/scripts/fix_perms/fix_cache_permissions.sh index a3fa3e15..4225b376 100755 --- a/scripts/fix_perms/fix_cache_permissions.sh +++ b/scripts/fix_perms/fix_cache_permissions.sh @@ -1,11 +1,23 @@ #!/bin/bash # LEDMatrix Cache Permissions Fix Script -# This script fixes permissions on all known cache directories so they're writable by the daemon or current user -# Also sets up placeholder logo directories for sports managers +# +# /var/cache/ledmatrix is shared by the display service (root) and the web +# interface (your user) through the ledmatrix group: root:ledmatrix, 2775, +# files 660. scripts/install/setup_cache.sh is what sets that up (the +# installer's Step 2 runs it, and install_web_service.sh keeps the group), so +# this script runs it rather than applying a model of its own. It used to set +# the directory 777 and re-group it to your own group, replacing the ledmatrix +# group everything else relies on. +# +# It also repairs ~/.ledmatrix_cache, the cache manager's fallback when +# /var/cache/ledmatrix is unusable. echo "Fixing LEDMatrix cache directory permissions..." +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SETUP_CACHE="$SCRIPT_DIR/../install/setup_cache.sh" + # Get the real user (not root when running with sudo) REAL_USER=${SUDO_USER:-$USER} # Resolve the home directory of the real user robustly @@ -16,72 +28,36 @@ else fi REAL_GROUP=$(id -gn "$REAL_USER") -# Known cache directories for LEDMatrix. Use the actual user's home instead of a hard-coded path. -CACHE_DIRS=( - "/var/cache/ledmatrix" - "$REAL_HOME/.ledmatrix_cache" -) - -for CACHE_DIR in "${CACHE_DIRS[@]}"; do - echo "" - echo "Checking cache directory: $CACHE_DIR" - if [ ! -d "$CACHE_DIR" ]; then - echo " - Directory does not exist. Creating it..." - sudo mkdir -p "$CACHE_DIR" - fi - echo " - Current permissions:" - ls -ld "$CACHE_DIR" - echo " - Fixing permissions..." - # Make directory writable by services regardless of user context - sudo chmod 777 "$CACHE_DIR" - sudo chown "$REAL_USER":"$REAL_GROUP" "$CACHE_DIR" - echo " - Updated permissions:" - ls -ld "$CACHE_DIR" - echo " - Testing write access as $REAL_USER..." - if sudo -u "$REAL_USER" test -w "$CACHE_DIR"; then - echo " ✓ $CACHE_DIR is now writable by $REAL_USER" - else - echo " ✗ $CACHE_DIR is still not writable by $REAL_USER" - fi - echo " - Permissions fix complete for $CACHE_DIR." -done - -# Set up placeholder logos directory for sports managers echo "" -echo "Setting up placeholder logos directory for sports managers..." - -PLACEHOLDER_DIR="/var/cache/ledmatrix/placeholder_logos" -if [ ! -d "$PLACEHOLDER_DIR" ]; then - echo "Creating placeholder logos directory: $PLACEHOLDER_DIR" - sudo mkdir -p "$PLACEHOLDER_DIR" - sudo chown "$REAL_USER":"$REAL_GROUP" "$PLACEHOLDER_DIR" - sudo chmod 777 "$PLACEHOLDER_DIR" +echo "Checking cache directory: /var/cache/ledmatrix" +if [ -f "$SETUP_CACHE" ]; then + bash "$SETUP_CACHE" else - echo "Placeholder logos directory already exists: $PLACEHOLDER_DIR" - sudo chmod 777 "$PLACEHOLDER_DIR" - sudo chown "$REAL_USER":"$REAL_GROUP" "$PLACEHOLDER_DIR" + echo " ✗ $SETUP_CACHE not found; /var/cache/ledmatrix left unchanged." fi +CACHE_DIR="$REAL_HOME/.ledmatrix_cache" +echo "" +echo "Checking cache directory: $CACHE_DIR" +if [ ! -d "$CACHE_DIR" ]; then + echo " - Directory does not exist. Creating it..." + sudo mkdir -p "$CACHE_DIR" +fi echo " - Current permissions:" -ls -ld "$PLACEHOLDER_DIR" +ls -ld "$CACHE_DIR" +echo " - Fixing permissions..." +sudo chmod 777 "$CACHE_DIR" +sudo chown "$REAL_USER":"$REAL_GROUP" "$CACHE_DIR" +echo " - Updated permissions:" +ls -ld "$CACHE_DIR" echo " - Testing write access as $REAL_USER..." -if sudo -u "$REAL_USER" test -w "$PLACEHOLDER_DIR"; then - echo " ✓ Placeholder logos directory is writable by $REAL_USER" +if sudo -u "$REAL_USER" test -w "$CACHE_DIR"; then + echo " ✓ $CACHE_DIR is now writable by $REAL_USER" else - echo " ✗ Placeholder logos directory is not writable by $REAL_USER" -fi - -# Test with daemon user (which the system might run as) -if sudo -u daemon test -w "$PLACEHOLDER_DIR" 2>/dev/null; then - echo " ✓ Placeholder logos directory is writable by daemon user" -else - echo " ✗ Placeholder logos directory is not writable by daemon user" + echo " ✗ $CACHE_DIR is still not writable by $REAL_USER" fi +echo " - Permissions fix complete for $CACHE_DIR." echo "" echo "All cache directory permission fixes attempted." echo "If you still see errors, check which user is running the LEDMatrix service and ensure it matches the owner above." -echo "" -echo "The system will now create placeholder logos in:" -echo " $PLACEHOLDER_DIR" -echo "This should eliminate the permission denied warnings for sports logos." \ No newline at end of file diff --git a/scripts/fix_perms/fix_plugin_permissions.sh b/scripts/fix_perms/fix_plugin_permissions.sh index 94d96255..e5bd515d 100755 --- a/scripts/fix_perms/fix_plugin_permissions.sh +++ b/scripts/fix_perms/fix_plugin_permissions.sh @@ -51,9 +51,10 @@ fi echo "Setting ownership to root:$ACTUAL_USER..." sudo chown -R root:"$ACTUAL_USER" "$PLUGINS_DIR" -# Set directory permissions (775: rwxrwxr-x) -# Root: read/write/execute, Group (ACTUAL_USER): read/write/execute, Others: read/execute -echo "Setting directory permissions to 2775 (rwxrwxr-x + sticky bit)..." +# Set directory permissions (2775: rwxrwsr-x) +# Owner (root) and group (ACTUAL_USER): read/write/execute, others: read/execute. +# The setgid bit makes new entries inherit the ACTUAL_USER group. +echo "Setting directory permissions to 2775 (rwxrwsr-x, setgid)..." find "$PLUGINS_DIR" -type d -exec sudo chmod 2775 {} \; # Set file permissions (664: rw-rw-r--) @@ -71,7 +72,7 @@ fi echo "Setting ownership of plugin-repos to root:$ACTUAL_USER..." sudo chown -R root:"$ACTUAL_USER" "$PLUGIN_REPOS_DIR" -echo "Setting plugin-repos directory permissions to 2775 (rwxrwxr-x + sticky bit)..." +echo "Setting plugin-repos directory permissions to 2775 (rwxrwsr-x, setgid)..." find "$PLUGIN_REPOS_DIR" -type d -exec sudo chmod 2775 {} \; echo "Setting plugin-repos file permissions to 664..." @@ -87,7 +88,7 @@ echo "plugin-repos/:" ls -la "$PLUGIN_REPOS_DIR" 2>/dev/null || echo " (empty or not accessible)" echo "" echo "Permissions summary:" -echo "- Root service: Can read/write plugins (for PWM hardware access)" +echo "- Root service: Can read/write plugins (as root it needs no permission bits)" echo "- Web service ($ACTUAL_USER): Can read/write plugins (for installation)" echo "- Others: Can read plugins" diff --git a/scripts/fix_perms/fix_web_permissions.sh b/scripts/fix_perms/fix_web_permissions.sh index 51b8fb7f..2a916f42 100755 --- a/scripts/fix_perms/fix_web_permissions.sh +++ b/scripts/fix_perms/fix_web_permissions.sh @@ -25,8 +25,9 @@ echo "" echo "This script will:" echo "1. Add the web user to the 'systemd-journal' group for log access" echo "2. Add the web user to the 'adm' group for additional system access" -echo "3. Configure sudoers for passwordless access to system commands" -echo "4. Set proper file permissions" +echo "3. Make the project directory yours again, keeping the root-owned sudo" +echo " helpers and config_secrets.json as the installer leaves them" +echo " (sudoers rules are configure_web_sudo.sh's job, not this script's)" echo "" # Ask for confirmation @@ -62,6 +63,51 @@ else echo "✗ Failed to set project ownership" fi +# The chown above also takes back two kinds of file that first_time_install.sh +# deliberately keeps from the web user. Put them back the way the installer +# leaves them (its Steps 11 and 11.1), whether or not the chown succeeded. +# +# 1. The helpers /etc/sudoers.d/ledmatrix_web lets the web user run as root +# (scripts/install/lib_sudoers.sh). A copy the web user owns is a root shell +# for whoever can edit it, so they stay root-owned and writable by root only. +# Keep this list in step with the installer's Step 11.1 loop; +# test/test_web_sudoers_installers_agree.py checks both against the grants. +for helper in safe_plugin_rm.sh safe_pip_install.sh; do + HELPER_PATH="$PROJECT_DIR/scripts/fix_perms/$helper" + if [ -f "$HELPER_PATH" ]; then + if sudo chown root:root "$HELPER_PATH" && sudo chmod 755 "$HELPER_PATH"; then + echo "✓ $helper is root-owned again (sudo runs it as root)" + else + echo "⚠ Could not make $HELPER_PATH root-owned, mode 755." + echo " Fix it by hand: sudo chown root:root $HELPER_PATH && sudo chmod 755 $HELPER_PATH" + fi + fi +done + +# 2. config_secrets.json: owned by the account ledmatrix-web.service runs as, +# group ledmatrix, mode 640 -- the same owner, group and mode as the +# installer's Step 11 gives it. +SECRETS_FILE="$PROJECT_DIR/config/config_secrets.json" +if [ -f "$SECRETS_FILE" ]; then + SECRETS_OWNER="" + if [ -f /etc/systemd/system/ledmatrix-web.service ]; then + SECRETS_OWNER=$(grep -m1 "^User=" /etc/systemd/system/ledmatrix-web.service | cut -d'=' -f2 || true) + fi + SECRETS_OWNER="${SECRETS_OWNER:-$WEB_USER}" + if getent group ledmatrix >/dev/null 2>&1; then + SECRETS_OWNERSHIP="$SECRETS_OWNER:ledmatrix" + else + # No ledmatrix group means the installer never ran; keep the chown's group. + SECRETS_OWNERSHIP="$SECRETS_OWNER" + fi + if sudo chown "$SECRETS_OWNERSHIP" "$SECRETS_FILE" && sudo chmod 640 "$SECRETS_FILE"; then + echo "✓ config_secrets.json restored to $SECRETS_OWNERSHIP, mode 640" + else + echo "⚠ Could not restore $SECRETS_FILE to $SECRETS_OWNERSHIP, mode 640." + echo " Fix it by hand: sudo chown $SECRETS_OWNERSHIP $SECRETS_FILE && sudo chmod 640 $SECRETS_FILE" + fi +fi + # Set proper permissions for config files if sudo chmod 644 "$PROJECT_DIR/config/config.json" 2>/dev/null; then echo "✓ Set config file permissions" @@ -86,7 +132,7 @@ echo "Step 5: Testing sudo access..." if sudo -n systemctl status ledmatrix.service > /dev/null 2>&1; then echo "✓ Sudo access test passed" else - echo "⚠ Sudo access test failed - you may need to run configure_web_sudo.sh" + echo "⚠ Sudo access test failed - you may need to run scripts/install/configure_web_sudo.sh" fi echo "" @@ -101,5 +147,5 @@ echo "" echo "After logging back in, test journal access with:" echo " journalctl --no-pager --lines=5" echo "" -echo "If you still have sudo issues, run:" -echo " ./configure_web_sudo.sh" +echo "If you still have sudo issues, run (as this user, without sudo):" +echo " $PROJECT_DIR/scripts/install/configure_web_sudo.sh" diff --git a/scripts/install/README.md b/scripts/install/README.md index 544ea336..530960f7 100644 --- a/scripts/install/README.md +++ b/scripts/install/README.md @@ -19,6 +19,21 @@ This directory contains scripts for installing and configuring the LEDMatrix sys (the user who runs the script, i.e. the one you installed LEDMatrix as; there is no `ledmatrix` system user) the passwordless `nmcli` and related WiFi permissions the web interface needs +- **`install_dns_fix.sh`** - Optional. Installs `ledmatrix-dns-fix.service`, + which adds `options single-request` to the resolver when API calls time + out (see `systemd/README.md`) +- **`install_mqtt_bridge.sh`** - Optional. Installs the Home Assistant MQTT + bridge service (see `integrations/mqtt_bridge/README.md`) + +Libraries (sourced, not run): + +- **`lib_sudoers.sh`** - The web interface's sudo allow-list + (`/etc/sudoers.d/ledmatrix_web`), shared by `first_time_install.sh` and + `configure_web_sudo.sh` +- **`lib_systemd_render.sh`** - `sed_escape_replacement`, used by every + script that renders a unit from `systemd/*.service` +- **`lib_lowmem.sh`** - Build-job sizing and temporary swap for the C++ + build on low-memory Pis (`first_time_install.sh` Step 6) ## Usage diff --git a/scripts/install/configure_web_sudo.sh b/scripts/install/configure_web_sudo.sh index 91ba1d3b..c0fc0a06 100755 --- a/scripts/install/configure_web_sudo.sh +++ b/scripts/install/configure_web_sudo.sh @@ -26,7 +26,6 @@ fi # Get the full paths to commands and validate each one MISSING_CMDS=() -PYTHON_PATH=$(command -v python3) || true SYSTEMCTL_PATH=$(command -v systemctl) || true REBOOT_PATH=$(command -v reboot) || true POWEROFF_PATH=$(command -v poweroff) || true @@ -35,8 +34,8 @@ JOURNALCTL_PATH=$(command -v journalctl) || true SAFE_RM_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh" SAFE_PIP_INSTALL_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_pip_install.sh" -# Validate required commands (systemctl, bash, python3 are essential) -for CMD_NAME in SYSTEMCTL_PATH BASH_PATH PYTHON_PATH; do +# Validate required commands (systemctl and bash are essential) +for CMD_NAME in SYSTEMCTL_PATH BASH_PATH; do CMD_VAL="${!CMD_NAME}" if [ -z "$CMD_VAL" ]; then MISSING_CMDS+=("$CMD_NAME") @@ -70,7 +69,6 @@ fi . "$SUDOERS_LIB" echo "Command paths:" -echo " Python: $PYTHON_PATH" echo " Systemctl: $SYSTEMCTL_PATH" echo " Reboot: ${REBOOT_PATH:-(not found, skipping)}" echo " Poweroff: ${POWEROFF_PATH:-(not found, skipping)}" @@ -79,14 +77,24 @@ echo " Journalctl: ${JOURNALCTL_PATH:-(not found, skipping)}" echo " Safe plugin rm: $SAFE_RM_PATH" echo " Safe pip install: $SAFE_PIP_INSTALL_PATH" -# Create a temporary sudoers file -TEMP_SUDOERS="/tmp/ledmatrix_web_sudoers_$$" +# Create a temporary sudoers file. A predictable name in a world-writable +# directory is a symlink target, and these rules end up in /etc/sudoers.d, so +# let mktemp pick the name; the trap removes it however the script ends. +TEMP_SUDOERS=$(mktemp "${TMPDIR:-/tmp}/ledmatrix_web_sudoers.XXXXXX") || { + echo "Error: could not create a temporary file" >&2 + exit 1 +} +trap 'rm -f "$TEMP_SUDOERS"' EXIT web_sudoers_rules "$WEB_USER" "$PROJECT_ROOT" "$SYSTEMCTL_PATH" "$BASH_PATH" \ "$REBOOT_PATH" "$POWEROFF_PATH" "$JOURNALCTL_PATH" > "$TEMP_SUDOERS" # Never offer to install rules we have not parsed. A malformed drop-in in # /etc/sudoers.d makes sudo refuse every command for every user. +# visudo lives in /usr/sbin, which is not on every user's PATH. +if ! command -v visudo >/dev/null 2>&1 && [ -x /usr/sbin/visudo ]; then + PATH="$PATH:/usr/sbin" +fi if command -v visudo >/dev/null 2>&1; then if ! visudo -c -f "$TEMP_SUDOERS" >/dev/null 2>&1; then echo "" @@ -96,6 +104,8 @@ if command -v visudo >/dev/null 2>&1; then rm -f "$TEMP_SUDOERS" exit 1 fi +else + echo "⚠ visudo not found; the rules below have not been validated" fi echo "" @@ -124,7 +134,7 @@ if [[ ! $REPLY =~ ^[Yy]$ ]]; then exit 0 fi -# Apply the configuration using visudo +# Apply the configuration echo "Applying sudoers configuration..." # Harden the helper script: root-owned, not writable by web user echo "Hardening safe_plugin_rm.sh ownership..." @@ -143,21 +153,29 @@ if ! sudo chmod 755 "$SAFE_PIP_INSTALL_PATH"; then fi if sudo cp "$TEMP_SUDOERS" /etc/sudoers.d/ledmatrix_web; then + # sudo reads /etc/sudoers.d files that are root-owned and not writable by + # group or other; 440 is the mode visudo and first_time_install.sh use. + if ! sudo chmod 440 /etc/sudoers.d/ledmatrix_web; then + echo "Warning: could not set mode 440 on /etc/sudoers.d/ledmatrix_web" + fi echo "Configuration applied successfully!" echo "" echo "Testing sudo access..." - # Test a few commands - if sudo -n systemctl status ledmatrix.service > /dev/null 2>&1; then + # Ask sudo whether two of the new rules let this user in without a + # password. `sudo -l CMD` answers from the rules without running CMD, so + # this does not depend on whether ledmatrix.service is running, and it + # tests commands the rules actually grant. + if sudo -n -l "$SYSTEMCTL_PATH" status ledmatrix.service > /dev/null 2>&1; then echo "✓ systemctl status ledmatrix.service - OK" else - echo "✗ systemctl status ledmatrix.service - Failed" + echo "✗ systemctl status ledmatrix.service - not allowed without a password" fi - - if sudo -n test -f "$PROJECT_ROOT/start_display.sh"; then - echo "✓ File access test - OK" + + if sudo -n -l "$BASH_PATH" "$SAFE_RM_PATH" "$PROJECT_ROOT/plugin-repos/example" > /dev/null 2>&1; then + echo "✓ safe_plugin_rm.sh helper - OK" else - echo "✗ File access test - Failed" + echo "✗ safe_plugin_rm.sh helper - not allowed without a password" fi echo "" diff --git a/scripts/install/configure_wifi_permissions.sh b/scripts/install/configure_wifi_permissions.sh index 37e2c7db..3db862d1 100755 --- a/scripts/install/configure_wifi_permissions.sh +++ b/scripts/install/configure_wifi_permissions.sh @@ -144,6 +144,11 @@ $WEB_USER ALL=(ALL) NOPASSWD: $MKDIR_PATH -p /etc/NetworkManager/dnsmasq-shared. $WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/cp /tmp/hostapd.conf /etc/hostapd/hostapd.conf $WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/cp /tmp/dnsmasq.conf /etc/dnsmasq.d/ledmatrix-captive.conf $WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/rm -f /etc/dnsmasq.d/ledmatrix-captive.conf +# The same captive-portal DNS drop-in for NetworkManager's shared-mode dnsmasq +# (wifi_manager._write_nm_dnsmasq_captive_conf / _remove_nm_dnsmasq_captive_conf), +# exact paths. +$WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/cp /tmp/ledmatrix-nm-dnsmasq.conf /etc/NetworkManager/dnsmasq-shared.d/ledmatrix-captive.conf +$WEB_USER ALL=(ALL) NOPASSWD: /usr/bin/rm -f /etc/NetworkManager/dnsmasq-shared.d/ledmatrix-captive.conf EOF echo "Generated sudoers configuration:" @@ -151,6 +156,21 @@ echo "--------------------------------" cat "$TEMP_SUDOERS" echo "--------------------------------" +# Never install rules we have not parsed. A malformed drop-in in +# /etc/sudoers.d makes sudo refuse every command for every user, which on a +# headless Pi leaves no way in at all. first_time_install.sh and +# configure_web_sudo.sh check their rules the same way. +if command -v visudo >/dev/null 2>&1; then + if ! visudo -c -f "$TEMP_SUDOERS" >/dev/null 2>&1; then + echo "✗ The generated sudoers rules did not parse:" >&2 + visudo -c -f "$TEMP_SUDOERS" >&2 || true + echo " Leaving $SUDOERS_FILE unchanged." >&2 + exit 1 + fi +else + echo "⚠ visudo not found; installing the sudoers rules unvalidated" +fi + # Apply the sudoers configuration echo "" echo "Applying sudoers configuration..." @@ -213,11 +233,14 @@ rm -f "$TEMP_POLKIT" echo "" echo "Step 3: Testing permissions..." -# Test sudo access -if sudo -n "$NMCLI_PATH" device status > /dev/null 2>&1; then - echo "✓ nmcli device status - OK" +# Ask sudo whether one of the new rules lets this user in without a password. +# `sudo -l CMD` answers from the rules without running CMD, so the radio is +# left alone. (This used to run `nmcli device status`, which is not granted, +# so it could only ever report a failure.) +if sudo -n -l "$NMCLI_PATH" radio wifi on > /dev/null 2>&1; then + echo "✓ nmcli radio wifi on - OK" else - echo "✗ nmcli device status - Failed (this is expected if not connected)" + echo "✗ nmcli radio wifi on - not allowed without a password" fi echo "" diff --git a/scripts/install/install_dns_fix.sh b/scripts/install/install_dns_fix.sh index 1edf2785..99060517 100755 --- a/scripts/install/install_dns_fix.sh +++ b/scripts/install/install_dns_fix.sh @@ -10,6 +10,10 @@ set -e PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd) + +# shellcheck source=scripts/install/lib_systemd_render.sh +source "$PROJECT_ROOT_DIR/scripts/install/lib_systemd_render.sh" + SERVICE_NAME="ledmatrix-dns-fix" UNIT_SRC="$PROJECT_ROOT_DIR/systemd/$SERVICE_NAME.service" UNIT_DEST="/etc/systemd/system/$SERVICE_NAME.service" @@ -34,7 +38,8 @@ fi chmod +x "$PROJECT_ROOT_DIR/scripts/utils/apply_dns_single_request.sh" echo "Installing $UNIT_DEST..." -sed "s|__PROJECT_ROOT_DIR__|$PROJECT_ROOT_DIR|g" "$UNIT_SRC" \ +ESCAPED_PROJECT_ROOT_DIR=$(sed_escape_replacement "$PROJECT_ROOT_DIR") +sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g" "$UNIT_SRC" \ | $SUDO tee "$UNIT_DEST" > /dev/null # Order ledmatrix.service after the fix. `Before=` in the unit itself only diff --git a/scripts/install/install_mqtt_bridge.sh b/scripts/install/install_mqtt_bridge.sh index 7e65ed08..86eea9bd 100755 --- a/scripts/install/install_mqtt_bridge.sh +++ b/scripts/install/install_mqtt_bridge.sh @@ -9,6 +9,10 @@ set -e PROJECT_ROOT_DIR=$(cd "$(dirname "$0")/../.." && pwd) + +# shellcheck source=scripts/install/lib_systemd_render.sh +source "$PROJECT_ROOT_DIR/scripts/install/lib_systemd_render.sh" + BRIDGE_DIR="$PROJECT_ROOT_DIR/integrations/mqtt_bridge" SERVICE_NAME="ledmatrix-mqtt-bridge" UNIT_SRC="$PROJECT_ROOT_DIR/systemd/$SERVICE_NAME.service" @@ -40,7 +44,8 @@ python3 -m pip install -r "$BRIDGE_DIR/requirements.txt" 2>/dev/null \ || python3 -m pip install --break-system-packages -r "$BRIDGE_DIR/requirements.txt" echo "Installing $UNIT_DEST..." -sed "s|__PROJECT_ROOT_DIR__|$PROJECT_ROOT_DIR|g" "$UNIT_SRC" \ +ESCAPED_PROJECT_ROOT_DIR=$(sed_escape_replacement "$PROJECT_ROOT_DIR") +sed "s|__PROJECT_ROOT_DIR__|$ESCAPED_PROJECT_ROOT_DIR|g" "$UNIT_SRC" \ | $SUDO tee "$UNIT_DEST" > /dev/null $SYSTEMCTL_CMD daemon-reload diff --git a/scripts/install/install_wifi_monitor.sh b/scripts/install/install_wifi_monitor.sh index 0c24b0b8..af934aa4 100755 --- a/scripts/install/install_wifi_monitor.sh +++ b/scripts/install/install_wifi_monitor.sh @@ -51,20 +51,25 @@ if [ ${#MISSING_PACKAGES[@]} -gt 0 ]; then # Install packages automatically (no prompt) # Use apt directly if running as root, otherwise use sudo + PACKAGES_OK=true if [ "$EUID" -eq 0 ]; then apt update || echo "⚠ apt update failed, continuing anyway..." apt install -y "${MISSING_PACKAGES[@]}" || { + PACKAGES_OK=false echo "⚠ Package installation failed, but continuing with WiFi monitor setup" echo " You may need to install packages manually: apt install -y ${MISSING_PACKAGES[*]}" } else sudo apt update || echo "⚠ apt update failed, continuing anyway..." sudo apt install -y "${MISSING_PACKAGES[@]}" || { + PACKAGES_OK=false echo "⚠ Package installation failed, but continuing with WiFi monitor setup" echo " You may need to install packages manually: sudo apt install -y ${MISSING_PACKAGES[*]}" } fi - echo "✓ Package installation completed" + if [ "$PACKAGES_OK" = true ]; then + echo "✓ Package installation completed" + fi fi # Render the unit from systemd/ledmatrix-wifi-monitor.service rather than diff --git a/scripts/install/lib_systemd_render.sh b/scripts/install/lib_systemd_render.sh index d07dcec1..1431b06e 100644 --- a/scripts/install/lib_systemd_render.sh +++ b/scripts/install/lib_systemd_render.sh @@ -2,9 +2,10 @@ # # Shared helper for rendering systemd unit templates via sed. # -# Sourced by install_service.sh, install_web_service.sh and -# install_wifi_monitor.sh so all three escape sed replacement text the same -# way instead of carrying three copies of the same fix. +# Sourced by install_service.sh, install_web_service.sh, +# install_wifi_monitor.sh, install_dns_fix.sh and install_mqtt_bridge.sh so +# every unit renderer escapes sed replacement text the same way instead of +# carrying its own copy of the fix. # sed_escape_replacement VALUE # diff --git a/scripts/install/one-shot-install.sh b/scripts/install/one-shot-install.sh index 37276027..cec278b0 100755 --- a/scripts/install/one-shot-install.sh +++ b/scripts/install/one-shot-install.sh @@ -205,27 +205,6 @@ check_sudo() { print_success "Sudo access confirmed" } -# Fix /tmp permissions if needed (common issue when running via curl | bash) -# Note: /tmp permission fixing is now done inline before running first_time_install.sh -# This function is kept for backward compatibility but not actively used -fix_tmp_permissions() { - CURRENT_STEP="TMP directory check" - # Only fix if /tmp is actually not writable (don't preemptively fix) - if [ ! -w /tmp ]; then - print_warning "/tmp is not writable, attempting to fix..." - if [ "$EUID" -eq 0 ]; then - chmod 1777 /tmp 2>/dev/null || true - else - sudo chmod 1777 /tmp 2>/dev/null || true - fi - fi - - # Ensure TMPDIR is set correctly - if [ -z "${TMPDIR:-}" ] || [ ! -w "${TMPDIR:-/tmp}" ]; then - export TMPDIR=/tmp - fi -} - # Main installation function main() { print_step "LED Matrix One-Shot Installation" @@ -429,6 +408,13 @@ main() { print_step "Installation Complete!" print_success "LED Matrix has been successfully installed!" echo "" + # first_time_install.sh -y reboots as its last action, so by now the + # reboot is under way (unless LEDMATRIX_SKIP_REBOOT_PROMPT=1 was set). + if [ "${LEDMATRIX_SKIP_REBOOT_PROMPT:-0}" != "1" ]; then + echo "The installer has just started a reboot to finish setup, so this" + echo "session may disconnect now. Give the Pi a few minutes to come back, then:" + echo "" + fi echo "Next steps:" echo " 1. Configure your settings: sudo nano $REPO_DIR/config/config.json" if command -v hostname >/dev/null 2>&1; then @@ -449,7 +435,7 @@ main() { else echo " 2. Or use the web interface: http://:5000" fi - echo " 3. Start the service: sudo systemctl start ledmatrix.service" + echo " 3. The display service starts on boot; to start it by hand: sudo systemctl start ledmatrix.service" echo "" else print_error "Main installation script exited with code $INSTALL_EXIT_CODE" diff --git a/scripts/utils/README.md b/scripts/utils/README.md index 9339d65d..6df34130 100644 --- a/scripts/utils/README.md +++ b/scripts/utils/README.md @@ -9,6 +9,7 @@ This directory contains utility scripts for maintenance and system operations. - **`wifi_monitor_daemon.py`** - Background daemon that monitors WiFi/Ethernet connection and manages access point mode - **`pixlet_config_editor.sh`** - Opens Pixlet's own config UI for one installed Starlark app - **`apply_dns_single_request.sh`** - Adds `options single-request` to the resolver (run by `ledmatrix-dns-fix.service`) +- **`auto_update_verify.py`** - Health check after an automatic update, rolling back if it fails (the updater copies it to `data/` before pulling and `ledmatrix-update-verify.service` runs that copy) ## Usage diff --git a/src/common/README.md b/src/common/README.md index 4c682574..68c22404 100644 --- a/src/common/README.md +++ b/src/common/README.md @@ -1,62 +1,242 @@ -# Common Utilities +# src/common -This directory contains reusable utilities and helpers for LEDMatrix plugins and core modules. +Helpers shared by core and plugins. This page lists every module, what it is +for, and whether plugins are expected to import it. -## Adaptive Layout & Images (`src/adaptive_layout.py`, `src/adaptive_images.py`) +Rules for the package: -The recommended way to lay out plugins that render legibly on **any** panel -size (64x32 through 256x128+) without hand-tuned coordinates. Re-exported -from `src.common` for convenience; canonical import paths are -`src.adaptive_layout` / `src.adaptive_images`. +- Every module must import without display hardware: nothing here may import + `src.display_manager` or `src.plugin_system` at module level + ([`test/test_common_is_hardware_free.py`](../../test/test_common_is_hardware_free.py)). + That keeps plugins that use it loadable by the web preview, + `scripts/check_plugin.py` and tests on a laptop. +- A plugin that imports a module added in a given core release must declare + that release as its minimum (`ledmatrix_min_version` in the manifest's + `versions` entry). The "Since" column gives the release; "—" means it + predates 3.1.0, "n/a" that plugins should not import it. +- `from src.common import ...` re-exports `APIHelper`, `ScrollHelper`, + `LogoHelper`, `TextHelper`, `scroll_config` (plus `ScrollSettings`, + `configure_scroll`, `resolve_scroll_settings`, `refresh_hz_from_config`) and + the adaptive layout names below ([`__init__.py`](__init__.py)). + +## Summary + +| Module | For | Plugins import it? | Since | +|---|---|---|---| +| [`api_helper`](#api_helper) | HTTP GET/POST with caching and rate limiting | Yes | — | +| [`bdf_font`](#bdf_font) | Load and draw BDF bitmap fonts | Yes, if drawing BDF text directly | Unreleased | +| [`espn_dates`](#espn_dates) | Fetch ESPN scoreboards across a date range | Yes (scoreboards) | 3.5.0 | +| [`font_layout`](#font_layout) | Reproducible TrueType loading, crisp sizes | Yes | 3.4.0 | +| [`logo_helper`](#logo_helper) | Load, resize and cache team logos | Yes | — | +| [`path_safety`](#path_safety) | Turn request-supplied names into safe paths | No, core-internal | n/a | +| [`permission_utils`](#permission_utils) | File modes and shared-group ownership | Rarely | — | +| [`scroll_config`](#scroll_config) | Plugin scroll config → configured `ScrollHelper` | Yes (scrollers) | 3.4.0 | +| [`scroll_helper`](#scroll_helper) | Pre-rendered horizontal scrolling | Yes | — | +| [`snapshot_policy`](#snapshot_policy) | When to write the web preview frame | No, core-internal | n/a | +| [`sports_card`](#sports_card) | Scoreboard card settings, colours, fonts, dates | Yes (scoreboards) | 3.3.0 | +| [`sports_game_renderer`](#sports_game_renderer) | Scoreboard scroll/Vegas card geometry | Yes (scoreboards) | 3.3.0 | +| [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 | +| [`sports_scroll`](#sports_scroll) | Scoreboard scroll-display orchestration | Yes (scoreboards) | 3.2.0 | +| [`sports_shared`](#sports_shared) | Sport-independent `sports.py` methods | Yes (scoreboards) | 3.3.0 | +| [`sync_manager`](#sync_manager) | Leader/follower sync between two displays | No, core-internal | n/a | +| [`text_helper`](#text_helper) | Outlined text, wrapping, measurement | Yes | — | + +The four `sports_*` mixin and card modules hold code the scoreboard plugins +used to carry as identical copies. Each module docstring lists what a host +class must provide. The plan behind them is in +[docs/SPORTS_UNIFICATION.md](../../docs/SPORTS_UNIFICATION.md). + +## Adaptive layout and images + +`src/adaptive_layout.py` and `src/adaptive_images.py` live outside this +package but are re-exported from `src.common`. They are the recommended way +to lay out a plugin that renders legibly on any panel size. Every +`BasePlugin` already has `self.layout`, `self.draw_fit()` and +`self.draw_image()`: ```python -# Every BasePlugin already has self.layout and the draw helpers: regs = scoreboard_regions(self.layout.bounds, ctx=self.layout) self.draw_image(away_logo, regs.away_slot, mode="fill_height", crop_to_ink=True, cache_key=f"logo:{abbr}") self.draw_fit(score_text, regs.score_area) # largest crisp font that fits -self.draw_fit(status, regs.status_band) ``` -Key pieces: `Region` (rect algebra: bands/columns/splits/offset), -font ladders (`LADDER_GRID`, `LADDER_ARCADE` — discrete crisp sizes, never -fractional scaling), `LayoutContext` (`fit_text`, `fit_image`, `by_tier`, -`px`), and composite carvers `scoreboard_regions()` / `media_row()`. -Full guide: [docs/ADAPTIVE_LAYOUT.md](../../docs/ADAPTIVE_LAYOUT.md). +Key pieces: `Region`, the font ladders `LADDER_GRID` / `LADDER_ARCADE`, +`LayoutContext` (`fit_text`, `fit_image`, `by_tier`, `px`), and +`scoreboard_regions()` / `media_row()`. Guide: +[docs/ADAPTIVE_LAYOUT.md](../../docs/ADAPTIVE_LAYOUT.md). -## API Helpers (`api_helper.py`) +## Modules -Utilities for making HTTP requests and handling API responses. +### api_helper -## Logo Helpers (`logo_helper.py`) +[`api_helper.py`](api_helper.py). `APIHelper(cache_manager=None, ...)`: +`get()` and `post()` with retries, optional caching through the cache +manager, and a minimum interval between requests (`set_rate_limit()`). Also has +`fetch_espn_scoreboard()`, `fetch_espn_standings()` and +`fetch_espn_rankings()`. -Utilities for loading and managing team logos. +### bdf_font -## Text Helpers (`text_helper.py`) +[`bdf_font.py`](bdf_font.py). The one BDF loader and rasterizer. +`load_bdf_face(path, size)` returns `(face, realised_px)`, falling back to +the file's native strike when it has none at `size`; +`draw_bdf_text(draw, text, x, y, face, color)` draws top-left anchored onto a +PIL `ImageDraw` the same way the panel does. `read_bdf_native_size(path)` +and `clear_face_cache()` round it out. Faces are cached per thread (FreeType +faces are not thread-safe). `DisplayManager`, `FontManager`, `element_style` +and the plugin test harness all use it. Most plugins get BDF text through +`display_manager.draw_text()` or `FontManager` and never import this. -Utilities for text processing and formatting. +### espn_dates -## BDF Fonts (`bdf_font.py`) +[`espn_dates.py`](espn_dates.py). ESPN's site API rejects `dates=` ranges +and truncates results when `limit` is above 500. `fetch_espn_scoreboard()` +splits a range into month and day requests ESPN accepts and merges the +results; `espn_date_chunks()`, `fetch_espn_date_chunks()`, +`clamp_espn_limit()` and `merge_scoreboard_payloads()` are the pieces. +Scoreboard plugins also bundle a copy for older cores. -The one way to load and draw BDF bitmap fonts. `load_bdf_face(path, size)` -returns `(face, realised_px)`, falling back to the file's native strike when -it has none at `size`; `draw_bdf_text(draw, text, x, y, face, color)` draws -top-left anchored onto a PIL `ImageDraw` exactly as the panel does. -`DisplayManager`, `FontManager`, `element_style` and the plugin test harness -all go through it. +### font_layout -## Scroll Helpers (`scroll_helper.py`) +[`font_layout.py`](font_layout.py). `load_truetype(path, size)` is +`ImageFont.truetype` with PIL's Basic layout engine pinned, so text lays out +the same whether or not the host Pillow has libraqm; use it for anything +drawn to the panel or compared against a golden image. `crisp_size()` gives +the size a bundled face renders on whole pixels at. `resolve_asset_path()` +resolves `assets/fonts/...` against the install root rather than the +working directory. -Utilities for scrolling text on the display. +### logo_helper -## Permission Utilities (`permission_utils.py`) +[`logo_helper.py`](logo_helper.py). `LogoHelper(display_width, +display_height, ...)`: `load_logo()`, `load_logo_with_download()`, +`get_logo_variations()`, `normalize_abbreviation()`, with an in-memory cache. -Helpers for ensuring directory permissions and ownership are correct -when running as a service (used by `CacheManager` to set up its -persistent cache directory). +### path_safety -## Best Practices +[`path_safety.py`](path_safety.py). Core-internal, used by web handlers that +open files named in a request. `safe_path_component(value)` returns the +value if it is one harmless path segment, else `None`; +`resolve_under(base, *parts)` returns the resolved path, or `None` if a part +is unsafe or the result would leave `base`; `safe_relative_parts()` splits a +relative path the same way. Both return the sanitised value rather than a +boolean, so a caller cannot check one string and open another. -1. **Use centralized logging**: Import from `src.logging_config` instead of creating loggers directly -2. **Reuse utilities**: Check existing utilities before creating new ones -3. **Document additions**: Add documentation when adding new utilities +### permission_utils + +[`permission_utils.py`](permission_utils.py). The modes and ownership that +let the root display service and the web user share files: +`ensure_directory_permissions()`, `ensure_file_permissions()`, the +`get_*_mode()` functions, `ensure_shared_group_ownership()`, +`sudo_remove_directory()` and `install_requirements_file()` (the sudo +`safe_pip_install.sh` path). `ConfigManager`, `CacheManager` and the store +already call these; a plugin needs them only when it creates its own files +outside the cache. See [docs/PERMISSIONS.md](../../docs/PERMISSIONS.md). + +### scroll_config + +[`scroll_config.py`](scroll_config.py). `configure(scroll_helper, +plugin_config=, global_config=, display_manager=, plugin_logger=)` reads a +plugin's scroll settings, snaps the speed to a whole number of pixels per +panel refresh, puts the helper in fixed-step mode and returns +`ScrollSettings`. Pass `settings.frame_hold` to +`display_manager.set_scrolling_state(True, frame_hold=...)` or the scroll +runs too fast. `resolve()` does the calculation without touching a helper. +See [docs/SCROLL_PERFORMANCE.md](../../docs/SCROLL_PERFORMANCE.md). + +### scroll_helper + +[`scroll_helper.py`](scroll_helper.py). `ScrollHelper(display_width, +display_height, logger=None)`: build a wide image once +(`create_scrolling_image()` or `set_scrolling_image()`), then per frame +`update_scroll_position()` and `get_visible_portion()`; +`is_scroll_complete()`, `calculate_dynamic_duration()` and +`get_dynamic_duration()` for timing. Configure it with `scroll_config` +rather than the `set_*` methods. Vegas mode reads a plugin's +`scroll_helper` image when the plugin has no `get_vegas_content()`. + +### snapshot_policy + +[`snapshot_policy.py`](snapshot_policy.py). Core-internal. `decide()` +tells `DisplayManager` whether to write `/tmp/led_matrix_preview.png`, only +touch its mtime, or skip, based on whether a browser is watching the preview. +The web health check reads the file's age. + +### sports_card + +[`sports_card.py`](sports_card.py). Free functions taking `config`, `fonts` +and `logger` explicitly: card options (`scroll_card_option()`, +`vs_text()`, `upcoming_center_mode()`), colours (`element_color()`, +`font_color()`, `score_color_for()`, `recent_score_color()`), favourite-team +rules (`favorite_teams_for()`, `side_is_favorite()`, `favorite_result()`), +dates (`format_game_date()`, `format_game_time()`, `card_tzinfo()`) and font +sizes (`schema_font_size()`, `resolve_font_size()`). A plugin keeps its own +method and delegates the body. + +### sports_game_renderer + +[`sports_game_renderer.py`](sports_game_renderer.py). +`SportsGameRendererMixin`: the scroll/Vegas card geometry (centre gap, logo +slot, layout offsets, upcoming-card date and time). No `__init__` and no +state; add it as a base class of the plugin's game renderer and override +what differs. + +### sports_helpers + +[`sports_helpers.py`](sports_helpers.py). Free functions `clamp_window()`, +`clamp_seconds()`, `logo_needs_refresh()`, `spread_weighted_order()`, and +`SportsHelpersMixin` with the scoreboards' `_mode_customization`, +`_setting_int`, `_reset_dwell_on_reentry`, `_next_switch_index`, +`_odds_color` and `_upcoming_date_and_time_text` under their existing names. +Nothing in core uses it. + +### sports_scroll + +[`sports_scroll.py`](sports_scroll.py). `SportsScrollDisplay` and +`SportsScrollDisplayManager`: the scroll-display orchestration the +scoreboards share (Vegas items, dynamic duration, frame loop), paced through +`scroll_config`. Subclasses supply `prepare_scroll_content()` and set +`SCROLL_LEAGUE_KEYS`; see the module docstring for an example. + +### sports_shared + +[`sports_shared.py`](sports_shared.py). `SportsCoreSharedMixin`, +`SportsLiveSharedMixin`, `SportsRecentSharedMixin`: the `sports.py` methods +that were identical in every scoreboard (game selection and rotation, +fonts, colours, dates, the switch-mode upcoming card). The docstring lists +the attributes the host class must have and the three methods deliberately +left out. + +### sync_manager + +[`sync_manager.py`](sync_manager.py). Core-internal. `DisplaySyncManager` +links two displays as leader and follower (`sync.role` in config) over UDP +port 5765, plus TCP on the next port for scroll images. The leader drives the +scroll and sends the follower its part of each frame; a follower falls back +to its own plugins when the leader goes quiet. Rows and columns must match. +Created by `DisplayController`; works with any plugin. + +### text_helper + +[`text_helper.py`](text_helper.py). `TextHelper(font_dir=None, ...)`: +`load_fonts()`, `draw_text_with_outline()`, `get_text_width()`, +`get_text_dimensions()`, `center_text()`, `wrap_text()`, +`draw_multiline_text()`, `create_text_image()`. + +## Logging + +Modules here create their logger with `logging.getLogger(__name__)`, which is +the same logger `src.logging_config.get_logger(__name__)` returns. The helper +classes (`APIHelper`, `LogoHelper`, `ScrollHelper`, `TextHelper`) and +`espn_dates` take an optional `logger`. In a plugin, pass `self.logger`: it is +created by `get_logger(..., plugin_id=...)` in `BasePlugin`, so messages carry +the plugin id. + +## Adding a module + +- Keep it importable without hardware (see the test above). +- Give it a module docstring that says what it is for and, if it is a mixin, + what the host class must provide. +- Add it to the table on this page and, if plugins may import it, to the + CHANGELOG with the release to floor on. diff --git a/start_display.sh b/start_display.sh index df9fcb92..cbe4ca4a 100755 --- a/start_display.sh +++ b/start_display.sh @@ -1,9 +1,6 @@ #!/bin/bash -# Get the current user -CURRENT_USER=$(whoami) - -echo "Starting LED Matrix Display Service for user: $CURRENT_USER..." +echo "Starting LED Matrix Display Service..." # Start the service sudo systemctl start ledmatrix.service diff --git a/stop_display.sh b/stop_display.sh index da37b27c..533463c6 100755 --- a/stop_display.sh +++ b/stop_display.sh @@ -1,9 +1,6 @@ #!/bin/bash -# Get the current user -CURRENT_USER=$(whoami) - -echo "Stopping LED Matrix Display Service for user: $CURRENT_USER..." +echo "Stopping LED Matrix Display Service..." # Stop the service sudo systemctl stop ledmatrix.service diff --git a/test/test_fix_web_permissions_rehardens.py b/test/test_fix_web_permissions_rehardens.py new file mode 100644 index 00000000..fc0855c6 --- /dev/null +++ b/test/test_fix_web_permissions_rehardens.py @@ -0,0 +1,109 @@ +"""scripts/fix_perms/fix_web_permissions.sh must not undo the installer's hardening. + +The script chowns the whole project to the web user. That used to include the +two helpers /etc/sudoers.d/ledmatrix_web lets the web user run as root +(safe_plugin_rm.sh, safe_pip_install.sh) -- a helper the web user owns is a +root shell for anyone who can edit it -- and config_secrets.json, which lost +the ledmatrix group first_time_install.sh gives it. After the chown the script +now puts both back the way the installer's Steps 11 and 11.1 leave them. + +The behavioural test runs the real script against a scratch copy of the +project with `sudo`, `getent` and `journalctl` stubbed, and checks the order +of what it asked sudo to do. +""" +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parent.parent +SCRIPT = ROOT / "scripts" / "fix_perms" / "fix_web_permissions.sh" +LIB = ROOT / "scripts" / "install" / "lib_sudoers.sh" + + +def _text(path): + return path.read_text(encoding="utf-8", errors="replace").replace("\r\n", "\n") + + +def _granted_helpers(): + helpers = set(re.findall(r"scripts/fix_perms/([\w.-]+\.sh) \*", _text(LIB))) + assert helpers, "no fix_perms helper grant found in lib_sudoers.sh" + return helpers + + +def test_every_granted_helper_is_rehardened_after_the_chown(): + text = _text(SCRIPT) + chown = text.index('sudo chown -R "$WEB_USER:$WEB_USER" "$PROJECT_DIR"') + loop = re.search(r"for helper in ([^;]+); do\n(.*?)\ndone", text, re.S) + assert loop, "no helper-hardening loop in fix_web_permissions.sh" + assert "sudo chown root:root" in loop.group(2) and "sudo chmod 755" in loop.group(2) + assert loop.start() > chown, "helpers are hardened before the chown that undoes it" + assert _granted_helpers() <= set(loop.group(1).split()) + + +def test_no_longer_claims_to_configure_sudoers(): + text = _text(SCRIPT) + assert "Configure sudoers for passwordless access" not in text + assert "./configure_web_sudo.sh" not in text.replace("scripts/install/configure_web_sudo.sh", "") + + +_STUB_SUDO = """#!/bin/bash +printf '%s\\n' "$*" >> "$SUDO_LOG" +# `sudo -n ...` probes and `sudo -u ...` tests: report failure, run nothing. +case "$1" in -n|-u) exit 1 ;; esac +exit 0 +""" + + +@pytest.mark.skipif(sys.platform == "win32" or shutil.which("bash") is None, + reason="needs a POSIX bash") +def test_script_rehardens_helpers_and_secrets(tmp_path): + project = tmp_path / "LED Matrix" + (project / "scripts" / "fix_perms").mkdir(parents=True) + (project / "config").mkdir() + script = project / "scripts" / "fix_perms" / "fix_web_permissions.sh" + script.write_text(_text(SCRIPT), encoding="utf-8") + for helper in ("safe_plugin_rm.sh", "safe_pip_install.sh"): + (project / "scripts" / "fix_perms" / helper).write_text("#!/bin/bash\n") + (project / "config" / "config_secrets.json").write_text("{}\n") + + stubs = tmp_path / "stubs" + stubs.mkdir() + for name, body in (("sudo", _STUB_SUDO), + ("getent", "#!/bin/sh\nexit 0\n"), + ("journalctl", "#!/bin/sh\nexit 1\n")): + (stubs / name).write_text(body) + (stubs / name).chmod(0o755) + log = tmp_path / "sudo.log" + env = dict(os.environ, SUDO_LOG=str(log), + PATH=os.pathsep.join([str(stubs), os.environ.get("PATH", "")])) + + result = subprocess.run(["bash", str(script)], input="y", env=env, + capture_output=True, text=True) + if os.geteuid() == 0: + # The script refuses to run as root; that refusal is the whole test. + assert result.returncode == 1 and "should not be run as root" in result.stdout + return + assert result.returncode == 0, result.stdout + result.stderr + + calls = log.read_text().splitlines() + user = subprocess.run(["whoami"], capture_output=True, text=True).stdout.strip() + chown_all = calls.index(f"chown -R {user}:{user} {project}") + for helper in ("safe_plugin_rm.sh", "safe_pip_install.sh"): + path = project / "scripts" / "fix_perms" / helper + assert calls.index(f"chown root:root {path}") > chown_all, calls + assert calls.index(f"chmod 755 {path}") > chown_all, calls + secrets = project / "config" / "config_secrets.json" + # The owner is the installed web unit's User= when there is one. + owner = user + unit = Path("/etc/systemd/system/ledmatrix-web.service") + if unit.is_file(): + m = re.search(r"^User=(.*)$", unit.read_text(), re.M) + if m and m.group(1): + owner = m.group(1) + assert calls.index(f"chown {owner}:ledmatrix {secrets}") > chown_all, calls + assert calls.index(f"chmod 640 {secrets}") > chown_all, calls diff --git a/test/test_install_reboot_is_last.py b/test/test_install_reboot_is_last.py new file mode 100644 index 00000000..3ae62832 --- /dev/null +++ b/test/test_install_reboot_is_last.py @@ -0,0 +1,134 @@ +"""first_time_install.sh prints its completion summary before it reboots. + +With -y (and so with the one-shot `curl | bash` installer, which always +passes -y) the reboot used to be issued ~180 lines before the "Installation +Complete / Web UI Access" summary. `reboot` returns at once and the script +carried on printing while the system went down, so the SSH session usually +dropped before the user saw the web UI address. + +first_time_install.sh exits on anything but Raspberry Pi OS Trixie before it +parses its arguments, so the behavioural test runs only the tail of the +script -- from the summary to the end -- with systemctl, nmcli, hostname, ip +and reboot stubbed. +""" +import os +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parent.parent +FIRST_TIME = ROOT / "first_time_install.sh" +SUMMARY_START = 'echo "Installation Complete!"' + + +def _text(): + return FIRST_TIME.read_text(encoding="utf-8").replace("\r\n", "\n") + + +def test_every_reboot_comes_after_the_summary(): + text = _text() + summary = text.index(SUMMARY_START) + lines = text.splitlines() + reboots = [i for i, line in enumerate(lines) if line.strip() == "reboot"] + assert reboots, "no reboot call found" + summary_line = text[:summary].count("\n") + enjoy_line = text[:text.index('echo "Enjoy your LED Matrix display!"')].count("\n") + assert all(i > enjoy_line > summary_line for i in reboots), ( + f"reboot at line(s) {[i + 1 for i in reboots]} runs before the summary " + f"(line {summary_line + 1}) has finished printing") + + +def _tail(): + """The script from the summary header to the end, header rule included.""" + text = _text() + start = text.rindex('echo "=========================================="', 0, + text.index(SUMMARY_START)) + return text[start:] + + +_POSIX = pytest.mark.skipif(sys.platform == "win32" or shutil.which("bash") is None, + reason="needs a POSIX bash") + + +def _run(tmp_path, env_extra, nmcli_active_line=True, stdin="", hostapd_active=False): + stubs = tmp_path / "stubs" + stubs.mkdir() + log = tmp_path / "calls.log" + active = 'echo "yes:HomeNet"' if nmcli_active_line else ":" + hostapd = 'case "$*" in *"is-active --quiet hostapd"*) exit 0 ;; esac\n' if hostapd_active else "" + bodies = { + "reboot": f'#!/bin/sh\necho REBOOT-CALLED\necho reboot >> "{log}"\n', + "systemctl": f"#!/bin/sh\n{hostapd}exit 3\n", + "hostname": '#!/bin/sh\necho "192.168.1.50 fe80::1"\n', + "ip": "#!/bin/sh\nexit 1\n", + # device status -> one connected wifi device; device wifi -> active line + "nmcli": ('#!/bin/sh\ncase "$*" in\n' + ' *"device status"*) echo "wlan0:wifi:connected" ;;\n' + f' *"device wifi"*) {active} ;;\n' + "esac\n"), + } + for name, body in bodies.items(): + (stubs / name).write_text(body) + (stubs / name).chmod(0o755) + script = "\n".join([ + "set -Eeuo pipefail", + "on_error() { echo \"ERR-TRAP line $1\" >&2; exit 1; }", + "trap 'on_error $LINENO' ERR", + "PROJECT_ROOT_DIR=/home/pi/LEDMatrix", + "ASSUME_YES=${ASSUME_YES:-0}", + "SKIP_REBOOT_PROMPT=${SKIP_REBOOT_PROMPT:-0}", + _tail(), + ]) + env = dict(os.environ, PATH=os.pathsep.join([str(stubs), "/usr/bin", "/bin"]), **env_extra) + result = subprocess.run(["bash", "-c", script], env=env, input=stdin, + capture_output=True, text=True) + calls = log.read_text().splitlines() if log.exists() else [] + return result, calls + + +@_POSIX +@pytest.mark.parametrize("nmcli_active_line", [True, False], ids=["ssid", "no-ssid"]) +def test_assume_yes_prints_the_summary_then_reboots(tmp_path, nmcli_active_line): + result, calls = _run(tmp_path, {"ASSUME_YES": "1"}, nmcli_active_line) + out = result.stdout + assert result.returncode == 0, out + result.stderr + assert calls == ["reboot"] + for text in ("Installation Complete!", "Web UI Access:", "http://192.168.1.50:5000", + "Enjoy your LED Matrix display!"): + assert out.index(text) < out.index("REBOOT-CALLED"), text + assert "Password: ledmatrix123" not in out + + +@_POSIX +def test_setup_access_point_is_described_as_open(tmp_path): + """wifi_manager creates the setup AP with no security ("No password" on + the panel); the summary used to print a password it does not have.""" + result, _ = _run(tmp_path, {"ASSUME_YES": "1"}, hostapd_active=True) + assert result.returncode == 0, result.stdout + result.stderr + assert "AP Mode is ACTIVE" in result.stdout + assert "Open network, no password" in result.stdout + assert "Password:" not in result.stdout + + +@_POSIX +def test_no_reboot_prompt_prints_the_summary_and_does_not_reboot(tmp_path): + result, calls = _run(tmp_path, {"ASSUME_YES": "1", "SKIP_REBOOT_PROMPT": "1"}) + assert result.returncode == 0, result.stdout + result.stderr + assert calls == [] + assert "Enjoy your LED Matrix display!" in result.stdout + assert "Skipping reboot prompt" in result.stdout + + +@_POSIX +@pytest.mark.parametrize("answer,expected", [("y", ["reboot"]), ("n", [])]) +def test_interactive_prompt_comes_after_the_summary(tmp_path, answer, expected): + result, calls = _run(tmp_path, {}, stdin=answer) + assert result.returncode == 0, result.stdout + result.stderr + assert calls == expected + out = result.stdout + assert "Enjoy your LED Matrix display!" in out + if expected: + assert out.index("Enjoy your LED Matrix display!") < out.index("REBOOT-CALLED") diff --git a/test/test_sudo_allowlist_covers_calls.py b/test/test_sudo_allowlist_covers_calls.py index 8f3b88f7..eb1004fd 100644 --- a/test/test_sudo_allowlist_covers_calls.py +++ b/test/test_sudo_allowlist_covers_calls.py @@ -11,12 +11,16 @@ Four such calls were ungranted, all of them captive-portal teardown/setup: rfkill unblock wifi wifi_manager.py:1811 mkdir -p .../dnsmasq-shared.d wifi_manager.py:922 +The drop-in written into that directory was missing too: the literal +`cp /tmp/ledmatrix-nm-dnsmasq.conf .../dnsmasq-shared.d/ledmatrix-captive.conf` +and `rm -f` of the same file, so the directory was granted but not the file. + It goes unnoticed because a stock Raspberry Pi image ships /etc/sudoers.d/010_pi-nopasswd granting the default user `ALL=(ALL) NOPASSWD: ALL`, which satisfies every gap in both files. It only bites once that blanket rule is removed or the service runs as another user. -Scope, deliberately narrow: this pins the four commands above, each of which +Scope, deliberately narrow: this pins the commands above, each of which can be written out literally. The portal makes further sudo calls whose arguments are built at runtime -- iptables and nft rules carrying an interface name and a port, `ip addr`, `ip link` -- and those cannot be granted safely @@ -50,6 +54,11 @@ REQUIRED = ( ("nft", "delete", "table", "ip", "ledmatrix"), ("rfkill", "unblock", "wifi"), ("mkdir", "-p", "/etc/NetworkManager/dnsmasq-shared.d"), + # The drop-in that directory exists for, written and removed by + # _write_nm_dnsmasq_captive_conf / _remove_nm_dnsmasq_captive_conf. + ("cp", "/tmp/ledmatrix-nm-dnsmasq.conf", + "/etc/NetworkManager/dnsmasq-shared.d/ledmatrix-captive.conf"), + ("rm", "-f", "/etc/NetworkManager/dnsmasq-shared.d/ledmatrix-captive.conf"), ) #: Tools with an option that executes a program of the caller's choosing. diff --git a/test/test_sudoers_is_validated.py b/test/test_sudoers_is_validated.py index a65bbd0e..5e6aeafb 100644 --- a/test/test_sudoers_is_validated.py +++ b/test/test_sudoers_is_validated.py @@ -19,6 +19,7 @@ import pytest REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) FIRST_TIME = os.path.join(REPO_ROOT, "first_time_install.sh") CONFIGURE = os.path.join(REPO_ROOT, "scripts", "install", "configure_web_sudo.sh") +WIFI = os.path.join(REPO_ROOT, "scripts", "install", "configure_wifi_permissions.sh") VISUDO = shutil.which("visudo") or ( "/usr/sbin/visudo" if os.path.exists("/usr/sbin/visudo") else None @@ -62,6 +63,54 @@ def test_configure_web_sudo_validates_before_installing(): assert validate < install, "the rules must be checked before they are installed" +def test_configure_web_sudo_does_not_use_a_predictable_temp_file(): + body = _read(CONFIGURE) + assert 'TEMP_SUDOERS=$(mktemp' in body + assert "/tmp/ledmatrix_web_sudoers_$$" not in body + assert "trap 'rm -f \"$TEMP_SUDOERS\"' EXIT" in body + + +def test_configure_web_sudo_installs_mode_440(): + body = _read(CONFIGURE) + install = body.index('cp "$TEMP_SUDOERS" /etc/sudoers.d/ledmatrix_web') + assert body.index("chmod 440 /etc/sudoers.d/ledmatrix_web") > install + + +def test_configure_wifi_permissions_validates_before_installing(): + """The third sudoers writer. It installed its rules unchecked.""" + body = _read(WIFI) + # The check itself, as a condition -- not merely the command appearing in + # the error report that follows it. + validate = body.index('if ! visudo -c -f "$TEMP_SUDOERS"') + install = body.index('sudo cp "$TEMP_SUDOERS" "$SUDOERS_FILE"') + assert validate < install, "the rules must be checked before they are installed" + # ...and a failed check stops the script before the copy. + assert "exit 1" in body[validate:install] + assert "TEMP_SUDOERS=$(mktemp" in body + + +@pytest.mark.skipif(sys.platform == "win32", reason="visudo is POSIX only") +@pytest.mark.skipif(VISUDO is None, reason="visudo not installed") +def test_the_wifi_rules_actually_parse(tmp_path): + """Render configure_wifi_permissions.sh's heredoc with realistic paths.""" + body = _read(WIFI) + opener = 'cat > "$TEMP_SUDOERS" << EOF\n' + start = body.index(opener) + len(opener) + end = body.index("\nEOF\n", start) + out = tmp_path / "wifi" + script = "\n".join([ + "WEB_USER=ledmatrix", "NMCLI_PATH=/usr/bin/nmcli", + "SYSTEMCTL_PATH=/usr/bin/systemctl", "SYSCTL_PATH=/usr/sbin/sysctl", + "NFT_PATH=/usr/sbin/nft", "RFKILL_PATH=/usr/sbin/rfkill", + "MKDIR_PATH=/usr/bin/mkdir", + f"cat > '{out}' << EOF", body[start:end], "EOF", + ]) + subprocess.run(["bash", "-c", script], check=True) + os.chmod(out, 0o440) + result = subprocess.run([VISUDO, "-c", "-f", str(out)], capture_output=True, text=True) + assert result.returncode == 0, result.stdout + result.stderr + + def test_a_missing_rules_library_installs_nothing(): """If lib_sudoers.sh is missing, nothing is generated -- and an empty file would pass `visudo -c` -- so that branch must set the flag the install is diff --git a/test/test_systemd_unit_drift.py b/test/test_systemd_unit_drift.py index 8afaadb0..9673beff 100644 --- a/test/test_systemd_unit_drift.py +++ b/test/test_systemd_unit_drift.py @@ -17,6 +17,7 @@ editing files under /etc and restarting services is the installer's job, not something a display process should do to a machine while it boots. """ import logging +import re import shlex import subprocess from pathlib import Path @@ -253,6 +254,24 @@ def test_sed_escape_replacement_preserves_special_characters(): "a sed-special character in the replacement was not preserved literally") +def test_every_unit_renderer_escapes_its_replacement(): + """Each `sed s|__PLACEHOLDER__|$VALUE|` in an install script uses an escaped value. + + install_dns_fix.sh and install_mqtt_bridge.sh interpolated the raw project + path while the other three renderers went through sed_escape_replacement, + so a checkout under a path containing `&` rendered a broken unit from + those two only. + """ + project_root = Path("src/startup_validator.py").resolve().parent.parent + offenders = [] + for script in sorted((project_root / "scripts" / "install").glob("*.sh")): + text = script.read_text(encoding="utf-8") + for m in re.finditer(r"s\|__[A-Z_]+__\|\$\{?([A-Za-z_][A-Za-z0-9_]*)\}?\|", text): + if not m.group(1).startswith("ESCAPED_") and m.group(1) != "root": + offenders.append(f"{script.name}: ${m.group(1)}") + assert not offenders, "unescaped sed replacement(s): " + ", ".join(offenders) + + def test_no_installer_carries_its_own_copy_of_a_unit(): """The regression guard. diff --git a/web_interface/static/v3/js/widgets/README.md b/web_interface/static/v3/js/widgets/README.md index be1f2189..36378fb5 100644 --- a/web_interface/static/v3/js/widgets/README.md +++ b/web_interface/static/v3/js/widgets/README.md @@ -1,20 +1,63 @@ # LEDMatrix Widget Development Guide -## Overview +Widgets are the controls the web UI draws for fields in a plugin's +`config_schema.json`. A field picks one with `"x-widget": ""`. This +directory holds the built-in widgets, the registry they register with, and +the loader for widgets a plugin ships itself. -The LEDMatrix Widget Registry system allows plugins to use reusable UI components (widgets) for configuration forms. This system enables: +The page loads every file here as one bundle, `/assets/widgets.js`, built by +[`web_interface/widget_bundle.py`](../../../../widget_bundle.py) in the order +set by `BUNDLE_ORDER` there. -- **Reusable Components**: Use existing widgets (file upload, checkboxes, etc.) without custom code -- **Custom Widgets**: Create plugin-specific widgets without modifying the LEDMatrix codebase -- **Backwards Compatibility**: Existing plugins continue to work without changes +## Built-in widgets -## Available Core Widgets +| `x-widget` | Field type | What it draws | +|---|---|---| +| `text-input` | string | Text field with optional length limits | +| `textarea` | string | Multi-line text | +| `email-input` | string | Email field with format check | +| `url-input` | string | URL field with format check | +| `password-input` | string | Password field with show/hide toggle | +| `select-dropdown` | string | Dropdown for an `enum` | +| `radio-group` | string | Radio buttons for an `enum` | +| `date-picker` | string | Date input | +| `time-picker` | string | Time input, `HH:MM` (24-hour) | +| `color-picker` | string or array | Colour picker; hex string, or `[r, g, b]` on an array field | +| `font-selector` | string | Font from `assets/fonts/` (TTF and BDF), fetched from the API | +| `timezone-selector` | string | IANA timezone, grouped by region | +| `file-upload-single` | string | One image upload; stores the uploaded file's relative path | +| `google-oauth` | string | Step 2 of the calendar plugin's Google sign-in | +| `plugin-file-manager` | null | Inline file manager driven by the plugin's `web_ui_actions` | +| `json-file-manager` | null | JSON data-file manager driven by `web_ui_actions` | +| `toggle-switch` | boolean | On/off switch | +| `slider` | integer / number | Range slider using `minimum` / `maximum` | +| `number-input` | integer / number | Number field with min/max check | +| `file-upload` | array | Multi-image upload with preview, delete and scheduling | +| `checkbox-group` | array | Checkboxes for an array of `enum` items | +| `day-selector` | array | Days of the week | +| `custom-feeds` | array | RSS feed table with per-feed logo upload | +| `array-table` | array | Table editor for an array of objects | +| `google-calendar-picker` | array | Calendars from the user's Google account | +| `schedule-picker` | object | Enable toggle, global/per-day mode and times | +| `time-range` | object | Start and end time pair | +| `style-editor` | object | One row per display element: font, size, colour, alignment, offsets | -### 1. File Upload Widget (`file-upload`) +Other files here: -Upload and manage image files with drag-and-drop support, preview, delete, and scheduling. +| File | Purpose | +|---|---| +| `registry.js` | `window.LEDMatrixWidgets`: `register()`, `get()` | +| `base-widget.js` | Shared helpers (`escapeHtml`, `sanitizeId`) other widgets use | +| `notification.js` | Toast notifications; owns `window.showNotification` | +| `plugin-order-list.js` | Drag-and-drop plugin order list used by the Display and Durations tabs (`window.PluginOrderList`) | +| `plugin-loader.js` | Loads a plugin-supplied widget on demand | +| `example-color-picker.js` | Example custom widget. Not bundled: it registers `color-picker` and would replace the real one | + +Each widget file's header comment gives its schema options. The sections +below cover the ones that need more than a line. + +### `file-upload` -**Schema Configuration:** ```json { "type": "array", @@ -28,45 +71,39 @@ Upload and manage image files with drag-and-drop support, preview, delete, and s } ``` -**Features:** -- Drag and drop file upload -- Image preview with thumbnails -- Delete functionality -- Schedule images to show at specific times -- Progress indicators during upload +### `file-upload-single` -### 2. Checkbox Group Widget (`checkbox-group`) +Uploads one image to the plugin's asset folder +(`assets/plugins//uploads/`) and stores the returned relative path +in a string field. `plugin_id` is filled in from the page; don't put it in +the schema. Use it for per-row images inside an `array-table`. -Multi-select checkboxes for array fields with enum items. - -**Schema Configuration:** ```json { - "type": "array", - "x-widget": "checkbox-group", - "items": { + "image_path": { "type": "string", - "enum": ["option1", "option2", "option3"] - }, - "x-options": { - "labels": { - "option1": "Option 1 Label", - "option2": "Option 2 Label" + "x-widget": "file-upload-single", + "x-upload-config": { + "allowed_types": ["image/png", "image/jpeg", "image/bmp", "image/gif"], + "max_size_mb": 5 } } } ``` -**Features:** -- Multiple selection from enum list -- Custom labels for each option -- Automatic JSON array serialization +### `checkbox-group` -### 3. Custom Feeds Widget (`custom-feeds`) +```json +{ + "type": "array", + "x-widget": "checkbox-group", + "items": {"type": "string", "enum": ["option1", "option2", "option3"]}, + "x-options": {"labels": {"option1": "Option 1 Label", "option2": "Option 2 Label"}} +} +``` -Table-based RSS feed editor with logo uploads. +### `custom-feeds` -**Schema Configuration:** ```json { "type": "array", @@ -74,539 +111,260 @@ Table-based RSS feed editor with logo uploads. "items": { "type": "object", "properties": { - "name": { "type": "string" }, - "url": { "type": "string", "format": "uri" }, - "enabled": { "type": "boolean" }, - "logo": { "type": "object" } + "name": {"type": "string"}, + "url": {"type": "string", "format": "uri"}, + "enabled": {"type": "boolean"}, + "logo": {"type": "object"} } }, "maxItems": 50 } ``` -**Features:** -- Add/remove feed rows -- Logo upload per feed -- Enable/disable individual feeds -- Automatic row re-indexing +### `plugin-file-manager` -### Other Built-in Widgets - -In addition to the three documented above, these widgets are -registered and ready to use via `x-widget`: - -**Inputs:** -- `text-input` — Plain text field with optional length constraints -- `textarea` — Multi-line text input -- `number-input` — Numeric input with min/max validation -- `email-input` — Email field with format validation -- `url-input` — URL field with format validation -- `password-input` — Password field with show/hide toggle - -**Selectors:** -- `select-dropdown` — Single-select dropdown for `enum` fields -- `radio-group` — Radio buttons for `enum` fields (alternative to dropdown) -- `toggle-switch` — Boolean toggle (alternative to a checkbox) -- `slider` — Numeric range slider for `integer`/`number` with `min`/`max` -- `color-picker` — RGB color picker; outputs `[r, g, b]` arrays -- `font-selector` — Picks from fonts in `assets/fonts/` (TTF + BDF) -- `timezone-selector` — IANA timezone picker - -**Date / time / scheduling:** -- `date-picker` — Single date input -- `day-selector` — Days-of-week multi-select (Mon–Sun checkboxes) -- `time-range` — Start/end time pair (e.g. for dim schedules) -- `schedule-picker` — Full cron-style or weekday/time schedule editor - -**Composite / data-source:** -- `array-table` — Generic table editor for arrays of objects -- `google-calendar-picker` — Picks from the user's authenticated Google - Calendars (used by the calendar plugin) - -**Internal (typically not used directly by plugins):** -- `notification` — Toast notification helper -- `base-widget` — Base class other widgets extend - -The canonical source for each widget's exact schema and options is the -file in this directory (e.g., `slider.js`, `color-picker.js`). If you -need a feature one of these doesn't support, see "Creating Custom -Widgets" below. - -## Using Existing Widgets - -To use an existing widget in your plugin's `config_schema.json`, simply add the `x-widget` property to your field definition: +A card grid, upload zone, create/delete dialogs and a table editor for the +plugin's data files, rendered inline. File operations call +`/api/v3/plugins/action` as soon as the user acts; they are not part of +**Save Configuration**. `plugin_id` is filled in from the page. ```json { - "properties": { - "my_images": { - "type": "array", - "x-widget": "file-upload", - "x-upload-config": { - "plugin_id": "my-plugin", - "max_files": 5 - } + "file_manager": { + "type": "null", + "title": "Data Files", + "x-widget": "plugin-file-manager", + "x-widget-config": { + "actions": { + "list": "list-files", + "get": "get-file", + "save": "save-file", + "upload": "upload-file", + "delete": "delete-file", + "create": "create-file", + "toggle": "toggle-category" + }, + "upload_hint": "JSON files with day numbers 1–365 as keys", + "directory_label": "my_data/", + "create_fields": [ + {"key": "category_name", "label": "Category Name", + "pattern": "^[a-z0-9_]+$", "hint": "Lowercase letters, numbers, underscores"}, + {"key": "display_name", "label": "Display Name", "hint": "Optional"} + ] } } } ``` -The widget will be automatically rendered when the plugin configuration form is loaded. +The action ids refer to entries in the plugin's `web_ui_actions` +([docs/PLUGIN_WEB_UI_ACTIONS.md](../../../../../docs/PLUGIN_WEB_UI_ACTIONS.md)). +`list` is required: without it the widget stays on its loading state. Leave +out any other action to hide its control. The editor shows a table when a +file is an object of objects with the same keys, otherwise a JSON text area. -## Creating Custom Widgets +## Schema keywords the form understands -### Step 1: Create Widget File +These work on any field, with or without a widget. -Create a JavaScript file in your plugin directory (e.g., `widgets/my-widget.js`): +### Option labels: `x-options.labels` + +A plain `enum` renders as a dropdown whose option text is the value with +underscores replaced and title case applied (`day_first` → "Day First"). +`x-options.labels` sets the visible text instead: + +```json +{ + "date_format": { + "type": "string", + "enum": ["abbrev", "numeric", "day_first"], + "default": "abbrev", + "x-options": {"labels": {"abbrev": "Sep 19", "numeric": "9/19", "day_first": "19 Sep"}} + } +} +``` + +Labels are display only; the stored value is still the enum value. The map +may be partial. Older cores ignore `x-options` and show the fallback text. +`array-table` columns accept the same `x-options.labels`, but their fallback +is the raw value (so a ticker symbol `aapl` stays `aapl`). + +### Advanced settings: `x-advanced` + +`"x-advanced": true` on a top-level, non-object property moves it into a +collapsed **Advanced Settings** section at the bottom of the plugin's page. +Use it for settings most users never change (timeouts, cache TTLs, styling +overrides); keep anything needed to get the plugin working in the main form. +The settings search still finds and expands advanced fields. It is ignored +on `object` properties and by older cores. + +### Hidden fields: `x-display: "hidden"` + +`"x-display": "hidden"` keeps a property in the schema without drawing a +control, for a deprecated key that existing configs still carry or an +internal value such as a generated row id. + +- Not rendered at any depth: top level, inside an object section, or as a + column or row-editor field of an array of objects. Hidden fields are left + out of Advanced Settings and the settings search. +- Saving the form never changes a hidden value. Array rows carry it through; + a new row gets no value. +- A JSON `POST /api/v3/plugins/config` can still set it. +- Older cores ignore the flag and render the field. + +## Creating a custom widget + +### 1. Write the widget + +Put it in your plugin's `widgets/` directory as `widgets/.js`. That +directory is the only place the core serves plugin widgets from. ```javascript -// Ensure LEDMatrixWidgets registry is available -if (typeof window.LEDMatrixWidgets === 'undefined') { - console.error('LEDMatrixWidgets registry not found'); - return; -} +(function () { + 'use strict'; + if (typeof window.LEDMatrixWidgets === 'undefined') { + console.error('LEDMatrixWidgets registry not found'); + return; + } -// Register your widget -window.LEDMatrixWidgets.register('my-custom-widget', { - name: 'My Custom Widget', - version: '1.0.0', - - /** - * Render the widget HTML - * @param {HTMLElement} container - Container element to render into - * @param {Object} config - Widget configuration from schema - * @param {*} value - Current value - * @param {Object} options - Additional options (fieldId, pluginId, etc.) - */ - render: function(container, config, value, options) { - const fieldId = options.fieldId || container.id; - // Sanitize fieldId for safe use in DOM IDs and selectors - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const safeFieldId = sanitizeId(fieldId); - - const html = ` -
- -
- `; - container.innerHTML = html; - - // Attach event listeners - const input = container.querySelector(`#${safeFieldId}_input`); - if (input) { - input.addEventListener('change', (e) => { - this.handlers.onChange(fieldId, e.target.value); - }); - } - }, - - /** - * Get current value from widget - * @param {string} fieldId - Field ID - * @returns {*} Current value - */ - getValue: function(fieldId) { - // Sanitize fieldId for safe selector use - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const safeFieldId = sanitizeId(fieldId); - const input = document.querySelector(`#${safeFieldId}_input`); - return input ? input.value : null; - }, - - /** - * Set value programmatically - * @param {string} fieldId - Field ID - * @param {*} value - Value to set - */ - setValue: function(fieldId, value) { - // Sanitize fieldId for safe selector use - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const safeFieldId = sanitizeId(fieldId); - const input = document.querySelector(`#${safeFieldId}_input`); - if (input) { - input.value = value || ''; - } - }, - - /** - * Event handlers - */ - handlers: { - onChange: function(fieldId, value) { - // Trigger form change event - const event = new CustomEvent('widget-change', { - detail: { fieldId, value }, - bubbles: true - }); - document.dispatchEvent(event); - } - }, - - /** - * Helper: Escape HTML to prevent XSS - */ - escapeHtml: function(text) { + const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); + const escapeHtml = (text) => { const div = document.createElement('div'); div.textContent = text; return div.innerHTML; - }, - - /** - * Helper: Sanitize identifier for use in DOM IDs and CSS selectors - */ - sanitizeId: function(id) { - return String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - } -}); + }; + + window.LEDMatrixWidgets.register('my-custom-widget', { + name: 'My Custom Widget', + version: '1.0.0', + + render: function (container, config, value, options) { + const fieldId = options.fieldId || container.id; + const safeId = sanitizeId(fieldId); + container.innerHTML = ` + `; + const input = container.querySelector(`#${safeId}_input`); + input.addEventListener('change', (e) => { + this.handlers.onChange(fieldId, e.target.value); + }); + }, + + getValue: function (fieldId) { + const input = document.querySelector(`#${sanitizeId(fieldId)}_input`); + return input ? input.value : null; + }, + + setValue: function (fieldId, value) { + const input = document.querySelector(`#${sanitizeId(fieldId)}_input`); + if (input) input.value = value || ''; + }, + + handlers: { + onChange: function (fieldId, value) { + document.dispatchEvent(new CustomEvent('widget-change', { + detail: { fieldId, value }, bubbles: true + })); + } + } + }); +})(); ``` -### Step 2: Reference Widget in Schema +[`example-color-picker.js`](example-color-picker.js) is a longer example. -In your plugin's `config_schema.json`: +### 2. Reference it in the schema ```json { "properties": { - "my_field": { - "type": "string", - "description": "My custom field", - "x-widget": "my-custom-widget", - "default": "" - } + "my_field": {"type": "string", "x-widget": "my-custom-widget", "default": ""} } } ``` -### Step 3: Declare the Widget in `manifest.json` +### 3. Declare it in `manifest.json` -The manifest is the allowlist -- a widget is served only if the plugin declares -it, so shipping a file under `widgets/` does not by itself publish it: +The manifest is the allowlist: a widget is served only if the plugin +declares it. ```json { "widgets": [ - { "name": "my-custom-widget", "script": "my-custom-widget.js" } + {"name": "my-custom-widget", "script": "my-custom-widget.js", + "description": "What this widget is for"} ] } ``` -`script` is optional and defaults to `[name].js`. It must be a plain filename -directly inside the plugin's `widgets/` directory. +`name` is what `x-widget` uses. `script` is optional, defaults to +`.js`, and must be a plain filename directly inside `widgets/`. Both +are validated against `schema/manifest_schema.json`. -### Step 4: Widget Loading +### 4. How it loads -The widget is loaded on demand when the config form renders a field that -references it. The system will: +When the config form reaches a field whose `x-widget` is not a built-in: -1. Check if the widget is registered in the core registry -2. If not, fetch `/static/plugin-widgets/[plugin-id]/[widget-name].js`, which - serves the declared script from the plugin's `widgets/` directory -3. Render the widget using the registered `render` function +1. If the name is already registered, that widget renders the field. +2. Otherwise the page fetches `/static/plugin-widgets//.js` + (`serve_plugin_widget` in + [`web_interface/blueprints/pages_v3.py`](../../../../blueprints/pages_v3.py)), + which serves the declared script from the plugin's `widgets/` directory. +3. The widget's `render()` draws the field. -The fetch is a dynamic `import()`, so the file must parse as an ES module (a -plain IIFE does). If anything fails, the field falls back to a plain text input +The fetch is a dynamic `import()`, so the file must parse as an ES module. +An IIFE does; modules are strict mode, and a `return` outside a function is a +syntax error. + +If the widget fails to load (not declared, file missing, script throws, or +it never calls `register`), the field falls back to a plain text input holding the current value, so a broken widget never costs the user their -configured value. +setting. -Only `string`-typed fields take this path today; see `docs/widget-guide.md` -for the full details and limitations. +**Limitation:** only `string` fields without an `enum` take this path. +[`plugin_config.html`](../../../../templates/v3/partials/plugin_config.html) +renders `object`, `array`, `boolean`, `integer`, `number` and `enum` fields +with its own branches, which only know the built-in names, so a plugin's own +widget on one of those is ignored. -## Widget API Reference - -### Widget Definition Object +## Widget API ```javascript { - name: string, // Human-readable widget name - version: string, // Widget version - render: function, // Required: Render function - getValue: function, // Optional: Get current value - setValue: function, // Optional: Set value programmatically - handlers: object // Optional: Event handlers + name: string, // human-readable name + version: string, + render: function, // required: render(container, config, value, options) + getValue: function, // optional: getValue(fieldId) -> value + setValue: function, // optional: setValue(fieldId, value) + handlers: object // optional: e.g. onChange(fieldId, value) } ``` -### Render Function +`render()` arguments: -```javascript -render(container, config, value, options) -``` +- `container` — element to render into +- `config` — the field's schema, including `x-widget-config` / `x-options` +- `value` — current value +- `options` — `fieldId`, `pluginId`, `fullKey` (dotted path of the field) -**Parameters:** -- `container` (HTMLElement): Container element to render into -- `config` (Object): Widget configuration from schema (`x-widget-config` or schema properties) -- `value` (*): Current field value -- `options` (Object): Additional options - - `fieldId` (string): Field ID - - `pluginId` (string): Plugin ID - - `fullKey` (string): Full field key path +## Guidelines -### Get Value Function - -```javascript -getValue(fieldId) -``` - -**Returns:** Current widget value - -### Set Value Function - -```javascript -setValue(fieldId, value) -``` - -**Parameters:** -- `fieldId` (string): Field ID -- `value` (*): Value to set - -### Event Handlers - -Widgets can define custom event handlers in the `handlers` object: - -```javascript -handlers: { - onChange: function(fieldId, value) { - // Handle value change - }, - onFocus: function(fieldId) { - // Handle focus - } -} -``` - -## Best Practices - -### Security - -1. **Always escape HTML**: Use `escapeHtml()` or `textContent` to prevent XSS -2. **Validate inputs**: Validate user input before processing -3. **Sanitize values**: Clean values before storing -4. **Sanitize identifiers**: Always sanitize identifiers (like `fieldId`) used as element IDs and in CSS selectors to prevent selector injection/XSS: - - Use `sanitizeId()` helper function (available in BaseWidget) or create your own - - Allow only safe characters: `[A-Za-z0-9_-]` - - Replace or remove invalid characters before using in: - - `getElementById()`, `querySelector()`, `querySelectorAll()` - - Setting `id` attributes - - Building CSS selectors - - Never interpolate raw `fieldId` into HTML strings or selectors without sanitization - - Example: `const safeId = fieldId.replace(/[^a-zA-Z0-9_-]/g, '_');` - -### Performance - -1. **Lazy loading**: Load widget scripts only when needed -2. **Event delegation**: Use event delegation for dynamic content -3. **Debounce**: Debounce frequent events (e.g., input changes) - -### Accessibility - -1. **Labels**: Always associate labels with inputs -2. **ARIA attributes**: Use appropriate ARIA attributes -3. **Keyboard navigation**: Ensure keyboard accessibility - -### Error Handling - -1. **Graceful degradation**: Handle missing dependencies -2. **User feedback**: Show clear error messages -3. **Logging**: Log errors for debugging - -## Examples - -### Example 1: Color Picker Widget - -```javascript -window.LEDMatrixWidgets.register('color-picker', { - name: 'Color Picker', - version: '1.0.0', - - render: function(container, config, value, options) { - const fieldId = options.fieldId; - // Sanitize fieldId for safe use in DOM IDs and selectors - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const sanitizedFieldId = sanitizeId(fieldId); - - container.innerHTML = ` -
- - -
- `; - - const colorInput = container.querySelector(`#${sanitizedFieldId}_color`); - const hexInput = container.querySelector(`#${sanitizedFieldId}_hex`); - - if (colorInput && hexInput) { - colorInput.addEventListener('change', (e) => { - hexInput.value = e.target.value; - this.handlers.onChange(fieldId, e.target.value); - }); - - hexInput.addEventListener('change', (e) => { - if (/^#[0-9A-Fa-f]{6}$/.test(e.target.value)) { - colorInput.value = e.target.value; - this.handlers.onChange(fieldId, e.target.value); - } - }); - } - }, - - getValue: function(fieldId) { - // Sanitize fieldId for safe selector use - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const sanitizedFieldId = sanitizeId(fieldId); - const colorInput = document.querySelector(`#${sanitizedFieldId}_color`); - return colorInput ? colorInput.value : null; - }, - - setValue: function(fieldId, value) { - // Sanitize fieldId for safe selector use - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const sanitizedFieldId = sanitizeId(fieldId); - const colorInput = document.querySelector(`#${sanitizedFieldId}_color`); - const hexInput = document.querySelector(`#${sanitizedFieldId}_hex`); - if (colorInput && hexInput) { - colorInput.value = value; - hexInput.value = value; - } - }, - - handlers: { - onChange: function(fieldId, value) { - const event = new CustomEvent('widget-change', { - detail: { fieldId, value }, - bubbles: true - }); - document.dispatchEvent(event); - } - } -}); -``` - -### Example 2: Slider Widget - -```javascript -window.LEDMatrixWidgets.register('slider', { - name: 'Slider Widget', - version: '1.0.0', - - render: function(container, config, value, options) { - const fieldId = options.fieldId; - // Sanitize fieldId for safe use in DOM IDs and selectors - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const sanitizedFieldId = sanitizeId(fieldId); - - const min = config.minimum || 0; - const max = config.maximum || 100; - const step = config.step || 1; - const currentValue = value !== undefined ? value : (config.default || min); - - container.innerHTML = ` -
- -
- ${min} - ${currentValue} - ${max} -
-
- `; - - const slider = container.querySelector(`#${sanitizedFieldId}_slider`); - const valueDisplay = container.querySelector(`#${sanitizedFieldId}_value`); - - if (slider && valueDisplay) { - slider.addEventListener('input', (e) => { - valueDisplay.textContent = e.target.value; - this.handlers.onChange(fieldId, parseFloat(e.target.value)); - }); - } - }, - - getValue: function(fieldId) { - // Sanitize fieldId for safe selector use - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const sanitizedFieldId = sanitizeId(fieldId); - const slider = document.querySelector(`#${sanitizedFieldId}_slider`); - return slider ? parseFloat(slider.value) : null; - }, - - setValue: function(fieldId, value) { - // Sanitize fieldId for safe selector use - const sanitizeId = (id) => String(id).replace(/[^a-zA-Z0-9_-]/g, '_'); - const sanitizedFieldId = sanitizeId(fieldId); - const slider = document.querySelector(`#${sanitizedFieldId}_slider`); - const valueDisplay = document.querySelector(`#${sanitizedFieldId}_value`); - if (slider) { - slider.value = value; - if (valueDisplay) { - valueDisplay.textContent = value; - } - } - }, - - handlers: { - onChange: function(fieldId, value) { - const event = new CustomEvent('widget-change', { - detail: { fieldId, value }, - bubbles: true - }); - document.dispatchEvent(event); - } - } -}); -``` +- Escape values before putting them in HTML (`textContent` or an + `escapeHtml` helper). +- Sanitise `fieldId` before using it in an `id`, `getElementById()` or a CSS + selector: allow only `[A-Za-z0-9_-]`. `BaseWidget` has `sanitizeId()`. +- Associate labels with inputs and keep the widget usable from the keyboard. +- Debounce events that fire on every keystroke. ## Troubleshooting -### Widget Not Loading +**Widget not loading** +- Check the browser console. +- The widget must be declared in `manifest.json` and live in `widgets/`. +- The name passed to `register()` must match `x-widget`. +- The field must be a non-enum `string` (see the limitation above). -1. Check browser console for errors -2. Verify widget file path is correct -3. Ensure `LEDMatrixWidgets.register()` is called -4. Check that widget name matches schema `x-widget` value - -### Widget Not Rendering - -1. Verify `render` function is defined -2. Check container element exists -3. Ensure widget is registered before form loads -4. Check for JavaScript errors in console - -### Value Not Saving - -1. Ensure widget triggers `widget-change` event -2. Verify form submission includes widget value -3. Check `getValue` function returns correct type -4. Verify field name matches schema property - -## Migration from Server-Side Rendering - -Currently, widgets are server-side rendered via Jinja2 templates. The registry system provides: - -1. **Backwards Compatibility**: Existing server-side rendered widgets continue to work -2. **Future Enhancement**: Client-side rendering support for custom widgets -3. **Handler Availability**: All widget handlers are available globally - -Future versions may support full client-side rendering, but server-side rendering remains the primary method for core widgets. - -## Support - -For questions or issues: -- Check existing widget implementations for examples -- Review browser console for errors -- Test with simple widget first before complex implementations +**Value not saving** +- Fire a `widget-change` event on change. +- `getValue()` must return the type the schema expects. +- Check the field name matches the schema property.