From b7f073a11b44b24c5dc573f66ec26d684c680442 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:52:01 -0400 Subject: [PATCH 01/13] docs(api): make the REST reference match the api_v3 package Every documented request body, query parameter and response shape was re-checked against the handlers in web_interface/blueprints/api_v3/. Fixes calls that failed as documented (repo_url, action_id/params, files/image_id, font_file+font_family, ?font=, cache key, auto_enable_ap_mode, plugin limit keys), removes the font-override endpoints dropped in #566, corrects response shapes (plugins/config, plugins/schema, health, metrics, operation history, github-status, fonts/catalog, cache/list, logs, wifi, on-demand, SSE streams), and adds the 26 routes it omitted (backup, system auto-update/git, wifi radio, starlark editor, MQTT bridge, status endpoints, skins). Documents the merge semantics of partial JSON saves to /config/main and /plugins/config and the dim-schedule POST accepting GET's days shape, which land in the same change set. Replaces app.py line numbers and the removed api_v3.py path with file and function names. Co-Authored-By: Claude Opus 5 --- docs/REST_API_REFERENCE.md | 1212 +++++++++++++++++++++++++----------- 1 file changed, 832 insertions(+), 380 deletions(-) diff --git a/docs/REST_API_REFERENCE.md b/docs/REST_API_REFERENCE.md index 2f312151..5a9865bb 100644 --- a/docs/REST_API_REFERENCE.md +++ b/docs/REST_API_REFERENCE.md @@ -1,10 +1,10 @@ # LEDMatrix REST API Reference -Complete reference for all REST API endpoints available in the LEDMatrix web interface. +Reference for the REST API served by the LEDMatrix web interface. **Base URL**: `http://your-pi-ip:5000/api/v3` -All endpoints return JSON responses with a standard format: +Most endpoints answer JSON in this envelope: ```json { "status": "success" | "error", @@ -13,6 +13,11 @@ All endpoints return JSON responses with a standard format: } ``` +Not every endpoint follows it exactly. Where a response puts fields at the +top level instead of under `data` (install-from-url, registry-from-url, the +auth endpoints, upload endpoints, `system/git-info`, `system/check-update`), +the entry below says so. + ## Table of Contents - [Configuration](#configuration) @@ -20,21 +25,29 @@ All endpoints return JSON responses with a standard format: - [Plugins](#plugins) - [Plugin Store](#plugin-store) - [System](#system) +- [Backup and Restore](#backup-and-restore) - [Fonts](#fonts) - [Cache](#cache) - [WiFi](#wifi) - [Streams](#streams) - [Logs](#logs) - [Error tracking](#error-tracking) -- [Health](#health) +- [Health and Status](#health-and-status) - [Schedule (dim/power)](#schedule-dimpower) +- [Integrations](#integrations) - [Plugin-specific endpoints](#plugin-specific-endpoints) - [Starlark Apps](#starlark-apps) +- [Skins](#skins) -> The API blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). -> SSE stream endpoints (`/api/v3/stream/*`) are defined directly on the -> Flask app at `app.py:799-809`. There are 111 routes total — see -> `web_interface/blueprints/api_v3.py` for the canonical list. +> The API blueprint is the `api_v3` package in +> `web_interface/blueprints/api_v3/` (one module per area: `config.py`, +> `display.py`, `plugins.py`, `system.py`, `backup.py`, `fonts.py`, +> `misc.py`, `wifi.py`, `starlark.py`). `web_interface/app.py` registers it +> at `/api/v3` (`app.register_blueprint(api_v3, url_prefix='/api/v3')`). +> The three SSE endpoints (`/api/v3/stream/*`) are defined directly on the +> Flask app in `app.py` (`stream_stats`, `stream_display`, `stream_logs`). +> `test/fixtures/api_v3_url_map.json` is the canonical list of blueprint +> routes (116 URL rules); a test fails if the code and that fixture differ. --- @@ -44,7 +57,9 @@ All endpoints return JSON responses with a standard format: **GET** `/api/v3/config/main` -Retrieve the complete main configuration file. +Return `config/config.json`. Fields whose names look like credentials +(`api_key`, `token`, `password`, `secret`, ...) are blanked to `""` in the +response; `config/config_secrets.json` values are never included. **Response**: ```json @@ -67,19 +82,36 @@ Retrieve the complete main configuration file. **POST** `/api/v3/config/main` -Update the main configuration. Accepts both JSON and form data. +Update the main configuration. Accepts JSON (`Content-Type: application/json`) +or form data. The body uses the web UI's flat field names, which the handler +maps into the nested config: + +| Fields | Stored at | +|--------|-----------| +| `timezone`, `city`, `state`, `country` | `timezone`, `location.*` | +| `web_display_autostart`, `auto_update_enabled` | `web_display_autostart`, `auto_update.enabled` | +| `plugins_directory`, `auto_discover`, `auto_load_enabled`, `development_mode` | `plugin_system.*` | +| `target_fps` (30-200) | `target_fps` | +| `rows`, `cols`, `chain_length`, `parallel`, `brightness`, `hardware_mapping`, `pwm_bits`, `led_rgb_sequence`, `panel_type`, `pixel_mapper_config`, `disable_hardware_pulsing`, `inverse_colors`, `show_refresh_rate`, ... | `display.hardware.*` | +| `gpio_slowdown`, `rp1_rio` | `display.runtime.*` | +| `use_short_date_format` | `display.use_short_date_format` | +| `max_dynamic_duration_seconds` | `display.dynamic_duration.max_duration_seconds` | +| `double_sided_*`, `vegas_*`, `sync_*` | `display.double_sided`, `display.vegas_scroll`, `sync` | +| `_duration`, `default_duration`, `duration__` | `display.display_durations.*` | +| `plugin_rotation_order` (list of plugin ids) | `display.plugin_rotation_order` | + +Any other top-level key is deep-merged into the config as given. + +A JSON body changes only the keys it contains; everything else keeps its +stored value. (Form posts from the web UI send every field of a tab, and +there an unchecked checkbox — which the browser omits — is saved as +`false`.) **Request Body** (JSON): ```json { "timezone": "America/New_York", "city": "New York", - "state": "NY", - "country": "US", - "web_display_autostart": true, - "rows": 32, - "cols": 64, - "chain_length": 2, "brightness": 90 } ``` @@ -92,11 +124,15 @@ Update the main configuration. Accepts both JSON and form data. } ``` +Invalid values (e.g. an out-of-range `target_fps`, a hardware option the +Raspberry Pi 5 driver cannot use) are rejected with `400` and nothing is +saved. + ### Get Schedule Configuration **GET** `/api/v3/config/schedule` -Retrieve the current schedule configuration. +Retrieve the current on/off schedule. **Response**: ```json @@ -134,7 +170,7 @@ Retrieve the current schedule configuration. **POST** `/api/v3/config/schedule` -Update the schedule configuration. +Replace the schedule configuration. **Request Body** (Global mode): ```json @@ -146,7 +182,7 @@ Update the schedule configuration. } ``` -**Request Body** (Per-day mode): +**Request Body** (Per-day mode, flat form-field names): ```json { "enabled": true, @@ -160,6 +196,9 @@ Update the schedule configuration. } ``` +A day whose `_enabled` key is absent counts as enabled, with default +times `07:00`-`23:00`. At least one day must be enabled. + **Response**: ```json { @@ -172,19 +211,17 @@ Update the schedule configuration. **GET** `/api/v3/config/secrets` -Retrieve the secrets configuration (API keys, tokens, etc.). Secret values are masked for security. +Retrieve `config/config_secrets.json` with every set value replaced by eight +bullet characters (`"••••••••"`). Empty values and `YOUR_*` placeholders +are returned as-is, so a client can tell "set" from "not set". **Response**: ```json { "status": "success", "data": { - "weather": { - "api_key": "***" - }, - "spotify": { - "client_id": "***", - "client_secret": "***" + "ledmatrix-weather": { + "api_key": "••••••••" } } } @@ -194,11 +231,14 @@ Retrieve the secrets configuration (API keys, tokens, etc.). Secret values are m **POST** `/api/v3/config/raw/main` -Save raw JSON configuration (advanced use only). +Replace `config/config.json` with the JSON body (advanced use only). **POST** `/api/v3/config/raw/secrets` -Save raw secrets configuration (advanced use only). +Save the secrets file (advanced use only). Masked values (`"••••••••"`) +and blank strings in the body are dropped, and the rest is merged onto the +stored secrets, so posting back the GET response unchanged changes nothing. +A secret cannot be cleared by blanking it here. --- @@ -208,7 +248,8 @@ Save raw secrets configuration (advanced use only). **GET** `/api/v3/display/current` -Get the current display state and preview image. +Get the latest display snapshot as a base64 PNG (`image` is `null` when no +snapshot is available). **Response**: ```json @@ -223,6 +264,27 @@ Get the current display state and preview image. } ``` +### Get Current Display Status + +**GET** `/api/v3/display/current-status` + +The mode and plugin the display service is currently showing, as published +by the display process (stale after 120 seconds). + +**Response**: +```json +{ + "status": "success", + "data": { + "mode": "nfl_live", + "plugin_id": "football-scoreboard", + "last_updated": 1234567890.123 + } +} +``` + +When nothing has been published, every field is `null`. + ### List Display Modes **GET** `/api/v3/display/modes` @@ -295,12 +357,17 @@ Get the current on-demand display state. }, "service": { "active": true, - "returncode": 0 + "returncode": 0, + "stdout": "active", + "stderr": "" } } } ``` +With no on-demand request, `state` is +`{"active": false, "status": "idle", "last_updated": null}`. + ### Start On-Demand Display **POST** `/api/v3/display/on-demand/start` @@ -318,12 +385,12 @@ Request a specific plugin to display on-demand. } ``` -**Parameters**: +**Parameters** (at least one of `plugin_id` and `mode` is required): - `plugin_id` (string, optional): Plugin identifier - `mode` (string, optional): Display mode name (plugin_id inferred if not provided) - `duration` (number, optional): Duration in seconds (0 = until stopped) - `pinned` (boolean, optional): Pin display (pause rotation) -- `start_service` (boolean, optional): Auto-start display service if not running (default: true) +- `start_service` (boolean, optional): (Re)start the display service so it picks the request up (default: true) **Response**: ```json @@ -333,11 +400,15 @@ Request a specific plugin to display on-demand. "request_id": "uuid-here", "plugin_id": "football-scoreboard", "mode": "nfl_live", - "active": true + "duration": 45, + "pinned": true, + "service": { "active": true, "returncode": 0, "stdout": "", "stderr": "" } } } ``` +`service` is `null` when `start_service` is false. + ### Stop On-Demand Display **POST** `/api/v3/display/on-demand/stop` @@ -358,7 +429,10 @@ Stop the current on-demand display. ```json { "status": "success", - "message": "On-demand display stopped" + "data": { + "request_id": "uuid-here", + "service": null + } } ``` @@ -381,6 +455,9 @@ List all installed plugins with their status and metadata. { "id": "football-scoreboard", "name": "Football Scoreboard", + "version": "1.2.3", + "latest_version": "1.2.4", + "update_available": true, "author": "ChuckBuilds", "category": "Sports", "description": "NFL and NCAA Football scores", @@ -388,11 +465,15 @@ List all installed plugins with their status and metadata. "enabled": true, "verified": true, "loaded": true, + "state": "loaded", + "error_info": null, "last_updated": "2025-01-15T10:30:00Z", "last_commit": "abc1234", "last_commit_message": "feat: Add live game updates", "branch": "main", - "web_ui_actions": [] + "web_ui_actions": [], + "vegas_mode": null, + "vegas_content_type": null } ] } @@ -403,7 +484,8 @@ List all installed plugins with their status and metadata. **GET** `/api/v3/plugins/config?plugin_id=` -Get configuration for a specific plugin. +Get a plugin's configuration, with schema defaults filled in for keys that +are not stored. `data` is the configuration object itself. **Query Parameters**: - `plugin_id` (required): Plugin identifier @@ -413,12 +495,9 @@ Get configuration for a specific plugin. { "status": "success", "data": { - "plugin_id": "football-scoreboard", - "config": { - "enabled": true, - "display_duration": 30, - "favorite_teams": ["TB", "DAL"] - } + "enabled": true, + "display_duration": 30, + "favorite_teams": ["TB", "DAL"] } } ``` @@ -427,14 +506,17 @@ Get configuration for a specific plugin. **POST** `/api/v3/plugins/config` -Update plugin configuration. +Update a plugin's configuration. With a JSON body, the keys in `config` are +merged onto the plugin's stored configuration: keys you do not send keep +their stored values. Fields the schema marks `"x-secret": true` are written +to `config/config_secrets.json` instead of `config.json`. The web UI posts +form data instead (`?plugin_id=` in the query string, fields as form fields). **Request Body**: ```json { "plugin_id": "football-scoreboard", "config": { - "enabled": true, "display_duration": 30, "favorite_teams": ["TB", "DAL"] } @@ -445,15 +527,19 @@ Update plugin configuration. ```json { "status": "success", - "message": "Plugin configuration saved successfully" + "message": "Plugin football-scoreboard configuration saved successfully" } ``` +A config that fails schema validation is rejected with `400` and nothing is +saved. + ### Get Plugin Schema **GET** `/api/v3/plugins/schema?plugin_id=` -Get the JSON schema for a plugin's configuration. +Get the JSON schema for a plugin's configuration. A plugin without a +`config_schema.json` gets a minimal default schema. **Query Parameters**: - `plugin_id` (required): Plugin identifier @@ -463,27 +549,53 @@ Get the JSON schema for a plugin's configuration. { "status": "success", "data": { - "type": "object", - "properties": { - "enabled": { - "type": "boolean", - "default": true - }, - "display_duration": { - "type": "number", - "minimum": 1, - "maximum": 300 + "schema": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean", + "default": true + }, + "display_duration": { + "type": "number", + "minimum": 1, + "maximum": 300 + } } } } } ``` +### Reset Plugin Configuration + +**POST** `/api/v3/plugins/config/reset` + +Reset a plugin's configuration to its schema defaults. + +**Request Body**: +```json +{ + "plugin_id": "football-scoreboard", + "preserve_secrets": true +} +``` + +**Response**: +```json +{ + "status": "success", + "message": "Plugin football-scoreboard configuration reset to defaults", + "data": { "config": { ... } } +} +``` + ### Toggle Plugin **POST** `/api/v3/plugins/toggle` -Enable or disable a plugin. +Enable or disable a plugin. A `plugin_id` of the form `starlark:` +toggles a Starlark app. **Request Body**: ```json @@ -497,7 +609,7 @@ Enable or disable a plugin. ```json { "status": "success", - "message": "Plugin football-scoreboard enabled" + "message": "Plugin football-scoreboard enabled successfully" } ``` @@ -510,22 +622,25 @@ Install a plugin from the plugin store. **Request Body**: ```json { - "plugin_id": "football-scoreboard" + "plugin_id": "football-scoreboard", + "branch": "main" } ``` -**Response**: +`branch` is optional. + +**Response** (queued; poll `/plugins/operation/`): ```json { "status": "success", - "data": { - "operation_id": "uuid-here", - "plugin_id": "football-scoreboard", - "status": "installing" - } + "data": { "operation_id": "uuid-here" }, + "message": "Plugin football-scoreboard installation queued" } ``` +When the operation queue is unavailable the install runs synchronously and +the response has only a `message`. + ### Uninstall Plugin **POST** `/api/v3/plugins/uninstall` @@ -535,15 +650,17 @@ Remove an installed plugin. **Request Body**: ```json { - "plugin_id": "football-scoreboard" + "plugin_id": "football-scoreboard", + "preserve_config": false } ``` -**Response**: +**Response** (queued): ```json { "status": "success", - "message": "Plugin football-scoreboard uninstalled" + "data": { "operation_id": "uuid-here" }, + "message": "Plugin uninstallation queued" } ``` @@ -551,7 +668,7 @@ Remove an installed plugin. **POST** `/api/v3/plugins/update` -Update a plugin to the latest version. +Update a plugin to the latest version. Runs synchronously. **Request Body**: ```json @@ -564,10 +681,10 @@ Update a plugin to the latest version. ```json { "status": "success", + "message": "Plugin football-scoreboard updated ...", "data": { - "operation_id": "uuid-here", - "plugin_id": "football-scoreboard", - "status": "updating" + "last_updated": "2025-01-15T10:30:00Z", + "commit": "abc1234..." } } ``` @@ -576,31 +693,31 @@ Update a plugin to the latest version. **POST** `/api/v3/plugins/install-from-url` -Install a plugin directly from a GitHub repository URL. +Install a plugin directly from a GitHub repository URL. Runs synchronously. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-my-plugin", + "repo_url": "https://github.com/user/ledmatrix-my-plugin", "branch": "main", "plugin_path": null } ``` **Parameters**: -- `url` (required): GitHub repository URL -- `branch` (optional): Branch name (default: "main") -- `plugin_path` (optional): Path within repository for monorepo plugins +- `repo_url` (required): GitHub repository URL +- `branch` (optional): Branch name (default: `main`, then `master`) +- `plugin_path` (optional): Path within the repository, for monorepo plugins +- `plugin_id` (optional): Plugin id, for monorepo installations -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "operation_id": "uuid-here", - "plugin_id": "my-plugin", - "status": "installing" - } + "message": "Plugin my-plugin installed successfully", + "plugin_id": "my-plugin", + "name": "My Plugin", + "branch": "main" } ``` @@ -608,28 +725,27 @@ Install a plugin directly from a GitHub repository URL. **POST** `/api/v3/plugins/registry-from-url` -Load a plugin registry from a GitHub repository URL. +Load a `plugins.json` registry from a GitHub repository URL. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-plugins" + "repo_url": "https://github.com/user/ledmatrix-plugins" } ``` -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "plugins": [ - { - "id": "plugin-1", - "name": "Plugin One", - "description": "..." - } - ] - } + "plugins": [ + { + "id": "plugin-1", + "name": "Plugin One", + "description": "..." + } + ], + "registry_url": "https://github.com/user/ledmatrix-plugins" } ``` @@ -637,7 +753,7 @@ Load a plugin registry from a GitHub repository URL. **GET** `/api/v3/plugins/health` -Get health metrics for all plugins. +Get health state for all installed plugins, keyed by plugin id. **Response**: ```json @@ -645,10 +761,20 @@ Get health metrics for all plugins. "status": "success", "data": { "football-scoreboard": { - "status": "healthy", - "last_update": 1234567890.123, - "error_count": 0, - "last_error": null + "plugin_id": "football-scoreboard", + "circuit_state": "closed", + "consecutive_failures": 0, + "total_failures": 2, + "total_successes": 1500, + "success_rate": 99.87, + "last_success_time": 1234567890.123, + "last_failure_time": 1234560000.0, + "last_error": null, + "is_healthy": true, + "degraded": false, + "degraded_reason": null, + "circuit_opened_time": null, + "half_open_start_time": null } } } @@ -658,20 +784,8 @@ Get health metrics for all plugins. **GET** `/api/v3/plugins/health/` -Get health metrics for a specific plugin. - -**Response**: -```json -{ - "status": "success", - "data": { - "status": "healthy", - "last_update": 1234567890.123, - "error_count": 0, - "last_error": null - } -} -``` +Health state for one plugin; `data` has the same fields as one entry above. +Answers `503` when health tracking is unavailable. ### Reset Plugin Health @@ -691,7 +805,7 @@ Reset health state for a plugin (manual recovery). **GET** `/api/v3/plugins/metrics` -Get resource usage metrics for all plugins. +Get resource usage metrics for all installed plugins, keyed by plugin id. **Response**: ```json @@ -699,21 +813,34 @@ Get resource usage metrics for all plugins. "status": "success", "data": { "football-scoreboard": { - "update_count": 150, - "display_count": 500, - "avg_update_time": 0.5, - "avg_display_time": 0.1, - "memory_usage": 1024000 + "plugin_id": "football-scoreboard", + "memory_mb": 24.5, + "cpu_percent": 3.2, + "execution_time": 0.12, + "avg_execution_time": 0.1, + "min_execution_time": 0.05, + "max_execution_time": 0.9, + "call_count": 500, + "last_update_time": 1234567890.123, + "limits": { + "max_memory_mb": 50, + "max_cpu_percent": 50, + "max_execution_time": 5.0, + "warning_threshold": 0.8 + } } } } ``` +`limits` (and usage percentages derived from it) appear only when limits +are configured for the plugin. + ### Get Plugin Metrics (Single) **GET** `/api/v3/plugins/metrics/` -Get resource usage metrics for a specific plugin. +Metrics for one plugin; `data` has the same fields as one entry above. ### Reset Plugin Metrics @@ -725,18 +852,33 @@ Reset metrics for a plugin. **GET** `/api/v3/plugins/limits/` -Get rate limits and resource limits for a plugin. +Get a plugin's resource limits. `data` is `null` when none are configured. + +**Response**: +```json +{ + "status": "success", + "data": { + "max_memory_mb": 50, + "max_cpu_percent": 50, + "max_execution_time": 5.0, + "warning_threshold": 0.8 + } +} +``` **POST** `/api/v3/plugins/limits/` -Update rate limits and resource limits for a plugin. +Set a plugin's resource limits. The body replaces all four limits: a key you +omit is stored as no limit (`warning_threshold` defaults to `0.8`). **Request Body**: ```json { - "max_update_interval": 60, - "max_display_time": 5.0, - "max_memory_mb": 50 + "max_memory_mb": 50, + "max_cpu_percent": 50, + "max_execution_time": 5.0, + "warning_threshold": 0.8 } ``` @@ -744,7 +886,8 @@ Update rate limits and resource limits for a plugin. **GET** `/api/v3/plugins/state` -Get the current state of all plugins. +Get the state manager's record for every plugin, keyed by plugin id. Pass +`?plugin_id=` for one plugin (`data` is then that record). **Response**: ```json @@ -752,9 +895,14 @@ Get the current state of all plugins. "status": "success", "data": { "football-scoreboard": { - "state": "loaded", + "plugin_id": "football-scoreboard", + "status": "loaded", "enabled": true, - "last_update": 1234567890.123 + "version": "1.2.3", + "installed_at": "2025-01-15T10:30:00", + "last_updated": "2025-01-15T10:30:00", + "config_version": 1, + "metadata": {} } } } @@ -764,21 +912,57 @@ Get the current state of all plugins. **POST** `/api/v3/plugins/state/reconcile` -Reconcile plugin state with configuration (fix inconsistencies). +Reconcile plugin state across config, disk and the state manager. + +**Request Body** (optional): +```json +{ + "force": false +} +``` **Response**: ```json { "status": "success", - "message": "Plugin state reconciled" + "message": "...", + "data": { + "inconsistencies_found": 1, + "inconsistencies_fixed": 1, + "inconsistencies_manual": 0, + "inconsistencies": [ + {"plugin_id": "...", "type": "...", "description": "...", "fix_action": "..."} + ], + "fixed": [ ... ], + "manual_fix_required": [ ... ] + } } ``` +### Get Reconciliation Status + +**GET** `/api/v3/plugins/reconciliation-status` + +Result of the last startup reconciliation, as written by the display service. + +**Response**: +```json +{ + "status": "success", + "data": { + "done": true, + "unresolved": [] + } +} +``` + +Before a run has finished, `data` is `{"done": false, "unresolved": []}`. + ### Get Plugin Operation **GET** `/api/v3/plugins/operation/` -Get status of an async plugin operation (install, update, etc.). +Get status of a queued plugin operation (install, uninstall). **Response**: ```json @@ -786,67 +970,80 @@ Get status of an async plugin operation (install, update, etc.). "status": "success", "data": { "operation_id": "uuid-here", - "type": "install", + "operation_type": "install", "plugin_id": "football-scoreboard", + "parameters": {}, "status": "completed", "progress": 100, - "message": "Installation completed successfully" + "message": "Installation completed successfully", + "error": null, + "result": { ... }, + "created_at": "2025-01-15T10:30:00", + "started_at": "2025-01-15T10:30:01", + "completed_at": "2025-01-15T10:30:20" } } ``` ### Get Operation History -**GET** `/api/v3/plugins/operation/history?limit=100` +**GET** `/api/v3/plugins/operation/history?limit=50` -Get history of plugin operations. +Get the plugin operation audit log. `data` is a list. **Query Parameters**: -- `limit` (optional): Maximum number of operations to return (default: 100) +- `limit` (optional): Maximum number of records (default: 50) +- `plugin_id` (optional): Only records for this plugin +- `operation_type` (optional): Only records of this type (`install`, `update`, `enable`, ...) **Response**: ```json { "status": "success", - "data": { - "operations": [ - { - "operation_id": "uuid-here", - "type": "install", - "plugin_id": "football-scoreboard", - "status": "completed", - "timestamp": 1234567890.123 - } - ] - } + "data": [ + { + "operation_id": "uuid-here", + "operation_type": "install", + "plugin_id": "football-scoreboard", + "timestamp": "2025-01-15T10:30:00", + "status": "success", + "user": null, + "details": null, + "error": null + } + ] } ``` +### Clear Operation History + +**DELETE** `/api/v3/plugins/operation/history` + +Clear the operation audit log. + ### Execute Plugin Action **POST** `/api/v3/plugins/action` -Execute a custom plugin action (defined in plugin's web_ui_actions). +Execute an action declared in the plugin manifest's `web_ui_actions`. See +[PLUGIN_WEB_UI_ACTIONS.md](PLUGIN_WEB_UI_ACTIONS.md). **Request Body**: ```json { "plugin_id": "football-scoreboard", - "action": "refresh_games", - "parameters": {} + "action_id": "refresh_games", + "params": {} } ``` -### Reset Plugin Configuration - -**POST** `/api/v3/plugins/config/reset` - -Reset a plugin's configuration to defaults. - -**Request Body**: +**Response** (fields at the top level; a script that prints JSON can return +its own object instead): ```json { - "plugin_id": "football-scoreboard" + "status": "success", + "message": "Action completed successfully", + "output": "script stdout" } ``` @@ -854,21 +1051,27 @@ Reset a plugin's configuration to defaults. **POST** `/api/v3/plugins/assets/upload` -Upload assets (images, files) for a plugin. +Upload images for a plugin. Stored under +`assets/plugins//uploads/`. **Request**: Multipart form data - `plugin_id` (required): Plugin identifier -- `file` (required): File to upload -- `asset_type` (optional): Type of asset (logo, image, etc.) +- `files` (required, repeatable, up to 10): PNG, JPEG, BMP or GIF images, 5 MB each, 50 MB total per plugin -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "filename": "logo.png", - "path": "plugins/football-scoreboard/assets/logo.png" - } + "uploaded_files": [ + { + "id": "uuid-here", + "filename": "image_1700000000_abcd1234.png", + "path": "assets/plugins/football-scoreboard/uploads/image_1700000000_abcd1234.png", + "size": 1024, + "uploaded_at": "2025-01-15T10:30:00Z" + } + ], + "total_files": 3 } ``` @@ -876,13 +1079,13 @@ Upload assets (images, files) for a plugin. **POST** `/api/v3/plugins/assets/delete` -Delete a plugin asset. +Delete an uploaded plugin image by its id (the `id` from upload or list). **Request Body**: ```json { "plugin_id": "football-scoreboard", - "filename": "logo.png" + "image_id": "uuid-here" } ``` @@ -890,7 +1093,7 @@ Delete a plugin asset. **GET** `/api/v3/plugins/assets/list?plugin_id=` -List all assets for a plugin. +List uploaded images for a plugin. **Query Parameters**: - `plugin_id` (required): Plugin identifier @@ -902,9 +1105,12 @@ List all assets for a plugin. "data": { "assets": [ { - "filename": "logo.png", - "path": "plugins/football-scoreboard/assets/logo.png", - "size": 1024 + "id": "uuid-here", + "filename": "image_1700000000_abcd1234.png", + "path": "assets/plugins/football-scoreboard/uploads/image_1700000000_abcd1234.png", + "size": 1024, + "uploaded_at": "2025-01-15T10:30:00Z", + "original_filename": "logo.png" } ] } @@ -915,59 +1121,76 @@ List all assets for a plugin. **POST** `/api/v3/plugins/authenticate/spotify` -Initiate Spotify authentication flow for music plugin. +Spotify OAuth for the music plugin (`ledmatrix-music`; the plugin is fixed, +not taken from the body). Two steps: call with an empty body to get the +authorization URL, then call again with the URL Spotify redirected to. -**Request Body**: +**Request Body** (step 2): ```json { - "plugin_id": "music" + "redirect_url": "http://127.0.0.1:8888/callback?code=..." } ``` -**Response**: +**Response** (step 1, fields at the top level): ```json { "status": "success", - "data": { - "auth_url": "https://accounts.spotify.com/authorize?..." - } + "message": "Authorization URL generated", + "auth_url": "https://accounts.spotify.com/authorize?..." } ``` +Step 2 returns `status`, `message` and the script's `output`. + ### Authenticate YouTube Music **POST** `/api/v3/plugins/authenticate/ytm` -Initiate YouTube Music authentication flow. - -**Request Body**: -```json -{ - "plugin_id": "music" -} -``` +Run the music plugin's YouTube Music authentication script. No body. Returns +`status`, `message` and the script's `output`. ### Upload Calendar Credentials **POST** `/api/v3/plugins/calendar/upload-credentials` -Upload Google Calendar credentials file. +Upload the Google OAuth client file for the calendar plugin. **Request**: Multipart form data -- `file` (required): credentials.json file +- `file` (required): `credentials.json` (JSON, max 1 MB) + +### Authenticate Calendar + +**POST** `/api/v3/plugins/calendar/authenticate` + +Google OAuth for the calendar plugin, in two steps. Step 1 (no body) returns +the consent URL. Step 2 posts the URL Google redirected to (it fails to load +in the browser, but its address carries the authorization code): + +```json +{ + "redirect_url": "http://localhost/?code=..." +} +``` + +Requires `credentials.json` to have been uploaded first (`400` otherwise). --- ## Plugin Store -### List Store Plugins +### List / Search Store Plugins -**GET** `/api/v3/plugins/store/list?fetch_commit_info=true` +**GET** `/api/v3/plugins/store/list` -Get list of available plugins from the plugin store. +List plugins from the registry and saved repositories. The same endpoint +searches. **Query Parameters**: -- `fetch_commit_info` (optional): Include commit information (default: false) +- `query` (optional): Text search over name, description, id +- `category` (optional): Category filter +- `tags` (optional, repeatable): Tag filter +- `fetch_commit_info` (optional): `false` to skip fetching commit metadata from GitHub (default: fetched) **Response**: ```json @@ -978,13 +1201,22 @@ Get list of available plugins from the plugin store. { "id": "football-scoreboard", "name": "Football Scoreboard", - "description": "NFL and NCAA Football scores", "author": "ChuckBuilds", "category": "Sports", + "description": "NFL and NCAA Football scores", + "tags": ["sports"], + "stars": 0, + "verified": true, + "repo": "https://github.com/ChuckBuilds/ledmatrix-plugins", + "last_updated": "2025-01-15", + "last_updated_iso": "2025-01-15T10:30:00Z", + "last_commit": "abc1234", + "last_commit_message": "...", + "last_commit_author": "...", "version": "1.2.3", - "repository_url": "https://github.com/ChuckBuilds/ledmatrix-football-scoreboard", - "installed": true, - "update_available": false + "branch": "main", + "default_branch": "main", + "plugin_path": "plugins/football-scoreboard" } ] } @@ -995,31 +1227,37 @@ Get list of available plugins from the plugin store. **GET** `/api/v3/plugins/store/github-status` -Get GitHub API rate limit status. +Whether a GitHub token is configured and valid. **Response**: ```json { "status": "success", "data": { + "token_status": "valid", + "authenticated": true, "rate_limit": 5000, - "rate_remaining": 4500, - "rate_reset": 1234567890 + "message": "GitHub API authenticated", + "error": null } } ``` +`token_status` is `none`, `valid` or `invalid`; `rate_limit` is the nominal +hourly limit (60 unauthenticated), not a live count. + ### Refresh Plugin Store **POST** `/api/v3/plugins/store/refresh` -Force refresh of the plugin store cache. +Force refresh of the registry cache. **Response**: ```json { "status": "success", - "message": "Plugin store refreshed" + "message": "Plugin store refreshed", + "plugin_count": 42 } ``` @@ -1027,7 +1265,7 @@ Force refresh of the plugin store cache. **GET** `/api/v3/plugins/saved-repositories` -Get list of saved custom plugin repositories. +Get the list of saved custom plugin repositories. **Response**: ```json @@ -1037,8 +1275,8 @@ Get list of saved custom plugin repositories. "repositories": [ { "url": "https://github.com/user/ledmatrix-plugins", - "name": "Custom Plugins", - "auto_load": true + "name": "ledmatrix-plugins", + "type": "registry" } ] } @@ -1049,14 +1287,14 @@ Get list of saved custom plugin repositories. **POST** `/api/v3/plugins/saved-repositories` -Save a custom plugin repository for easy access. +Save a custom plugin repository. Returns the updated list in +`data.repositories`. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-plugins", - "name": "Custom Plugins", - "auto_load": true + "repo_url": "https://github.com/user/ledmatrix-plugins", + "name": "Custom Plugins" } ``` @@ -1064,12 +1302,12 @@ Save a custom plugin repository for easy access. **DELETE** `/api/v3/plugins/saved-repositories` -Remove a saved repository. +Remove a saved repository. Returns the updated list in `data.repositories`. **Request Body**: ```json { - "url": "https://github.com/user/ledmatrix-plugins" + "repo_url": "https://github.com/user/ledmatrix-plugins" } ``` @@ -1081,7 +1319,7 @@ Remove a saved repository. **GET** `/api/v3/system/status` -Get system status and metrics. +Get system status and metrics (cached for 10 seconds). **Response**: ```json @@ -1089,12 +1327,18 @@ Get system status and metrics. "status": "success", "data": { "timestamp": 1234567890.123, - "uptime": "Running", + "uptime": "3d 4h", + "uptime_seconds": 273600, "service_active": true, "cpu_percent": 25.5, "memory_used_percent": 45.2, + "memory_total_mb": 3794.0, + "memory_used_mb": 1715.0, + "memory_available_mb": 1900.0, "cpu_temp": 45.0, - "disk_used_percent": 60.0 + "disk_used_percent": 60.0, + "disk_total_gb": 29.0, + "disk_used_gb": 17.4 } } ``` @@ -1115,17 +1359,68 @@ Get LEDMatrix repository version. } ``` +### Check for Update + +**GET** `/api/v3/system/check-update` + +Whether `origin/main` has commits the checkout lacks. Cached briefly. +Fields at the top level (no envelope): + +```json +{ + "update_available": true, + "remote_sha": "abc123...", + "commits_behind": 3 +} +``` + +When git cannot run the check, the response also carries +`"check_failed": true` and an `error` explaining why. + +### Automatic Update Status + +**GET** `/api/v3/system/auto-update` + +Weekly automatic-update status for the General tab and the Overview banner: +`last_run`, `summary`, `status`, `next_due`, `alert`, `alert_id`, +`verifier_installed`, `setup_status`, `setup_message`, `verifying` (in +`data`). + +**POST** `/api/v3/system/auto-update/dismiss` + +Hide the current automatic-update alert until a new one replaces it. + +```json +{ + "alert_id": "..." +} +``` + +### Git Info + +**GET** `/api/v3/system/git-info` + +Branch, dirty state, recent commits and remote for the Tools tab. Fields at +the top level: `branch`, `dirty`, `status`, `recent_commits`, `remote_url` +(credentials scrubbed), `upstream`, `can_pull`. + +### Git Branches + +**GET** `/api/v3/system/git-branches` + +Fetches `origin` and lists branches to switch to: `current`, `upstream`, +`local` (list), `remote_only` (list), at the top level. + ### Execute System Action **POST** `/api/v3/system/action` -Execute system-level actions. +Execute system-level actions. JSON or form data. **Request Body**: ```json { - "action": "start_display", - "mode": "nfl_live" + "action": "restart_display_service" } ``` @@ -1137,19 +1432,87 @@ Execute system-level actions. - `enable_autostart`: Enable display service autostart - `disable_autostart`: Disable display service autostart - `reboot_system`: Reboot the Raspberry Pi -- `git_pull`: Update code from git repository +- `shutdown_system`: Power off the Raspberry Pi +- `git_pull`: Update LEDMatrix from git (the Update button) +- `checkout_branch`: Switch branch; takes `branch` and optional `stash` +- `force_git_reset`: `git reset --hard origin/main` +- `install_base_requirements`: pip install `requirements.txt` and `web_interface/requirements.txt` +- `install_plugin_requirements`: pip install every plugin's `requirements.txt` +- `clear_pycache`: Delete `__pycache__` directories -**Response**: +**Response** (service actions): ```json { "status": "success", - "message": "Action start_display completed", - "returncode": 0, - "stdout": "...", - "stderr": "" + "message": "Action completed" } ``` +A failed service action returns `"status": "error"` with `returncode` and +`stderr`. `git_pull` returns `message`, `restart_required` and +`dependency_failures`; the install actions return `output` or `details`. + +--- + +## Backup and Restore + +Backups are ZIP files kept in the backup export directory. + +### Preview + +**GET** `/api/v3/backup/preview` + +Summary of what a new backup would include. + +### List + +**GET** `/api/v3/backup/list` + +Stored backups, newest first. `data` is a list of +`{"filename", "size", "created_at"}`. + +### Export + +**POST** `/api/v3/backup/export` + +Create a backup. Returns `{"status": "success", "filename": "..."}`. + +### Validate + +**POST** `/api/v3/backup/validate` + +Check an uploaded backup and return its manifest in `data`. + +**Request**: Multipart form data +- `backup_file` (required): the ZIP + +### Restore + +**POST** `/api/v3/backup/restore` + +Restore an uploaded backup. + +**Request**: Multipart form data +- `backup_file` (required): the ZIP +- `options` (optional): JSON object; keys `restore_config`, `restore_secrets`, + `restore_wifi`, `restore_fonts`, `restore_plugin_uploads`, + `reinstall_plugins` (each defaults to `true`; unknown keys are rejected) + +A partial restore answers `500` with `"status": "error"`, a message listing +what did and didn't restore, and the result in `data`. + +### Download + +**GET** `/api/v3/backup/download/` + +Download a stored backup. + +### Delete + +**DELETE** `/api/v3/backup/` + +Delete a stored backup. + --- ## Fonts @@ -1158,20 +1521,26 @@ Execute system-level actions. **GET** `/api/v3/fonts/catalog` -Get list of available fonts. +Fonts in `assets/fonts/`, keyed by file name without extension. **Response**: ```json { "status": "success", "data": { - "fonts": [ - { - "family": "Press Start 2P", - "files": ["PressStart2P-Regular.ttf"], - "sizes": [8, 10, 12] + "catalog": { + "press_start": { + "filename": "press_start.ttf", + "family_name": "Press Start 2P", + "display_name": "Press Start 2P", + "path": "assets/fonts/press_start.ttf", + "type": "ttf", + "is_system": true, + "scalable": true, + "native_size": null, + "metadata": { ... } } - ] + } } } ``` @@ -1192,71 +1561,31 @@ Get font size token definitions. "sm": 8, "md": 10, "lg": 12, - "xl": 16 + "xl": 14, + "xxl": 16 } } } ``` -### Get Font Overrides - -**GET** `/api/v3/fonts/overrides` - -Get current font overrides. - -**Response**: -```json -{ - "status": "success", - "data": { - "overrides": { - "plugin.football-scoreboard.title": { - "family": "Arial", - "size_px": 12 - } - } - } -} -``` - -### Set Font Override - -**POST** `/api/v3/fonts/overrides` - -Set a font override for a specific element. - -**Request Body**: -```json -{ - "element_key": "plugin.football-scoreboard.title", - "family": "Arial", - "size_px": 12 -} -``` - -### Delete Font Override - -**DELETE** `/api/v3/fonts/overrides/` - -Remove a font override. - ### Upload Font **POST** `/api/v3/fonts/upload` -Upload a custom font file. +Upload a custom font file. It is saved as `assets/fonts/`. **Request**: Multipart form data -- `file` (required): Font file (.ttf, .otf, etc.) +- `font_file` (required): `.ttf`, `.otf` or `.bdf`, max 10 MB +- `font_family` (required): name for the font (letters, numbers, `_`, `-`) -**Response**: +**Response** (fields at the top level): ```json { "status": "success", - "data": { - "family": "Custom Font", - "filename": "custom-font.ttf" - } + "message": "Font custom_font uploaded successfully", + "font_family": "custom_font", + "filename": "custom_font.ttf", + "path": "assets/fonts/custom_font.ttf" } ``` @@ -1264,13 +1593,36 @@ Upload a custom font file. **DELETE** `/api/v3/fonts/` -Delete an uploaded font. +Delete an uploaded font (`` is the file name without +extension). System fonts answer `403`. ### Font Preview -**GET** `/api/v3/fonts/preview?family=&text=` +**GET** `/api/v3/fonts/preview?font=&text=&size=12` -Render a small preview image of a font for use in the web UI font picker. +Render text in a font, for the web UI font picker. BDF fonts are not +previewed (`400`). + +**Query Parameters**: +- `font` (required): font file name in `assets/fonts/` (e.g. `press_start.ttf`) +- `text` (optional): up to 100 characters (default `Sample Text 123`) +- `size` (optional): 4-72 (default 12) +- `bg`, `fg` (optional): hex colours without `#` (default `000000` / `ffffff`) + +**Response**: +```json +{ + "status": "success", + "data": { + "image": "data:image/png;base64,...", + "width": 140, + "height": 32 + } +} +``` + +> Font overrides (`/api/v3/fonts/overrides`) were removed. Per-plugin font +> choices are made in each plugin's own settings. --- @@ -1280,20 +1632,16 @@ Render a small preview image of a font for use in the web UI font picker. **GET** `/api/v3/cache/list` -List all cache entries. +List cache files. **Response**: ```json { "status": "success", "data": { - "entries": [ - { - "key": "weather_current_12345", - "age": 300, - "size": 1024 - } - ] + "cache_files": [ ... ], + "cache_dir": "/var/cache/ledmatrix", + "total_files": 12 } } ``` @@ -1302,7 +1650,8 @@ List all cache entries. **POST** `/api/v3/cache/delete` -Delete a cache entry or clear all cache. +Delete one cache entry by key. There is no clear-all option here; use +`scripts/utils/clear_cache.py --clear-all` on the Pi for that. **Request Body**: ```json @@ -1311,13 +1660,6 @@ Delete a cache entry or clear all cache. } ``` -**Or clear all**: -```json -{ - "clear_all": true -} -``` - --- ## WiFi @@ -1336,7 +1678,10 @@ Get current WiFi connection status. "connected": true, "ssid": "MyNetwork", "ip_address": "192.168.1.100", - "signal_strength": -50 + "signal": 70, + "ap_mode_active": false, + "auto_enable_ap_mode": true, + "last_connect_attempt": null } } ``` @@ -1345,22 +1690,21 @@ Get current WiFi connection status. **GET** `/api/v3/wifi/scan` -Scan for available WiFi networks. +Scan for available WiFi networks. `data` is a list. If AP mode is active it is +turned off for the scan and back on afterwards, and `message` says so. **Response**: ```json { "status": "success", - "data": { - "networks": [ - { - "ssid": "MyNetwork", - "signal_strength": -50, - "encryption": "WPA2", - "connected": true - } - ] - } + "data": [ + { + "ssid": "MyNetwork", + "signal": 70, + "security": "WPA2", + "frequency": 2437 + } + ] } ``` @@ -1378,13 +1722,10 @@ Connect to a WiFi network. } ``` -**Response**: -```json -{ - "status": "success", - "message": "Connecting to MyNetwork..." -} -``` +**Response**: `"status": "success"` when connected; `"status": "pending"` +(with `data.ssid`) when the connection continues in the background — poll +`/wifi/status` and read `last_connect_attempt`. A wrong password answers +`400` with `"error_type": "wrong_password"`. ### Disconnect from WiFi @@ -1396,7 +1737,7 @@ Disconnect from current WiFi network. **POST** `/api/v3/wifi/ap/enable` -Enable WiFi access point mode. +Enable WiFi access point mode. Optional body `{"force": true}`. ### Disable Access Point Mode @@ -1404,19 +1745,16 @@ Enable WiFi access point mode. Disable WiFi access point mode. -### Get Auto-Enable AP Status +### Get Auto-Enable AP Setting **GET** `/api/v3/wifi/ap/auto-enable` -Get access point auto-enable configuration. - **Response**: ```json { "status": "success", "data": { - "auto_enable": true, - "timeout_seconds": 300 + "auto_enable_ap_mode": true } } ``` @@ -1425,13 +1763,29 @@ Get access point auto-enable configuration. **POST** `/api/v3/wifi/ap/auto-enable` -Configure access point auto-enable settings. - **Request Body**: ```json { - "auto_enable": true, - "timeout_seconds": 300 + "auto_enable_ap_mode": true +} +``` + +### WiFi Radio + +**GET** `/api/v3/wifi/radio` + +Radio state: `data.enabled` (`null` if unknown), `data.ethernet_connected`, +`data.available`. + +**POST** `/api/v3/wifi/radio` + +Turn the WiFi radio on or off. Turning it off is refused unless Ethernet is +connected or `force` is true, so you don't cut off your own connection. + +```json +{ + "enabled": false, + "force": false } ``` @@ -1439,39 +1793,34 @@ Configure access point auto-enable settings. ## Streams +Server-Sent Events, defined in `web_interface/app.py`. Each event is one +`data: ` line; idle connections get `: heartbeat` comments. + ### System Statistics Stream **GET** `/api/v3/stream/stats` -Server-Sent Events (SSE) stream for real-time system statistics. - -**Response**: SSE stream ``` -data: {"cpu_percent": 25.5, "memory_used_percent": 45.2, ...} - -data: {"cpu_percent": 26.0, "memory_used_percent": 45.3, ...} +data: {"timestamp": 1234567890.1, "uptime": "Running", "service_active": true, "cpu_percent": 25.5, "memory_used_percent": 45.2, "memory_available_mb": 1900.0, "cpu_temp": 45.0, "disk_used_percent": 60.0, "power": {...}} ``` ### Display Preview Stream **GET** `/api/v3/stream/display` -Server-Sent Events (SSE) stream for real-time display preview images. - -**Response**: SSE stream with base64-encoded images ``` -data: {"image": "base64_data_here", "timestamp": 1234567890.123} +data: {"timestamp": 1234567890.123, "width": 128, "height": 32, "image": "base64_data_here"} ``` ### Service Logs Stream **GET** `/api/v3/stream/logs` -Server-Sent Events (SSE) stream for real-time service logs. +Each event carries the latest journal lines for `ledmatrix` and +`ledmatrix-web` as one text block: -**Response**: SSE stream ``` -data: {"level": "INFO", "message": "Plugin loaded", "timestamp": 1234567890.123} +data: {"timestamp": 1234567890.123, "logs": "2025-01-15T10:30:00+0000 host python[123]: ..."} ``` --- @@ -1480,26 +1829,17 @@ data: {"level": "INFO", "message": "Plugin loaded", "timestamp": 1234567890.123} ### Get Logs -**GET** `/api/v3/logs?limit=100&level=INFO` +**GET** `/api/v3/logs` -Get recent log entries. - -**Query Parameters**: -- `limit` (optional): Maximum number of log entries (default: 100) -- `level` (optional): Filter by log level (DEBUG, INFO, WARNING, ERROR) +The last 100 journal lines for `ledmatrix.service` and +`ledmatrix-web.service`, as one text block. Takes no parameters. **Response**: ```json { "status": "success", "data": { - "logs": [ - { - "level": "INFO", - "message": "Plugin loaded: football-scoreboard", - "timestamp": 1234567890.123 - } - ] + "logs": "2025-01-15T10:30:00+0000 host python[123]: Plugin loaded: football-scoreboard\n..." } } ``` @@ -1512,31 +1852,54 @@ Get recent log entries. **GET** `/api/v3/errors/summary` -Aggregated counts of recent errors across all plugins and core -components, used by the web UI's error indicator. +Aggregated counts, detected patterns and recent errors across plugins and +core components. ### Get Plugin Errors **GET** `/api/v3/errors/plugin/` -Recent errors for a specific plugin. +Error health and statistics for one plugin. ### Clear Errors **POST** `/api/v3/errors/clear` -Clear the in-memory error aggregator. +Clear error records older than `max_age_hours` (default 24, 1-8760). +Returns `data.cleared_count`. + +```json +{ + "max_age_hours": 24 +} +``` --- -## Health +## Health and Status ### Health Check **GET** `/api/v3/health` -Lightweight liveness check used by the WiFi monitor and external -monitoring tools. +Health of the web interface, display service, config file, plugin system and +display snapshot. `data.status` is `healthy` or `degraded`, with +`data.services` and `data.checks`. + +### Hardware Status + +**GET** `/api/v3/hardware/status` + +LED matrix initialization result written by the display service at startup. +Before the service has written it, `data` is +`{"ok": null, "error": "Display service not yet started"}`. + +### Sync Status + +**GET** `/api/v3/sync/status` + +Live multi-display sync status from the display process; before it has +written one, `data` is `{"role", "port", "state": "starting"}` from config. --- @@ -1546,44 +1909,97 @@ monitoring tools. **GET** `/api/v3/config/dim-schedule` -Read the dim/power schedule that automatically reduces brightness or -turns the display off at configured times. +Read the schedule that lowers brightness at configured times. + +**Response**: +```json +{ + "status": "success", + "data": { + "enabled": true, + "dim_brightness": 30, + "mode": "per-day", + "days": { + "monday": { "enabled": true, "start_time": "20:00", "end_time": "07:00" }, + "tuesday": { ... } + } + } +} +``` + +In `global` mode, `start_time` and `end_time` sit at the top level instead +of `days`. ### Update Dim Schedule **POST** `/api/v3/config/dim-schedule` -Update the dim schedule. Body matches the structure returned by GET. +Replace the dim schedule. `dim_brightness` is 0-100 (default 30). In +`per-day` mode the days can be sent either as the `days` object that GET +returns, or as the web form's flat fields (`monday_enabled`, +`monday_start`, `monday_end`, ...). A day that is not sent counts as +enabled with default times `20:00`-`07:00`; at least one day must be +enabled. + +--- + +## Integrations + +### MQTT Bridge + +**GET** `/api/v3/integrations/mqtt-bridge` + +Home Assistant MQTT bridge service state and settings: `data.service`, +`data.config_exists`, `data.config_path`, `data.config` (password +omitted), `data.password_set`, `data.env_override_prefix`. + +**PUT** `/api/v3/integrations/mqtt-bridge/config` + +Write `integrations/mqtt_bridge/bridge_config.json`. Only the keys you send +change. The password is write-only: omit `mqtt_password` to keep it, send a +value to replace it, or send `"clear_password": true`. A password with +`mqtt_tls` off is refused unless `allow_insecure_mqtt` is true. Returns +`data.password_set` and `data.restart_required` (the bridge must be +restarted to pick up changes). See +[integrations/mqtt_bridge/README.md](../integrations/mqtt_bridge/README.md). --- ## Plugin-specific endpoints -A handful of endpoints belong to individual built-in or shipped plugins. +A handful of endpoints belong to individual plugins. ### Calendar **GET** `/api/v3/plugins/calendar/list-calendars` -List the calendars available on the authenticated Google account. -Used by the calendar plugin's config UI. +List the calendars on the authenticated Google account. Used by the calendar +plugin's config UI. Returns `calendars` at the top level. The upload and +authenticate endpoints are under [Plugins](#upload-calendar-credentials). ### Of The Day **POST** `/api/v3/plugins/of-the-day/json/upload` -Upload a JSON data file for the Of-The-Day plugin's category data. +Upload JSON data files (multipart field `files`) as Of-The-Day categories. +Returns `uploaded_files` and `total_files` at the top level. **POST** `/api/v3/plugins/of-the-day/json/delete` -Delete a previously uploaded Of-The-Day data file. +Delete an uploaded data file. + +```json +{ + "file_id": "category_name" +} +``` ### Plugin Static Assets **GET** `/api/v3/plugins//static/` -Serve a static asset (image, font, etc.) from a plugin's directory. -Used internally by the web UI to render plugin previews and icons. +Serve a static file from a plugin's directory. Used internally by the web UI +to render plugin previews and icons. --- @@ -1613,40 +2029,77 @@ Download and install the Pixlet binary on the Pi. **GET** `/api/v3/starlark/apps//config` — get app config schema **PUT** `/api/v3/starlark/apps//config` — update app config **POST** `/api/v3/starlark/apps//render` — render app to a frame -**POST** `/api/v3/starlark/apps//toggle` — enable/disable app +**POST** `/api/v3/starlark/apps//toggle` — enable/disable app (`{"enabled": bool}`; omit to flip) ### Repository (Tronbyt community apps) -**GET** `/api/v3/starlark/repository/categories` — browse categories -**GET** `/api/v3/starlark/repository/browse?category=` — browse apps -**POST** `/api/v3/starlark/repository/install` — install an app from the -community repository +**GET** `/api/v3/starlark/repository/categories` — list categories +**GET** `/api/v3/starlark/repository/browse` — every app with metadata (filtering happens client-side; cached for 2 hours) +**POST** `/api/v3/starlark/repository/install` — install an app: `{"app_id": "...", "render_interval": 300, "display_duration": 15}` ### Upload custom app **POST** `/api/v3/starlark/upload` -Upload a custom Starlark `.star` file as a new app. +Upload a custom Starlark `.star` file as a new app. Multipart fields: `file` +(required, max 5 MB), `name`, `app_id`, `render_interval`, +`display_duration`. + +### Editor + +A Pixlet editing session for one app. The display is stopped while a session +runs. + +**GET** `/api/v3/starlark/editor/apps` — apps the editor can open (`data.apps`, `data.apps_dir`, `data.pixlet_available`) +**GET** `/api/v3/starlark/editor/status` — `data.running`, plus `app_id`, `port`, `pid`, `started_at`, `timeout`, `seconds_remaining`, `host_bound` while running +**POST** `/api/v3/starlark/editor/start` — `{"app_id": "...", "timeout": 1800, "port": 8080}` (`timeout` and `port` optional) +**POST** `/api/v3/starlark/editor/stop` — end the session and restart the display + +--- + +## Skins + +**GET** `/api/v3/skins` + +Installed scoreboard skins (optional `?plugin_id=` filter). Skins are not +supported by the current scoreboard plugins, so the response carries +`data.supported: false` and a `data.message`; clients must not offer these +as selectable. See [SKIN_SYSTEM.md](SKIN_SYSTEM.md). --- ## Error Responses -All endpoints may return error responses in the following format: +Errors use one of two shapes. Most endpoints answer: ```json { "status": "error", "message": "Error description", - "error_code": "ERROR_CODE", "details": "Additional error details (optional)" } ``` +Endpoints built on the structured error helper add a code and category: + +```json +{ + "status": "error", + "error_code": "CONFIG_SAVE_FAILED", + "error_category": "configuration", + "message": "Error description", + "details": "optional", + "context": { }, + "suggested_fixes": [ ] +} +``` + **Common HTTP Status Codes**: - `200`: Success - `400`: Bad Request (invalid parameters) +- `403`: Forbidden (e.g. deleting a system font) - `404`: Not Found (resource doesn't exist) +- `408`: Timed out (plugin actions, auth scripts) - `500`: Internal Server Error - `503`: Service Unavailable (feature not available) @@ -1657,4 +2110,3 @@ All endpoints may return error responses in the following format: - [Plugin API Reference](PLUGIN_API_REFERENCE.md) - API for plugin developers - [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - Complete plugin development guide - [Web Interface README](../web_interface/README.md) - Web interface documentation - From 04aaf39ae2a9077745d636379dadc781b16ea596 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:52:58 -0400 Subject: [PATCH 02/13] docs(fonts): point plugins at plugin_manager.font_manager; drop removed overrides UI FONT_MANAGER.md told plugins to read display_manager.font_manager, which does not exist, so a plugin following it failed to load with AttributeError. The shared FontManager lives on the PluginManager and BasePlugin._get_font_manager() returns it (with a fallback for harnesses). Also removes the Fonts-tab override workflow and element-override panels that #566 deleted, from FONT_MANAGER.md and WEB_INTERFACE_GUIDE.md. Co-Authored-By: Claude Opus 5 --- docs/FONT_MANAGER.md | 71 ++++++++++++++++++++----------------- docs/WEB_INTERFACE_GUIDE.md | 15 ++++---- 2 files changed, 45 insertions(+), 41 deletions(-) diff --git a/docs/FONT_MANAGER.md b/docs/FONT_MANAGER.md index e6691dd7..3678e955 100644 --- a/docs/FONT_MANAGER.md +++ b/docs/FONT_MANAGER.md @@ -12,10 +12,28 @@ The enhanced FontManager provides comprehensive font management for the LEDMatrix application with support for: - Manager font registration and detection - Plugin font management -- Manual font overrides via web interface +- Programmatic per-element font overrides - Performance monitoring and caching - Dynamic font discovery +## Getting the FontManager + +There is one shared FontManager per display process. The display controller +creates it and hands it to the `PluginManager`, so a plugin reaches it +through its `plugin_manager`: + +```python +class MyPlugin(BasePlugin): + def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager): + super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) + self.font_manager = self._get_font_manager() +``` + +`BasePlugin._get_font_manager()` returns `plugin_manager.font_manager`, or a +standalone FontManager when none is available (test harnesses, mocks). +`DisplayManager` has **no** `font_manager` attribute — +`display_manager.font_manager` raises `AttributeError`. + ## Architecture ### Manager-Centric Design @@ -40,8 +58,9 @@ Manager requests font → Check manual overrides → Apply manager choice → Ca from src.font_manager import FontManager class MyManager: - def __init__(self, config, display_manager, cache_manager): - self.font_manager = display_manager.font_manager # Access shared FontManager + def __init__(self, config, display_manager, cache_manager, plugin_manager): + self.display_manager = display_manager + self.font_manager = plugin_manager.font_manager # Shared FontManager self.manager_id = "my_manager" def display(self): @@ -80,8 +99,9 @@ class MyManager: ```python class AdvancedManager: - def __init__(self, config, display_manager, cache_manager): - self.font_manager = display_manager.font_manager + def __init__(self, config, display_manager, cache_manager, plugin_manager): + self.display_manager = display_manager + self.font_manager = plugin_manager.font_manager self.manager_id = "advanced_manager" # Define your font specifications @@ -152,19 +172,13 @@ font = self.font_manager.resolve_font( > URIs documented below are resolved relative to the plugin's > install directory. > -> The **Fonts** tab in the web UI that lists detected -> manager-registered fonts is still a **placeholder -> implementation** — fonts that managers register through -> `register_manager_font()` do not yet appear there. The -> programmatic per-element override workflow described in -> [Manual Font Overrides](#manual-font-overrides) below -> (`set_override()` / `remove_override()` / the -> `config/font_overrides.json` store) **does** work today and is -> the supported way to override a font for an element until the -> Fonts tab is wired up. If you can't wait and need a workaround -> right now, you can also just load the font directly with PIL -> (or `freetype-py` for BDF) inside your plugin's `manager.py` -> and skip the override system entirely. +> The web UI's **Fonts** tab lists, uploads, previews and deletes the +> font files in `assets/fonts/`. It does not show fonts registered +> through `register_manager_font()` and has no override editor (the +> override panels and `/api/v3/fonts/overrides` endpoints were removed). +> The programmatic override workflow in +> [Manual Font Overrides](#manual-font-overrides) below still works. +> Let users pick fonts through your plugin's own config schema. ### Plugin Font Registration @@ -200,10 +214,10 @@ In your plugin's `manifest.json`: ### Using Plugin Fonts ```python -class PluginManager: - def __init__(self, config, display_manager, cache_manager, plugin_id): - self.font_manager = display_manager.font_manager - self.plugin_id = plugin_id +class MyPlugin(BasePlugin): + def __init__(self, plugin_id, config, display_manager, cache_manager, plugin_manager): + super().__init__(plugin_id, config, display_manager, cache_manager, plugin_manager) + self.font_manager = self._get_font_manager() def display(self): # Use plugin font (automatically namespaced) @@ -219,17 +233,8 @@ class PluginManager: ## Manual Font Overrides -Users can override any font through the web interface: - -1. Navigate to **Fonts** tab -2. View **Detected Manager Fonts** to see what's currently in use -3. In **Element Overrides** section: - - Select the element (e.g., "nfl.live.score") - - Choose a different font family - - Choose a different size - - Click **Add Override** - -Overrides are stored in `config/font_overrides.json` and persist across restarts. +Overrides are set in code (there is no web UI or REST endpoint for them). +They are stored in `config/font_overrides.json` and persist across restarts. ### Programmatic Overrides diff --git a/docs/WEB_INTERFACE_GUIDE.md b/docs/WEB_INTERFACE_GUIDE.md index 10ef082b..64b51ba3 100644 --- a/docs/WEB_INTERFACE_GUIDE.md +++ b/docs/WEB_INTERFACE_GUIDE.md @@ -206,11 +206,11 @@ Manage fonts for your display: - See font previews - Check font sizes and styles -**Font Overrides:** -- Overrides are set per display *element* (e.g. a specific score or - clock text element), not per plugin -- Override default font choices for individual elements -- Preview font changes +**Font Preview:** +- Render sample text in any TTF/OTF font at a chosen size + +Fonts used by a plugin are chosen in that plugin's own settings tab; the +Fonts tab has no per-element override editor. **Delete Fonts:** - Remove unused fonts @@ -327,9 +327,8 @@ The web interface is built on a REST API that you can access programmatically: http://your-pi-ip:5000/api/v3 ``` -The API blueprint mounts at `/api/v3` (see -`web_interface/app.py:199`). All endpoints below are relative to that -base. +The API blueprint (`web_interface/blueprints/api_v3/`) is registered at +`/api/v3` in `web_interface/app.py`. **Common Endpoints:** - `GET /api/v3/config/main` — Get main configuration From e70ccdfb6c47dad0b486ff4d19f41e9416f054ac Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:53:49 -0400 Subject: [PATCH 03/13] docs(store): search via /plugins/store/list?query=; send Content-Type on registry curls /plugins/store/search does not exist (404) and the list endpoint reads query, not q. The registry guide's curl examples omitted the JSON Content-Type, so the handlers saw an empty body and answered 400. Co-Authored-By: Claude Opus 5 --- docs/PLUGIN_REGISTRY_SETUP_GUIDE.md | 3 +++ docs/PLUGIN_STORE_GUIDE.md | 9 ++++----- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md index d1426979..87457c14 100644 --- a/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md +++ b/docs/PLUGIN_REGISTRY_SETUP_GUIDE.md @@ -103,6 +103,7 @@ All plugins can be installed through the LEDMatrix web interface: Or via API: ```bash curl -X POST http://your-pi-ip:5000/api/v3/plugins/install \ + -H "Content-Type: application/json" \ -d '{"plugin_id": "clock-simple"}' ``` @@ -153,6 +154,7 @@ Before submitting, ensure your plugin: ```bash # Install via URL on your Pi curl -X POST http://your-pi:5000/api/v3/plugins/install-from-url \ + -H "Content-Type: application/json" \ -d '{"repo_url": "https://github.com/you/ledmatrix-your-plugin"}' ``` @@ -312,6 +314,7 @@ git push # 2. Review using VERIFICATION.md checklist # 3. Test installation: curl -X POST http://pi:5000/api/v3/plugins/install-from-url \ + -H "Content-Type: application/json" \ -d '{"repo_url": "https://github.com/contributor/plugin"}' # 4. If approved, merge PR diff --git a/docs/PLUGIN_STORE_GUIDE.md b/docs/PLUGIN_STORE_GUIDE.md index 421c99de..fbec2a36 100644 --- a/docs/PLUGIN_STORE_GUIDE.md +++ b/docs/PLUGIN_STORE_GUIDE.md @@ -131,13 +131,13 @@ else: **Via REST API:** ```bash # Search by query -curl "http://your-pi-ip:5000/api/v3/plugins/store/search?q=hockey" +curl "http://your-pi-ip:5000/api/v3/plugins/store/list?query=hockey" # Filter by category -curl "http://your-pi-ip:5000/api/v3/plugins/store/search?category=sports" +curl "http://your-pi-ip:5000/api/v3/plugins/store/list?category=sports" # Filter by tags -curl "http://your-pi-ip:5000/api/v3/plugins/store/search?tags=nhl&tags=hockey" +curl "http://your-pi-ip:5000/api/v3/plugins/store/list?tags=nhl&tags=hockey" ``` **Via Python:** @@ -351,8 +351,7 @@ All API endpoints return JSON with this structure: | Method | Endpoint | Description | |--------|----------|-------------| -| GET | `/api/v3/plugins/store/list` | List all plugins in store | -| GET | `/api/v3/plugins/store/search` | Search for plugins | +| GET | `/api/v3/plugins/store/list` | List plugins in store; `?query=`, `?category=`, `?tags=` search and filter | | GET | `/api/v3/plugins/installed` | List installed plugins | | POST | `/api/v3/plugins/install` | Install from registry | | POST | `/api/v3/plugins/install-from-url` | Install from GitHub URL | From 3a0d126fdd8e054969f8ed7d76825376c5c25b9a Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:55:24 -0400 Subject: [PATCH 04/13] docs: replace stale API names, line numbers and the api_v3.py path - ADVANCED_FEATURES: StreamManager methods that exist (get_next_segment, take_next_group, refresh, advance_cycle, ...), and the real on-demand status envelope ({status, data: {state, service}}) - app.py:199 / :144 / :607-619 line citations and web_interface/blueprints/api_v3.py (now a package) replaced with file and function names in ADVANCED_FEATURES, CONFIG_DEBUGGING, PLUGIN_ARCHITECTURE_SPEC, PLUGIN_QUICK_REFERENCE, PLUGIN_CONFIGURATION_TABS, TROUBLESHOOTING and web_interface/README - CONFIG_DEBUGGING: partial /config/main saves change only sent keys; use /config/raw/main to replace the file; describe where validation runs - TROUBLESHOOTING: clear_cache.py needs --clear-all (no args only prints usage) Co-Authored-By: Claude Opus 5 --- docs/ADVANCED_FEATURES.md | 40 ++++++++++++++++++++++--------- docs/CONFIG_DEBUGGING.md | 19 +++++++++++---- docs/PLUGIN_ARCHITECTURE_SPEC.md | 3 ++- docs/PLUGIN_CONFIGURATION_TABS.md | 3 ++- docs/PLUGIN_QUICK_REFERENCE.md | 3 ++- docs/TROUBLESHOOTING.md | 4 ++-- web_interface/README.md | 13 ++++++---- 7 files changed, 59 insertions(+), 26 deletions(-) diff --git a/docs/ADVANCED_FEATURES.md b/docs/ADVANCED_FEATURES.md index d620fef7..a942b152 100644 --- a/docs/ADVANCED_FEATURES.md +++ b/docs/ADVANCED_FEATURES.md @@ -377,9 +377,16 @@ Vegas mode consists of four core components working together to provide smooth 1 5. Compose into continuous stream with separators **Key Methods:** -- `get_stream_content()` - Returns current stream content as PIL Image -- `advance_stream(pixels)` - Advances stream by N pixels -- `refresh_stream()` - Regenerates stream from current plugins +- `get_next_segment()` - Returns the next buffered `ContentSegment` (or `None`) +- `take_next_group(count=None, offscreen_only=False)` - Hands over the next + slice of the rotation as `(plugin_id, images)` groups +- `get_grouped_content_for_composition()` - Buffered images grouped by plugin +- `mark_plugin_updated(plugin_id)` / `process_updates()` - Refresh one + plugin's segment in place when its data changes +- `refresh()` - Re-read the plugin list and config +- `advance_cycle()` - Clear the active buffer when a scroll cycle completes + +(`src/vegas_mode/stream_manager.py`) #### 3. PluginAdapter @@ -552,7 +559,8 @@ time when something is active. ### REST API Reference -The API is mounted at `/api/v3` (`web_interface/app.py:199`). +The API is mounted at `/api/v3` (the `api_v3` blueprint, registered in +`web_interface/app.py`). Full details: [REST_API_REFERENCE.md](REST_API_REFERENCE.md#display-control). #### Start On-Demand Display @@ -608,20 +616,30 @@ curl http://localhost:5000/api/v3/display/on-demand/status # Response: { - "active": true, - "plugin_id": "weather", - "mode": "weather", - "remaining": 25.5, - "pinned": false, - "status": "active" + "status": "success", + "data": { + "state": { + "active": true, + "plugin_id": "weather", + "mode": "weather", + "duration": 30, + "pinned": false, + "status": "running", + "last_updated": 1234567890.1 + }, + "service": {"active": true, "returncode": 0, "stdout": "active", "stderr": ""} + } } ``` +When nothing is running on demand, `data.state` is +`{"active": false, "status": "idle", "last_updated": null}`. + > There is no public Python on-demand API. The display controller's > on-demand machinery is internal — drive it through the REST endpoints > above (or the web UI buttons). The API handlers > (`start_on_demand_display()` / `stop_on_demand_display()` in -> `web_interface/blueprints/api_v3.py`) write a request into the cache +> `web_interface/blueprints/api_v3/display.py`) write a request into the cache > manager under the `display_on_demand_request` key, which > `DisplayController._poll_on_demand_requests()` > (`src/display_controller.py`) picks up. A separate diff --git a/docs/CONFIG_DEBUGGING.md b/docs/CONFIG_DEBUGGING.md index 2454f755..4adae1fb 100644 --- a/docs/CONFIG_DEBUGGING.md +++ b/docs/CONFIG_DEBUGGING.md @@ -250,14 +250,21 @@ WARNING - Plugin ID 'Football-Scoreboard' may conflict with 'football-scoreboard ## Checking Configuration via API -The API blueprint mounts at `/api/v3` (`web_interface/app.py:144`). +The API blueprint (`web_interface/blueprints/api_v3/`) is registered at +`/api/v3` in `web_interface/app.py`. ```bash -# Get full main config (includes all plugin sections) +# Get full main config (includes all plugin sections; credential-named +# fields are blanked in the response) curl http://localhost:5000/api/v3/config/main -# Save updated main config +# Change some settings: only the keys you send are changed curl -X POST http://localhost:5000/api/v3/config/main \ + -H "Content-Type: application/json" \ + -d '{"timezone": "America/Chicago", "brightness": 80}' + +# Replace config.json wholesale (advanced) +curl -X POST http://localhost:5000/api/v3/config/raw/main \ -H "Content-Type: application/json" \ -d @new-config.json @@ -269,8 +276,10 @@ curl "http://localhost:5000/api/v3/plugins/config?plugin_id=football-scoreboard" ``` > There is no dedicated `/config/plugin/` or `/config/validate` -> endpoint — config validation runs server-side automatically when you -> POST to `/config/main` or `/plugins/config`. See +> endpoint. `POST /plugins/config` validates against the plugin's schema +> and rejects an invalid config with `400`; `POST /config/main` checks the +> individual fields it knows (display hardware values, durations, Vegas +> and sync settings). See > [REST_API_REFERENCE.md](REST_API_REFERENCE.md) for the full list. ## Backup and Recovery diff --git a/docs/PLUGIN_ARCHITECTURE_SPEC.md b/docs/PLUGIN_ARCHITECTURE_SPEC.md index 3c8ddb6f..8df78fd9 100644 --- a/docs/PLUGIN_ARCHITECTURE_SPEC.md +++ b/docs/PLUGIN_ARCHITECTURE_SPEC.md @@ -8,7 +8,8 @@ > - Code paths reference `web_interface_v2.py`; the current web UI is > `web_interface/app.py` with v3 Blueprint-based templates. > - The example Flask routes use `/api/plugins/*`; the real API -> blueprint is mounted at `/api/v3` (`web_interface/app.py:199`). +> blueprint (`web_interface/blueprints/api_v3/`) is mounted at `/api/v3` +> in `web_interface/app.py`. > - The default plugin location is `plugin-repos/` (configurable via > `plugin_system.plugins_directory`), not `./plugins/`. > - Example imports use `src/plugin_system/base_classes/*_plugin.py`; diff --git a/docs/PLUGIN_CONFIGURATION_TABS.md b/docs/PLUGIN_CONFIGURATION_TABS.md index ed5de33d..9fcecef4 100644 --- a/docs/PLUGIN_CONFIGURATION_TABS.md +++ b/docs/PLUGIN_CONFIGURATION_TABS.md @@ -6,7 +6,8 @@ > in the "Implementation Details" section below still reference the > pre-v3 file layout (`web_interface_v2.py`, `templates/index_v2.html`). > The current implementation lives in `web_interface/app.py`, -> `web_interface/blueprints/api_v3.py`, and `web_interface/templates/v3/`. +> `web_interface/blueprints/api_v3/` (plugin config handlers in +> `plugins.py`), and `web_interface/templates/v3/`. > The user-facing description (Overview, Features, Form Generation > Process) is still accurate. diff --git a/docs/PLUGIN_QUICK_REFERENCE.md b/docs/PLUGIN_QUICK_REFERENCE.md index 850fbb22..09c54b76 100644 --- a/docs/PLUGIN_QUICK_REFERENCE.md +++ b/docs/PLUGIN_QUICK_REFERENCE.md @@ -127,7 +127,8 @@ git push origin v1.0.0 ### REST API -The API is mounted at `/api/v3` (`web_interface/app.py:199`). +The API is mounted at `/api/v3` (the `api_v3` blueprint in +`web_interface/blueprints/api_v3/`, registered in `web_interface/app.py`). ```bash # Install plugin from the registry diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index fe3a1a74..db5b3cac 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -273,7 +273,7 @@ sudo systemctl cat ledmatrix-web | grep User 1. **Verify file structure:** ```bash ls -l web_interface/app.py - ls -l web_interface/blueprints/api_v3.py + ls -ld web_interface/blueprints/api_v3/ ls -l web_interface/blueprints/pages_v3.py ``` @@ -531,7 +531,7 @@ sudo systemctl cat ledmatrix-web | grep User ```bash # Clear the cache with the helper script - sudo python3 scripts/utils/clear_cache.py + sudo python3 scripts/utils/clear_cache.py --clear-all # Or remove files manually from the cache dir in use, e.g.: sudo rm -rf /var/cache/ledmatrix/* diff --git a/web_interface/README.md b/web_interface/README.md index b6235ded..9dd13376 100644 --- a/web_interface/README.md +++ b/web_interface/README.md @@ -20,7 +20,9 @@ web_interface/ ├── run.sh # Shell runner script ├── requirements.txt # Python dependencies ├── blueprints/ # Flask blueprints -│ ├── api_v3.py # API endpoints +│ ├── api_v3/ # API endpoints (package: config, display, +│ │ # plugins, system, backup, fonts, misc, +│ │ # wifi, starlark) │ └── pages_v3.py # Page routes ├── templates/ # HTML templates │ └── v3/ @@ -75,7 +77,8 @@ The web interface reads configuration from: ## API Documentation -The V3 API is mounted at `/api/v3/` (`app.py:144`). For the complete +The V3 API is the `api_v3` blueprint, registered at `/api/v3/` in +`app.py`. For the complete list and request/response formats, see [`docs/REST_API_REFERENCE.md`](../docs/REST_API_REFERENCE.md). Quick reference for the most common endpoints: @@ -115,9 +118,9 @@ reference for the most common endpoints: - `POST /api/v3/plugins/store/refresh` - Refresh registry from GitHub ### Real-time Streams (SSE) -SSE stream endpoints are defined directly on the Flask app -(`app.py:607-619` — includes the CSRF exemption and rate-limit hookup -alongside the three route definitions), not on the api_v3 blueprint: +SSE stream endpoints are defined directly on the Flask app in `app.py` +(`stream_stats`, `stream_display`, `stream_logs`, followed by their CSRF +exemption and rate-limit hookup), not on the api_v3 blueprint: - `GET /api/v3/stream/stats` - System statistics stream - `GET /api/v3/stream/display` - Display preview stream - `GET /api/v3/stream/logs` - Service logs stream From df3524305f2379c7bb3c43ddb3cb47f1e4acd864 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:56:12 -0400 Subject: [PATCH 05/13] docs(plugins): one display-size contract: display_manager.width/height CLAUDE.md (#580) says to read display_manager.width/height because matrix is None when hardware init fails; the development guide, the safety-harness doc and two DisplayManager docstrings still recommended matrix.width/height. The bundled starlark-apps plugin read matrix.width unguarded, so its magnify recommendation and frame scaling raised in fallback mode (e.g. after the Pi 5 hardware refusal). Co-Authored-By: Claude Opus 5 --- docs/PLUGIN_DEVELOPMENT_GUIDE.md | 6 ++++-- docs/plugin-safety-harness.md | 3 ++- plugin-repos/starlark-apps/manager.py | 14 +++++++------- src/display_manager.py | 14 +++++++++----- 4 files changed, 22 insertions(+), 15 deletions(-) diff --git a/docs/PLUGIN_DEVELOPMENT_GUIDE.md b/docs/PLUGIN_DEVELOPMENT_GUIDE.md index a9db48c4..b4c8a28d 100644 --- a/docs/PLUGIN_DEVELOPMENT_GUIDE.md +++ b/docs/PLUGIN_DEVELOPMENT_GUIDE.md @@ -3,8 +3,10 @@ This guide explains how to set up a development workflow for plugins that are maintained in separate Git repositories while still being able to test them within the LEDMatrix project. > **Rendering guidance:** plugins should read the display size dynamically -> (`self.display_manager.matrix.width/height`) rather than hardcoding one -> panel. For plugins that want to *scale* their layout to any panel, the +> (`self.display_manager.width/height`) rather than hardcoding one +> panel. Don't read `display_manager.matrix.width/height`: `matrix` is +> `None` when hardware init fails, while the `width`/`height` properties +> fall back to the canvas size. For plugins that want to *scale* their layout to any panel, the > opt-in adaptive layout system ([ADAPTIVE_LAYOUT.md](ADAPTIVE_LAYOUT.md)) > provides the shared helpers — fonts, images, and composite layouts that > scale. Existing plugins keep their classic rendering unless they adopt diff --git a/docs/plugin-safety-harness.md b/docs/plugin-safety-harness.md index eb127f18..5a869ad2 100644 --- a/docs/plugin-safety-harness.md +++ b/docs/plugin-safety-harness.md @@ -10,7 +10,8 @@ plugin without breaking a size or screen you didn't think to test. There is **no fixed set of supported panel sizes** — an RGB matrix build can be any width/height and configuration (square, rectangle, 2×2, 4×4, 8×2, long strips, tall stacks). Plugins are expected to read dimensions dynamically -(`self.display_manager.matrix.width/height`) and lay themselves out +(`self.display_manager.width/height` — not `matrix.width/height`, since +`matrix` is `None` when hardware init fails) and lay themselves out accordingly, so a hardcoded coordinate or unscaled font shows up as a failure here. diff --git a/plugin-repos/starlark-apps/manager.py b/plugin-repos/starlark-apps/manager.py index b3f26118..5a810079 100644 --- a/plugin-repos/starlark-apps/manager.py +++ b/plugin-repos/starlark-apps/manager.py @@ -239,7 +239,7 @@ class StarlarkAppsPlugin(BasePlugin): # Calculate optimal magnification based on display size self.calculated_magnify = self._calculate_optimal_magnify() if self.calculated_magnify > 1: - self.logger.info(f"Display size: {self.display_manager.matrix.width}x{self.display_manager.matrix.height}, " + self.logger.info(f"Display size: {self.display_manager.width}x{self.display_manager.height}, " f"recommended magnify: {self.calculated_magnify}") # Load installed apps @@ -323,8 +323,8 @@ class StarlarkAppsPlugin(BasePlugin): Recommended magnify value (1-8) """ try: - display_width = self.display_manager.matrix.width - display_height = self.display_manager.matrix.height + display_width = self.display_manager.width + display_height = self.display_manager.height # Tronbyte native resolution NATIVE_WIDTH = 64 @@ -362,8 +362,8 @@ class StarlarkAppsPlugin(BasePlugin): Dictionary with recommendation details """ try: - display_width = self.display_manager.matrix.width - display_height = self.display_manager.matrix.height + display_width = self.display_manager.width + display_height = self.display_manager.height NATIVE_WIDTH = 64 NATIVE_HEIGHT = 32 @@ -852,8 +852,8 @@ class StarlarkAppsPlugin(BasePlugin): # Scale frames if needed if self.config.get("scale_output", True): - width = self.display_manager.matrix.width - height = self.display_manager.matrix.height + width = self.display_manager.width + height = self.display_manager.height # Get scaling method from config scale_method_str = self.config.get("scale_method", "nearest") diff --git a/src/display_manager.py b/src/display_manager.py index a0b105fa..3c44ea0c 100644 --- a/src/display_manager.py +++ b/src/display_manager.py @@ -91,9 +91,11 @@ class _LogicalMatrix: """Proxy that reports a logical (per-screen) size for a physical matrix. In double-sided mode the physical panel chain shows N identical copies of a - smaller logical screen. Plugins size themselves from ``matrix.width`` / - ``matrix.height`` (the documented convention, used at 30+ call sites), so - this proxy reports the logical dimensions while delegating every real + smaller logical screen. Plugins size themselves from + ``display_manager.width`` / ``height`` (the documented convention), which + defer to ``matrix.width`` / ``matrix.height`` -- and many older plugins read + ``matrix.width`` directly -- so this proxy reports the logical dimensions + while delegating every real operation — ``CreateFrameCanvas``, ``SwapOnVSync``, ``brightness``, ``Clear`` and so on — to the underlying physical matrix. The duplication itself happens once per frame in :meth:`DisplayManager.update_display`. @@ -682,8 +684,10 @@ class DisplayManager: def render_size(self, width: int, height: Optional[int] = None): """Temporarily present a smaller logical canvas to plugins. - Plugins lay out against ``display_manager.matrix.width`` (and the - ``width``/``height`` properties, which defer to it), so the only way to + Plugins lay out against the ``display_manager.width``/``height`` + properties (which defer to ``matrix.width`` when hardware is present, + and to the canvas when it is not; some older plugins read + ``matrix.width`` directly), so the only way to get a *narrower layout* rather than a cropped one is to tell the plugin the screen is narrower while it renders. Trimming after the fact cannot fix a forecast spread across five columns or a progress bar drawn at From c3465c35f5204d62ab92b854cf1ba99eab88b7d2 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:58:03 -0400 Subject: [PATCH 06/13] docs(deps): describe how plugin dependencies are really installed The guides said the web service runs as root, that installs pick --user from os.geteuid(), and quoted a warning and a PluginManager._install_plugin_dependencies() method that don't exist. The web unit runs as the installing user; store installs go through install_requirements_file() and sudo safe_pip_install.sh (root), with a user-level fallback that says so, and load-time installs run in the display service's own (root) interpreter. Manual paths now use the configured plugins directory (plugin-repos/ by default) instead of plugins/, which store installs no longer use, and install_plugin_dependencies.sh is described as scanning that directory. Co-Authored-By: Claude Opus 5 --- docs/PLUGIN_DEPENDENCY_GUIDE.md | 294 ++++++++-------------- docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md | 159 ++++++------ 2 files changed, 187 insertions(+), 266 deletions(-) diff --git a/docs/PLUGIN_DEPENDENCY_GUIDE.md b/docs/PLUGIN_DEPENDENCY_GUIDE.md index 1388a6fc..5396b0c6 100644 --- a/docs/PLUGIN_DEPENDENCY_GUIDE.md +++ b/docs/PLUGIN_DEPENDENCY_GUIDE.md @@ -2,234 +2,160 @@ ## Overview -The LEDMatrix system has smart dependency installation that adapts based on who is running it. This guide explains how it works and potential pitfalls. +A plugin lists its Python packages in its `requirements.txt`. LEDMatrix +installs them for you when a plugin is installed, updated or loaded. This +guide explains where they end up and what to do when a plugin can't import a +package. -## How It Works +The rule to remember: **packages must be importable by `ledmatrix.service`, +which runs as root.** Anything installed only into another user's +`~/.local/` is invisible to it. -### Execution Context Detection +## Who Runs What -The plugin manager checks if it's running as root: -```python -running_as_root = os.geteuid() == 0 +| Service | Runs as | Set by | +|---------|---------|--------| +| `ledmatrix.service` (display) | `root` | `systemd/ledmatrix.service` | +| `ledmatrix-web.service` (web UI) | the user who ran the installer (e.g. `ledpi`) | `User=__USER__` in `systemd/ledmatrix-web.service`, filled in by `scripts/install/install_service.sh` | + +## How Dependencies Get Installed + +### 1. Installing or updating a plugin from the web UI + +The web interface is not root, so it installs through a narrow sudo helper: + +1. `PluginStoreManager._install_dependencies()` + (`src/plugin_system/store_manager.py`) calls + `install_requirements_file()` (`src/common/permission_utils.py`). +2. That runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh /requirements.txt`. + The helper checks the path is the project's own `requirements.txt` or a + `requirements.txt` under `plugin-repos/` or `plugins/`, then runs + `python3 -m pip install --break-system-packages --ignore-installed -r ...` + **as root**, so the display service can import the packages. +3. The sudoers rule that allows this is written by the installer + (`first_time_install.sh`) or by `scripts/install/configure_web_sudo.sh`. + +If sudo refuses (the rule isn't installed), `install_requirements_file()` +falls back to installing with the web process's own interpreter, as the web +user, and prefixes the pip output with a note like: + +``` +[Root install unavailable (...); installed for the current process's user only. +Packages may not be visible to ledmatrix.service if it runs as a different +user — run scripts/install/configure_web_sudo.sh to fix this.] ``` -Based on this, it chooses the appropriate installation method: +Fix it by running `./scripts/install/configure_web_sudo.sh` as the web +user (not with `sudo`; it asks for your password itself), then +reinstall the plugin (or use the manual install below). -| Running As | Installation Method | Location | Accessible To | -|------------|-------------------|----------|---------------| -| **root** (systemd service) | System-wide (`--break-system-packages`) | `/usr/local/lib/python3.X/dist-packages/` | All users | -| **ledpi** or other user | User-specific (`--user`) | `~/.local/lib/python3.X/site-packages/` | Only that user | +The **Reinstall Plugin Deps** button on the web UI's Tools tab goes +through the same helper for every installed plugin. + +### 2. Loading a plugin + +When a plugin loads, `PluginLoader.install_dependencies()` +(`src/plugin_system/plugin_loader.py`) checks its `requirements.txt`. If the +requirements are already satisfied it does nothing; otherwise it runs +`python3 -m pip install --break-system-packages -r requirements.txt` with the +interpreter of the process doing the loading (retrying with +`--ignore-installed` when a system package without a pip RECORD file is in +the way). + +In `ledmatrix.service` that process is root, so restarting the display +service installs anything missing system-wide: + +```bash +sudo systemctl restart ledmatrix +``` + +If you run `python3 run.py` by hand as a normal user instead, pip cannot +write to the system site-packages and installs into your `~/.local/`. That +works for your manual run but not for the service. ## Common Scenarios -### ✅ Scenario 1: Normal Production Use (Recommended) +### Installing plugins from the web UI (recommended) -**What:** Services running via systemd +Use the **Plugin Manager** tab. Dependencies are installed as root through +the sudo helper and the display service can use them. + +### Running the display manually for debugging ```bash -sudo systemctl start ledmatrix -sudo systemctl start ledmatrix-web +cd ~/LEDMatrix +sudo python3 run.py # same user as the service ``` -- **Runs as:** root (configured in .service files) -- **Installs to:** System-wide -- **Result:** ✅ Works perfectly, all dependencies accessible +Running as your own user works for plugins whose packages are already +installed system-wide, but any *missing* package lands in `~/.local/`. -### ✅ Scenario 2: Web Interface Plugin Installation +### A plugin works when run manually but fails in the service -**What:** Installing/enabling plugins via web interface at `http://pi-ip:5000` +Its packages were installed for your user only. Install them as root (see +below) and restart the service. -- **Web service runs as:** root (ledmatrix-web.service) -- **Installs to:** System-wide -- **Result:** ✅ Works perfectly, systemd service can access them +## Manual Installation -### ✅ Scenario 3: Manual Testing as ledpi (Read-only) - -**What:** Running display manually as ledpi to test/debug +### All plugins ```bash -# As ledpi user -cd /home/ledpi/LEDMatrix -python3 run.py -``` - -- **Runs as:** ledpi -- **Can import:** ✅ System-wide packages (installed by root) -- **Result:** ✅ Works! Can use existing plugins with root-installed dependencies - -### ⚠️ Scenario 4: Manual Plugin Installation as ledpi (Problematic) - -**What:** Enabling a NEW plugin and running manually as ledpi - -```bash -# As ledpi user -cd /home/ledpi/LEDMatrix -# Edit config to enable new plugin -nano config/config.json -# Run display - will try to install new plugin dependencies -python3 run.py -``` - -**What Happens:** -1. Plugin manager runs as `ledpi` -2. Installs dependencies with `--user` flag -3. Dependencies go to `~/.local/lib/python3.X/site-packages/` -4. ⚠️ **Warning logged:** "Installing plugin dependencies for current user (not root)" - -**Problem:** -- When systemd service restarts (as root), it **can't see** `~/.local/` packages -- Plugin will fail to load for the systemd service - -**Solution:** -After testing, restart the service to install dependencies system-wide: -```bash +sudo ~/LEDMatrix/scripts/install_plugin_dependencies.sh sudo systemctl restart ledmatrix ``` -## Best Practices +The script installs every `requirements.txt` found in the plugins directory +configured by `plugin_system.plugins_directory` in `config/config.json` +(default `plugin-repos/`). Run it with `sudo` so the packages are installed +system-wide. -### For Production/Normal Use +### One plugin -1. **Always use the web interface** to install/enable plugins -2. **Or restart the systemd service** after config changes: - ```bash - sudo systemctl restart ledmatrix - ``` - -### For Development/Testing - -1. **Read existing plugins:** Safe to run as `ledpi` - can import system packages -2. **Test new plugins:** Use sudo or restart service to install dependencies: - ```bash - # Option 1: Run as root - sudo python3 run.py - - # Option 2: Install deps manually - sudo pip3 install --break-system-packages -r plugins/my-plugin/requirements.txt - python3 run.py - - # Option 3: Let service install them - sudo systemctl restart ledmatrix - ``` - -## Warning Messages - -### If you see this warning: -``` -Installing plugin dependencies for current user (not root). -These will NOT be accessible to the systemd service. -For production use, install plugins via the web interface or restart the ledmatrix service. -``` - -**What it means:** -- You're running as a non-root user -- Dependencies were installed to your user directory only -- The systemd service won't be able to use this plugin - -**What to do:** ```bash -# Restart the service to install dependencies system-wide +cd ~/LEDMatrix/plugin-repos/PLUGIN-NAME # or your configured plugins directory +sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt sudo systemctl restart ledmatrix ``` +`--no-cache-dir` avoids errors about `/root/.cache/pip` not being writable. + ## Troubleshooting -### Plugin works when I run manually but fails in systemd service - -**Cause:** Dependencies installed to user directory (`~/.local/`) instead of system-wide - -**Fix:** -```bash -# Check where package is installed -pip3 list -v | grep - -# If it shows ~/.local/, reinstall system-wide: -sudo pip3 install --break-system-packages - -# Or just restart the service: -sudo systemctl restart ledmatrix -``` - ### Permission denied when installing dependencies -**If you see errors like:** ``` ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/root/.local' WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable ``` -**Quick Fix - Use the Helper Script:** -```bash -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh -sudo systemctl restart ledmatrix -``` +Use one of the manual installs above (they pass `--no-cache-dir`). -**Manual Fix:** -```bash -# Install dependencies with --no-cache-dir to avoid cache permission issues -cd /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME -sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt -sudo systemctl restart ledmatrix -``` - -**For more detailed troubleshooting, see:** [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md) - -## Architecture Summary - -``` -┌─────────────────────────────────────────────────────────────┐ -│ LEDMatrix Services │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ ledmatrix.service (User=root) │ -│ ledmatrix-web.service (User=root) │ -│ ├── Install dependencies system-wide │ -│ └── Accessible to all users │ -│ │ -├─────────────────────────────────────────────────────────────┤ -│ │ -│ Manual execution as ledpi │ -│ ├── Can READ system-wide packages ✅ │ -│ ├── WRITES go to ~/.local/ ⚠️ │ -│ └── Not accessible to root service │ -│ │ -└─────────────────────────────────────────────────────────────┘ -``` - -## Recommendations - -1. **For end users:** Always use the web interface for plugin management -2. **For developers:** Be aware of the user context when testing -3. **For plugin authors:** Test with `sudo systemctl restart ledmatrix` to ensure dependencies install correctly -4. **For CI/CD:** Always run installation as root or use the service - -## Helper Scripts - -### Install Plugin Dependencies Script - -Located at: `scripts/install_plugin_dependencies.sh` - -This script automatically finds and installs dependencies for all plugins: +### Checking where a package is installed ```bash -# Run as root (recommended for production) -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +# How the service sees it +sudo python3 -c "import package_name; print(package_name.__file__)" -# Make executable if needed -chmod +x /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +# A path under /home//.local/ means it was installed for that user only +python3 -m pip show -f package_name ``` -Features: -- Auto-detects all plugins with requirements.txt -- Uses correct installation method (system-wide vs user) -- Bypasses pip cache to avoid permission issues -- Provides detailed logging and error messages +For more, see the [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TROUBLESHOOTING.md). + +## For Plugin Authors + +1. Keep `requirements.txt` minimal and pin only what you need. +2. Test that it installs the way the Pi will install it: + ```bash + sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt + ``` +3. Note any `apt` packages your plugin needs in its README. ## Files to Reference -- Service configs: `ledmatrix.service`, `ledmatrix-web.service` -- Plugin manager: `src/plugin_system/plugin_manager.py` -- Installation script: `first_time_install.sh` -- Dependency installer: `scripts/install_plugin_dependencies.sh` -- Troubleshooting guide: `PLUGIN_DEPENDENCY_TROUBLESHOOTING.md` - +- Service units: `systemd/ledmatrix.service`, `systemd/ledmatrix-web.service` +- Store installs: `src/plugin_system/store_manager.py` (`_install_dependencies`) +- Root install helper: `src/common/permission_utils.py` (`install_requirements_file`), `scripts/fix_perms/safe_pip_install.sh` +- Load-time installs: `src/plugin_system/plugin_loader.py` (`install_dependencies`) +- Sudo rules: `scripts/install/configure_web_sudo.sh` +- Manual installer: `scripts/install_plugin_dependencies.sh` diff --git a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md index 1f33cb88..a62b9644 100644 --- a/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md +++ b/docs/PLUGIN_DEPENDENCY_TROUBLESHOOTING.md @@ -1,6 +1,7 @@ # Plugin Dependency Installation Troubleshooting -This guide helps resolve issues with automatic plugin dependency installation in the LEDMatrix system. +This guide helps resolve problems installing a plugin's Python packages. For +how installation works, see the [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md). ## Common Error Symptoms @@ -10,109 +11,118 @@ ERROR: Could not install packages due to an OSError: [Errno 13] Permission denie WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable ``` -### Context Mismatch +### Installed for the wrong user +The pip output shown after a web-UI install starts with: ``` -WARNING: Installing plugin dependencies for current user (not root). -These will NOT be accessible to the systemd service. +[Root install unavailable (...); installed for the current process's user only. +Packages may not be visible to ledmatrix.service if it runs as a different +user — run scripts/install/configure_web_sudo.sh to fix this.] ``` +### Plugin fails to load with `ModuleNotFoundError` +The display service can't see a package the plugin needs. + ## Root Cause -Plugin dependencies must be installed in a context accessible to the LEDMatrix systemd service, which runs as root. Permission errors typically occur when: +Plugin packages must be importable by `ledmatrix.service`, which runs as +root. The web interface (`ledmatrix-web.service`) runs as the user who +installed LEDMatrix, so it installs through a sudo helper +(`scripts/fix_perms/safe_pip_install.sh`). Problems usually come from: -1. The pip cache directory has incorrect permissions -2. The process tries to install to user directories without proper permissions -3. Environment variables (like HOME) are not set correctly for the service context +1. The sudoers rule for that helper missing, so the web UI installed the + packages for its own user only +2. Running `python3 run.py` by hand as a normal user, which installs missing + packages into `~/.local/` +3. pip's cache directory not being writable for root ## Solutions -### Solution 1: Use the Manual Installation Script (Recommended) - -We provide a helper script that handles dependency installation correctly: +### Solution 1: Restore the sudo rule, then reinstall ```bash -# Run as root to install system-wide (for production) -sudo /home/ledpi/LEDMatrix/scripts/install_plugin_dependencies.sh +cd ~/LEDMatrix +./scripts/install/configure_web_sudo.sh # as the web user, not with sudo +``` -# After installation, restart the service +Then reinstall the plugin from the **Plugin Manager** tab, or click +**Reinstall Plugin Deps** on the **Tools** tab. + +### Solution 2: Install every plugin's dependencies from the terminal + +```bash +sudo ~/LEDMatrix/scripts/install_plugin_dependencies.sh sudo systemctl restart ledmatrix ``` -This script: -- Detects all plugins with requirements.txt files -- Installs dependencies with correct permissions -- Uses `--no-cache-dir` to avoid cache permission issues -- Provides detailed logging for troubleshooting +The script finds each `requirements.txt` in the plugins directory set by +`plugin_system.plugins_directory` in `config/config.json` (default +`plugin-repos/`), installs with `--no-cache-dir`, and reports what it found. -### Solution 2: Manual Installation per Plugin - -If you need to install dependencies for a specific plugin: +### Solution 3: Install one plugin's dependencies ```bash -# Navigate to the plugin directory -cd /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME +# Your configured plugins directory; plugin-repos/ by default +cd ~/LEDMatrix/plugin-repos/PLUGIN-NAME -# Install as root (system-wide) -sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt +sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt -# Restart the service sudo systemctl restart ledmatrix ``` -### Solution 3: Fix Cache Directory Permissions +### Solution 4: Let the display service install them -If you specifically have cache permission issues: +When a plugin loads, the display service installs any missing requirements +itself, as root: + +```bash +sudo systemctl restart ledmatrix +sudo journalctl -u ledmatrix -f # watch for "Installing dependencies for plugin ..." +``` + +### Solution 5: Fix pip cache permissions ```bash # Option A: Skip the cache (recommended) -sudo pip3 install --no-cache-dir --break-system-packages -r requirements.txt +sudo python3 -m pip install --no-cache-dir --break-system-packages -r requirements.txt -# Option B: Fix cache permissions (if needed) +# Option B: Fix cache permissions sudo mkdir -p /root/.cache/pip sudo chown -R root:root /root/.cache sudo chmod -R 755 /root/.cache ``` -### Solution 4: Install via Web Interface - -The web interface handles dependency installation correctly in the service context: - -1. Access the web interface (`http://ledpi:5000` or `http://your-pi-ip:5000`) -2. Open the **Plugin Manager** tab (use the **Plugin Store** section to - find the plugin, or **Install from GitHub**) -3. Install the plugin through the web UI -4. The system automatically handles dependency installation in the - service context (which has the right permissions) - ## Prevention ### For Plugin Developers -When creating plugins with dependencies: - 1. **Keep requirements minimal**: Only include essential packages -2. **Test installation**: Verify your requirements.txt works with: +2. **Test installation** the way the Pi does it: ```bash - sudo pip3 install --break-system-packages --no-cache-dir -r requirements.txt + sudo python3 -m pip install --break-system-packages --no-cache-dir -r requirements.txt ``` 3. **Document dependencies**: Note any system packages needed (via apt) ### For Users -1. **Use web interface**: Install plugins via the web UI when possible -2. **Install as root**: When using SSH/terminal, use sudo for plugin installations -3. **Restart service**: After manual installations, restart the ledmatrix service +1. **Use the web interface** to install plugins +2. **Use sudo** for installs from SSH/terminal +3. **Restart the service** after manual installations ## Technical Details -### How Dependency Installation Works +### Where installs happen -The `PluginManager._install_plugin_dependencies()` method: - -1. Detects if running as root using `os.geteuid() == 0` -2. If root: Uses system-wide installation with `--break-system-packages --no-cache-dir` -3. If not root: Uses user installation with `--user --break-system-packages --no-cache-dir` -4. The `--no-cache-dir` flag prevents cache-related permission issues +- **Web UI install/update:** `PluginStoreManager._install_dependencies()` + → `install_requirements_file()` in `src/common/permission_utils.py`, which + runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh `. + The helper only accepts the project's `requirements.txt` or one under + `plugin-repos/` or `plugins/`, and runs + `pip install --break-system-packages --ignore-installed` as root. If sudo + refuses, it falls back to a pip install as the web user and says so. +- **Plugin load:** `PluginLoader.install_dependencies()` in + `src/plugin_system/plugin_loader.py` skips satisfied requirements and + otherwise runs `pip install --break-system-packages` with the loading + process's interpreter — root in `ledmatrix.service`. ### Why `--break-system-packages`? @@ -120,26 +130,20 @@ Debian 12+ (Bookworm) and Raspberry Pi OS based on it implement PEP 668, which p ### Service Context -The ledmatrix.service runs as: -- **User**: root -- **WorkingDirectory**: /home/ledpi/LEDMatrix -- **Python**: /usr/bin/python3 +- `ledmatrix.service` runs as **root** with `/usr/bin/python3` +- `ledmatrix-web.service` runs as **the installing user** -Dependencies must be installed in root's Python environment or system-wide to be accessible. +Dependencies must be installed system-wide (as root) to be visible to the +display service. ## Checking Installation -Verify dependencies are installed correctly: - ```bash # Check as root (how the service sees it) -sudo python3 -c "import package_name" +sudo python3 -c "import package_name; print(package_name.__file__)" -# List installed packages -pip3 list - -# Check specific package -pip3 show package_name +# A path under /home//.local/ means a user-only install +python3 -m pip show -f package_name ``` ## Getting Help @@ -151,19 +155,11 @@ If you continue to experience issues: sudo journalctl -u ledmatrix -f ``` -2. Check pip logs (created by manual script): +2. Verify the plugin manifest and requirements (default plugins directory + shown): ```bash - cat /tmp/pip_install_*.log - ``` - -3. Verify plugin manifest is correct: - ```bash - cat /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME/manifest.json - ``` - -4. Check plugin requirements: - ```bash - cat /home/ledpi/LEDMatrix/plugins/PLUGIN-NAME/requirements.txt + cat ~/LEDMatrix/plugin-repos/PLUGIN-NAME/manifest.json + cat ~/LEDMatrix/plugin-repos/PLUGIN-NAME/requirements.txt ``` ## Related Documentation @@ -171,4 +167,3 @@ If you continue to experience issues: - [Plugin Dependency Guide](PLUGIN_DEPENDENCY_GUIDE.md) - [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md) - [Troubleshooting](TROUBLESHOOTING.md) - From 9378901054bd95626897a0ab8d0836ad4604c8eb Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 08:58:37 -0400 Subject: [PATCH 07/13] docs(install): what install_service.sh installs; verify script port; no sudo for --help install_service.sh installs and starts ledmatrix, ledmatrix-web and the update-verify units, not only ledmatrix.service (systemd/README.md, README.md). MIGRATION_GUIDE presented 'sudo install_service.sh --help' as a harmless check; it now shows --help without sudo and warns what a real run does. SSH_UNAVAILABLE_AFTER_INSTALL: verify_installation.sh checks the web interface on port 5000. Co-Authored-By: Claude Opus 5 --- README.md | 8 +++++--- docs/MIGRATION_GUIDE.md | 7 +++++-- docs/SSH_UNAVAILABLE_AFTER_INSTALL.md | 4 +++- systemd/README.md | 8 +++++--- 4 files changed, 18 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 0a77ce1f..cca9503d 100644 --- a/README.md +++ b/README.md @@ -834,9 +834,11 @@ sudo ./scripts/install/install_service.sh The script will: - Detect your user account and home directory -- Install the service file with the correct paths -- Enable the service to start on boot -- Start the service immediately +- Install `ledmatrix.service` (display, runs as root), `ledmatrix-web.service` + (web interface, runs as your user) and the `ledmatrix-update-verify` units, + with the correct paths +- Enable them to start on boot +- Start them immediately ### Managing the Service diff --git a/docs/MIGRATION_GUIDE.md b/docs/MIGRATION_GUIDE.md index b64269b9..c6b17c94 100644 --- a/docs/MIGRATION_GUIDE.md +++ b/docs/MIGRATION_GUIDE.md @@ -59,9 +59,12 @@ sudo ./scripts/install/install_service.sh After updating your scripts, verify they still work: ```bash -# Test installation scripts (if needed) +# Check the installation scripts are at their new paths ls scripts/install/*.sh -sudo ./scripts/install/install_service.sh --help +./scripts/install/install_service.sh --help # prints usage only +# Note: running install_service.sh for real (with sudo, no --help) +# reinstalls, enables and restarts ledmatrix.service, ledmatrix-web.service +# and the update-verify units. # Test permission scripts ls scripts/fix_perms/*.sh diff --git a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md index bc065eb1..4978b4c6 100644 --- a/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md +++ b/docs/SSH_UNAVAILABLE_AFTER_INSTALL.md @@ -158,9 +158,11 @@ This script will check: - Python dependencies - Configuration files - File permissions -- Web interface availability +- Web interface availability (`ledmatrix-web` listening on port 5000) - Network connectivity +Once it passes, the web interface is at `http://:5000`. + ## Quick Reference Commands ```bash diff --git a/systemd/README.md b/systemd/README.md index a3a8cd5d..cdc65ec6 100644 --- a/systemd/README.md +++ b/systemd/README.md @@ -52,9 +52,11 @@ This directory contains systemd service unit files for LEDMatrix services. ## Installation These service files are installed by the installation scripts in `scripts/install/`: -- `install_service.sh` installs `ledmatrix.service` -- `install_web_service.sh` installs `ledmatrix-web.service` and the - `ledmatrix-update-verify` service and path units +- `install_service.sh` installs `ledmatrix.service`, `ledmatrix-web.service` + and the `ledmatrix-update-verify` service and path units, then enables and + starts them +- `install_web_service.sh` installs only `ledmatrix-web.service` and the + `ledmatrix-update-verify` units - `install_wifi_monitor.sh` installs `ledmatrix-wifi-monitor.service` - `install_dns_fix.sh` installs `ledmatrix-dns-fix.service` (opt-in, not run by the normal installer) From 4d9c52f182ff31fdf7b07f30f6d98754a1263b1e Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 09:03:52 -0400 Subject: [PATCH 08/13] fix(dev): link-github links plugins from the ledmatrix-plugins monorepo link-github cloned https://github.com/ChuckBuilds/ledmatrix-.git, and those per-plugin repositories no longer exist: official plugins are directories in the ledmatrix-plugins monorepo. It now clones (or pulls) the monorepo once into the dev directory, finds plugins/, plugins/ledmatrix- or the plugin whose manifest id is , and links it under its manifest id. With an explicit repo URL it still links a single-repository plugin as before. dev_plugins.json: github_user is honoured again (monorepo owner, e.g. a fork), plus plugins_repo and plugins_branch; github_pattern, which was documented but never read, is dropped and warned about. Ships dev_plugins.json.example and git-ignores dev_plugins.json, both of which the guide promised. Reading JSON falls back to python3 when jq is missing (get_plugin_id silently returned nothing without jq). update/status/list find the git checkout above a monorepo plugin directory (its .git is not in the plugin dir), and update pulls a shared checkout once. status no longer exits 1 when nothing is broken. Docs: PLUGIN_DEVELOPMENT_GUIDE (quick start, link-github, configuration, workflow, store integration, hello-world link, submission), and the nonexistent scripts/git-hooks/pre-push-plugin-version and scripts/bump_plugin_version.py replaced with the real rule: bump the manifest version and run update_registry.py. scripts/dev/README.md and CLAUDE.md updated to match. Co-Authored-By: Claude Opus 5 --- .gitignore | 7 +- CLAUDE.md | 2 +- dev_plugins.json.example | 6 + docs/PLUGIN_DEVELOPMENT_GUIDE.md | 191 ++++++++++++---------- scripts/dev/README.md | 10 ++ scripts/dev/dev_plugin_setup.sh | 268 ++++++++++++++++++++++--------- 6 files changed, 319 insertions(+), 165 deletions(-) create mode 100644 dev_plugins.json.example diff --git a/.gitignore b/.gitignore index afb9b3df..3b59b144 100644 --- a/.gitignore +++ b/.gitignore @@ -39,11 +39,12 @@ htmlcov/ # Cache directory (root level only, not src/cache which is source code) /cache/ -# Development plugins directory -# Plugins are managed as separate repositories via multi-root workspace -# See docs/MULTI_ROOT_WORKSPACE_SETUP.md for details +# Development plugins directory: symlinks into a ledmatrix-plugins checkout +# See docs/PLUGIN_DEVELOPMENT_GUIDE.md and docs/MULTI_ROOT_WORKSPACE_SETUP.md plugins/* !plugins/.gitkeep +# Local settings for scripts/dev/dev_plugin_setup.sh (template: dev_plugins.json.example) +/dev_plugins.json # Binary files and backups bin/pixlet/ diff --git a/CLAUDE.md b/CLAUDE.md index 764b637a..1a3cd990 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,7 @@ `config.get(...)`, never a separate accessor ## Dev Workflow -- Link a plugin for development: `./scripts/dev/dev_plugin_setup.sh link-github ` (or `link `); symlinks land in `plugins/` — set `plugin_system.plugins_directory` to `plugins` so discovery picks them up +- Link a plugin for development: `./scripts/dev/dev_plugin_setup.sh link-github ` clones the `ledmatrix-plugins` monorepo into `~/.ledmatrix-dev-plugins/` and links its `plugins/` under the manifest id (add a repo URL for a plugin with its own repo; or `link `); symlinks land in `plugins/` — set `plugin_system.plugins_directory` to `plugins` so discovery picks them up. Fork/location overrides: `dev_plugins.json` (from `dev_plugins.json.example`) - Browser preview without the display loop: `python3 scripts/dev_server.py` → http://localhost:5001 - Full display in emulator mode: `python3 run.py -e` (or `EMULATOR=true python3 run.py`) - Validate one plugin headlessly: `python3 scripts/check_plugin.py --plugin ` diff --git a/dev_plugins.json.example b/dev_plugins.json.example new file mode 100644 index 00000000..64f26abb --- /dev/null +++ b/dev_plugins.json.example @@ -0,0 +1,6 @@ +{ + "dev_plugins_dir": "~/.ledmatrix-dev-plugins", + "github_user": "ChuckBuilds", + "plugins_repo": "ledmatrix-plugins", + "plugins_branch": "main" +} diff --git a/docs/PLUGIN_DEVELOPMENT_GUIDE.md b/docs/PLUGIN_DEVELOPMENT_GUIDE.md index b4c8a28d..491bd8ad 100644 --- a/docs/PLUGIN_DEVELOPMENT_GUIDE.md +++ b/docs/PLUGIN_DEVELOPMENT_GUIDE.md @@ -45,28 +45,45 @@ The solution uses **symbolic links** to connect plugin repositories to the `plug ## Quick Start -### 1. Link a Plugin from GitHub +Official plugins all live in one repository, +[ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins), with +one directory per plugin under `plugins/` (there are no per-plugin +`ledmatrix-` repositories). The helper script links a plugin directory +from a checkout of that monorepo into LEDMatrix's `plugins/` directory. -The easiest way to link a plugin that's already on GitHub: +### 1. Link an Official Plugin ```bash -./scripts/dev/dev_plugin_setup.sh link-github music +./scripts/dev/dev_plugin_setup.sh link-github football-scoreboard ``` This will: -- Clone `https://github.com/ChuckBuilds/ledmatrix-music.git` to `~/.ledmatrix-dev-plugins/ledmatrix-music` -- Create a symbolic link from `plugins/music` to the cloned repository -- Validate that the plugin has a proper `manifest.json` +- Clone `https://github.com/ChuckBuilds/ledmatrix-plugins.git` to + `~/.ledmatrix-dev-plugins/ledmatrix-plugins` (or `git pull` it if it is + already there) +- Find `plugins/football-scoreboard` in it (also accepted: + `plugins/ledmatrix-`, or a plugin whose manifest `id` is the name) +- Validate that it has a `manifest.json` +- Create a symbolic link named after the plugin's manifest id, e.g. + `plugins/football-scoreboard` → `~/.ledmatrix-dev-plugins/ledmatrix-plugins/plugins/football-scoreboard` -### 2. Link a Local Plugin Repository +`link-github music` finds the monorepo's `plugins/ledmatrix-music` directory +and links it into LEDMatrix as `plugins/ledmatrix-music`, because +`ledmatrix-music` is that plugin's manifest id. -If you already have a plugin repository cloned locally: +To work from your fork of the monorepo, set `github_user` in +`dev_plugins.json` (see [Configuration](#configuration)). + +### 2. Link a Local Plugin Directory + +If you already have the monorepo (or a third-party plugin repository) cloned +locally: ```bash -./scripts/dev/dev_plugin_setup.sh link music ../ledmatrix-music +./scripts/dev/dev_plugin_setup.sh link hello-world ../ledmatrix-plugins/plugins/hello-world ``` -This creates a symlink from `plugins/music` to your local repository path. +This creates a symlink from `plugins/hello-world` to that directory. ### 3. Check Status @@ -79,13 +96,17 @@ See which plugins are linked and their git status: ### 4. Work on Your Plugin ```bash -cd plugins/music # Actually editing the linked repository -# Make your changes +cd plugins/football-scoreboard # Actually editing the monorepo checkout +# Make your changes, then bump "version" in manifest.json git add . -git commit -m "feat: add new feature" -git push origin main +git commit -m "feat(football-scoreboard): add new feature" +git push # to your fork, then open a PR against ledmatrix-plugins ``` +In the monorepo, every plugin change must bump `version` in the plugin's +`manifest.json` and run `python update_registry.py`, or users won't receive +the update. + ### 5. Update Plugins Pull latest changes from remote: @@ -118,7 +139,7 @@ Links a local plugin repository to the plugins directory. **Example:** ```bash -./scripts/dev/dev_plugin_setup.sh link football-scoreboard ../ledmatrix-football-scoreboard +./scripts/dev/dev_plugin_setup.sh link football-scoreboard ../ledmatrix-plugins/plugins/football-scoreboard ``` **Notes:** @@ -131,23 +152,25 @@ Links a local plugin repository to the plugins directory. Clones a plugin from GitHub and links it. **Arguments:** -- `plugin-name`: The name of the plugin (will be the directory name in `plugins/`) -- `repo-url`: (Optional) Full GitHub repository URL. If omitted, constructs from pattern: `https://github.com/ChuckBuilds/ledmatrix-.git` +- `plugin-name`: Without `repo-url`, the plugin to link from the monorepo: a + directory under `plugins/` (`` or `ledmatrix-`) or a manifest + id. The link is named after the plugin's manifest id. With `repo-url`, the + name of the link in `plugins/`. +- `repo-url`: (Optional) A plugin that has its own repository (e.g. a + third-party plugin). The repository root is linked. **Examples:** ```bash -# Auto-construct URL from plugin name -./scripts/dev/dev_plugin_setup.sh link-github music +# Official plugin, from the ledmatrix-plugins monorepo +./scripts/dev/dev_plugin_setup.sh link-github stocks -# Use explicit URL -./scripts/dev/dev_plugin_setup.sh link-github stocks https://github.com/ChuckBuilds/ledmatrix-stocks.git - -# Link from a different GitHub user +# Third-party plugin with its own repository ./scripts/dev/dev_plugin_setup.sh link-github custom-plugin https://github.com/OtherUser/custom-plugin.git ``` **Notes:** - Repositories are cloned to `~/.ledmatrix-dev-plugins/` by default (configurable) +- The monorepo is cloned once and shared by every plugin you link from it - If the repository already exists, it will be updated with `git pull` instead of re-cloning - The cloned repository is preserved when you unlink the plugin @@ -219,30 +242,28 @@ Updates plugin(s) by running `git pull` in their repositories. ### Custom Development Directory -By default, GitHub repositories are cloned to `~/.ledmatrix-dev-plugins/`. You can customize this by creating a `dev_plugins.json` file: +By default, GitHub repositories are cloned to `~/.ledmatrix-dev-plugins/` +and official plugins come from `ChuckBuilds/ledmatrix-plugins`. To change +either, copy `dev_plugins.json.example` (in the LEDMatrix root) to +`dev_plugins.json` and edit it. `dev_plugins.json` is git-ignored. ```json { - "dev_plugins_dir": "/path/to/your/dev/plugins", - "github_user": "ChuckBuilds", - "github_pattern": "ledmatrix-", - "plugins": { - "music": { - "source": "github", - "url": "https://github.com/ChuckBuilds/ledmatrix-music.git", - "branch": "main" - } - } + "dev_plugins_dir": "~/.ledmatrix-dev-plugins", + "github_user": "your-github-user", + "plugins_repo": "ledmatrix-plugins", + "plugins_branch": "main" } ``` -**Configuration options:** +**Configuration options** (all optional): - `dev_plugins_dir`: Where to clone GitHub repositories (default: `~/.ledmatrix-dev-plugins`) -- `github_user`: Default GitHub username for auto-constructing URLs -- `github_pattern`: Pattern for repository names (default: `ledmatrix-`) -- `plugins`: Plugin definitions (optional, for future auto-discovery features) +- `github_user`: Owner of the plugin monorepo that `link-github ` clones — set it to use your fork (default: `ChuckBuilds`) +- `plugins_repo`: Name of that monorepo (default: `ledmatrix-plugins`) +- `plugins_branch`: Branch to clone it at (default: the repository's default branch). Only applies when the clone is first made. -**Note:** Copy `dev_plugins.json.example` to `dev_plugins.json` and customize it. The `dev_plugins.json` file is git-ignored. +`github_pattern` from older versions of this guide is no longer used (the +script warns if it is set). ## Development Workflow @@ -250,43 +271,46 @@ By default, GitHub repositories are cloned to `~/.ledmatrix-dev-plugins/`. You c 1. **Link your plugin for development:** ```bash - ./scripts/dev/dev_plugin_setup.sh link-github music + ./scripts/dev/dev_plugin_setup.sh link-github clock-simple ``` 2. **Test in LEDMatrix:** ```bash - # Run LEDMatrix with your plugin - python run.py + # Run LEDMatrix with your plugin (emulator shown) + python3 run.py -e ``` 3. **Make changes:** ```bash - cd plugins/music + cd plugins/clock-simple # Edit files... # Test changes... ``` -4. **Commit to plugin repository:** +4. **Commit to the plugin repository:** ```bash - cd plugins/music # This is actually your repo + cd plugins/clock-simple # This is inside your monorepo checkout + # bump "version" in manifest.json, then from the monorepo root: + # python update_registry.py git add . - git commit -m "feat: add new feature" - git push origin main + git commit -m "feat(clock-simple): add new feature" + git push ``` 5. **Update from remote (if needed):** ```bash - ./scripts/dev/dev_plugin_setup.sh update music + ./scripts/dev/dev_plugin_setup.sh update clock-simple ``` 6. **When done developing:** ```bash - ./scripts/dev/dev_plugin_setup.sh unlink music + ./scripts/dev/dev_plugin_setup.sh unlink clock-simple ``` ### Working with Multiple Plugins -You can have multiple plugins linked simultaneously: +You can have multiple plugins linked simultaneously. Plugins linked from the +monorepo share one checkout: ```bash ./scripts/dev/dev_plugin_setup.sh link-github music @@ -296,7 +320,7 @@ You can have multiple plugins linked simultaneously: # Check status of all ./scripts/dev/dev_plugin_setup.sh status -# Update all at once +# Update all at once (the shared monorepo checkout is pulled once) ./scripts/dev/dev_plugin_setup.sh update ``` @@ -411,7 +435,7 @@ If you have conflicts when updating: 1. **Manually resolve in the plugin repository:** ```bash - cd ~/.ledmatrix-dev-plugins/ledmatrix-music + cd ~/.ledmatrix-dev-plugins/ledmatrix-plugins git pull # Resolve conflicts... git add . @@ -472,18 +496,19 @@ You can mix local and GitHub plugins: The development workflow is separate from the plugin store installation: -- **Plugin Store:** Installs plugins to `plugins/` as regular directories -- **Development Setup:** Links plugin repositories as symlinks +- **Plugin Store:** Installs plugins as regular directories in the configured + plugins directory (`plugin-repos/` by default) +- **Development Setup:** Links plugin directories as symlinks in `plugins/` -If you install a plugin via the store, you can still link it for development: +The plugin loader scans only one directory, so while developing set +`plugin_system.plugins_directory` to `plugins` (see the note at the top of +this guide). If `plugins/` already holds a regular directory of the same +name, `link`/`link-github` offers to rename it to +`.backup.` before linking. -```bash -# Store installs to plugins/music (regular directory) -# Link for development (will prompt to replace) -./scripts/dev/dev_plugin_setup.sh link-github music -``` - -When you unlink, the directory is removed. If you want to switch back to the store version, re-install it via the plugin store. +`unlink` removes only the symlink. To switch back to the store version, set +`plugins_directory` back to `plugin-repos` (or reinstall the plugin from the +store). ## API Reference @@ -527,7 +552,7 @@ Want to create and share your own plugin? Here's everything you need to know. - [Advanced Plugin Development](ADVANCED_PLUGIN_DEVELOPMENT.md) - Patterns and examples 2. **Start with a template**: - - Use the [Hello World plugin](https://github.com/ChuckBuilds/ledmatrix-hello-world) as a starting point + - Use the [Hello World plugin](https://github.com/ChuckBuilds/ledmatrix-plugins/tree/main/plugins/hello-world) as a starting point - Or fork an existing plugin and modify it 3. **Follow the plugin structure**: @@ -591,24 +616,16 @@ Your plugin must: ### Versioning Best Practices - **Use semantic versioning**: `MAJOR.MINOR.PATCH` (e.g., `1.2.3`) -- **GitHub as source of truth**: the plugin store resolves versions in this - order: GitHub Releases → GitHub Tags → manifest from branch → git commit hash -- **Automatic version bumping**: install the self-contained pre-push hook in - your plugin repo and patch versions bump themselves on push (a git tag - `v{version}` is created and `manifest.json` staged automatically): - - ```bash - # From your plugin repository directory - cp /path/to/LEDMatrix/scripts/git-hooks/pre-push-plugin-version .git/hooks/pre-push - chmod +x .git/hooks/pre-push - ``` - - Set `SKIP_TAG=1` in the environment to skip auto-tagging for one push. -- **Manual versioning**: only needed for major/minor bumps, CI pipelines that - bypass hooks, or forks without the hook — use - `scripts/bump_plugin_version.py`. -- **Registry stores no versions**: `plugins.json` holds only metadata (name, - description, repo URL). +- **Bump `version` in `manifest.json` by hand** for every change you ship. + There is no automatic version-bump hook or bump script. +- **Official (monorepo) plugins**: after bumping the manifest, run + `python update_registry.py` in the `ledmatrix-plugins` checkout. It copies + each manifest's version into `plugins.json` as `latest_version`, which is + what the store compares installed versions against. Without it, users + won't be offered the update. +- **Plugins in their own repository**: still bump the manifest `version`, + so users can see which version they run; tagging releases (`v1.2.3`) to + match is a good habit. ### Submitting to Official Registry @@ -620,12 +637,14 @@ To have your plugin added to the official plugin store: - Follows best practices - Tested on Raspberry Pi hardware -2. **Create GitHub repository**: - - Repository name: `ledmatrix-` - - Public repository - - Proper README.md with installation instructions +2. **Choose where it lives** (see `SUBMISSION.md` in + [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins)): + - **In the monorepo (preferred):** fork ledmatrix-plugins, add + `plugins//`, and open a pull request + - **In your own public repository** (conventionally + `ledmatrix-`), with a README that covers installation -3. **Contact maintainers**: +3. **Contact maintainers** (own-repository plugins): - Open a GitHub issue in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) repository - Or reach out on Discord: https://discord.gg/uW36dVAtcT - Include: Repository URL, plugin description, why it's useful diff --git a/scripts/dev/README.md b/scripts/dev/README.md index 3768f17e..cec5a631 100644 --- a/scripts/dev/README.md +++ b/scripts/dev/README.md @@ -12,9 +12,19 @@ This directory contains scripts and utilities for development and testing. ### Plugin Development Setup ```bash +# Official plugin: clones ChuckBuilds/ledmatrix-plugins (once) and links +# its plugins/ into plugins/ ./scripts/dev/dev_plugin_setup.sh link-github + +# Plugin with its own repository +./scripts/dev/dev_plugin_setup.sh link-github ``` +Set `plugin_system.plugins_directory` to `plugins` so the loader finds the +links. To use a fork or another clone location, copy +`dev_plugins.json.example` to `dev_plugins.json`. Details: +[docs/PLUGIN_DEVELOPMENT_GUIDE.md](../../docs/PLUGIN_DEVELOPMENT_GUIDE.md). + ### Running Emulator ```bash ./scripts/dev/run_emulator.sh diff --git a/scripts/dev/dev_plugin_setup.sh b/scripts/dev/dev_plugin_setup.sh index f9f6363b..1d4d1d39 100755 --- a/scripts/dev/dev_plugin_setup.sh +++ b/scripts/dev/dev_plugin_setup.sh @@ -10,8 +10,14 @@ PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" PLUGINS_DIR="$PROJECT_ROOT/plugins" CONFIG_FILE="$PROJECT_ROOT/dev_plugins.json" DEFAULT_DEV_DIR="$HOME/.ledmatrix-dev-plugins" -GITHUB_USER="ChuckBuilds" -GITHUB_PATTERN="ledmatrix-" +# Official plugins live in one monorepo: /, one +# directory per plugin under plugins/. Both can be overridden in +# dev_plugins.json (e.g. to work from a fork). +DEFAULT_GITHUB_USER="ChuckBuilds" +DEFAULT_PLUGINS_REPO="ledmatrix-plugins" +GITHUB_USER="$DEFAULT_GITHUB_USER" +PLUGINS_REPO="$DEFAULT_PLUGINS_REPO" +PLUGINS_BRANCH="" # Colors for output RED='\033[0;31m' @@ -37,18 +43,55 @@ log_error() { echo -e "${RED}[ERROR]${NC} $1" } +# Print a top-level string field of a JSON file, or nothing if it is absent. +# Uses jq when installed, else python3. +json_field() { + local file="$1" + local key="$2" + if command -v jq >/dev/null 2>&1; then + jq -r --arg k "$key" '.[$k] // empty | select(type == "string")' "$file" 2>/dev/null || true + elif command -v python3 >/dev/null 2>&1; then + python3 - "$file" "$key" <<'PY' 2>/dev/null || true +import json, sys +try: + with open(sys.argv[1], encoding="utf-8") as f: + value = json.load(f).get(sys.argv[2]) +except Exception: + value = None +if isinstance(value, str): + print(value) +PY + fi +} + # Load configuration file load_config() { + DEV_PLUGINS_DIR="$DEFAULT_DEV_DIR" if [[ -f "$CONFIG_FILE" ]]; then - DEV_PLUGINS_DIR=$(jq -r '.dev_plugins_dir // "'"$DEFAULT_DEV_DIR"'"' "$CONFIG_FILE" 2>/dev/null || echo "$DEFAULT_DEV_DIR") - # Expand ~ in path - DEV_PLUGINS_DIR="${DEV_PLUGINS_DIR/#\~/$HOME}" - else - DEV_PLUGINS_DIR="$DEFAULT_DEV_DIR" + local value + value=$(json_field "$CONFIG_FILE" dev_plugins_dir) + [[ -n "$value" ]] && DEV_PLUGINS_DIR="$value" + value=$(json_field "$CONFIG_FILE" github_user) + [[ -n "$value" ]] && GITHUB_USER="$value" + value=$(json_field "$CONFIG_FILE" plugins_repo) + [[ -n "$value" ]] && PLUGINS_REPO="$value" + value=$(json_field "$CONFIG_FILE" plugins_branch) + [[ -n "$value" ]] && PLUGINS_BRANCH="$value" + if [[ -n "$(json_field "$CONFIG_FILE" github_pattern)" ]]; then + log_warn "dev_plugins.json: github_pattern is no longer used (official plugins are in the $PLUGINS_REPO monorepo)" + fi fi + # Expand ~ in path + DEV_PLUGINS_DIR="${DEV_PLUGINS_DIR/#\~/$HOME}" mkdir -p "$DEV_PLUGINS_DIR" } +# Top level of the git checkout containing a path, or nothing. +# A monorepo plugin is a subdirectory, so its .git is not in the plugin dir. +git_root_of() { + git -C "$1" rev-parse --show-toplevel 2>/dev/null || true +} + # Validate plugin structure validate_plugin() { local plugin_path="$1" @@ -63,7 +106,7 @@ validate_plugin() { get_plugin_id() { local plugin_path="$1" if [[ -f "$plugin_path/manifest.json" ]]; then - jq -r '.id // empty' "$plugin_path/manifest.json" 2>/dev/null || echo "" + json_field "$plugin_path/manifest.json" id fi } @@ -176,50 +219,104 @@ clone_from_github() { return 0 } +# Clone a repository into DEV_PLUGINS_DIR, or update the existing clone. +# Prints the clone's path on stdout (log output goes to stderr). +ensure_clone() { + local repo_url="$1" + local branch="${2:-}" + local repo_name + repo_name=$(basename "$repo_url" .git) + local target_dir="$DEV_PLUGINS_DIR/$repo_name" + + if [[ -d "$target_dir" ]]; then + log_info "Repository already exists at $target_dir" >&2 + if [[ -d "$target_dir/.git" ]]; then + log_info "Updating repository..." >&2 + (cd "$target_dir" && git pull --rebase) >&2 || true + fi + else + if ! clone_from_github "$repo_url" "$target_dir" "$branch" >&2; then + return 1 + fi + fi + echo "$target_dir" +} + +# Find a plugin's directory inside a monorepo clone: plugins/, +# plugins/ledmatrix-, or the directory whose manifest id is . +find_monorepo_plugin() { + local repo_dir="$1" + local name="$2" + local candidate + for candidate in "$repo_dir/plugins/$name" "$repo_dir/plugins/ledmatrix-$name"; do + if [[ -f "$candidate/manifest.json" ]]; then + echo "$candidate" + return 0 + fi + done + for candidate in "$repo_dir"/plugins/*/; do + candidate="${candidate%/}" + [[ -f "$candidate/manifest.json" ]] || continue + if [[ "$(get_plugin_id "$candidate")" == "$name" ]]; then + echo "$candidate" + return 0 + fi + done + return 1 +} + # Link plugin from GitHub link_github_plugin() { - local plugin_name="$1" + local plugin_name="${1:-}" local repo_url="${2:-}" - + if [[ -z "$plugin_name" ]]; then log_error "Usage: $0 link-github [repo-url]" exit 1 fi - + load_config - - # Construct repo URL if not provided - if [[ -z "$repo_url" ]]; then - repo_url="https://github.com/${GITHUB_USER}/${GITHUB_PATTERN}${plugin_name}.git" - log_info "Using default GitHub URL: $repo_url" - fi - - # Determine target directory name from URL - local repo_name=$(basename "$repo_url" .git) - local target_dir="$DEV_PLUGINS_DIR/$repo_name" - - # Check if already cloned - if [[ -d "$target_dir" ]]; then - log_info "Repository already exists at $target_dir" - if [[ -d "$target_dir/.git" ]]; then - log_info "Updating repository..." - (cd "$target_dir" && git pull --rebase) || true - fi - else - # Clone the repository - if ! clone_from_github "$repo_url" "$target_dir"; then + + if [[ -n "$repo_url" ]]; then + # A plugin with its own repository (e.g. a third-party plugin): the + # repository root is the plugin. + local target_dir + if ! target_dir=$(ensure_clone "$repo_url"); then exit 1 fi + if ! validate_plugin "$target_dir"; then + log_error "Cloned repository does not appear to be a valid plugin" + exit 1 + fi + link_plugin "$plugin_name" "$target_dir" + return fi - - # Validate plugin structure - if ! validate_plugin "$target_dir"; then - log_error "Cloned repository does not appear to be a valid plugin" + + # Official plugins: clone the monorepo once, link plugins/ from it. + repo_url="https://github.com/${GITHUB_USER}/${PLUGINS_REPO}.git" + log_info "Using plugin monorepo: $repo_url" + local repo_dir + if ! repo_dir=$(ensure_clone "$repo_url" "$PLUGINS_BRANCH"); then exit 1 fi - - # Link the plugin - link_plugin "$plugin_name" "$target_dir" + + local plugin_dir + if ! plugin_dir=$(find_monorepo_plugin "$repo_dir" "$plugin_name"); then + log_error "No plugin named '$plugin_name' in $repo_dir/plugins" + log_info "Plugins are the directory names under $repo_dir/plugins, or their manifest ids" + exit 1 + fi + + # Link under the manifest id: that is the name the plugin loader and + # config.json use, and it can differ from the directory name + # (plugins/ledmatrix-music has id ledmatrix-music, not music). + local link_name + link_name=$(get_plugin_id "$plugin_dir") + [[ -n "$link_name" ]] || link_name=$(basename "$plugin_dir") + if [[ "$link_name" != "$plugin_name" ]]; then + log_info "Linking as '$link_name' (the plugin's manifest id)" + fi + link_plugin "$link_name" "$plugin_dir" } # Unlink a plugin @@ -274,7 +371,7 @@ list_plugins() { echo " → $target" # Check git status if it's a git repo - if [[ -d "$target/.git" ]]; then + if [[ -n "$(git_root_of "$target")" ]]; then local branch=$(cd "$target" && git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown") local status=$(cd "$target" && git status --porcelain 2>/dev/null | head -1) if [[ -n "$status" ]]; then @@ -327,7 +424,7 @@ check_status() { echo -e "${GREEN}✓${NC} ${BLUE}$plugin_name${NC}" echo " Path: $target" - if [[ -d "$target/.git" ]]; then + if [[ -n "$(git_root_of "$target")" ]]; then local branch=$(cd "$target" && git rev-parse --abbrev-ref HEAD 2>/dev/null || echo "unknown") local remote=$(cd "$target" && git remote get-url origin 2>/dev/null || echo "no remote") local commits_behind=$(cd "$target" && git rev-list --count HEAD..@{upstream} 2>/dev/null || echo "0") @@ -360,9 +457,13 @@ check_status() { done echo "Summary:" - echo " ${GREEN}Clean: $clean_count${NC}" - echo " ${YELLOW}Needs attention: $dirty_count${NC}" - [[ $broken_count -gt 0 ]] && echo -e " ${RED}Broken: $broken_count${NC}" + echo -e " ${GREEN}Clean: $clean_count${NC}" + echo -e " ${YELLOW}Needs attention: $dirty_count${NC}" + # An if, not `[[ ]] &&`: as the function's last command a false test made + # `status` exit 1 whenever nothing was broken. + if [[ $broken_count -gt 0 ]]; then + echo -e " ${RED}Broken: $broken_count${NC}" + fi } # Update plugin(s) @@ -384,39 +485,46 @@ update_plugins() { fi local target=$(get_symlink_target "$plugin_name") - - if [[ ! -d "$target/.git" ]]; then + local root + root=$(git_root_of "$target") + + if [[ -z "$root" ]]; then log_error "Plugin repository is not a git repository: $target" exit 1 fi - - log_info "Updating $plugin_name from $target" - (cd "$target" && git pull --rebase) + + log_info "Updating $plugin_name from $root" + (cd "$root" && git pull --rebase) log_success "Updated $plugin_name" else - # Update all linked plugins + # Update all linked plugins. Plugins linked from the monorepo share one + # checkout, which is pulled once. log_info "Updating all linked plugins..." local updated=0 local failed=0 - + local pulled_roots=" " + for item in "$PLUGINS_DIR"/*; do [[ -e "$item" ]] || continue [[ -d "$item" ]] || continue - + local name=$(basename "$item") [[ "$name" =~ ^\.|^_ ]] && continue - + if is_symlink "$item"; then local target=$(get_symlink_target "$name") - if [[ -d "$target/.git" ]]; then - log_info "Updating $name..." - if (cd "$target" && git pull --rebase); then - log_success "Updated $name" - updated=$((updated + 1)) - else - log_error "Failed to update $name" - failed=$((failed + 1)) - fi + local root + root=$(git_root_of "$target") + [[ -n "$root" ]] || continue + [[ "$pulled_roots" == *" $root "* ]] && continue + pulled_roots="$pulled_roots$root " + log_info "Updating $root (for $name)..." + if (cd "$root" && git pull --rebase); then + log_success "Updated $root" + updated=$((updated + 1)) + else + log_error "Failed to update $root" + failed=$((failed + 1)) fi fi done @@ -439,7 +547,12 @@ Commands: link-github [repo-url] Clone and link a plugin from GitHub - If repo-url is not provided, uses: https://github.com/${GITHUB_USER}/${GITHUB_PATTERN}.git + Without repo-url: clones (or updates) the official plugin monorepo, + https://github.com/${DEFAULT_GITHUB_USER}/${DEFAULT_PLUGINS_REPO}.git, and links its + plugins/ (or plugins/ledmatrix-) under the + plugin's manifest id + With repo-url: clones a plugin that has its own repository and links + the repository root unlink Remove symlink for a plugin (preserves repository) @@ -458,25 +571,30 @@ Commands: Show this help message Examples: - # Link a local plugin - $0 link music ../ledmatrix-music - - # Link from GitHub (auto-detects URL) - $0 link-github music - - # Link from GitHub with custom URL - $0 link-github stocks https://github.com/ChuckBuilds/ledmatrix-stocks.git - + # Link an official plugin from the monorepo + $0 link-github football-scoreboard + + # Link a plugin from a local monorepo checkout + $0 link hello-world ../ledmatrix-plugins/plugins/hello-world + + # Link a third-party plugin from its own repository + $0 link-github my-plugin https://github.com/OtherUser/ledmatrix-my-plugin.git + # Check status $0 status - + # Update all plugins $0 update Configuration: - Create dev_plugins.json in project root to customize: + Copy dev_plugins.json.example to dev_plugins.json (git-ignored) to customize: - dev_plugins_dir: Where to clone GitHub repos (default: ~/.ledmatrix-dev-plugins) - - plugins: Plugin definitions (optional, for auto-discovery) + - github_user: Owner of the plugin monorepo, e.g. your fork (default: ${DEFAULT_GITHUB_USER}) + - plugins_repo: Name of the plugin monorepo (default: ${DEFAULT_PLUGINS_REPO}) + - plugins_branch: Branch to clone the monorepo at (default: its default branch) + + Symlinks are created in plugins/. Set plugin_system.plugins_directory to + "plugins" in config/config.json so the plugin loader discovers them. EOF } From f2241f393881edb8f3e92403383ba6cc852b383e Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 09:05:00 -0400 Subject: [PATCH 09/13] docs(scripts): monorepo workspace layout; fix_perms and install READMEs MULTI_ROOT_WORKSPACE_SETUP described one sibling repository per plugin; setup_plugin_repos.py links ../ledmatrix-plugins/plugins/* into plugin-repos/ and update_plugin_repos.py pulls only the monorepo, and the workspace file opens LEDMatrix plus ../ledmatrix-plugins. scripts/fix_perms/README.md listed cache directories fix_cache_permissions.sh never touches and a 'ledmatrix' service user that doesn't exist (also in scripts/install/README.md); adds safe_pip_install.sh. install/README: install_service.sh installs the web and update-verify units too. Co-Authored-By: Claude Opus 5 --- docs/MULTI_ROOT_WORKSPACE_SETUP.md | 177 +++++++++++++---------------- scripts/fix_perms/README.md | 22 ++-- scripts/install/README.md | 14 ++- 3 files changed, 105 insertions(+), 108 deletions(-) diff --git a/docs/MULTI_ROOT_WORKSPACE_SETUP.md b/docs/MULTI_ROOT_WORKSPACE_SETUP.md index 2fc0a833..56bbab2f 100644 --- a/docs/MULTI_ROOT_WORKSPACE_SETUP.md +++ b/docs/MULTI_ROOT_WORKSPACE_SETUP.md @@ -1,169 +1,154 @@ # Multi-Root Workspace Setup Guide -This document explains how the LEDMatrix project uses a multi-root workspace to manage plugins as separate Git repositories. +This document explains how to work on LEDMatrix and the official plugins side +by side, with one editor workspace and the plugins loaded straight from your +plugin checkout. ## Overview -The LEDMatrix project has been migrated from a git submodule implementation to a **multi-root workspace** implementation for managing plugins. This allows: +Official plugins live in a single repository, +[ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins), with one +directory per plugin under `plugins/`. There are no separate per-plugin +repositories. For development you clone that monorepo **next to** LEDMatrix +and symlink its plugin directories into LEDMatrix's `plugin-repos/`, which is +where the plugin loader looks by default. -- ✅ Plugins to exist as independent Git repositories -- ✅ Updates to plugins without modifying the LEDMatrix project -- ✅ Easy development workflow with all repos in one workspace -- ✅ Plugin system discovers plugins via symlinks in `plugin-repos/` +- ✅ Plugin code stays in the monorepo checkout, with its own git history +- ✅ LEDMatrix discovers the plugins through symlinks in `plugin-repos/` +- ✅ `LEDMatrix.code-workspace` opens both repositories in VS Code/Cursor ## Directory Structure ```text -/home/chuck/Github/ -├── LEDMatrix/ # Main project -│ ├── plugin-repos/ # Symlinks to actual repos (managed automatically) -│ │ ├── ledmatrix-clock-simple -> ../../ledmatrix-clock-simple -│ │ ├── ledmatrix-weather -> ../../ledmatrix-weather +~/Github/ +├── LEDMatrix/ # Main project +│ ├── plugin-repos/ # Plugin directory the loader scans +│ │ ├── starlark-apps/ # Bundled with LEDMatrix (tracked in git) +│ │ ├── web-ui-info/ # Bundled with LEDMatrix (tracked in git) +│ │ ├── clock-simple -> ../../ledmatrix-plugins/plugins/clock-simple +│ │ ├── ledmatrix-weather -> ../../ledmatrix-plugins/plugins/ledmatrix-weather │ │ └── ... -│ ├── LEDMatrix.code-workspace # Multi-root workspace configuration +│ ├── LEDMatrix.code-workspace # Opens LEDMatrix and ../ledmatrix-plugins │ └── ... -├── ledmatrix-clock-simple/ # Plugin repository (actual git repo) -├── ledmatrix-weather/ # Plugin repository (actual git repo) -├── ledmatrix-football-scoreboard/ # Plugin repository (actual git repo) -└── ... # Other plugin repos +└── ledmatrix-plugins/ # Plugin monorepo (git repo) + ├── plugins/ + │ ├── clock-simple/ + │ ├── ledmatrix-weather/ + │ └── ... + ├── plugins.json # Store registry + └── update_registry.py ``` ## How It Works -### 1. Plugin Repositories +### 1. The plugin monorepo -All plugin repositories are cloned to `/home/chuck/Github/` (parent directory of LEDMatrix) as regular Git repositories: +Clone ledmatrix-plugins into the same parent directory as LEDMatrix (the +scripts below look for `../ledmatrix-plugins` relative to the LEDMatrix +root): -- `ledmatrix-clock-simple/` -- `ledmatrix-weather/` -- `ledmatrix-football-scoreboard/` -- etc. +```bash +cd ~/Github +git clone https://github.com/ChuckBuilds/ledmatrix-plugins.git +``` ### 2. Symlinks in plugin-repos/ -The `LEDMatrix/plugin-repos/` directory contains symlinks pointing to the actual repositories in the parent directory. This allows the plugin system to discover plugins without modifying the project structure. +`scripts/setup_plugin_repos.py` creates one symlink per plugin in +`LEDMatrix/plugin-repos/`, named after the plugin's manifest `id` and pointing +at `../ledmatrix-plugins/plugins/`. -### 3. Multi-Root Workspace +### 3. Multi-root workspace -The `LEDMatrix.code-workspace` file configures VS Code/Cursor to open all plugin repositories as separate workspace roots, allowing easy development across all repos. +`LEDMatrix.code-workspace` has two roots: LEDMatrix itself and +`../ledmatrix-plugins`. ## Setup Scripts ### Initial Setup -If you already have plugin repositories cloned, use the setup script: - ```bash -cd /home/chuck/Github/LEDMatrix +cd ~/Github/LEDMatrix python3 scripts/setup_plugin_repos.py ``` This script: -- Reads the workspace configuration -- Creates symlinks in `plugin-repos/` pointing to actual repos -- Verifies all links are created correctly +- Reads each `manifest.json` under `../ledmatrix-plugins/plugins/` +- Creates `plugin-repos/` symlinks (relative) to those directories +- Leaves correct links alone, replaces links that point elsewhere, and skips + (does not overwrite) a real directory of the same name — for example a + plugin you installed from the Plugin Store. Remove that directory first if + you want the linked copy. ### Updating Plugins -To update all plugin repositories: - ```bash -cd /home/chuck/Github/LEDMatrix +cd ~/Github/LEDMatrix python3 scripts/update_plugin_repos.py ``` -This script: -- Finds all plugins in the workspace -- Runs `git pull` on each repository -- Reports which plugins were updated +This runs `git pull` in `../ledmatrix-plugins` and prints the result. The +symlinks pick up the new code; restart the display to load it. ## Configuration -The plugin system is configured in `config/config.json`: +The loader reads plugins from `plugin_system.plugins_directory` in +`config/config.json`. The default is already right for this setup: ```json { "plugin_system": { - "plugins_directory": "plugin-repos", - "auto_discover": true, - "auto_load_enabled": true + "plugins_directory": "plugin-repos" } } ``` -The `plugins_directory` points to `plugin-repos/`, which contains symlinks to the actual repositories. - ## Workflow ### Daily Development 1. **Open Workspace**: Open `LEDMatrix.code-workspace` in VS Code/Cursor -2. **All Repos Available**: All plugin repos appear as separate folders in the workspace -3. **Edit Plugins**: Edit plugin code directly in their repositories -4. **Update Plugins**: Run `update_plugin_repos.py` to pull latest changes +2. **Edit Plugins**: Edit code under `ledmatrix-plugins/plugins//` +3. **Test**: `python3 run.py -e` (emulator) or + `python3 scripts/check_plugin.py --plugin ` from LEDMatrix +4. **Ship**: Bump `version` in the plugin's `manifest.json`, run + `python update_registry.py` in ledmatrix-plugins, commit there ### Adding New Plugins -1. **Clone Repository**: Clone the new plugin repo to `/home/chuck/Github/` -2. **Add to Workspace**: Add the plugin folder to `LEDMatrix.code-workspace` -3. **Create Symlink**: Run `setup_plugin_repos.py` to create the symlink - -### Updating Individual Plugins - -Since plugins are regular Git repositories, you can update them individually: - -```bash -cd /home/chuck/Github/ledmatrix-weather -git pull origin master -``` - -Or update all at once: - -```bash -cd /home/chuck/Github/LEDMatrix -python3 scripts/update_plugin_repos.py -``` - -## Benefits - -1. **No Submodule Hassle**: No need to update `.gitmodules` or run `git submodule update` -2. **Independent Updates**: Update plugins independently without touching LEDMatrix -3. **Clean Separation**: Each plugin is a separate repository with its own history -4. **Easy Development**: Multi-root workspace makes it easy to work across repos -5. **Automatic Discovery**: Plugin system automatically discovers plugins via symlinks +1. Create `plugins//` in the monorepo checkout +2. Run `python3 scripts/setup_plugin_repos.py` in LEDMatrix to link it ## Troubleshooting -### Symlinks Not Working - -If plugins aren't being discovered: +### Plugins not discovered ```bash -cd /home/chuck/Github/LEDMatrix -python3 scripts/setup_plugin_repos.py +cd ~/Github/LEDMatrix +ls -la plugin-repos/ # links present and not broken? +python3 scripts/setup_plugin_repos.py # recreate them ``` -This will recreate all symlinks. +Also check that `plugin_system.plugins_directory` is `plugin-repos`. -### Missing Plugins +### "Monorepo plugins directory not found" -If a plugin is in the workspace but not found: +`setup_plugin_repos.py` expects the monorepo at `../ledmatrix-plugins`. Clone +it there (or symlink it there). -1. Check if the repo exists in `/home/chuck/Github/` -2. Check if the symlink exists in `plugin-repos/` -3. Run `setup_plugin_repos.py` to recreate symlinks +### Plugin updates not showing -### Plugin Updates Not Showing - -If changes to plugins aren't appearing: - -1. Verify the symlink points to the correct directory: `ls -la plugin-repos/ledmatrix-weather` -2. Check that you're editing in the actual repo, not a copy -3. Restart the LEDMatrix service if running +1. Verify the link target: `ls -la plugin-repos/` +2. Check that you're editing the monorepo checkout, not a store-installed copy +3. Restart the LEDMatrix service (or `run.py`) ## Notes -- The `plugin-repos/` directory is tracked in git, but only contains symlinks -- Actual plugin code lives in `/home/chuck/Github/ledmatrix-*/` -- Each plugin repo can be updated independently via `git pull` -- The LEDMatrix project doesn't need to be updated when plugins change +- `plugin-repos/` is tracked in git only for the bundled plugins + (`starlark-apps`, `web-ui-info`). The symlinks you create are untracked + files; don't commit them. +- For linking a single plugin into `plugins/` instead (without a sibling + checkout), see `scripts/dev/dev_plugin_setup.sh` in the + [Plugin Development Guide](PLUGIN_DEVELOPMENT_GUIDE.md). +- When changing a plugin in the monorepo, bump its manifest `version` and run + `python update_registry.py`, or users won't receive the update. diff --git a/scripts/fix_perms/README.md b/scripts/fix_perms/README.md index 5616e51a..77575a71 100644 --- a/scripts/fix_perms/README.md +++ b/scripts/fix_perms/README.md @@ -7,8 +7,10 @@ install as the wrong user, after a manual file copy that didn't preserve ownership, or after a permissions-related error from the display or web service. -Most of these scripts require `sudo` since they touch directories -owned by the `ledmatrix` service user or by `root`. +Most of these scripts require `sudo` since they touch directories owned +by `root` (the display service's user) or by the user you installed +LEDMatrix as (the web service's user). There is no dedicated `ledmatrix` +system user. ## Scripts @@ -16,11 +18,12 @@ owned by the `ledmatrix` service user or by `root`. permissions on the `assets/` tree so plugins can download and cache team logos, fonts, and other static content. -- **`fix_cache_permissions.sh`** — Fixes permissions on every cache - directory the project may use (`/var/cache/ledmatrix/`, - `~/.cache/ledmatrix/`, `/opt/ledmatrix/cache/`, project-local - `cache/`). Also creates placeholder logo subdirectories used by the - sports plugins. +- **`fix_cache_permissions.sh`** — Creates (if missing) and fixes + permissions on `/var/cache/ledmatrix/` and `~/.ledmatrix_cache/` of the + user running `sudo`, and creates + `/var/cache/ledmatrix/placeholder_logos/` for the sports plugins. It does + not touch the cache manager's other fallbacks (`/opt/ledmatrix/cache`, + `$TMPDIR/ledmatrix_cache`). - **`fix_plugin_permissions.sh`** — Fixes ownership on the plugins directory so both the root display service and the web service user @@ -31,6 +34,11 @@ owned by the `ledmatrix` service user or by `root`. systemd journal access, and the sudoers entries the web interface needs to control the display service. +- **`safe_pip_install.sh`** — Installs a `requirements.txt` as root + after checking it is the project's own or one under `plugin-repos/` or + `plugins/`. Used by the web interface (via sudo) to install plugin + dependencies where `ledmatrix.service` can import them. + - **`safe_plugin_rm.sh`** — Validates that a plugin removal path is inside an allowed base directory before deleting it. Used by the web interface (via sudo) when a user clicks **Uninstall** on a plugin — diff --git a/scripts/install/README.md b/scripts/install/README.md index 0813f286..dc092d9d 100644 --- a/scripts/install/README.md +++ b/scripts/install/README.md @@ -7,14 +7,18 @@ This directory contains scripts for installing and configuring the LEDMatrix sys - **`one-shot-install.sh`** - Single-command installer; clones the repo, checks prerequisites, then runs `first_time_install.sh`. Invoked via `curl ... | bash` from the project root README. -- **`install_service.sh`** - Installs the main LED Matrix display service (systemd) -- **`install_web_service.sh`** - Installs the web interface service (systemd) +- **`install_service.sh`** - Installs, enables and starts the display + service (`ledmatrix.service`), the web interface service + (`ledmatrix-web.service`) and the update-verify units (systemd) +- **`install_web_service.sh`** - Installs only the web interface service + and the update-verify units (systemd) - **`install_wifi_monitor.sh`** - Installs the WiFi monitor daemon service - **`setup_cache.sh`** - Sets up persistent cache directory with proper permissions - **`configure_web_sudo.sh`** - Configures passwordless sudo access for web interface actions -- **`configure_wifi_permissions.sh`** - Grants the `ledmatrix` user - the WiFi management permissions needed by the web interface and - the WiFi monitor service +- **`configure_wifi_permissions.sh`** - Grants the web interface's user + (the user who runs the script, i.e. the one you installed LEDMatrix as; + there is no `ledmatrix` system user) the passwordless `nmcli` and related + WiFi permissions the web interface needs - **`migrate_config.sh`** - Migrates configuration files to new formats (if needed) - **`debug_install.sh`** - Diagnostic helper used when an install fails; collects environment info and recent logs From 945d1997aab36b3c3155df2215897679b5fd7ad8 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 09:06:08 -0400 Subject: [PATCH 10/13] docs(config): rewrite PLUGIN_CONFIG_ARCHITECTURE for the v3 web UI It described web_interface_v2.py and index_v2.html (both gone), client-side form generation, one POST per field with {key, value}, and 'no nested objects'. The v3 UI renders plugin forms server-side from the schema (pages_v3 partial + plugin_config.html macros, nested sections and x-widgets), posts the whole form once, and save_plugin_config() merges onto the stored section, validates, splits x-secret fields and notifies the plugin. Co-Authored-By: Claude Opus 5 --- docs/PLUGIN_CONFIG_ARCHITECTURE.md | 516 ++++++++--------------------- 1 file changed, 134 insertions(+), 382 deletions(-) diff --git a/docs/PLUGIN_CONFIG_ARCHITECTURE.md b/docs/PLUGIN_CONFIG_ARCHITECTURE.md index 823bb4db..b34ffc89 100644 --- a/docs/PLUGIN_CONFIG_ARCHITECTURE.md +++ b/docs/PLUGIN_CONFIG_ARCHITECTURE.md @@ -11,427 +11,179 @@ ### Component Overview ``` -┌─────────────────────────────────────────────────────────────────┐ -│ Web Browser │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Tab Navigation Bar │ │ -│ │ [Overview] [General] ... [Plugins] [Plugin X] [Plugin Y]│ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────┐ ┌──────────────────────────────────┐ │ -│ │ Plugins Tab │ │ Plugin X Configuration Tab │ │ -│ │ │ │ │ │ -│ │ • Install │ │ Form Generated from Schema: │ │ -│ │ • Update │ │ • Boolean → Toggle │ │ -│ │ • Uninstall │ │ • Number → Number Input │ │ -│ │ • Enable │ │ • String → Text Input │ │ -│ │ • [Configure]──────→ • Array → Comma Input │ │ -│ │ │ │ • Enum → Dropdown │ │ -│ └─────────────────┘ │ │ │ -│ │ [Save] [Back] [Reset] │ │ -│ └──────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Web browser (templates/v3/base.html, Alpine.js + HTMX) │ +│ │ +│ Second nav row: one tab per installed plugin │ +│ Clicking a tab: GET /v3/partials/plugin-config/ │ +│ → server-rendered form swapped into the tab │ +│ │ +│ Save: hx-post="/api/v3/plugins/config?plugin_id=" (form data) │ +└──────────────────────────────────────────────────────────────────┘ │ - │ HTTP API ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Flask Backend │ -│ ┌───────────────────────────────────────────────────────┐ │ -│ │ /api/v3/plugins/installed │ │ -│ │ • Discover plugins in plugins/ directory │ │ -│ │ • Load manifest.json for each plugin │ │ -│ │ • Load config_schema.json if exists │ │ -│ │ • Load current config from config.json │ │ -│ │ • Return combined data to frontend │ │ -│ └───────────────────────────────────────────────────────┘ │ -│ │ -│ ┌───────────────────────────────────────────────────────┐ │ -│ │ /api/v3/plugins/config │ │ -│ │ • Receive key-value pair │ │ -│ │ • Update config.json │ │ -│ │ • Return success/error │ │ -│ └───────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Flask (web_interface/app.py) │ +│ │ +│ pages_v3 blueprint (blueprints/pages_v3.py) │ +│ _load_plugin_config_partial(plugin_id) │ +│ • SchemaManager.load_schema() → config_schema.json │ +│ • config.json section for the plugin │ +│ • masks x-secret fields │ +│ • renders partials/plugin_config.html (render_field macros) │ +│ │ +│ api_v3 blueprint (blueprints/api_v3/plugins.py) │ +│ save_plugin_config() POST /api/v3/plugins/config │ +│ get_plugin_config() GET /api/v3/plugins/config │ +│ get_plugin_schema() GET /api/v3/plugins/schema │ +│ reset_plugin_config() POST /api/v3/plugins/config/reset │ +└──────────────────────────────────────────────────────────────────┘ │ - │ File System ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ File System │ -│ │ -│ plugins/ │ -│ ├── hello-world/ │ -│ │ ├── manifest.json ───┐ │ -│ │ ├── config_schema.json ─┼─→ Defines UI structure │ -│ │ ├── manager.py │ │ -│ │ └── requirements.txt │ │ -│ └── clock-simple/ │ │ -│ ├── manifest.json │ │ -│ └── config_schema.json ──┘ │ -│ │ -│ config/ │ -│ └── config.json ────────────→ Stores configuration values │ -│ { │ -│ "hello-world": { │ -│ "enabled": true, │ -│ "message": "Hello!", │ -│ ... │ -│ } │ -│ } │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Files │ +│ plugin-repos//config_schema.json JSON Schema (Draft-7) │ +│ config/config.json { "": { ... } } │ +│ config/config_secrets.json { "": { secrets } } │ +└──────────────────────────────────────────────────────────────────┘ ``` +The plugins directory is `plugin_system.plugins_directory` in +`config/config.json` (default `plugin-repos/`). Plugin configuration lives in +`config/config.json`, not in the plugin directory, so it survives reinstalls. + ## Data Flow -### 1. Page Load Sequence +### 1. Rendering a plugin's tab ``` -User Opens Web Interface - │ - ▼ -DOMContentLoaded Event - │ - ▼ -refreshPlugins() - │ - ▼ -GET /api/v3/plugins/installed - │ - ├─→ For each plugin directory: - │ ├─→ Read manifest.json - │ ├─→ Read config_schema.json (if exists) - │ └─→ Read config from config.json - │ - ▼ -Return JSON Array: -[{ - id: "hello-world", - name: "Hello World", - config: { enabled: true, message: "Hello!" }, - config_schema_data: { - properties: { - enabled: { type: "boolean", ... }, - message: { type: "string", ... } - } - } -}, ...] - │ - ▼ -generatePluginTabs(plugins) - │ - ├─→ For each plugin: - │ ├─→ Create tab button - │ ├─→ Create tab content div - │ └─→ generatePluginConfigForm(plugin) - │ │ - │ ├─→ Read schema properties - │ ├─→ Get current config values - │ └─→ Generate HTML form inputs - │ - ▼ -Tabs Rendered in UI +User opens the plugin's tab + │ + ▼ +GET /v3/partials/plugin-config/ (pages_v3) + │ + ├─→ Load schema (SchemaManager, no cache) + ├─→ Load config.json[] + ├─→ Mask "x-secret" values (fails closed if the schema is unusable) + └─→ render partials/plugin_config.html + │ + └─→ render_field() per property, recursively: + boolean → toggle, number/integer → input or slider, + string → input / textarea / select (enum), + array → list or table widget, + object → collapsible nested section, + "x-widget" → a registered widget + (static/v3/js/widgets/, or one the plugin ships) ``` -### 2. Configuration Save Sequence +Nested objects are supported: a nested field is posted with a dotted name +(e.g. `transition.type`). + +### 2. Saving ``` -User Modifies Form - │ - ▼ -User Clicks "Save" - │ - ▼ -savePluginConfiguration(pluginId) - │ - ├─→ Get form data - ├─→ For each field: - │ ├─→ Get schema type - │ ├─→ Convert value to correct type - │ │ • boolean: checkbox.checked - │ │ • integer: parseInt() - │ │ • number: parseFloat() - │ │ • array: split(',') - │ │ • string: as-is - │ │ - │ └─→ POST /api/v3/plugins/config - │ { - │ plugin_id: "hello-world", - │ key: "message", - │ value: "Hello, World!" - │ } - │ - ▼ -Backend Updates config.json - │ - ▼ -Return Success - │ - ▼ -Show Notification - │ - ▼ -Refresh Plugins +User clicks Save + │ + ▼ +validatePluginConfigForm() (client-side checks) + │ + ▼ +POST /api/v3/plugins/config?plugin_id= (form data, all fields of the form) + │ + ▼ +save_plugin_config() (api_v3/plugins.py) + ├─→ Start from the stored config.json[] + ├─→ Apply form fields: dotted names → nested keys, "[]" checkbox + │ groups → lists, values coerced to the schema's types + ├─→ Merge schema defaults for keys that are still missing + ├─→ Validate against the schema (plus core per-plugin properties); + │ invalid → 400 with the validation errors, nothing saved + ├─→ Split "x-secret" fields out; masked/blank secrets are dropped so + │ an untouched secret keeps its stored value + ├─→ Deep-merge regular fields into config.json[] (atomic save) + ├─→ Merge secrets into config_secrets.json[] + └─→ Call the loaded plugin's on_config_change() (and + on_enable/on_disable if "enabled" changed) + │ + ▼ +One response for the whole form → notification in the UI ``` -## Class and Function Hierarchy +The display service picks up the new config through its config hot reload +(ConfigService) without a restart. -### Frontend (JavaScript) +JSON clients can post `{"plugin_id": ..., "config": {...}}` instead; the keys +sent are merged onto the stored config the same way. See +[REST_API_REFERENCE.md](REST_API_REFERENCE.md#save-plugin-configuration). -``` -Window Load - └── DOMContentLoaded - └── refreshPlugins() - ├── fetch('/api/v3/plugins/installed') - ├── renderInstalledPlugins(plugins) - └── generatePluginTabs(plugins) - └── For each plugin: - ├── Create tab button - ├── Create tab content - └── generatePluginConfigForm(plugin) - ├── Read config_schema_data - ├── Read current config - └── Generate form HTML - ├── Boolean → Toggle switch - ├── Number → Number input - ├── String → Text input - ├── Array → Comma-separated input - └── Enum → Select dropdown +### 3. Reset -User Interactions - ├── configurePlugin(pluginId) - │ └── showTab(`plugin-${pluginId}`) - │ - ├── savePluginConfiguration(pluginId) - │ ├── Process form data - │ ├── Convert types per schema - │ └── For each field: - │ └── POST /api/v3/plugins/config - │ - └── resetPluginConfig(pluginId) - ├── Get schema defaults - └── For each field: - └── POST /api/v3/plugins/config -``` - -### Backend (Python) - -``` -Flask Routes - ├── /api/v3/plugins/installed (GET) - │ └── api_plugins_installed() - │ ├── PluginManager.discover_plugins() - │ ├── For each plugin: - │ │ ├── PluginManager.get_plugin_info() - │ │ ├── Load config_schema.json - │ │ └── Load config from config.json - │ └── Return JSON response - │ - └── /api/v3/plugins/config (POST) - └── api_plugin_config() - ├── Parse request JSON - ├── Load current config - ├── Update config[plugin_id][key] = value - └── Save config.json -``` - -## File Structure - -``` -LEDMatrix/ -│ -├── web_interface_v2.py -│ └── Flask backend with plugin API endpoints -│ -├── templates/ -│ └── index_v2.html -│ └── Frontend with dynamic tab generation -│ -├── config/ -│ └── config.json -│ └── Stores all plugin configurations -│ -├── plugins/ -│ ├── hello-world/ -│ │ ├── manifest.json ← Plugin metadata -│ │ ├── config_schema.json ← UI schema definition -│ │ ├── manager.py ← Plugin logic -│ │ └── requirements.txt -│ │ -│ └── clock-simple/ -│ ├── manifest.json -│ ├── config_schema.json -│ └── manager.py -│ -└── docs/ - ├── PLUGIN_CONFIGURATION_TABS.md ← Full documentation - ├── PLUGIN_CONFIG_TABS_SUMMARY.md ← Implementation summary - ├── PLUGIN_CONFIG_QUICK_START.md ← Quick start guide - └── PLUGIN_CONFIG_ARCHITECTURE.md ← This file -``` +`POST /api/v3/plugins/config/reset` replaces the plugin's section with the +schema defaults (keeping secrets unless `preserve_secrets` is false). ## Key Design Decisions -### 1. Dynamic Tab Generation +### 1. Server-side rendered forms -**Why**: Plugins are installed/uninstalled dynamically -**How**: JavaScript creates/removes tab elements on plugin list refresh -**Benefit**: No server-side template rendering needed +**Why**: One renderer for every plugin, no per-plugin frontend code +**How**: Jinja macros in `partials/plugin_config.html` walk the schema +**Benefit**: The settings search index is built from the same rendered HTML +(`/v3/settings/search-index`) -### 2. JSON Schema as Source of Truth +### 2. JSON Schema as source of truth -**Why**: Standard, well-documented, validation-ready -**How**: Frontend interprets schema to generate forms -**Benefit**: Plugin developers use familiar format +**Why**: Standard, well-documented, validation-ready +**How**: The same schema drives the form, the defaults and server-side validation +**Benefit**: Plugin developers use a familiar format -### 3. Individual Config Updates +### 3. Whole-form saves that merge -**Why**: Simplifies backend API -**How**: Each field saved separately via `/api/v3/plugins/config` -**Benefit**: Atomic updates, easier error handling +**Why**: A partial form (or a field the form doesn't show) must not wipe +stored values +**How**: The handler starts from the stored section and merges what was posted +**Benefit**: One request per save, atomic write -### 4. Type Conversion in Frontend +### 4. Secrets kept out of config.json -**Why**: HTML forms only return strings -**How**: JavaScript converts based on schema type before sending -**Benefit**: Backend receives correctly-typed values - -### 5. No Nested Objects - -**Why**: Keeps UI simple -**How**: Only flat property structures supported -**Benefit**: Easy form generation, clear to users +**Why**: `config.json` is shown in the raw editor and returned by the API +**How**: `"x-secret": true` fields go to `config_secrets.json`, which is +deep-merged back into the plugin's config at load time +**Benefit**: Plugins read secrets with plain `config.get(...)` ## Extension Points -### Adding New Input Types +### Custom input widgets -Location: `generatePluginConfigForm()` in `index_v2.html` +Set `"x-widget": ""` on a property. Core widgets are in +`web_interface/static/v3/js/widgets/` (see its README); a plugin can ship its +own widget script, served from `/static/plugin-widgets//.js`. +See [widget-guide.md](widget-guide.md). -```javascript -if (type === 'your-new-type') { - formHTML += ` - - `; -} -``` +### Custom actions -### Custom Validation +Buttons that run plugin scripts are declared in the manifest's +`web_ui_actions`. See [PLUGIN_WEB_UI_ACTIONS.md](PLUGIN_WEB_UI_ACTIONS.md). -Location: `savePluginConfiguration()` in `index_v2.html` +### Reacting to changes -```javascript -// Add validation before sending -if (!validateCustomConstraint(value, propSchema)) { - throw new Error('Validation failed'); -} -``` +Implement `on_config_change(new_config)` in the plugin (see +[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md)). -### Backend Hook +## Where to Look -Location: `api_plugin_config()` in `web_interface_v2.py` - -```python -# Add custom logic before saving -if plugin_id == 'special-plugin': - value = transform_value(value) -``` - -## Performance Considerations - -### Frontend - -- **Tab Generation**: O(n) where n = number of plugins (typically < 20) -- **Form Generation**: O(m) where m = number of config properties (typically < 10) -- **Memory**: Each plugin tab ~5KB HTML -- **Total Impact**: Negligible for typical use cases - -### Backend - -- **Schema Loading**: Cached after first load -- **Config Updates**: Single file write (atomic) -- **API Calls**: One per config field on save (sequential) -- **Optimization**: Could batch updates in single API call - -## Security Considerations - -1. **Input Validation**: Schema constraints enforced client-side (UX) and should be enforced server-side -2. **Path Traversal**: Plugin paths validated against known plugin directory -3. **XSS**: All user inputs escaped before rendering in HTML -4. **CSRF**: Flask CSRF tokens should be used in production -5. **File Permissions**: config.json requires write access +| Concern | File | +|---------|------| +| Tab partial loader | `web_interface/blueprints/pages_v3.py` (`_load_plugin_config_partial`) | +| Form template and field macros | `web_interface/templates/v3/partials/plugin_config.html` | +| Save / get / schema / reset handlers | `web_interface/blueprints/api_v3/plugins.py` | +| Schema loading, defaults, validation | `src/plugin_system/schema_manager.py` | +| Secret masking and splitting | `src/web_interface/secret_helpers.py` | +| Widgets | `web_interface/static/v3/js/widgets/` | ## Error Handling -### Frontend - -- Network errors: Show notification, don't crash -- Schema errors: Graceful fallback to no config tab -- Type errors: Log to console, continue processing other fields - -### Backend - -- Invalid plugin_id: 400 Bad Request -- Schema not found: Return null, frontend handles gracefully -- Config save error: 500 Internal Server Error with message - -## Testing Strategy - -### Unit Tests - -- `generatePluginConfigForm()` for each schema type -- Type conversion logic in `savePluginConfiguration()` -- Backend schema loading logic - -### Integration Tests - -- Full save flow: form → API → config.json -- Tab generation from API response -- Reset to defaults - -### E2E Tests - -- Install plugin → verify tab appears -- Configure plugin → verify config saved -- Uninstall plugin → verify tab removed - -## Monitoring - -### Frontend Metrics - -- Time to generate tabs -- Form submission success rate -- User interactions (configure, save, reset) - -### Backend Metrics - -- API response times -- Config update success rate -- Schema loading errors - -### User Feedback - -- Are users finding the configuration interface? -- Are validation errors clear? -- Are default values sensible? - -## Future Roadmap - -### Phase 2: Enhanced Validation -- Real-time validation feedback -- Custom error messages -- Dependent field validation - -### Phase 3: Advanced Inputs -- Color pickers for RGB arrays -- File upload for assets -- Rich text editor for descriptions - -### Phase 4: Configuration Management -- Export/import configurations -- Configuration presets -- Version history/rollback - -### Phase 5: Developer Tools -- Schema editor in web UI -- Live preview while editing schema -- Validation tester - +- Unknown plugin or unreadable schema: the partial renders an error message +- Validation failure: `400` with `details` and `context.validation_errors`; + the form shows them and nothing is saved +- Save failure: `500` with an error message; config.json is written + atomically, so a failed save leaves the previous file intact From 2e031ea7401e63427ad6a6dccf077c6d60523457 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 09:06:33 -0400 Subject: [PATCH 11/13] docs(mqtt): brightness saves apply via hot reload and leave other settings alone The bridge README said brightness is applied on the display's next restart; the display controller's config hot reload applies it within seconds. It also now states that the bridge's partial JSON save changes only brightness (the /config/main merge fix in this change set). Co-Authored-By: Claude Opus 5 --- integrations/mqtt_bridge/README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/integrations/mqtt_bridge/README.md b/integrations/mqtt_bridge/README.md index 29d31b04..dae255d4 100644 --- a/integrations/mqtt_bridge/README.md +++ b/integrations/mqtt_bridge/README.md @@ -114,5 +114,9 @@ python3 integrations/mqtt_bridge/ledmatrix_mqtt_bridge.py --config integrations/ discovery itself. Discovery is lazy and normally happens because somebody opened the dashboard; without that endpoint a bridge that never does would see an empty list. -- Brightness writes `display.hardware.brightness` through `/api/v3/config/main`. - The display service picks it up on its next restart, not instantly. +- Brightness writes `display.hardware.brightness` by posting + `{"brightness": N}` to `/api/v3/config/main`. A JSON save changes only the + keys it sends, so the other display settings are left as they were. The + display service's config hot reload notices the change within a few seconds + and applies it without a restart (while a dim schedule is dimming the panel, + the dim level wins until the dim period ends). From 69ddcdadb72fd67e6a2dc23984f01b57406c6c7b Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 09:07:27 -0400 Subject: [PATCH 12/13] docs(config): mark the plugin_system toggles as unused legacy keys auto_discover, auto_load_enabled and development_mode are read by nothing and leave the General tab in this change set (F40). CONFIG_REFERENCE said they were read by the plugin loader; PLUGIN_CONFIGURATION_GUIDE and the REST reference listed them as live settings. Co-Authored-By: Claude Opus 5 --- docs/CONFIG_REFERENCE.md | 10 ++++------ docs/PLUGIN_CONFIGURATION_GUIDE.md | 10 ++++------ docs/REST_API_REFERENCE.md | 2 +- 3 files changed, 9 insertions(+), 13 deletions(-) diff --git a/docs/CONFIG_REFERENCE.md b/docs/CONFIG_REFERENCE.md index f4d991ca..f55e19c1 100644 --- a/docs/CONFIG_REFERENCE.md +++ b/docs/CONFIG_REFERENCE.md @@ -152,14 +152,12 @@ Read by `src/common/sync_manager.py` and `src/display_controller.py`. ## `plugin_system` -Read by the plugin loader/manager (`src/plugin_system/`). - | Key | Type / default | Meaning | |---|---|---| -| `plugins_directory` | string, `"plugin-repos"` | Where the Plugin Store installs plugins | -| `auto_discover` | bool, `true` | Scan the plugins directory at startup | -| `auto_load_enabled` | bool, `true` | Load discovered plugins automatically | -| `development_mode` | bool, `false` | Development conveniences in the web UI (editable under General settings) | +| `plugins_directory` | string, `"plugin-repos"` | Where the Plugin Store installs plugins and the only directory the plugin loader scans. Read by `PluginManager` and `PluginStoreManager` (`src/plugin_system/`); editable under General settings | +| `auto_discover` | bool, `true` | **Unused.** Legacy key, read by nothing. Plugins are always discovered, and every plugin with `enabled: true` is loaded. Not shown in the web UI; may be left in or removed from config.json | +| `auto_load_enabled` | bool, `true` | **Unused.** Legacy key, read by nothing (see `auto_discover`). To keep a plugin installed but dormant, set its own `enabled` to `false` | +| `development_mode` | bool, `false` | **Unused.** Legacy key, read by nothing | ## Plugin config blocks diff --git a/docs/PLUGIN_CONFIGURATION_GUIDE.md b/docs/PLUGIN_CONFIGURATION_GUIDE.md index 7471130a..5c4956fa 100644 --- a/docs/PLUGIN_CONFIGURATION_GUIDE.md +++ b/docs/PLUGIN_CONFIGURATION_GUIDE.md @@ -67,9 +67,7 @@ The main configuration file (`config/config.json`) now contains only essential s "time_format": "%I:%M %p" }, "plugin_system": { - "plugins_directory": "plugin-repos", - "auto_discover": true, - "auto_load_enabled": true + "plugins_directory": "plugin-repos" } } ``` @@ -93,9 +91,9 @@ The main configuration file (`config/config.json`) now contains only essential s #### 4. Plugin System - **plugin_system**: Plugin system configuration - - **plugins_directory**: Directory where plugins are stored - - **auto_discover**: Automatically discover plugins - - **auto_load_enabled**: Automatically load enabled plugins + - **plugins_directory**: Directory where plugins are stored (the only one the loader scans) + - `auto_discover`, `auto_load_enabled`, `development_mode` may still appear in + older configs; nothing reads them (see [CONFIG_REFERENCE.md](CONFIG_REFERENCE.md#plugin_system)) ## Plugin Configuration diff --git a/docs/REST_API_REFERENCE.md b/docs/REST_API_REFERENCE.md index 5a9865bb..4f36386a 100644 --- a/docs/REST_API_REFERENCE.md +++ b/docs/REST_API_REFERENCE.md @@ -90,7 +90,7 @@ maps into the nested config: |--------|-----------| | `timezone`, `city`, `state`, `country` | `timezone`, `location.*` | | `web_display_autostart`, `auto_update_enabled` | `web_display_autostart`, `auto_update.enabled` | -| `plugins_directory`, `auto_discover`, `auto_load_enabled`, `development_mode` | `plugin_system.*` | +| `plugins_directory` (and the unused legacy flags `auto_discover`, `auto_load_enabled`, `development_mode`, stored only when sent) | `plugin_system.*` | | `target_fps` (30-200) | `target_fps` | | `rows`, `cols`, `chain_length`, `parallel`, `brightness`, `hardware_mapping`, `pwm_bits`, `led_rgb_sequence`, `panel_type`, `pixel_mapper_config`, `disable_hardware_pulsing`, `inverse_colors`, `show_refresh_rate`, ... | `display.hardware.*` | | `gpio_slowdown`, `rp1_rio` | `display.runtime.*` | From 708b69a2eb87626718906acf75edbc107ce193b2 Mon Sep 17 00:00:00 2001 From: Chuck <33324927+ChuckBuilds@users.noreply.github.com> Date: Thu, 17 Sep 2026 09:07:51 -0400 Subject: [PATCH 13/13] docs(changelog): docs and developer tools group Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ea497352..877e7d97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -135,6 +135,34 @@ Core: `ensure_shared_group_ownership()` now returns immediately when `os.geteuid` or `os.chown` is missing. No behaviour change on the Pi. +Docs and developer tools: + +- `docs/REST_API_REFERENCE.md` rechecked against every handler: request + fields that made documented calls fail (`repo_url`, `action_id`/`params`, + `files`/`image_id`, `font_file`+`font_family`, `?font=`, cache `key`, + `auto_enable_ap_mode`, plugin limit keys) and response shapes are fixed, the + removed font-override endpoints are gone, and the 26 undocumented routes + (backup, git/auto-update, WiFi radio, Starlark editor, MQTT bridge, status + endpoints, skins) are listed. Store search is `/plugins/store/list?query=`. +- `FONT_MANAGER.md` no longer tells plugins to read + `display_manager.font_manager`, which does not exist; use + `plugin_manager.font_manager` / `BasePlugin._get_font_manager()`. +- Plugin docs, `DisplayManager` docstrings and the bundled `starlark-apps` + plugin now all read the display size from `display_manager.width/height`, + which works in fallback mode where `matrix` is `None`. +- `scripts/dev/dev_plugin_setup.sh link-github ` links the plugin from a + clone of the `ledmatrix-plugins` monorepo (per-plugin `ledmatrix-` + repositories no longer exist). `dev_plugins.json` honours `github_user`, + `plugins_repo` and `plugins_branch`; `dev_plugins.json.example` ships and + `dev_plugins.json` is git-ignored. `update`/`status` handle monorepo links, + and `status` no longer exits 1 when nothing is broken. +- Rewritten for current behaviour: plugin dependency installation (web service + runs as the installing user and installs through `safe_pip_install.sh`), + `PLUGIN_CONFIG_ARCHITECTURE.md`, `MULTI_ROOT_WORKSPACE_SETUP.md`; stale + `app.py` line numbers, `api_v3.py` paths, StreamManager method names, + nonexistent version-bump scripts and `ledmatrix` service user references + removed. + ## 3.4.0 Plugin-facing changes since 3.3.0 (tag `v3.3.1`) not covered further down: