diff --git a/docs/PLUGIN_CONFIG_ARCHITECTURE.md b/docs/PLUGIN_CONFIG_ARCHITECTURE.md index 823bb4db..b34ffc89 100644 --- a/docs/PLUGIN_CONFIG_ARCHITECTURE.md +++ b/docs/PLUGIN_CONFIG_ARCHITECTURE.md @@ -11,427 +11,179 @@ ### Component Overview ``` -┌─────────────────────────────────────────────────────────────────┐ -│ Web Browser │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Tab Navigation Bar │ │ -│ │ [Overview] [General] ... [Plugins] [Plugin X] [Plugin Y]│ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────┐ ┌──────────────────────────────────┐ │ -│ │ Plugins Tab │ │ Plugin X Configuration Tab │ │ -│ │ │ │ │ │ -│ │ • Install │ │ Form Generated from Schema: │ │ -│ │ • Update │ │ • Boolean → Toggle │ │ -│ │ • Uninstall │ │ • Number → Number Input │ │ -│ │ • Enable │ │ • String → Text Input │ │ -│ │ • [Configure]──────→ • Array → Comma Input │ │ -│ │ │ │ • Enum → Dropdown │ │ -│ └─────────────────┘ │ │ │ -│ │ [Save] [Back] [Reset] │ │ -│ └──────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Web browser (templates/v3/base.html, Alpine.js + HTMX) │ +│ │ +│ Second nav row: one tab per installed plugin │ +│ Clicking a tab: GET /v3/partials/plugin-config/ │ +│ → server-rendered form swapped into the tab │ +│ │ +│ Save: hx-post="/api/v3/plugins/config?plugin_id=" (form data) │ +└──────────────────────────────────────────────────────────────────┘ │ - │ HTTP API ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Flask Backend │ -│ ┌───────────────────────────────────────────────────────┐ │ -│ │ /api/v3/plugins/installed │ │ -│ │ • Discover plugins in plugins/ directory │ │ -│ │ • Load manifest.json for each plugin │ │ -│ │ • Load config_schema.json if exists │ │ -│ │ • Load current config from config.json │ │ -│ │ • Return combined data to frontend │ │ -│ └───────────────────────────────────────────────────────┘ │ -│ │ -│ ┌───────────────────────────────────────────────────────┐ │ -│ │ /api/v3/plugins/config │ │ -│ │ • Receive key-value pair │ │ -│ │ • Update config.json │ │ -│ │ • Return success/error │ │ -│ └───────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Flask (web_interface/app.py) │ +│ │ +│ pages_v3 blueprint (blueprints/pages_v3.py) │ +│ _load_plugin_config_partial(plugin_id) │ +│ • SchemaManager.load_schema() → config_schema.json │ +│ • config.json section for the plugin │ +│ • masks x-secret fields │ +│ • renders partials/plugin_config.html (render_field macros) │ +│ │ +│ api_v3 blueprint (blueprints/api_v3/plugins.py) │ +│ save_plugin_config() POST /api/v3/plugins/config │ +│ get_plugin_config() GET /api/v3/plugins/config │ +│ get_plugin_schema() GET /api/v3/plugins/schema │ +│ reset_plugin_config() POST /api/v3/plugins/config/reset │ +└──────────────────────────────────────────────────────────────────┘ │ - │ File System ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ File System │ -│ │ -│ plugins/ │ -│ ├── hello-world/ │ -│ │ ├── manifest.json ───┐ │ -│ │ ├── config_schema.json ─┼─→ Defines UI structure │ -│ │ ├── manager.py │ │ -│ │ └── requirements.txt │ │ -│ └── clock-simple/ │ │ -│ ├── manifest.json │ │ -│ └── config_schema.json ──┘ │ -│ │ -│ config/ │ -│ └── config.json ────────────→ Stores configuration values │ -│ { │ -│ "hello-world": { │ -│ "enabled": true, │ -│ "message": "Hello!", │ -│ ... │ -│ } │ -│ } │ -└─────────────────────────────────────────────────────────────────┘ +┌──────────────────────────────────────────────────────────────────┐ +│ Files │ +│ plugin-repos//config_schema.json JSON Schema (Draft-7) │ +│ config/config.json { "": { ... } } │ +│ config/config_secrets.json { "": { secrets } } │ +└──────────────────────────────────────────────────────────────────┘ ``` +The plugins directory is `plugin_system.plugins_directory` in +`config/config.json` (default `plugin-repos/`). Plugin configuration lives in +`config/config.json`, not in the plugin directory, so it survives reinstalls. + ## Data Flow -### 1. Page Load Sequence +### 1. Rendering a plugin's tab ``` -User Opens Web Interface - │ - ▼ -DOMContentLoaded Event - │ - ▼ -refreshPlugins() - │ - ▼ -GET /api/v3/plugins/installed - │ - ├─→ For each plugin directory: - │ ├─→ Read manifest.json - │ ├─→ Read config_schema.json (if exists) - │ └─→ Read config from config.json - │ - ▼ -Return JSON Array: -[{ - id: "hello-world", - name: "Hello World", - config: { enabled: true, message: "Hello!" }, - config_schema_data: { - properties: { - enabled: { type: "boolean", ... }, - message: { type: "string", ... } - } - } -}, ...] - │ - ▼ -generatePluginTabs(plugins) - │ - ├─→ For each plugin: - │ ├─→ Create tab button - │ ├─→ Create tab content div - │ └─→ generatePluginConfigForm(plugin) - │ │ - │ ├─→ Read schema properties - │ ├─→ Get current config values - │ └─→ Generate HTML form inputs - │ - ▼ -Tabs Rendered in UI +User opens the plugin's tab + │ + ▼ +GET /v3/partials/plugin-config/ (pages_v3) + │ + ├─→ Load schema (SchemaManager, no cache) + ├─→ Load config.json[] + ├─→ Mask "x-secret" values (fails closed if the schema is unusable) + └─→ render partials/plugin_config.html + │ + └─→ render_field() per property, recursively: + boolean → toggle, number/integer → input or slider, + string → input / textarea / select (enum), + array → list or table widget, + object → collapsible nested section, + "x-widget" → a registered widget + (static/v3/js/widgets/, or one the plugin ships) ``` -### 2. Configuration Save Sequence +Nested objects are supported: a nested field is posted with a dotted name +(e.g. `transition.type`). + +### 2. Saving ``` -User Modifies Form - │ - ▼ -User Clicks "Save" - │ - ▼ -savePluginConfiguration(pluginId) - │ - ├─→ Get form data - ├─→ For each field: - │ ├─→ Get schema type - │ ├─→ Convert value to correct type - │ │ • boolean: checkbox.checked - │ │ • integer: parseInt() - │ │ • number: parseFloat() - │ │ • array: split(',') - │ │ • string: as-is - │ │ - │ └─→ POST /api/v3/plugins/config - │ { - │ plugin_id: "hello-world", - │ key: "message", - │ value: "Hello, World!" - │ } - │ - ▼ -Backend Updates config.json - │ - ▼ -Return Success - │ - ▼ -Show Notification - │ - ▼ -Refresh Plugins +User clicks Save + │ + ▼ +validatePluginConfigForm() (client-side checks) + │ + ▼ +POST /api/v3/plugins/config?plugin_id= (form data, all fields of the form) + │ + ▼ +save_plugin_config() (api_v3/plugins.py) + ├─→ Start from the stored config.json[] + ├─→ Apply form fields: dotted names → nested keys, "[]" checkbox + │ groups → lists, values coerced to the schema's types + ├─→ Merge schema defaults for keys that are still missing + ├─→ Validate against the schema (plus core per-plugin properties); + │ invalid → 400 with the validation errors, nothing saved + ├─→ Split "x-secret" fields out; masked/blank secrets are dropped so + │ an untouched secret keeps its stored value + ├─→ Deep-merge regular fields into config.json[] (atomic save) + ├─→ Merge secrets into config_secrets.json[] + └─→ Call the loaded plugin's on_config_change() (and + on_enable/on_disable if "enabled" changed) + │ + ▼ +One response for the whole form → notification in the UI ``` -## Class and Function Hierarchy +The display service picks up the new config through its config hot reload +(ConfigService) without a restart. -### Frontend (JavaScript) +JSON clients can post `{"plugin_id": ..., "config": {...}}` instead; the keys +sent are merged onto the stored config the same way. See +[REST_API_REFERENCE.md](REST_API_REFERENCE.md#save-plugin-configuration). -``` -Window Load - └── DOMContentLoaded - └── refreshPlugins() - ├── fetch('/api/v3/plugins/installed') - ├── renderInstalledPlugins(plugins) - └── generatePluginTabs(plugins) - └── For each plugin: - ├── Create tab button - ├── Create tab content - └── generatePluginConfigForm(plugin) - ├── Read config_schema_data - ├── Read current config - └── Generate form HTML - ├── Boolean → Toggle switch - ├── Number → Number input - ├── String → Text input - ├── Array → Comma-separated input - └── Enum → Select dropdown +### 3. Reset -User Interactions - ├── configurePlugin(pluginId) - │ └── showTab(`plugin-${pluginId}`) - │ - ├── savePluginConfiguration(pluginId) - │ ├── Process form data - │ ├── Convert types per schema - │ └── For each field: - │ └── POST /api/v3/plugins/config - │ - └── resetPluginConfig(pluginId) - ├── Get schema defaults - └── For each field: - └── POST /api/v3/plugins/config -``` - -### Backend (Python) - -``` -Flask Routes - ├── /api/v3/plugins/installed (GET) - │ └── api_plugins_installed() - │ ├── PluginManager.discover_plugins() - │ ├── For each plugin: - │ │ ├── PluginManager.get_plugin_info() - │ │ ├── Load config_schema.json - │ │ └── Load config from config.json - │ └── Return JSON response - │ - └── /api/v3/plugins/config (POST) - └── api_plugin_config() - ├── Parse request JSON - ├── Load current config - ├── Update config[plugin_id][key] = value - └── Save config.json -``` - -## File Structure - -``` -LEDMatrix/ -│ -├── web_interface_v2.py -│ └── Flask backend with plugin API endpoints -│ -├── templates/ -│ └── index_v2.html -│ └── Frontend with dynamic tab generation -│ -├── config/ -│ └── config.json -│ └── Stores all plugin configurations -│ -├── plugins/ -│ ├── hello-world/ -│ │ ├── manifest.json ← Plugin metadata -│ │ ├── config_schema.json ← UI schema definition -│ │ ├── manager.py ← Plugin logic -│ │ └── requirements.txt -│ │ -│ └── clock-simple/ -│ ├── manifest.json -│ ├── config_schema.json -│ └── manager.py -│ -└── docs/ - ├── PLUGIN_CONFIGURATION_TABS.md ← Full documentation - ├── PLUGIN_CONFIG_TABS_SUMMARY.md ← Implementation summary - ├── PLUGIN_CONFIG_QUICK_START.md ← Quick start guide - └── PLUGIN_CONFIG_ARCHITECTURE.md ← This file -``` +`POST /api/v3/plugins/config/reset` replaces the plugin's section with the +schema defaults (keeping secrets unless `preserve_secrets` is false). ## Key Design Decisions -### 1. Dynamic Tab Generation +### 1. Server-side rendered forms -**Why**: Plugins are installed/uninstalled dynamically -**How**: JavaScript creates/removes tab elements on plugin list refresh -**Benefit**: No server-side template rendering needed +**Why**: One renderer for every plugin, no per-plugin frontend code +**How**: Jinja macros in `partials/plugin_config.html` walk the schema +**Benefit**: The settings search index is built from the same rendered HTML +(`/v3/settings/search-index`) -### 2. JSON Schema as Source of Truth +### 2. JSON Schema as source of truth -**Why**: Standard, well-documented, validation-ready -**How**: Frontend interprets schema to generate forms -**Benefit**: Plugin developers use familiar format +**Why**: Standard, well-documented, validation-ready +**How**: The same schema drives the form, the defaults and server-side validation +**Benefit**: Plugin developers use a familiar format -### 3. Individual Config Updates +### 3. Whole-form saves that merge -**Why**: Simplifies backend API -**How**: Each field saved separately via `/api/v3/plugins/config` -**Benefit**: Atomic updates, easier error handling +**Why**: A partial form (or a field the form doesn't show) must not wipe +stored values +**How**: The handler starts from the stored section and merges what was posted +**Benefit**: One request per save, atomic write -### 4. Type Conversion in Frontend +### 4. Secrets kept out of config.json -**Why**: HTML forms only return strings -**How**: JavaScript converts based on schema type before sending -**Benefit**: Backend receives correctly-typed values - -### 5. No Nested Objects - -**Why**: Keeps UI simple -**How**: Only flat property structures supported -**Benefit**: Easy form generation, clear to users +**Why**: `config.json` is shown in the raw editor and returned by the API +**How**: `"x-secret": true` fields go to `config_secrets.json`, which is +deep-merged back into the plugin's config at load time +**Benefit**: Plugins read secrets with plain `config.get(...)` ## Extension Points -### Adding New Input Types +### Custom input widgets -Location: `generatePluginConfigForm()` in `index_v2.html` +Set `"x-widget": ""` on a property. Core widgets are in +`web_interface/static/v3/js/widgets/` (see its README); a plugin can ship its +own widget script, served from `/static/plugin-widgets//.js`. +See [widget-guide.md](widget-guide.md). -```javascript -if (type === 'your-new-type') { - formHTML += ` - - `; -} -``` +### Custom actions -### Custom Validation +Buttons that run plugin scripts are declared in the manifest's +`web_ui_actions`. See [PLUGIN_WEB_UI_ACTIONS.md](PLUGIN_WEB_UI_ACTIONS.md). -Location: `savePluginConfiguration()` in `index_v2.html` +### Reacting to changes -```javascript -// Add validation before sending -if (!validateCustomConstraint(value, propSchema)) { - throw new Error('Validation failed'); -} -``` +Implement `on_config_change(new_config)` in the plugin (see +[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md)). -### Backend Hook +## Where to Look -Location: `api_plugin_config()` in `web_interface_v2.py` - -```python -# Add custom logic before saving -if plugin_id == 'special-plugin': - value = transform_value(value) -``` - -## Performance Considerations - -### Frontend - -- **Tab Generation**: O(n) where n = number of plugins (typically < 20) -- **Form Generation**: O(m) where m = number of config properties (typically < 10) -- **Memory**: Each plugin tab ~5KB HTML -- **Total Impact**: Negligible for typical use cases - -### Backend - -- **Schema Loading**: Cached after first load -- **Config Updates**: Single file write (atomic) -- **API Calls**: One per config field on save (sequential) -- **Optimization**: Could batch updates in single API call - -## Security Considerations - -1. **Input Validation**: Schema constraints enforced client-side (UX) and should be enforced server-side -2. **Path Traversal**: Plugin paths validated against known plugin directory -3. **XSS**: All user inputs escaped before rendering in HTML -4. **CSRF**: Flask CSRF tokens should be used in production -5. **File Permissions**: config.json requires write access +| Concern | File | +|---------|------| +| Tab partial loader | `web_interface/blueprints/pages_v3.py` (`_load_plugin_config_partial`) | +| Form template and field macros | `web_interface/templates/v3/partials/plugin_config.html` | +| Save / get / schema / reset handlers | `web_interface/blueprints/api_v3/plugins.py` | +| Schema loading, defaults, validation | `src/plugin_system/schema_manager.py` | +| Secret masking and splitting | `src/web_interface/secret_helpers.py` | +| Widgets | `web_interface/static/v3/js/widgets/` | ## Error Handling -### Frontend - -- Network errors: Show notification, don't crash -- Schema errors: Graceful fallback to no config tab -- Type errors: Log to console, continue processing other fields - -### Backend - -- Invalid plugin_id: 400 Bad Request -- Schema not found: Return null, frontend handles gracefully -- Config save error: 500 Internal Server Error with message - -## Testing Strategy - -### Unit Tests - -- `generatePluginConfigForm()` for each schema type -- Type conversion logic in `savePluginConfiguration()` -- Backend schema loading logic - -### Integration Tests - -- Full save flow: form → API → config.json -- Tab generation from API response -- Reset to defaults - -### E2E Tests - -- Install plugin → verify tab appears -- Configure plugin → verify config saved -- Uninstall plugin → verify tab removed - -## Monitoring - -### Frontend Metrics - -- Time to generate tabs -- Form submission success rate -- User interactions (configure, save, reset) - -### Backend Metrics - -- API response times -- Config update success rate -- Schema loading errors - -### User Feedback - -- Are users finding the configuration interface? -- Are validation errors clear? -- Are default values sensible? - -## Future Roadmap - -### Phase 2: Enhanced Validation -- Real-time validation feedback -- Custom error messages -- Dependent field validation - -### Phase 3: Advanced Inputs -- Color pickers for RGB arrays -- File upload for assets -- Rich text editor for descriptions - -### Phase 4: Configuration Management -- Export/import configurations -- Configuration presets -- Version history/rollback - -### Phase 5: Developer Tools -- Schema editor in web UI -- Live preview while editing schema -- Validation tester - +- Unknown plugin or unreadable schema: the partial renders an error message +- Validation failure: `400` with `details` and `context.validation_errors`; + the form shows them and nothing is saved +- Save failure: `500` with an error message; config.json is written + atomically, so a failed save leaves the previous file intact