rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-10-04 06:15:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(web): ES-module page lifecycle and one schema field model (stage 1) (#703)

Browse Source 
Adds a native ES-module layer to the web UI (core/boot, registry, api, facade; window.LEDMatrix as the one global), a page lifecycle that the Cache tab is converted to as the reference, text/javascript serving and revalidation for unversioned module requests, and src/plugin_system/field_model.py with a parity test against the render_field macro. Also: the cache page toggles its grey 'Not configured' style instead of only adding it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-10-01 10:00:26 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 795834811f
commit c6701ac00d
27 changed files with 3006 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/HOW_TO_RUN_TESTS.md
+25
View File
@@ -170,6 +170,31 @@ pytest test/test_config_manager.py
pytest
```
### Web UI JavaScript Tests
The suites in `test/js` need node; the DOM ones also need jsdom and a running
web interface (details in [`test/js/README.md`](../test/js/README.md)):
```bash
npm install --no-audit --no-fund --prefix test/js # jsdom; node_modules is gitignored
EMULATOR=true python3 web_interface/app.py # in another shell
BASE=http://localhost:5000 REQUIRE_DOM=1 node test/js/run_all.js
```
`pytest test/test_js_unit_suites.py` runs just the unit suites.
### Plugin Config Form Parity
`test/test_field_model_parity.py` checks `build_field_model` against the
`render_field` macro for every plugin schema it finds
([WEB_FRONTEND_ARCHITECTURE.md](WEB_FRONTEND_ARCHITECTURE.md)). It always
covers `plugin-repos/` and the test fixtures; point it at a checkout of the
official plugins to cover those too:
```bash
LEDMATRIX_MONOREPO_PLUGINS=../ledmatrix-plugins/plugins pytest test/test_field_model_parity.py
```
### Debug a Failing Test
```bash
docs/README.md
+1
View File
@@ -72,6 +72,7 @@ Going deeper:
## Contributing to LEDMatrix itself
- [ARCHITECTURE.md](ARCHITECTURE.md) — processes, display loop, plugin system, web UI; where to start reading
- [WEB_FRONTEND_ARCHITECTURE.md](WEB_FRONTEND_ARCHITECTURE.md) — the web UI's ES modules, page lifecycle and form model, and the page-by-page migration to them
- [DEVELOPMENT.md](DEVELOPMENT.md) — environment setup
- [HOW_TO_RUN_TESTS.md](HOW_TO_RUN_TESTS.md) — running the test suite
- [MULTI_ROOT_WORKSPACE_SETUP.md](MULTI_ROOT_WORKSPACE_SETUP.md) — multi-repo workspace
docs/WEB_FRONTEND_ARCHITECTURE.md
+284
View File
@@ -0,0 +1,284 @@
# Web frontend architecture
This page covers where the web UI's JavaScript is going and how it gets
there one page at a time. The UI is Flask + HTMX + Alpine.js. Templates
live in `web_interface/templates/v3/` and static files in
`web_interface/static/v3/`.
Two rules hold at every step:
- **The Pi never builds anything.** It serves the files that are committed.
CI builds the generated CSS (`scripts/build_css.py`, see #685) and checks
it. The JavaScript needs no build at all: it is native ES modules that the
browser loads as they are.
- **Every page keeps working, and so does every plugin.** Third-party plugin
forms, `x-widget` scripts and plugin web UIs use the existing `window.*`
names. Each name keeps working as an alias until a release announces that
it will be removed.
## Where it started
- About 195 `window.*` globals. Their load order is held together by comments
repeated in the headers of `app-early.js`, `app-shell.js` and
`plugins_manager.js`.
- About 5,700 lines of inline `<script>` in the tab partials.
`js/htmx-config.js` re-runs every one of them after every htmx swap, so
each partial's code had to cope with running twice.
- The installed-plugin list is kept in four places.
- Plugin config forms are drawn by the `render_field` macro in
`partials/plugin_config.html`, which is about 1,100 lines of Jinja. It
duplicates the JS widgets. The server then needs about 430 lines to
rebuild JSON from the flat dotted keys the form posts. The soccer form
renders to 1.2 MB of HTML.
- `plugins_manager.js` is 3,800 lines. The owner decided that it needs a
namespace refactor before it is split, which is what this plan provides.
## Target
```
static/v3/js/
core/ ES modules ("type": "module" in core/package.json)
boot.js entry point; base.html loads it with <script type="module">
registry.js page lifecycle: init/destroy on htmx swaps
api.js fetch wrapper for /api/v3 (JSON envelope, login redirect)
facade.js window.LEDMatrix and deprecated aliases
(later) escape.js, notify.js, dialog.js, streams.js, visibility.js,
store.js (the one installed-plugin store), form/renderer.js
pages/ one module per tab partial
cache.js export init(root, ctx), destroy(root, ctx)
...
```
### The page lifecycle
A converted partial has no `<script>`. Its root element names its page:
```html
<div class="..." data-page="cache"> ... </div>
```
`core/boot.js` registers each page with a loader:
`registry.register('cache', () => import('../pages/cache.js'))`. A page's
module is fetched only when its partial first appears.
`core/registry.js` handles the rest:
| Event | What the registry does |
|---|---|
| `htmx:beforeSwap` (on `document`, so it runs after the body-level handlers that can veto a swap) | If `detail.shouldSwap` is still true, destroys every mounted page inside the swap target |
| `htmx:afterSwap` | Destroys any mounted page whose root has left the document, then mounts every `data-page` root not mounted yet |
| `LEDMatrix.pages.refresh()` | Same as afterSwap. `loadPartialDirect` (the no-htmx fallback in `base.html`) calls it |
| `start()` | Mounts whatever is already on the page. Module scripts run deferred, so a partial may arrive first |
Mounting is idempotent: a root is never initialised twice.
Each mount gets a `ctx` object:
| Field | Contents |
|---|---|
| `ctx.root` | The page's root element |
| `ctx.name` | The page name |
| `ctx.signal` | An `AbortSignal` that is aborted after `destroy()` |
| `ctx.state` | A per-mount object for the page's own state |
| `ctx.api` | Shared service from `boot.js` |
| `ctx.notify` | Shared service from `boot.js` |
A page that passes `{ signal: ctx.signal }` to `addEventListener` and
`fetch` needs no teardown code. Its listeners and in-flight requests go
away when the partial is swapped out. `pages/cache.js` is the worked
example: its delete buttons use one delegated listener, rows are built with
`textContent` rather than markup strings, and a newer load supersedes an
older one.
### One facade
`window.LEDMatrix` is the only global the module code adds:
| Member | What it is |
|---|---|
| `api` | `core/api.js`: `get`/`post`/`put`/`del`. Resolves to the JSON body, rejects with an `ApiError` |
| `pages` | `register`, `refresh`, `list` |
| `notify(message, type)` | Calls `window.showNotification`, looked up at call time |
| `escape` | Read-through to `window.LEDEscape` |
| `widgets` | Read-through to `window.LEDMatrixWidgets` |
| `deprecate(name, target, replacement)` | Keeps an old `window.*` name working. It warns once in the console, then forwards |
`ApiError` carries `status`, `body`, `network` and `loginRequired`.
`escape`, `widgets` and `notify` are read at call time. The classic scripts
that define them are deferred, and a plugin may replace them.
Login: `base.html` wraps `window.fetch` before any other script runs, and
the wrapper sends a 401 with `X-LEDMatrix-Login` to the login page (#683).
`api.js` calls `window.fetch` at call time, so its requests get the same
redirect. It also rejects that answer quietly with `loginRequired`, so no
error message flashes up while the page navigates away.
### Serving modules from the Pi
- **MIME type.** A browser runs a module only when it is served with a
JavaScript MIME type. `app.py` pins `.js` and `.mjs` to `text/javascript`
rather than trusting the host's mimetypes table, and
`test/web_interface/test_es_modules.py` checks it.
- **Caching.** `url_for` adds `?v=<mtime>` to the entry script, but modules
import each other by plain relative URL, without the version. A static
`.js` request without `v` is therefore served `Cache-Control: no-cache`
(revalidated, so 304 when unchanged) instead of being cached as immutable
for a year. Versioned URLs keep the long cache. A later optimisation is an
import map that maps each module to its versioned URL.
- **Load order.** `boot.js` loads after every classic script. Modules are
deferred and run in document order with the deferred classic scripts.
Nothing classic may depend on a module at load time. A classic script that
needs a module service calls `window.LEDMatrix` at run time.
### One form model
`src/plugin_system/field_model.py` provides
`build_field_model(schema, config, plugin_id)`. It walks a plugin's schema
once and returns a JSON tree with these keys for each field:
- path, label, help, widget
- starting value, default
- constraints, options, secret flag
- the exact form controls the macro posts today (`inputs`)
- the JS widget it mounts (`mount`)
`test/test_field_model_parity.py` renders the real macro for every schema it
can find and checks that the model names the same controls, with the same
starting values, in the same order, and the same widget mounts. The schemas
come from `plugin-repos/`, `test/fixtures/plugins/`, the ledmatrix-plugins
monorepo when a checkout is present, and a synthetic schema that reaches
every branch of the macro. The test was mutation-checked when it was
written. Each of these deliberate model bugs makes it fail:
- dropping the checkbox-group sentinel
- dropping a table's `00:00` time default
- picking the first matching `<select>` option instead of the last
- dropping the `None` quirk
- missing all-hidden objects
The model mirrors the macro's quirks on purpose. The parity run surfaced
these:
- 83 number fields whose schema default is `null` render `value="None"`.
- Four array fields name an `x-widget` the core does not ship (`color`,
`tag-input`) and fall back to a comma-separated text box.
- A list-typed `type` uses its first entry, so `["null", "string"]` draws a
text box.
- Eleven objects with no properties and no widget render nothing.
These get fixed once, in the renderer, after the switch below.
## Switching forms to the model, behind a flag
Stage 1 (this change) only proves the model is complete. Rendering does not
change. The switch is staged so either path can be turned back on at any
point:
1. **Model endpoint.** `GET /api/v3/plugins/config/model?plugin_id=<id>`
returns `build_field_model(schema, prepared_masked_config)`. It uses the
same preparation as the partial: defaults merged, secrets masked.
2. **Renderer module.** `core/form/renderer.js` walks the model. It draws
plain fields itself and hands every widget to `LEDMatrixWidgets` through
one `mount(el, field)` adapter. The adapter keeps plugin widgets' existing
`render(container, config, value, options)` signature (the hard
constraint in PRODUCT.md). `getValue()` results are assembled into one
JSON object.
3. **Flag.** `plugin_config.html` renders the macro unless the form-model
flag is on. The flag is a `web_interface.form_model` setting in
`config.json` (default off), plus a per-browser override
(`localStorage.ledmatrixFormModel`) so a tester can compare both paths on
one device. With the flag on, the partial renders only a
`<div data-page="plugin-config" data-plugin-id="...">` root, and
`pages/plugin-config.js` fetches the model and renders it.
4. **JSON submit.** With the flag on, Save posts
`Content-Type: application/json` to the existing
`POST /api/v3/plugins/config` JSON path (`plugin_config.py`, `is_json`).
That path already validates against the schema and keeps secrets. No
dotted keys, no `__rendered_section`, no checkbox reconstruction.
5. **Save parity test.** This gates turning the flag on by default. For every
schema, posting the macro form's data and posting the renderer's JSON
must store the same config.
6. **Retire.** Once the flag has been on by default for a release with no
regressions, the macro shrinks to a no-JS fallback for plain fields, and
the form-encoded reconstruction (`_parse_form_value_with_schema`,
`_set_nested_value`, `_set_missing_booleans_to_false` and friends in
`api_v3/__init__.py`) is deleted. The settings search index is then built
from the model instead of from rendered HTML.
## Migration order
Smallest and most isolated first. `plugins_manager.js` goes last. Line counts
are the inline script in each partial today.
| # | Page | Inline JS | Why it is here |
|---|---|---|---|
| 1 | Cache (`cache.html`) | 163 lines, now 0 | **Done in stage 1.** One endpoint pair, no globals other pages use. The reference conversion |
| 2 | Rotation (`durations.html`) | 29 | Tiny. One htmx form |
| 3 | Operation History | 293 | No globals, read-only list |
| 4 | Config Editor (`raw_json.html`) | 212 | No globals. CodeMirror is set up and torn down in init/destroy |
| 5 | Backup & Restore | 232 | 5 globals used only by its own `onclick`s; these become delegated listeners plus deprecated aliases |
| 6 | Schedule | 193 | 2 globals used as `hx-on` response handlers. Moves `hx-on` handlers into page listeners |
| 7 | General | 147 | `webLogin` global and the security section. The first page that touches login |
| 8 | Display | 231 | First page with `LEDVisibility` timers: those move to a `ctx.visibility` service that stops on destroy |
| 9 | Overview | 410 (4 scripts) | First-run surface: Getting Started, update banner, live preview. Five globals |
| 10 | WiFi | 364 | `x-data="wifiSetup()"` is defined by its own script. Moves to `Alpine.data()` registered from the module. AP-mode first screen, so it needs the AP-mode test on a real device |
| 11 | Fonts | 681 | Large, but self-contained (6 globals) |
| 12 | Logs | 801 | 14 globals, a stream and timers. Uses the visibility service from step 8 |
| 13 | Tools | 1,022 | 21 globals, MQTT bridge, Pixlet editor, diagnostics polling |
| 14 | Starlark app config, plugin config (`plugin_config.html`) | 123 + 294 | Plugin panels sit inside an Alpine `x-if` that removes them without an htmx swap. The registry's sweep covers that on the next swap; this step adds a MutationObserver or an `x-if` hook. Then the form-model flag (above) |
| 15 | Plugin Manager (`plugins.html` + `plugins_manager.js`) | 3,836-line file | Last. Split along the seams that already exist (installed grid, store, registries, Starlark section, on-demand) into `pages/plugins/*.js`. Its 42 globals become aliases. The four installed-plugin stores merge into one `core/store.js`, and `window.installedPlugins` becomes a getter over it |
The shell moves in parallel, a service at a time, with no page depending on
the order:
| Service | Current home | New module |
|---|---|---|
| `showNotification` | 4 versions | `core/notify.js` |
| The modal helper | `utils/dialog.js` | `core/dialog.js` |
| SSE streams | `app-shell.js` | `core/streams.js` |
| `LEDVisibility` | `app-shell.js` | `core/visibility.js` |
Each move leaves the old global as an alias. When the last inline script is
gone, the script re-execution in `htmx-config.js` and the "HTMX never
loaded" fallbacks in `base.html` can go too (keep the captive-page path).
## How the tests cover each step
The JS suites live in `test/js` (see `test/js/README.md` and
`docs/HOW_TO_RUN_TESTS.md`). In CI, the **Web UI JS tests** job installs
jsdom, starts the web interface and runs `node test/js/run_all.js` with
`REQUIRE_DOM=1`, so a skipped DOM suite fails the job.
`test/test_js_unit_suites.py` also runs every unit suite under pytest.
Unit suites need only node. They import the shipped modules directly:
`core/package.json` and `pages/package.json` mark those directories
`"type": "module"`.
| Suite | Kind | What it covers |
|---|---|---|
| `unit/test_page_registry.js` | Unit, minimal DOM shim | The lifecycle: one init per root, destroy on swap, a veto keeps the page, swaps elsewhere leave it alone, the sweep, lazy loading, a destroy while loading, error containment |
| `unit/test_core_modules.js` | Unit | `api.js` (envelope, errors, abort, login redirect, path check) and `facade.js` (facade, aliases) |
| `dom/test_cache_page.js` | DOM: real partial, real API shape | No inline script; one request per swap and per Refresh after five swaps; a cancelled request draws nothing; hostile keys stay text; delete, empty, error, network and login states |
| `test/web_interface/test_es_modules.py` | pytest | MIME type; `no-cache` without `?v` and immutable with it; `boot.js` loads last; every import resolves inside `core/` and `pages/`; every registered page has its module and exactly one partial root; a converted partial has no `<script>` |
| `test/test_field_model_parity.py` | pytest | The model against the macro for every available schema |
What each future step adds:
- **A page conversion** adds `dom/test_<page>_page.js`, built like the cache
suite: the real partial from the server, the real API's payload shape, N
swaps followed by one action that must make exactly one request, the
destroy and cancel behaviour, and escaping. `test_es_modules.py` picks up
the new page automatically. A unit suite that today slices a function out
of a template or `plugins_manager.js` and `eval`s it is rewritten to
import the module once that code moves (stage C of the plan).
- **A shell service move** adds a unit suite for the module and an alias
test showing the old global still works.
- **The form switch** adds the save parity test (macro form data and
renderer JSON store the same config, for every schema) and a DOM suite
for `pages/plugin-config.js`. Both run with the flag on and off.
- **`plugins_manager.js`.** The existing DOM suites (`test_installed_dom.js`,
`test_store_dom.js`, `test_no_double_fetch.js`) already test the real
Plugin Manager in jsdom. They stay green throughout the split and are the
gate for it, alongside the unit suites that pin its card rendering and
escaping.