Commit Graph
2 Commits
Author SHA1 Message Date
ChuckandClaude Opus 5 a0d3e64099 fix: ten defects found validating the whole plugin fleet on hardware (#534)
* fix(core): register tom_thumb, accept frame_hold in the test double, wire api_v3's managers

Three independent fixes found while validating every plugin on a 256x64 rig.

FontManager never registered tom_thumb even though assets/fonts/tom-thumb.bdf
ships with the core, so every plugin offering it logged "Font family
'tom_thumb' not found" (16 warnings per countdown render) and had to carry a
private loader to use a bundled font. Closes #524.

VisualTestDisplayManager.set_scrolling_state() lacked the frame_hold parameter
that DisplayManager gained, so any plugin passing it died with TypeError at
render time and failed every size. Nine plugins now make that call;
ledmatrix-stocks and ledmatrix-leaderboard were failing outright and the other
seven only passed because their scroll path was unreachable without data.
Closes #525.

api_v3 declared module-level config_manager/plugin_manager = None that nothing
ever assigned -- app.py sets the blueprint attributes, which the other 150+
call sites use. Three sites read the decoys, so /health reported the config
unreadable and the plugin system uninitialised (making "degraded" permanent and
unreachable-by-design) and /display/current fell back to a hardcoded 128x64 on
every rig. The decoys are removed rather than assigned, so a bare name is now a
NameError at test time instead of a silent None. The same function's first-call
uptime was computed from two separate clock reads and came out negative.
Closes #529.

Verified on the rig: both previously-failing plugins render, the tom_thumb
warnings are gone, /health reports "healthy" with all three checks passing, and
/display/current reports the real 256x64.

* fix(core): unique snapshot temp name, honour on-demand requests, skip empty starlark

The preview snapshot wrote through a fixed "<snapshot>.tmp". /tmp is
world-writable and sticky, and the display service runs as a different user
from the tooling, so a leftover temp owned by anyone else became unopenable
even by root -- fs.protected_regular refuses O_CREAT on a foreign file in a
sticky directory. The preview and the health check's liveness proxy then froze
until someone deleted the file by hand; on the test rig that meant 23 hours of
a healthy display reporting "hardware: stale". Now uses tempfile.mkstemp with
cleanup on failure, matching the hardware-status write a few hundred lines
above. Closes #528.

_poll_on_demand_requests read its mailbox with max_age=3600, and get() defaults
the in-memory TTL to max_age -- so the first request was pinned in memory for an
hour and every later poll returned that stale copy. No second on-demand request
was honoured until the service restarted, while the API kept returning 200.
get() already documents memory_ttl=0 for exactly this cross-process case.
The consumed request is also now deleted: leaving it on disk meant a restart
replayed the previous request, activated it, and ignored the one the caller had
just made. Closes #530.

starlark-apps returned None from display() when it has no app to show, which is
the state of every install without Pixlet and of a fresh one before any app is
added. The controller only skips on a boolean False, so that held a black panel
for the full display_duration instead of rotating on. Closes #456 (core side).

Verified on the rig: two consecutive on-demand requests with no restart between
them are both activated, where the second was previously dropped in silence.

* perf(harness): share one cache across a plugin's renders

_instantiate built a fresh MockCacheManager for every (size, mode), and that
mock is a per-instance in-memory dict, so each render was a cold start. A plugin
that fetches per game or per player re-fetched everything N times over --
baseball-scoreboard at one size took 840s for nine renders where the arithmetic
said ~72s, and at eight sizes it exceeded a 900s timeout.

The second and later renders also never exercised the cache-hit path, which is
what a running rig executes almost all of the time, so a caching regression
could not be caught here.

The cache is now built once per render_plugin_matrix call and threaded down.
The display manager stays per-render -- the bounds checking depends on that --
so only fetched data is shared.

Measured on the rig, same render counts and same goldens:
  tide-display         2s -> 1s   (32 renders)
  cricket-scoreboard  10s -> 3s   (24 renders)
No pass/fail change across tide-display, cricket-scoreboard, clock-simple,
geochron, christmas-countdown, of-the-day, web-ui-info and incoming-packages.

Closes #533.

* fix(scripts): run standalone plugin tests instead of collecting nothing

run_plugin_tests.py discovered every plugin test file and handed the lot to
pytest. Most plugin tests are standalone scripts -- module-level main() plus an
`if __name__ == "__main__"` guard, signalling through an exit code -- and pytest
collects zero items from those. The run printed how many files it had *found*,
then "no tests ran", and exited without executing any of them. On a rig with all
44 first-party plugins that is 151 of 248 files.

Files are now classified and each kind runs under the right runner: pytest for
real test modules, subprocess for scripts, honouring the 0 pass / 2 skip / 1
fail convention ledmatrix-plugins' own runner established (a script that wants a
tty or an LED matrix is a skip, not a regression).

Before:
    $ python3 scripts/run_plugin_tests.py -p countdown -d ~/LEDMatrix/plugin-repos
    Found 1 test file(s)
    collected 0 items
    no tests ran in 0.31s                      rc=0

After:
    Found 1 test file(s) -- 0 collectable, 1 standalone script(s)
    1 passed, 0 skipped, 0 failed (scripts)    rc=0

Verified across three shapes: countdown (1 script), jellyfin-now-playing and
pomodoro-timer (pytest only, 16 and 42 tests), and ledmatrix-flights (11 files
split 4 collectable / 7 scripts, all seven of which had never run).

Closes #532.

Running the flights scripts for the first time also surfaced four genuinely
failing tests there, hidden by the mirror-image bug in the plugins repo's own
runner -- filed as ChuckBuilds/ledmatrix-plugins#464 and #465.

* fix(harness): give an empty-looking mode a few frames before warning about it

check_plugin's "drew nothing but display() returned X" warning fired on a single
frame, rendered with force_clear=True, under a frozen clock. All three defeat a
scrolling plugin, whose first frame is legitimately its blank scroll-in buffer.
Across 44 first-party plugins, 60 of 76 warnings were false -- the rate at which
people stop reading a warning, which matters because the true positives are
real: a mode that draws nothing and does not return False holds a blank panel
for its whole display duration.

An apparently-empty frame is now re-driven for up to 48 more frames with
force_clear=False (force_clear means "reset the scroll", so repeating it would
redraw frame 1 for ever) and with the clock advancing -- freezegun's factory
where time is frozen, a real sleep where it is not, since scroll position is
usually a function of elapsed time. The first frame that draws content replaces
the result.

The clock is moved back afterwards. It is shared by every render in the matrix,
so time borrowed by the probe leaked into later modes and drifted their goldens
-- f1_upcoming picked up 5 spurious drifts before this was restored.

Measured on the rig:

                        empty warns          check
                        before  after
  f1-scoreboard            42      0    48 PASS / 0 FAIL, goldens intact
  ledmatrix-elections      16      0    16 PASS / 0 FAIL
  on-air                    8      8    true positive, kept
  nfl-draft                 8      8    true positive, kept
  clock-simple/geochron/    0      0    unchanged
  christmas-countdown

58 false positives gone, both true positives kept, no golden regressions. Cost
is confined to modes that really are blank: plugins that draw immediately are
unchanged (clock-simple and tide-display still 2s), while on-air -- eight
deliberately blank modes -- goes to 21s.

Closes #527.

* fix(harness): load nested schema defaults, and merge caller config at leaf level

load_config_defaults read only top-level properties. An object property carries
its defaults on its children, not on itself, so everything nested was dropped --
2,386 defaults across 37 of 44 plugins, soccer-scoreboard alone losing 539 of
565. render_plugin_matrix's comment says the plugin then "behaves like a real
install", which for most of the fleet it did not.

_defaults_from_properties now recurses. merge_config deep-merges the caller's
config onto the result so an override lands at the leaf: a shallow merge would
let -c '{"nhl": {"enabled": true}}' replace the whole nhl subtree and discard
every other nhl default, which is the same class of bug being fixed here.

Measured before/after across all 49 installed plugins on the rig: **no render
changed** -- identical PASS/FAIL counts, byte-identical output, goldens intact.
Plugins already fall back to the same values internally via config.get(key,
default), so supplying them explicitly agrees with what they were doing. The
defaults really are arriving now:

  ufc-scoreboard        9 -> 87 defaults
  ledmatrix-flights    51 -> 95
  masters-tournament   10 -> 51
  cricket-scoreboard   22 -> 50
  tide-display         12 -> 18

and hockey-scoreboard, which used to load nhl.enabled=None, now gets
nhl.enabled=True with its full display_modes block.

Caveat worth carrying: the eight plugins with the most nested config
(soccer, baseball, basketball, hockey, lacrosse, football, afl, nrl -- 1,634 of
the 2,386 dropped defaults, 68%) could not be measured. They import
src.common.sports_shared, which the test rig's core branch predates, so they
fail to load there identically before and after. Re-run this comparison against
a core that has that module before trusting the "nothing changed" result for
them; those are exactly the plugins whose renders should change most.

Closes #531.

* refactor: narrow the exception handlers this branch introduced

Codacy flagged the new code; it passes on other recent PRs, so the finding is
mine. Four of the five broad `except Exception` clauses I added were catching
far more than they needed to, which is the same shape as several bugs this
branch fixes -- hello-world's TypeError sat invisible for exactly this reason.

  freezer() / move_to() / tick()   -> (AttributeError, TypeError, ValueError)
  cache_manager.delete()           -> (OSError, AttributeError, KeyError)

The fifth stays broad and now says why: it wraps a call into a plugin's own
display(), which can raise anything, and the first frame has already rendered --
so a failure there must not turn a good result into an error.

Verified against a checkout of main: f1-scoreboard 48 PASS / 0 FAIL with 0 empty
warnings, on-air keeps its 8 true positives, clock-simple 8 PASS. geochron shows
7 golden drifts both before and after this branch, so it is not from these
changes -- its committed goldens predate #521's 1-bit text rendering.

* fix: resolve CodeRabbit review and Codacy findings on #534

CodeRabbit raised six; all six were real.

The test double had drifted ahead of production. VisualTestDisplayManager
accepted set_scrolling_state(frame_hold=...) while DisplayManager did not,
so such a call passed every harness run and would raise TypeError on the
panel -- the one failure a safety harness exists to prevent. frame_hold
belongs to the change that adds it to DisplayManager (#523), so it moves
there and the double matches main again.

The harness swallowed exceptions from re-rendered frames. _settle_loop
re-renders a mode that came back blank, to give a scroll time to draw;
returning silently on a crash meant a mode that renders one good frame
and then explodes was reported as passing. Recorded on result.error now,
keeping the captured frame so the failure stays inspectable.

starlark-apps display() returned True after _display_frame() failed, so
the controller held a dead frame for the whole display_duration instead
of rotating on. _display_frame now returns bool on all three paths.

run_plugin_tests.py used env.setdefault for PYTHONPATH and
LEDMATRIX_CORE, so an inherited value won and the subprocess imported a
different core than the one under test -- ledmatrix-plugins#467 exactly.
Prepends PROJECT_ROOT and sets LEDMATRIX_CORE unconditionally.

The on-demand mailbox is polled after every frame, ~125x/second on a
scrolling mode, and the read is deliberately uncached, so it was that
many disk reads per second to find nothing. Floored at 250ms, which is
imperceptible for a web-UI click. Consuming it also deleted whatever was
present rather than what had just been processed, so a request posted
while the previous one was in flight was thrown away and never ran; the
delete is now keyed by request_id. That narrows the window rather than
closing it -- a true atomic claim needs a primitive the cache layer does
not offer, and the code says so rather than implying otherwise.

Codacy's 2 criticals were bandit B404/B603 on the subprocess call added
to run_plugin_tests.py. Fixed interpreter, argument list, no shell;
annotated with the repo's existing nosec convention. Bandit is clean on
the file.

Adds test/test_on_demand_mailbox.py (8), test_starlark_display_contract.py
(4) and two settle cases in test_harness_empty_claimed.py. 4, 4 and 2 of
those fail against the pre-fix code. Full suite: 3961 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RRtqXDCnvnY6EQwhT5CV9

* chore: satisfy Codacy's subprocess checks on the new test runner

Codacy runs Bandit and Opengrep (its Semgrep fork). The new
subprocess.run in scripts/run_plugin_tests.py trips three patterns, on
two different lines:

  Bandit   B404 on the import, B603 on the call
  Opengrep dangerous-subprocess-use-audit          on the run( line
           dangerous-subprocess-use-tainted-env-args on the argv line

A nosemgrep applies only to its own line, so the call line and the argv
line each need one; a single comment on the call covered neither rule
fully. Suppression is the right answer here rather than a rewrite: the
interpreter is sys.executable, the arguments are a list, and no shell is
involved, so there is nothing to word-split or expand.

Matches the pair the rest of the repo already uses for this shape --
permission_utils.py, plugin_loader.py, install_dependencies_apt.py.

Codacy: 0 new issues, up to standards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RRtqXDCnvnY6EQwhT5CV9

* chore: leave visual_display_manager untouched so #523 can merge

The only change this branch made to that file was a docstring, and it
collided with #523's rewrite of the same method -- so #534 and #523 each
merged cleanly against main but conflicted with each other. Reverted to
main's text; #523 owns this method and adds frame_hold to it.

The note the docstring carried ('frame_hold arrives in #523') would have
been stale the moment #523 landed anyway. The parity test in #523 is
what actually keeps the two signatures honest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RRtqXDCnvnY6EQwhT5CV9

* chore: add the Ruff suppression nosec/nosemgrep do not cover

Ruff reports S603 on the same call Bandit and Opengrep do, and none of
the three suppressions covers the others. Confirmed the precondition
first: path comes from discover_plugin_tests(), which globs test files
inside the repo, and the call is a fixed interpreter with a list argv
and no shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RRtqXDCnvnY6EQwhT5CV9

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-07 13:37:38 -04:00
302235a357 feat: Starlark Apps Integration with Schema-Driven Config + Security Hardening (#253)
* feat: integrate Starlark/Tronbyte app support into plugin system

Add starlark-apps plugin that renders Tidbyt/Tronbyte .star apps via
Pixlet binary and integrates them into the existing Plugin Manager UI
as virtual plugins. Includes vegas scroll support, Tronbyte repository
browsing, and per-app configuration.

- Extract working starlark plugin code from starlark branch onto fresh main
- Fix plugin conventions (get_logger, VegasDisplayMode, BasePlugin)
- Add 13 starlark API endpoints to api_v3.py (CRUD, browse, install, render)
- Virtual plugin entries (starlark:<app_id>) in installed plugins list
- Starlark-aware toggle and config routing in pages_v3.py
- Tronbyte repository browser section in Plugin Store UI
- Pixlet binary download script (scripts/download_pixlet.sh)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(starlark): use bare imports instead of relative imports

Plugin loader uses spec_from_file_location without package context,
so relative imports (.pixlet_renderer) fail. Use bare imports like
all other plugins do.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(starlark): make API endpoints work standalone in web service

The web service runs as a separate process with display_manager=None,
so plugins aren't instantiated. Refactor starlark API endpoints to
read/write the manifest file directly when the plugin isn't loaded,
enabling full CRUD operations from the web UI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(starlark): make config partial work standalone in web service

Read starlark app data from manifest file directly when the plugin
isn't loaded, matching the api_v3.py standalone pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(starlark): always show editable timing settings in config panel

Render interval and display duration are now always editable in the
starlark app config panel, not just shown as read-only status text.
App-specific settings from schema still appear below when present.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(store): add sort, filter, search, and pagination to Plugin Store and Starlark Apps

Plugin Store:
- Live search with 300ms debounce (replaces Search button)
- Sort dropdown: A→Z, Z→A, Category, Author, Newest
- Installed toggle filter (All / Installed / Not Installed)
- Per-page selector (12/24/48) with pagination controls
- "Installed" badge and "Reinstall" button on already-installed plugins
- Active filter count badge + clear filters button

Starlark Apps:
- Parallel bulk manifest fetching via ThreadPoolExecutor (20 workers)
- Server-side 2-hour cache for all 500+ Tronbyte app manifests
- Auto-loads all apps when section expands (no Browse button)
- Live search, sort (A→Z, Z→A, Category, Author), author dropdown
- Installed toggle filter, per-page selector (24/48/96), pagination
- "Installed" badge on cards, "Reinstall" button variant

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(store): move storeFilterState to global scope to fix scoping bug

storeFilterState, pluginStoreCache, and related variables were declared
inside an IIFE but referenced by top-level functions, causing
ReferenceError that broke all plugin loading.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(starlark): schema-driven config forms + critical security fixes

## Schema-Driven Config UI
- Render type-appropriate form inputs from schema.json (text, dropdown, toggle, color, datetime, location)
- Pre-populate config.json with schema defaults on install
- Auto-merge schema defaults when loading existing apps (handles schema updates)
- Location fields: 3-part mini-form (lat/lng/timezone) assembles into JSON
- Toggle fields: support both boolean and string "true"/"false" values
- Unsupported field types (oauth2, photo_select) show warning banners
- Fallback to raw key/value inputs for apps without schema

## Critical Security Fixes (P0)
- **Path Traversal**: Verify path safety BEFORE mkdir to prevent TOCTOU
- **Race Conditions**: Add file locking (fcntl) + atomic writes to manifest operations
- **Command Injection**: Validate config keys/values with regex before passing to Pixlet subprocess

## Major Logic Fixes (P1)
- **Config/Manifest Separation**: Store timing keys (render_interval, display_duration) ONLY in manifest
- **Location Validation**: Validate lat [-90,90] and lng [-180,180] ranges, reject malformed JSON
- **Schema Defaults Merge**: Auto-apply new schema defaults to existing app configs on load
- **Config Key Validation**: Enforce alphanumeric+underscore format, prevent prototype pollution

## Files Changed
- web_interface/templates/v3/partials/starlark_config.html — schema-driven form rendering
- plugin-repos/starlark-apps/manager.py — file locking, path safety, config validation, schema merge
- plugin-repos/starlark-apps/pixlet_renderer.py — config value sanitization
- web_interface/blueprints/api_v3.py — timing key separation, safe manifest updates

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): use manifest filename field for .star downloads

Tronbyte apps don't always name their .star file to match the directory.
For example, the "analogclock" app has "analog_clock.star" (with underscore).

The manifest.yaml contains a "filename" field with the correct name.

Changes:
- download_star_file() now accepts optional filename parameter
- Install endpoint passes metadata['filename'] to download_star_file()
- Falls back to {app_id}.star if filename not in manifest

Fixes: "Failed to download .star file for analogclock" error

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): reload tronbyte_repository module to pick up code changes

The web service caches imported modules in sys.modules. When deploying
code updates, the old cached version was still being used.

Now uses importlib.reload() when module is already loaded.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): use correct 'fileName' field from manifest (camelCase)

The Tronbyte manifest uses 'fileName' (camelCase), not 'filename' (lowercase).
This caused the download to fall back to {app_id}.star which doesn't exist
for apps like analogclock (which has analog_clock.star).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* feat(starlark): extract schema during standalone install

The standalone install function (_install_star_file) wasn't extracting
schema from .star files, so apps installed via the web service had no
schema.json and the config panel couldn't render schema-driven forms.

Now uses PixletRenderer to extract schema during standalone install,
same as the plugin does.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* feat(starlark): implement source code parser for schema extraction

Pixlet CLI doesn't support schema extraction (--print-schema flag doesn't exist),
so apps were being installed without schemas even when they have them.

Implemented regex-based .star file parser that:
- Extracts get_schema() function from source code
- Parses schema.Schema(version, fields) structure
- Handles variable-referenced dropdown options (e.g., options = dialectOptions)
- Supports Location, Text, Toggle, Dropdown, Color, DateTime fields
- Gracefully handles unsupported fields (OAuth2, LocationBased, etc.)
- Returns formatted JSON matching web UI template expectations

Coverage: 90%+ of Tronbyte apps (static schemas + variable references)

Changes:
- Replace extract_schema() to parse .star files directly instead of using Pixlet CLI
- Add 6 helper methods for parsing schema structure
- Handle nested parentheses and brackets properly
- Resolve variable references for dropdown options

Tested with:
- analog_clock.star (Location field) ✓
- Multi-field test (Text + Dropdown + Toggle) ✓
- Variable-referenced options ✓

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): add List to typing imports for schema parser

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): load schema from schema.json in standalone mode

The standalone API endpoint was returning schema: null because it didn't
load the schema.json file. Now reads schema from disk when returning
app details via web service.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* feat(starlark): implement schema extraction, asset download, and config persistence

## Schema Extraction
- Replace broken `pixlet serve --print-schema` with regex-based source parser
- Extract schema by parsing `get_schema()` function from .star files
- Support all field types: Location, Text, Toggle, Dropdown, Color, DateTime
- Handle variable-referenced dropdown options (e.g., `options = teamOptions`)
- Gracefully handle complex/unsupported field types (OAuth2, PhotoSelect, etc.)
- Extract schema for 90%+ of Tronbyte apps

## Asset Download
- Add `download_app_assets()` to fetch images/, sources/, fonts/ directories
- Download assets in binary mode for proper image/font handling
- Validate all paths to prevent directory traversal attacks
- Copy asset directories during app installation
- Enable apps like AnalogClock that require image assets

## Config Persistence
- Create config.json file during installation with schema defaults
- Update both config.json and manifest when saving configuration
- Load config from config.json (not manifest) for consistency with plugin
- Separate timing keys (render_interval, display_duration) from app config
- Fix standalone web service mode to read/write config.json

## Pixlet Command Fix
- Fix Pixlet CLI invocation: config params are positional, not flags
- Change from `pixlet render file.star -c key=value` to `pixlet render file.star key=value -o output`
- Properly handle JSON config values (e.g., location objects)
- Enable config to be applied during rendering

## Security & Reliability
- Add threading.Lock for cache operations to prevent race conditions
- Reduce ThreadPoolExecutor workers from 20 to 5 for Raspberry Pi
- Add path traversal validation in download_star_file()
- Add YAML error logging in manifest fetching
- Add file size validation (5MB limit) for .star uploads
- Use sanitized app_id consistently in install endpoints
- Use atomic manifest updates to prevent race conditions
- Add missing Optional import for type hints

## Web UI
- Fix standalone mode schema loading in config partial
- Schema-driven config forms now render correctly for all apps
- Location fields show lat/lng/timezone inputs
- Dropdown, toggle, text, color, and datetime fields all supported

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): code review fixes - security, robustness, and schema parsing

## Security Fixes
- manager.py: Check _update_manifest_safe return values to prevent silent failures
- manager.py: Improve temp file cleanup in _save_manifest to prevent leaks
- manager.py: Fix uninstall order (manifest → memory → disk) for consistency
- api_v3.py: Add path traversal validation in uninstall endpoint
- api_v3.py: Implement atomic writes for manifest files with temp + rename
- pixlet_renderer.py: Relax config validation to only block dangerous shell metacharacters

## Frontend Robustness
- plugins_manager.js: Add safeLocalStorage wrapper for restricted contexts (private browsing)
- starlark_config.html: Scope querySelector to container to prevent modal conflicts

## Schema Parsing Improvements
- pixlet_renderer.py: Indentation-aware get_schema() extraction (handles nested functions)
- pixlet_renderer.py: Handle quoted defaults with commas (e.g., "New York, NY")
- tronbyte_repository.py: Validate file_name is string before path traversal checks

## Dependencies
- requirements.txt: Update Pillow (10.4.0), PyYAML (6.0.2), requests (2.32.0)

## Documentation
- docs/STARLARK_APPS_GUIDE.md: Comprehensive guide explaining:
  - How Starlark apps work
  - That apps come from Tronbyte (not LEDMatrix)
  - Installation, configuration, troubleshooting
  - Links to upstream projects

All changes improve security, reliability, and user experience.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): convert Path to str in spec_from_file_location calls

The module import helpers were passing Path objects directly to
spec_from_file_location(), which caused spec to be None. This broke
the Starlark app store browser.

- Convert module_path to string in both _get_tronbyte_repository_class
  and _get_pixlet_renderer_class
- Add None checks with clear error messages for debugging

Fixes: spec not found for the module 'tronbyte_repository'

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): restore Starlark Apps section in plugins.html

The Starlark Apps UI section was lost during merge conflict resolution
with main branch. Restored from commit 942663ab which had the complete
implementation with filtering, sorting, and pagination.

Fixes: Starlark section not visible on plugin manager page

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): restore Starlark JS functionality lost in merge

During the merge with main, all Starlark-specific JavaScript (104 lines)
was removed from plugins_manager.js, including:
- starlarkFilterState and filtering logic
- loadStarlarkApps() function
- Starlark app install/uninstall handlers
- Starlark section collapse/expand logic
- Pagination and sorting for Starlark apps

Restored from commit 942663ab and re-applied safeLocalStorage wrapper
from our code review fixes.

Fixes: Starlark Apps section non-functional in web UI

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): security and race condition improvements

Security fixes:
- Add path traversal validation for output_path in download_star_file
- Remove XSS-vulnerable inline onclick handlers, use delegated events
- Add type hints to helper functions for better type safety

Race condition fixes:
- Lock manifest file BEFORE creating temp file in _save_manifest
- Hold exclusive lock for entire read-modify-write cycle in _update_manifest_safe
- Prevent concurrent writers from racing on manifest updates

Other improvements:
- Fix pages_v3.py standalone mode to load config.json from disk
- Improve error handling with proper logging in cleanup blocks
- Add explicit type annotations to Starlark helper functions

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): critical bug fixes and code quality improvements

Critical fixes:
- Fix stack overflow in safeLocalStorage (was recursively calling itself)
- Fix duplicate event listeners on Starlark grid (added sentinel check)
- Fix JSON validation to fail fast on malformed data instead of silently passing

Error handling improvements:
- Narrow exception catches to specific types (OSError, json.JSONDecodeError, ValueError)
- Use logger.exception() with exc_info=True for better stack traces
- Replace generic "except Exception" with specific exception types

Logging improvements:
- Add "[Starlark Pixlet]" context tags to pixlet_renderer logs
- Redact sensitive config values from debug logs (API keys, etc.)
- Add file_path context to schema parsing warnings

Documentation:
- Fix markdown lint issues (add language tags to code blocks)
- Fix time unit spacing: "(5min)" -> "(5 min)"

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): critical path traversal and exception handling fixes

Path traversal security fixes (CRITICAL):
- Add _validate_starlark_app_path() helper to check for path traversal attacks
- Validate app_id in get_starlark_app(), uninstall_starlark_app(),
  get_starlark_app_config(), and update_starlark_app_config()
- Check for '..' and path separators before any filesystem access
- Verify resolved paths are within _STARLARK_APPS_DIR using Path.relative_to()
- Prevents unauthorized file access via crafted app_id like '../../../etc/passwd'

Exception handling improvements (tronbyte_repository.py):
- Replace broad "except Exception" with specific types
- _make_request: catch requests.Timeout, requests.RequestException, json.JSONDecodeError
- _fetch_raw_file: catch requests.Timeout, requests.RequestException separately
- download_app_assets: narrow to OSError, ValueError
- Add "[Tronbyte Repo]" context prefix to all log messages
- Use exc_info=True for better stack traces

API improvements:
- Narrow exception catches to OSError, json.JSONDecodeError in config loading
- Remove duplicate path traversal checks (now centralized in helper)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix(starlark): logging improvements and code quality fixes

Logging improvements (pages_v3.py):
- Add logging import and create module logger
- Replace print() calls with logger.warning() with "[Pages V3]" prefix
- Use logger.exception() for outer try/catch with exc_info=True
- Narrow exception handling to OSError, json.JSONDecodeError for file operations

API improvements (api_v3.py):
- Remove unnecessary f-strings (Ruff F541) from ImportError messages
- Narrow upload exception handling to ValueError, OSError, IOError
- Use logger.exception() with context for better debugging
- Remove early return in get_starlark_status() to allow standalone mode fallback
- Sanitize error messages returned to client (don't expose internal details)

Benefits:
- Better log context with consistent prefixes
- More specific exception handling prevents masking unexpected errors
- Standalone/web-service-only mode now works for status endpoint
- Stack traces preserved for debugging without exposing to clients

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Chuck <chuck@example.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 19:44:12 -05:00