Compare commits

..
29 Commits
Author SHA1 Message Date
Chuck 853b5aa618 Merge remote-tracking branch 'origin/main' into claude/scan-order-compensation
# Conflicts:
#	CHANGELOG.md
#	src/display_manager.py
2026-09-24 19:39:12 -04:00
Chuck 86c27970ef Merge branch 'claude/frame-timing-harness' into claude/scan-order-compensation 2026-09-24 18:58:49 -04:00
ChuckandClaude Opus 5.5 ab39552cdb refactor(display): stop the frame-timing watchdog at the end of cleanup()
#628 adds its snapshot-writer stop at the top of cleanup(); with this at the
top as well, merging #629 after #628 conflicted. Same behaviour, no
overlap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 18:58:42 -04:00
Chuck 83fd3491c7 Merge branch 'claude/frame-timing-harness' into claude/scan-order-compensation 2026-09-24 18:34:11 -04:00
ChuckandClaude Opus 5.5 37203e635c fix(perf): recorder review fixes -- period adoption, timed frames, snapshot copy, watchdog stop
From review (CodeRabbit):

- The first refresh period is adopted only once two windows in a row agree.
  One loaded startup window, most frames a refresh late, used to fix a
  period twice the real one for the life of the process.
- totals.timed_frames counts the frames judged against a known period, and
  frame_soak rates late and early frames over it. Before, frames seen before
  any period was known counted as on time, so a short run could PASS
  having judged nothing; now it has no verdict.
- snapshot() copies totals. render_bench kept the snapshot object and
  differenced it against a later one sharing the same live dict, so every
  graded run reported zero frames.
- held_refresh_hz uses the p50 bucket's midpoint, not its upper edge (a
  1-2% low bias, the size of the idle-vs-held gap it exists to show).
- StallWatchdog.stop() and FrameTimingRecorder.close(); DisplayManager's
  cleanup() calls it, so the watchdog no longer outlives its manager.
- A stall that outlasts the 2s scroll-state expiry still gets its end
  reported (tracking is dropped only past GAP_SECONDS).
- test: EMULATOR through monkeypatch; docs: the 1s resume rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 18:33:34 -04:00
Chuck 695be34a1b Merge branch 'claude/frame-timing-harness' into claude/scan-order-compensation 2026-09-24 18:22:42 -04:00
ChuckandClaude Opus 5.5 ffdd0efd43 style(bench): say when the panel could not be blanked after a run (Codacy B110)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 18:22:38 -04:00
Chuck cfcdbfb18a Merge branch 'claude/frame-timing-harness' into claude/scan-order-compensation 2026-09-24 17:44:33 -04:00
Chuck 4c17e18786 Merge commit 'b9416ef8' into claude/frame-timing-harness 2026-09-24 17:42:43 -04:00
Chuck 764fc6fcdb Merge branch 'claude/frame-timing-harness' into claude/scan-order-compensation 2026-09-24 17:36:54 -04:00
Chuck 99a3608516 Merge remote-tracking branch 'origin/main' into claude/frame-timing-harness 2026-09-24 17:36:48 -04:00
Chuck b1510c209d Merge branch 'claude/frame-timing-harness' into claude/scan-order-compensation
# Conflicts:
#	CHANGELOG.md
2026-09-24 17:36:39 -04:00
Chuck 48a433328a Merge remote-tracking branch 'origin/main' into claude/scan-order-compensation
# Conflicts:
#	CHANGELOG.md
2026-09-24 17:36:25 -04:00
Chuck edfcd9e2a1 Merge remote-tracking branch 'origin/main' into claude/frame-timing-harness
# Conflicts:
#	CHANGELOG.md
2026-09-24 17:35:58 -04:00
Chuck 74ba36a059 Merge remote-tracking branch 'origin/claude/frame-timing-harness' into claude/scan-order-compensation 2026-09-24 17:02:24 -04:00
ChuckandClaude Opus 5.5 d37a3a712a style(perf): say why render_bench fell back; mark the stats path as a safe fixed name (Codacy)
Codacy (Bandit B110, B108). The bench's silent except now prints why it read
config.json directly. The stats file's fixed name in /dev/shm is safe:
write() goes through mkstemp and os.replace, which replaces a planted
symlink instead of following it; the comment says so and marks it nosec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 17:02:15 -04:00
Chuck dc26baa134 Merge remote-tracking branch 'origin/claude/frame-timing-harness' into claude/scan-order-compensation
# Conflicts:
#	src/display_manager.py
2026-09-24 16:58:46 -04:00
ChuckandClaude Opus 5.5 6aef54598b docs(display): the mid-panel tear can be compensated; say how, and when it cannot
main's new section called the 1px step at mid-height a panel-scan effect with
nothing to fix in the render path. The scan explanation holds, but at one
pixel per refresh the step is one refresh of motion and lagging one half
cancels it, which this branch does. Rewrite that paragraph, list what
compensation covers and what it leaves to the old advice (held frames,
other layouts, the emulator), and add a CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 16:52:33 -04:00
Chuck 7b16e953d2 Merge remote-tracking branch 'origin/main' into claude/scan-order-compensation 2026-09-24 16:49:41 -04:00
Chuck 52bc520335 Merge remote-tracking branch 'origin/main' into claude/frame-timing-harness
# Conflicts:
#	CHANGELOG.md
#	docs/SCROLL_PERFORMANCE.md
2026-09-24 16:30:49 -04:00
ChuckandClaude Opus 5.5 14c38a3189 fix(perf): count a stall even when the scroll state went missing across it
On hdpi the stall watchdog logged a 1.9s stall during the hourly sports
refresh that the soak report never had: its worst gap was 655ms. The frame
that ended the stall was recorded as static, so its interval was dropped.

"Scrolling" is DisplayManager's scroll state at the moment a frame is
presented, and it goes missing mid-scroll: it expires after 2s without
activity, and any thread can clear it. Plugins call
set_scrolling_state(False) from their own display() (news, stocks, the odds
ticker's fallback), and Vegas captures some of those on the render thread
between two of its own frames. Vegas sets the state again only after its
next frame, so that frame is recorded as static -- along with the capture
or stall it followed.

One static frame between two scrolling frames, with the scroll resuming
within RESUME_SECONDS (1s), is now a frame of the scroll and both of its
intervals count, the first at the scroll's own hold (clearing the state
drops the hold to 1 too). Two static frames in a row still end the scroll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 15:23:18 -04:00
ChuckandClaude Opus 5.5 b67818d5c5 feat(perf): LEDMATRIX_STALL_WATCHDOG_MS lowers the stall watchdog's threshold
250ms catches freezes; the hitches left on hdpi are frames 2-5 refreshes
late, which look like the render thread waiting for the GIL. At 30ms the
watchdog dumps those too, naming what the other threads were running when
the frame missed. It polls at a third of the threshold so a stall one poll
long is still seen, which costs some GIL time of its own: a diagnostic
setting, not one to soak with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 14:26:04 -04:00
ChuckandClaude Opus 5.5 c5281d9a45 feat(display): compensate for the panel's scan order while scrolling
A 1:N-scan HUB75 panel lights its rows in pairs, row d of the top half
with row d of the bottom half, d running 0..N-1 across each refresh. The
two rows either side of the middle of a panel are therefore lit at
opposite ends of every refresh, and a strip moving a whole pixel per
refresh shows a crisp 1px step across the middle of every panel -- in a
phone video as well as by eye.

Established on hdpi's panel (4x128x64, one chain, rotated 180) on
2026-09-24: interlaced scanning (scan_mode 1) made the step vanish, and
halving the scroll speed halved it. It is the scan order, not a torn
frame, and crisp vsync-locked pacing (#523, #628) makes it visible where
uneven, blended motion used to hide it. Showing the upper half one
refresh behind removed it completely at full speed.

src/scan_order.py works out which rows lag how many refreshes from the
layout: walking the logical rows, wherever a row lit near the start of a
refresh follows one lit near the end, the section below takes one more
refresh of lag (one less the other way), so the result is a uniform lean
rather than a step. Stacked parallel chains lean further. It covers plain
and parallel chains at 0 or 180 degrees with standard multiplexing and
progressive scan; anything else (U-mapper, 90/270, multiplexing,
interlaced, double-sided) is left alone, as is the emulator, which has no
scan order.

DisplayManager applies it only mid-scroll at one frame per refresh, when
consecutive frames are consecutive refreshes: lagging rows come from the
previous input frames, so it works for Vegas and every plugin ticker
without knowing how they scroll. display.scan_order_compensation
("auto" | "off") controls it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 13:36:51 -04:00
ChuckandClaude Opus 5.5 f79618d4f7 refactor(bench): grade render_bench with the shared frame-timing recorder
render_bench.py (from the parallel perf/render-bench work) had its own
grading module, frame_pacing, with its own definition of a missed frame
and its own refresh estimate. The soak already had both in frame_timing,
so the two could have drifted apart on what "late" means.

The bench now gives the display manager a fresh FrameTimingRecorder,
drains it synchronously at the start and end of the graded run, and prints
frame_soak's report with frame_soak's verdict. Its workload is unchanged:
the synthetic strip, --busy load, the shared speed resolver, the
per-frame scrolling announcement. frame_pacing, its tests and its
src.common exports are removed; measure_refresh_hz moves to frame_timing,
where scroll_speeds.py now finds it.

Two ideas from frame_pacing carry over. The bench seeds the recorder with
the idle refresh it measures, so a loop that free-runs (the 827fps bug
the first bench caught) shows as early frames and one stuck at half rate
as late frames, where an estimate taken from their own intervals finds
both self-consistent. And the soak, which has no idle measurement, now
calls a run NOT LOCKED when its refresh estimate beats the configured cap.
The report also gives the rate held while rendering.

Docs: the bench becomes "Without the service" under "Soaking a rig",
keeping its hdpi numbers and the idle-vs-rendering refresh finding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 11:56:44 -04:00
ChuckBuildsandClaude Opus 5 d56ec2ab3a feat(bench): measure a rig against the refresh it actually holds
There was no way to answer "does this hardware present every frame on
time?" other than watching the panel. `scripts/render_bench.py` drives the
production path -- a real DisplayManager and ScrollHelper, configured
through the same `scroll_config` resolver every ticker uses -- and grades
the run with a new `src.common.frame_pacing`, exiting non-zero when more
than 0.1% of frames slipped a refresh. Exit 2 when the run could not be set
up at all, so a rig that was never measured cannot pass by accident.

A missed frame is defined exactly: an interval that rounds up to at least
one more refresh than its frame hold asked for. The half-refresh rounding
boundary keeps a frame that ran 1ms long on a 10ms refresh out of the
count, because it still presented on the refresh it was meant to.

The verdict that matters more is NOT LOCKED. A loop that never blocked on
vsync reports a perfect zero misses while presenting nothing -- 8ms frames
on a 100Hz panel all land in the one-refresh bucket while running 25% too
fast -- so the report also checks the typical frame is not shorter than the
panel could physically present. That is what caught the first version of
this benchmark announcing its scrolling state once instead of per frame:
the state expires on an inactivity threshold, the dirty-tracking skip then
fires mid-scroll, and the loop free-ran at 827fps.

And the refresh is read back out of the frames rather than taken from an
idle measurement. Driving the matrix is bit-banging on the same machine, so
pushing frames slows the refresh: a Pi 4 on 512x64 measures 100.4Hz idle
and holds 96.3Hz while scrolling. Both are real, and grading against the
idle figure reports a locked loop as 4% slow -- or, once the gap passes
half a refresh, as missing every frame. The gap between the two is itself
worth watching: a rise in it is a render-cost regression even when nothing
is missed.

Measured on hdpi (Pi 4, 512x64, pwm_bits 8), two minutes each:

  plain      95.44 fps, 8 missed of 11,449 (0.070%)  PASS
  --busy 2   95.41 fps, 3 missed of 11,445 (0.026%)  PASS

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014RRtqXDCnvnY6EQwhT5CV9
2026-09-24 11:08:43 -04:00
ChuckandClaude Opus 5.5 c883a2fd1e feat(perf): a stall watchdog that logs what the render thread is waiting on
The recorder counts freezes; it cannot say why. hdpi showed 1-2s freezes
in both the #628 and offscreen builds, one lining up with hockey's 2s
NHL fetch on the update thread, and nothing in the logs explained it.

StallWatchdog polls every 50ms from its own thread. When a scroll's last
frame is more than 250ms old (and a scroll is still running, so the end
of a scroll is not a stall), it logs the stack of the thread that
presented that frame and the top of every other thread's, then the
stall's length when frames resume. It also measures how late its own
wake-up was: if it was held up as long as the render thread, the whole
interpreter was blocked (C code holding the GIL), not one thread on a
lock. One dump per 30s at most; LEDMATRIX_STALL_WATCHDOG=0 disables it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 10:58:27 -04:00
ChuckandClaude Opus 5.5 ac841f4583 docs(perf): hdpi soak results, main vs #628
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 08:56:49 -04:00
ChuckandClaude Opus 5.5 98728d3b81 fix(perf): count 1-2s stalls, flag early frames, and keep the refresh estimate honest
Three gaps found by the first hdpi soaks:

- Intervals of 1s or more between two scrolling frames were dropped as
  "gaps between scrolls". But the scrolling state lapses only after 2s, so
  every 1-2s stall inside a scroll vanished from the report. Those are now
  freezes (the gap bound is a 5s sanity limit), with a breakdown by length.
- A frame a whole refresh early means the swap did not wait for the panel.
  Those are counted, and a soak with more than the threshold of them fails
  as NOT LOCKED instead of reporting a flattering late rate.
- The refresh estimate took the lowest window it had seen, so one window of
  non-blocking swaps halved it and made every early frame look on time. A
  window may now lower it by at most 20%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 08:49:37 -04:00
ChuckandClaude Opus 5.5 3eb7a2e349 feat(perf): time every presented frame, and a soak script to judge a rig
Each scroller already logs its own stats line, but in different formats,
per source, and Vegas logs a healthy window only at DEBUG. None of it
answers the question a release has to answer on each rig: over a long
run, how often did a moving frame reach the panel late?

Every frame reaches the panel through DisplayManager.update_display, so
it is timed there once, whoever drew it: the blit (SetImage), the vsync
wait, and the interval since the previous frame. The render thread only
appends a tuple. A worker thread aggregates cumulative counters and
histograms and rewrites /dev/shm/ledmatrix_frame_stats.json every 10s
(RAM, so no SD wear).

A frame due after `hold` refreshes that lands one or more refreshes
later is "late": the panel repeated the previous frame, a visible hitch.
Gaps of 250ms+ inside a scroll are "freezes" (recomposes, handovers,
blocking calls), counted separately so one handover does not read as 40
missed refreshes. Static frames, the first frame of a scroll and gaps
between scrolls are not timed. The refresh period is estimated from the
frames themselves.

scripts/frame_soak.py runs next to the service as any user, diffs two
snapshots over a run (default 10 minutes), optionally keeps the web
preview's viewer marker fresh, and exits non-zero above 0.1% late
frames. It also reports whether the loaded rgbmatrix binding releases
the GIL. Documented under "Soaking a rig" in docs/SCROLL_PERFORMANCE.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 21:51:32 -04:00
238 changed files with 7781 additions and 16842 deletions
+1
View File
@@ -4,3 +4,4 @@ exclude_paths:
- "plugins/**" - "plugins/**"
- "assets/**" - "assets/**"
- "test/**" - "test/**"
- "scripts/debug/**"
-6
View File
@@ -1,8 +1,2 @@
# Auto detect text files and perform LF normalization # Auto detect text files and perform LF normalization
* text=auto * text=auto
# Files the Pi executes must stay LF even in a Windows checkout with
# core.autocrlf=true: a CRLF shebang line fails with "bad interpreter",
# and systemd rejects CRLF unit files.
*.sh text eol=lf
*.service text eol=lf
+1 -5
View File
@@ -6,10 +6,6 @@ on:
jobs: jobs:
claude-review: claude-review:
# Pull requests from forks get no repository secrets, so without this
# guard every outside contributor's PR showed this check red for a reason
# they can't fix. Skipped checks don't block merging.
if: github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
contents: read contents: read
@@ -25,7 +21,7 @@ jobs:
- name: Run Claude Code Review - name: Run Claude Code Review
id: claude-review id: claude-review
uses: anthropics/claude-code-action@756cc22e19660d20e8cc9496b4f242475a7f7790 # v1 uses: anthropics/claude-code-action@v1
with: with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Review PRs opened by the Claude GitHub App. Without this the action # Review PRs opened by the Claude GitHub App. Without this the action
+1 -1
View File
@@ -32,7 +32,7 @@ jobs:
- name: Run Claude Code - name: Run Claude Code
id: claude id: claude
uses: anthropics/claude-code-action@756cc22e19660d20e8cc9496b4f242475a7f7790 # v1 uses: anthropics/claude-code-action@v1
with: with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+3 -70
View File
@@ -8,7 +8,7 @@ on:
# needs a re-run or didn't get created. # needs a re-run or didn't get created.
workflow_dispatch: workflow_dispatch:
# The jobs only check out the repo and run the tests. # Both jobs only check out the repo and run pytest.
permissions: permissions:
contents: read contents: read
@@ -35,7 +35,7 @@ jobs:
- name: Install dependencies - name: Install dependencies
run: | run: |
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install -r requirements.txt -r web_interface/requirements.txt -r requirements-test.txt pip install -r requirements.txt -r requirements-test.txt
pip install RGBMatrixEmulator pip install RGBMatrixEmulator
- name: Run plugin safety harness - name: Run plugin safety harness
@@ -58,7 +58,7 @@ jobs:
- name: Install dependencies - name: Install dependencies
run: | run: |
python -m pip install --upgrade pip python -m pip install --upgrade pip
pip install -r requirements.txt -r web_interface/requirements.txt -r requirements-test.txt pip install -r requirements.txt -r requirements-test.txt
pip install RGBMatrixEmulator pip install RGBMatrixEmulator
# Run the ENTIRE test tree (except test/plugins, which the # Run the ENTIRE test tree (except test/plugins, which the
@@ -73,70 +73,3 @@ jobs:
--cov=src --cov=web_interface \ --cov=src --cov=web_interface \
--cov-report=term \ --cov-report=term \
--cov-fail-under=52 --cov-fail-under=52
js-tests:
name: Web UI JS tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.12"
cache: pip
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r web_interface/requirements.txt
npm install --no-audit --no-fund --prefix test/js
# The DOM suites test the real server-rendered pages and API, so they
# need the web interface running. REQUIRE_DOM turns "couldn't reach it"
# into a failure instead of a silent skip.
- name: Start the web interface
run: |
EMULATOR=true python -c "from web_interface.app import app; app.run(host='127.0.0.1', port=5000, threaded=True)" > web.log 2>&1 &
for i in $(seq 60); do curl -sf -o /dev/null http://127.0.0.1:5000/ && exit 0; sleep 1; done
cat web.log
exit 1
- name: Run JS suites
env:
BASE: http://127.0.0.1:5000
REQUIRE_DOM: "1"
run: node test/js/run_all.js
type-check:
name: Type check (mypy ratchet)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
python-version: "3.12"
cache: pip
# The runtime requirements are installed so mypy sees the real types of
# PIL, requests, psutil and friends -- missing, they'd be Any and the
# result would differ from a developer's machine. mypy and the stubs are
# pinned so a new release can't turn this red without a code change.
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt -r web_interface/requirements.txt
pip install mypy==1.20.2 types-requests==2.33.0.20260906 types-pytz==2026.4.0.20260926
# mypy on exactly the modules in mypy-clean.txt; fails on any error in
# them, or if a listed file is missing. See CONTRIBUTING.md.
- name: Run mypy on the ratchet list
run: python scripts/check_types.py
+9 -7
View File
@@ -3,13 +3,15 @@ __pycache__/
*.py[cod] *.py[cod]
*$py.class *$py.class
# Secrets and per-device state. Everything the software writes into config/ # Secrets
# is local to one device -- config.json, config_secrets.json, wifi_config.json, config/config_secrets.json
# ytm_auth.json (a login session), saved_repositories.json, font_overrides.json, # Atomic writes leave these behind when a save or a test is interrupted;
# and the temp files atomic writes leave behind when interrupted -- so only # the suite drops several per run.
# the templates are tracked. Listing files one by one missed several. config/.config_secrets.json.tmp.*
config/* config/config.json
!config/*.template.json config/config.json.backup
config/wifi_config.json
config/uninstalled_plugins.json
credentials.json credentials.json
token.pickle token.pickle
+5 -13
View File
@@ -37,22 +37,14 @@ repos:
types: [python] types: [python]
pass_filenames: false pass_filenames: false
# The mypy ratchet -- the same check as CI's "Type check (mypy ratchet)" - repo: https://github.com/pre-commit/mirrors-mypy
# job: mypy on exactly the modules listed in mypy-clean.txt. Run it with rev: v1.8.0
# pre-commit run mypy --hook-stage manual
# A local hook rather than mirrors-mypy so mypy sees the packages installed
# from requirements.txt, as CI does; an isolated hook env without them types
# PIL, requests and friends as Any and reports different errors. Needs
# mypy==1.20.2 (the version CI pins) in the environment you commit from.
- repo: local
hooks: hooks:
- id: mypy - id: mypy
name: mypy (ratchet, mypy-clean.txt) additional_dependencies: [types-requests, types-pytz]
entry: python scripts/check_types.py args: [--ignore-missing-imports, --no-error-summary]
language: system
pass_filenames: false pass_filenames: false
always_run: true files: ^src/
stages: [manual]
- repo: https://github.com/PyCQA/bandit - repo: https://github.com/PyCQA/bandit
rev: 1.8.3 rev: 1.8.3
+243 -522
View File
@@ -19,76 +19,108 @@ accepts both, but the store flags the old spelling as deprecated
## Unreleased ## Unreleased
## 3.6.1 - Scripts and installer:
- `fix_web_permissions.sh` makes `safe_plugin_rm.sh` and `safe_pip_install.sh` root-owned again after resetting ownership. A web-user-owned copy of either is a root shell, since sudo lets the web user run them as root. It also restores `config_secrets.json` to mode 640.
- `configure_wifi_permissions.sh` checks its rules with `visudo -c` before installing them, and grants the NetworkManager captive-portal `cp` and `rm` commands `wifi_manager` runs.
- `configure_web_sudo.sh` uses a random temp file and installs its rules with mode 440.
- The installer prints its completion summary before the `-y` reboot, and describes the setup access point as an open network (it was shown with a password it doesn't have).
- `fix_cache_permissions.sh` applies `setup_cache.sh`'s `ledmatrix`-group model instead of setting 777.
- `check_system_compatibility.sh` reports anything but Debian 13 (Trixie) as unsupported, and reaches its summary.
- New `scripts/README.md` lists every script.
- Docs:
- New `docs/ARCHITECTURE.md` (processes, shared state, display loop, plugin system, web UI) and `docs/PERMISSIONS.md` (owners, modes, both sudoers files, repair scripts).
- Deprecated plugin APIs are marked in the plugin docs.
- `src/common/README.md` covers every module.
- Stale setup, service and troubleshooting claims are corrected.
A fix to `src.common.favorite_team_check` (#667). Plugins that drop their - Plugin store and plugin manager fixes:
bundled copy of it should floor on 3.6.1, not 3.6.0. - Updating a plugin that was installed from a ZIP no longer tries to reinstall it from the LEDMatrix repository's own URL.
- Repository URLs with `.git` in the middle are no longer mangled. The URL helpers now live in `src/plugin_system/repo_urls.py`.
- Installing from a URL works when the repository's only branch isn't `main` or `master`.
- A missing required config field is reported once, by name.
- A plugin that went over `max_memory_mb` once is no longer refused on every call after that.
- `reload_plugin` reads the manifest from the plugin's discovered directory.
- Removed: `last_display` from plugin state info and `get_last_display()` (nothing recorded them); `PluginOperationQueue`'s `history_file` and `lazy_load` arguments; and `data/plugin_operations.json`, which nothing read.
### Fixes - Core service fixes:
- `/api/v3/errors` shows each exception's real stack trace instead of `NoneType: None`.
- Wi-Fi disconnect takes the saved connection profile down.
- `wifi_config.json` is written atomically, and a save that fails now gets a 500.
- `plugin://` fonts load from the plugin's own install directory. `FontManager.register_plugin_fonts()` takes an optional `plugin_dir`.
- `APIHelper` keeps cached responses for the `cache_ttl` it was given, instead of always 300 s.
- Logo scales from 0.1 to 10 are honoured everywhere; values outside that range are clamped.
- `LogoHelper` and `logo_downloader`: an empty ESPN logo list counts as a failed download, and the placeholder is written at the requested path.
- Bundled font paths no longer depend on the directory the process was started from.
- Backups record `src.__version__`.
- Removed: `BackgroundDataService`'s `queue_size` stat and `clear_completed_requests()`.
- The favourite-team check no longer logs "the season has finished" for a - Web API fixes:
league that is still playing. ESPN's default scoreboard keeps showing the - A plugin save drops repeated entries in lists whose schema says `uniqueItems`, instead of failing validation.
last slate after it: MLB's regular-season games two days into the - `/api/v3/health` reports the real plugin count.
postseason, a soccer league's previous matchday between rounds. When every - A malformed `vegas_plugin_order` or `vegas_excluded_plugins` is refused with a 400 and nothing is saved. It used to wipe the saved list.
event is in the past, the check now looks first at the league's phase (a - The per-plugin health and metrics routes return the display service's latest state.
regular season or postseason that has moved past the events shown draws no - Resetting a plugin's config takes a backup first and reports a failed save.
conclusion) and at a match-day calendar (`calendarType` "day" with - System metrics that can't be read are `null` everywhere: `cpu_temp` off a Pi, and every metric without psutil, where `/system/status` now answers 200 instead of 503.
`calendarIsWhitelist`, as soccer, the NHL and the NBA use), whose next date - `/plugins/store/refresh` no longer claims a commit-metadata refresh it doesn't do.
becomes "nothing on until <date>". An offseason, or a payload without these - The plugin-config list repair code is in one place, `src/web_interface/config_arrays.py`.
fields, is reported as before.
## 3.6.0 - Web UI:
- Cache tab errors no longer show up in the Logs tab.
- A tab that fails to load shows "Try again" instead of a skeleton that never goes away.
- Plugin Store search and registry errors appear as a notification, and the Plugin Manager stays on screen.
- The image schedule button works on uploaded images, and the editor stays open while you edit.
- A failed plugin toggle moves the switch back.
- Each save shows one notification; a failed Durations save says it failed.
- Stats the server can't read show `--`.
- New `window.LEDEscape` (`html`, `attr`, `jsStringAttr`) replaces about 30 copied escapers. `window.escapeHtml` and `window.escapeAttribute` remain as aliases for plugin pages.
New modules a plugin may import via `src.*` (floor on 3.6.0). Both are - Display and Vegas:
promoted from files the scoreboard plugins carry as copies; the plugins keep - Vegas `max_cycle_duration` defaults to 240 s when unset, as documented (it was 600 s). The Vegas defaults are now defined once.
their copies as a fallback until they floor on 3.6.0. No other change since - The display controller stops Vegas mode on shutdown.
3.5.0. - Startup validation warnings are logged once, not twice.
- Vegas logs one INFO line per plugin-list refresh.
- `run.py -d` shows `display_manager` debug output.
- Removed: the Vegas staging buffer that was never filled (`swap_buffers()`, and `staging_count` / `current_index` in `get_buffer_status()`), unread `ContentSegment` fields, and `geometry.find_blank_cut()`.
- `src/common/favorite_team_check.py` — `FavoriteTeamCheck(logger, leagues)`: - The web service (`ledmatrix-web`) logs through `src.logging_config` like the
checks configured favourite team codes against ESPN once per league, on a display service, so `journalctl -p err -u ledmatrix-web` works. Successful
daemon thread, and logs why a league shows nothing (a wrong code, with the GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of
nearest real one, or a season that has not started). The seven copies INFO; 4xx at WARNING, 5xx at ERROR. `LEDMATRIX_DEBUG=true` shows them again.
(`<sport>_favorite_check.py`) were byte-identical; this is the same code, `web_interface/logging_config.py` is removed. The web cache
with type annotations added. (`web_interface/cache.py`) now honours the TTL a value was stored with and is
- `src/common/sports_timezone.py` — `resolve_timezone_name()` / thread-safe.
`resolve_timezone()` (plus `system_timezone_name()`): the timezone a
scoreboard draws start times in. The ten copies (`<sport>_timezone.py`)
differed only in two values, which are keyword-only arguments here:
`plugin_label` (named in the warning logged when nothing resolves) and
`writeback_fixed_in` (for a plugin that once wrote `"UTC"` back into the
saved config; `None` otherwise). Same resolution order and log messages.
## 3.5.0 - One plugin-directory resolver, `src/plugin_system/plugin_dirs.py`, behind
discovery, `PluginManager.get_plugin_directory`, `PluginLoader`, the store and
state reconciliation. A manifest's `id` wins over a directory merely named for
the id; hidden and `.standalone-backup-` directories are never treated as
plugins (auto-update could previously try to update a backup); ids like
`a/b` or `..` resolve to nothing everywhere. Installs where each directory is
named for its manifest id, the installer's layout, behave as before.
New modules a plugin may import via `src.*` (floor on 3.5.0): - `/api/v3` routes answer an exception they don't handle themselves from one
blueprint error handler, with the same `{status, message, details}` body the
53 removed per-route catch-alls returned. `ErrorCategory` and the
`error_category` key are removed from `src.web_interface.errors` (nothing read
them); `exception_error_response()` replaces the `from_exception` +
`error_response` pairs. A failing plugin action script's error now names the
real failure instead of `UnboundLocalError`.
- `src/common/sports_helpers.py` — the helpers the scoreboards' `sports.py` - `FontManager.get_font()` returns a BDF font at its native size when asked for
carry byte-identical copies of: `clamp_window`, `clamp_seconds`, a size the file doesn't contain (5x7.bdf at 8 or 10px, say). It used to
`logo_needs_refresh`, `spread_weighted_order` (+ `MIN_WINDOW_DAYS`, return PIL's default font, a different typeface, so a plugin that relied on
`MAX_WINDOW_DAYS`), and `SportsHelpersMixin` with `_mode_customization`, that will now render the font it asked for.
`_setting_int`, `_reset_dwell_on_reentry`, `_next_switch_index`, - `src.wifi_manager.get_wifi_status_path()` — where WiFi status messages for
`_spread_weighted_order`, `_odds_color`, `_upcoming_date_and_time_text` under the display are written (`config/wifi_status.json`).
the plugins' names and signatures, plus the `_favorite_key` override point. - `src.device_location` — a blank `Location` field on a Starlark (Tidbyt) app
Constructor-free; keeps lazy state on its host (see the module docstring, now renders at the device's City / State / Country (geocoded once via
which also gives the host contract). Open-Meteo and cached) instead of the app author's hard-coded default,
A new module rather than more methods on `sports_shared`: a plugin that usually San Francisco. A location saved on the app still wins. With no
deletes a copy and leans on an older module having grown the method fails at device city set, or when the lookup fails or finds no match, the app keeps
runtime with `AttributeError`, which no load-time check sees, while a missing its own default (a failed lookup is retried after 30 minutes). Clearing an
module fails at load. Nothing in core uses it yet. app's location in the web UI now actually clears it; the save used to drop
- `src/common/espn_dates.py` — `fetch_espn_scoreboard`, the blank field, so the old value stayed.
`fetch_espn_date_chunks`, `espn_date_chunks`, `clamp_espn_limit`, - `src.common.bdf_font` — `load_bdf_face(path, size)` (a cached
`ESPN_MAX_LIMIT`: fetch an ESPN scoreboard date range now that ESPN rejects
ranges (see Sports data below). Plugins bundle a copy of it.
- `src/common/json_body.py` — `response_json(response)`: `response.json()`,
parsed by orjson when it is installed (an optional dependency) and by the
stdlib otherwise; an orjson parse error falls back to `response.json()` so
requests raises its usual error. Same Python objects either way; a season
schedule parses about 1.7x faster on a Pi 4, and the parse holds the GIL (so
freezes the display) for that much less time. `espn_dates` and
`BackgroundDataService` use it; `espn_dates` falls back to `response.json()`
when it is missing, so the plugins' bundled copies of `espn_dates` still load
on an older core.
- `src/common/bdf_font.py` — `load_bdf_face(path, size)` (a cached
`freetype.Face` plus the pixel size it really renders at, falling back to `freetype.Face` plus the pixel size it really renders at, falling back to
the file's native strike) and `draw_bdf_text(draw, text, x, y, face, color)`. the file's native strike) and `draw_bdf_text(draw, text, x, y, face, color)`.
`DisplayManager`, `FontManager`, `element_style` and the plugin test harness `DisplayManager`, `FontManager`, `element_style` and the plugin test harness
@@ -99,27 +131,12 @@ New modules a plugin may import via `src.*` (floor on 3.5.0):
`dev_server` previews its text sat 6px above where the panel draws it (off `dev_server` previews its text sat 6px above where the panel draws it (off
the canvas entirely near the top) and `get_font_height()` returned 0. the canvas entirely near the top) and `get_font_height()` returned 0.
Also new under `src/` since 3.4.0, but internal to core rather than for plugins: - The web UI's Fonts tab has a **Used by** column: the loaded plugins that
`src/common/frame_timing.py` and `src/common/render_gate.py` (see Scrolling), registered each font with `FontManager.register_manager_font()`, published
`src/core_config_keys.py`, `src/deprecation.py`, `src/device_location.py`, by the display service to the shared cache (`src/font_usage.py`) and merged
`src/font_usage.py`, `src/matrix_support.py`, `src/pi5_matrix_support.py`, into `GET /api/v3/fonts/catalog` as `used_by`. Deleting a font a plugin
`src/redaction.py`, `src/scan_order.py`, `src/web_interface/config_arrays.py`, uses now names those plugins in the confirmation (it is not blocked).
and in `src/plugin_system/`: `plugin_dirs.py`, `repo_urls.py`, `FontManager.forget_manager_fonts()` is new; unloading a plugin calls it.
`store_install.py`, `store_registry.py` and `store_update.py`.
New names in existing modules (a plugin using these must floor on 3.5.0):
- `src.common.api_helper`: `USER_AGENT`, `DEFAULT_HTTP_HEADERS` (read-only).
- `src.logo_downloader`: `fetch_logo`, `save_png_atomically`,
`shared_downloader`.
- `src.common.sports_card.unshare_element_fonts` takes an optional third
argument, `element_for_font` (default: the module's `ELEMENT_FOR_FONT`, so
existing calls are unchanged).
- `src.wifi_manager.get_wifi_status_path()` — where WiFi status messages for
the display are written (`config/wifi_status.json`).
- `BackgroundDataService.handles_espn_date_ranges` (see Sports data).
- `FontManager.register_plugin_fonts()` takes an optional `plugin_dir`, and
`FontManager.forget_manager_fonts()` is new (see Fonts).
Deprecated, removed in 3.7.0 (each logs a warning on first use; see Deprecated, removed in 3.7.0 (each logs a warning on first use; see
`docs/PLUGIN_API_REFERENCE.md#deprecated-apis` for replacements). Nothing in `docs/PLUGIN_API_REFERENCE.md#deprecated-apis` for replacements). Nothing in
@@ -139,6 +156,147 @@ core, the monorepo or the registry's third-party plugins calls them:
`unregister_plugin_fonts`. `unregister_plugin_fonts`.
- `PluginManager.get_enabled_plugins`. - `PluginManager.get_enabled_plugins`.
### Config writes
- A power cut or crash mid-save can no longer leave `config/config.json`
truncated. `ConfigManager.save_config()` wrote the file in place; it,
`save_config_atomic()`, `save_raw_file_content()` and backup rollback now
share one writer (`atomic_write_text` in `src/config_manager_atomic.py`)
that fsyncs a temp file, renames it into place and fsyncs the directory.
- `save_config_atomic()` no longer rewrites `config_secrets.json` on every
save, only when its content changes, and rotating backups no longer re-reads
every backup. The backups themselves are unchanged:
`config/backups/config.json.backup.<version>` plus its paired secrets
backup, five newest kept.
- A save by the root-run display service keeps the file's previous owner
instead of handing `config.json` to root, and an install path with
"secrets" in a directory name no longer makes `config.json` mode 0640.
New names in existing modules (no new modules; a plugin importing these must
floor on the release that ships them):
- `src.common.api_helper`: `USER_AGENT`, `DEFAULT_HTTP_HEADERS` (read-only).
- `src.logo_downloader`: `fetch_logo`, `save_png_atomically`,
`shared_downloader`.
- `src.common.sports_card.unshare_element_fonts` takes an optional third
argument, `element_for_font` (default: the module's `ELEMENT_FOR_FONT`, so
existing calls are unchanged).
### Sports twins
- The `SportsCoreSharedMixin` helpers that behave identically to their
`sports_card` twins (`_card_option`, `_vs_text`, `_format_game_time`,
`_coerce_rgb`, `_crisp_size`, `_unshare_element_fonts`, the colour/month/
weekday/font-grid tables) are now thin wrappers over the `sports_card`
functions, and `_format_game_date` / `_schema_font_size` share its
formatting body and schema parser. No method was removed or renamed and
nothing renders differently: `test/test_sports_twins.py` checks each pair
against the same inputs, and the old and new mixin agree on every input
there. The pairs that do differ -- favourite-result colours on nested
payloads, the weekday's timezone, the element-name map, per-mode colours --
are left as they are and pinned in that test.
### Logo downloads
- `download_missing_logo` / `LogoDownloader.download_logo` (the path the
scoreboard plugins use) now stream the logo with a 10 MB cap, accept only an
`image/*` response that Pillow can decode, and move the finished RGBA PNG
into place atomically. A failed, oversized or non-image download no longer
leaves a partial file behind, and no longer replaces a logo already on disk.
`LogoHelper._download_logo` goes through the same code. Signatures and return
values are unchanged; saved files are pixel-identical to before.
- `download_missing_logo` reuses one downloader (one `requests.Session`) per
thread instead of building a new one for every logo.
- Placeholder logos are written atomically, without the `test_write.tmp`
probe file.
### HTTP headers
- The logo downloader and the background data service send the real
`LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)` User-Agent
instead of a `yourusername` / `contact@example.com` placeholder, and no
longer set `Accept-Encoding: ... br` by hand (brotli is not installed, so a
`br` response could not be decoded); requests picks the encodings.
### Plugin error reporting
- `/api/v3/errors/summary` and `/api/v3/errors/plugin/<id>` report the errors
the display service recorded. They used to read the web process's own error
aggregator, which never records anything, so they always answered "no
errors". The display service now publishes a bounded snapshot to the shared
cache (`plugin_error_snapshot`, at most every 10 seconds and only on change;
`src/error_aggregator.py`, started from `DisplayController.__init__`).
Responses keep their shape and add `snapshot_available`, `generated_at` and
`clear_pending`; exception text has credentials redacted.
- `POST /api/v3/errors/clear` records a request (`plugin_error_clear_request`)
the display service applies within about 5 seconds; reads hide the cleared
errors at once. It accepts `"all": true`, and `cleared_count` can be `null`
when the count is only known to the display service.
- The Logs tab has a **Plugin errors** panel: per-plugin counts, repeating
errors and a Clear button.
- Credential redaction in exception text (`src/redaction.py`) takes time
proportional to the text, not its square. Two patterns were quadratic: URL
`user:password@`, on a long unbroken run of letters or digits (a hex digest,
an ID), and `Authorization:` followed by a long run of whitespace. Either
used to stall every thread of the display service for up to seconds each
time the snapshot was published: about 0.5s for 20k characters of hex, 8s
for 20k spaces. What gets redacted is unchanged.
### Removed
- **The skin system.** Skins never rendered with the current scoreboard
plugins, so they are gone rather than "not supported yet": `src/skin_system/`,
`skins/`, `scripts/validate_skin.py`, `GET /api/v3/skins`, the store's
`"type": "skin"` handling and `docs/SKIN_SYSTEM.md` / `docs/CREATING_SKINS.md`.
A `skin` or `skin_options` key left in a plugin's saved config still loads
and saves without a validation error; it is ignored, and the next save of
that plugin's settings removes it (unless the plugin's own schema declares
the key).
- **`src/base_classes/`** (`SportsCore`, the sport and mode classes,
`CelebrationMixin`, the rotation strategies, `data_sources`,
`api_extractors`). No known plugin imports it. A plugin that does must use
`src.common` or its own copy of the code.
- `src.common.frame_timing` -- times every frame the display presents, whoever
drew it, and writes cumulative counters to `/dev/shm`. Two tools read it:
`scripts/frame_soak.py` judges a running service (late frames, freezes,
where the time goes), and `scripts/render_bench.py` judges the hardware and
render path alone on a synthetic strip. Both fail a run above 0.1% late
frames, and both call a loop that never waited for the panel NOT LOCKED. A
stall watchdog logs the stack of whatever holds a scroll up for 250 ms or
more. See `docs/SCROLL_PERFORMANCE.md`, "Soaking a rig".
- `display.scan_order_compensation` (`"auto"` by default): while something
scrolls at one pixel per refresh, one half of each panel is shown a refresh
behind the other, which removes the 1px step a 1:N-scan panel shows across
its middle. Only for layouts whose row order is known; `"off"` disables it.
See `docs/SCROLL_PERFORMANCE.md`, "A tear across the middle on fast scrolls".
## 3.5.0
New modules a plugin may import via `src.*` (floor on 3.5.0):
- `src/common/sports_helpers.py` — the helpers the scoreboards' `sports.py`
carry byte-identical copies of: `clamp_window`, `clamp_seconds`,
`logo_needs_refresh`, `spread_weighted_order` (+ `MIN_WINDOW_DAYS`,
`MAX_WINDOW_DAYS`), and `SportsHelpersMixin` with `_mode_customization`,
`_setting_int`, `_reset_dwell_on_reentry`, `_next_switch_index`,
`_spread_weighted_order`, `_odds_color`, `_upcoming_date_and_time_text` under
the plugins' names and signatures, plus the `_favorite_key` override point.
Constructor-free; keeps lazy state on its host (see the module docstring,
which also gives the host contract).
A new module rather than more methods on `sports_shared`: a plugin that
deletes a copy and leans on an older module having grown the method fails at
runtime with `AttributeError`, which no load-time check sees, while a missing
module fails at load. Nothing in core uses it yet.
- `test/test_common_is_hardware_free.py` — `src/common` must import without
`rgbmatrix` and never import `src.base_classes`, `src.display_manager` or
`src.plugin_system` at module level.
- `src/common/espn_dates.py` — `fetch_espn_scoreboard`,
`fetch_espn_date_chunks`, `espn_date_chunks`, `clamp_espn_limit`,
`ESPN_MAX_LIMIT`: fetch an ESPN scoreboard date range now that ESPN rejects
ranges (see Sports data below). Plugins bundle a copy of it.
### Config saves and plugin config preparation ### Config saves and plugin config preparation
- A JSON `POST /api/v3/config/main` changes only the keys it sends. The MQTT - A JSON `POST /api/v3/config/main` changes only the keys it sends. The MQTT
@@ -184,23 +342,8 @@ core, the monorepo or the registry's third-party plugins calls them:
"enabled but not found in plugins directory", and plugin ids that collide "enabled but not found in plugins directory", and plugin ids that collide
with any core config section are flagged: the last private copies of the with any core config section are flagged: the last private copies of the
core-key list now use `src/core_config_keys.py`. core-key list now use `src/core_config_keys.py`.
- Plugin config saves recombine position-keyed inputs for nullable array fields (`"type": ["array", "null"]`).
- A blank Max Dynamic Duration keeps the stored value instead of failing the Display save with a 500; other values must be whole seconds from 30 to 1800.
- A power cut or crash mid-save can no longer leave `config/config.json`
truncated. `ConfigManager.save_config()` wrote the file in place; it,
`save_config_atomic()`, `save_raw_file_content()` and backup rollback now
share one writer (`atomic_write_text` in `src/config_manager_atomic.py`)
that fsyncs a temp file, renames it into place and fsyncs the directory.
- `save_config_atomic()` no longer rewrites `config_secrets.json` on every
save, only when its content changes, and rotating backups no longer re-reads
every backup. The backups themselves are unchanged:
`config/backups/config.json.backup.<version>` plus its paired secrets
backup, five newest kept.
- A save by the root-run display service keeps the file's previous owner
instead of handing `config.json` to root, and an install path with
"secrets" in a directory name no longer makes `config.json` mode 0640.
### Sports data, logos and odds ### Sports data
- Since 2026-09-15 ESPN answers `dates=YYYYMMDD-YYYYMMDD` scoreboard queries - Since 2026-09-15 ESPN answers `dates=YYYYMMDD-YYYYMMDD` scoreboard queries
with `400 Bad Request` for every sport, so season schedules, the weeks window with `400 Bad Request` for every sport, so season schedules, the weeks window
@@ -248,44 +391,6 @@ core, the monorepo or the registry's third-party plugins calls them:
longer swallowed as a missing poll. This is the implementation the football, longer swallowed as a missing poll. This is the implementation the football,
baseball and hockey boards already ship; core was the last copy on the old baseball and hockey boards already ship; core was the last copy on the old
one. one.
- `BaseOddsManager.get_odds()` no longer returns the cached "no odds" marker (`{"no_odds": True}`) as if it were odds. A game ESPN had no odds for is cached that way so it isn't re-requested every update; on the next update the cache hit handed the marker back, and callers saw a truthy dict. It now returns `None` for it, on the cache hit and in the stale-cache fallback after a failed fetch, as the plugins' bundled copies already did.
- Background data fetches retry at one level instead of two. The session adapter retried a connection error three times inside every attempt of the service's own retry loop, so a dead network cost up to 16 connection attempts per request and held one of the few worker threads throughout; now it is the loop's `max_retries + 1` attempts. ESPN date-range chunks, which don't go through that loop and skip a chunk that fails, keep a small connection retry of their own so a brief blip doesn't drop a month from a cached season.
- `LogoHelper.load_logo_with_download()` sizes its placeholder to the scaled logo box, like a real logo (only differs when `scale` isn't 1).
- The AP Top 25 resolver remembers a failed or empty rankings fetch for 5 minutes, so an ESPN outage no longer costs every scoreboard update a 30s timeout. Its duplicate INFO log line is gone.
- `BackgroundDataService` runs a cache-hit callback outside its lock, as the fetch path does.
- `LogoHelper.load_logo_with_download()` waits an hour before retrying a download that failed for a missing logo, instead of retrying (with a 30 s timeout) on every call.
- Restamping a placeholder logo writes the file atomically.
- The odds manager logs cache hits, misses and fetches at DEBUG, and a bad JSON body is logged as a parse error rather than a failed fetch.
- `APIHelper` keeps cached responses for the `cache_ttl` it was given, instead of always 300 s.
- Logo scales from 0.1 to 10 are honoured everywhere; values outside that range are clamped.
- `LogoHelper` and `logo_downloader`: an empty ESPN logo list counts as a failed download, and the placeholder is written at the requested path.
- The `SportsCoreSharedMixin` helpers that behave identically to their
`sports_card` twins (`_card_option`, `_vs_text`, `_format_game_time`,
`_coerce_rgb`, `_crisp_size`, `_unshare_element_fonts`, the colour/month/
weekday/font-grid tables) are now thin wrappers over the `sports_card`
functions, and `_format_game_date` / `_schema_font_size` share its
formatting body and schema parser. No method was removed or renamed and
nothing renders differently: `test/test_sports_twins.py` checks each pair
against the same inputs, and the old and new mixin agree on every input
there. The pairs that do differ -- favourite-result colours on nested
payloads, the weekday's timezone, the element-name map, per-mode colours --
are left as they are and pinned in that test.
- `download_missing_logo` / `LogoDownloader.download_logo` (the path the
scoreboard plugins use) now stream the logo with a 10 MB cap, accept only an
`image/*` response that Pillow can decode, and move the finished RGBA PNG
into place atomically. A failed, oversized or non-image download no longer
leaves a partial file behind, and no longer replaces a logo already on disk.
`LogoHelper._download_logo` goes through the same code. Signatures and return
values are unchanged; saved files are pixel-identical to before.
- `download_missing_logo` reuses one downloader (one `requests.Session`) per
thread instead of building a new one for every logo.
- Placeholder logos are written atomically, without the `test_write.tmp`
probe file.
- The logo downloader and the background data service send the real
`LEDMatrix/1.0 (+https://github.com/ChuckBuilds/LEDMatrix)` User-Agent
instead of a `yourusername` / `contact@example.com` placeholder, and no
longer set `Accept-Encoding: ... br` by hand (brotli is not installed, so a
`br` response could not be decoded); requests picks the encodings.
### Scrolling ### Scrolling
@@ -316,92 +421,6 @@ core, the monorepo or the registry's third-party plugins calls them:
held 20 ms frame as missed refreshes, and Vegas `frame_based_scrolling` / held 20 ms frame as missed refreshes, and Vegas `frame_based_scrolling` /
`scroll_delay` are described as the speed clamp they are rather than frame `scroll_delay` are described as the speed clamp they are rather than frame
stepping. Scoreboard `scroll_delay` is documented as ignored for pacing. stepping. Scoreboard `scroll_delay` is documented as ignored for pacing.
- `ScrollHelper.set_scrolling_image()` accepts RGBA, L and palette images (transparent pixels become black), and a new scrolling image no longer jumps ahead by the time the helper sat idle.
- `src.common.frame_timing` -- times every frame the display presents, whoever
drew it, and writes cumulative counters to `/dev/shm`. Two tools read it:
`scripts/frame_soak.py` judges a running service (late frames, freezes,
where the time goes), and `scripts/render_bench.py` judges the hardware and
render path alone on a synthetic strip. Both fail a run above 0.1% late
frames, and both call a loop that never waited for the panel NOT LOCKED. A
stall watchdog logs the stack of whatever holds a scroll up for 250 ms or
more. See `docs/SCROLL_PERFORMANCE.md`, "Soaking a rig".
- `display.scan_order_compensation` (`"auto"` by default): while something
scrolls at one pixel per refresh, one half of each panel is shown a refresh
behind the other, which removes the 1px step a 1:N-scan panel shows across
its middle. Only for layouts whose row order is known; `"off"` disables it.
See `docs/SCROLL_PERFORMANCE.md`, "A tear across the middle on fast scrolls".
### Display and Vegas
- Vegas scrolls in step with the panel's refresh (#628). With `smooth_scroll`
(on by default) the strip moves a whole number of pixels per presented
frame, each held for `frame_hold` refreshes and timed by `SwapOnVSync`,
the same pacing as the plugin tickers; it used to advance by elapsed time
and sleep to `target_fps`, missing a vsync every few frames. The speed is
solved against the panel's measured refresh when that is below its
`limit_refresh_rate_hz` cap. The old sub-pixel blend, which the panel shows
as shimmer, is kept as `vegas_scroll.sub_pixel_blend` (default off). While
scrolling, the web preview's PNG is encoded on a background writer instead
of the render thread. On a Pi 4 driving 512x64, late frames went from about
6.4% to 0.7%.
- Vegas prepares plugin content off the render thread (#630). A plugin that
needed the shared canvas used to be fetched on the render thread, stalling
the scroll for as long as it took (320 ms for news, 660 ms for a hockey
scoreboard, measured). `DisplayManager.offscreen(width, height)` gives the
calling thread a canvas of its own: `image`, `draw` and `matrix` are now
properties that resolve to it inside the block, where `update_display()`,
the hardware half of `clear()` and `set_scrolling_state()` /
`set_frame_hold()` do nothing. Background fetches take the plugin's lock,
waiting up to 2 s for a running `update()` and otherwise skipping the plugin
that round. A GIL gate (`src/common/render_gate.py`) pauses the prefetch
thread outside a window around each vsync swap, so the render thread finds
the GIL free; it needs the rebuilt binding that releases the GIL in
`SwapOnVSync` and stays off (one INFO line per Vegas run) on a stock one.
New `vegas_scroll` keys: `offscreen_prefetch` and `prefetch_gate` (both on by
default) and `switch_interval_ms` (experimental, default 0, off). Design in
`docs/OFFSCREEN_RENDERING.md`.
- On-demand requests, the display on/off schedule and brightness take effect
within about a quarter of a second instead of at the next screen (#618). A
screen can stay up for a minute and a Vegas iteration for 240 s, so an
on-demand request during Vegas waited for the iteration to end and a
brightness save mid-screen could be lost. Vegas now stops for an on-demand
request and for the display being scheduled off, and a brightness change
re-sends the current frame. Plugin enable/disable, screen durations and Vegas
settings still apply at the next screen.
- The Rotation & Durations page takes effect (#605). A saved
`display.display_durations` value now wins over the plugin's own duration;
the plugin was asked first, and every plugin inherits
`get_display_duration()`, so saved values did nothing. The page shows an
unsaved screen blank with the plugin's own duration as the placeholder (it
showed 30 where the real default is 15), and saving a blank removes the
override. Durations saved before this now apply.
- Vegas settings reach a running scroll (#605): they are queued when
`display.vegas_scroll` changes (unrelated saves don't rebuild the strip) and
also applied while Vegas is stopped. The sync follower's scroll-speed
default (75) now matches `VegasModeConfig`'s (50). The Vegas live-priority
scan is throttled to 4 Hz.
- `DisplayManager.defer_update()` from a plugin's update thread no longer loses queued updates while the render thread processes the queue; the queue is locked, and the queued callables still run outside the lock.
- **Behaviour change:** when `display.hardware.limit_refresh_rate_hz` is missing from config, the panel is now capped at 100 Hz (the config template's value) instead of 90 Hz. Scroll pacing already assumed 100 Hz in that case, so it now matches what the panel does. Configs that set the key (every config migrated from the template) are unaffected.
- A sync follower adopts the leader's scroll image between frames on the render thread, instead of the TCP thread swapping the image, array and width while a frame is being drawn.
- `update_display()` errors are logged once with a traceback, then at most once a minute with a count, instead of an untraced line every frame. Several swallowed exceptions in `DisplayController` now log at DEBUG.
- The repo-root `display_controller.py` now runs `run.py` (the real entry point), so it gets run.py's `-e`/`-d` flags, logging setup and `sys.dont_write_bytecode`.
- Vegas: a plugin set to `vegas_mode: "static"` pauses the scroll for its turn again. The pause was triggered by peeking at the front of a segment buffer that continuous scrolling (the default) never advances, so a static plugin paused only if it happened to be first, once, at startup, and otherwise scrolled past as ordinary content; swap mode had the same problem for any static plugin not first in its cycle. The render pipeline now marks where each static plugin's turn falls in the strip and the scroll pauses when it gets there. The pause runs the plugin's `display()` under its plugin lock, and a static plugin's content is no longer rendered for the strip.
- The display loop no longer spins at 100% CPU when no enabled mode has anything to show (for example, only a sports plugin enabled in its off-season). After one full rotation of empty modes it checks one mode per second until something shows; live content still takes over at once.
- Stopping `ledmatrix.service` runs the controller's cleanup (SIGTERM now takes the Ctrl-C path).
- Turning Vegas on in the web UI works without a restart when it was off at startup.
- Vegas comes back after live content interrupts it. It stayed paused, and the display fell back to normal rotation until a restart.
- A day with dimming turned off in a per-day dim schedule stays at normal brightness. Before, brightness went back to dim for most of each minute.
- Stopping on-demand after a second request resumes rotation where it was first interrupted, not at the first request's screen.
- Turning Vegas off and on no longer shows content prepared for the previous run, including plugins disabled in between.
- How long a Vegas iteration runs is timed with the monotonic clock, so an NTP clock step on a Pi without an RTC doesn't cut it short or stretch it.
- The sync status file is removed when the display service stops, and at startup in standalone mode, so the web UI no longer reports a peer from an earlier run. Concurrent writes each use their own temp file.
- `render_gate.swap_releases_gil()` delegates to `frame_timing.binding_releases_gil()` instead of duplicating it.
- Vegas `max_cycle_duration` defaults to 240 s when unset, as documented (it was 600 s). The Vegas defaults are now defined once.
- The display controller stops Vegas mode on shutdown.
- Startup validation warnings are logged once, not twice.
- Vegas logs one INFO line per plugin-list refresh.
- `run.py -d` shows `display_manager` debug output.
- Removed: the Vegas staging buffer that was never filled (`swap_buffers()`, and `staging_count` / `current_index` in `get_buffer_status()`), unread `ContentSegment` fields, and `geometry.find_blank_cut()`.
### Web interface ### Web interface
@@ -466,142 +485,8 @@ core, the monorepo or the registry's third-party plugins calls them:
discover when nothing has been discovered yet, and rescan once when a discover when nothing has been discovered yet, and rescan once when a
specific plugin id (or, for on-demand by mode, a mode) is not found, so a specific plugin id (or, for on-demand by mode, a mode) is not found, so a
plugin installed since the last scan is found too. plugin installed since the last scan is found too.
- `web_interface/blueprints/api_v3/plugins.py` (3,285 lines) is split by area into `plugins.py` (installed list, enable/disable, plugin actions), `plugin_store.py`, `plugin_config.py`, `plugin_assets.py`, `plugin_health.py`, `plugin_operations.py` and `plugin_calendar.py`. Pure move: every function body and route decorator is byte-identical, and URLs and endpoint names are unchanged.
- Plugins installed as `ledmatrix-<id>` (or in a directory not named after their id) work in the installed list, the update button, recorded versions, the plugin config form and plugin web UI pages. Those routes built `plugins_dir/<id>` themselves instead of asking the plugin manager.
- Uploading several plugin images checks every file before saving any, so a rejected file no longer leaves the others saved; the images' `.metadata.json` and the calendar plugin's `credentials.json` are written atomically, and the credentials upload no longer returns the server's absolute path.
- `"false"` sent as a string no longer counts as true when toggling a plugin (including Starlark apps) or starting on-demand mode (`pinned`, `start_service`); `force` on the AP-enable route is parsed like every other WiFi boolean (`"yes"` and `1` now force).
- The live-preview stream starts a new broadcast thread for a client that connects while the previous one is shutting down; that client got no updates.
- The web server's log filter no longer raises when werkzeug logs with `exc_info=True`.
- The raw secrets editor's save errors carry `error_code` like the main config's; the asset delete route answers 400 for a missing body instead of 415/500. Dead code removed: an unused manifest scan on each Plugins-tab load, backup routes' duplicate catch-alls, redundant imports.
- A plugin's own config widget (`/static/plugin-widgets/<id>/<widget>.js`) is requested with `?v=<plugin version>`, so an updated plugin's widget reaches browsers instead of the copy cached as immutable for a year.
- A failed installed-plugins reload after a toggle, install or uninstall shows one error, not a second generic "unexpected error" toast.
- The timezone picker on the General tab renders again when the tab is reloaded in the same page session.
- Removed dead code: the plugin-action button's six plugin-id fallbacks (the button always passes its id) and its `[DEBUG]` logging, `window.currentPluginConfig` (never set to anything but `null`), the file-upload widget's JSON delete branch (its endpoint never existed), unused `PluginAPI` / `PluginInstallManager` / `PluginStateManager` helpers, `loadPluginWidgetsFromManifest`, no-longer-reachable fallbacks for a stale `install_manager.js` and a missing `LEDVisibility`, and 13 unused CSS utility rules.
- The Logs tab's "Now showing" no longer reads "unknown" when one screen stays up longer than 2 minutes.
- A network failure fetching GitHub repo info logs a warning, not an error.
- The Operation History plugin filter lists installed plugins (it showed one option, "plugins").
- Ctrl/Cmd+S submits the active tab's visible form (with its validation) instead of the first form in the page; it does nothing inside a dialog or on a tab without a form. The Ctrl/Cmd+R override (the browser's own reload) and the textarea auto-resize (no textarea exists at load) are removed.
- Tools tab actions and diagnostics show the server's error message; only a non-JSON error falls back to `HTTP <status>`.
- An uninstalled plugin no longer reappears in the installed list: writes through `PluginAPI` clear its 5s GET cache, and Refresh and the post-uninstall reload bypass both list caches.
- Plugin widgets load from `/static/plugin-widgets/` only; the two other paths it tried have no route.
- The raw JSON editor escapes the parse error, and the slider widget escapes its value, min, max and step.
- Removed unused array-of-objects and key-value helpers from `plugins_manager.js` (about 640 lines, no callers) and a redundant `?v=` on its script tag.
- Plugin tabs show the manifest's `icon`: `/api/v3/plugins/installed` now includes it.
- `POST /api/v3/starlark/apps/<id>/toggle` goes through the same code as `/plugins/toggle`: `"false"` disables, a failed save no longer leaves the running app out of step with disk, and a loaded app with no manifest entry no longer answers 500.
- `/api/v3/` JSON responses are sent `Cache-Control: no-store`, so a reload right after an install, toggle or Wi-Fi connect shows the new state. Non-JSON files served through the API keep the 5 s cache.
- Startup plugin validation no longer gives up on a `null` plugin block, and plugins are discovered once at startup instead of twice.
- A plugin save drops repeated entries in lists whose schema says `uniqueItems`, instead of failing validation.
- `/api/v3/health` reports the real plugin count.
- A malformed `vegas_plugin_order` or `vegas_excluded_plugins` is refused with a 400 and nothing is saved. It used to wipe the saved list.
- The per-plugin health and metrics routes return the display service's latest state.
- Resetting a plugin's config takes a backup first and reports a failed save.
- System metrics that can't be read are `null` everywhere: `cpu_temp` off a Pi, and every metric without psutil, where `/system/status` now answers 200 instead of 503.
- `/plugins/store/refresh` no longer claims a commit-metadata refresh it doesn't do.
- The plugin-config list repair code is in one place, `src/web_interface/config_arrays.py`.
- Cache tab errors no longer show up in the Logs tab.
- A tab that fails to load shows "Try again" instead of a skeleton that never goes away.
- Plugin Store search and registry errors appear as a notification, and the Plugin Manager stays on screen.
- The image schedule button works on uploaded images, and the editor stays open while you edit.
- A failed plugin toggle moves the switch back.
- Each save shows one notification; a failed Durations save says it failed.
- Stats the server can't read show `--`.
- New `window.LEDEscape` (`html`, `attr`, `jsStringAttr`) replaces about 30 copied escapers. `window.escapeHtml` and `window.escapeAttribute` remain as aliases for plugin pages.
- The web service (`ledmatrix-web`) logs through `src.logging_config` like the
display service, so `journalctl -p err -u ledmatrix-web` works. Successful
GET/HEAD/OPTIONS requests (the UI's polling) are logged at DEBUG instead of
INFO; 4xx at WARNING, 5xx at ERROR. `LEDMATRIX_DEBUG=true` shows them again.
`web_interface/logging_config.py` is removed. The web cache
(`web_interface/cache.py`) now honours the TTL a value was stored with and is
thread-safe.
- `/api/v3` routes answer an exception they don't handle themselves from one
blueprint error handler, with the same `{status, message, details}` body the
53 removed per-route catch-alls returned. `ErrorCategory` and the
`error_category` key are removed from `src.web_interface.errors` (nothing read
them); `exception_error_response()` replaces the `from_exception` +
`error_response` pairs. A failing plugin action script's error now names the
real failure instead of `UnboundLocalError`.
- Installing a Starlark app works on a fresh install (#604). `starlark-apps/`
is created by whichever service reaches it first, and on a fresh install
that was usually the root display service, so the web interface could not
write to it and every install path answered "Failed to install from
repository". The display service now hands the directory and its contents
to the checkout's owner on every start (a no-op when not root or when the
checkout belongs to root), which also repairs devices already affected; a
permission error from the install routes names the directory and the fix.
- Clicks on plugin cards reach `handlePluginAction` (#605). Every click took
a copied fallback that asked twice before uninstalling and sent Starlark app
uninstalls to `POST /plugins/uninstall` instead of
`DELETE /starlark/apps/<id>`. A failed plugin toggle no longer always says
"A plugin operation is already in progress".
- The web interface starts with an absolute `plugin_system.plugins_directory`
(#616); it crashed at import with `NameError: project_root`.
- Stopping a Pixlet editor that ignores SIGTERM restarts the display instead
of answering 500 and leaving the panel dark (#625).
- Removed dead routes and files (#609): `POST /plugins/authenticate/spotify`
and `/ytm` (the music plugin runs its auth scripts through `web_ui_actions`),
`POST /plugins/of-the-day/json/upload` and `/json/delete` (they used the
wrong plugin id), `js/plugins/store_manager.js`, `js/config/diff_viewer.js`
and `js/htmx-sse.js`, and `web_interface/run.sh`. `htmx-config.js` no longer
replaces `console.error` / `console.warn`, which hid some real errors.
### Plugin error reporting ### Security (request paths and inline handlers, siblings of #561)
- `/api/v3/errors/summary` and `/api/v3/errors/plugin/<id>` report the errors
the display service recorded. They used to read the web process's own error
aggregator, which never records anything, so they always answered "no
errors". The display service now publishes a bounded snapshot to the shared
cache (`plugin_error_snapshot`, at most every 10 seconds and only on change;
`src/error_aggregator.py`, started from `DisplayController.__init__`).
Responses keep their shape and add `snapshot_available`, `generated_at` and
`clear_pending`; exception text has credentials redacted.
- `POST /api/v3/errors/clear` records a request (`plugin_error_clear_request`)
the display service applies within about 5 seconds; reads hide the cleared
errors at once. It accepts `"all": true`, and `cleared_count` can be `null`
when the count is only known to the display service.
- The Logs tab has a **Plugin errors** panel: per-plugin counts, repeating
errors and a Clear button.
- Credential redaction in exception text (`src/redaction.py`) takes time
proportional to the text, not its square. Two patterns were quadratic: URL
`user:password@`, on a long unbroken run of letters or digits (a hex digest,
an ID), and `Authorization:` followed by a long run of whitespace. Either
used to stall every thread of the display service for up to seconds each
time the snapshot was published: about 0.5s for 20k characters of hex, 8s
for 20k spaces. What gets redacted is unchanged.
### Wi-Fi
- WiFi status messages reach the panel (#605). The display controller looked
for `wifi_status.json` one directory above the repo; both sides now use
`wifi_manager.get_wifi_status_path()`, the file is written atomically, and
the plugin that resumes afterwards redraws the whole panel.
- The captive-portal checks (`/generate_204` and friends) also detect an access point brought up through NetworkManager, the fallback `enable_ap_mode` uses without hostapd; only hostapd was checked, so phones on that AP were told the internet worked.
- The WiFi monitor daemon re-reads `wifi_config.json` when it changes, so the "auto-enable AP mode" toggle takes effect without restarting the daemon.
- Disconnecting from WiFi in the web UI no longer runs an AP-mode check that could never enable the AP; it only added seconds of waiting. The daemon still enables the AP after its grace period.
- The WiFi status message file follows each WiFi manager's own config directory, and the config path falls back to this checkout rather than `/home/ledpi/LEDMatrix`.
- A wrong Wi-Fi password is reported as one again ("Incorrect password for ..."); the fallback that restores the old network or brings up the setup AP was replacing the signal.
- Wi-Fi disconnect takes the saved connection profile down.
- `wifi_config.json` is written atomically, and a save that fails now gets a 500.
### Fonts
- Fonts tab: the preview endpoint renders BDF fonts with the panel's own rasterizer instead of refusing them. (The Fonts page still skips the request for `.bdf`; enabling it there is a separate template change.)
- A plugin font declared as a `.zip` URL is served as the font extracted from it after a restart, instead of registering the archive itself. Font downloads time out after 30s and land in the cache only once complete, so an interrupted download is retried rather than served forever.
- BDF fonts: `FontManager.get_font()` and `element_style.load_font()` no longer hand one `freetype.Face` to every thread. BDF faces come from `load_bdf_face`, which already caches them per thread; TrueType fonts are cached as before. `element_style`'s font cache is locked (a concurrent eviction could raise `KeyError`).
- `FontManager.clear_cache()` and unregistering a plugin's fonts bump `cache_generation`, so cached layouts are rebuilt.
- `plugin://` fonts load from the plugin's own install directory. `FontManager.register_plugin_fonts()` takes an optional `plugin_dir`.
- Bundled font paths no longer depend on the directory the process was started from.
- `FontManager.get_font()` returns a BDF font at its native size when asked for
a size the file doesn't contain (5x7.bdf at 8 or 10px, say). It used to
return PIL's default font, a different typeface, so a plugin that relied on
that will now render the font it asked for.
- The web UI's Fonts tab has a **Used by** column: the loaded plugins that
registered each font with `FontManager.register_manager_font()`, published
by the display service to the shared cache (`src/font_usage.py`) and merged
into `GET /api/v3/fonts/catalog` as `used_by`. Deleting a font a plugin
uses now names those plugins in the confirmation (it is not blocked).
`FontManager.forget_manager_fonts()` is new; unloading a plugin calls it.
### Security
- `POST /api/v3/plugins/assets/upload`, `GET .../assets/list` and - `POST /api/v3/plugins/assets/upload`, `GET .../assets/list` and
`POST .../assets/delete` validate `plugin_id` with `src/common/path_safety` `POST .../assets/delete` validate `plugin_id` with `src/common/path_safety`
@@ -620,24 +505,6 @@ core, the monorepo or the registry's third-party plugins calls them:
and add its own script. The store's View button opens only `http(s)` links. and add its own script. The store's View button opens only `http(s)` links.
- The uploaded-images list escapes each file's original name, path and ids; a - The uploaded-images list escapes each file's original name, path and ids; a
name like `<img src=x onerror=...>.png` was inserted as markup. name like `<img src=x onerror=...>.png` was inserted as markup.
- Installing from a URL (and a registry install whose manifest renames the plugin) refuses a plugin id that isn't a single safe name, so `../x` can no longer delete and replace a directory outside the plugins directory.
- Plugin uninstall and config reset refuse core config sections (`display`, `schedule`, ...) and ids with path parts. Uninstall still cleans the config of a plugin whose directory is already gone.
- A config field marked `x-secret` whose value is an object or array is saved to `config_secrets.json`, not to `config.json` in plain text.
- Restoring a backup onto a device without `config_secrets.json`, `wifi_config.json` or `ytm_auth.json` creates them with mode 640 instead of world-readable 644.
- Backup export skips a plugin `manifest.json` that isn't a JSON object instead of failing, and two exports in the same second no longer share a temp file or overwrite each other (the second gets a `-2` suffix).
- Every font that ships in `assets/fonts/` is protected from deletion; `MatrixChunky8X`, `MatrixLight6X`, `MatrixLight8X` and `ic8x8u` could be deleted from the Fonts tab.
- The raw config and secrets editors, and endpoints using `validate_request_json`, answer 400 for a JSON body that isn't an object.
- `fix_web_permissions.sh` makes `safe_plugin_rm.sh` and `safe_pip_install.sh` root-owned again after resetting ownership. A web-user-owned copy of either is a root shell, since sudo lets the web user run them as root. It also restores `config_secrets.json` to mode 640.
- Wi-Fi passwords are no longer stored in `config/wifi_config.json` (#608).
`WiFiManager` appended every joined network's SSID and password, in plain
text, to `saved_networks`, and nothing read them back (NetworkManager keeps
its own credentials). Loading the config now drops a `saved_networks` key and
rewrites the file, so passwords already on disk are removed.
- The installers no longer grant the web user passwordless root on
`display_controller.py`, `start_display.sh` and `stop_display.sh` (#606).
Those files are owned by the user, so the web user could rewrite them and
run them as root; nothing ran them through sudo. Existing devices keep the
old rules until the installer or `configure_web_sudo.sh` is run again.
### Display hardware settings the library refuses ### Display hardware settings the library refuses
@@ -680,38 +547,6 @@ core, the monorepo or the registry's third-party plugins calls them:
(`legacy_bool_as_object` in `src/plugin_system/schema_manager.py`). Nothing (`legacy_bool_as_object` in `src/plugin_system/schema_manager.py`). Nothing
is written at load; the next save of that plugin's settings stores the object. is written at load; the next save of that plugin's settings stores the object.
Other type mismatches still warn. Other type mismatches still warn.
- A plugin that is reloaded (switched off and on again from the web UI) imports its own modules again, not another plugin's. Plugins import their own files by bare name (`from sports import ...`), which resolves to the first plugin directory on `sys.path` that has the file; the loader only added a directory that was missing, so a reloaded plugin's directory stayed behind any loaded since. On a Pi, re-enabling UFC with hockey running failed with "cannot import name '_status_is_final' from 'sports'". A loading plugin's directory is now always moved to the front.
- `src/plugin_system/store_manager.py` (2,977 lines) is split into mixins: `store_registry.py` (registry, GitHub metadata, search, manifest validation), `store_install.py` (install paths and dependencies) and `store_update.py` (updates, rollback, local git state). `PluginStoreManager` is still imported from `store_manager.py` and has exactly the same methods and attributes; every method body is byte-identical.
- Unloading a plugin waits (up to 5s) for an in-flight `update()` before running `cleanup()`/`on_disable()`, and an update that finishes after the unload no longer puts the plugin back to ENABLED.
- A plugin whose load fails after its module was imported (constructor, `validate_config()` or `on_enable()` raising) no longer leaves that module cached: fixing the plugin and reloading it runs the new code without a restart. Its font registrations are dropped too.
- `POST /api/v3/plugins/limits/<id>` answers 400 for a limit that isn't a non-negative number (a string limit used to make every later update of that plugin raise). A bad cached limits record is ignored with a warning instead of raising.
- The config schema is found for a plugin installed as `ledmatrix-<id>` or in a directory named differently from its manifest id, resolved the way the loader resolves it (plugins/ is still searched before plugin-repos/). A plugin with no schema is logged once at DEBUG instead of a warning on every lookup.
- Installing from a URL over an existing install sets the old copy aside and restores it if the move fails, under the same per-plugin lock as a registry install.
- The operation queue refuses a second operation for a plugin whose first is still waiting (a double-clicked Install ran twice), and no longer keeps every finished operation in memory.
- `get_vegas_render_width()` reads `display_manager.width` first, as plugins are told to.
- Store and state files are read as UTF-8 regardless of the system locale.
- Docs: `update_interval` in `config.json` sets the scheduler's cadence only for a plugin whose manifest has none (TROUBLESHOOTING, PLUGIN_CONFIGURATION_GUIDE). The health/metrics reset and limits routes note that they only change the web process's view.
- A plugin whose `on_enable()` raises is no longer left registered: the next load retries it instead of reporting "already loaded" for a plugin that never ran.
- One plugin's `get_info()` raising no longer breaks the installed-plugins list; it is logged and shown with empty runtime info.
- `plugin_state.json` and the operation history are written atomically (temp file + rename) under their lock, so concurrent saves or a failed save can't leave a truncated file.
- Plugin dependency installs run one `pip` at a time during parallel startup loading.
- A failed store download no longer leaves its extraction directory in the temp dir.
- Test doubles: `draw_image()` on `MockDisplayManager`, `VisualTestDisplayManager` and `BoundsCheckingDisplayManager` now emits a `DeprecationWarning` — the real `DisplayManager` has no such method; use `display_manager.image.paste(img, (x, y))`. `MockDisplayManager.draw_text` accepts the real signature's `small_font`/`centered` and default `x`/`y`, and `VisualTestDisplayManager` logs draw errors at WARNING.
- Removed the unused `PluginOperationQueue.get_active_operations()`.
- Updating a plugin that was installed from a ZIP no longer tries to reinstall it from the LEDMatrix repository's own URL.
- Repository URLs with `.git` in the middle are no longer mangled. The URL helpers now live in `src/plugin_system/repo_urls.py`.
- Installing from a URL works when the repository's only branch isn't `main` or `master`.
- A missing required config field is reported once, by name.
- A plugin that went over `max_memory_mb` once is no longer refused on every call after that.
- `reload_plugin` reads the manifest from the plugin's discovered directory.
- Removed: `last_display` from plugin state info and `get_last_display()` (nothing recorded them); `PluginOperationQueue`'s `history_file` and `lazy_load` arguments; and `data/plugin_operations.json`, which nothing read.
- One plugin-directory resolver, `src/plugin_system/plugin_dirs.py`, behind
discovery, `PluginManager.get_plugin_directory`, `PluginLoader`, the store and
state reconciliation. A manifest's `id` wins over a directory merely named for
the id; hidden and `.standalone-backup-` directories are never treated as
plugins (auto-update could previously try to update a backup); ids like
`a/b` or `..` resolve to nothing everywhere. Installs where each directory is
named for its manifest id, the installer's layout, behave as before.
### Core ### Core
@@ -730,39 +565,6 @@ core, the monorepo or the registry's third-party plugins calls them:
handling, so the restore stopped at `config.json` with nothing restored. The handling, so the restore stopped at `config.json` with nothing restored. The
ownership step is now skipped where `os.chown` is missing. No behaviour ownership step is now skipped where `os.chown` is missing. No behaviour
change on the Pi. change on the Pi.
- `APIHelper`'s rate limit and the display-sync heartbeat/leader timeouts measure elapsed time with `time.monotonic()`. A wall-clock step (NTP correcting a Pi with no RTC) could stall API requests for as long as the step or fake a sync timeout. `get_request_stats()['last_request_time']` is still wall-clock time.
- `sudo_remove_directory()` tries each bash path the sudoers rule might name, as `install_requirements_file()` already did.
- An element's saved layout `scale` equal to its schema default is no longer treated as a user choice when the default is declared under an alias (`score` for `score_text`).
- `CacheError`/`ConfigError`/`PluginError`/`DisplayError` no longer write their key into the caller's `context` dict; the JSON log formatter stringifies values it can't encode instead of dropping the record.
- Removed `ErrorAggregator`'s unused JSON export (`export_path`, `export_to_file()`); nothing called it. Docstring fixes in `validate_file_upload`, `StartupValidator.raise_on_errors`, `DisplaySyncManager.set_on_new_cycle`, `dynamic_team_resolver` and `config_arrays`.
- `/api/v3/errors` shows each exception's real stack trace instead of `NoneType: None`.
- Backups record `src.__version__`.
- Removed: `BackgroundDataService`'s `queue_size` stat and `clear_completed_requests()`.
- `src.device_location` — a blank `Location` field on a Starlark (Tidbyt) app
now renders at the device's City / State / Country (geocoded once via
Open-Meteo and cached) instead of the app author's hard-coded default,
usually San Francisco. A location saved on the app still wins. With no
device city set, or when the lookup fails or finds no match, the app keeps
its own default (a failed lookup is retried after 30 minutes). Clearing an
app's location in the web UI now actually clears it; the save used to drop
the blank field, so the old value stayed.
- Fixed a memory leak in the display service (#605): `ErrorAggregator`
appended every plugin in the time window to a pattern's `affected_plugins`
on each repeat (3,000 errors from three plugins reached 2.5 million
entries).
- An expired cache record is refused without being parsed (#633).
`CacheManager.set` writes `timestamp` and `ttl` ahead of `data`, and
`DiskCache.get` reads the first 256 bytes to decide staleness, with the same
rules as before. A 53 MB MLB season file used to be parsed in full (about
1.8 s holding the GIL on a Pi 4, freezing the display) only to be thrown
away. Files in the old layout are parsed as before and convert when
rewritten.
- Cache internals (#613): `CacheManager` delegates memory-tier cleanup and
stats to `MemoryCache`; `list_cache_files` no longer holds the memory lock
during directory I/O; `BackgroundDataService.get_sport_cache_key()` formats
the key instead of building a whole `CacheManager` (and probing the cache
directory) on every call; the unused request queue is gone, and `priority=`
is accepted and documented as ignored.
### Cache permissions ### Cache permissions
@@ -817,8 +619,6 @@ core, the monorepo or the registry's third-party plugins calls them:
timeouts with a second bash path, and all reinstalls share a 10-minute timeouts with a second bash path, and all reinstalls share a 10-minute
budget, so a rollback finishes inside the unit's 30-minute limit instead of budget, so a rollback finishes inside the unit's 30-minute limit instead of
being killed mid-way. being killed mid-way.
- A hand-edited non-object `auto_update` value reads as off instead of raising at startup, and a failed result write no longer leaves a temp file behind.
- Overview "Check Updates" asks for the same confirmation as "Update Code" and shows the server's message. Both, and the Tools tab's git pull, show the restart-pending banner when the update needs a restart.
### Installers ### Installers
@@ -832,22 +632,6 @@ core, the monorepo or the registry's third-party plugins calls them:
`configure_web_sudo.sh` does the same before offering the rules for `configure_web_sudo.sh` does the same before offering the rules for
confirmation. `first_time_install.sh` also built that file at a fixed `/tmp` confirmation. `first_time_install.sh` also built that file at a fixed `/tmp`
path as root; `mktemp` now picks the name. path as root; `mktemp` now picks the name.
- `check_system_compatibility.sh` treats Python 3.13 (what Trixie ships) as supported and anything below 3.10 as an error.
- `configure_web_sudo.sh` run as the web user keeps the reboot/poweroff rules.
- `check_system_compatibility.sh` no longer reports installed packages as missing.
- `configure_wifi_permissions.sh` checks its rules with `visudo -c` before installing them, and grants the NetworkManager captive-portal `cp` and `rm` commands `wifi_manager` runs.
- `configure_web_sudo.sh` uses a random temp file and installs its rules with mode 440.
- The installer prints its completion summary before the `-y` reboot, and describes the setup access point as an open network (it was shown with a password it doesn't have).
- `fix_cache_permissions.sh` applies `setup_cache.sh`'s `ledmatrix`-group model instead of setting 777.
- `check_system_compatibility.sh` reports anything but Debian 13 (Trixie) as unsupported, and reaches its summary.
- `one-shot-install.sh`'s `retry()` retries (#606). It read `$?` after `!`,
which is always 0, so a failed command ran once and was reported as a
success. It now tries three times and returns the command's status; both
apt steps still warn and continue after their retries, and a clone that
keeps failing stops the install sooner, with its own message.
- One generator for the web sudoers rules, `scripts/install/lib_sudoers.sh`,
used by `first_time_install.sh` and `configure_web_sudo.sh` (#622); the two
copies had drifted.
### Small fixes (update-all, plugin system settings, scripts) ### Small fixes (update-all, plugin system settings, scripts)
@@ -885,9 +669,7 @@ core, the monorepo or the registry's third-party plugins calls them:
(only an explicit `web_display_autostart: false` keeps the web interface (only an explicit `web_display_autostart: false` keeps the web interface
down), so a missing key no longer shows as disabled. The shell scripts also down), so a missing key no longer shows as disabled. The shell scripts also
check `web_interface/blueprints/api_v3/`, which became a package, instead of check `web_interface/blueprints/api_v3/`, which became a package, instead of
reporting `api_v3.py` as missing. (`scripts/verify_web_ui.sh`, reporting `api_v3.py` as missing.
`scripts/diagnose_web_ui.sh` and `scripts/debug/debug_web_manual.py` were
later deleted as unreferenced; see Docs and developer tools.)
### Docs and developer tools ### Docs and developer tools
@@ -916,67 +698,6 @@ core, the monorepo or the registry's third-party plugins calls them:
`app.py` line numbers, `api_v3.py` paths, StreamManager method names, `app.py` line numbers, `api_v3.py` paths, StreamManager method names,
nonexistent version-bump scripts and `ledmatrix` service user references nonexistent version-bump scripts and `ledmatrix` service user references
removed. removed.
- A mypy ratchet in CI. `mypy-clean.txt` lists the 71 modules under `src/` that type-check clean, and the new "Type check (mypy ratchet)" job runs `python scripts/check_types.py` (mypy 1.20.2 on exactly those files) so they stay clean; add a module when you make it clean (see CONTRIBUTING.md). The manual pre-commit `mypy` hook runs the same script. 35 modules were made clean for it with annotation-only fixes, no behaviour change. Their public signatures only widened (`declared_min_version()` now says it returns the manifest's value as-is, `Any`); `DynamicTeamResolver._rankings_cache` is annotated as the abbreviation-to-rank dict it holds. `mypy.ini` treats numpy and orjson as `Any`, so it parses with `python_version = 3.10` against numpy 2.3+ stubs and gives the same result whether orjson is installed or not.
- CI runs the web UI's DOM test suites (jsdom against the real server-rendered pages and API) in a new **Web UI JS tests** job, with the web interface started in emulator mode; `REQUIRE_DOM=1` makes a suite that can't run fail instead of being skipped. Two suites that had gone stale were fixed: the Tools suite now installs `LEDEscape` the way `base.html` does and supplies sample Starlark apps when the server has none, and the Store suite no longer assumes the registry has 48 plugins or fewer.
- CI installs `web_interface/requirements.txt` too, so flask-limiter, flask-compress and the web floors are tested. `test_api_helper_does_not_hand_set_brotli` now checks what it meant: core doesn't add `br` itself, and `requests` may advertise it when a brotli decoder is installed.
- All Discord links point to the LEDMatrix server's invite.
- `pytz` may be any release before 2027, so current timezone data installs; `requirements-test.txt` caps `psutil` below 7 like the runtime requirements and allows `pytest-cov` up to 7.x (checked against pytest 9 with the CI coverage run).
- The Claude GitHub Actions workflows pin `anthropics/claude-code-action` to a commit SHA like the other actions.
- `mypy.ini` parses again. A multi-line `exclude` and trailing comments on values made mypy refuse the whole file, so none of its settings applied and the pre-commit hook failed with "Missing target". The mypy hook is now manual (`pre-commit run mypy --hook-stage manual`) while the ~500 existing type errors in `src/` are paid down.
- `.gitignore` ignores everything in `config/` except the templates; `ytm_auth.json`, `saved_repositories.json`, `wifi_status.json` and `font_overrides.json` weren't ignored.
- `.sh` and `.service` files are always checked out with LF line endings.
- The Claude code-review check is skipped on pull requests from forks, which get no secrets and always failed it.
- Doc fixes: emulator guide (Python 3.10+, `emulator_config.json` isn't in the repo), README's nonexistent "API Metrics" feature, a stale route count, and missing index entries for the scroll-performance and offscreen-rendering docs and the frame-soak and render-bench scripts.
- `src/common/README.md` lists `frame_timing`, `json_body` and `render_gate`.
- New `scripts/README.md` lists every script.
- New `docs/ARCHITECTURE.md` (processes, shared state, display loop, plugin system, web UI) and `docs/PERMISSIONS.md` (owners, modes, both sudoers files, repair scripts).
- Deprecated plugin APIs are marked in the plugin docs.
- `src/common/README.md` covers every module.
- Stale setup, service and troubleshooting claims are corrected.
- Deleted 13 scripts nothing referenced (#607): `utils/cleanup_venv.sh`,
`utils/clear_python_cache.sh`, `install/migrate_config.sh`,
`install/debug_install.sh`, `debug/debug_web_manual.py`,
`diagnose_web_ui.sh`, `verify_web_ui.sh`, `fix_internet_connectivity.sh`,
`diagnose_plugin_permissions.sh`, `dev/validate_python.py`,
`download_nba_logos.py` (with `README_NBA_LOGOS.md`) and
`setup_plugin_repos.py`, all under `scripts/`; also `docs/archive/` and
`PLUGIN_IMPLEMENTATION_SUMMARY.md`. `config.template.json` no longer carries
`plugin_system.auto_discover`, `auto_load_enabled` or `development_mode`,
which nothing reads (existing configs keep them). About 20 docs had stale
claims corrected against the code.
- `test/test_js_unit_suites.py` runs every `test/js/unit/*.js` suite under
pytest; CI used to run one of the eight (#605).
- New test `test/test_common_is_hardware_free.py`: `src/common` must import
without `rgbmatrix`, and never import `src.base_classes`,
`src.display_manager` or `src.plugin_system` at module level, so plugins can
use it on machines with no panel library.
### Removed
- **The skin system.** Skins never rendered with the current scoreboard
plugins, so they are gone rather than "not supported yet": `src/skin_system/`,
`skins/`, `scripts/validate_skin.py`, `GET /api/v3/skins`, the store's
`"type": "skin"` handling and `docs/SKIN_SYSTEM.md` / `docs/CREATING_SKINS.md`.
A `skin` or `skin_options` key left in a plugin's saved config still loads
and saves without a validation error; it is ignored, and the next save of
that plugin's settings removes it (unless the plugin's own schema declares
the key).
- **`src/base_classes/`** (`SportsCore`, the sport and mode classes,
`CelebrationMixin`, the rotation strategies, `data_sources`,
`api_extractors`). No known plugin imports it. A plugin that does must use
`src.common` or its own copy of the code.
- **Unused `src.common` modules and plugin-system helpers** (#608):
`src/common/config_helper.py`, `display_helper.py`, `game_helper.py`,
`utils.py` and `error_handler.py` (its re-exports leave `src.common`'s
`__all__`), `src/plugin_system/health_monitor.py` (`PluginHealthMonitor`,
whose loop did nothing; `PluginHealthTracker` is unchanged), and
`src.plugin_system.get_store_manager` / `__api_version__`. Nothing in core,
the scripts or the plugin monorepo imported them. `APIHelper`, `TextHelper`,
`ScrollHelper`, `LogoHelper` and the adaptive-layout exports of `src.common`
are unchanged. The same change removed unused methods from `ConfigService`,
`PluginStateManager`, `PluginManager`, `PluginExecutor`, `PluginLoader`,
`PluginStoreManager`, `VegasModeConfig` and `DisplayController`; none had
callers in core, the scripts or the monorepo.
## 3.4.0 ## 3.4.0
+1 -1
View File
@@ -63,7 +63,7 @@ ChuckBuilds, and any other forums hosted by or affiliated with the project.
Instances of abusive, harassing, or otherwise unacceptable behavior may be Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement on the reported to the community leaders responsible for enforcement on the
[LEDMatrix Discord](https://discord.gg/RdrC37rEag) (DM a moderator or [LEDMatrix Discord](https://discord.gg/uW36dVAtcT) (DM a moderator or
ChuckBuilds directly) or by opening a private GitHub Security Advisory if ChuckBuilds directly) or by opening a private GitHub Security Advisory if
the issue involves account safety. All complaints will be reviewed and the issue involves account safety. All complaints will be reviewed and
investigated promptly and fairly. investigated promptly and fairly.
+4 -12
View File
@@ -9,7 +9,7 @@ improvements, and code changes.
- **Bugs / feature requests**: open an issue using one of the templates - **Bugs / feature requests**: open an issue using one of the templates
in [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/). in [`.github/ISSUE_TEMPLATE/`](.github/ISSUE_TEMPLATE/).
- **Real-time discussion**: the - **Real-time discussion**: the
[LEDMatrix Discord](https://discord.gg/RdrC37rEag). [LEDMatrix Discord](https://discord.gg/uW36dVAtcT).
- **Plugin development**: - **Plugin development**:
[`docs/PLUGIN_DEVELOPMENT_GUIDE.md`](docs/PLUGIN_DEVELOPMENT_GUIDE.md) [`docs/PLUGIN_DEVELOPMENT_GUIDE.md`](docs/PLUGIN_DEVELOPMENT_GUIDE.md)
and the [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins) and the [`ledmatrix-plugins`](https://github.com/ChuckBuilds/ledmatrix-plugins)
@@ -58,18 +58,10 @@ integration tests.
3. **Keep PRs focused.** One conceptual change per PR. If you find 3. **Keep PRs focused.** One conceptual change per PR. If you find
adjacent bugs while working, fix them in a separate PR. adjacent bugs while working, fix them in a separate PR.
4. **Follow the existing code style.** The pre-commit hooks run 4. **Follow the existing code style.** The pre-commit hooks run
`flake8` (E9, F63, F7, F82 plus bugbear `B` checks), `bandit`, `flake8` (E9, F63, F7, F82 plus bugbear `B` checks), `mypy` on
and `gitleaks` — install the CLI with `src/`, `bandit`, and `gitleaks` — install the CLI with
`python -m pip install pre-commit`, then run `python -m pip install pre-commit`, then run
`pre-commit install` so they run on every commit. Type checking `pre-commit install` so they run on every commit; HTML/JS in
is a ratchet while the existing mypy errors in `src/` are paid
down: `mypy-clean.txt` lists the modules that type-check clean, and
CI runs `python scripts/check_types.py` (also the manual hook
`pre-commit run mypy --hook-stage manual`) to keep every listed
module clean. When you make another module clean, add it to the
list (sorted); don't take one off to get CI green. Keep type fixes
annotation-only where you can -- widen a hint rather than delete a
defensive runtime check mypy calls unreachable. HTML/JS in
`web_interface/` follows the patterns already in `templates/v3/` `web_interface/` follows the patterns already in `templates/v3/`
and `static/v3/`. and `static/v3/`.
5. **Update documentation** alongside code changes. If you add a 5. **Update documentation** alongside code changes. If you add a
+2 -2
View File
@@ -33,7 +33,7 @@ I'm trying to be open to constructive criticism and support, as long as it's a r
- Show support on Youtube: https://www.youtube.com/@ChuckBuilds - Show support on Youtube: https://www.youtube.com/@ChuckBuilds
- Check out the write-up on my website: https://www.chuck-builds.com/led-matrix/ - Check out the write-up on my website: https://www.chuck-builds.com/led-matrix/
- Stay in touch on Instagram: https://www.instagram.com/ChuckBuilds/ - Stay in touch on Instagram: https://www.instagram.com/ChuckBuilds/
- Want to chat? Reach out on the LEDMatrix Discord: [https://discord.gg/RdrC37rEag](https://discord.gg/RdrC37rEag) - Want to chat? Reach out on the LEDMatrix Discord: [https://discord.com/invite/uW36dVAtcT](https://discord.gg/dfFwsasa6W)
- Feeling Generous? Consider sponsoring this project or sending a donation (these AI credits aren't cheap!) - Feeling Generous? Consider sponsoring this project or sending a donation (these AI credits aren't cheap!)
----------------------------------------------------------------------------------- -----------------------------------------------------------------------------------
@@ -948,7 +948,7 @@ sudo systemctl enable ledmatrix-web.service
- **On-Demand Controls**: Start specific displays (weather, stocks, sports) on demand - **On-Demand Controls**: Start specific displays (weather, stocks, sports) on demand
- **Service Management**: Start/stop the main display service - **Service Management**: Start/stop the main display service
- **System Controls**: Restart, update code, and manage the system - **System Controls**: Restart, update code, and manage the system
- **System Stats**: CPU, memory and temperature on the Overview tab - **API Metrics**: Monitor API usage and system performance
- **Logs**: View system logs in real-time - **Logs**: View system logs in real-time
### Troubleshooting Web Interface ### Troubleshooting Web Interface
+1 -1
View File
@@ -16,7 +16,7 @@ Use one of these channels, in order of preference:
maintainer. maintainer.
- Direct link: <https://github.com/ChuckBuilds/LEDMatrix/security/advisories/new> - Direct link: <https://github.com/ChuckBuilds/LEDMatrix/security/advisories/new>
2. **Discord DM**. Send a direct message to a moderator on the 2. **Discord DM**. Send a direct message to a moderator on the
[LEDMatrix Discord](https://discord.gg/RdrC37rEag). Don't post in [LEDMatrix Discord](https://discord.gg/uW36dVAtcT). Don't post in
public channels. public channels.
Please include: Please include:
+7 -15
View File
@@ -1,20 +1,12 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Legacy entry point: runs ``run.py``, which is the one to use.
``python3 run.py`` (``-e`` for the emulator, ``-d`` for debug logging) is how
the display service and the docs start LEDMatrix. This file used to import
``src.display_controller.main`` directly, which skipped what run.py sets up
first -- ``sys.dont_write_bytecode`` (root-owned ``__pycache__`` in plugin
directories blocks the web service from updating them), the ``-e``/``-d``
flags, and the logging configuration. It now runs run.py exactly as
``python3 run.py`` would, with the same arguments.
"""
import os import os
import runpy import sys
# Add the project root directory to Python path
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from src.display_controller import main
if __name__ == "__main__": if __name__ == "__main__":
runpy.run_path( main()
os.path.join(os.path.dirname(os.path.abspath(__file__)), "run.py"),
run_name="__main__",
)
+3 -8
View File
@@ -127,7 +127,7 @@ then normal rotation.
| Circuit breaker | [`plugin_health.py`](../src/plugin_system/plugin_health.py) (`PluginHealthTracker`: 3 consecutive failures open the circuit for 300 s) | | Circuit breaker | [`plugin_health.py`](../src/plugin_system/plugin_health.py) (`PluginHealthTracker`: 3 consecutive failures open the circuit for 300 s) |
| Resource metrics | [`resource_monitor.py`](../src/plugin_system/resource_monitor.py) | | Resource metrics | [`resource_monitor.py`](../src/plugin_system/resource_monitor.py) |
| Config schemas and defaults | [`schema_manager.py`](../src/plugin_system/schema_manager.py) | | Config schemas and defaults | [`schema_manager.py`](../src/plugin_system/schema_manager.py) |
| Install, update, uninstall | [`store_manager.py`](../src/plugin_system/store_manager.py) (`PluginStoreManager`), with its methods split across [`store_registry.py`](../src/plugin_system/store_registry.py) (registry, GitHub), [`store_install.py`](../src/plugin_system/store_install.py) and [`store_update.py`](../src/plugin_system/store_update.py) | | Install, update, uninstall | [`store_manager.py`](../src/plugin_system/store_manager.py) (`PluginStoreManager`) |
| Core-version gate | [`compatibility.py`](../src/plugin_system/compatibility.py) | | Core-version gate | [`compatibility.py`](../src/plugin_system/compatibility.py) |
Discovery scans only `plugin_system.plugins_directory` (default Discovery scans only `plugin_system.plugins_directory` (default
@@ -158,13 +158,8 @@ everything else through `_reinstall_with_rollback()`.
- **API.** [`blueprints/api_v3/`](../web_interface/blueprints/api_v3/) is one - **API.** [`blueprints/api_v3/`](../web_interface/blueprints/api_v3/) is one
blueprint at `/api/v3`, split by area: `backup.py`, `config.py`, blueprint at `/api/v3`, split by area: `backup.py`, `config.py`,
`display.py`, `fonts.py`, `misc.py` (health, logs, errors, cache, sync), `display.py`, `fonts.py`, `misc.py` (health, logs, errors, cache, sync),
`starlark.py`, `system.py` (service actions, updates, git), `wifi.py`, and `plugins.py`, `starlark.py`, `system.py` (service actions, updates, git),
the plugin routes: `plugins.py` (installed list, enable/disable, plugin `wifi.py`. `__init__.py` defines the blueprint and shared helpers and
actions), `plugin_store.py` (install, update, uninstall, store),
`plugin_config.py` (config, schema, reset), `plugin_assets.py` (uploads,
plugin static files), `plugin_health.py` (health, metrics, limits),
`plugin_operations.py` (operation history, state reconciliation) and
`plugin_calendar.py`. `__init__.py` defines the blueprint and shared helpers and
imports the modules so their routes register. Endpoints are listed in imports the modules so their routes register. Endpoints are listed in
[REST_API_REFERENCE.md](REST_API_REFERENCE.md). [REST_API_REFERENCE.md](REST_API_REFERENCE.md).
- **Front end.** HTMX loads each tab's partial on first open - **Front end.** HTMX loads each tab's partial on first open
+1 -4
View File
@@ -128,9 +128,6 @@ Read by `src/vegas_mode/config.py` (`VegasScrollConfig.from_config`). See
| `min_content_separation` | int, `24` | | `min_content_separation` | int, `24` |
| `min_cut_gap` | int, `6` | | `min_cut_gap` | int, `6` |
| `continuous_scroll` | bool, `true` | | `continuous_scroll` | bool, `true` |
| `offscreen_prefetch` | bool, `true` — render every plugin's ticker content on the background thread, each on its own canvas. `false` restores handing canvas-bound plugins to the render thread, one pause at a time. Temporary; see [OFFSCREEN_RENDERING.md](OFFSCREEN_RENDERING.md) |
| `prefetch_gate` | bool, `true` — let that background thread run Python only while the render thread is waiting for the panel, so the render thread never waits for the GIL when a refresh comes round. Only takes effect with the rebuilt rgbmatrix binding (`scripts/build_rgbmatrix_nogil.sh`). See [OFFSCREEN_RENDERING.md](OFFSCREEN_RENDERING.md) |
| `switch_interval_ms` | float, `0` — experimental: shorten Python's GIL switch interval to this many ms while Vegas runs. `0` leaves the default (5 ms) alone |
| `smooth_scroll` | bool, `true` — move a whole number of pixels per panel refresh, locked to vsync. `scroll_speed` is snapped to the nearest speed the panel can show that way (at 95Hz: 95, 47.5, 31.7 px/s…), measured against the panel's real refresh rate once scrolling starts | | `smooth_scroll` | bool, `true` — move a whole number of pixels per panel refresh, locked to vsync. `scroll_speed` is snapped to the nearest speed the panel can show that way (at 95Hz: 95, 47.5, 31.7 px/s…), measured against the panel's real refresh rate once scrolling starts |
| `sub_pixel_blend` | bool, `false` — the older smoothing: advance by elapsed time and blend neighbouring pixel columns. Looks anti-aliased in the web preview but shimmers on the panel and is not locked to the refresh. Overrides `smooth_scroll` when on | | `sub_pixel_blend` | bool, `false` — the older smoothing: advance by elapsed time and blend neighbouring pixel columns. Looks anti-aliased in the web preview but shimmers on the panel and is not locked to the refresh. Overrides `smooth_scroll` when on |
| `extend_threshold_screens` | float, `2.0` | | `extend_threshold_screens` | float, `2.0` |
@@ -180,5 +177,5 @@ See [PLUGIN_CONFIG_CORE_PROPERTIES.md](PLUGIN_CONFIG_CORE_PROPERTIES.md).
| Key | Meaning | | Key | Meaning |
|---|---| |---|---|
| `github.api_token` | Optional GitHub token the Plugin Store uses to avoid API rate limits (`src/plugin_system/store_registry.py`) | | `github.api_token` | Optional GitHub token the Plugin Store uses to avoid API rate limits (`src/plugin_system/store_manager.py`) |
| `<plugin-id>.*` | Secrets a plugin declares with `"x-secret": true` in its config schema; merged into that plugin's config at load time | | `<plugin-id>.*` | Secrets a plugin declares with `"x-secret": true` in its config schema; merged into that plugin's config at load time |
+6 -6
View File
@@ -17,13 +17,13 @@ The LEDMatrix emulator allows you to run and test LEDMatrix displays on your com
## Prerequisites ## Prerequisites
### System Requirements ### System Requirements
- Python 3.10 or higher - Python 3.7 or higher
- Windows, macOS, or Linux - Windows, macOS, or Linux
- At least 2GB RAM (4GB recommended) - At least 2GB RAM (4GB recommended)
- Internet connection for plugin downloads - Internet connection for plugin downloads
### Required Software ### Required Software
- Python 3.10+ - Python 3.7+
- pip (Python package manager) - pip (Python package manager)
- Git (for plugin management) - Git (for plugin management)
@@ -50,7 +50,8 @@ pip install -r requirements-emulator.txt
``` ```
This installs: This installs:
- `RGBMatrixEmulator` - the emulation library (and whatever it depends on) - `RGBMatrixEmulator` - The core emulation library
- Additional dependencies for display adapters
### 3. Install Standard Dependencies ### 3. Install Standard Dependencies
@@ -62,9 +63,8 @@ pip install -r requirements.txt
### 1. Emulator Configuration File ### 1. Emulator Configuration File
The emulator uses `emulator_config.json` for configuration. It isn't in The emulator uses `emulator_config.json` for configuration. Here's the
the repo (it's gitignored): RGBMatrixEmulator writes it on first run. default configuration as it ships in the repo:
A typical file looks like this:
```json ```json
{ {
-388
View File
@@ -1,388 +0,0 @@
# Offscreen Rendering
**Status (2026-09-24):** step 1, offscreen rendering, is implemented
(`DisplayManager.offscreen()`, the adapter on the prefetch thread, the plugin
lock). Steps 2 and 3 are proposed. When all three land, this file becomes the
reference for how plugin content is rendered off the render thread.
First soak of step 1 on hdpi (50 px/s, `pwm_bits` 7, preview open, 8-minute
runs, A/B/B/A):
| build | late | by 1 | 2 | 3–5 | 6+ | freezes | render-thread fetches |
|---|---|---|---|---|---|---|---|
| #628 | 0.53% | 82 | 2 | 3 | 2 | 3 | 6 |
| step 1 | 0.63% | 78 | 63 | 17 | 2 | 1 | 0 |
| step 1 | 0.42% | 77 | 23 | 10 | 0 | 0 | 0 |
| #628 | 0.37% | 84 | 5 | 3 | 3 | 2 | 14 |
It does what it was built to: no plugin is fetched on the render thread, and
freezes fell from 5 to 1. But frames 2–5 refreshes late rose. The rendering
moved to the prefetch thread still needs the GIL, and the render thread waits
for it (risk 5 below). The late rate did not improve overall. The 1–2 s
freezes appear in both builds and have a separate, not yet identified cause.
The GIL fix, measured on hdpi (90 px/s, `pwm_bits` 8, preview open, 8-minute
runs after a 2-minute warm-up, order A B C C B A, 2026-09-24). Each arm pools
two runs, about 81,000 frames:
| arm | late | by 1 | 2 | 3–5 | 6+ | 2+ late per 10k frames | freezes |
|---|---|---|---|---|---|---|---|
| A: step 1 as is | 0.90% | 575 | 64 | 91 | 9 | 20.1 | 0 |
| B: `switch_interval_ms` 1 | 0.78% | 510 | 105 | 23 | 2 | 15.8 | 0 |
| C: `prefetch_gate` | **0.60%** | 471 | 11 | 7 | 2 | **2.5** | 0 |
The gate removes the frames the render thread spent waiting for the GIL, and
it costs the prefetch nothing that shows: it parked the thread for 3–6 s per
run, and the next group was ready at every strip extension in every arm.
`prefetch_gate` is therefore on by default; `switch_interval_ms` stays an
off-by-default experiment. What is left is almost all one refresh late, which
is the per-frame budget (a 6.75 ms p50 blit in a refresh the panel holds at
83–85 Hz while rendering), not contention.
The runs restart the service, so the hourly sports refresh never fell inside
one. That refresh is its own case: about twenty ESPN chunk-fetch threads at
once, which the gate does not cover (it gates only the prefetch thread).
## The problem
Vegas mode builds its ticker from every plugin's content. Most of that work
already happens on a background prefetch thread
(`RenderPipeline.start_prefetch`). But any plugin whose content needs the
**shared display canvas** is deferred to the render thread
(`RenderPipeline.drain_deferred`), one plugin every two seconds. The code's
own comments put each of those at 40–600 ms, and the render thread presents no
frames while one runs.
On hdpi (Pi 4, 512×64) most plugins take that path: geochron, tide-display,
news, hockey-scoreboard, ledmatrix-stocks, incoming-packages, clock-simple,
countdown, birdnet-go, ledmatrix-music and odds-ticker. They arrive in bursts
("Whole group deferred; strip will extend as it drains") every minute or so,
12 fetches in five minutes. That is the "occasional pause" a viewer sees.
An 8-minute soak (`scripts/frame_soak.py --preview`) of the #628 build on
hdpi:
| late by | frames |
|---|---|
| 1 refresh | 238 |
| 2 | 32 |
| 3–5 | 30 |
| 6+ | 5 |
| freezes ≥ 250 ms | 2 (0.97 s total) |
The 3+ rows and the freezes are the pauses. The single-refresh row is a
separate problem: the blit is 6 ms of a 10 ms refresh, so there is little
slack. It is covered under *What this does not fix*.
## Why a plugin is canvas-bound
The plugin-facing canvas is a set of shared attributes on `DisplayManager`:
`image`, `draw`, `matrix`, and the `width`/`height` properties that read from
`matrix`. Three adapter paths (`src/vegas_mode/plugin_adapter.py`) need them,
and each returns `None` under `offscreen_only=True` so the plugin is queued for
the render thread:
1. **Display capture** (`_capture_display_content`): clear the canvas, call
`plugin.display()`, copy `display_manager.image`. Used by any plugin
without `get_vegas_content()` or a populated `scroll_helper`.
2. **Scroll-content generation** (`_trigger_scroll_content_generation`): a
ticker plugin whose `scroll_helper.cached_image` is empty is made to build
it by calling `display(force_clear=True)` or `_create_scrolling_display()`.
Both draw on the canvas.
3. **Narrowed rendering** (`DisplayManager.render_size`): swaps the shared
`matrix`, `image` and `draw` for a narrower set so the plugin lays out for
`render_width_pct`. The render thread would see the swap mid-frame.
The render thread keeps the canvas coherent only because nothing else touches
it at the same time. A background thread can't use it.
## The design: a per-thread render target
`capture_mode()` is already per-thread (#423 made its state a
`threading.local`, so a background capture no longer suppresses the render
loop's pushes). The same move applies to the canvas itself:
```python
with display_manager.offscreen(width=None, height=None) as surface:
plugin.display(force_clear=True)
content = surface.image.copy()
```
For the **calling thread only**, inside the block:
| accessor | resolves to |
|---|---|
| `display_manager.image`, `.draw` | the surface's own image and draw: a fresh black canvas, `fontmode = "1"` |
| `display_manager.matrix` | a logical proxy reporting the surface size, so `width`/`height` and plugins that read `matrix.width` follow it. Hardware calls through it (`SetImage`, `SwapOnVSync`, `Clear`, brightness writes) are inert. |
| `update_display()`, `clear()` | canvas-only: the block implies capture mode, which is already per-thread |
| `set_scrolling_state()`, `set_frame_hold()` | no-ops, so a plugin's `display()` cannot re-pace the live scroll. Today it can, when it is captured on the render thread. |
Every other thread sees the real canvas, unchanged. The render loop in
particular keeps presenting while a plugin draws elsewhere.
### Implementation sketch
- `image`, `draw` and `matrix` become properties over `_image`, `_draw` and
`_matrix`, plus a thread-local current surface. The getter returns the
surface's value when the calling thread has one, else the shared one; setters
mirror that. That costs about 0.1 µs per access, and `update_display()` reads
each a handful of times per frame. Every existing `self.image = ...` in
`DisplayManager` (`clear()`, setup, fallback) keeps working and becomes
thread-correct for free.
- `render_size()` is rebuilt on `offscreen()`: it creates or narrows the
calling thread's surface instead of swapping shared state.
- `offscreen()` nests and always restores on exit, including when the plugin
raises.
- `VisualDisplayManager` (the plugin test harness) gets the same method, for
parity.
### Adapter changes
- `get_content(offscreen_only=True)` stops returning `None` for the three
paths above. Each runs inside `display_manager.offscreen(render_width)`.
- `_capture_display_content` and `_trigger_scroll_content_generation` drop
their "copy the shared image, restore it afterwards" bookkeeping, since the
shared image is never touched.
- **Take the plugin's lock.** `PluginManager.get_plugin_lock()` keeps
`update()` and `display()` mutually exclusive in normal rotation, but Vegas
never takes it, so today's render-thread captures already race
`update()`. Off the render thread the adapter can afford to wait: blocking
acquire with a timeout (proposed 2 s). On timeout it keeps the cached segment
and tries again next group.
- `drain_deferred()` and the deferred queue are deleted. The only render-thread
fetch left is the inline fallback when no prepared group is ready, which in
practice is the first extension. Prefetching at start removes that too.
## Keeping live content fresh
Offscreen rendering is also what makes fresh sports scores possible. Today a
plugin's segment is drawn when its group is prefetched, and the strip carries
7,000–10,000 px of content ahead of the viewport (hdpi logs: "7153px still
ahead", "9842px ahead"). At ~100 px/s, a score drawn now reaches the screen
70–100 seconds later. When a plugin reports new data, Vegas only drops its
cache (`invalidate_pending_updates`), so the change is drawn on the plugin's
*next* turn, several minutes later. A segment already in the strip scrolls by
with the data it was drawn with.
That was the right trade while every redraw of a canvas-bound plugin stalled
the scroll. Off the render thread a redraw costs the scroll nothing, so the
strip can afford three things.
### 1. Refresh at the gate
Before a segment enters the viewport, check whether its plugin has updated
since the segment was drawn. If it has, redraw it offscreen and replace it
while it is still out of sight. Width changes are fine here, because
everything from that segment onward is still invisible.
The gate sits `lead` pixels ahead of the viewport's right edge:
`lead = max(one screen, speed × (render time + margin))`. The render time is
the plugin's own, measured on each render (sports cards take the longest,
hundreds of ms up to seconds per the prefetch notes). A plugin whose render
does not finish before its segment reaches the viewport keeps the old segment.
The scroll never waits for it.
Content is then at most `lead / speed` seconds old when it appears, a few
seconds instead of minutes, without changing how far ahead the rotation
fetches.
### 2. Replace ahead of the screen
When a plugin reports new data (the Vegas update tick already names them), any
of its segments that are **anywhere ahead of the viewport** are redrawn and
replaced straight away, not only at the gate. That covers the long stretch of
strip between prefetch and the gate.
### 3. Update on screen
A segment that is already **visible** is patched in place when the redrawn
version has the same geometry: the same total width, and the same width for
each card (a sports plugin returns one image per game, joined with
`intra_plugin_gap`). Scoreboard cards keep a fixed layout, so a score change
patches in and the digits update as the card scrolls past. The patch is a
pixel copy of one card (a 150×64 card is ~29 KB) applied by the render thread
between frames, so a frame never shows half of a patch.
When the geometry differs (a game added or dropped, a card that grew), the
visible part cannot change without a jump. Only the cards not yet on screen
are replaced, and only if the geometry up to that point is unchanged. Otherwise
the segment keeps its snapshot until it has scrolled off.
### Avoiding wasted work
- **Change detection.** `run_scheduled_updates_with_changes()` names a plugin
whenever its `update()` ran, not when its data changed. On hdpi
`clock-simple` and `ledmatrix-music` are named on every 4-second tick. A
redraw whose pixels hash the same as the segment's is discarded without a
swap.
- **Redraw on real updates only.** Vegas makes no API calls. Each plugin
fetches on its own schedule, and a redraw is triggered only when the
plugin's `update()` has run since its segment was drawn. On hdpi live
football, baseball and hockey poll every 30 s (live odds every 60 s,
everything else hourly), so a live sports card is redrawn once per poll.
- **Floor.** A plugin is redrawn at most once per
`vegas_scroll.refresh_min_interval` (proposed 10 s), and never while its
previous redraw is still running. The floor never holds back a sports card
polling every 30 s. It exists for chatty plugins: `clock-simple` updates
every second and `ledmatrix-music` polls every 2 s.
- **One worker.** Redraws go through the same background worker as prefetch,
one plugin at a time at `nice 10`, under the plugin's lock.
Data freshness is still bounded by each plugin's own fetch interval (how often
it polls live scores). Drawing faster cannot beat the data source.
### The strip becomes a list of segments
All three need the strip to be replaceable by segment. Today it is one
image (`ScrollHelper.cached_array`, 8,000–20,000 px wide, 1.5–3.8 MB), and
`append_content()` rebuilds the whole thing on the render thread for every
appended block. That is also a pause source.
Proposed `SegmentStrip`, used by Vegas in place of the single image:
- an ordered list of segments: plugin id, card boundaries, a pixel array, the
render time, and the plugin data version it was drawn from, plus its
x-offset in the strip;
- `visible(x, width)` assembles the viewport by slicing across at most a few
segments: the same ~100 KB copy per frame that slicing the single image
costs today;
- append and trim become O(block) list operations, not a copy of the strip;
- replace swaps one list entry and shifts the offsets of the segments after it
(dozens at most). A same-geometry patch copies pixels into the existing array.
Every mutation is prepared off the render thread and applied by the render
thread at a frame boundary, so the strip the render loop reads is never
half-changed.
### Multi-display sync
The follower renders from its own copy of the strip, offset from the leader's
scroll position. Today the leader sends that copy whole, and only in
`start_new_cycle()` (`send_scroll_image`), plus the scroll position every
frame. Continuous scroll, the default, extends and trims the strip without
starting a new cycle, and nothing sends those changes. From reading the code,
the follower therefore probably falls out of step after the first extension
already, before any of this design. That is untested; it needs a two-Pi rig.
With a segment strip, keeping the follower identical becomes **replaying the
leader's operations**:
- Every strip mutation (append, trim, replace, patch) is one operation in
strip coordinates. The leader applies it and sends the same operation to the
follower over the existing TCP channel. Segments are small: a card is ~29 KB
raw and compresses well.
- Operations on off-screen segments apply on arrival. A patch to a segment
that is on either panel carries an *apply at scroll position X* stamp a
couple of hundred milliseconds ahead. Both sides apply it when their scroll
position passes X, so both panels change on the same frame, within the
existing position-sync jitter.
- Each operation carries a sequence number. A follower that sees a gap (a
reconnect, a dropped message) asks for a full snapshot, which is today's
`send_scroll_image` path.
That also fixes the probable continuous-mode gap as a side effect, since
appends and trims become operations too. Until it is in place, fresh-content
updates are disabled while sync is active.
## Risks, and what was checked
1. **Plugins holding their own reference to the shared `draw` or `image`.**
They would keep drawing into the shared canvas, and routing by thread can't
redirect them. A grep of the 49 plugins installed on hdpi found none storing
`display_manager.draw` or `.image` in an attribute (a pattern search, so
indirect aliasing would slip past it). A plugin that did would
draw into an image nobody displays, which trims to a blank segment. That is
not corruption, and it is no worse than today.
2. **Plugins calling the matrix directly.** None in the audit. Inside
`offscreen()` the proxy makes it inert anyway.
3. **Font thread-safety.** `FontManager` shares font objects across plugins.
Measured on Pillow 12.3, two threads rendering text take 1.94× as long as
one, so text rendering holds the GIL and FreeType is never entered
concurrently. Re-check if Pillow changes that.
4. **Plugin thread-safety.** `display()` moves to the prefetch thread. The
plugin lock makes it exclusive with `update()`, which is more protection
than it has today. Threads a plugin starts itself are not covered, as today.
5. **The GIL.** Moving 40–600 ms of plugin rendering off the render thread
removes the pauses, but the work still needs the GIL. Pillow drawing holds
it, and a waiting thread only gets it back after the switch interval
(default 5 ms). Expect some single-refresh late frames while a prefetch
runs. Measure with the soak. A render process separate from plugin work
is the structural answer (the "native presenter" step). Two experiments
get most of the way first (results under Status, above):
- `vegas_scroll.switch_interval_ms` lowers the switch interval for a Vegas
run (1 ms is the obvious try), so the render thread waits at most that
long behind bytecode. It does nothing for a C call that keeps the GIL.
- `vegas_scroll.prefetch_gate` (`src/common/render_gate.py`) lets the
prefetch thread run Python only while the render thread is blocked in
`SwapOnVSync`, up to just before the refresh the swap returns on, and
parks it the rest of the time. That covers C calls too, since the gate is
checked before each one starts. It never parks the thread while it holds
a lock the render thread takes, and never for more than 50 ms. It needs
the rebuilt binding, which releases the GIL during the swap. On by
default.
## What this does not fix
- **The blit.** Copying a 512×64 frame into the matrix (`SetImage`) is ~6 ms at
8 PWM bits on a Pi 4, leaving ~4 ms of slack per refresh. That is the main
source of the single-refresh late frames. Holding frames for two refreshes
(≈50 px/s) doubles the budget. Cutting the blit itself is the native-presenter
step.
- **Live refreshes pushed from `update()`.** Some sports plugins call
`display()` and `update_display()` from inside `update()`, which runs on the
update worker and can push to the panel mid-Vegas. That is a separate
hazard. `offscreen()` gives a tool for it (run the update worker offscreen
while Vegas owns the panel), but it is out of scope here.
## Test plan
- **Unit, `DisplayManager`:** one thread inside `offscreen()` draws while
another reads `image`/`draw`/`matrix`/`width`/`height` and sees the real
canvas. Also: `update_display()` and `set_scrolling_state()` are inert inside;
`render_size()` narrows only the calling thread; nesting and exceptions
restore state.
- **Unit, adapter:** a stub display-capture plugin and a stub scroll-helper
plugin both return content with `offscreen_only=True`, and nothing is queued
for the render thread. The plugin lock is taken, and a timeout keeps the cached
segment.
- **Emulator integration:** a stub canvas-bound plugin whose `display()` sleeps
300 ms. The Vegas render loop never goes a frame without presenting (frame
timing recorder: zero freezes).
- **Unit, `SegmentStrip`:** the viewport assembled across segment boundaries
matches slicing one concatenated image, pixel for pixel. Append, trim,
replace-ahead and same-geometry patch each leave every other column
unchanged. A geometry-changing patch of a visible segment is refused.
- **Freshness:** a stub sports plugin whose score changes every second. The
score on screen is never older than `lead / speed` plus the plugin's fetch
interval. A visible card's digits change without the frame-timing recorder
seeing a late frame. An unchanged redraw is discarded.
- **Hardware:** an hdpi soak, A/B against the #628 build, alternating order.
Targets: no freezes, an empty 6+ bucket, the 3–5 bucket near zero, and the late
rate below 0.66%. Plus, for freshness: log each segment's age when it enters
the viewport, and compare the median and max before and after.
## Rollout
Three changes, each soaked on hdpi before the next:
1. **Offscreen rendering:** `offscreen()`, the adapter on the prefetch thread,
and the plugin lock. Removes the render-thread pauses.
2. **`SegmentStrip`:** Vegas's strip becomes a list of segments. Removes the
whole-strip copy on append. No visible behaviour change.
3. **Fresh content:** refresh at the gate, replace ahead, patch on screen,
with change detection and the rate limit.
`display.vegas_scroll.offscreen_prefetch` (default `true`) restores today's
deferred path when `false`, and `display.vegas_scroll.live_refresh` (default
`true`) turns off step 3. Keep both for one release, then delete the old paths.
## Open questions
1. Keep the kill switch, or ship without one?
2. Plugin lock timeout: skip the plugin and keep its cached segment (proposed),
or wait longer?
3. `refresh_min_interval`: 10 s proposed. It only limits chatty plugins;
live sports are redrawn once per 30 s poll regardless.
4. Multi-display sync: is there a two-Pi rig to test on? Operation replay is
proposed as part of the segment strip (step 2), with fresh content
disabled under sync until it has been verified on real hardware.
+1 -1
View File
@@ -26,7 +26,7 @@ fields:
| Check | Fields | What happens when one is missing | | Check | Fields | What happens when one is missing |
|---|---|---| |---|---|---|
| JSON schema, [`schema/manifest_schema.json`](../schema/manifest_schema.json) | `id`, `name`, `version`, `author`, `entry_point`, `class_name`, `compatible_versions` | Install from URL logs a warning (`PluginStoreManager._validate_manifest_schema()`); nothing is refused | | JSON schema, [`schema/manifest_schema.json`](../schema/manifest_schema.json) | `id`, `name`, `version`, `author`, `entry_point`, `class_name`, `compatible_versions` | Install from URL logs a warning (`PluginStoreManager._validate_manifest_schema()`); nothing is refused |
| Plugin Store install, [`src/plugin_system/store_install.py`](../src/plugin_system/store_install.py) | `id`, `name`, `class_name`, `display_modes` | Install is refused. A registry install first tries to detect a missing `class_name` from the entry-point file | | Plugin Store install, [`src/plugin_system/store_manager.py`](../src/plugin_system/store_manager.py) | `id`, `name`, `class_name`, `display_modes` | Install is refused. A registry install first tries to detect a missing `class_name` from the entry-point file |
| Plugin loader, [`src/plugin_system/plugin_loader.py`](../src/plugin_system/plugin_loader.py) | `class_name` | The plugin fails to load | | Plugin loader, [`src/plugin_system/plugin_loader.py`](../src/plugin_system/plugin_loader.py) | `class_name` | The plugin fails to load |
Defaults and other uses: Defaults and other uses:
+1 -9
View File
@@ -124,14 +124,6 @@ Plugins are configured by adding their plugin ID as a top-level key in the confi
} }
``` ```
How often the core calls a plugin's `update()`: the plugin's
`get_update_interval()` if it returns a number, else `update_interval` in the
plugin's `manifest.json`, else `update_interval` in its `config.json` section
as above, else 60 seconds. A config `update_interval` therefore only sets the
scheduler's cadence for a plugin whose manifest does not; plugins that expose
it in their config schema typically also honour it themselves inside
`update()`. See [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#get_update_interval---optionalfloat).
### Plugin Display Durations ### Plugin Display Durations
Add plugin display modes to the `display_durations` section: Add plugin display modes to the `display_durations` section:
@@ -202,7 +194,7 @@ plugin-repos/
``` ```
The Plugin Store refuses a manifest that lacks any of `id`, `name`, The Plugin Store refuses a manifest that lacks any of `id`, `name`,
`class_name` or `display_modes` (`store_install.py`); the loader itself `class_name` or `display_modes` (`store_manager.py`); the loader itself
needs `class_name`. `version` is not required, but the store compares it needs `class_name`. `version` is not required, but the store compares it
with the registry's `latest_version` to offer updates, so set it. with the registry's `latest_version` to offer updates, so set it.
`entry_point` defaults to `manager.py` if omitted. The config schema is not `entry_point` defaults to `manager.py` if omitted. The config schema is not
+1 -1
View File
@@ -203,7 +203,7 @@ Forms are rendered on the server, not generated in the browser:
from the schema (widgets named by `x-widget` are rendered by the scripts in from the schema (widgets named by `x-widget` are rendered by the scripts in
`web_interface/static/v3/js/widgets/`) `web_interface/static/v3/js/widgets/`)
4. **Save Configuration** posts the form to `/api/v3/plugins/config` 4. **Save Configuration** posts the form to `/api/v3/plugins/config`
(`web_interface/blueprints/api_v3/plugin_config.py`), which validates it against (`web_interface/blueprints/api_v3/plugins.py`), which validates it against
the schema, writes `config.json` (secret fields go to the schema, writes `config.json` (secret fields go to
`config_secrets.json`) and shows a notification `config_secrets.json`) and shows a notification
+3 -3
View File
@@ -32,7 +32,7 @@
│ • masks x-secret fields │ │ • masks x-secret fields │
│ • renders partials/plugin_config.html (render_field macros) │ │ • renders partials/plugin_config.html (render_field macros) │
│ │ │ │
│ api_v3 blueprint (blueprints/api_v3/plugin_config.py) │ │ api_v3 blueprint (blueprints/api_v3/plugins.py) │
│ save_plugin_config() POST /api/v3/plugins/config │ │ save_plugin_config() POST /api/v3/plugins/config │
│ get_plugin_config() GET /api/v3/plugins/config │ │ get_plugin_config() GET /api/v3/plugins/config │
│ get_plugin_schema() GET /api/v3/plugins/schema │ │ get_plugin_schema() GET /api/v3/plugins/schema │
@@ -91,7 +91,7 @@ validatePluginConfigForm() (client-side checks)
POST /api/v3/plugins/config?plugin_id=<id> (form data, all fields of the form) POST /api/v3/plugins/config?plugin_id=<id> (form data, all fields of the form)
│ │
▼ ▼
save_plugin_config() (api_v3/plugin_config.py) save_plugin_config() (api_v3/plugins.py)
├─→ Start from the stored config.json[<id>] ├─→ Start from the stored config.json[<id>]
├─→ Apply form fields: dotted names → nested keys, "[]" checkbox ├─→ Apply form fields: dotted names → nested keys, "[]" checkbox
│ groups → lists, values coerced to the schema's types │ groups → lists, values coerced to the schema's types
@@ -175,7 +175,7 @@ Implement `on_config_change(new_config)` in the plugin (see
|---------|------| |---------|------|
| Tab partial loader | `web_interface/blueprints/pages_v3.py` (`_load_plugin_config_partial`) | | Tab partial loader | `web_interface/blueprints/pages_v3.py` (`_load_plugin_config_partial`) |
| Form template and field macros | `web_interface/templates/v3/partials/plugin_config.html` | | Form template and field macros | `web_interface/templates/v3/partials/plugin_config.html` |
| Save / get / schema / reset handlers | `web_interface/blueprints/api_v3/plugin_config.py` | | Save / get / schema / reset handlers | `web_interface/blueprints/api_v3/plugins.py` |
| Schema loading, defaults, validation | `src/plugin_system/schema_manager.py` | | Schema loading, defaults, validation | `src/plugin_system/schema_manager.py` |
| Secret masking and splitting | `src/web_interface/secret_helpers.py` | | Secret masking and splitting | `src/web_interface/secret_helpers.py` |
| Widgets | `web_interface/static/v3/js/widgets/` | | Widgets | `web_interface/static/v3/js/widgets/` |
+7 -5
View File
@@ -5,9 +5,11 @@
A plugin can name an icon for its tab in the web interface's second nav row A plugin can name an icon for its tab in the web interface's second nav row
(next to **Plugin Manager**) with the `icon` field in `manifest.json`. (next to **Plugin Manager**) with the `icon` field in `manifest.json`.
`GET /api/v3/plugins/installed` passes the manifest's `icon` through (a > **Status:** the tab code honors `icon`, but `GET /api/v3/plugins/installed`
non-string value comes back as `null`), and a plugin without one gets the > (`web_interface/blueprints/api_v3/plugins.py`) does not currently include
default puzzle piece. > the manifest's `icon` in its response, so every tab shows the default
> puzzle piece. Setting `icon` is harmless and will take effect once the API
> passes it through again.
## Font Awesome classes only ## Font Awesome classes only
@@ -53,8 +55,8 @@ With no `icon` (or an empty one) the tab shows `fas fa-puzzle-piece`.
or misspelled class renders as a blank space. or misspelled class renders as a blank space.
2. Include the style prefix (`fas`, `far` or `fab`) as well as the icon 2. Include the style prefix (`fas`, `far` or `fab`) as well as the icon
class. class.
3. The manifest is re-read on each plugin list load; reload the page after 3. See the status note above: the icon is currently not passed through by
editing `icon`. the API.
## Related Documentation ## Related Documentation
+2 -2
View File
@@ -25,7 +25,7 @@ which runs as root.** Anything installed only into another user's
The web interface is not root, so it installs through a narrow sudo helper: The web interface is not root, so it installs through a narrow sudo helper:
1. `PluginStoreManager._install_dependencies()` 1. `PluginStoreManager._install_dependencies()`
(`src/plugin_system/store_install.py`) calls (`src/plugin_system/store_manager.py`) calls
`install_requirements_file()` (`src/common/permission_utils.py`). `install_requirements_file()` (`src/common/permission_utils.py`).
2. That runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh <plugin>/requirements.txt`. 2. That runs `sudo -n bash scripts/fix_perms/safe_pip_install.sh <plugin>/requirements.txt`.
The helper checks the path is the project's own `requirements.txt` or a The helper checks the path is the project's own `requirements.txt` or a
@@ -154,7 +154,7 @@ For more, see the [Plugin Dependency Troubleshooting Guide](PLUGIN_DEPENDENCY_TR
## Files to Reference ## Files to Reference
- Service units: `systemd/ledmatrix.service`, `systemd/ledmatrix-web.service` - Service units: `systemd/ledmatrix.service`, `systemd/ledmatrix-web.service`
- Store installs: `src/plugin_system/store_install.py` (`_install_dependencies`) - Store installs: `src/plugin_system/store_manager.py` (`_install_dependencies`)
- Root install helper: `src/common/permission_utils.py` (`install_requirements_file`), `scripts/fix_perms/safe_pip_install.sh` - Root install helper: `src/common/permission_utils.py` (`install_requirements_file`), `scripts/fix_perms/safe_pip_install.sh`
- Load-time installs: `src/plugin_system/plugin_loader.py` (`install_dependencies`) - Load-time installs: `src/plugin_system/plugin_loader.py` (`install_dependencies`)
- Sudo rules: `scripts/install/lib_sudoers.sh` (written by `first_time_install.sh` - Sudo rules: `scripts/install/lib_sudoers.sh` (written by `first_time_install.sh`
+1 -1
View File
@@ -645,7 +645,7 @@ To have your plugin added to the official plugin store:
3. **Contact maintainers** (own-repository plugins): 3. **Contact maintainers** (own-repository plugins):
- Open a GitHub issue in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) repository - Open a GitHub issue in the [ledmatrix-plugins](https://github.com/ChuckBuilds/ledmatrix-plugins) repository
- Or reach out on Discord: https://discord.gg/RdrC37rEag - Or reach out on Discord: https://discord.gg/uW36dVAtcT
- Include: Repository URL, plugin description, why it's useful - Include: Repository URL, plugin description, why it's useful
4. **Review process**: 4. **Review process**:
-2
View File
@@ -56,8 +56,6 @@ Going deeper:
- [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) — Vegas scroll, on-demand display, - [ADVANCED_FEATURES.md](ADVANCED_FEATURES.md) — Vegas scroll, on-demand display,
cache management, background services, permissions cache management, background services, permissions
- [FONT_MANAGER.md](FONT_MANAGER.md) — font system - [FONT_MANAGER.md](FONT_MANAGER.md) — font system
- [SCROLL_PERFORMANCE.md](SCROLL_PERFORMANCE.md) — how scrolling is paced, and how to make a plugin's marquee smooth
- [OFFSCREEN_RENDERING.md](OFFSCREEN_RENDERING.md) — rendering plugin content off the render thread
- [PERMISSIONS.md](PERMISSIONS.md) — file ownership, sudo rules, repair scripts - [PERMISSIONS.md](PERMISSIONS.md) — file ownership, sudo rules, repair scripts
- [MQTT bridge](../integrations/mqtt_bridge/README.md) — control the display from Home Assistant over MQTT - [MQTT bridge](../integrations/mqtt_bridge/README.md) — control the display from Home Assistant over MQTT
+4 -11
View File
@@ -40,14 +40,13 @@ the entry below says so.
> The API blueprint is the `api_v3` package in > The API blueprint is the `api_v3` package in
> `web_interface/blueprints/api_v3/` (one module per area: `config.py`, > `web_interface/blueprints/api_v3/` (one module per area: `config.py`,
> `display.py`, `system.py`, `backup.py`, `fonts.py`, `misc.py`, `wifi.py`, > `display.py`, `plugins.py`, `system.py`, `backup.py`, `fonts.py`,
> `starlark.py`, and `plugins.py` plus the `plugin_*.py` modules for the > `misc.py`, `wifi.py`, `starlark.py`). `web_interface/app.py` registers it
> plugin routes). `web_interface/app.py` registers it
> at `/api/v3` (`app.register_blueprint(api_v3, url_prefix='/api/v3')`). > at `/api/v3` (`app.register_blueprint(api_v3, url_prefix='/api/v3')`).
> The three SSE endpoints (`/api/v3/stream/*`) are defined directly on the > The three SSE endpoints (`/api/v3/stream/*`) are defined directly on the
> Flask app in `app.py` (`stream_stats`, `stream_display`, `stream_logs`). > Flask app in `app.py` (`stream_stats`, `stream_display`, `stream_logs`).
> `test/fixtures/api_v3_url_map.json` is the canonical list of blueprint > `test/fixtures/api_v3_url_map.json` is the canonical list of blueprint
> routes; a test fails if the code and that fixture differ. > routes (116 URL rules); a test fails if the code and that fixture differ.
--- ---
@@ -870,13 +869,7 @@ Get a plugin's resource limits. `data` is `null` when none are configured.
**POST** `/api/v3/plugins/limits/<plugin_id>` **POST** `/api/v3/plugins/limits/<plugin_id>`
Set a plugin's resource limits. The body replaces all four limits: a key you Set a plugin's resource limits. The body replaces all four limits: a key you
omit is stored as no limit (`warning_threshold` defaults to `0.8`). Each value omit is stored as no limit (`warning_threshold` defaults to `0.8`).
must be a non-negative number or `null`; anything else is a 400.
The limits are stored in the shared cache. A display service that has already
read limits for the plugin keeps using those until it restarts; likewise the
health and metrics reset routes clear the stored record and the web process's
copy, not the display service's in-memory state.
**Request Body**: **Request Body**:
```json ```json
-2
View File
@@ -81,8 +81,6 @@ more. Shared sports code lives in `src/common`:
| `sports_shared.py` | 3.3.0 | `SportsCoreSharedMixin`, `SportsLiveSharedMixin`, `SportsRecentSharedMixin` — the sport-independent `sports.py` methods | | `sports_shared.py` | 3.3.0 | `SportsCoreSharedMixin`, `SportsLiveSharedMixin`, `SportsRecentSharedMixin` — the sport-independent `sports.py` methods |
| `sports_helpers.py` | 3.5.0 | clamp/logo/rotation free functions and `SportsHelpersMixin`, plus the `_favorite_key` seam | | `sports_helpers.py` | 3.5.0 | clamp/logo/rotation free functions and `SportsHelpersMixin`, plus the `_favorite_key` seam |
| `espn_dates.py` | 3.5.0 | ESPN date-range and `limit` workarounds | | `espn_dates.py` | 3.5.0 | ESPN date-range and `limit` workarounds |
| `favorite_team_check.py` | 3.6.0 | `FavoriteTeamCheck` — logs why a favourite team code shows nothing |
| `sports_timezone.py` | 3.6.0 | Which timezone start times are drawn in (`resolve_timezone_name`) |
Each is described in [src/common/README.md](../src/common/README.md). Each is described in [src/common/README.md](../src/common/README.md).
-9
View File
@@ -616,15 +616,6 @@ sudo systemctl cat ledmatrix-web | grep User
``` ```
**Note:** Minimum recommended: 300 seconds (5 minutes) **Note:** Minimum recommended: 300 seconds (5 minutes)
How often the core calls the plugin's `update()` comes from the plugin
itself first: its `get_update_interval()` if it has one, then
`update_interval` in its `manifest.json`. The `update_interval` in
`config.json` is used by the scheduler only when the manifest sets none.
Many plugins also read their own config `update_interval` and skip the
API call inside `update()` until it has elapsed, which is what makes the
setting above effective; check the plugin's settings form or
`config_schema.json` for the option it actually honours.
2. **Check current rate limit usage:** 2. **Check current rate limit usage:**
- OpenWeatherMap free tier: 1,000 calls/day, 60 calls/minute - OpenWeatherMap free tier: 1,000 calls/day, 60 calls/minute
- With 300s interval: 288 calls/day (well within limits) - With 300s interval: 288 calls/day (well within limits)
-83
View File
@@ -1,83 +0,0 @@
# The mypy ratchet: modules that type-check clean, one path per line, sorted.
#
# CI ("Type check (mypy ratchet)") runs `python scripts/check_types.py`, which
# runs mypy on exactly these files (imports followed silently, so errors in an
# unlisted module they import don't count) and fails on any error, so a listed
# module stays clean. Most of src/ isn't clean yet. When you make a module
# clean, add it here. Don't take a module off to get CI green -- fix the error
# (annotation-only where you can: hints, typing.cast, TYPE_CHECKING imports;
# widen an annotation rather than delete a defensive runtime check).
src/__init__.py
src/adaptive_images.py
src/auto_update_setup.py
src/backup_manager.py
src/base_odds_manager.py
src/cache/__init__.py
src/cache/cache_metrics.py
src/cache/cache_strategy.py
src/cache/memory_cache.py
src/common/__init__.py
src/common/api_helper.py
src/common/bdf_font.py
src/common/espn_dates.py
src/common/favorite_team_check.py
src/common/font_layout.py
src/common/frame_timing.py
src/common/json_body.py
src/common/logo_helper.py
src/common/path_safety.py
src/common/permission_utils.py
src/common/render_gate.py
src/common/scroll_config.py
src/common/snapshot_policy.py
src/common/sports_card.py
src/common/sports_scroll.py
src/common/sports_timezone.py
src/config_service.py
src/core_config_keys.py
src/deprecation.py
src/device_location.py
src/display_geometry.py
src/dynamic_team_resolver.py
src/exceptions.py
src/font_usage.py
src/logging_config.py
src/logo_downloader.py
src/matrix_support.py
src/pi5_matrix_support.py
src/plugin_system/__init__.py
src/plugin_system/compatibility.py
src/plugin_system/operation_history.py
src/plugin_system/operation_queue.py
src/plugin_system/operation_types.py
src/plugin_system/plugin_dirs.py
src/plugin_system/plugin_executor.py
src/plugin_system/plugin_health.py
src/plugin_system/plugin_loader.py
src/plugin_system/plugin_state.py
src/plugin_system/repo_urls.py
src/plugin_system/resource_monitor.py
src/plugin_system/saved_repositories.py
src/plugin_system/schema_manager.py
src/plugin_system/state_reconciliation.py
src/plugin_system/testing/__init__.py
src/plugin_system/testing/bounds_display_manager.py
src/plugin_system/testing/loading.py
src/plugin_system/testing/mocks.py
src/plugin_system/testing/plugin_test_base.py
src/plugin_system/testing/sizes.py
src/redaction.py
src/scan_order.py
src/startup_validator.py
src/vegas_mode/__init__.py
src/vegas_mode/config.py
src/vegas_mode/coordinator.py
src/vegas_mode/geometry.py
src/vegas_mode/stream_manager.py
src/web_interface/api_helpers.py
src/web_interface/config_arrays.py
src/web_interface/error_handler.py
src/web_interface/errors.py
src/web_interface/secret_helpers.py
src/web_interface/validators.py
+10 -25
View File
@@ -1,11 +1,6 @@
[mypy] [mypy]
# Mypy configuration for LEDMatrix # Mypy configuration for LEDMatrix
# What a bare `mypy` checks. ini values can't span lines or carry trailing
# comments -- this file used to have both, so mypy refused to read it at all.
files = src
exclude = (^|/)(test|__pycache__)/
# Python version # Python version
python_version = 3.10 python_version = 3.10
@@ -30,11 +25,11 @@ warn_unreachable = True
# Strict optional checking # Strict optional checking
strict_optional = True strict_optional = True
# Disallow untyped definitions (set to True once all code is typed) # Disallow untyped definitions
disallow_untyped_defs = False disallow_untyped_defs = False # Set to True once all code is typed
# Disallow untyped calls (set to True once all code is typed) # Disallow untyped calls
disallow_untyped_calls = False disallow_untyped_calls = False # Set to True once all code is typed
# Check untyped definitions # Check untyped definitions
check_untyped_defs = True check_untyped_defs = True
@@ -101,20 +96,10 @@ ignore_missing_imports = True
[mypy-spotipy.*] [mypy-spotipy.*]
ignore_missing_imports = True ignore_missing_imports = True
# Exclude test files and generated files
exclude = (?x)(
^test/.*|
^.*/__pycache__/.*|
^.*\.pyc$
)
# numpy's own stubs (numpy>=2.3) use Python 3.12 `type` statements, which mypy
# refuses to parse under python_version = 3.10 -- and 3.10 is the floor this
# code has to run on, so it stays. Treat numpy as Any instead: skip it, and
# follow_imports_for_stubs makes the skip apply to its .pyi files too.
[mypy-numpy.*]
follow_imports = skip
follow_imports_for_stubs = True
# orjson is optional (see requirements.txt): the modules that use it fall back
# to the stdlib when `import orjson` fails. Whether mypy sees its stubs would
# otherwise depend on whether it happens to be installed -- installed, the
# `orjson = None` fallback is a type error and the stdlib branch "unreachable";
# not installed, silencing either is an unused ignore. Treat it as Any always.
[mypy-orjson.*]
follow_imports = skip
follow_imports_for_stubs = True
+2 -2
View File
@@ -1,10 +1,10 @@
# Test/dev-only dependencies (not needed on a running display). # Test/dev-only dependencies (not needed on a running display).
# Install alongside requirements.txt: pip install -r requirements.txt -r requirements-test.txt # Install alongside requirements.txt: pip install -r requirements.txt -r requirements-test.txt
pytest>=9.0.3,<10.0.0 pytest>=9.0.3,<10.0.0
pytest-cov>=4.1.0,<8.0.0 pytest-cov>=4.1.0,<5.0.0
pytest-mock>=3.11.0,<4.0.0 pytest-mock>=3.11.0,<4.0.0
freezegun>=1.2,<2 # deterministic time for golden-image tests freezegun>=1.2,<2 # deterministic time for golden-image tests
psutil>=6.0.0,<7.0.0 # optional at runtime; installed for tests so the psutil>=6.0.0,<8.0.0 # optional at runtime; installed for tests so the
# /system/status endpoint's real path is exercised # /system/status endpoint's real path is exercised
mypy>=1.5.0,<2.0.0 # static type checking (also pinned in .pre-commit-config.yaml) mypy>=1.5.0,<2.0.0 # static type checking (also pinned in .pre-commit-config.yaml)
PyYAML>=6.0.2,<7.0.0 # not a core dependency: test_starlark_pixlet_routes loads PyYAML>=6.0.2,<7.0.0 # not a core dependency: test_starlark_pixlet_routes loads
+1 -1
View File
@@ -7,7 +7,7 @@ Pillow>=12.2.0,<13.0.0
numpy>=1.24.0 # For fast array operations in ScrollHelper (compatible with 2.x) numpy>=1.24.0 # For fast array operations in ScrollHelper (compatible with 2.x)
# Timezone handling # Timezone handling
pytz>=2024.2,<2027.0 # Updated for latest timezone data pytz>=2024.2,<2025.0 # Updated for latest timezone data
# HTTP requests # HTTP requests
requests>=2.33.0,<3.0.0 requests>=2.33.0,<3.0.0
-2
View File
@@ -31,11 +31,9 @@ display; **diagnostic** — run by hand on a Pi when something is wrong.
| `diagnose_web_interface.sh` | diagnostic | Checks why the web interface is not reachable | | `diagnose_web_interface.sh` | diagnostic | Checks why the web interface is not reachable |
| `download_pixlet.sh` | keep | Downloads the bundled Pixlet binaries for Starlark apps (also run from the web UI) | | `download_pixlet.sh` | keep | Downloads the bundled Pixlet binaries for Starlark apps (also run from the web UI) |
| `emergency_reconnect.sh` | diagnostic | Reconnects to your WiFi network if captive-portal testing leaves the Pi offline | | `emergency_reconnect.sh` | diagnostic | Reconnects to your WiFi network if captive-portal testing leaves the Pi offline |
| `frame_soak.py` | diagnostic | Soaks a running display and reports how often frames reached the panel late (docs/SCROLL_PERFORMANCE.md) |
| `install_dependencies_apt.py` | keep | Dependency installer that tries apt packages first, then pip (installer Step 7, plugin loader) | | `install_dependencies_apt.py` | keep | Dependency installer that tries apt packages first, then pip (installer Step 7, plugin loader) |
| `install_plugin_dependencies.sh` | diagnostic | Installs plugin requirements by hand when the automatic install fails | | `install_plugin_dependencies.sh` | diagnostic | Installs plugin requirements by hand when the automatic install fails |
| `prove_security.py` | keep | Security property checks run by pre-commit | | `prove_security.py` | keep | Security property checks run by pre-commit |
| `render_bench.py` | diagnostic | Benchmarks the render loop against the panel's real refresh rate on a synthetic strip |
| `render_plugin.py` | dev-only | Runs a plugin's `update()` + `display()` and saves the frame as a PNG | | `render_plugin.py` | dev-only | Runs a plugin's `update()` + `display()` and saves the frame as a PNG |
| `run_plugin_tests.py` | dev-only | Discovers and runs plugin test suites | | `run_plugin_tests.py` | dev-only | Discovers and runs plugin test suites |
| `scroll_speeds.py` | keep | Shows and tries the scroll speeds your panel can display cleanly | | `scroll_speeds.py` | keep | Shows and tries the scroll speeds your panel can display cleanly |
+1 -1
View File
@@ -194,7 +194,7 @@ def process_schema_file(schema_path: Path) -> bool:
print(f" ✓ Modified {len(modified_fields)} fields") print(f" ✓ Modified {len(modified_fields)} fields")
return True return True
else: else:
print(" ✓ No changes needed") print(f" ✓ No changes needed")
return False return False
+3 -2
View File
@@ -87,6 +87,7 @@ def find_duplicate_fields(schema: Dict[str, Any], path: str = "") -> List[str]:
def validate_schema_syntax(schema_path: Path) -> tuple[bool, List[str]]: def validate_schema_syntax(schema_path: Path) -> tuple[bool, List[str]]:
"""Validate JSON Schema syntax.""" """Validate JSON Schema syntax."""
errors = []
try: try:
with open(schema_path, 'r', encoding='utf-8') as f: with open(schema_path, 'r', encoding='utf-8') as f:
schema = json.load(f) schema = json.load(f)
@@ -163,7 +164,7 @@ def analyze_schema(schema_path: Path) -> Dict[str, Any]:
if "update_interval_seconds" in properties: if "update_interval_seconds" in properties:
analysis["update_interval_variant"] = "update_interval_seconds" analysis["update_interval_variant"] = "update_interval_seconds"
analysis["naming_issues"].append( analysis["naming_issues"].append(
"Uses 'update_interval_seconds' instead of 'update_interval'" f"Uses 'update_interval_seconds' instead of 'update_interval'"
) )
else: else:
analysis["missing_common_fields"].append(field_name) analysis["missing_common_fields"].append(field_name)
@@ -238,7 +239,7 @@ def main():
print(f" Missing common fields: {', '.join(result['missing_common_fields'])}") print(f" Missing common fields: {', '.join(result['missing_common_fields'])}")
if result['naming_issues']: if result['naming_issues']:
print(" Naming issues:") print(f" Naming issues:")
for issue in result['naming_issues']: for issue in result['naming_issues']:
print(f" - {issue}") print(f" - {issue}")
+7 -8
View File
@@ -112,13 +112,15 @@ if command -v python3 >/dev/null 2>&1; then
echo "Python: $PYTHON_VERSION" echo "Python: $PYTHON_VERSION"
if [ "$PYTHON_MAJOR" -eq "3" ]; then if [ "$PYTHON_MAJOR" -eq "3" ]; then
if [ "$PYTHON_MINOR" -ge "10" ] && [ "$PYTHON_MINOR" -le "13" ]; then if [ "$PYTHON_MINOR" -ge "10" ] && [ "$PYTHON_MINOR" -le "12" ]; then
print_success "Python version is supported (3.10-3.13)" print_success "Python version is fully supported (3.10-3.12)"
elif [ "$PYTHON_MINOR" -eq "13" ]; then
print_warning "Python 3.13 detected - most packages compatible, but some may have limited testing"
print_warning "Please report any compatibility issues you encounter"
elif [ "$PYTHON_MINOR" -ge "14" ]; then elif [ "$PYTHON_MINOR" -ge "14" ]; then
print_warning "Python 3.${PYTHON_MINOR} is very new - some packages may not be compatible yet" print_warning "Python 3.${PYTHON_MINOR} is very new - some packages may not be compatible yet"
else else
# Pillow 12 and the pinned test tools need 3.10+, so this won't install. print_warning "Python 3.${PYTHON_MINOR} is outdated - upgrade to 3.10+ recommended"
print_error "Python 3.${PYTHON_MINOR} is too old - Python 3.10+ is required"
fi fi
else else
print_error "Python 2.x detected - Python 3.10+ is required" print_error "Python 2.x detected - Python 3.10+ is required"
@@ -153,10 +155,7 @@ ESSENTIAL_PACKAGES=(
for pkg_info in "${ESSENTIAL_PACKAGES[@]}"; do for pkg_info in "${ESSENTIAL_PACKAGES[@]}"; do
IFS=':' read -r pkg desc <<< "$pkg_info" IFS=':' read -r pkg desc <<< "$pkg_info"
# dpkg-query rather than `dpkg -l | grep -q`: under pipefail, grep -q if dpkg -l | grep -q "^ii $pkg "; then
# exiting on its first match kills dpkg with SIGPIPE and fails the pipeline,
# which reported installed packages as missing.
if [ "$(dpkg-query -W -f='${Status}' "$pkg" 2>/dev/null)" = "install ok installed" ]; then
print_success "$desc ($pkg) is installed" print_success "$desc ($pkg) is installed"
else else
print_warning "$desc ($pkg) not installed - will be installed during setup" print_warning "$desc ($pkg) not installed - will be installed during setup"
-99
View File
@@ -1,99 +0,0 @@
#!/usr/bin/env python3
"""Type-check the modules listed in mypy-clean.txt (the mypy ratchet).
Most of src/ still has mypy errors, so CI can't require a clean `mypy src`.
Instead mypy-clean.txt lists the modules that *are* clean, and this script
fails if any of them regresses. When you make another module clean, add it to
the list; nothing ever comes off it.
Imports are followed silently: a listed module is checked against the types of
everything it imports, but errors inside those imported modules are not
reported, so a clean file isn't failed by an unlisted neighbour.
Usage:
python scripts/check_types.py # check the listed modules
python scripts/check_types.py --list # print the list and exit
Extra arguments after ``--`` are passed to mypy.
Exit status: 0 clean, 1 mypy errors, 2 a bad list (missing file, duplicate,
unsorted, or empty).
"""
import argparse
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
LIST_FILE = REPO_ROOT / "mypy-clean.txt"
def read_list(path: Path = LIST_FILE) -> list:
"""The listed paths, in file order, with comments and blank lines dropped."""
entries = []
for raw in path.read_text(encoding="utf-8").splitlines():
line = raw.split("#", 1)[0].strip()
if line:
entries.append(line)
return entries
def list_problems(entries: list, root: Path = REPO_ROOT) -> list:
"""Why the list can't be used as-is; empty when it is fine."""
problems = []
if not entries:
problems.append(f"{LIST_FILE.name} lists no modules")
seen = set()
for entry in entries:
if entry in seen:
problems.append(f"listed twice: {entry}")
seen.add(entry)
if "\\" in entry:
problems.append(f"use forward slashes: {entry}")
elif not (root / entry).is_file():
problems.append(f"listed but not found (renamed or deleted? update the list): {entry}")
if entries != sorted(entries):
problems.append(f"{LIST_FILE.name} is not sorted")
return problems
def main(argv=None) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
parser.add_argument("--list", action="store_true", help="print the listed modules and exit")
parser.add_argument("mypy_args", nargs="*", help="extra mypy arguments (after --)")
args = parser.parse_args(argv)
entries = read_list()
problems = list_problems(entries)
if problems:
for problem in problems:
print(f"check_types: {problem}", file=sys.stderr)
return 2
if args.list:
print("\n".join(entries))
return 0
cmd = [
sys.executable, "-m", "mypy",
"--config-file", str(REPO_ROOT / "mypy.ini"),
"--follow-imports=silent",
*args.mypy_args,
*entries,
]
print(f"check_types: mypy on {len(entries)} modules from {LIST_FILE.name}", flush=True)
# This interpreter's mypy, fixed flags, and paths from the checked-in list.
result = subprocess.run(cmd, cwd=REPO_ROOT) # nosec B603 - list-form argv, no shell # nosemgrep
if result.returncode > 1: # mypy itself failed (bad config, crash)
return result.returncode
if result.returncode != 0:
print(
"check_types: a module on the mypy ratchet has type errors. Fix them "
f"(annotation-only where possible) rather than taking it off {LIST_FILE.name}.",
file=sys.stderr,
)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
+1 -1
View File
@@ -423,7 +423,7 @@ def main():
global _extra_dirs global _extra_dirs
_extra_dirs = args.extra_dir _extra_dirs = args.extra_dir
print("LEDMatrix Dev Preview Server") print(f"LEDMatrix Dev Preview Server")
print(f"Open http://{args.host}:{args.port} in your browser") print(f"Open http://{args.host}:{args.port} in your browser")
print(f"Plugin search dirs: {[str(d) for d in get_search_dirs()]}") print(f"Plugin search dirs: {[str(d) for d in get_search_dirs()]}")
print() print()
+5 -20
View File
@@ -26,26 +26,11 @@ fi
# Get the full paths to commands and validate each one # Get the full paths to commands and validate each one
MISSING_CMDS=() MISSING_CMDS=()
# Full path of a command, also looking in the sbin directories. This script runs SYSTEMCTL_PATH=$(command -v systemctl) || true
# as the web user, whose PATH usually lacks /usr/sbin and /sbin -- where reboot REBOOT_PATH=$(command -v reboot) || true
# and poweroff live -- so `command -v` alone silently dropped their rules. POWEROFF_PATH=$(command -v poweroff) || true
find_command() { BASH_PATH=$(command -v bash) || true
local found JOURNALCTL_PATH=$(command -v journalctl) || true
found=$(command -v "$1" 2>/dev/null) && { printf '%s\n' "$found"; return 0; }
for dir in /usr/sbin /sbin /usr/bin /bin; do
if [ -x "$dir/$1" ]; then
printf '%s\n' "$dir/$1"
return 0
fi
done
return 1
}
SYSTEMCTL_PATH=$(find_command systemctl) || true
REBOOT_PATH=$(find_command reboot) || true
POWEROFF_PATH=$(find_command poweroff) || true
BASH_PATH=$(find_command bash) || true
JOURNALCTL_PATH=$(find_command journalctl) || true
SAFE_RM_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh" SAFE_RM_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_plugin_rm.sh"
SAFE_PIP_INSTALL_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_pip_install.sh" SAFE_PIP_INSTALL_PATH="$PROJECT_ROOT/scripts/fix_perms/safe_pip_install.sh"
+1
View File
@@ -299,5 +299,6 @@ def main():
if __name__ == '__main__': if __name__ == '__main__':
import importlib.util import importlib.util
from typing import Optional
sys.exit(main()) sys.exit(main())
+2 -31
View File
@@ -42,8 +42,6 @@ class WiFiMonitorDaemon:
""" """
self.check_interval = check_interval self.check_interval = check_interval
self.wifi_manager = WiFiManager() self.wifi_manager = WiFiManager()
# mtime of wifi_config.json as last loaded; see _reload_config_if_changed.
self._config_mtime = self._config_file_mtime()
self.running = True self.running = True
self.last_state = None self.last_state = None
# Counts consecutive checks where nmcli says "connected" but internet is unreachable. # Counts consecutive checks where nmcli says "connected" but internet is unreachable.
@@ -59,32 +57,7 @@ class WiFiMonitorDaemon:
"""Handle shutdown signals""" """Handle shutdown signals"""
logger.info(f"Received signal {signum}, shutting down...") logger.info(f"Received signal {signum}, shutting down...")
self.running = False self.running = False
def _config_file_mtime(self):
try:
return self.wifi_manager.config_path.stat().st_mtime_ns
except OSError:
return None
def _reload_config_if_changed(self):
"""Re-read wifi_config.json when it has changed on disk.
The web UI's auto-enable toggle (POST /api/v3/wifi/ap/auto-enable)
only writes the file; this process read it once at startup, so the
toggle did nothing until the daemon restarted. One stat per check.
"""
mtime = self._config_file_mtime()
if mtime is None or mtime == self._config_mtime:
return
before = self.wifi_manager.config.get("auto_enable_ap_mode", True)
self.wifi_manager._load_config()
# _load_config can itself save (it fills in missing keys), so take
# the mtime after it, or that save would trigger another reload.
self._config_mtime = self._config_file_mtime()
after = self.wifi_manager.config.get("auto_enable_ap_mode", True)
if after != before:
logger.info(f"wifi_config.json changed: auto_enable_ap_mode={after}")
def run(self): def run(self):
"""Main daemon loop""" """Main daemon loop"""
logger.info("WiFi Monitor Daemon started") logger.info("WiFi Monitor Daemon started")
@@ -105,8 +78,6 @@ class WiFiMonitorDaemon:
while self.running: while self.running:
try: try:
self._reload_config_if_changed()
# One combined check that also returns the state it observed — # One combined check that also returns the state it observed —
# the previous flow fetched status before AND after the check # the previous flow fetched status before AND after the check
# on top of the check's own internal fetch, each one several # on top of the check's own internal fetch, each one several
@@ -248,7 +219,7 @@ def main():
parser.add_argument( parser.add_argument(
'--foreground', '--foreground',
action='store_true', action='store_true',
help='Accepted for compatibility; the daemon always runs in the foreground' help='Run in foreground (for debugging)'
) )
args = parser.parse_args() args = parser.parse_args()
+1 -1
View File
@@ -4,5 +4,5 @@ LEDMatrix Display System
Core source package for the LED Matrix Display project. Core source package for the LED Matrix Display project.
""" """
__version__ = "3.6.1" __version__ = "3.5.0"
+9 -25
View File
@@ -70,14 +70,7 @@ def _read(path):
def is_enabled(config): def is_enabled(config):
# Only the {"enabled": true} object turns this on. A hand-edited return bool((config.get('auto_update') or {}).get('enabled', False))
# non-dict (e.g. "auto_update": true) raised AttributeError here and
# aborted startup setup; the web UI's save replaces such a value with {}
# (disabled), so read it the same way.
section = config.get('auto_update')
if not isinstance(section, dict):
return False
return bool(section.get('enabled', False))
class UpdateHelperSetup: class UpdateHelperSetup:
@@ -208,26 +201,17 @@ class UpdateHelperSetup:
try: try:
self.result_file.parent.mkdir(parents=True, exist_ok=True) self.result_file.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(self.result_file.parent), prefix='.auto_update_setup_') fd, tmp = tempfile.mkstemp(dir=str(self.result_file.parent), prefix='.auto_update_setup_')
try: with os.fdopen(fd, 'w', encoding='utf-8') as f:
with os.fdopen(fd, 'w', encoding='utf-8') as f: json.dump(result, f, indent=2)
json.dump(result, f, indent=2) os.chmod(tmp, 0o644)
os.chmod(tmp, 0o644) if self._web_ids and hasattr(os, 'chown'):
if self._web_ids and hasattr(os, 'chown'): # Only root can give the file away; the result is readable
# Only root can give the file away; the result is readable # (0644) either way, so a failed chown must not lose it.
# (0644) either way, so a failed chown must not lose it.
try:
os.chown(tmp, *self._web_ids)
except OSError:
pass
os.replace(tmp, self.result_file)
except BaseException:
# Don't leave a .auto_update_setup_* file behind in the
# project dir every time the write fails.
try: try:
os.unlink(tmp) os.chown(tmp, *self._web_ids)
except OSError: except OSError:
pass pass
raise os.replace(tmp, self.result_file)
except OSError as e: except OSError as e:
logger.warning("Could not record automatic update setup result: %s", e) logger.warning("Could not record automatic update setup result: %s", e)
return result return result
+12 -48
View File
@@ -103,32 +103,6 @@ class FetchResult:
# FAILED, which turns "you cancelled this" into "this errored". # FAILED, which turns "you cancelled this" into "this errored".
final_status: Optional[FetchStatus] = None final_status: Optional[FetchStatus] = None
class _ConnectionRetryingSession:
"""``session.get`` that retries a connection error a few times.
For ESPN date chunks, which bypass _make_request_with_retry: a failed
chunk is logged and skipped, so a brief network blip would otherwise drop
a month from a cached season. That protection used to come from the
session adapter's own retries, which every other request stacked with the
retry loop.
"""
ATTEMPTS = 3
DELAY = 0.5
def __init__(self, session):
self._session = session
def get(self, *args, **kwargs):
for attempt in range(self.ATTEMPTS):
try:
return self._session.get(*args, **kwargs)
except requests.ConnectionError:
if attempt == self.ATTEMPTS - 1:
raise
time.sleep(self.DELAY * (attempt + 1))
class BackgroundDataService: class BackgroundDataService:
""" """
Background data service for fetching season data without blocking the main thread. Background data service for fetching season data without blocking the main thread.
@@ -189,16 +163,10 @@ class BackgroundDataService:
'average_fetch_time': 0.0 'average_fetch_time': 0.0
} }
# Session for HTTP requests. No retries at the adapter: a fetch goes # Session for HTTP requests
# through _make_request_with_retry (max_retries + 1 attempts with
# exponential backoff, logged), and date-range chunks through
# _ConnectionRetryingSession. With the adapter also retrying
# connection errors three times, a dead network cost up to 16
# connection attempts per request and held one of the few worker
# threads for all of them.
self.session = requests.Session() self.session = requests.Session()
self.session.mount('http://', requests.adapters.HTTPAdapter(max_retries=0)) self.session.mount('http://', requests.adapters.HTTPAdapter(max_retries=3))
self.session.mount('https://', requests.adapters.HTTPAdapter(max_retries=0)) self.session.mount('https://', requests.adapters.HTTPAdapter(max_retries=3))
# Default headers: core's shared set (real User-Agent, no hand-set # Default headers: core's shared set (real User-Agent, no hand-set
# Accept-Encoding) -- see src/common/api_helper.py. # Accept-Encoding) -- see src/common/api_helper.py.
@@ -279,19 +247,15 @@ class BackgroundDataService:
# same object the dict holds. # same object the dict holds.
self.completed_requests[request_id] = result self.completed_requests[request_id] = result
# The callback runs outside the lock, as on the worker path: it is if callback:
# plugin code, and holding the service lock through it blocked try:
# every worker's result bookkeeping (and any other thread's callback(result)
# submit) for as long as the callback took. except Exception as e:
if callback: logger.error(f"Error in callback for request {request_id}: {e}")
try: self._release_payload(result)
callback(result)
except Exception as e:
logger.error(f"Error in callback for request {request_id}: {e}")
self._release_payload(result)
logger.debug(f"Cache hit for {sport} {year} data") logger.debug(f"Cache hit for {sport} {year} data")
return request_id return request_id
# limit above 500 makes an ESPN *scoreboard* return a truncated list # limit above 500 makes an ESPN *scoreboard* return a truncated list
# (src/common/espn_dates.py). Other endpoints need more: /teams has 762 # (src/common/espn_dates.py). Other endpoints need more: /teams has 762
@@ -596,7 +560,7 @@ class BackgroundDataService:
""" """
logger.info("Recovering %s %s from a rejected date range", request.sport, request.year) logger.info("Recovering %s %s from a rejected date range", request.sport, request.year)
return fetch_espn_date_chunks( return fetch_espn_date_chunks(
_ConnectionRetryingSession(self.session), self.session,
request.url, request.url,
params=request.params, params=request.params,
headers=request.headers, headers=request.headers,
+5 -41
View File
@@ -102,12 +102,6 @@ _SINGLE_FILE_SECTIONS: Tuple[Tuple[str, Path, str], ...] = (
("ytm_auth", _YTM_REL, "restore_wifi"), ("ytm_auth", _YTM_REL, "restore_wifi"),
) )
#: Sections holding credentials. Restored onto a device that has no copy yet,
#: they would otherwise take the extracted temp file's umask mode (0o644,
#: world-readable); 0o640 matches what config_manager_atomic gives secrets.
_PRIVATE_SECTION_RELS = frozenset({_SECRETS_REL, _WIFI_REL, _YTM_REL})
_PRIVATE_FILE_MODE = 0o640
MANIFEST_NAME = "manifest.json" MANIFEST_NAME = "manifest.json"
PLUGINS_MANIFEST_NAME = "plugins.json" PLUGINS_MANIFEST_NAME = "plugins.json"
@@ -241,10 +235,6 @@ def list_installed_plugins(project_root: Path) -> List[Dict[str, Any]]:
data = json.load(f) data = json.load(f)
except (OSError, json.JSONDecodeError): except (OSError, json.JSONDecodeError):
continue continue
# Valid JSON that is not an object (a list, a bare string) would
# raise AttributeError on .get() and abort the whole export.
if not isinstance(data, dict):
continue
plugin_id = data.get("id") or entry.name plugin_id = data.get("id") or entry.name
if plugin_id not in plugins: if plugin_id not in plugins:
plugins[plugin_id] = { plugins[plugin_id] = {
@@ -320,11 +310,7 @@ def create_backup(
contents: List[str] = [] contents: List[str] = []
# Stream directly to a temp file so we never hold the whole ZIP in memory. # Stream directly to a temp file so we never hold the whole ZIP in memory.
# The name is unique per call: a fixed "<zip>.tmp" was shared by two tmp_path = zip_path.with_suffix(".zip.tmp")
# exports started in the same second, which then wrote the same file.
fd, tmp_name = tempfile.mkstemp(dir=str(output_dir), prefix=f".{zip_name}.", suffix=".tmp")
os.close(fd)
tmp_path = Path(tmp_name)
try: try:
with zipfile.ZipFile(tmp_path, "w", compression=zipfile.ZIP_DEFLATED) as zf: with zipfile.ZipFile(tmp_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
for section, rel, _flag in _SINGLE_FILE_SECTIONS: for section, rel, _flag in _SINGLE_FILE_SECTIONS:
@@ -361,24 +347,7 @@ def create_backup(
manifest = _build_manifest(contents) manifest = _build_manifest(contents)
zf.writestr(MANIFEST_NAME, json.dumps(manifest, indent=2)) zf.writestr(MANIFEST_NAME, json.dumps(manifest, indent=2))
# Same-second exports share a timestamp; number the later one rather os.replace(tmp_path, zip_path)
# than replacing the backup the first one just returned. The name is
# claimed with an exclusive create (O_EXCL fails if it exists), so two
# exports finishing together can't both pick the same free name; the
# replace then swaps the finished archive in over our own placeholder.
suffix = 2
while True:
try:
os.close(os.open(zip_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600))
break
except FileExistsError:
zip_path = output_dir / f"{Path(zip_name).stem}-{suffix}.zip"
suffix += 1
try:
os.replace(tmp_path, zip_path)
except BaseException:
zip_path.unlink(missing_ok=True)
raise
except Exception: except Exception:
tmp_path.unlink(missing_ok=True) tmp_path.unlink(missing_ok=True)
raise raise
@@ -481,8 +450,7 @@ def validate_backup(zip_path: Path) -> Tuple[bool, str, Dict[str, Any]]:
): ):
detected.append("plugin_uploads") detected.append("plugin_uploads")
# Whatever the archive's manifest holds; checked below. plugins: List[Dict[str, Any]] = []
plugins: Any = []
if PLUGINS_MANIFEST_NAME in names: if PLUGINS_MANIFEST_NAME in names:
try: try:
plugins = json.loads(zf.read(PLUGINS_MANIFEST_NAME).decode("utf-8")) plugins = json.loads(zf.read(PLUGINS_MANIFEST_NAME).decode("utf-8"))
@@ -525,7 +493,7 @@ def _extract_zip_safe(zip_path: Path, dest_dir: Path) -> None:
shutil.copyfileobj(src, dst, length=64 * 1024) shutil.copyfileobj(src, dst, length=64 * 1024)
def _copy_file(src: Path, dst: Path, new_mode: Optional[int] = None) -> None: def _copy_file(src: Path, dst: Path) -> None:
"""Replace ``dst`` with ``src``, atomically, without needing to own ``dst``. """Replace ``dst`` with ``src``, atomically, without needing to own ``dst``.
``shutil.copy2`` opens the destination for writing, so it needs write ``shutil.copy2`` opens the destination for writing, so it needs write
@@ -541,7 +509,6 @@ def _copy_file(src: Path, dst: Path, new_mode: Optional[int] = None) -> None:
The destination's existing mode is preserved when there is one, so The destination's existing mode is preserved when there is one, so
restoring secrets does not silently widen them to the umask default. restoring secrets does not silently widen them to the umask default.
When there is none, ``new_mode`` (if given) is used instead of ``src``'s.
""" """
dst.parent.mkdir(parents=True, exist_ok=True) dst.parent.mkdir(parents=True, exist_ok=True)
@@ -563,8 +530,6 @@ def _copy_file(src: Path, dst: Path, new_mode: Optional[int] = None) -> None:
shutil.copyfile(src, tmp_path) shutil.copyfile(src, tmp_path)
if existing_mode is not None: if existing_mode is not None:
os.chmod(tmp_path, existing_mode) os.chmod(tmp_path, existing_mode)
elif new_mode is not None:
os.chmod(tmp_path, new_mode)
else: else:
shutil.copymode(src, tmp_path) shutil.copymode(src, tmp_path)
if existing_owner is not None and hasattr(os, 'chown'): if existing_owner is not None and hasattr(os, 'chown'):
@@ -628,8 +593,7 @@ def restore_backup(
result.skipped.append(section) result.skipped.append(section)
continue continue
try: try:
_copy_file(tmp_dir / rel, project_root / rel, _copy_file(tmp_dir / rel, project_root / rel)
new_mode=_PRIVATE_FILE_MODE if rel in _PRIVATE_SECTION_RELS else None)
result.restored.append(section) result.restored.append(section)
except OSError as e: except OSError as e:
logger.error("[Backup] Failed to restore %s: %s", rel.name, e, exc_info=True) logger.error("[Backup] Failed to restore %s: %s", rel.name, e, exc_info=True)
+12 -33
View File
@@ -16,16 +16,11 @@ import time
import requests import requests
import json import json
from typing import Dict, Any, Optional, List, cast from typing import Dict, Any, Optional, List
from src.common.api_helper import DEFAULT_HTTP_HEADERS from src.common.api_helper import DEFAULT_HTTP_HEADERS
def _is_no_odds_marker(data: Any) -> bool:
"""Whether a cached odds entry is the "ESPN had none" marker, not odds."""
return isinstance(data, dict) and bool(data.get("no_odds"))
class BaseOddsManager: class BaseOddsManager:
""" """
Base class for odds data fetching and management. Base class for odds data fetching and management.
@@ -105,7 +100,7 @@ class BaseOddsManager:
_FAILURE_COOLDOWN = 60.0 _FAILURE_COOLDOWN = 60.0
def get_odds(self, sport: str | None, league: str | None, event_id: str, def get_odds(self, sport: str | None, league: str | None, event_id: str,
update_interval_seconds: Optional[int] = None) -> Optional[Dict[str, Any]]: update_interval_seconds: int = None) -> Optional[Dict[str, Any]]:
""" """
Fetch odds data for a specific game. Fetch odds data for a specific game.
@@ -126,20 +121,10 @@ class BaseOddsManager:
cache_key = f"odds_espn_{sport}_{league}_{event_id}" cache_key = f"odds_espn_{sport}_{league}_{event_id}"
# Check cache first # Check cache first
cached_data: Optional[Dict[str, Any]] = self.cache_manager.get_with_auto_strategy(cache_key) cached_data = self.cache_manager.get_with_auto_strategy(cache_key)
# Per-game chatter, logged on every update of every game on the
# slate: debug, not the journal.
if cached_data: if cached_data:
# A game ESPN had no odds for is cached as {"no_odds": True} so it self.logger.info(f"Using cached odds from ESPN for {cache_key}")
# isn't re-requested every update. That marker is a cache hit --
# its ttl decides when to ask again -- but it is not odds: returned
# as-is, a caller saw a truthy dict and treated the game as having
# odds. The plugins' bundled copies already did this.
if _is_no_odds_marker(cached_data):
self.logger.debug("Cached no-odds marker for %s", cache_key)
return None
self.logger.debug(f"Using cached odds from ESPN for {cache_key}")
return cached_data return cached_data
if time.monotonic() < self._skip_network_until: if time.monotonic() < self._skip_network_until:
@@ -152,7 +137,7 @@ class BaseOddsManager:
self._skip_network_until - time.monotonic()) self._skip_network_until - time.monotonic())
return None return None
self.logger.debug(f"Cache miss - fetching fresh odds from ESPN for {cache_key}") self.logger.info(f"Cache miss - fetching fresh odds from ESPN for {cache_key}")
try: try:
# Map league names to ESPN API format # Map league names to ESPN API format
@@ -166,7 +151,7 @@ class BaseOddsManager:
espn_league = league_mapping.get(league, league) espn_league = league_mapping.get(league, league)
url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds" url = f"{self.base_url}/{sport}/leagues/{espn_league}/events/{event_id}/competitions/{event_id}/odds"
self.logger.debug(f"Requesting odds from URL: {url}") self.logger.info(f"Requesting odds from URL: {url}")
response = self.session.get(url, timeout=self.request_timeout) response = self.session.get(url, timeout=self.request_timeout)
response.raise_for_status() response.raise_for_status()
@@ -178,9 +163,9 @@ class BaseOddsManager:
odds_data = self._extract_espn_data(raw_data) odds_data = self._extract_espn_data(raw_data)
if odds_data: if odds_data:
self.logger.debug(f"Successfully extracted odds data: {odds_data}") self.logger.info(f"Successfully extracted odds data: {odds_data}")
self.cache_manager.set(cache_key, odds_data, ttl=interval) self.cache_manager.set(cache_key, odds_data, ttl=interval)
self.logger.debug(f"Saved odds data to cache for {cache_key} with TTL {interval}s") self.logger.info(f"Saved odds data to cache for {cache_key} with TTL {interval}s")
else: else:
self.logger.debug(f"No odds data available for {cache_key}") self.logger.debug(f"No odds data available for {cache_key}")
# Cache the absence too, so the game is not re-requested # Cache the absence too, so the game is not re-requested
@@ -189,22 +174,16 @@ class BaseOddsManager:
return odds_data return odds_data
# Before RequestException: requests' JSONDecodeError subclasses it, so
# listed second this branch never ran and a bad body was reported as a
# failed fetch. It holds off like a failed fetch did, so only the
# message changes.
except (json.JSONDecodeError, requests.exceptions.JSONDecodeError):
self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN
self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.")
except requests.exceptions.RequestException as e: except requests.exceptions.RequestException as e:
self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN self._skip_network_until = time.monotonic() + self._FAILURE_COOLDOWN
self.logger.error( self.logger.error(
"Error fetching odds from ESPN API for %s: %s. Holding off on odds " "Error fetching odds from ESPN API for %s: %s. Holding off on odds "
"for %.0fs so a slate of games does not pay this timeout each.", "for %.0fs so a slate of games does not pay this timeout each.",
cache_key, e, self._FAILURE_COOLDOWN) cache_key, e, self._FAILURE_COOLDOWN)
except json.JSONDecodeError:
cached = self.cache_manager.get_with_auto_strategy(cache_key) self.logger.error(f"Error decoding JSON response from ESPN API for {cache_key}.")
return None if _is_no_odds_marker(cached) else cast(Optional[Dict[str, Any]], cached)
return self.cache_manager.get_with_auto_strategy(cache_key)
def _extract_espn_data(self, data: Dict[str, Any]) -> Optional[Dict[str, Any]]: def _extract_espn_data(self, data: Dict[str, Any]) -> Optional[Dict[str, Any]]:
""" """
+3 -6
View File
@@ -8,7 +8,7 @@ import os
import time import time
import threading import threading
import logging import logging
from typing import Dict, Any, Optional, Union from typing import Dict, Any, Optional
# Historical fixed ceiling, kept as the fallback when RAM cannot be read. # Historical fixed ceiling, kept as the fallback when RAM cannot be read.
DEFAULT_MAX_SIZE = 1000 DEFAULT_MAX_SIZE = 1000
@@ -70,15 +70,13 @@ class MemoryCache:
""" """
self.logger = logging.getLogger(__name__) self.logger = logging.getLogger(__name__)
self._cache: Dict[str, Dict[str, Any]] = {} self._cache: Dict[str, Dict[str, Any]] = {}
# Values are time.time() floats; get()/cleanup also accept a numeric self._timestamps: Dict[str, float] = {}
# string, as a timestamp may have been restored from serialized data.
self._timestamps: Dict[str, Union[float, str]] = {}
self._lock = threading.Lock() self._lock = threading.Lock()
self._max_size = max_size self._max_size = max_size
self._cleanup_interval = cleanup_interval self._cleanup_interval = cleanup_interval
self._last_cleanup = time.time() self._last_cleanup = time.time()
def get(self, key: str, max_age: Optional[float] = None) -> Optional[Dict[str, Any]]: def get(self, key: str, max_age: Optional[int] = None) -> Optional[Dict[str, Any]]:
""" """
Get value from memory cache. Get value from memory cache.
@@ -202,7 +200,6 @@ class MemoryCache:
max_age_for_cleanup = 3600 # 1 hour max_age_for_cleanup = 3600 # 1 hour
expired_keys = [] expired_keys = []
timestamp: Optional[Union[float, str]]
for key, timestamp in list(self._timestamps.items()): for key, timestamp in list(self._timestamps.items()):
if isinstance(timestamp, str): if isinstance(timestamp, str):
try: try:
+1 -55
View File
@@ -24,16 +24,12 @@ Rules for the package:
| Module | For | Plugins import it? | Since | | Module | For | Plugins import it? | Since |
|---|---|---|---| |---|---|---|---|
| [`api_helper`](#api_helper) | HTTP GET/POST with caching and rate limiting | Yes | — | | [`api_helper`](#api_helper) | HTTP GET/POST with caching and rate limiting | Yes | — |
| [`bdf_font`](#bdf_font) | Load and draw BDF bitmap fonts | Yes, if drawing BDF text directly | 3.5.0 | | [`bdf_font`](#bdf_font) | Load and draw BDF bitmap fonts | Yes, if drawing BDF text directly | Unreleased |
| [`espn_dates`](#espn_dates) | Fetch ESPN scoreboards across a date range | Yes (scoreboards) | 3.5.0 | | [`espn_dates`](#espn_dates) | Fetch ESPN scoreboards across a date range | Yes (scoreboards) | 3.5.0 |
| [`favorite_team_check`](#favorite_team_check) | Log why a favourite team code shows nothing | Yes (scoreboards) | 3.6.0 |
| [`font_layout`](#font_layout) | Reproducible TrueType loading, crisp sizes | Yes | 3.4.0 | | [`font_layout`](#font_layout) | Reproducible TrueType loading, crisp sizes | Yes | 3.4.0 |
| [`frame_timing`](#frame_timing) | Timing of every presented frame, stall watchdog | No, core-internal | n/a |
| [`json_body`](#json_body) | Parse a response body as JSON, with orjson if installed | Optional (large payloads) | 3.5.0 |
| [`logo_helper`](#logo_helper) | Load, resize and cache team logos | Yes | — | | [`logo_helper`](#logo_helper) | Load, resize and cache team logos | Yes | — |
| [`path_safety`](#path_safety) | Turn request-supplied names into safe paths | No, core-internal | n/a | | [`path_safety`](#path_safety) | Turn request-supplied names into safe paths | No, core-internal | n/a |
| [`permission_utils`](#permission_utils) | File modes and shared-group ownership | Rarely | — | | [`permission_utils`](#permission_utils) | File modes and shared-group ownership | Rarely | — |
| [`render_gate`](#render_gate) | Keep background Python off the GIL while the panel swaps | No, core-internal | n/a |
| [`scroll_config`](#scroll_config) | Plugin scroll config → configured `ScrollHelper` | Yes (scrollers) | 3.4.0 | | [`scroll_config`](#scroll_config) | Plugin scroll config → configured `ScrollHelper` | Yes (scrollers) | 3.4.0 |
| [`scroll_helper`](#scroll_helper) | Pre-rendered horizontal scrolling | Yes | — | | [`scroll_helper`](#scroll_helper) | Pre-rendered horizontal scrolling | Yes | — |
| [`snapshot_policy`](#snapshot_policy) | When to write the web preview frame | No, core-internal | n/a | | [`snapshot_policy`](#snapshot_policy) | When to write the web preview frame | No, core-internal | n/a |
@@ -42,7 +38,6 @@ Rules for the package:
| [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 | | [`sports_helpers`](#sports_helpers) | Small helpers every scoreboard `sports.py` copies | Yes (scoreboards) | 3.5.0 |
| [`sports_scroll`](#sports_scroll) | Scoreboard scroll-display orchestration | Yes (scoreboards) | 3.2.0 | | [`sports_scroll`](#sports_scroll) | Scoreboard scroll-display orchestration | Yes (scoreboards) | 3.2.0 |
| [`sports_shared`](#sports_shared) | Sport-independent `sports.py` methods | Yes (scoreboards) | 3.3.0 | | [`sports_shared`](#sports_shared) | Sport-independent `sports.py` methods | Yes (scoreboards) | 3.3.0 |
| [`sports_timezone`](#sports_timezone) | Which timezone a scoreboard draws start times in | Yes (scoreboards) | 3.6.0 |
| [`sync_manager`](#sync_manager) | Leader/follower sync between two displays | No, core-internal | n/a | | [`sync_manager`](#sync_manager) | Leader/follower sync between two displays | No, core-internal | n/a |
| [`text_helper`](#text_helper) | Outlined text, wrapping, measurement | Yes | — | | [`text_helper`](#text_helper) | Outlined text, wrapping, measurement | Yes | — |
@@ -102,17 +97,6 @@ results; `espn_date_chunks()`, `fetch_espn_date_chunks()`,
`clamp_espn_limit()` and `merge_scoreboard_payloads()` are the pieces. `clamp_espn_limit()` and `merge_scoreboard_payloads()` are the pieces.
Scoreboard plugins also bundle a copy for older cores. Scoreboard plugins also bundle a copy for older cores.
### favorite_team_check
[`favorite_team_check.py`](favorite_team_check.py).
`FavoriteTeamCheck(logger, leagues)`, where `leagues` maps a league key to
`(display name, ESPN sport/league path)`. `schedule(league_key, favorites)`
checks the configured favourite team codes against ESPN's team list once per
league, on a daemon thread, and logs a bad code with the nearest real one, or
says the league has nothing on yet; `reset()` re-arms it after a config edit.
Diagnostics only: every failure is swallowed. Scoreboard plugins also bundle
a copy for older cores.
### font_layout ### font_layout
[`font_layout.py`](font_layout.py). `load_truetype(path, size)` is [`font_layout.py`](font_layout.py). `load_truetype(path, size)` is
@@ -123,24 +107,6 @@ the size a bundled face renders on whole pixels at. `resolve_asset_path()`
resolves `assets/fonts/...` against the install root rather than the resolves `assets/fonts/...` against the install root rather than the
working directory. working directory.
### frame_timing
[`frame_timing.py`](frame_timing.py). Core-internal. `DisplayManager`
records every presented frame in a `FrameTimingRecorder`, which writes
cumulative late-frame counters and histograms to `/dev/shm` for
`scripts/frame_soak.py` and `scripts/render_bench.py`. `StallWatchdog` logs
the stack of whatever holds up a scroll. See
[docs/SCROLL_PERFORMANCE.md](../../docs/SCROLL_PERFORMANCE.md).
### json_body
[`json_body.py`](json_body.py). `response_json(response)` is
`response.json()` parsed by orjson when it is installed, falling back to the
stdlib parser (and requests' own error) otherwise. For multi-MB payloads such
as a season schedule, where the parse holds the GIL and freezes the display.
A plugin that also runs on older cores should guard the import, as
`espn_dates` does.
### logo_helper ### logo_helper
[`logo_helper.py`](logo_helper.py). `LogoHelper(display_width, [`logo_helper.py`](logo_helper.py). `LogoHelper(display_width,
@@ -168,14 +134,6 @@ let the root display service and the web user share files:
already call these; a plugin needs them only when it creates its own files already call these; a plugin needs them only when it creates its own files
outside the cache. See [docs/PERMISSIONS.md](../../docs/PERMISSIONS.md). outside the cache. See [docs/PERMISSIONS.md](../../docs/PERMISSIONS.md).
### render_gate
[`render_gate.py`](render_gate.py). Core-internal. `RenderGate` is opened by
the render thread around each vsync swap; a background thread inside
`gate.yielding()` (Vegas's prefetch) parks while the gate is closed, so the
render thread finds the GIL free when its refresh arrives. It never parks a
thread holding a guarded lock or inside logging, threading or import code.
### scroll_config ### scroll_config
[`scroll_config.py`](scroll_config.py). `configure(scroll_helper, [`scroll_config.py`](scroll_config.py). `configure(scroll_helper,
@@ -250,18 +208,6 @@ fonts, colours, dates, the switch-mode upcoming card). The docstring lists
the attributes the host class must have and the three methods deliberately the attributes the host class must have and the three methods deliberately
left out. left out.
### sports_timezone
[`sports_timezone.py`](sports_timezone.py).
`resolve_timezone_name(config, plugin_manager, cache_manager, log, *,
plugin_label, writeback_fixed_in=None)` and `resolve_timezone(...)` (the same
as a pytz zone): the plugin's own `timezone`, then the global one via either
manager's `config_manager`, then the host's zone (`system_timezone_name()`),
then UTC. `plugin_label` names the plugin in the warning logged when nothing
resolves; `writeback_fixed_in` is for a plugin that once wrote `"UTC"` into
the saved config (a bare plugin-level `"UTC"` is then ignored when another
source disagrees). Scoreboard plugins also bundle a copy for older cores.
### sync_manager ### sync_manager
[`sync_manager.py`](sync_manager.py). Core-internal. `DisplaySyncManager` [`sync_manager.py`](sync_manager.py). Core-internal. `DisplaySyncManager`
+15 -27
View File
@@ -11,16 +11,12 @@ import time
from datetime import datetime from datetime import datetime
from types import MappingProxyType from types import MappingProxyType
from src.common.espn_dates import ESPN_MAX_LIMIT from src.common.espn_dates import ESPN_MAX_LIMIT
from typing import TYPE_CHECKING, Any, Dict, Mapping, Optional, cast from typing import Any, Dict, Mapping, Optional
import requests import requests
from requests.adapters import HTTPAdapter from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry from urllib3.util.retry import Retry
if TYPE_CHECKING:
# What Session() puts in .headers; the stubs only promise a MutableMapping.
from requests.structures import CaseInsensitiveDict
#: The User-Agent core sends to ESPN and other data APIs. It names the client #: The User-Agent core sends to ESPN and other data APIs. It names the client
#: and links to it: around 2026-08-04 ESPN began 403ing bare custom tokens #: and links to it: around 2026-08-04 ESPN began 403ing bare custom tokens
@@ -88,11 +84,7 @@ class APIHelper:
self.session.headers.update({**DEFAULT_HTTP_HEADERS, 'Connection': 'keep-alive'}) self.session.headers.update({**DEFAULT_HTTP_HEADERS, 'Connection': 'keep-alive'})
# Rate limiting # Rate limiting
self._last_request_time: float = 0 # wall clock, reported by get_request_stats() self._last_request_time = 0
# The interval is measured on time.monotonic(): a wall-clock step
# back (NTP correcting a Pi with no RTC) made time_since_last
# negative and the "remaining interval" sleep as long as the step.
self._last_request_monotonic: Optional[float] = None
self._min_request_interval = 1.0 # Minimum seconds between requests self._min_request_interval = 1.0 # Minimum seconds between requests
def get(self, url: str, params: Optional[Dict] = None, def get(self, url: str, params: Optional[Dict] = None,
@@ -116,14 +108,14 @@ class APIHelper:
cached = self._get_from_cache(cache_key, cache_ttl) cached = self._get_from_cache(cache_key, cache_ttl)
if cached is not None: if cached is not None:
self.logger.debug(f"Using cached response for {cache_key}") self.logger.debug(f"Using cached response for {cache_key}")
return cast(Dict[Any, Any], cached) return cached
# Rate limiting # Rate limiting
self._enforce_rate_limit() self._enforce_rate_limit()
try: try:
# Prepare request # Prepare request
request_headers = cast('CaseInsensitiveDict[Any]', self.session.headers).copy() request_headers = self.session.headers.copy()
if headers: if headers:
request_headers.update(headers) request_headers.update(headers)
@@ -137,7 +129,7 @@ class APIHelper:
response.raise_for_status() response.raise_for_status()
# Parse JSON response # Parse JSON response
data: Dict[Any, Any] = response.json() data = response.json()
# Cache response if cache key provided # Cache response if cache key provided
if cache_key and self.cache_manager: if cache_key and self.cache_manager:
@@ -251,7 +243,7 @@ class APIHelper:
self._enforce_rate_limit() self._enforce_rate_limit()
try: try:
request_headers = cast('CaseInsensitiveDict[Any]', self.session.headers).copy() request_headers = self.session.headers.copy()
if headers: if headers:
request_headers.update(headers) request_headers.update(headers)
@@ -264,7 +256,7 @@ class APIHelper:
) )
response.raise_for_status() response.raise_for_status()
return cast(Optional[Dict[Any, Any]], response.json()) return response.json()
except requests.exceptions.RequestException as e: except requests.exceptions.RequestException as e:
self.logger.error(f"POST request failed for {url}: {e}") self.logger.error(f"POST request failed for {url}: {e}")
@@ -341,14 +333,13 @@ class APIHelper:
def _enforce_rate_limit(self) -> None: def _enforce_rate_limit(self) -> None:
"""Enforce rate limiting between requests.""" """Enforce rate limiting between requests."""
if self._last_request_monotonic is not None: current_time = time.time()
time_since_last = time.monotonic() - self._last_request_monotonic time_since_last = current_time - self._last_request_time
if time_since_last < self._min_request_interval: if time_since_last < self._min_request_interval:
sleep_time = self._min_request_interval - time_since_last sleep_time = self._min_request_interval - time_since_last
time.sleep(sleep_time) time.sleep(sleep_time)
self._last_request_monotonic = time.monotonic()
self._last_request_time = time.time() self._last_request_time = time.time()
def set_rate_limit(self, min_interval: float) -> None: def set_rate_limit(self, min_interval: float) -> None:
@@ -371,8 +362,5 @@ class APIHelper:
return { return {
'min_request_interval': self._min_request_interval, 'min_request_interval': self._min_request_interval,
'last_request_time': self._last_request_time, 'last_request_time': self._last_request_time,
'time_since_last_request': ( 'time_since_last_request': time.time() - self._last_request_time
time.monotonic() - self._last_request_monotonic
if self._last_request_monotonic is not None
else time.time() - self._last_request_time),
} }
+4 -4
View File
@@ -37,7 +37,7 @@ import time
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import date, timedelta from datetime import date, timedelta
from functools import partial from functools import partial
from typing import Any, Dict, List, Optional, Tuple, cast from typing import Any, Dict, List, Optional, Tuple
try: try:
from src.common.json_body import response_json from src.common.json_body import response_json
@@ -159,7 +159,7 @@ def espn_date_chunks(start: date, end: date) -> List[str]:
return chunks return chunks
def merge_scoreboard_payloads(payloads: List[Any]) -> Dict[str, Any]: def merge_scoreboard_payloads(payloads: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Fold chunk responses into one scoreboard payload. """Fold chunk responses into one scoreboard payload.
Events are de-duplicated by id and keep first-seen order. Non-event keys Events are de-duplicated by id and keep first-seen order. Non-event keys
@@ -202,7 +202,7 @@ def _fetch_one_chunk(
timeout=timeout, timeout=timeout,
) )
response.raise_for_status() response.raise_for_status()
return cast(Optional[Dict[str, Any]], response_json(response)) return response_json(response)
except Exception as exc: # noqa: BLE001 - see docstring except Exception as exc: # noqa: BLE001 - see docstring
if logger: if logger:
logger.warning("ESPN chunk %s failed, skipping it: %s", chunk, exc) logger.warning("ESPN chunk %s failed, skipping it: %s", chunk, exc)
@@ -379,4 +379,4 @@ def fetch_espn_scoreboard(
if data is not None: if data is not None:
return data return data
response.raise_for_status() response.raise_for_status()
return cast(Dict[str, Any], response_json(response)) return response_json(response)
-369
View File
@@ -1,369 +0,0 @@
"""
Explain an empty screen: a wrong team code, or a season that has not started.
Favourite teams are matched by exact ESPN abbreviation, so a plausible-looking
code silently matches nothing and the plugin shows an empty screen with no hint
that the code is at fault. The codes are not always guessable — ESPN calls
Alabama ``ALA`` rather than ``BAMA``, and Golden State ``GS`` rather than
``GSW``. Between seasons a perfectly correct code produces the same empty
screen for a completely different reason, and the two were indistinguishable
from the logs.
This module is diagnostics only. It runs on a daemon thread, once per league per
process, and every failure is swallowed: it must never delay a frame or change
what is displayed.
"""
import difflib
import logging
import re
import threading
from datetime import datetime, timezone
from typing import Dict, Iterable, List, Optional, Set, Tuple
TEAMS_URL = "https://site.api.espn.com/apis/site/v2/sports/{path}/teams?limit=1000"
SCOREBOARD_URL = "https://site.api.espn.com/apis/site/v2/sports/{path}/scoreboard"
REQUEST_TIMEOUT = 15
class FavoriteTeamCheck:
"""
Validates configured favourite team codes against ESPN, and says so in the log.
``leagues`` maps the plugin's own league key to a
``(human readable name, ESPN sport/league path)`` pair, e.g.
``{'nhl': ('NHL', 'hockey/nhl')}``.
"""
# How far out the next fixture has to be before it is worth mentioning.
# An off day or two is normal mid-season and saying so would just be noise.
GAP_DAYS = 3
def __init__(self, logger: Optional[logging.Logger],
leagues: Dict[str, Tuple[str, str]]) -> None:
self.logger = logger or logging.getLogger(__name__)
self.leagues = leagues
self._checked: Set[str] = set()
self._lock = threading.Lock()
def reset(self) -> None:
"""Re-check on the next call, e.g. after the user edits the config."""
with self._lock:
self._checked.clear()
def schedule(self, league_key: str, favorites: Iterable[str]) -> None:
"""Check one league in the background, at most once per process."""
try:
favorites = [str(f) for f in (favorites or []) if str(f).strip()]
if not favorites or league_key not in self.leagues:
return
with self._lock:
if league_key in self._checked:
return
self._checked.add(league_key)
threading.Thread(
target=self._run, args=(league_key, favorites),
name="favorite-team-check", daemon=True,
).start()
except Exception:
pass # nosec B110 - a diagnostic must never be the reason an update fails # nosemgrep
def _run(self, league_key: str, favorites) -> None:
try:
self._check(league_key, favorites)
except Exception as exc:
self.logger.debug("Favorite team check failed for %s: %s",
league_key, exc)
def _check(self, league_key: str, favorites) -> None:
name, path = self.leagues[league_key]
try:
teams = self._fetch_teams(path)
except Exception as exc:
self.logger.debug("Could not verify %s favorite teams: %s", name, exc)
return
if not teams:
# Some ESPN endpoints (college lacrosse) return no teams at all.
# Nothing can be concluded, so say nothing.
return
# Dynamic groups like AP_TOP_25 are expanded elsewhere; they are not
# team codes and must not be reported as bad ones.
codes = [f for f in favorites if not self._is_dynamic(f)]
recognised = [f for f in codes if f in teams]
unknown = [f for f in codes if f not in teams]
for code in unknown:
self.logger.warning(
"%s favorite team %r is not a %s team code.%s "
"Every code this league accepts is listed at %s.",
name, code, name, self._suggest(code, teams),
TEAMS_URL.format(path=path),
)
if codes and not recognised:
self.logger.warning(
"%s has no recognised favorite teams, so nothing will be shown "
"for it. Codes must be ESPN abbreviations, e.g. %s.",
name, ", ".join("{} ({})".format(a, n)
for a, n in list(sorted(teams.items()))[:3]),
)
return
if not recognised:
return
# Codes are fine, so check the other cause of an empty screen.
try:
note = self._schedule_note(path)
except Exception as exc:
self.logger.debug("Could not check the %s schedule: %s", name, exc)
return
if note:
self.logger.info(
"%s favorite teams %s look correct, but %s. An empty display "
"until then is expected, not a configuration problem.",
name, ", ".join(recognised), note,
)
else:
self.logger.info("%s favorite teams recognised: %s",
name, ", ".join(recognised))
@staticmethod
def _is_dynamic(code: str) -> bool:
upper = (code or "").strip().upper()
return upper.startswith("AP_") or upper.startswith("TOP_") or "TOP_" in upper
@staticmethod
def _fetch_teams(path: str) -> Dict[str, str]:
"""ESPN's {abbreviation: display name} for a league.
``limit=1000`` is required: the default page size truncates the NCAA
responses to roughly half their teams, which makes valid codes look wrong.
"""
import requests
payload = requests.get(TEAMS_URL.format(path=path),
timeout=REQUEST_TIMEOUT).json()
entries = payload['sports'][0]['leagues'][0]['teams']
return {
t['team']['abbreviation']: t['team']['displayName']
for t in entries if t.get('team', {}).get('abbreviation')
}
@classmethod
def _schedule_note(cls, path: str) -> Optional[str]:
"""
Why the league has nothing to show, as a clause, or ``None`` if it does.
Two things make this harder than reading ``events``:
* An out-of-season league does not come back empty. ESPN rolls the
scoreboard forward to the next day that has fixtures, so in July the
NHL endpoint returns seven September games. Emptiness cannot be the
signal; the date of those games is, and it is more useful anyway.
* A *finished* season rolls nowhere and returns its last game instead,
months in the past — so dates have to be filtered to the future
before the soonest one means anything.
"""
import requests
payload = requests.get(SCOREBOARD_URL.format(path=path),
timeout=REQUEST_TIMEOUT).json()
event_dates = [cls._parse_date(e.get('date'))
for e in payload.get('events') or []]
calendar_dates = []
for entry in (payload.get('leagues') or [{}])[0].get('calendar') or []:
calendar_dates.append(cls._parse_date(
entry if isinstance(entry, str) else entry.get('startDate')))
# Count the last day as current, rather than filtering on "later than
# right now": a game that began a few hours ago still means the league
# has something on, and dropping it would report a live slate as a
# finished season. A day's grace also keeps this correct whatever the
# user's timezone, since these timestamps are UTC.
now = datetime.now(timezone.utc)
def future(candidates):
return sorted(d for d in candidates if d and (now - d).days < 1)
# Events are fixtures; the calendar is week and phase boundaries,
# which routinely open days before their first game (an NFL week 1
# calendar entry starts the weekend before the Thursday opener).
# Reading the two together reported the earliest boundary as a game
# date -- "nothing on until 06 September" for a league whose first
# snap is the 10th. The calendar only gets a say when the scoreboard
# has no events at all to roll forward to: events that exist but are
# all in the past mean the season is over, and an offseason calendar
# phase must not be dressed up as its next game.
#
# The exception is a calendar of match days. With calendarType "day"
# and calendarIsWhitelist true, every entry is a day that has games,
# so a future entry is a real next fixture. Soccer needs it: between
# matchdays the scoreboard keeps showing the last one, so on
# 2026-09-29 every Premier League event was from 20 September and the
# next games (10 October) were only in the calendar. A day calendar
# that is not a whitelist (MLB's) lists days *without* games.
if any(event_dates):
upcoming = future(event_dates)
if not upcoming and cls._calendar_is_match_days(payload):
upcoming = future(calendar_dates)
else:
upcoming = future(calendar_dates)
if not upcoming:
if not any(event_dates) and not any(calendar_dates):
return None # Nothing published either way; draw no conclusion.
if cls._moved_to_later_phase(payload):
return None # e.g. postseason under way; see the method.
return ("the season has finished and the next one's fixtures are "
"not published yet")
# A day or two out is just an off day, and saying so would be noise.
if (upcoming[0] - now).days < cls.GAP_DAYS:
return None
return "the league has nothing on until {}".format(
upcoming[0].strftime('%d %B %Y'))
@staticmethod
def _calendar_is_match_days(payload) -> bool:
"""Whether the league calendar lists the days that have games."""
league = (payload.get('leagues') or [{}])[0] or {}
return (league.get('calendarType') == 'day'
and league.get('calendarIsWhitelist') is True)
@staticmethod
def _moved_to_later_phase(payload) -> bool:
"""
Whether the league is in a later in-season phase than its events.
ESPN does not roll the scoreboard forward into a postseason. The day
after MLB's regular season ended, the default scoreboard still returned
that last regular-season day, while ``leagues[0].season`` already said
Postseason and the wild-card games were two days out. Past events alone
then read as a finished season while the same process's upcoming
manager was showing the favourite's playoff games.
Only regular season (2) and postseason (3) count as "later". The
offseason (4) follows the postseason too, and there past events really
do mean the season is over.
"""
season = ((payload.get('leagues') or [{}])[0] or {}).get('season') or {}
league_type = (season.get('type') or {}).get('type')
if league_type not in (2, 3):
return False
event_types = [(e.get('season') or {}).get('type')
for e in payload.get('events') or []]
known = [t for t in event_types if isinstance(t, int)]
return bool(known) and all(t < league_type for t in known)
@staticmethod
def _parse_date(raw) -> Optional[datetime]:
if not raw or not isinstance(raw, str):
return None
try:
return datetime.fromisoformat(raw.replace('Z', '+00:00'))
except ValueError:
return None
@classmethod
def _suggest(cls, code: str, teams: Dict[str, str]) -> str:
"""Nearest matching code for a typo, as a ready-to-log clause."""
upper = (code or "").strip().upper()
if not upper or code in teams:
return ""
# Right code, wrong case — matching is case-sensitive. Guard on the case
# actually differing, so a valid code never draws this message.
for abbr in teams:
if abbr.upper() == upper:
return " Codes are case-sensitive; use {!r} ({}).".format(
abbr, teams[abbr])
ranked = cls._rank(upper, (a for a, n in teams.items()
if cls._abbreviates(upper, n)), teams)
if not ranked:
# Nicknames are often a fragment of a word rather than its initials:
# 'BAMA' sits inside 'Alabama' but abbreviates nothing in it. Require
# three characters, since shorter fragments match far too much.
if len(upper) >= 3:
ranked = cls._rank(
upper,
(a for a, n in teams.items()
if any(upper in w for w in cls._words(n))),
teams)
if len(ranked) == 1:
return " Closest match is {!r} ({}).".format(
ranked[0], teams[ranked[0]])
if ranked:
return " Did you mean {}?".format(", ".join(
"{!r} ({})".format(a, teams[a]) for a in ranked[:3]))
# Otherwise fall back to similarity, against names before codes: a name
# gives more characters to compare and so produces fewer ties.
names = {n.upper(): a for a, n in teams.items()}
hits = difflib.get_close_matches(upper, list(names), n=1, cutoff=0.6)
if hits:
abbr = names[hits[0]]
return " Closest match is {!r} ({}).".format(abbr, teams[abbr])
code_hits = difflib.get_close_matches(upper, list(teams), n=1, cutoff=0.6)
if code_hits:
return " Closest match is {!r} ({}).".format(
code_hits[0], teams[code_hits[0]])
return ""
@staticmethod
def _words(name: str):
return [w for w in re.split(r'[^A-Za-z0-9]+', (name or '').upper()) if w]
@classmethod
def _rank(cls, code: str, candidates, teams: Dict[str, str]):
"""
Order candidate codes best-first.
A code that picks up the *first* word of the name wins, because that is
how people shorten team names: 'SCAR' for South Carolina starts at
'South', whereas for Rutgers Scarlet Knights it starts mid-name. Without
this the tie is broken alphabetically and the obvious answer can land
third in the list.
"""
def key(abbr):
words = cls._words(teams.get(abbr, ''))
first_word_hit = bool(words) and words[0].startswith(code[:1])
return (not first_word_hit, len(abbr), abbr)
return sorted(set(candidates), key=key)
@staticmethod
def _abbreviates(code: str, name: str) -> bool:
"""
Whether ``code`` reads as an abbreviation of ``name``.
Each part of the code must be a prefix of one of the name's words, taken
in order — which is how people actually shorten team names. Plain string
similarity is no use for three-letter codes: 'MUN' scores identically
against 'MAN' and 'SUN', so Manchester United and Sunderland tie and the
suggestion is a coin flip. This rule separates them, because 'MUN'
splits as M-anchester UN-ited while Sunderland has no word starting M.
"""
words = [w for w in re.split(r'[^A-Za-z0-9]+', (name or '').upper()) if w]
def consume(rest: str, remaining: List[str]) -> bool:
if not rest:
return True
if not remaining:
return False
head, tail = remaining[0], remaining[1:]
# Skip this word entirely, as in "Manchester United" -> "UTD".
if consume(rest, tail):
return True
for size in range(1, min(len(rest), len(head)) + 1):
if head.startswith(rest[:size]) and consume(rest[size:], tail):
return True
return False
return consume((code or "").strip().upper(), words)
+2 -8
View File
@@ -93,7 +93,7 @@ import tempfile
import threading import threading
import time import time
import traceback import traceback
from typing import Any, Callable, Dict, List, Optional, Tuple, TypedDict from typing import Any, Callable, Dict, List, Optional, Tuple
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -483,13 +483,7 @@ class FrameTimingRecorder:
raise raise
class _WatchdogSettings(TypedDict, total=False): def watchdog_settings() -> Dict[str, float]:
"""The StallWatchdog keyword arguments watchdog_settings() may set."""
threshold: float
poll: float
def watchdog_settings() -> _WatchdogSettings:
"""StallWatchdog arguments from ``LEDMATRIX_STALL_WATCHDOG_MS``, if set. """StallWatchdog arguments from ``LEDMATRIX_STALL_WATCHDOG_MS``, if set.
The poll comes down with the threshold, or a stall shorter than one poll The poll comes down with the threshold, or a stall shorter than one poll
+17 -42
View File
@@ -8,7 +8,7 @@ Extracted from LEDMatrix core to provide reusable functionality for plugins.
import logging import logging
import time import time
from pathlib import Path from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union from typing import Dict, List, Optional, Union
import requests import requests
from PIL import Image, ImageDraw from PIL import Image, ImageDraw
@@ -80,11 +80,6 @@ class LogoHelper:
# Time-bounded rather than permanent so a logo that appears later (the # Time-bounded rather than permanent so a logo that appears later (the
# downloader writes them at runtime) is still picked up. # downloader writes them at runtime) is still picked up.
self._missing_logos: Dict[str, float] = {} self._missing_logos: Dict[str, float] = {}
# Failed downloads by logo path. A logo that is absent (not a stale
# placeholder) has no on-disk timestamp to back off on, so without this
# every call retried the download -- up to a 30s timeout each time.
self._download_failures: Dict[str, float] = {}
# Session for HTTP requests # Session for HTTP requests
self.session = requests.Session() self.session = requests.Session()
@@ -125,7 +120,17 @@ class LogoHelper:
# Resolve the effective target size BEFORE the cache lookup so the # Resolve the effective target size BEFORE the cache lookup so the
# key is size-qualified — a panel-size change must not return a # key is size-qualified — a panel-size change must not return a
# logo resized for the old dimensions. # logo resized for the old dimensions.
max_width, max_height, scale = self._scaled_box(max_width, max_height, scale) if max_width is None:
max_width = int(self.display_width * DEFAULT_LOGO_BOX_FACTOR)
if max_height is None:
max_height = int(self.display_height * DEFAULT_LOGO_BOX_FACTOR)
# Imported here: src.element_style imports src.common (for bdf_font),
# whose __init__ imports this module.
from src.element_style import coerce_scale
scale = coerce_scale(scale, 1.0)
if scale != 1.0:
max_width = max(1, int(round(max_width * scale)))
max_height = max(1, int(round(max_height * scale)))
# The key carries the scaled box, so two elements scaled differently # The key carries the scaled box, so two elements scaled differently
# cannot be served each other's image. # cannot be served each other's image.
cache_key = f"{team_abbr}_{logo_path}_{max_width}x{max_height}" cache_key = f"{team_abbr}_{logo_path}_{max_width}x{max_height}"
@@ -154,7 +159,7 @@ class LogoHelper:
return None return None
# Load image # Load image
logo: Image.Image = Image.open(logo_path) logo = Image.open(logo_path)
if logo.mode != 'RGBA': if logo.mode != 'RGBA':
logo = logo.convert('RGBA') logo = logo.convert('RGBA')
@@ -199,12 +204,7 @@ class LogoHelper:
return self.load_logo(team_abbr, logo_path, max_width, max_height, return self.load_logo(team_abbr, logo_path, max_width, max_height,
scale) scale)
# Download if URL provided and file doesn't exist, unless the last # Download if URL provided and file doesn't exist
# attempt for this path failed recently.
failed_at = self._download_failures.get(str(logo_path))
if (logo_url and failed_at is not None
and time.time() - failed_at < MISSING_LOGO_RECHECK_SECONDS):
logo_url = None
if logo_url: if logo_url:
try: try:
self.logger.info(f"Downloading logo for {team_abbr} from {logo_url}") self.logger.info(f"Downloading logo for {team_abbr} from {logo_url}")
@@ -218,7 +218,6 @@ class LogoHelper:
scale) scale)
except Exception as e: except Exception as e:
self.logger.error(f"Failed to download logo for {team_abbr}: {e}") self.logger.error(f"Failed to download logo for {team_abbr}: {e}")
self._download_failures[str(logo_path)] = time.time()
# The retry failed, so restart the back-off. The stale # The retry failed, so restart the back-off. The stale
# placeholder is still on disk with its old timestamp, and # placeholder is still on disk with its old timestamp, and
# leaving it there means the next call retries immediately -- # leaving it there means the next call retries immediately --
@@ -226,30 +225,8 @@ class LogoHelper:
# exists to prevent. # exists to prevent.
self._refresh_stale_placeholder(logo_path) self._refresh_stale_placeholder(logo_path)
# Create placeholder if all else fails. Sized to the same scaled box # Create placeholder if all else fails
# a real logo gets, so a scaled element doesn't jump in size while return self._create_placeholder_logo(team_abbr, max_width, max_height)
# its logo is missing.
box_width, box_height, _ = self._scaled_box(max_width, max_height, scale)
return self._create_placeholder_logo(team_abbr, box_width, box_height)
def _scaled_box(self, max_width: Optional[int], max_height: Optional[int],
scale: float) -> Tuple[int, int, float]:
"""The logo box after defaults and the user's scale are applied.
Returns ``(width, height, coerced_scale)``.
"""
if max_width is None:
max_width = int(self.display_width * DEFAULT_LOGO_BOX_FACTOR)
if max_height is None:
max_height = int(self.display_height * DEFAULT_LOGO_BOX_FACTOR)
# Imported here: src.element_style imports src.common (for bdf_font),
# whose __init__ imports this module.
from src.element_style import coerce_scale
scale = coerce_scale(scale, 1.0)
if scale != 1.0:
max_width = max(1, int(round(max_width * scale)))
max_height = max(1, int(round(max_height * scale)))
return max_width, max_height, scale
def _invalidate_cached_logo(self, team_abbr: str, logo_path: Path) -> None: def _invalidate_cached_logo(self, team_abbr: str, logo_path: Path) -> None:
"""Drop every cached size of one logo after its file changed on disk.""" """Drop every cached size of one logo after its file changed on disk."""
@@ -263,7 +240,6 @@ class LogoHelper:
# leaving it would hide a logo we just downloaded. # leaving it would hide a logo we just downloaded.
for key in [k for k in self._missing_logos if k.startswith(prefix)]: for key in [k for k in self._missing_logos if k.startswith(prefix)]:
del self._missing_logos[key] del self._missing_logos[key]
self._download_failures.pop(str(logo_path), None)
@staticmethod @staticmethod
def _refresh_stale_placeholder(logo_path: Path) -> None: def _refresh_stale_placeholder(logo_path: Path) -> None:
@@ -356,10 +332,9 @@ class LogoHelper:
self._logo_cache.clear() self._logo_cache.clear()
self._cache_order.clear() self._cache_order.clear()
self._missing_logos.clear() self._missing_logos.clear()
self._download_failures.clear()
self.logger.debug("Logo cache cleared") self.logger.debug("Logo cache cleared")
def get_cache_stats(self) -> Dict[str, float]: def get_cache_stats(self) -> Dict[str, int]:
""" """
Get cache statistics. Get cache statistics.
+24 -41
View File
@@ -290,26 +290,6 @@ def get_cache_dir_mode() -> int:
return 0o2775 # rwxrwsr-x (setgid + group writable) return 0o2775 # rwxrwsr-x (setgid + group writable)
def _sudo_bash_candidates() -> list:
"""Bash paths to try, in order, when running a vetted helper via sudo.
sudoers matches the exact argv, so ``sudo -n <bash> <helper> ...`` only
works if <bash> is the same path configure_web_sudo.sh wrote into the
rule -- whatever ``command -v bash`` said on the machine that ran it.
On merged-/usr systems /usr/bin/bash and /bin/bash are the same file but
different strings to sudo, and the web user's PATH can differ from the
installer's, so no single guess is reliable. Callers try each in turn and
move on only when sudo refused the command line (SUDO_REFUSAL_PHRASES).
The helper is invoked through bash rather than its shebang for the same
reason: the rule names bash, not the script.
"""
candidates = []
for candidate in ("/usr/bin/bash", "/bin/bash", _shutil.which("bash")):
if candidate and candidate not in candidates:
candidates.append(candidate)
return candidates
def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> bool: def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> bool:
""" """
Remove a directory using sudo as a last resort. Remove a directory using sudo as a last resort.
@@ -370,25 +350,22 @@ def sudo_remove_directory(path: Path, allowed_bases: Optional[list] = None) -> b
logger.error(f"Safe removal helper not found: {helper_script}") logger.error(f"Safe removal helper not found: {helper_script}")
return False return False
bash_path = _shutil.which('bash') or '/bin/bash'
try: try:
for bash_path in _sudo_bash_candidates(): result = subprocess.run(
result = subprocess.run( ['sudo', '-n', bash_path, str(helper_script), str(resolved)],
['sudo', '-n', bash_path, str(helper_script), str(resolved)], capture_output=True,
capture_output=True, text=True,
text=True, timeout=30
timeout=30 )
) if result.returncode == 0 and not resolved.exists():
if result.returncode == 0 and not resolved.exists(): logger.info(f"Successfully removed {path} via sudo helper")
logger.info(f"Successfully removed {path} via sudo helper") return True
return True else:
# Only a refused command line is worth another bash path; if the stderr = result.stderr.strip()
# helper itself ran and failed, a retry would just repeat it. logger.error(f"sudo helper failed for {path}: {stderr}")
if result.returncode == 0 or not any( return False
phrase in (result.stderr or '') for phrase in SUDO_REFUSAL_PHRASES):
break
stderr = (result.stderr or '').strip()
logger.error(f"sudo helper failed for {path}: {stderr}")
return False
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
logger.error(f"sudo helper timed out for {path}") logger.error(f"sudo helper timed out for {path}")
return False return False
@@ -440,10 +417,16 @@ def install_requirements_file(req_file: Path, timeout: int = 300) -> subprocess.
wrapper = project_root / "scripts" / "fix_perms" / "safe_pip_install.sh" wrapper = project_root / "scripts" / "fix_perms" / "safe_pip_install.sh"
if wrapper.exists(): if wrapper.exists():
# See _sudo_bash_candidates for why bash is invoked by explicit path # See sudo_remove_directory / configure_web_sudo.sh for why bash must
# and why there is more than one to try. # be invoked with an explicit, known path rather than relying on the
# wrapper's shebang: sudoers matches the exact command line.
bash_candidates = []
for candidate in ("/usr/bin/bash", "/bin/bash", _shutil.which("bash")):
if candidate and candidate not in bash_candidates:
bash_candidates.append(candidate)
result = None result = None
for bash_path in _sudo_bash_candidates(): for bash_path in bash_candidates:
# bash_path and wrapper are fixed, known-good paths, and # bash_path and wrapper are fixed, known-good paths, and
# safe_pip_install.sh independently re-validates req_file is an # safe_pip_install.sh independently re-validates req_file is an
# allowed requirements.txt before installing anything as root. # allowed requirements.txt before installing anything as root.
-238
View File
@@ -1,238 +0,0 @@
"""Let a background thread run Python only while the render thread waits on vsync.
With plugin rendering moved to Vegas's prefetch thread (DisplayManager.offscreen,
#630) the render thread no longer stops for it, but it still shares the GIL
with it. The render thread spends most of each refresh inside SwapOnVSync,
which releases the GIL, and needs it back the moment the swap returns. If the
prefetch thread is running Python right then, the render thread waits: up to
the switch interval (5ms) behind bytecode, and for as long as a C call that
keeps the GIL takes. On hdpi that showed up as frames 2-5 refreshes late while
a group was being prepared.
The gate turns that around. The display manager opens it just before each swap,
with a deadline shortly ahead of the refresh the swap will return on, and
closes it when the swap returns. A thread inside ``gate.yielding()`` checks it on
every Python and C call through a profile hook, and once the window has closed
it parks -- blocked on a condition, GIL released -- until the next swap opens
it. The render thread then finds the GIL free when its refresh arrives, and the
background work runs in time the render thread was only spending waiting.
Parking a thread is only safe if nothing the render thread needs is stuck
behind it, so it is never parked:
* while it holds a lock registered with ``guard()`` (the Vegas buffers and
caches the render thread also takes);
* inside logging, threading, importlib or the cache, all of which take locks the
render thread can take too;
* when there is no render loop to protect -- no swap for ``STALE_SECONDS``, as
on a static screen or a stalled frame.
And a parked thread is never held more than ``MAX_WAIT_SECONDS`` at a time, so
whatever the gate gets wrong costs a frame, not a freeze. The render thread
itself is never gated, whatever it calls.
It gates the prefetch thread only. Gating the ESPN fetch threads as well was
tried for the hourly sports refresh, twenty-odd of them at once, and measured
worse on hdpi (0.85% late frames without it, 1.14% with it, across a burst every
five minutes): each parked thread has to take the GIL again just to park at the
end of every window, and the fetches ran two to three times as long.
"""
from __future__ import annotations
import math
import sys
import threading
import time
from collections import deque
from typing import Any, Callable, Deque, List, Optional, cast
from src.common.frame_timing import binding_releases_gil
#: Park background threads this long before the refresh a swap will return on,
#: so a short C call already under way has finished by then.
MARGIN_SECONDS = 0.002
#: The longest a background thread is parked in one go.
MAX_WAIT_SECONDS = 0.05
#: No swap for this long means there is no render loop running to protect.
STALE_SECONDS = 0.05
#: Swaps needed before the refresh period is trusted enough to open a window.
MIN_SAMPLES = 8
#: Parking inside any of these modules could hold a lock the render thread
#: takes: logging handler locks, Condition and Event internals, the module
#: import locks, and the disk and memory cache locks. Matched by module name,
#: not file path: a path can say "cache" or "logging" for reasons of its own --
#: a virtualenv under ~/.cache, or GitHub's /opt/hostedtoolcache, where every
#: stdlib frame would otherwise count and the gate would never park anything.
_UNSAFE_MODULES = frozenset({
"logging", "threading", "importlib", "src.cache_manager", "src.cache",
})
_UNSAFE_PREFIXES = ("logging.", "importlib.", "_frozen_importlib", "src.cache.")
def _unsafe(frame: Any, base: Any) -> bool:
"""True if a frame above ``base`` comes from somewhere parking could deadlock.
``base`` is the frame that entered ``yielding()``; what lies below it (the
thread's own bootstrap in threading.py) holds nothing.
"""
while frame is not None and frame is not base:
name = frame.f_globals.get("__name__") or ""
if name in _UNSAFE_MODULES or name.startswith(_UNSAFE_PREFIXES):
return True
frame = frame.f_back
return False
def swap_releases_gil() -> Optional[bool]:
"""Whether the loaded rgbmatrix binding releases the GIL, or None if none is loaded.
A thin delegate to src.common.frame_timing.binding_releases_gil (#629),
which this used to duplicate line for line. The name stays because the
coordinator calls it here and tests replace it here.
"""
return binding_releases_gil()
def _held(lock: Any) -> bool:
"""Is ``lock`` held? RLocks report this thread's ownership; plain locks, anyone's."""
is_owned = getattr(lock, "_is_owned", None)
if is_owned is not None:
return cast(bool, is_owned())
return cast(bool, lock.locked())
class RenderGate:
"""Opened by the render thread around each swap; honoured by background threads."""
def __init__(self, clock: Callable[[], float] = time.monotonic):
self.clock = clock
self._cond = threading.Condition()
self._generation = 0
self._open_until = 0.0
self._last_return: Optional[float] = None
self._periods: Deque[float] = deque(maxlen=64)
self._period: Optional[float] = None
self._guarded: List[Any] = []
self._local = threading.local()
self._render_ident: Optional[int] = None
#: How often, and for how long in all, background threads were parked.
self.parks = 0
self.parked_seconds = 0.0
def guard(self, *locks: Any) -> None:
"""Never park a thread while it holds (or, for a plain Lock, anyone holds) these."""
self._guarded.extend(lock for lock in locks if lock is not None)
# -- render thread -----------------------------------------------------
def refresh_period(self) -> Optional[float]:
"""The panel's refresh period from recent swaps, or None until known.
The 10th percentile of the gaps between swap returns, each divided by
the hold: a late frame only ever lengthens a gap, so the low end is
the panel's own period.
"""
return self._period
def before_swap(self, hold: int) -> None:
"""The render thread is about to block in SwapOnVSync: open the window."""
hold = max(1, int(hold))
period = self._period
now = self.clock()
last = self._last_return
if period and last is not None and now - last < STALE_SECONDS:
# The swap returns on the first refresh boundary after both the
# current frame's hold is up and this frame has been handed over;
# boundaries fall a whole period apart from the last return.
refreshes = max(hold, math.ceil((now - last) / period))
open_until = last + refreshes * period - MARGIN_SECONDS
else:
open_until = 0.0 # no rhythm to predict from: leave threads be
with self._cond:
self._open_until = open_until
self._generation += 1
self._cond.notify_all()
def after_swap(self, hold: int) -> None:
"""The swap returned and the render thread needs the GIL: close the window."""
now = self.clock()
self._open_until = 0.0
if self._render_ident is None:
# The first thread to swap is the render loop. A plugin pushing a
# live refresh from its update thread swaps too, but must not take
# over its exemption.
self._render_ident = threading.get_ident()
last = self._last_return
if last is not None and now - last < STALE_SECONDS:
self._periods.append((now - last) / max(1, int(hold)))
if len(self._periods) >= MIN_SAMPLES:
ordered = sorted(self._periods)
self._period = ordered[len(ordered) // 10]
self._last_return = now
# -- background threads ------------------------------------------------
def _should_park(self, frame: Any, now: float) -> bool:
if now < self._open_until:
return False # inside the window
last = self._last_return
if last is None or now - last > STALE_SECONDS or self._period is None:
return False # no render loop to protect
for lock in self._guarded:
if _held(lock):
return False
return not _unsafe(frame, getattr(self._local, "base", None))
def _hook(self, frame: Any, _event: str, _arg: Any) -> None:
now = self.clock()
if not self._should_park(frame, now):
return
generation = self._generation
with self._cond:
self._cond.wait_for(lambda: self._generation != generation,
timeout=MAX_WAIT_SECONDS)
self.parks += 1
self.parked_seconds += self.clock() - now
def yielding(self) -> "_Yielding":
"""``with gate.yielding():`` runs the block giving way to the render thread."""
return _Yielding(self)
class _Yielding:
"""Installs a gate's profile hook on the thread for the length of a block."""
def __init__(self, gate: RenderGate):
self.gate = gate
self._previous: Any = None
self._previous_base: Any = None
self._skipped = False
def __enter__(self) -> RenderGate:
gate = self.gate
# pylint: disable=protected-access
if threading.get_ident() == gate._render_ident:
self._skipped = True # parking the render thread parks the display
return gate
local = gate._local
self._previous_base = getattr(local, "base", None)
if self._previous_base is None:
# Nested blocks keep the outermost frame, so everything the thread
# entered since it first gave way is still checked for locks.
local.base = sys._getframe(1)
self._previous = sys.getprofile()
sys.setprofile(gate._hook)
return gate
def __exit__(self, *_exc: Any) -> None:
if self._skipped:
return
sys.setprofile(self._previous)
self.gate._local.base = self._previous_base # pylint: disable=protected-access
+6 -10
View File
@@ -49,9 +49,7 @@ from __future__ import annotations
import logging import logging
from dataclasses import dataclass, replace from dataclasses import dataclass, replace
from typing import Any, Dict, List, Optional, cast from typing import Any, Dict, Optional
from src.matrix_support import DEFAULT_REFRESH_LIMIT_HZ
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -65,9 +63,8 @@ MIN_PIXELS_PER_SECOND = 1.0
MAX_PIXELS_PER_SECOND = 500.0 MAX_PIXELS_PER_SECOND = 500.0
#: Assumed refresh when the caller does not say. Matches the usual #: Assumed refresh when the caller does not say. Matches the usual
#: ``display.hardware.limit_refresh_rate_hz``, and is the cap DisplayManager #: ``display.hardware.limit_refresh_rate_hz``.
#: applies when that key is missing. DEFAULT_REFRESH_HZ = 100.0
DEFAULT_REFRESH_HZ = float(DEFAULT_REFRESH_LIMIT_HZ)
#: How far px/s may sit from a whole number of pixels per refresh before it is #: How far px/s may sit from a whole number of pixels per refresh before it is
#: worth warning about. 0.05px per frame is invisible; a third of a pixel is not. #: worth warning about. 0.05px per frame is invisible; a third of a pixel is not.
@@ -132,14 +129,14 @@ def crisp_ladder(
refresh_hz: float = DEFAULT_REFRESH_HZ, refresh_hz: float = DEFAULT_REFRESH_HZ,
max_frame_hold: int = MAX_FRAME_HOLD, max_frame_hold: int = MAX_FRAME_HOLD,
max_pixels_per_frame: int = MAX_PIXELS_PER_FRAME, max_pixels_per_frame: int = MAX_PIXELS_PER_FRAME,
) -> List[CrispSpeed]: ):
"""Every whole-pixel speed this panel can show, slowest first. """Every whole-pixel speed this panel can show, slowest first.
Duplicates are collapsed keeping the gentlest option: 100 px/s is reachable Duplicates are collapsed keeping the gentlest option: 100 px/s is reachable
as 1px every refresh or 2px every 2nd refresh, and the former moves in as 1px every refresh or 2px every 2nd refresh, and the former moves in
smaller increments, so that is the one worth offering. smaller increments, so that is the one worth offering.
""" """
best: Dict[float, CrispSpeed] = {} best = {}
for hold in range(1, max_frame_hold + 1): for hold in range(1, max_frame_hold + 1):
for ppf in range(1, max_pixels_per_frame + 1): for ppf in range(1, max_pixels_per_frame + 1):
pps = refresh_hz / hold * ppf pps = refresh_hz / hold * ppf
@@ -434,8 +431,7 @@ def configure(
# they start scrolling. configure() only reports what is needed. # they start scrolling. configure() only reports what is needed.
if choice: if choice:
# Set whenever there is a crisp choice (see the replace() above). requested = settings.requested_pixels_per_second
requested = cast(float, settings.requested_pixels_per_second)
if abs(requested - applied) > 0.05: if abs(requested - applied) > 0.05:
log.info( log.info(
"Scroll configured: %s (asked for %.1f px/s from %s; " "Scroll configured: %s (asked for %.1f px/s from %s; "
-18
View File
@@ -254,9 +254,6 @@ class ScrollHelper:
now = time.time() now = time.time()
self.scroll_start_time = now self.scroll_start_time = now
self.last_progress_log_time = now self.last_progress_log_time = now
# The position just went back to 0; the first update must not advance
# it by however long the helper sat idle (off-screen) before this.
self.last_update_time = now
self.logger.info( self.logger.info(
"Dynamic duration target set to %ds (min=%ds, max=%ds, buffer=%.2f)", "Dynamic duration target set to %ds (min=%ds, max=%ds, buffer=%.2f)",
self.calculated_duration, self.calculated_duration,
@@ -779,18 +776,6 @@ class ScrollHelper:
self.clear_cache() self.clear_cache()
return return
# Every frame is cut from cached_array with Image.frombytes('RGB', ...),
# which reads a 4-channel (RGBA) array as garbage and raises on a
# 1-channel (L) one. Transparent pixels go to black, the panel's
# background, rather than to whatever colour hides under the alpha.
if image.mode != 'RGB':
if 'A' in image.mode or 'transparency' in image.info:
rgba = image.convert('RGBA')
image = Image.new('RGB', rgba.size, (0, 0, 0))
image.paste(rgba, (0, 0), rgba)
else:
image = image.convert('RGB')
# Set the cached image # Set the cached image
self.cached_image = image self.cached_image = image
@@ -817,9 +802,6 @@ class ScrollHelper:
self.scroll_start_time = now self.scroll_start_time = now
self.last_progress_log_time = now self.last_progress_log_time = now
self.last_step_time = now # Initialize step timer for frame-based scrolling self.last_step_time = now # Initialize step timer for frame-based scrolling
# The position just went back to 0; the first update must not advance
# it by however long the helper sat idle before this image arrived.
self.last_update_time = now
self.logger.debug("Set scrolling image: %dx%d, total_scroll_width=%d", self.logger.debug("Set scrolling image: %dx%d, total_scroll_width=%d",
image.width, image.height, self.total_scroll_width) image.width, image.height, self.total_scroll_width)
+1 -2
View File
@@ -144,8 +144,7 @@ def resolve_font_color(config: Optional[Dict[str, Any]],
if len(matches) > 1: if len(matches) > 1:
configured = [] configured = []
for element in matches: for element in matches:
# None as the default makes it come back when unconfigured. colour = element_color(config, element, None, mode)
colour = element_color(config, element, None, mode) # type: ignore[arg-type]
if colour is not None and colour not in configured: if colour is not None and colour not in configured:
configured.append(colour) configured.append(colour)
if len(configured) == 1: if len(configured) == 1:
-262
View File
@@ -1,262 +0,0 @@
"""Timezone resolution for the scoreboard plugins.
Game start times arrive from ESPN in UTC and have to be converted to the
user's local zone before they are drawn. This module owns the "which zone?"
decision so every part of a scoreboard (its scorebug, its scroll-mode game
card and its plugin manager) agrees.
The scoreboards each carried a copy of this module as ``<sport>_timezone.py``.
The copies were identical apart from two per-plugin values, which are
keyword-only arguments here: ``plugin_label``, the name the final warning
tells the user to open, and ``writeback_fixed_in`` (see below).
Resolution order, first valid wins:
1. ``timezone`` in the plugin's own config (explicit per-plugin override)
2. The LEDMatrix global timezone via ``plugin_manager.config_manager``
3. The LEDMatrix global timezone via ``cache_manager.config_manager``
4. The host system's zone (``TZ``, ``/etc/timezone``, ``/etc/localtime``)
5. UTC
Steps 2 and 3 matter because the core does not consistently hang
``config_manager`` off both objects -- reading only one of them is what made
a scoreboard fall through to UTC while the clock plugin (which checks
``plugin_manager`` first) showed the right time on the same device. Step 4 is
the backstop for cores that expose no ``config_manager`` at all: a Pi with its
system clock set correctly should never end up rendering UTC.
Two traps this module exists to avoid, both of which render every start time
in UTC on a correctly-configured device:
* **The stale ``"UTC"`` artifact.** Some plugins once wrote
``"timezone": "UTC"`` into the *saved* config whenever resolution failed, and
that write-back stuck -- thereafter shadowing the real global timezone. Such
a plugin passes ``writeback_fixed_in`` (the release that fixed it), and step 1
then treats a bare ``"UTC"`` as suspect: it is honored only when nothing
downstream disagrees. ``Etc/UTC`` is the unambiguous spelling for "I really
do want UTC"; the bug never produced it, so it is always honored. A plugin
that never had the bug leaves ``writeback_fixed_in`` as ``None``, and its
plugin-level ``"UTC"`` is honored verbatim.
* **``get_timezone()``'s own default.** The core's
``ConfigManager.get_timezone()`` is ``self.config.get('timezone', 'UTC')``, so
it hands back ``"UTC"`` for a config that simply has no ``timezone`` key.
Steps 2 and 3 read the raw config dict instead, so an absent key falls
through to the system zone rather than latching onto that default.
"""
import logging
import os
from typing import Any, Dict, Iterable, Optional, Tuple
import pytz
logger = logging.getLogger(__name__)
def _from_config_manager(config_manager: Any, log: logging.Logger) -> Optional[str]:
"""Pull the global timezone out of a core ConfigManager, if it has one.
Reads the raw config dict in preference to ``get_timezone()``. The core's
``ConfigManager.get_timezone()`` is ``self.config.get('timezone', 'UTC')`` --
it substitutes its own ``"UTC"`` when the key is absent, which is
indistinguishable from the user deliberately choosing UTC. Taking that at
face value would mask a missing global setting and stop resolution ever
reaching the host system zone. So: if the raw config is readable and has no
``timezone`` key, report "nothing here" and let the caller fall through.
``get_timezone()`` is only consulted for cores that expose no raw config.
"""
if config_manager is None:
return None
raw_readable = False
for loader_name in ("get_config", "load_config"):
loader = getattr(config_manager, loader_name, None)
if not callable(loader):
continue
try:
main_config = loader()
except Exception:
log.debug("config_manager.%s() failed", loader_name, exc_info=True)
continue
if isinstance(main_config, dict):
raw_readable = True
name: Optional[str] = main_config.get("timezone")
if name:
return name
if raw_readable:
return None
getter = getattr(config_manager, "get_timezone", None)
if callable(getter):
try:
name = getter()
if name:
return name
except Exception:
log.debug("config_manager.get_timezone() failed", exc_info=True)
return None
def system_timezone_name() -> Optional[str]:
"""Best-effort IANA name for the host's configured timezone."""
name = os.environ.get("TZ")
if name:
return name
# Debian / Raspberry Pi OS record the zone name here.
try:
with open("/etc/timezone", "r", encoding="utf-8") as handle:
name = handle.read().strip()
if name:
return name
except OSError:
pass
# Otherwise /etc/localtime is a symlink into the zoneinfo tree.
try:
path = os.path.realpath("/etc/localtime")
marker = "zoneinfo" + os.sep
if marker in path:
return path.split(marker, 1)[1]
except OSError:
pass
return None
def _validated(name: object, source: str, log: logging.Logger) -> Optional[str]:
"""Return a usable IANA name, or None if blank/absent/not a real zone."""
if not isinstance(name, str):
return None
name = name.strip()
if not name:
return None
try:
pytz.timezone(name)
except pytz.UnknownTimeZoneError:
log.warning("Ignoring invalid timezone %r from %s", name, source)
return None
except Exception:
# Not an unknown-zone error, so something else went wrong inside pytz.
# Log it loudly rather than silently reclassifying it as "invalid" --
# but still don't propagate: this runs in the render path, and a
# mislabelled zone beats taking the whole display down.
log.warning(
"Unexpected error validating timezone %r from %s; ignoring it",
name, source, exc_info=True,
)
return None
return name
def resolve_timezone_name(
config: Optional[Dict[str, Any]] = None,
plugin_manager: Any = None,
cache_manager: Any = None,
log: Optional[logging.Logger] = None,
*,
plugin_label: str,
writeback_fixed_in: Optional[str] = None,
) -> str:
"""Return the IANA timezone name to render game times in.
Never raises and never returns an empty string; falls back to ``"UTC"``
only when every source is missing or invalid.
``plugin_label`` names the plugin in the warning logged when nothing
resolves (e.g. ``"hockey scoreboard"``). ``writeback_fixed_in`` is the
plugin release that stopped writing ``"UTC"`` back into the saved config,
for a plugin that ever did; ``None`` (the default) otherwise.
"""
log = log or logger
def downstream():
"""Yield (source, name) for everything except the plugin's own config.
Lazy: ``SportsCore._get_timezone()`` runs this once per game and the
answer is almost always already in the plugin config, so evaluating on
demand keeps the common case from calling into both config managers and
stat-ing the host timezone files every time.
"""
yield (
"plugin_manager.config_manager",
_from_config_manager(getattr(plugin_manager, "config_manager", None), log),
)
yield (
"cache_manager.config_manager",
_from_config_manager(getattr(cache_manager, "config_manager", None), log),
)
yield "system timezone", system_timezone_name()
def first_valid(
sources: Iterable[Tuple[str, Any]],
) -> Tuple[Optional[str], Optional[str]]:
for source, name in sources:
name = _validated(name, source, log)
if name:
return source, name
return None, None
plugin_value = _validated((config or {}).get("timezone"), "plugin config", log)
if writeback_fixed_in is not None and plugin_value and plugin_value.lower() == "utc":
# Before writeback_fixed_in this plugin wrote "timezone": "UTC" into
# the saved config whenever it failed to resolve a global timezone, and
# that write-back persisted. A bare "UTC" is therefore far more likely
# to be that artifact than a deliberate choice -- it only ever appeared
# on failure. Honor it only when nothing downstream disagrees; a user
# who genuinely wants UTC writes the unambiguous "Etc/UTC", which the
# bug never produced and which falls through to the normal path below.
source, downstream_name = first_valid(downstream())
if downstream_name and downstream_name.lower() not in ("utc", "etc/utc"):
log.warning(
"Ignoring the plugin-level timezone 'UTC': it is almost "
"certainly left over from the write-back bug fixed in %s, and "
"%s says %s. Using %s. If you really do want UTC here, set "
"this plugin's timezone to 'Etc/UTC' instead.",
writeback_fixed_in, source, downstream_name, downstream_name,
)
return downstream_name
log.debug("Plugin-level timezone 'UTC' agrees with %s; using UTC", source or "no other source")
return "UTC"
if plugin_value:
log.debug("Resolved timezone %s from plugin config", plugin_value)
return plugin_value
source, name = first_valid(downstream())
if name:
log.debug("Resolved timezone %s from %s", name, source)
return name
log.warning(
"Could not determine a timezone from the plugin config, the LEDMatrix "
"config or the system; game times will be shown in UTC. Set a timezone "
"in the %s's Advanced Settings to override.",
plugin_label,
)
return "UTC"
def resolve_timezone(
config: Optional[Dict[str, Any]] = None,
plugin_manager: Any = None,
cache_manager: Any = None,
log: Optional[logging.Logger] = None,
*,
plugin_label: str,
writeback_fixed_in: Optional[str] = None,
):
"""``resolve_timezone_name`` as a ready-to-use tzinfo object."""
return pytz.timezone(
resolve_timezone_name(
config=config,
plugin_manager=plugin_manager,
cache_manager=cache_manager,
log=log,
plugin_label=plugin_label,
writeback_fixed_in=writeback_fixed_in,
)
)
+14 -76
View File
@@ -32,7 +32,6 @@ from typing import Callable, Optional
import numpy as np import numpy as np
from PIL import Image from PIL import Image
from src.config_manager_atomic import _replace
from src.display_geometry import DEFAULT_CHAIN_LENGTH, DEFAULT_COLS, DEFAULT_ROWS from src.display_geometry import DEFAULT_CHAIN_LENGTH, DEFAULT_COLS, DEFAULT_ROWS
# Raw-frame wire format: 8-byte magic + 4-byte header + raw RGB pixels # Raw-frame wire format: 8-byte magic + 4-byte header + raw RGB pixels
@@ -54,17 +53,6 @@ HEARTBEAT_INTERVAL = 2.0 # follower sends heartbeat every 2 s
PEER_TIMEOUT = 6.0 # leader: no heartbeat → follower gone PEER_TIMEOUT = 6.0 # leader: no heartbeat → follower gone
LEADER_TIMEOUT = 6.0 # follower: no frame → leader gone LEADER_TIMEOUT = 6.0 # follower: no frame → leader gone
STATUS_FILE = os.path.join(tempfile.gettempdir(), "led_matrix_sync_status.json") STATUS_FILE = os.path.join(tempfile.gettempdir(), "led_matrix_sync_status.json")
# Serialises writes to STATUS_FILE (several threads report status) against
# its removal in stop(), so a write already under way cannot put the file back
# after the display process has shut down.
_STATUS_LOCK = threading.Lock()
def _remove_status_file() -> None:
try:
os.remove(STATUS_FILE)
except FileNotFoundError:
pass
class SyncRole(Enum): class SyncRole(Enum):
@@ -95,10 +83,6 @@ class DisplaySyncManager:
back to its own plugins when the leader stops sending. back to its own plugins when the leader stops sending.
""" """
# Set by stop(); status writes after that are dropped. Class-level so
# instances built without __init__ (tests) have it too.
_status_closed = False
def __init__( def __init__(
self, self,
role_str: str, role_str: str,
@@ -128,9 +112,6 @@ class DisplaySyncManager:
self._peer_ip: Optional[str] = None self._peer_ip: Optional[str] = None
self._peer_compatible: bool = False self._peer_compatible: bool = False
self._peer_chain: int = 0 self._peer_chain: int = 0
# time.monotonic() readings, like _last_leader_frame_time: these only
# feed the timeout watchdogs, and a wall-clock step (NTP correcting a
# Pi with no RTC) would otherwise fake or mask a timeout.
self._last_heartbeat_time: float = 0.0 self._last_heartbeat_time: float = 0.0
self._leader_width: int = 0 # set by display_controller after init self._leader_width: int = 0 # set by display_controller after init
self._oversized_frame_warned: bool = False self._oversized_frame_warned: bool = False
@@ -157,14 +138,6 @@ class DisplaySyncManager:
self._send_sock: Optional[socket.socket] = None self._send_sock: Optional[socket.socket] = None
if self.role == SyncRole.STANDALONE: if self.role == SyncRole.STANDALONE:
# Standalone never writes a status file, so one still here is
# from an earlier run as leader or follower. The web UI would
# keep reporting that run's peer as if it were live.
try:
with _STATUS_LOCK:
_remove_status_file()
except OSError as exc:
logger.debug("Sync: could not remove stale status file: %s", exc)
return return
if self.role == SyncRole.LEADER: if self.role == SyncRole.LEADER:
@@ -210,7 +183,7 @@ class DisplaySyncManager:
self._handle_hello(msg, sender_ip) self._handle_hello(msg, sender_ip)
elif t == "hb": elif t == "hb":
if self._peer_ip == sender_ip: if self._peer_ip == sender_ip:
self._last_heartbeat_time = time.monotonic() self._last_heartbeat_time = time.time()
except socket.timeout: except socket.timeout:
continue continue
except Exception as exc: except Exception as exc:
@@ -233,7 +206,7 @@ class DisplaySyncManager:
self._peer_ip = sender_ip self._peer_ip = sender_ip
self._peer_compatible = compatible self._peer_compatible = compatible
self._peer_chain = peer_chain self._peer_chain = peer_chain
self._last_heartbeat_time = time.monotonic() self._last_heartbeat_time = time.time()
prev_state = self._leader_state prev_state = self._leader_state
if compatible: if compatible:
@@ -277,7 +250,7 @@ class DisplaySyncManager:
while self._running: while self._running:
time.sleep(1.0) time.sleep(1.0)
if self._leader_state == LeaderState.CONNECTED: if self._leader_state == LeaderState.CONNECTED:
if time.monotonic() - self._last_heartbeat_time > PEER_TIMEOUT: if time.time() - self._last_heartbeat_time > PEER_TIMEOUT:
self.logger.info( self.logger.info(
"Sync: follower heartbeat timeout — peer disconnected" "Sync: follower heartbeat timeout — peer disconnected"
) )
@@ -505,7 +478,7 @@ class DisplaySyncManager:
"""Note that the leader at ``sender_ip`` just sent something, and """Note that the leader at ``sender_ip`` just sent something, and
switch from standalone to follower mode if not already following. switch from standalone to follower mode if not already following.
Returns True if this call made the switch.""" Returns True if this call made the switch."""
self._last_leader_frame_time = time.monotonic() self._last_leader_frame_time = time.time()
self._leader_ip = sender_ip self._leader_ip = sender_ip
if self._follower_state != FollowerState.STANDALONE: if self._follower_state != FollowerState.STANDALONE:
return False return False
@@ -625,13 +598,11 @@ class DisplaySyncManager:
heartbeat = json.dumps({"t": "hb"}).encode("utf-8") heartbeat = json.dumps({"t": "hb"}).encode("utf-8")
dest = ("<broadcast>", self.port) dest = ("<broadcast>", self.port)
# -inf, not 0.0: monotonic time starts near boot, so "now - 0.0" can last_hello = 0.0
# be under the interval and would delay the first announcement. last_hb = 0.0
last_hello = float("-inf")
last_hb = float("-inf")
while self._running: while self._running:
now = time.monotonic() now = time.time()
if now - last_hello >= HELLO_INTERVAL: if now - last_hello >= HELLO_INTERVAL:
try: try:
self._send_sock.sendto(hello, dest) self._send_sock.sendto(hello, dest)
@@ -650,7 +621,7 @@ class DisplaySyncManager:
while self._running: while self._running:
time.sleep(1.0) time.sleep(1.0)
if self._follower_state == FollowerState.FOLLOWER: if self._follower_state == FollowerState.FOLLOWER:
if time.monotonic() - self._last_leader_frame_time > LEADER_TIMEOUT: if time.time() - self._last_leader_frame_time > LEADER_TIMEOUT:
self.logger.info( self.logger.info(
"Sync: leader frame timeout — returning to standalone mode" "Sync: leader frame timeout — returning to standalone mode"
) )
@@ -676,10 +647,7 @@ class DisplaySyncManager:
def set_on_new_cycle(self, callback: Callable[[], None]) -> None: def set_on_new_cycle(self, callback: Callable[[], None]) -> None:
"""Follower: register a callback fired when the leader starts a new scroll cycle. """Follower: register a callback fired when the leader starts a new scroll cycle.
Used to trigger a local start_new_cycle() so both Pis rebuild from same fresh data.
Nothing in core registers one: display_controller follows the leader
through set_on_scroll_image() and the scroll position instead of
rebuilding locally. The hook stays for callers that want the signal.
""" """
self._on_new_cycle = callback self._on_new_cycle = callback
@@ -724,40 +692,18 @@ class DisplaySyncManager:
def write_status_file(self) -> None: def write_status_file(self) -> None:
"""Write current sync status to STATUS_FILE for the web UI to read.""" """Write current sync status to STATUS_FILE for the web UI to read."""
tmp = None
try: try:
status = self.get_status() status = self.get_status()
status["ts"] = time.time() status["ts"] = time.time()
with _STATUS_LOCK: tmp = STATUS_FILE + ".tmp"
if self._status_closed: with open(tmp, "w") as f:
return json.dump(status, f)
# A unique temp name per write, like frame_timing's stats os.replace(tmp, STATUS_FILE)
# file: the receive loop, watchdog and hello handler all
# write, and with one fixed ".tmp" name one thread's
# os.replace() could move the other's half-written file.
fd, tmp = tempfile.mkstemp(
dir=os.path.dirname(STATUS_FILE) or ".",
prefix=".led_matrix_sync_status.", suffix=".tmp")
with os.fdopen(fd, "w") as f:
json.dump(status, f)
# mkstemp makes it owner-only; the web UI may run as a
# different user from the display service.
os.chmod(tmp, 0o644)
# _replace: on Windows a rename can briefly fail with
# "Access is denied" while a scanner holds the target open.
_replace(tmp, STATUS_FILE)
tmp = None
except Exception as exc: except Exception as exc:
self.logger.debug("Sync: status file write error: %s", exc) self.logger.debug("Sync: status file write error: %s", exc)
finally:
if tmp is not None:
try:
os.unlink(tmp)
except OSError:
pass
def stop(self) -> None: def stop(self) -> None:
"""Shut down threads, close sockets and withdraw the status file.""" """Shut down threads and close sockets."""
self._running = False self._running = False
for sock in (self._recv_sock, self._send_sock, self._img_server_sock): for sock in (self._recv_sock, self._send_sock, self._img_server_sock):
if sock: if sock:
@@ -765,11 +711,3 @@ class DisplaySyncManager:
sock.close() sock.close()
except Exception as exc: except Exception as exc:
self.logger.debug("Sync: error closing socket: %s", exc) self.logger.debug("Sync: error closing socket: %s", exc)
# The web UI reads this file as live status. Left behind, it went on
# reporting a connected peer after the display service had stopped.
try:
with _STATUS_LOCK:
self._status_closed = True
_remove_status_file()
except OSError as exc:
self.logger.debug("Sync: could not remove status file: %s", exc)
+1 -2
View File
@@ -206,8 +206,7 @@ class ConfigService:
# Sleep with periodic checks for stop signal # Sleep with periodic checks for stop signal
for _ in range(int(self._watch_interval)): for _ in range(int(self._watch_interval)):
if self._stop_watching: if self._stop_watching:
# Set from another thread; mypy keeps the while's narrowing. break
break # type: ignore[unreachable]
time.sleep(1) time.sleep(1)
except Exception as e: except Exception as e:
+1 -1
View File
@@ -41,7 +41,7 @@ def deprecated(removal: str, alternative: Optional[str] = None) -> Callable[[F],
warnings.warn(message, DeprecationWarning, stacklevel=2) warnings.warn(message, DeprecationWarning, stacklevel=2)
return func(*args, **kwargs) return func(*args, **kwargs)
wrapper.__deprecated__ = message # type: ignore[attr-defined] # functools' _Wrapped doesn't declare it wrapper.__deprecated__ = message
return wrapper # type: ignore[return-value] return wrapper # type: ignore[return-value]
return decorate return decorate
+3 -4
View File
@@ -24,7 +24,7 @@ next render. A location saved on the app itself always wins.
import json import json
import logging import logging
import time import time
from typing import Any, Callable, Dict, Iterable, List, Optional, cast from typing import Any, Callable, Dict, Iterable, List, Optional
GEOCODE_URL = "https://geocoding-api.open-meteo.com/v1/search" GEOCODE_URL = "https://geocoding-api.open-meteo.com/v1/search"
GEOCODE_TIMEOUT = 10 GEOCODE_TIMEOUT = 10
@@ -112,7 +112,6 @@ def parse_location(value: Any) -> Optional[Dict[str, Any]]:
``{"timezone": ...}`` when only the timezone box is filled) all mean the ``{"timezone": ...}`` when only the timezone box is filled) all mean the
user has not given the app a place. user has not given the app a place.
""" """
loc: Any
if isinstance(value, dict): if isinstance(value, dict):
loc = value loc = value
elif isinstance(value, str) and value.strip(): elif isinstance(value, str) and value.strip():
@@ -163,10 +162,10 @@ def geocode(city: str, state: Any = None, country: Any = None,
"""Look the city up on Open-Meteo. Raises on a network/HTTP failure.""" """Look the city up on Open-Meteo. Raises on a network/HTTP failure."""
import requests import requests
response = requests.get(GEOCODE_URL, params=cast(Dict[str, Any], { response = requests.get(GEOCODE_URL, params={
"name": city, "count": GEOCODE_RESULT_COUNT, "name": city, "count": GEOCODE_RESULT_COUNT,
"language": "en", "format": "json", "language": "en", "format": "json",
}), timeout=timeout) }, timeout=timeout)
response.raise_for_status() response.raise_for_status()
best = pick_geocode_result(response.json().get("results") or [], state, country) best = pick_geocode_result(response.json().get("results") or [], state, country)
if best is None: if best is None:
+31 -197
View File
@@ -23,13 +23,11 @@ Entry point: :func:`main` — instantiates :class:`DisplayController` and calls
import time import time
import os import os
import inspect import inspect
import signal
import json import json
import threading import threading
import types import types
from collections import deque
from contextlib import contextmanager from contextlib import contextmanager
from typing import Dict, Any, List, Optional, Callable, Tuple from typing import Dict, Any, List, Optional, Callable
from datetime import datetime from datetime import datetime
from concurrent.futures import ThreadPoolExecutor, as_completed # pylint: disable=no-name-in-module from concurrent.futures import ThreadPoolExecutor, as_completed # pylint: disable=no-name-in-module
import pytz import pytz
@@ -46,10 +44,6 @@ from src.vegas_mode.render_pipeline import SYNC_SEND_INTERVAL
# Get logger with consistent configuration # Get logger with consistent configuration
logger = get_logger(__name__) logger = get_logger(__name__)
# How often the unchanged current mode is republished for the web UI, which
# treats display_current_state older than 120 s as unknown.
CURRENT_STATE_REFRESH_SECONDS = 30
# How long startup will wait for plugins to fetch their first data before # How long startup will wait for plugins to fetch their first data before
# showing anything. Each plugin's update blocks for up to the executor's 30s # showing anything. Each plugin's update blocks for up to the executor's 30s
# timeout and they run one after another, so the uncapped total is the sum of # timeout and they run one after another, so the uncapped total is the sum of
@@ -102,16 +96,6 @@ class DisplayController:
it and start the run loop. it and start the run loop.
""" """
#: How long the run loop pauses per pass once a whole rotation has had
#: nothing to show. See _note_empty_pass.
EMPTY_ROTATION_PAUSE = 1.0
#: Consecutive passes whose mode had nothing to show, and the rotation
#: (on-demand or not, and its modes) they were counted in. Class-level so
#: controllers built without __init__ (tests) have them too.
_empty_pass_streak = 0
_empty_pass_rotation: Optional[Tuple[bool, Tuple[str, ...]]] = None
def __init__(self): def __init__(self):
start_time = time.time() start_time = time.time()
logger.info("Starting DisplayController initialization") logger.info("Starting DisplayController initialization")
@@ -197,12 +181,6 @@ class DisplayController:
# scroll image arrives. # scroll image arrives.
self._follower_pending_new_image = False self._follower_pending_new_image = False
self._follower_last_frame = None self._follower_last_frame = None
# (image, array) from the leader, handed from the sync TCP thread to
# the render thread, which adopts it at the start of a follower frame
# (_adopt_follower_scroll_image). One append / one popleft, each
# atomic, so the render thread never draws from a half-swapped
# cached_image / cached_array / total_scroll_width.
self._follower_incoming_image: deque = deque(maxlen=1)
self._follower_deadline: Optional[float] = None self._follower_deadline: Optional[float] = None
# Leader: time.time() of the last follower frame sent. # Leader: time.time() of the last follower frame sent.
self._last_follower_send = 0.0 self._last_follower_send = 0.0
@@ -238,10 +216,6 @@ class DisplayController:
# the main run loop reconciles (loads/unloads) on its own thread so # the main run loop reconciles (loads/unloads) on its own thread so
# mutating available_modes never races with rendering. # mutating available_modes never races with rendering.
self._pending_plugin_reconcile = False self._pending_plugin_reconcile = False
# Set by the config-watcher thread when Vegas is switched on but no
# coordinator exists (Vegas was off at startup). The render thread
# creates it in _is_vegas_mode_active(), never the watcher thread.
self._pending_vegas_init = False
# Monotonic stamp of the last mailbox disk read; see # Monotonic stamp of the last mailbox disk read; see
# _poll_on_demand_requests. None means "never polled", so the first # _poll_on_demand_requests. None means "never polled", so the first
# call always goes through. # call always goes through.
@@ -301,10 +275,7 @@ class DisplayController:
if os.path.isabs(plugins_dir_name): if os.path.isabs(plugins_dir_name):
plugins_dir = plugins_dir_name plugins_dir = plugins_dir_name
else: else:
# If relative, resolve against the current working directory. # If relative, resolve relative to the project root (LEDMatrix directory)
# That is the project root only because ledmatrix.service
# sets WorkingDirectory to it; run from anywhere else, a
# relative path resolves against wherever that is.
project_root = os.getcwd() project_root = os.getcwd()
plugins_dir = os.path.join(project_root, plugins_dir_name) plugins_dir = os.path.join(project_root, plugins_dir_name)
@@ -335,19 +306,13 @@ class DisplayController:
except Exception as e: except Exception as e:
logger.warning("Could not enable plugin health/resource monitoring: %s", e) logger.warning("Could not enable plugin health/resource monitoring: %s", e)
# Discover plugins. Before the plugin checks so they can reuse the
# list: each discover_plugins() call rescans the plugins directory
# and logs every plugin again.
discovered_plugins = self.plugin_manager.discover_plugins()
logger.info("Discovered %d plugin(s)", len(discovered_plugins))
# Only the plugin checks: validate_all() above has run the rest, # Only the plugin checks: validate_all() above has run the rest,
# and running it again logged every config warning twice. # and running it again logged every config warning twice.
try: try:
from src.startup_validator import StartupValidator from src.startup_validator import StartupValidator
validator = StartupValidator(self.config_manager, self.plugin_manager, validator = StartupValidator(self.config_manager, self.plugin_manager,
cache_manager=self.cache_manager) cache_manager=self.cache_manager)
validator._validate_plugins(discovered_plugins=discovered_plugins) validator._validate_plugins()
for warning in validator.warnings: for warning in validator.warnings:
logger.warning("Plugin validation warning: %s", warning) logger.warning("Plugin validation warning: %s", warning)
if validator.errors: if validator.errors:
@@ -356,6 +321,10 @@ class DisplayController:
except Exception as e: except Exception as e:
logger.warning("Plugin validation could not be completed: %s", e) logger.warning("Plugin validation could not be completed: %s", e)
# Discover plugins
discovered_plugins = self.plugin_manager.discover_plugins()
logger.info("Discovered %d plugin(s)", len(discovered_plugins))
# Check for on-demand plugin filter from cache # Check for on-demand plugin filter from cache
on_demand_config = self.cache_manager.get('display_on_demand_config', max_age=3600) on_demand_config = self.cache_manager.get('display_on_demand_config', max_age=3600)
enabled_plugins = self._select_startup_plugins(discovered_plugins, on_demand_config) enabled_plugins = self._select_startup_plugins(discovered_plugins, on_demand_config)
@@ -453,9 +422,8 @@ class DisplayController:
# Display rotation state # Display rotation state
self.current_mode_index = 0 self.current_mode_index = 0
self.current_display_mode = None self.current_display_mode = None
# Last mode written to the display_current_state cache key, and when. # Last mode written to the display_current_state cache key.
self._last_published_mode: Optional[str] = None self._last_published_mode: Optional[str] = None
self._last_published_at = 0.0
self.global_dynamic_config = ( self.global_dynamic_config = (
self.config.get("display", {}).get("dynamic_duration", {}) or {} self.config.get("display", {}).get("dynamic_duration", {}) or {}
) )
@@ -566,15 +534,21 @@ class DisplayController:
# temporarily replace the leader's correct one. # temporarily replace the leader's correct one.
# When the leader sends its scroll image (TCP), update our # When the leader sends its scroll image (TCP), update our
# cached_array so both Pis have pixel-identical images. This runs # cached_array so both Pis have pixel-identical images.
# on the sync TCP thread, so it only converts and queues the
# image; the render thread swaps it in between frames.
import numpy as _np import numpy as _np
def _on_leader_scroll_image(image): def _on_leader_scroll_image(image):
vc = self.vegas_coordinator vc = self.vegas_coordinator
if vc and vc.render_pipeline: if vc and vc.render_pipeline:
rp = vc.render_pipeline
arr = _np.asarray(image.convert("RGB"), dtype=_np.uint8) arr = _np.asarray(image.convert("RGB"), dtype=_np.uint8)
self._follower_incoming_image.append((image, arr)) rp.scroll_helper.cached_image = image
rp.scroll_helper.cached_array = arr
rp.scroll_helper.total_scroll_width = image.width
self._follower_pending_new_image = False
logger.info(
"Sync: follower adopted leader scroll image %dx%d",
image.width, image.height,
)
self.sync_manager.set_on_scroll_image(_on_leader_scroll_image) self.sync_manager.set_on_scroll_image(_on_leader_scroll_image)
if self.sync_manager.role == SyncRole.LEADER: if self.sync_manager.role == SyncRole.LEADER:
@@ -597,32 +571,8 @@ class DisplayController:
logger.error("Failed to initialize Vegas mode: %s", e, exc_info=True) logger.error("Failed to initialize Vegas mode: %s", e, exc_info=True)
self.vegas_coordinator = None self.vegas_coordinator = None
def _adopt_follower_scroll_image(self, rp) -> None:
"""Swap in the leader's latest scroll image, on the render thread.
The sync TCP thread used to set cached_image, cached_array and
total_scroll_width one after another while this thread read them, so
a frame could slice the new array with the old width. It now queues
the image and this applies it between frames.
"""
try:
image, arr = self._follower_incoming_image.popleft()
except IndexError:
return
if rp is None:
return
rp.scroll_helper.cached_image = image
rp.scroll_helper.cached_array = arr
rp.scroll_helper.total_scroll_width = image.width
self._follower_pending_new_image = False
logger.info(
"Sync: follower adopted leader scroll image %dx%d",
image.width, image.height,
)
def _is_vegas_mode_active(self) -> bool: def _is_vegas_mode_active(self) -> bool:
"""Check if Vegas mode should be running.""" """Check if Vegas mode should be running."""
self._apply_pending_vegas_init()
if not self.vegas_coordinator: if not self.vegas_coordinator:
return False return False
# A stopped coordinator never reaches run_frame(), where queued config # A stopped coordinator never reaches run_frame(), where queued config
@@ -634,19 +584,6 @@ class DisplayController:
return False # On-demand takes priority return False # On-demand takes priority
return True return True
def _apply_pending_vegas_init(self) -> None:
"""Create the Vegas coordinator if Vegas was switched on after startup.
Render thread only: the config watcher just sets _pending_vegas_init.
Called from _is_vegas_mode_active() and from the main loop before the
sync-follower branch, which skips _is_vegas_mode_active() while a
follower is connected but still needs the coordinator to show the
leader's scroll image.
"""
if not self.vegas_coordinator and self._pending_vegas_init:
self._pending_vegas_init = False
self._initialize_vegas_mode()
def _check_vegas_interrupt(self) -> bool: def _check_vegas_interrupt(self) -> bool:
""" """
Check if Vegas should yield control for higher priority events. Check if Vegas should yield control for higher priority events.
@@ -840,15 +777,7 @@ class DisplayController:
if use_per_day: if use_per_day:
day_config = days_config[current_day] day_config = days_config[current_day]
if not day_config.get('enabled', True): if not day_config.get('enabled', True):
# Past the minute gate, so the cache must say the same thing:
# returning here without it left the previous minute's dim
# value to be served for the rest of this one, and the
# brightness flipped between dim and normal every minute.
if self._was_dimmed:
logger.info(f"Dim schedule deactivated: brightness restored to {normal_brightness}%")
self.is_dimmed = False self.is_dimmed = False
self._was_dimmed = False
self._cached_target_brightness = normal_brightness # persist for minute-gate
return normal_brightness return normal_brightness
start_time_str = day_config.get('start_time', '20:00') start_time_str = day_config.get('start_time', '20:00')
end_time_str = day_config.get('end_time', '07:00') end_time_str = day_config.get('end_time', '07:00')
@@ -1122,38 +1051,6 @@ class DisplayController:
or self.on_demand_active != on_demand): or self.on_demand_active != on_demand):
break break
def _note_empty_pass(self) -> None:
"""Record a pass whose mode had nothing to show; pause once a whole
rotation has been empty.
A mode with no content rotates to the next one at once, with no dwell.
When every mode is empty -- say, only a sports plugin enabled in its
off-season -- the loop went round with no sleep at all: 100% of a core,
a plugin-executor thread per pass and several log lines each time,
indefinitely. After one full rotation of empty passes, each further
one pauses EMPTY_ROTATION_PAUSE seconds. Live content is still picked
up within that second (the live-priority check runs at the top of
every pass), the pause services plugin updates, and it returns early
on an on-demand request or a schedule change. The streak resets as
soon as any mode shows something, and when the rotation itself
changes (on-demand starting or stopping, a plugin enabled or
disabled): a streak counted in one rotation says nothing about the
modes of another, which haven't been tried yet.
"""
modes = self.on_demand_modes if self.on_demand_active else self.available_modes
rotation_key = (bool(self.on_demand_active), tuple(modes))
if rotation_key != self._empty_pass_rotation:
self._empty_pass_rotation = rotation_key
self._empty_pass_streak = 0
self._empty_pass_streak += 1
rotation = max(1, len(modes))
if self._empty_pass_streak < rotation:
return
if self._empty_pass_streak == rotation:
logger.info("No mode has anything to show; checking one mode every %.0fs "
"until one does", self.EMPTY_ROTATION_PAUSE)
self._sleep_with_plugin_updates(self.EMPTY_ROTATION_PAUSE)
def _get_display_duration(self, mode_key): def _get_display_duration(self, mode_key):
"""Seconds to show a mode: the Rotation & Durations page's value for it """Seconds to show a mode: the Rotation & Durations page's value for it
(display.display_durations), else the plugin's own duration. (display.display_durations), else the plugin's own duration.
@@ -1293,22 +1190,13 @@ class DisplayController:
} }
self.cache_manager.set('display_current_state', state) self.cache_manager.set('display_current_state', state)
self._last_published_mode = self.current_display_mode self._last_published_mode = self.current_display_mode
self._last_published_at = time.monotonic()
except (OSError, RuntimeError, ValueError, TypeError) as err: except (OSError, RuntimeError, ValueError, TypeError) as err:
logger.error("Failed to publish current display state: %s", err, exc_info=True) logger.error("Failed to publish current display state: %s", err, exc_info=True)
def _publish_current_mode_state_if_changed(self) -> None: def _publish_current_mode_state_if_changed(self) -> None:
"""Publish the current mode state when it changed, or when the last """Publish current mode state only when it actually changed, to avoid
publish is older than CURRENT_STATE_REFRESH_SECONDS. writing to the shared cache on every render tick."""
if self.current_display_mode != self._last_published_mode:
The web UI reads this key with a max_age (api_v3/display.py), so a mode
that stays on screen longer than that -- a live game under live
priority, a single enabled plugin -- has to be republished or the UI
reports it as unknown. Otherwise this writes only on a change, not on
every render tick.
"""
if (self.current_display_mode != self._last_published_mode
or time.monotonic() - self._last_published_at >= CURRENT_STATE_REFRESH_SECONDS):
self._publish_current_mode_state() self._publish_current_mode_state()
def _publish_on_demand_state(self) -> None: def _publish_on_demand_state(self) -> None:
@@ -1416,9 +1304,6 @@ class DisplayController:
self._check_on_demand_expiration() self._check_on_demand_expiration()
self._evaluate_schedule() self._evaluate_schedule()
self._apply_brightness_target(repaint=True) self._apply_brightness_target(repaint=True)
# A Vegas iteration or a long screen keeps the main loop away for
# minutes; keep the web UI's "Now showing" from going stale.
self._publish_current_mode_state_if_changed()
except Exception: # pylint: disable=broad-except except Exception: # pylint: disable=broad-except
# Called from inside Vegas and the render loops; a failure here # Called from inside Vegas and the render loops; a failure here
# must not take the display loop down with it. # must not take the display loop down with it.
@@ -1691,10 +1576,7 @@ class DisplayController:
if plugin_instance.has_live_content(): if plugin_instance.has_live_content():
live_with_content.append(live_mode) live_with_content.append(live_mode)
except Exception: except Exception:
# Treated as no live content; logged so a plugin whose pass
# check always raises is findable.
logger.debug("has_live_content() failed for %s", live_mode,
exc_info=True)
# Build mode list: live modes with content first, then other modes, then live modes without content # Build mode list: live modes with content first, then other modes, then live modes without content
if live_with_content: if live_with_content:
@@ -1792,16 +1674,10 @@ class DisplayController:
pinned = bool(request.get('pinned', False)) pinned = bool(request.get('pinned', False))
now = time.time() now = time.time()
# Only a request that starts a session records where rotation was. if self.available_modes:
# A request made while on-demand is already showing would otherwise self.rotation_resume_index = self.current_mode_index
# save the previous request's mode (current_mode_index points at it else:
# by now), and clearing would resume there instead of where the self.rotation_resume_index = None
# normal rotation was interrupted.
if not self.on_demand_active:
if self.available_modes:
self.rotation_resume_index = self.current_mode_index
else:
self.rotation_resume_index = None
if resolved_mode in self.available_modes: if resolved_mode in self.available_modes:
self.current_mode_index = self.available_modes.index(resolved_mode) self.current_mode_index = self.available_modes.index(resolved_mode)
@@ -1931,9 +1807,7 @@ class DisplayController:
if bg_service and hasattr(bg_service, 'log_memory_stats'): if bg_service and hasattr(bg_service, 'log_memory_stats'):
bg_service.log_memory_stats() bg_service.log_memory_stats()
except Exception: except Exception:
# Background service may not be initialized pass # Background service may not be initialized
logger.debug("Background service memory stats unavailable",
exc_info=True)
# Log deferred updates stats # Log deferred updates stats
if hasattr(self.display_manager, '_scrolling_state'): if hasattr(self.display_manager, '_scrolling_state'):
@@ -2113,7 +1987,6 @@ class DisplayController:
continue continue
self._publish_current_mode_state_if_changed() self._publish_current_mode_state_if_changed()
self._apply_pending_vegas_init()
logger.debug("Display active, processing mode: %s", self.current_display_mode) logger.debug("Display active, processing mode: %s", self.current_display_mode)
# Plugins update on their own schedules - no forced sync updates needed # Plugins update on their own schedules - no forced sync updates needed
@@ -2135,7 +2008,6 @@ class DisplayController:
vc = self.vegas_coordinator vc = self.vegas_coordinator
rp = vc.render_pipeline if (vc and vc.render_pipeline) else None rp = vc.render_pipeline if (vc and vc.render_pipeline) else None
width = self.display_manager.width width = self.display_manager.width
self._adopt_follower_scroll_image(rp)
local_x = self._follower_local_x local_x = self._follower_local_x
if local_x is None: if local_x is None:
@@ -2419,16 +2291,6 @@ class DisplayController:
# If display() returned False, skip to next mode immediately # If display() returned False, skip to next mode immediately
if not display_result: if not display_result:
was_on_demand = self.on_demand_active
self._note_empty_pass()
# The pause returns early when an on-demand request, its
# end, or the schedule decides what comes next. Rotating
# past this empty mode now would skip that: an on-demand
# start would advance past the mode just requested.
if (self.current_display_mode != active_mode
or self.on_demand_active != was_on_demand
or not self.is_display_active):
continue
if self.on_demand_active: if self.on_demand_active:
logger.info("No content for on-demand mode %s, skipping to next mode", active_mode) logger.info("No content for on-demand mode %s, skipping to next mode", active_mode)
if not self.on_demand_modes: if not self.on_demand_modes:
@@ -2477,7 +2339,6 @@ class DisplayController:
# If no exception (just no content), fall through to normal rotation logic # If no exception (just no content), fall through to normal rotation logic
# This allows trying other modes (recent, upcoming) from the same plugin # This allows trying other modes (recent, upcoming) from the same plugin
else: else:
self._empty_pass_streak = 0
# Get base duration for current mode # Get base duration for current mode
base_duration = self._get_display_duration(active_mode) base_duration = self._get_display_duration(active_mode)
dynamic_enabled = self._plugin_supports_dynamic(manager_to_display) dynamic_enabled = self._plugin_supports_dynamic(manager_to_display)
@@ -2906,8 +2767,7 @@ class DisplayController:
try: try:
self.wifi_status_file.unlink() self.wifi_status_file.unlink()
except Exception: except Exception:
logger.debug("Could not remove WiFi status file %s", pass
self.wifi_status_file, exc_info=True)
return None return None
# Validate required fields # Validate required fields
@@ -2940,8 +2800,7 @@ class DisplayController:
try: try:
self.wifi_status_file.unlink() self.wifi_status_file.unlink()
except Exception: except Exception:
logger.debug("Could not remove WiFi status file %s", pass
self.wifi_status_file, exc_info=True)
return None return None
# Message is valid and not expired — cache for the throttle window # Message is valid and not expired — cache for the throttle window
@@ -3348,14 +3207,10 @@ class DisplayController:
# new one only when it changed: applying it rebuilds the strip. # new one only when it changed: applying it rebuilds the strip.
# (getattr: this can fire before __init__ creates the coordinator.) # (getattr: this can fire before __init__ creates the coordinator.)
vegas = getattr(self, 'vegas_coordinator', None) vegas = getattr(self, 'vegas_coordinator', None)
new_vegas = (new_config.get('display', {}) or {}).get('vegas_scroll')
if vegas is not None and ( if vegas is not None and (
(old_config.get('display', {}) or {}).get('vegas_scroll') != new_vegas): (old_config.get('display', {}) or {}).get('vegas_scroll')
!= (new_config.get('display', {}) or {}).get('vegas_scroll')):
vegas.update_config(new_config) vegas.update_config(new_config)
elif vegas is None and (new_vegas or {}).get('enabled', False):
# No coordinator yet because Vegas was off at startup. Creating
# one here would race the render thread; flag it instead.
self._pending_vegas_init = True
# If a plugin was enabled/disabled, flag a reconcile for the main # If a plugin was enabled/disabled, flag a reconcile for the main
# loop to apply (loading/unloading off the watcher thread is unsafe). # loop to apply (loading/unloading off the watcher thread is unsafe).
if (self._enabled_set_changed(old_config, new_config) if (self._enabled_set_changed(old_config, new_config)
@@ -3414,13 +3269,6 @@ class DisplayController:
self.vegas_coordinator.cleanup() self.vegas_coordinator.cleanup()
except Exception as e: except Exception as e:
logger.warning("Error cleaning up Vegas mode: %s", e) logger.warning("Error cleaning up Vegas mode: %s", e)
# After Vegas, which sends through it. Stopping also withdraws the
# sync status file, which the web UI otherwise kept showing as live.
if getattr(self, 'sync_manager', None) is not None:
try:
self.sync_manager.stop()
except Exception as e:
logger.warning("Error stopping display sync: %s", e)
# Shutdown config service if it exists # Shutdown config service if it exists
if hasattr(self, 'config_service'): if hasattr(self, 'config_service'):
try: try:
@@ -3434,23 +3282,9 @@ class DisplayController:
self.display_manager.cleanup() self.display_manager.cleanup()
logger.info("Cleanup complete.") logger.info("Cleanup complete.")
def _raise_keyboard_interrupt(signum, frame):
"""SIGTERM handler: stop the way Ctrl-C does.
systemd stops ledmatrix.service with SIGTERM. Python's default action for
it ends the process at once, so run()'s ``finally: self.cleanup()`` --
which stops the update worker, tears down Vegas and clears the panel --
never ran. Raising KeyboardInterrupt sends SIGTERM down that same path.
"""
raise KeyboardInterrupt
def main(): def main():
"""Application entry point — create a DisplayController and run until interrupted.""" """Application entry point — create a DisplayController and run until interrupted."""
controller = DisplayController() controller = DisplayController()
# Installed after construction: a SIGTERM while plugins are still loading
# keeps the default immediate exit rather than waiting for the loads.
signal.signal(signal.SIGTERM, _raise_keyboard_interrupt)
controller.run() controller.run()
if __name__ == "__main__": if __name__ == "__main__":
-1
View File
@@ -132,7 +132,6 @@ def apply_pixel_mappers(width: int, height: int, mapper_config: str,
``multiplexing`` isn't modelled: its mappers give back the configured ``multiplexing`` isn't modelled: its mappers give back the configured
size for the panel sizes they are made for. size for the panel sizes they are made for.
""" """
param: Optional[str]
for entry in (mapper_config or '').split(';'): for entry in (mapper_config or '').split(';'):
name, colon, param = entry.partition(':') name, colon, param = entry.partition(':')
name = name.lower() name = name.lower()
+61 -260
View File
@@ -20,10 +20,7 @@ Key responsibilities
Singleton: only one ``DisplayManager`` instance exists per process. The Singleton: only one ``DisplayManager`` instance exists per process. The
first call to ``DisplayManager(config)`` creates it; subsequent calls return first call to ``DisplayManager(config)`` creates it; subsequent calls return
the same object, but ``__init__`` runs again on it each time, so it is the same object.
re-initialised (matrix included) with the new arguments rather than handed
back as it was. Construct it once and pass that instance around;
:meth:`DisplayManager.cleanup` clears the singleton.
""" """
import json import json
@@ -44,23 +41,18 @@ from src.display_geometry import (
DEFAULT_CHAIN_LENGTH, DEFAULT_COLS, DEFAULT_PARALLEL, DEFAULT_ROWS, DEFAULT_CHAIN_LENGTH, DEFAULT_COLS, DEFAULT_PARALLEL, DEFAULT_ROWS,
compose_pixel_mapper_config, physical_size, resolve_double_sided, compose_pixel_mapper_config, physical_size, resolve_double_sided,
) )
from src.matrix_support import ( from src.matrix_support import MatrixSettingsRefused, library_refusals, refusal_message
DEFAULT_REFRESH_LIMIT_HZ, MatrixSettingsRefused, library_refusals, refusal_message,
)
from src.pi5_matrix_support import is_raspberry_pi_5 from src.pi5_matrix_support import is_raspberry_pi_5
import threading import threading
import time import time
from collections import OrderedDict, deque from collections import OrderedDict, deque
from typing import Dict, Any, List, Optional, Tuple, TYPE_CHECKING from typing import Dict, Any, List, Optional, Tuple
import math import math
import zlib import zlib
import freetype import freetype
from src.common import snapshot_policy from src.common import snapshot_policy
from src.common.frame_timing import FrameTimingRecorder from src.common.frame_timing import FrameTimingRecorder
if TYPE_CHECKING:
from src.common.render_gate import RenderGate
from src.deprecation import deprecated from src.deprecation import deprecated
from src.logging_config import get_logger from src.logging_config import get_logger
from src.common.permission_utils import ( from src.common.permission_utils import (
@@ -77,10 +69,6 @@ logger = get_logger(__name__)
#: and therefore get_font_height() -- report anything but 0. #: and therefore get_font_height() -- report anything but 0.
_CALENDAR_FONT_PX = 7 _CALENDAR_FONT_PX = 7
#: Seconds between repeats of update_display()'s error log. It runs every
#: frame, so a fault that persists would otherwise log ~100 lines a second.
_UPDATE_ERROR_LOG_INTERVAL = 60.0
def _bdf_native_size(face) -> int: def _bdf_native_size(face) -> int:
"""The pixel height a BDF Face declares, or 0 if it does not say. """The pixel height a BDF Face declares, or 0 if it does not say.
@@ -145,84 +133,6 @@ class _LogicalMatrix:
setattr(object.__getattribute__(self, "_matrix"), name, value) setattr(object.__getattribute__(self, "_matrix"), name, value)
class _OffscreenMatrix(_LogicalMatrix):
"""``display_manager.matrix`` as a thread drawing off-screen sees it.
Reports the surface's size, so plugins that lay out from ``matrix.width``
follow it, and swallows every write that would reach the hardware. Nothing
drawn off-screen may touch the panel the render loop is driving. Method
names mirror the rgbmatrix API they stand in for.
"""
# pylint: disable=invalid-name
__slots__ = ()
def SetImage(self, *_args: Any, **_kwargs: Any) -> None:
"""Inert: off-screen drawing never reaches the panel."""
def SetPixel(self, *_args: Any, **_kwargs: Any) -> None:
"""Inert: off-screen drawing never reaches the panel."""
def Clear(self) -> None:
"""Inert: off-screen drawing never reaches the panel."""
def Fill(self, *_args: Any, **_kwargs: Any) -> None:
"""Inert: off-screen drawing never reaches the panel."""
def SwapOnVSync(self, canvas: Any, *_args: Any, **_kwargs: Any) -> Any:
"""Inert: hands the canvas straight back without waiting on the panel."""
return canvas
def __setattr__(self, name: str, value: Any) -> None:
"""Inert: brightness and other writes stay off the real matrix."""
class _OffscreenSurface:
"""One thread's private canvas while it renders off-screen.
See :meth:`DisplayManager.offscreen`.
"""
__slots__ = ("draw", "image", "matrix")
def __init__(self, width: int, height: int, real_matrix: Any) -> None:
self.image = Image.new('RGB', (width, height))
self.draw = ImageDraw.Draw(self.image)
# 1-bit text: the panel has no partial brightness, so AA only smears glyphs.
self.draw.fontmode = "1"
self.matrix = (_OffscreenMatrix(real_matrix, width, height)
if real_matrix is not None else None)
def _per_thread_canvas_attr(name: str) -> property:
"""A DisplayManager attribute that resolves per thread.
A thread inside :meth:`DisplayManager.offscreen` reads and writes its own
surface's ``name``; every other thread reads and writes the shared value,
exactly as when this was a plain attribute. Existing ``self.image = ...``
assignments therefore keep working and become thread-correct as they are.
"""
shared = "_shared_" + name
def fget(self: "DisplayManager") -> Any:
surface = self._current_surface() # pylint: disable=protected-access
if surface is not None:
return getattr(surface, name)
try:
return self.__dict__[shared]
except KeyError:
raise AttributeError(name) from None
def fset(self: "DisplayManager", value: Any) -> None:
surface = self._current_surface() # pylint: disable=protected-access
if surface is not None:
setattr(surface, name, value)
else:
self.__dict__[shared] = value
return property(fget, fset, doc=f"The plugin-facing ``{name}``, per thread.")
class DisplayManager: class DisplayManager:
""" """
@@ -245,21 +155,11 @@ class DisplayManager:
_instance = None _instance = None
# update_display()'s error-log throttle. Class defaults so instances built
# without __init__ (tests, doubles) have them too.
_update_error_logged_at: Optional[float] = None
_update_errors_suppressed = 0
def __new__(cls, *args, **kwargs): def __new__(cls, *args, **kwargs):
if cls._instance is None: if cls._instance is None:
cls._instance = super(DisplayManager, cls).__new__(cls) cls._instance = super(DisplayManager, cls).__new__(cls)
return cls._instance return cls._instance
# The plugin-facing canvas. Per thread: see offscreen().
image = _per_thread_canvas_attr("image")
draw = _per_thread_canvas_attr("draw")
matrix = _per_thread_canvas_attr("matrix")
def __init__(self, config: Dict[str, Any] = None, force_fallback: bool = False, suppress_test_pattern: bool = False): def __init__(self, config: Dict[str, Any] = None, force_fallback: bool = False, suppress_test_pattern: bool = False):
start_time = time.time() start_time = time.time()
self.config = config or {} self.config = config or {}
@@ -273,9 +173,6 @@ class DisplayManager:
# suppress the render loop's own frame pushes for the duration, freezing # suppress the render loop's own frame pushes for the duration, freezing
# the panel exactly when the point was to avoid a freeze. # the panel exactly when the point was to avoid a freeze.
self._capture_state = threading.local() self._capture_state = threading.local()
# Per-thread off-screen surface. While a thread is inside offscreen(),
# image, draw and matrix resolve to its own canvas; see offscreen().
self._surface_state = threading.local()
# Double-sided mode state (resolved in _setup_matrix). When disabled, # Double-sided mode state (resolved in _setup_matrix). When disabled,
# the logical image is blitted to the matrix unchanged. # the logical image is blitted to the matrix unchanged.
self._double_sided = None # dict {copies, axis, logical_width, logical_height} or None self._double_sided = None # dict {copies, axis, logical_width, logical_height} or None
@@ -341,11 +238,6 @@ class DisplayManager:
# See src/common/scroll_config.py and scripts/scroll_speeds.py. # See src/common/scroll_config.py and scripts/scroll_speeds.py.
self._frame_hold = 1 self._frame_hold = 1
# A src.common.render_gate.RenderGate while Vegas runs with
# vegas_scroll.prefetch_gate on: opened around each swap so the
# prefetch thread only runs Python while this thread waits on vsync.
self.render_gate: Optional['RenderGate'] = None
# Timing of every presented frame, whoever drew it, for # Timing of every presented frame, whoever drew it, for
# scripts/frame_soak.py. See src/common/frame_timing.py. # scripts/frame_soak.py. See src/common/frame_timing.py.
self.frame_timing = FrameTimingRecorder(info=self._frame_timing_info()) self.frame_timing = FrameTimingRecorder(info=self._frame_timing_info())
@@ -359,13 +251,6 @@ class DisplayManager:
'max_deferred_updates': 50, # Limit queue size to prevent memory issues 'max_deferred_updates': 50, # Limit queue size to prevent memory issues
'deferred_update_ttl': 300.0 # 5 minutes TTL for deferred updates 'deferred_update_ttl': 300.0 # 5 minutes TTL for deferred updates
} }
# Guards _scrolling_state['deferred_updates']. defer_update() is called
# from plugin update() on the update worker thread while
# process_deferred_updates() runs on the render thread, and both
# rebuild the list (TTL filter, [n:] slice) and assign it back -- an
# append landing between one side's read and its assignment was lost.
# Never held while a queued callable runs: those may defer again.
self._deferred_lock = threading.Lock()
self._setup_matrix() self._setup_matrix()
logger.info("Matrix setup completed in %.3f seconds", time.time() - start_time) logger.info("Matrix setup completed in %.3f seconds", time.time() - start_time)
@@ -759,10 +644,7 @@ class DisplayManager:
@property @property
def _capture_mode_active(self) -> bool: def _capture_mode_active(self) -> bool:
"""True while the calling thread is capturing content off-screen.""" """True while the calling thread is capturing content off-screen."""
# Read like _current_surface(): a DisplayManager built without return getattr(self._capture_state, 'active', False)
# __init__ (tests do) has no per-thread state, and captures nothing.
state = self.__dict__.get('_capture_state')
return getattr(state, 'active', False) if state is not None else False
@_capture_mode_active.setter @_capture_mode_active.setter
def _capture_mode_active(self, value: bool) -> None: def _capture_mode_active(self, value: bool) -> None:
@@ -778,67 +660,11 @@ class DisplayManager:
Entering this context prevents those writes without affecting the PIL Entering this context prevents those writes without affecting the PIL
image buffer, which the adapter reads to extract content. image buffer, which the adapter reads to extract content.
""" """
# Restore rather than clear: capture_mode() inside offscreen() must not
# switch suppression off for the rest of the off-screen block.
was_active = self._capture_mode_active
self._capture_mode_active = True self._capture_mode_active = True
try: try:
yield yield
finally: finally:
self._capture_mode_active = was_active self._capture_mode_active = False
def _current_surface(self) -> Optional[_OffscreenSurface]:
"""The calling thread's off-screen surface, or None."""
state = self.__dict__.get('_surface_state')
return getattr(state, 'surface', None) if state is not None else None
def _writes_suppressed(self) -> bool:
"""True when the calling thread must not touch the panel or its pacing."""
return self._capture_mode_active or self._current_surface() is not None
@contextmanager
def offscreen(self, width: Optional[int] = None, height: Optional[int] = None):
"""Give the calling thread its own canvas to draw on.
Inside the block, for the calling thread only, ``image``, ``draw`` and
``matrix`` (and so ``width``/``height``) are a fresh black canvas of the
requested size, and nothing reaches the hardware: ``update_display()``
and the hardware half of ``clear()`` are skipped, and
``set_scrolling_state()``/``set_frame_hold()`` cannot re-pace the live
scroll. Every other thread, the render loop above all, keeps seeing the
real canvas.
That is what lets Vegas mode render a plugin on its background prefetch
thread. The shared canvas used to be the only one, so any plugin that
drew on it (display capture, scroll-content generation, narrowed
rendering) had to be fetched on the render thread, stalling the scroll
for 40-600ms each. See docs/OFFSCREEN_RENDERING.md.
Blocks nest; each restores the one outside it, also on an exception.
Args:
width: Width of the surface, clamped to the size this thread sees
now. Defaults to that size.
height: Height, likewise.
Yields:
The surface. ``surface.image`` is what the plugin drew.
"""
state = self.__dict__.get('_surface_state')
if state is None:
state = self._surface_state = threading.local()
current_w, current_h = self.width, self.height
target_w = max(1, min(int(width), current_w)) if width else current_w
target_h = max(1, min(int(height), current_h)) if height else current_h
surface = _OffscreenSurface(target_w, target_h, self.matrix)
previous = getattr(state, 'surface', None)
state.surface = surface
try:
yield surface
finally:
state.surface = previous
@contextmanager @contextmanager
def render_size(self, width: int, height: Optional[int] = None): def render_size(self, width: int, height: Optional[int] = None):
@@ -858,14 +684,18 @@ class DisplayManager:
indirection that double-sided mode relies on, so plugins see a indirection that double-sided mode relies on, so plugins see a
consistent size from every accessor. consistent size from every accessor.
Built on :meth:`offscreen`, so the narrower canvas belongs to the Only meaningful inside :meth:`capture_mode` — this swaps the shared
calling thread alone; the render loop keeps drawing on the real one. image buffer, so the render loop must not be writing to it concurrently.
Args: Args:
width: Logical width to report, clamped to at least 1 and to the width: Logical width to report, clamped to at least 1 and to the
real panel width (a larger canvas would overflow the hardware). real panel width (a larger canvas would overflow the hardware).
height: Logical height, defaulting to the current height. height: Logical height, defaulting to the current height.
""" """
real_matrix = self.matrix
prev_image = getattr(self, 'image', None)
prev_draw = getattr(self, 'draw', None)
current_w = self.width current_w = self.width
current_h = self.height current_h = self.height
target_w = max(1, min(int(width), current_w)) target_w = max(1, min(int(width), current_w))
@@ -876,8 +706,19 @@ class DisplayManager:
yield yield
return return
with self.offscreen(target_w, target_h): try:
if real_matrix is not None:
self.matrix = _LogicalMatrix(real_matrix, target_w, target_h)
# With no hardware, the width/height properties fall through to
# self.image, so swapping the buffer below is enough on its own.
self._new_canvas(target_w, target_h)
yield yield
finally:
self.matrix = real_matrix
if prev_image is not None:
self.image = prev_image
if prev_draw is not None:
self.draw = prev_draw
def _composite_double_sided(self): def _composite_double_sided(self):
"""Tile the logical screen across the full physical chain. """Tile the logical screen across the full physical chain.
@@ -922,12 +763,6 @@ class DisplayManager:
need to know about it. need to know about it.
""" """
try: try:
if self._writes_suppressed():
# This thread is drawing off-screen. Checked before the lock,
# so it never contends with the render loop's swap, and before
# the fallback branch, so captured content never reaches the
# web preview either.
return
with self._update_lock: with self._update_lock:
if self.matrix is None: if self.matrix is None:
# Fallback mode - no actual hardware to update # Fallback mode - no actual hardware to update
@@ -936,6 +771,9 @@ class DisplayManager:
self._write_snapshot_if_due() self._write_snapshot_if_due()
return return
if self._capture_mode_active:
return # Skip hardware write — content is being captured off-screen
digest = None digest = None
frame_checksum = None frame_checksum = None
if self._dirty_tracking_enabled: if self._dirty_tracking_enabled:
@@ -978,12 +816,7 @@ class DisplayManager:
# Swap buffers immediately. framerate_fraction holds the frame # Swap buffers immediately. framerate_fraction holds the frame
# for N refreshes; SwapOnVSync blocks for all of them, which is # for N refreshes; SwapOnVSync blocks for all of them, which is
# what paces the render loop to the chosen frame rate. # what paces the render loop to the chosen frame rate.
gate = self.render_gate
if gate is not None:
gate.before_swap(self._frame_hold)
self.matrix.SwapOnVSync(self.offscreen_canvas, self._frame_hold) self.matrix.SwapOnVSync(self.offscreen_canvas, self._frame_hold)
if gate is not None:
gate.after_swap(self._frame_hold)
presented_at = time.perf_counter() presented_at = time.perf_counter()
self.frame_timing.record( self.frame_timing.record(
blit_done - blit_started, presented_at - blit_done, blit_done - blit_started, presented_at - blit_done,
@@ -997,21 +830,7 @@ class DisplayManager:
# Write a snapshot for the web preview (throttled) # Write a snapshot for the web preview (throttled)
self._write_snapshot_if_due(frame_checksum) self._write_snapshot_if_due(frame_checksum)
except Exception as e: except Exception as e:
# Once with the traceback, then at most every logger.error(f"Error updating display: {e}")
# _UPDATE_ERROR_LOG_INTERVAL with a count of what was skipped.
now = time.monotonic()
last = self._update_error_logged_at
if last is None:
self._update_error_logged_at = now
logger.error("Error updating display: %s", e, exc_info=True)
elif now - last >= _UPDATE_ERROR_LOG_INTERVAL:
skipped = self._update_errors_suppressed
self._update_error_logged_at = now
self._update_errors_suppressed = 0
logger.error("Error updating display: %s (%d more since the "
"last report)", e, skipped)
else:
self._update_errors_suppressed += 1
def _setup_scan_order_compensation(self) -> None: def _setup_scan_order_compensation(self) -> None:
"""Work out which rows to show a refresh behind while scrolling. """Work out which rows to show a refresh behind while scrolling.
@@ -1069,7 +888,7 @@ class DisplayManager:
self._new_canvas(self.matrix.width, self.matrix.height) self._new_canvas(self.matrix.width, self.matrix.height)
if not self._writes_suppressed(): if not self._capture_mode_active:
# Clear both canvases and the underlying matrix to ensure no artifacts. # Clear both canvases and the underlying matrix to ensure no artifacts.
# Failures are non-fatal — the image buffer is already black above, so # Failures are non-fatal — the image buffer is already black above, so
# the next update_display() call will push clean content regardless. # the next update_display() call will push clean content regardless.
@@ -1582,7 +1401,7 @@ class DisplayManager:
options.panel_type = hardware_config.get('panel_type', '') options.panel_type = hardware_config.get('panel_type', '')
options.disable_hardware_pulsing = hardware_config.get('disable_hardware_pulsing', False) options.disable_hardware_pulsing = hardware_config.get('disable_hardware_pulsing', False)
options.show_refresh_rate = hardware_config.get('show_refresh_rate', False) options.show_refresh_rate = hardware_config.get('show_refresh_rate', False)
options.limit_refresh_rate_hz = hardware_config.get('limit_refresh_rate_hz', DEFAULT_REFRESH_LIMIT_HZ) options.limit_refresh_rate_hz = hardware_config.get('limit_refresh_rate_hz', 90)
options.gpio_slowdown = runtime_config.get('gpio_slowdown', 3) options.gpio_slowdown = runtime_config.get('gpio_slowdown', 3)
# Disable internal privilege dropping - we manage this via systemd or remain root # Disable internal privilege dropping - we manage this via systemd or remain root
@@ -1628,7 +1447,7 @@ class DisplayManager:
value = float(hardware.get('limit_refresh_rate_hz') or 0) value = float(hardware.get('limit_refresh_rate_hz') or 0)
except (TypeError, ValueError): except (TypeError, ValueError):
value = 0.0 value = 0.0
return value if value > 0 else float(DEFAULT_REFRESH_LIMIT_HZ) return value if value > 0 else 100.0
def _scrolling_now(self) -> bool: def _scrolling_now(self) -> bool:
"""Whether a scroll is running, without is_currently_scrolling()'s """Whether a scroll is running, without is_currently_scrolling()'s
@@ -1661,8 +1480,6 @@ class DisplayManager:
Reset to 1 whenever scrolling stops, so one plugin's pacing cannot Reset to 1 whenever scrolling stops, so one plugin's pacing cannot
leak into the next thing on screen. leak into the next thing on screen.
""" """
if self._writes_suppressed():
return # a plugin drawing off-screen cannot re-pace the live scroll
try: try:
value = int(refreshes) value = int(refreshes)
except (TypeError, ValueError): except (TypeError, ValueError):
@@ -1691,10 +1508,6 @@ class DisplayManager:
the lifetime exactly the scroll, and the default of 1 means any caller the lifetime exactly the scroll, and the default of 1 means any caller
that does not care gets a new frame every refresh. that does not care gets a new frame every refresh.
""" """
if self._writes_suppressed():
# A plugin captured for Vegas calls this from its own display();
# it must not change the live scroll's state or frame hold.
return
current_time = time.time() current_time = time.time()
# Scrolling callers set this every frame; log transitions only. # Scrolling callers set this every frame; log transitions only.
changed = self._scrolling_state['is_scrolling'] != is_scrolling changed = self._scrolling_state['is_scrolling'] != is_scrolling
@@ -1739,28 +1552,26 @@ class DisplayManager:
""" """
current_time = time.time() current_time = time.time()
with self._deferred_lock: # Clean up expired updates before adding new ones
# Clean up expired updates before adding new ones self._cleanup_expired_deferred_updates(current_time)
self._cleanup_expired_deferred_updates(current_time)
# Limit queue size to prevent memory issues
# Limit queue size to prevent memory issues if len(self._scrolling_state['deferred_updates']) >= self._scrolling_state['max_deferred_updates']:
if len(self._scrolling_state['deferred_updates']) >= self._scrolling_state['max_deferred_updates']: # Remove oldest update to make room
# Remove oldest update to make room self._scrolling_state['deferred_updates'].pop(0)
self._scrolling_state['deferred_updates'].pop(0) logger.debug("Removed oldest deferred update due to queue size limit")
logger.debug("Removed oldest deferred update due to queue size limit")
self._scrolling_state['deferred_updates'].append({
self._scrolling_state['deferred_updates'].append({ 'func': update_func,
'func': update_func, 'priority': priority,
'priority': priority, 'timestamp': current_time
'timestamp': current_time })
})
# Only sort if we have a reasonable number of updates to avoid excessive sorting
# Only sort if we have a reasonable number of updates to avoid excessive sorting if len(self._scrolling_state['deferred_updates']) <= 20:
if len(self._scrolling_state['deferred_updates']) <= 20: self._scrolling_state['deferred_updates'].sort(key=lambda x: x['priority'])
self._scrolling_state['deferred_updates'].sort(key=lambda x: x['priority'])
logger.debug(f"Deferred update added. Total deferred: {len(self._scrolling_state['deferred_updates'])}")
queued = len(self._scrolling_state['deferred_updates'])
logger.debug(f"Deferred update added. Total deferred: {queued}")
def process_deferred_updates(self): def process_deferred_updates(self):
"""Process any deferred updates if not currently scrolling.""" """Process any deferred updates if not currently scrolling."""
@@ -1768,26 +1579,21 @@ class DisplayManager:
# Always clean up expired updates, even if scrolling # Always clean up expired updates, even if scrolling
# This prevents memory leaks from accumulated expired updates # This prevents memory leaks from accumulated expired updates
with self._deferred_lock: self._cleanup_expired_deferred_updates(current_time)
self._cleanup_expired_deferred_updates(current_time)
if self.is_currently_scrolling(): if self.is_currently_scrolling():
return return
with self._deferred_lock: if not self._scrolling_state['deferred_updates']:
if not self._scrolling_state['deferred_updates']: return
return
# Process only a limited number of updates per call to avoid blocking
# Process only a limited number of updates per call to avoid blocking max_updates_per_call = min(5, len(self._scrolling_state['deferred_updates']))
max_updates_per_call = min(5, len(self._scrolling_state['deferred_updates'])) updates_to_process = self._scrolling_state['deferred_updates'][:max_updates_per_call]
updates_to_process = self._scrolling_state['deferred_updates'][:max_updates_per_call] self._scrolling_state['deferred_updates'] = self._scrolling_state['deferred_updates'][max_updates_per_call:]
self._scrolling_state['deferred_updates'] = self._scrolling_state['deferred_updates'][max_updates_per_call:]
queued = len(self._scrolling_state['deferred_updates'])
logger.debug(f"Processing {len(updates_to_process)} deferred updates (queue size: {queued})") logger.debug(f"Processing {len(updates_to_process)} deferred updates (queue size: {len(self._scrolling_state['deferred_updates'])})")
# The callables run outside the lock: they are plugin code of any
# length, and one that defers again would deadlock on it.
failed_updates = [] failed_updates = []
for update_info in updates_to_process: for update_info in updates_to_process:
try: try:
@@ -1806,15 +1612,10 @@ class DisplayManager:
# Re-add failed updates to the end of the queue (not the beginning) # Re-add failed updates to the end of the queue (not the beginning)
if failed_updates: if failed_updates:
with self._deferred_lock: self._scrolling_state['deferred_updates'].extend(failed_updates)
self._scrolling_state['deferred_updates'].extend(failed_updates)
def _cleanup_expired_deferred_updates(self, current_time: float): def _cleanup_expired_deferred_updates(self, current_time: float):
"""Remove expired deferred updates to prevent memory leaks. """Remove expired deferred updates to prevent memory leaks."""
Callers hold ``_deferred_lock``: this reads the list and assigns a
filtered copy back.
"""
ttl = self._scrolling_state['deferred_update_ttl'] ttl = self._scrolling_state['deferred_update_ttl']
initial_count = len(self._scrolling_state['deferred_updates']) initial_count = len(self._scrolling_state['deferred_updates'])
+7 -20
View File
@@ -13,14 +13,13 @@ Supported dynamic teams:
Usage: Usage:
resolver = DynamicTeamResolver() resolver = DynamicTeamResolver()
resolved_teams = resolver.resolve_teams(["UGA", "AP_TOP_25", "AUB"]) resolved_teams = resolver.resolve_teams(["UGA", "AP_TOP_25", "AUB"])
# Returns: ["UGA", "MICH", "OSU", ..., "AUB"] -- AP_TOP_25 expanded in # Returns: ["UGA", "UGA", "AUB", "MICH", "OSU", ...] (AP_TOP_25 teams)
# place, and UGA (also ranked) kept once, at its first position
""" """
import logging import logging
import time import time
import requests import requests
from typing import Any, Dict, List from typing import Dict, List
from src.common.api_helper import DEFAULT_HTTP_HEADERS from src.common.api_helper import DEFAULT_HTTP_HEADERS
@@ -35,17 +34,12 @@ class DynamicTeamResolver:
""" """
# Cache for rankings data # Cache for rankings data
_rankings_cache: Dict[str, int] = {} # team abbreviation -> AP rank _rankings_cache: Dict[str, List[str]] = {}
_cache_timestamp: float = 0 _cache_timestamp: float = 0
_cache_duration: int = 3600 # 1 hour cache _cache_duration: int = 3600 # 1 hour cache
# A failed or empty fetch is remembered briefly too: during an ESPN
# outage every resolve would otherwise wait out request_timeout (30s)
# again, on each scoreboard's update.
_failure_timestamp: float = 0
_failure_backoff: int = 300 # 5 minutes
# Supported dynamic team patterns # Supported dynamic team patterns
DYNAMIC_PATTERNS: Dict[str, Dict[str, Any]] = { DYNAMIC_PATTERNS = {
'AP_TOP_25': {'sport': 'ncaa_fb', 'limit': 25}, 'AP_TOP_25': {'sport': 'ncaa_fb', 'limit': 25},
'AP_TOP_10': {'sport': 'ncaa_fb', 'limit': 10}, 'AP_TOP_10': {'sport': 'ncaa_fb', 'limit': 10},
'AP_TOP_5': {'sport': 'ncaa_fb', 'limit': 5}, 'AP_TOP_5': {'sport': 'ncaa_fb', 'limit': 5},
@@ -76,8 +70,8 @@ class DynamicTeamResolver:
if team in self.DYNAMIC_PATTERNS: if team in self.DYNAMIC_PATTERNS:
# Resolve dynamic team # Resolve dynamic team
dynamic_teams = self._resolve_dynamic_team(team, sport) dynamic_teams = self._resolve_dynamic_team(team, sport)
# _resolve_dynamic_team already logs the result.
resolved_teams.extend(dynamic_teams) resolved_teams.extend(dynamic_teams)
self.logger.info(f"Resolved {team} to {len(dynamic_teams)} teams: {dynamic_teams[:5]}{'...' if len(dynamic_teams) > 5 else ''}")
elif self._is_potential_dynamic_team(team): elif self._is_potential_dynamic_team(team):
# Unknown dynamic team, skip it # Unknown dynamic team, skip it
self.logger.warning(f"Unknown dynamic team '{team}' - skipping") self.logger.warning(f"Unknown dynamic team '{team}' - skipping")
@@ -144,11 +138,7 @@ class DynamicTeamResolver:
if (self._rankings_cache and if (self._rankings_cache and
current_time - self._cache_timestamp < self._cache_duration): current_time - self._cache_timestamp < self._cache_duration):
return self._rankings_cache return self._rankings_cache
# A recent attempt failed: don't pay the request timeout again yet.
if current_time - self._failure_timestamp < self._failure_backoff:
return {}
try: try:
self.logger.info("Fetching fresh NCAA Football rankings from ESPN API") self.logger.info("Fetching fresh NCAA Football rankings from ESPN API")
rankings_url = "https://site.api.espn.com/apis/site/v2/sports/football/college-football/rankings" rankings_url = "https://site.api.espn.com/apis/site/v2/sports/football/college-football/rankings"
@@ -195,9 +185,7 @@ class DynamicTeamResolver:
except Exception as e: except Exception as e:
self.logger.error(f"Error fetching NCAA Football rankings: {e}") self.logger.error(f"Error fetching NCAA Football rankings: {e}")
# On the class, for the same reason as the rankings cache above.
DynamicTeamResolver._failure_timestamp = current_time
return {} return {}
def get_available_dynamic_teams(self) -> List[str]: def get_available_dynamic_teams(self) -> List[str]:
@@ -241,7 +229,6 @@ class DynamicTeamResolver:
shadow the shared cache for this instance.""" shadow the shared cache for this instance."""
DynamicTeamResolver._rankings_cache = {} DynamicTeamResolver._rankings_cache = {}
DynamicTeamResolver._cache_timestamp = 0 DynamicTeamResolver._cache_timestamp = 0
DynamicTeamResolver._failure_timestamp = 0
self.logger.info("Cleared dynamic team rankings cache") self.logger.info("Cleared dynamic team rankings cache")
+19 -39
View File
@@ -48,7 +48,6 @@ import json
import logging import logging
import math import math
import os import os
import threading
from collections import OrderedDict from collections import OrderedDict
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any, Dict, Optional, Tuple, Union from typing import Any, Dict, Optional, Tuple, Union
@@ -69,10 +68,8 @@ _FONTS_SUBDIR = os.path.join('assets', 'fonts')
_FALLBACK_FONT_NAME = 'PressStart2P-Regular.ttf' _FALLBACK_FONT_NAME = 'PressStart2P-Regular.ttf'
# (resolved absolute path, requested size) -> (font face, realised size). # (resolved absolute path, requested size) -> (font face, realised size).
# TTF only: a BDF ``freetype.Face`` must never be shared between threads # BDF faces are stateful in principle, but the core's own FontManager shares
# (FreeType does not allow it, and ``load_char`` rewrites the face's glyph # faces the same way.
# slot), and this cache is process-wide. BDF faces come from
# ``load_bdf_face`` every time, which already caches them per thread.
# #
# Bounded LRU rather than the unbounded dict this started as: the display # Bounded LRU rather than the unbounded dict this started as: the display
# process runs for weeks, and every config save can introduce a new # process runs for weeks, and every config save can introduce a new
@@ -81,28 +78,14 @@ _FALLBACK_FONT_NAME = 'PressStart2P-Regular.ttf'
# every other hot cache (display_manager, font_manager, adaptive_layout). # every other hot cache (display_manager, font_manager, adaptive_layout).
_FONT_CACHE_MAX = 256 _FONT_CACHE_MAX = 256
_font_cache: 'OrderedDict[Tuple[str, int], Tuple[Any, int]]' = OrderedDict() _font_cache: 'OrderedDict[Tuple[str, int], Tuple[Any, int]]' = OrderedDict()
# load_font is called from the display thread and from plugin update threads.
# A get() then move_to_end() pair on an unguarded OrderedDict raises KeyError
# when another thread evicts the key in between.
_font_cache_lock = threading.Lock()
def _cache_get(key: Tuple[str, int]) -> Optional[Tuple[Any, int]]:
"""The cached entry for ``key`` (marked most recently used), or None."""
with _font_cache_lock:
cached = _font_cache.get(key)
if cached is not None:
_font_cache.move_to_end(key)
return cached
def _cache_put(key: Tuple[str, int], value: Tuple[Any, int]) -> None: def _cache_put(key: Tuple[str, int], value: Tuple[Any, int]) -> None:
"""Insert, evicting the least recently used entry past the bound.""" """Insert, evicting the least recently used entry past the bound."""
with _font_cache_lock: _font_cache[key] = value
_font_cache[key] = value _font_cache.move_to_end(key)
_font_cache.move_to_end(key) while len(_font_cache) > _FONT_CACHE_MAX:
while len(_font_cache) > _FONT_CACHE_MAX: _font_cache.popitem(last=False)
_font_cache.popitem(last=False)
# Config keys a style element block carries, in schema/UI order. # Config keys a style element block carries, in schema/UI order.
_STYLE_KEYS = ('font', 'font_size', 'text_color', 'visible', 'align') _STYLE_KEYS = ('font', 'font_size', 'text_color', 'visible', 'align')
@@ -239,15 +222,14 @@ def _load_font_sized(font_name: str, size: int) -> Tuple[Any, int]:
logger.warning("Font file not found: %s, using fallback", font_name) logger.warning("Font file not found: %s, using fallback", font_name)
return _load_fallback_font(size) return _load_fallback_font(size)
is_bdf = path.lower().endswith('.bdf')
cache_key = (path, size) cache_key = (path, size)
if not is_bdf: cached = _font_cache.get(cache_key)
cached = _cache_get(cache_key) if cached is not None:
if cached is not None: _font_cache.move_to_end(cache_key)
return cached return cached
try: try:
if is_bdf: if path.lower().endswith('.bdf'):
font, effective = _load_bdf(path, size) font, effective = _load_bdf(path, size)
else: else:
font, effective = load_truetype(path, size), size font, effective = load_truetype(path, size), size
@@ -256,9 +238,7 @@ def _load_font_sized(font_name: str, size: int) -> Tuple[Any, int]:
path, size, e) path, size, e)
return _load_fallback_font(size) return _load_fallback_font(size)
# Not BDF: load_bdf_face caches those per thread (see _font_cache). _cache_put(cache_key, (font, effective))
if not is_bdf:
_cache_put(cache_key, (font, effective))
return font, effective return font, effective
@@ -267,8 +247,9 @@ def _load_fallback_font(size: int) -> Tuple[Any, int]:
path = resolve_font_path(_FALLBACK_FONT_NAME) path = resolve_font_path(_FALLBACK_FONT_NAME)
if path is not None: if path is not None:
cache_key = (path, size) cache_key = (path, size)
cached = _cache_get(cache_key) cached = _font_cache.get(cache_key)
if cached is not None: if cached is not None:
_font_cache.move_to_end(cache_key)
return cached return cached
try: try:
entry = (load_truetype(path, size), size) entry = (load_truetype(path, size), size)
@@ -859,7 +840,7 @@ def defaults_from_schema(schema: Dict[str, Any]) -> Dict[str, Any]:
defaults['align'] = align_spec['default'] defaults['align'] = align_spec['default']
scale_spec = spec.get('scale') scale_spec = spec.get('scale')
if isinstance(scale_spec, dict) and 'default' in scale_spec: if isinstance(scale_spec, dict) and 'default' in scale_spec:
layout.setdefault(element_key, {})['scale'] = scale_spec['default'] layout.setdefault(element_key, {})['scale'] = scale_spec['default']
elif scale_spec is True: elif scale_spec is True:
layout.setdefault(element_key, {})['scale'] = 1.0 layout.setdefault(element_key, {})['scale'] = 1.0
if defaults: if defaults:
@@ -875,7 +856,7 @@ def defaults_from_schema(schema: Dict[str, Any]) -> Dict[str, Any]:
continue continue
scale_prop = (block.get('properties') or {}).get('scale') scale_prop = (block.get('properties') or {}).get('scale')
if isinstance(scale_prop, dict) and 'default' in scale_prop: if isinstance(scale_prop, dict) and 'default' in scale_prop:
layout.setdefault(element_key, {})['scale'] = scale_prop['default'] layout.setdefault(element_key, {})['scale'] = scale_prop['default']
for element_key, block in properties.items(): for element_key, block in properties.items():
if element_key in ('layout', 'modes') or element_key in elements: if element_key in ('layout', 'modes') or element_key in elements:
continue continue
@@ -1499,12 +1480,11 @@ class ElementStyleResolver:
mode_config, 'align') mode_config, 'align')
# scale is geometry, so it lives with the offsets rather than in the # scale is geometry, so it lives with the offsets rather than in the
# element block -- a logo has a scale and no font. # element block -- a logo has a scale and no font.
# The default is looked up through the aliases too, like the value: layout_defaults = self._defaults.get('layout', {})
# an exact-key lookup missed a default filed under another name, so
# a configured value equal to it counted as a user choice.
scale = self._forced( scale = self._forced(
self._layout_element(self._customization(), element_key), self._layout_element(self._customization(), element_key),
_lookup_element(self._defaults.get('layout', {}), element_key), layout_defaults.get(element_key, {})
if isinstance(layout_defaults, dict) else {},
self._layout_element(self._mode_block(mode), element_key), self._layout_element(self._mode_block(mode), element_key),
'scale') 'scale')
+40 -3
View File
@@ -6,8 +6,7 @@ for the LEDMatrix system. Enables automatic bug detection by tracking
error frequency, patterns, and context. error frequency, patterns, and context.
This is a local-only implementation with no external dependencies. This is a local-only implementation with no external dependencies.
Errors are stored in memory; ErrorSnapshotPublisher shares a summary with the Errors are stored in memory with optional JSON export.
web process through the cache.
""" """
import math import math
@@ -19,6 +18,7 @@ import uuid
from collections import defaultdict from collections import defaultdict
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import datetime, timedelta from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, List, Optional, Any, Callable, Tuple from typing import Dict, List, Optional, Any, Callable, Tuple
import logging import logging
@@ -105,6 +105,7 @@ class ErrorAggregator:
max_records: int = 1000, max_records: int = 1000,
pattern_threshold: int = 5, pattern_threshold: int = 5,
pattern_window_minutes: int = 60, pattern_window_minutes: int = 60,
export_path: Optional[Path] = None
): ):
""" """
Initialize the error aggregator. Initialize the error aggregator.
@@ -113,18 +114,20 @@ class ErrorAggregator:
max_records: Maximum number of error records to keep in memory max_records: Maximum number of error records to keep in memory
pattern_threshold: Number of occurrences to detect a pattern pattern_threshold: Number of occurrences to detect a pattern
pattern_window_minutes: Time window for pattern detection pattern_window_minutes: Time window for pattern detection
export_path: Optional path for JSON export (auto-export on pattern detection)
""" """
self.logger = logging.getLogger(__name__) self.logger = logging.getLogger(__name__)
self.max_records = max_records self.max_records = max_records
self.pattern_threshold = pattern_threshold self.pattern_threshold = pattern_threshold
self.pattern_window = timedelta(minutes=pattern_window_minutes) self.pattern_window = timedelta(minutes=pattern_window_minutes)
self.export_path = export_path
self._records: List[ErrorRecord] = [] self._records: List[ErrorRecord] = []
self._error_counts: Dict[str, int] = defaultdict(int) self._error_counts: Dict[str, int] = defaultdict(int)
self._plugin_error_counts: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int)) self._plugin_error_counts: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int))
self._patterns: Dict[str, ErrorPattern] = {} self._patterns: Dict[str, ErrorPattern] = {}
self._pattern_callbacks: List[Callable[[ErrorPattern], None]] = [] self._pattern_callbacks: List[Callable[[ErrorPattern], None]] = []
self._lock = threading.RLock() # RLock: build_snapshot and pattern callbacks re-enter self._lock = threading.RLock() # RLock allows nested acquisition for export_to_file
# Track session start for relative timing # Track session start for relative timing
self._session_start = datetime.now() self._session_start = datetime.now()
@@ -245,6 +248,10 @@ class ErrorAggregator:
callback(pattern) callback(pattern)
except Exception as e: except Exception as e:
self.logger.error(f"Pattern callback failed: {e}") self.logger.error(f"Pattern callback failed: {e}")
# Auto-export if path configured
if self.export_path:
self._auto_export()
else: else:
# Update existing pattern # Update existing pattern
self._patterns[pattern_key].count = count self._patterns[pattern_key].count = count
@@ -417,6 +424,33 @@ class ErrorAggregator:
# turned into a string here rather than failing the write. # turned into a string here rather than failing the write.
return json.loads(json.dumps(summary, default=str)) return json.loads(json.dumps(summary, default=str))
def export_to_file(self, filepath: Path) -> None:
"""
Export error data to JSON file.
Args:
filepath: Path to export file
"""
with self._lock:
data = {
"exported_at": datetime.now().isoformat(),
"summary": self.get_error_summary(),
"all_records": [r.to_dict() for r in self._records]
}
filepath.parent.mkdir(parents=True, exist_ok=True)
filepath.write_text(json.dumps(data, indent=2))
self.logger.info(f"Exported error data to {filepath}")
def _auto_export(self) -> None:
"""Auto-export on pattern detection (if export_path configured)."""
if self.export_path:
try:
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
filepath = self.export_path / f"errors_{timestamp}.json"
self.export_to_file(filepath)
except Exception as e:
self.logger.error(f"Auto-export failed: {e}")
# Global singleton instance # Global singleton instance
_error_aggregator: Optional[ErrorAggregator] = None _error_aggregator: Optional[ErrorAggregator] = None
@@ -427,6 +461,7 @@ def get_error_aggregator(
max_records: int = 1000, max_records: int = 1000,
pattern_threshold: int = 5, pattern_threshold: int = 5,
pattern_window_minutes: int = 60, pattern_window_minutes: int = 60,
export_path: Optional[Path] = None
) -> ErrorAggregator: ) -> ErrorAggregator:
""" """
Get or create the global error aggregator instance. Get or create the global error aggregator instance.
@@ -435,6 +470,7 @@ def get_error_aggregator(
max_records: Maximum records to keep (only used on first call) max_records: Maximum records to keep (only used on first call)
pattern_threshold: Pattern detection threshold (only used on first call) pattern_threshold: Pattern detection threshold (only used on first call)
pattern_window_minutes: Pattern detection window (only used on first call) pattern_window_minutes: Pattern detection window (only used on first call)
export_path: Export path for auto-export (only used on first call)
Returns: Returns:
The global ErrorAggregator instance The global ErrorAggregator instance
@@ -447,6 +483,7 @@ def get_error_aggregator(
max_records=max_records, max_records=max_records,
pattern_threshold=pattern_threshold, pattern_threshold=pattern_threshold,
pattern_window_minutes=pattern_window_minutes, pattern_window_minutes=pattern_window_minutes,
export_path=export_path
) )
return _error_aggregator return _error_aggregator
+9 -19
View File
@@ -5,13 +5,11 @@ Provides specific exception types for different error categories,
enabling better error handling and debugging. enabling better error handling and debugging.
""" """
from typing import Optional
class LEDMatrixError(Exception): class LEDMatrixError(Exception):
"""Base exception for all LEDMatrix errors.""" """Base exception for all LEDMatrix errors."""
def __init__(self, message: str, context: Optional[dict] = None): def __init__(self, message: str, context: dict = None):
""" """
Initialize the exception. Initialize the exception.
@@ -34,7 +32,7 @@ class LEDMatrixError(Exception):
class CacheError(LEDMatrixError): class CacheError(LEDMatrixError):
"""Exception raised for cache-related errors.""" """Exception raised for cache-related errors."""
def __init__(self, message: str, cache_key: Optional[str] = None, context: Optional[dict] = None): def __init__(self, message: str, cache_key: str = None, context: dict = None):
""" """
Initialize cache error. Initialize cache error.
@@ -44,9 +42,7 @@ class CacheError(LEDMatrixError):
context: Optional context dictionary context: Optional context dictionary
""" """
if cache_key: if cache_key:
# Copy so the caller's dict isn't mutated (a reused context context = context or {}
# dict would otherwise collect every error's keys).
context = dict(context or {})
context['cache_key'] = cache_key context['cache_key'] = cache_key
super().__init__(message, context) super().__init__(message, context)
self.cache_key = cache_key self.cache_key = cache_key
@@ -55,7 +51,7 @@ class CacheError(LEDMatrixError):
class ConfigError(LEDMatrixError): class ConfigError(LEDMatrixError):
"""Exception raised for configuration-related errors.""" """Exception raised for configuration-related errors."""
def __init__(self, message: str, config_path: Optional[str] = None, field: Optional[str] = None, context: Optional[dict] = None): def __init__(self, message: str, config_path: str = None, field: str = None, context: dict = None):
""" """
Initialize config error. Initialize config error.
@@ -66,9 +62,7 @@ class ConfigError(LEDMatrixError):
context: Optional context dictionary context: Optional context dictionary
""" """
if config_path or field: if config_path or field:
# Copy so the caller's dict isn't mutated (a reused context context = context or {}
# dict would otherwise collect every error's keys).
context = dict(context or {})
if config_path: if config_path:
context['config_path'] = config_path context['config_path'] = config_path
if field: if field:
@@ -81,7 +75,7 @@ class ConfigError(LEDMatrixError):
class PluginError(LEDMatrixError): class PluginError(LEDMatrixError):
"""Exception raised for plugin-related errors.""" """Exception raised for plugin-related errors."""
def __init__(self, message: str, plugin_id: Optional[str] = None, context: Optional[dict] = None): def __init__(self, message: str, plugin_id: str = None, context: dict = None):
""" """
Initialize plugin error. Initialize plugin error.
@@ -91,9 +85,7 @@ class PluginError(LEDMatrixError):
context: Optional context dictionary context: Optional context dictionary
""" """
if plugin_id: if plugin_id:
# Copy so the caller's dict isn't mutated (a reused context context = context or {}
# dict would otherwise collect every error's keys).
context = dict(context or {})
context['plugin_id'] = plugin_id context['plugin_id'] = plugin_id
super().__init__(message, context) super().__init__(message, context)
self.plugin_id = plugin_id self.plugin_id = plugin_id
@@ -102,7 +94,7 @@ class PluginError(LEDMatrixError):
class DisplayError(LEDMatrixError): class DisplayError(LEDMatrixError):
"""Exception raised for display-related errors.""" """Exception raised for display-related errors."""
def __init__(self, message: str, display_mode: Optional[str] = None, context: Optional[dict] = None): def __init__(self, message: str, display_mode: str = None, context: dict = None):
""" """
Initialize display error. Initialize display error.
@@ -112,9 +104,7 @@ class DisplayError(LEDMatrixError):
context: Optional context dictionary context: Optional context dictionary
""" """
if display_mode: if display_mode:
# Copy so the caller's dict isn't mutated (a reused context context = context or {}
# dict would otherwise collect every error's keys).
context = dict(context or {})
context['display_mode'] = display_mode context['display_mode'] = display_mode
super().__init__(message, context) super().__init__(message, context)
self.display_mode = display_mode self.display_mode = display_mode
+17 -70
View File
@@ -28,10 +28,10 @@ for accurate width/height calculations.
import os import os
import logging import logging
import freetype import freetype
import requests
import json import json
import hashlib import hashlib
import urllib.parse import urllib.parse
import urllib.request
import zipfile import zipfile
import tempfile import tempfile
import time import time
@@ -50,9 +50,6 @@ from src.deprecation import deprecated
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# Seconds before a stalled font download gives up (connect and per-read).
_FONT_DOWNLOAD_TIMEOUT = 30
class FontManager: class FontManager:
""" """
Comprehensive font management supporting TTF and BDF fonts with caching, Comprehensive font management supporting TTF and BDF fonts with caching,
@@ -323,59 +320,31 @@ class FontManager:
extension = self._get_font_extension(url) extension = self._get_font_extension(url)
cache_filename = f"{family}_{url_hash}{extension}" cache_filename = f"{family}_{url_hash}{extension}"
cache_path = self.temp_font_dir / cache_filename cache_path = self.temp_font_dir / cache_filename
is_zip = url.endswith('.zip')
extract_dir = self.temp_font_dir / f"{family}_{url_hash}"
# Check if already downloaded. For a zip the font is the file # Check if already downloaded
# extracted from it, so look there first -- returning the cached if cache_path.exists():
# .zip itself would register the archive as the font after a
# restart.
if is_zip:
extracted = self._find_extracted_font(extract_dir)
if extracted:
logger.info(f"Using cached font: {extracted}")
return extracted
elif cache_path.exists():
logger.info(f"Using cached font: {cache_path}") logger.info(f"Using cached font: {cache_path}")
return str(cache_path) return str(cache_path)
if not cache_path.exists(): # Download font — restrict to http/https to prevent file:// reads
# Download font — restrict to http/https to prevent file:// reads parsed = urllib.parse.urlparse(url)
parsed = urllib.parse.urlparse(url) if parsed.scheme not in ('http', 'https'):
if parsed.scheme not in ('http', 'https'): raise ValueError(f"Font URL must use http or https, got: {parsed.scheme!r}")
raise ValueError(f"Font URL must use http or https, got: {parsed.scheme!r}") logger.info(f"Downloading font from {url}")
logger.info(f"Downloading font from {url}") urllib.request.urlretrieve(url, cache_path) # nosec B310 - scheme validated above
# Download to a temp file and rename into place, with a
# timeout: writing straight to cache_path left a truncated
# file after a stalled/interrupted download, and the exists()
# check above then served it forever.
fd, tmp_name = tempfile.mkstemp(dir=self.temp_font_dir, suffix='.part')
try:
with os.fdopen(fd, 'wb') as tmp_file:
response = requests.get(url, timeout=_FONT_DOWNLOAD_TIMEOUT, stream=True)
response.raise_for_status()
for chunk in response.iter_content(chunk_size=65536):
if chunk:
tmp_file.write(chunk)
os.replace(tmp_name, cache_path)
except BaseException:
try:
os.unlink(tmp_name)
except OSError:
pass
raise
# Handle zip files # Handle zip files
if is_zip: if url.endswith('.zip'):
extract_dir = self.temp_font_dir / f"{family}_{url_hash}"
extract_dir.mkdir(exist_ok=True) extract_dir.mkdir(exist_ok=True)
with zipfile.ZipFile(cache_path, 'r') as zip_ref: with zipfile.ZipFile(cache_path, 'r') as zip_ref:
zip_ref.extractall(extract_dir) zip_ref.extractall(extract_dir)
# Find the actual font file # Find the actual font file
extracted = self._find_extracted_font(extract_dir) for file in extract_dir.iterdir():
if extracted: if file.suffix.lower() in ['.ttf', '.otf', '.bdf']:
return extracted return str(file)
return str(cache_path) return str(cache_path)
@@ -383,16 +352,6 @@ class FontManager:
logger.error(f"Error downloading font from {url}: {e}") logger.error(f"Error downloading font from {url}: {e}")
return None return None
@staticmethod
def _find_extracted_font(extract_dir: Path) -> Optional[str]:
"""Return the first font file in a zip's extract dir, if any."""
if not extract_dir.is_dir():
return None
for file in extract_dir.iterdir():
if file.suffix.lower() in ['.ttf', '.otf', '.bdf']:
return str(file)
return None
def _get_font_extension(self, url: str) -> str: def _get_font_extension(self, url: str) -> str:
"""Extract font file extension from URL.""" """Extract font file extension from URL."""
if '.ttf' in url.lower(): if '.ttf' in url.lower():
@@ -467,9 +426,6 @@ class FontManager:
keys_to_remove = [key for key in self.font_cache.keys() if key.startswith(f"{plugin_id}::")] keys_to_remove = [key for key in self.font_cache.keys() if key.startswith(f"{plugin_id}::")]
for key in keys_to_remove: for key in keys_to_remove:
del self.font_cache[key] del self.font_cache[key]
if keys_to_remove:
# Font objects someone may hold were dropped; see cache_generation.
self.cache_generation += 1
@deprecated("3.7.0") @deprecated("3.7.0")
def get_plugin_fonts(self, plugin_id: str) -> List[str]: def get_plugin_fonts(self, plugin_id: str) -> List[str]:
@@ -539,7 +495,6 @@ class FontManager:
self.performance_stats["cache_misses"] += 1 self.performance_stats["cache_misses"] += 1
# Load font # Load font
shareable = True
font_path = self.font_catalog.get(family) font_path = self.font_catalog.get(family)
if not font_path: if not font_path:
logger.warning(f"Font family '{family}' not found") logger.warning(f"Font family '{family}' not found")
@@ -549,7 +504,6 @@ class FontManager:
try: try:
if font_path.endswith('.bdf'): if font_path.endswith('.bdf'):
font = self._load_bdf_font(font_path, size_px) font = self._load_bdf_font(font_path, size_px)
shareable = False
else: else:
font = load_truetype(font_path, size_px) font = load_truetype(font_path, size_px)
except Exception as e: except Exception as e:
@@ -559,11 +513,7 @@ class FontManager:
self.performance_stats["failed_loads"] += 1 self.performance_stats["failed_loads"] += 1
font = ImageFont.load_default() font = ImageFont.load_default()
# A BDF face is not cached here: font_cache is shared by every self.font_cache[cache_key] = font
# thread, and a freetype.Face must never be (see load_bdf_face, which
# already caches BDF faces per thread).
if shareable:
self.font_cache[cache_key] = font
return font return font
def _load_bdf_font(self, font_path: str, size_px: int) -> freetype.Face: def _load_bdf_font(self, font_path: str, size_px: int) -> freetype.Face:
@@ -782,9 +732,6 @@ class FontManager:
"""Clear font and metrics cache.""" """Clear font and metrics cache."""
self.font_cache.clear() self.font_cache.clear()
self.metrics_cache.clear() self.metrics_cache.clear()
# Holders of derived caches (layout fits, font usage) key off this;
# without the bump they kept serving results for the dropped fonts.
self.cache_generation += 1
logger.info("Font cache cleared") logger.info("Font cache cleared")
@deprecated("3.7.0", "read font_catalog") @deprecated("3.7.0", "read font_catalog")
+3 -7
View File
@@ -43,10 +43,7 @@ class StructuredFormatter(logging.Formatter):
if hasattr(record, 'operation_id'): if hasattr(record, 'operation_id'):
log_data['operation_id'] = record.operation_id log_data['operation_id'] = record.operation_id
# default=str: record.context / extras can hold datetimes, Paths, return json.dumps(log_data)
# exceptions etc.; without it one such value raised TypeError and
# the whole record was dropped by the handler's error path.
return json.dumps(log_data, default=str)
class ContextualFormatter(logging.Formatter): class ContextualFormatter(logging.Formatter):
@@ -125,7 +122,6 @@ def setup_logging(
root_logger.handlers.clear() root_logger.handlers.clear()
# Create formatter based on type # Create formatter based on type
formatter: logging.Formatter
if format_type == 'json': if format_type == 'json':
formatter = StructuredFormatter() formatter = StructuredFormatter()
else: else:
@@ -245,7 +241,7 @@ class PluginLoggerAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs): def process(self, msg, kwargs):
extra = dict(kwargs.get('extra') or {}) extra = dict(kwargs.get('extra') or {})
extra.setdefault('plugin_id', self.extra.get('plugin_id')) # type: ignore[union-attr] # get_logger always passes a dict extra.setdefault('plugin_id', self.extra.get('plugin_id'))
kwargs['extra'] = extra kwargs['extra'] = extra
return msg, kwargs return msg, kwargs
@@ -291,7 +287,7 @@ def log_with_context(
operation_id: Optional operation ID for request tracking operation_id: Optional operation ID for request tracking
exc_info: Optional exception info for error logging exc_info: Optional exception info for error logging
""" """
extra: Dict[str, Any] = {} extra = {}
if context: if context:
extra['context'] = context extra['context'] = context
+4 -8
View File
@@ -13,7 +13,7 @@ import time
import logging import logging
import requests import requests
import json import json
from typing import Dict, List, Optional, Tuple, Union from typing import Dict, List, Optional, Tuple
from pathlib import Path from pathlib import Path
from PIL import Image, ImageDraw, ImageFont, UnidentifiedImageError from PIL import Image, ImageDraw, ImageFont, UnidentifiedImageError
from src.common.font_layout import load_truetype, resolve_asset_path from src.common.font_layout import load_truetype, resolve_asset_path
@@ -235,10 +235,7 @@ def refresh_placeholder_timestamp(filepath: Path) -> bool:
metadata = PngInfo() metadata = PngInfo()
metadata.add_text(PLACEHOLDER_MARKER, str(time.time())) metadata.add_text(PLACEHOLDER_MARKER, str(time.time()))
with Image.open(filepath) as img: with Image.open(filepath) as img:
image = img.copy() img.copy().save(filepath, "PNG", pnginfo=metadata)
# Atomically, like every other logo write: a renderer can open this
# file at any moment, and an in-place save exposes a truncated PNG.
save_png_atomically(image, filepath, pnginfo=metadata)
return True return True
except Exception: except Exception:
logger.debug("Could not refresh placeholder timestamp for %s", filepath, logger.debug("Could not refresh placeholder timestamp for %s", filepath,
@@ -481,7 +478,7 @@ class LogoDownloader:
logger.info(f"Fetching team data for {league} from ESPN API...") logger.info(f"Fetching team data for {league} from ESPN API...")
response = self.session.get(api_url, params={'limit':1000},headers=self.headers, timeout=self.request_timeout) response = self.session.get(api_url, params={'limit':1000},headers=self.headers, timeout=self.request_timeout)
response.raise_for_status() response.raise_for_status()
data: Dict = response.json() data = response.json()
logger.info(f"Successfully fetched team data for {league}") logger.info(f"Successfully fetched team data for {league}")
return data return data
@@ -505,7 +502,7 @@ class LogoDownloader:
logger.info(f"Fetching team data for team {team_id} in {league} from ESPN API...") logger.info(f"Fetching team data for team {team_id} in {league} from ESPN API...")
response = self.session.get(f"{api_url}/{team_id}", headers=self.headers, timeout=self.request_timeout) response = self.session.get(f"{api_url}/{team_id}", headers=self.headers, timeout=self.request_timeout)
response.raise_for_status() response.raise_for_status()
data: Dict = response.json() data = response.json()
logger.info(f"Successfully fetched team data for {team_id} in {league}") logger.info(f"Successfully fetched team data for {team_id} in {league}")
return data return data
@@ -815,7 +812,6 @@ class LogoDownloader:
draw = ImageDraw.Draw(logo) draw = ImageDraw.Draw(logo)
# Try to load a font, fallback to default # Try to load a font, fallback to default
font: Optional[Union[ImageFont.FreeTypeFont, ImageFont.ImageFont]]
try: try:
font = load_truetype(resolve_asset_path("assets/fonts/PressStart2P-Regular.ttf"), 12) font = load_truetype(resolve_asset_path("assets/fonts/PressStart2P-Regular.ttf"), 12)
except (OSError, IOError): except (OSError, IOError):
+1 -8
View File
@@ -74,13 +74,6 @@ MAPPING_OUTPUTS: Dict[str, int] = {
'classic-pi1': 1, 'classic-pi1': 1,
} }
#: The refresh cap (``display.hardware.limit_refresh_rate_hz``) when config
#: omits it -- config/config.template.json's value. DisplayManager passes it to
#: the library and reports it as ``refresh_hz`` for scroll pacing, so the two
#: must be the same number: they were 90 and 100, and pacing solved against a
#: rate the panel was capped below.
DEFAULT_REFRESH_LIMIT_HZ = 100
#: What DisplayManager passes when a key is missing from display.hardware / #: What DisplayManager passes when a key is missing from display.hardware /
#: display.runtime. Config migration normally fills these from #: display.runtime. Config migration normally fills these from
#: config/config.template.json first, so they rarely apply. #: config/config.template.json first, so they rarely apply.
@@ -88,7 +81,7 @@ DISPLAY_MANAGER_DEFAULTS: Dict[str, Any] = {
'rows': 32, 'cols': 64, 'chain_length': 2, 'parallel': 1, 'rows': 32, 'cols': 64, 'chain_length': 2, 'parallel': 1,
'hardware_mapping': 'adafruit-hat-pwm', 'brightness': 90, 'pwm_bits': 10, 'hardware_mapping': 'adafruit-hat-pwm', 'brightness': 90, 'pwm_bits': 10,
'pwm_lsb_nanoseconds': 150, 'led_rgb_sequence': 'RGB', 'pwm_lsb_nanoseconds': 150, 'led_rgb_sequence': 'RGB',
'row_address_type': 0, 'multiplexing': 0, 'limit_refresh_rate_hz': DEFAULT_REFRESH_LIMIT_HZ, 'row_address_type': 0, 'multiplexing': 0, 'limit_refresh_rate_hz': 90,
'gpio_slowdown': 3, 'gpio_slowdown': 3,
} }
+7 -14
View File
@@ -776,35 +776,28 @@ class BasePlugin(ABC):
tighter arrangement instead of being cropped afterwards. tighter arrangement instead of being cropped afterwards.
Vegas also narrows ``display_manager`` for the duration of the call, so Vegas also narrows ``display_manager`` for the duration of the call, so
a plugin that already sizes itself from ``display_manager.width`` needs a plugin that already sizes itself from ``matrix.width`` needs no
no changes. Read this only when you size content some other way. changes. Read this only when you size content some other way.
Controlled by the plugin's own ``vegas_width_pct`` config value, else Controlled by the plugin's own ``vegas_width_pct`` config value, else
the global ``display.vegas_scroll.render_width_pct``. the global ``display.vegas_scroll.render_width_pct``.
Returns: Returns:
Target width in pixels. Outside a Vegas content request, the full Target width in pixels. Outside a Vegas content request, the full
display width: ``display_manager.width``, which falls back to the display width.
canvas size when ``matrix`` is None (hardware init failed).
""" """
requested = getattr(self, '_vegas_render_width', None) requested = getattr(self, '_vegas_render_width', None)
if isinstance(requested, int) and requested > 0: if isinstance(requested, int) and requested > 0:
return requested return requested
# display_manager.width first, as CLAUDE.md asks of every plugin: it
# already reads matrix.width when there is a matrix. matrix.width is
# only the fallback for a display_manager without a width (a test
# double, an older wrapper).
display_manager = getattr(self, 'display_manager', None) display_manager = getattr(self, 'display_manager', None)
width = getattr(display_manager, 'width', None)
if callable(width):
width = width()
if width:
return int(width)
matrix = getattr(display_manager, 'matrix', None) matrix = getattr(display_manager, 'matrix', None)
if matrix is not None and getattr(matrix, 'width', None): if matrix is not None and getattr(matrix, 'width', None):
return int(matrix.width) return int(matrix.width)
return 128 width = getattr(display_manager, 'width', None)
if callable(width):
width = width()
return int(width) if width else 128
def get_vegas_content(self) -> Optional[Any]: def get_vegas_content(self) -> Optional[Any]:
""" """
+3 -9
View File
@@ -97,8 +97,6 @@ def parse_semver(value: Any) -> Optional[Tuple[int, int, int]]:
try: try:
nums = [int(''.join(ch for ch in p if ch.isdigit()) or 0) for p in parts[:3]] nums = [int(''.join(ch for ch in p if ch.isdigit()) or 0) for p in parts[:3]]
except ValueError: except ValueError:
# Reachable: str.isdigit() accepts characters int() rejects, such as
# a superscript "\u00b2" -- "1.\u00b2.0" lands here.
return None return None
while len(nums) < 3: while len(nums) < 3:
nums.append(0) nums.append(0)
@@ -191,11 +189,11 @@ def satisfies_compatible_versions(
return any(parsed) return any(parsed)
def declared_min_version(manifest: Dict[str, Any]) -> Any: def declared_min_version(manifest: Dict[str, Any]) -> Optional[str]:
"""The core version this plugin says it needs, or ``None`` if it doesn't say. """The core version this plugin says it needs, or ``None`` if it doesn't say.
Checked in order of specificity. `ledmatrix_min` is the deprecated spelling Checked in order of specificity. `ledmatrix_min` is the deprecated spelling
of `ledmatrix_min_version` (`store_manager._validate_manifest_version_fields` flags of `ledmatrix_min_version` (`store_manager._validate_manifest_fields` flags
it); both are read because a large share of published manifests still carry it); both are read because a large share of published manifests still carry
the old one. the old one.
@@ -206,10 +204,6 @@ def declared_min_version(manifest: Dict[str, Any]) -> Any:
untrustworthy-core branch of :func:`check` calls this for *every* manifest, untrustworthy-core branch of :func:`check` calls this for *every* manifest,
so one malformed file would take down the install path rather than just so one malformed file would take down the install path rather than just
itself. A shape we do not recognise means "no declared floor". itself. A shape we do not recognise means "no declared floor".
The value is returned as the manifest holds it -- normally a version
string, but nothing here checks that; callers hand it to
:func:`parse_semver`, which accepts anything.
""" """
declared = manifest.get('min_ledmatrix_version') declared = manifest.get('min_ledmatrix_version')
if not declared: if not declared:
@@ -226,7 +220,7 @@ def declared_min_version(manifest: Dict[str, Any]) -> Any:
return None return None
def is_update_available(installed_version: Any, latest_version: Any) -> bool: def is_update_available(installed_version: str, latest_version: str) -> bool:
"""Return True when the registry's ``latest_version`` is strictly newer """Return True when the registry's ``latest_version`` is strictly newer
than the installed version. than the installed version.
+6 -8
View File
@@ -11,7 +11,6 @@ from datetime import datetime
from pathlib import Path from pathlib import Path
from dataclasses import dataclass, asdict from dataclasses import dataclass, asdict
from src.config_manager_atomic import atomic_write_text
from src.logging_config import get_logger from src.logging_config import get_logger
@@ -181,16 +180,15 @@ class OperationHistory:
return return
try: try:
# Held across the write, and written via a temp file, so two
# threads saving at once can't interleave or truncate the file.
with self._lock: with self._lock:
history_data = [record.to_dict() for record in self._history] history_data = [record.to_dict() for record in self._history]
# Ensure directory exists # Ensure directory exists
self.history_file.parent.mkdir(parents=True, exist_ok=True) self.history_file.parent.mkdir(parents=True, exist_ok=True)
# Write to file # Write to file
atomic_write_text(self.history_file, json.dumps(history_data, indent=2)) with open(self.history_file, 'w') as f:
json.dump(history_data, f, indent=2)
except Exception as e: except Exception as e:
self.logger.error(f"Error saving operation history: {e}", exc_info=True) self.logger.error(f"Error saving operation history: {e}", exc_info=True)
+14 -18
View File
@@ -85,17 +85,6 @@ class PluginOperationQueue:
f"Plugin {plugin_id} already has an active operation: " f"Plugin {plugin_id} already has an active operation: "
f"{active_op.operation_id} ({active_op.operation_type.value})" f"{active_op.operation_id} ({active_op.operation_type.value})"
) )
# _active_operations only holds the *running* one, so a second
# request while the first still waits in the queue (a double-
# clicked Install) used to be queued too, and both ran back to
# back. Refuse it the same way.
for queued_op in self._operations.values():
if queued_op.plugin_id == plugin_id and queued_op.status == OperationStatus.PENDING:
raise ValueError(
f"Plugin {plugin_id} already has an active operation: "
f"{queued_op.operation_id} ({queued_op.operation_type.value})"
)
# Create operation # Create operation
operation = PluginOperation( operation = PluginOperation(
@@ -183,6 +172,20 @@ class PluginOperationQueue:
) )
return history[:limit] return history[:limit]
def get_active_operations(self) -> List[PluginOperation]:
"""
Get all currently active operations (pending or running).
Returns:
List of active operations
"""
with self._lock:
active = []
for operation in self._operations.values():
if operation.status in [OperationStatus.PENDING, OperationStatus.RUNNING]:
active.append(operation)
return active
def _start_worker(self) -> None: def _start_worker(self) -> None:
"""Start the worker thread that processes operations.""" """Start the worker thread that processes operations."""
if self._worker_thread and self._worker_thread.is_alive(): if self._worker_thread and self._worker_thread.is_alive():
@@ -299,14 +302,7 @@ class PluginOperationQueue:
if len(self._operation_history) > self.max_history: if len(self._operation_history) > self.max_history:
# Remove oldest operations # Remove oldest operations
self._operation_history.sort(key=lambda op: op.created_at) self._operation_history.sort(key=lambda op: op.created_at)
dropped = self._operation_history[:-self.max_history]
self._operation_history = self._operation_history[-self.max_history:] self._operation_history = self._operation_history[-self.max_history:]
# ...and forget them in the status map too, which otherwise kept
# every operation ever enqueued for the life of the process. A
# still-pending or running one is never dropped from lookups.
for op in dropped:
if op.status not in (OperationStatus.PENDING, OperationStatus.RUNNING):
self._operations.pop(op.operation_id, None)
def shutdown(self) -> None: def shutdown(self) -> None:
"""Shutdown the operation queue and worker thread.""" """Shutdown the operation queue and worker thread."""
+5 -6
View File
@@ -45,7 +45,7 @@ from __future__ import annotations
import json import json
from dataclasses import dataclass, field from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Set, Union, cast from typing import Any, Dict, Iterable, List, Optional, Set, Union
from src.common.path_safety import safe_path_component from src.common.path_safety import safe_path_component
@@ -91,7 +91,7 @@ class PluginDirEntry:
"""One candidate directory and its manifest, read once.""" """One candidate directory and its manifest, read once."""
path: Path path: Path
status: str status: str
manifest: Any = None manifest: Optional[Any] = None
error: Optional[BaseException] = None error: Optional[BaseException] = None
@property @property
@@ -103,8 +103,7 @@ class PluginDirEntry:
"""The manifest's ``id`` when the manifest is usable, else None.""" """The manifest's ``id`` when the manifest is usable, else None."""
if self.status != ManifestStatus.OK: if self.status != ManifestStatus.OK:
return None return None
# OK means a dict whose "id" is a non-empty string (_read_entry). return self.manifest['id']
return cast(str, self.manifest['id'])
@property @property
def manifest_parses(self) -> bool: def manifest_parses(self) -> bool:
@@ -241,7 +240,7 @@ class PluginDirectoryIndex:
# -- lookup ----------------------------------------------------------- # -- lookup -----------------------------------------------------------
def find(self, plugin_id: Any, *, prefix: bool, case_insensitive: bool, def find(self, plugin_id: str, *, prefix: bool, case_insensitive: bool,
by_manifest: bool = True) -> Optional[Path]: by_manifest: bool = True) -> Optional[Path]:
"""Resolve ``plugin_id`` within this directory (rules in the module doc).""" """Resolve ``plugin_id`` within this directory (rules in the module doc)."""
plugin_id = _lookup_id(plugin_id) plugin_id = _lookup_id(plugin_id)
@@ -271,7 +270,7 @@ def _lookup_id(plugin_id: Any) -> Optional[str]:
plugin_id = safe_path_component(plugin_id) plugin_id = safe_path_component(plugin_id)
if plugin_id is None or is_ignored_dir_name(plugin_id): if plugin_id is None or is_ignored_dir_name(plugin_id):
return None return None
return cast(str, plugin_id) # safe_path_component returned a str return plugin_id
def _candidate_names(plugin_id: str, prefix: bool) -> List[str]: def _candidate_names(plugin_id: str, prefix: bool) -> List[str]:
+2 -2
View File
@@ -6,7 +6,7 @@ error isolation, and performance monitoring.
""" """
import time import time
from typing import Any, Dict, Optional, Callable from typing import Any, Optional, Callable
from threading import Thread from threading import Thread
import logging import logging
@@ -62,7 +62,7 @@ class PluginExecutor:
plugin_context = f"plugin {plugin_id}" if plugin_id else "plugin" plugin_context = f"plugin {plugin_id}" if plugin_id else "plugin"
# Use threading-based timeout (more reliable than signal-based) # Use threading-based timeout (more reliable than signal-based)
result_container: Dict[str, Any] = {'value': None, 'exception': None, 'completed': False} result_container = {'value': None, 'exception': None, 'completed': False}
def target(): def target():
try: try:
+2 -3
View File
@@ -6,11 +6,10 @@ and circuit breaker state. Provides automatic recovery mechanisms.
""" """
import time import time
import logging
from typing import Dict, Optional, Any, Tuple from typing import Dict, Optional, Any, Tuple
from enum import Enum from enum import Enum
from src.logging_config import get_logger
class CircuitState(Enum): class CircuitState(Enum):
"""Circuit breaker states.""" """Circuit breaker states."""
@@ -44,7 +43,7 @@ class PluginHealthTracker:
self.failure_threshold = failure_threshold self.failure_threshold = failure_threshold
self.cooldown_period = cooldown_period self.cooldown_period = cooldown_period
self.half_open_timeout = half_open_timeout self.half_open_timeout = half_open_timeout
self.logger = get_logger(__name__) self.logger = logging.getLogger(__name__)
# In-memory health state (also persisted to cache) # In-memory health state (also persisted to cache)
self._health_state: Dict[str, Dict[str, Any]] = {} self._health_state: Dict[str, Dict[str, Any]] = {}
+87 -104
View File
@@ -23,11 +23,6 @@ from src.exceptions import PluginError
from src.logging_config import get_logger from src.logging_config import get_logger
from src.plugin_system.plugin_dirs import resolve_plugin_dir from src.plugin_system.plugin_dirs import resolve_plugin_dir
#: Serialises pip runs across threads. Startup loads plugins on a small
#: thread pool, and two concurrent ``pip install`` processes writing the same
#: site-packages can corrupt it or fail on each other's partial installs.
_PIP_INSTALL_LOCK = threading.Lock()
def requirements_has_real_deps(requirements_file: str) -> bool: def requirements_has_real_deps(requirements_file: str) -> bool:
""" """
@@ -277,7 +272,6 @@ class PluginLoader:
Path to plugin directory or None if not found. An id that is not Path to plugin directory or None if not found. An id that is not
one plain path segment finds nothing. one plain path segment finds nothing.
""" """
plugin_dir: Optional[Path]
# Strategy 1: Use mapping from discovery # Strategy 1: Use mapping from discovery
if plugin_directories and plugin_id in plugin_directories: if plugin_directories and plugin_id in plugin_directories:
plugin_dir = plugin_directories[plugin_id] plugin_dir = plugin_directories[plugin_id]
@@ -331,94 +325,93 @@ class PluginLoader:
if requirements_file is None: if requirements_file is None:
return True return True
with _PIP_INSTALL_LOCK: try:
try: self.logger.info("Installing dependencies for plugin %s...", plugin_id)
self.logger.info("Installing dependencies for plugin %s...", plugin_id) result = subprocess.run(
result = subprocess.run( [sys.executable, "-m", "pip", "install", "--break-system-packages", "-r", requirements_file],
[sys.executable, "-m", "pip", "install", "--break-system-packages", "-r", requirements_file], capture_output=True,
capture_output=True, text=True,
text=True, timeout=timeout,
timeout=timeout, check=False
check=False )
)
if result.returncode == 0: if result.returncode == 0:
self.logger.info("Dependencies installed successfully for %s", plugin_id) self.logger.info("Dependencies installed successfully for %s", plugin_id)
return True
else:
stderr = result.stderr or ""
# uninstall-no-record-file means a system-managed copy of a package
# (e.g. apt's python3-requests, which ships no pip RECORD file) is in
# the way of the version this requirements.txt pins. Retry with
# --ignore-installed so pip lays the pinned version down alongside
# the system copy instead of trying to replace it — matching the
# retry already used by install_dependencies_apt.py / safe_pip_install.sh.
# Without this retry, the plugin would silently keep running against
# whatever version the system happened to ship.
if "uninstall-no-record-file" in stderr:
self.logger.warning(
"Dependencies for %s conflict with a system-managed package "
"(no pip RECORD); retrying with --ignore-installed: %s",
plugin_id, stderr.strip()
)
# Wrapped in its own try/except so a retry timeout is
# tolerated the same way as a retry failure, instead of
# propagating to the outer handler and returning False
# (which would contradict the "assume satisfied" fallback
# below).
try:
# sys.executable is this process's own interpreter (not
# attacker-influenced), and requirements_file is rebuilt
# by contained_plugin_dir() from a trusted listing, never raw
# external input.
retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep
[sys.executable, "-m", "pip", "install", "--break-system-packages",
"--ignore-installed", "-r", requirements_file],
capture_output=True,
text=True,
timeout=timeout,
check=False
)
if retry_result.returncode != 0:
self.logger.warning(
"Retry with --ignore-installed also failed for %s; assuming the "
"system-managed version satisfies the requirement: %s",
plugin_id, (retry_result.stderr or "").strip()
)
except subprocess.TimeoutExpired:
self.logger.warning(
"Retry with --ignore-installed timed out for %s; assuming the "
"system-managed version satisfies the requirement",
plugin_id
)
return True
self.logger.warning(
"Dependency installation returned non-zero exit code for %s: %s",
plugin_id,
stderr
)
return False
except subprocess.TimeoutExpired:
self.logger.error("Dependency installation timed out for %s", plugin_id)
return False
except FileNotFoundError:
self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id)
return True return True
except OSError as e: else:
# A broken pipe (EPIPE) happens when pip's output pipe closes stderr = result.stderr or ""
# mid-download, usually a network interruption. # uninstall-no-record-file means a system-managed copy of a package
if e.errno == errno.EPIPE: # (e.g. apt's python3-requests, which ships no pip RECORD file) is in
self.logger.error( # the way of the version this requirements.txt pins. Retry with
"Broken pipe error during dependency installation for %s. " # --ignore-installed so pip lays the pinned version down alongside
"This usually indicates a network interruption or pip output buffer issue. " # the system copy instead of trying to replace it — matching the
"Try installing again or check your network connection.", plugin_id # retry already used by install_dependencies_apt.py / safe_pip_install.sh.
# Without this retry, the plugin would silently keep running against
# whatever version the system happened to ship.
if "uninstall-no-record-file" in stderr:
self.logger.warning(
"Dependencies for %s conflict with a system-managed package "
"(no pip RECORD); retrying with --ignore-installed: %s",
plugin_id, stderr.strip()
) )
else: # Wrapped in its own try/except so a retry timeout is
self.logger.error("OS error during dependency installation for %s: %s", plugin_id, e) # tolerated the same way as a retry failure, instead of
return False # propagating to the outer handler and returning False
except Exception as e: # (which would contradict the "assume satisfied" fallback
self.logger.error("Unexpected error installing dependencies for %s: %s", plugin_id, e, exc_info=True) # below).
try:
# sys.executable is this process's own interpreter (not
# attacker-influenced), and requirements_file is rebuilt
# by contained_plugin_dir() from a trusted listing, never raw
# external input.
retry_result = subprocess.run( # nosec B603 - no shell invoked (list-form argv) # nosemgrep
[sys.executable, "-m", "pip", "install", "--break-system-packages",
"--ignore-installed", "-r", requirements_file],
capture_output=True,
text=True,
timeout=timeout,
check=False
)
if retry_result.returncode != 0:
self.logger.warning(
"Retry with --ignore-installed also failed for %s; assuming the "
"system-managed version satisfies the requirement: %s",
plugin_id, (retry_result.stderr or "").strip()
)
except subprocess.TimeoutExpired:
self.logger.warning(
"Retry with --ignore-installed timed out for %s; assuming the "
"system-managed version satisfies the requirement",
plugin_id
)
return True
self.logger.warning(
"Dependency installation returned non-zero exit code for %s: %s",
plugin_id,
stderr
)
return False return False
except subprocess.TimeoutExpired:
self.logger.error("Dependency installation timed out for %s", plugin_id)
return False
except FileNotFoundError:
self.logger.warning("pip not found. Skipping dependency installation for %s", plugin_id)
return True
except OSError as e:
# A broken pipe (EPIPE) happens when pip's output pipe closes
# mid-download, usually a network interruption.
if e.errno == errno.EPIPE:
self.logger.error(
"Broken pipe error during dependency installation for %s. "
"This usually indicates a network interruption or pip output buffer issue. "
"Try installing again or check your network connection.", plugin_id
)
else:
self.logger.error("OS error during dependency installation for %s: %s", plugin_id, e)
return False
except Exception as e:
self.logger.error("Unexpected error installing dependencies for %s: %s", plugin_id, e, exc_info=True)
return False
@staticmethod @staticmethod
def _iter_plugin_bare_modules( def _iter_plugin_bare_modules(
@@ -592,21 +585,11 @@ class PluginLoader:
raise PluginError(error_msg, plugin_id=plugin_id, context={'entry_file': str(entry_file)}) raise PluginError(error_msg, plugin_id=plugin_id, context={'entry_file': str(entry_file)})
with self._module_load_lock: with self._module_load_lock:
# Put this plugin's directory first on sys.path -- moving it there # Add plugin directory to sys.path if not already there
# if it is already present. Plugins import their own modules by
# bare name (``from sports import ...``), and those resolve to the
# first directory that has the file. A directory added on an
# earlier load stays where it was, so reloading a plugin (a live
# re-enable from the web UI) after another scoreboard had loaded
# found that one's sports.py first and failed on a name only its
# own copy has.
plugin_dir_str = str(plugin_dir) plugin_dir_str = str(plugin_dir)
try: if plugin_dir_str not in sys.path:
sys.path.remove(plugin_dir_str) sys.path.insert(0, plugin_dir_str)
except ValueError: self.logger.debug("Added plugin %s's directory to sys.path", plugin_id)
pass
sys.path.insert(0, plugin_dir_str)
self.logger.debug("Put plugin %s's directory first on sys.path", plugin_id)
# Import the plugin module # Import the plugin module
module_name = f"plugin_{plugin_id.replace('-', '_')}" module_name = f"plugin_{plugin_id.replace('-', '_')}"
+3 -89
View File
@@ -52,10 +52,6 @@ class PluginManager:
- PluginExecutor: Handles plugin execution with timeout and error isolation - PluginExecutor: Handles plugin execution with timeout and error isolation
- PluginStateManager: Manages plugin state machine - PluginStateManager: Manages plugin state machine
""" """
# How long unload_plugin() waits for an in-flight update() to finish
# before tearing the instance down anyway.
UNLOAD_LOCK_TIMEOUT = 5.0
def __init__(self, plugins_dir: str = "plugins", def __init__(self, plugins_dir: str = "plugins",
config_manager: Optional[Any] = None, config_manager: Optional[Any] = None,
@@ -187,8 +183,6 @@ class PluginManager:
someone opened a page -- the same log-volume problem this is meant to someone opened a page -- the same log-volume problem this is meant to
help diagnose. help diagnose.
""" """
# setdefault rather than self._skip_reported: tests build a bare
# scanner with PluginManager.__new__ and skip __init__.
reported = self.__dict__.setdefault('_skip_reported', set()) reported = self.__dict__.setdefault('_skip_reported', set())
if key in reported: if key in reported:
return return
@@ -412,12 +406,10 @@ class PluginManager:
try: try:
if not plugin_instance.validate_config(): if not plugin_instance.validate_config():
self.logger.error("Plugin %s configuration validation failed", plugin_id) self.logger.error("Plugin %s configuration validation failed", plugin_id)
self._discard_failed_load(plugin_id)
self.state_manager.set_state(plugin_id, PluginState.ERROR) self.state_manager.set_state(plugin_id, PluginState.ERROR)
return False return False
except Exception as e: except Exception as e:
self.logger.error("Error validating plugin %s config: %s", plugin_id, e, exc_info=True) self.logger.error("Error validating plugin %s config: %s", plugin_id, e, exc_info=True)
self._discard_failed_load(plugin_id)
self.state_manager.set_state(plugin_id, PluginState.ERROR, error=e) self.state_manager.set_state(plugin_id, PluginState.ERROR, error=e)
return False return False
@@ -441,18 +433,7 @@ class PluginManager:
self.state_manager.set_state(plugin_id, PluginState.ENABLED) self.state_manager.set_state(plugin_id, PluginState.ENABLED)
# Call on_enable if plugin is enabled # Call on_enable if plugin is enabled
if hasattr(plugin_instance, 'on_enable'): if hasattr(plugin_instance, 'on_enable'):
try: plugin_instance.on_enable()
plugin_instance.on_enable()
except Exception:
# Undo the registration above before the outer
# handler marks it ERROR: left in self.plugins, the
# next load_plugin() would return True as "already
# loaded" for a plugin that never enabled.
self.plugins.pop(plugin_id, None)
with self._plugin_last_update_lock:
self.plugin_last_update.pop(plugin_id, None)
self._update_interval_cache.pop(plugin_id, None)
raise
else: else:
self.state_manager.set_state(plugin_id, PluginState.DISABLED) self.state_manager.set_state(plugin_id, PluginState.DISABLED)
@@ -462,38 +443,16 @@ class PluginManager:
except PluginError as e: except PluginError as e:
self.logger.error("Plugin error loading %s: %s", plugin_id, e, exc_info=True) self.logger.error("Plugin error loading %s: %s", plugin_id, e, exc_info=True)
self._discard_failed_load(plugin_id)
self.state_manager.set_state(plugin_id, PluginState.ERROR, error=e) self.state_manager.set_state(plugin_id, PluginState.ERROR, error=e)
return False return False
except Exception as e: except Exception as e:
self.logger.error("Unexpected error loading plugin %s: %s", plugin_id, e, exc_info=True) self.logger.error("Unexpected error loading plugin %s: %s", plugin_id, e, exc_info=True)
self._discard_failed_load(plugin_id)
self.state_manager.set_state(plugin_id, PluginState.ERROR, error=e) self.state_manager.set_state(plugin_id, PluginState.ERROR, error=e)
return False return False
def _discard_failed_load(self, plugin_id: str) -> None:
"""Forget a plugin's imported module and font registrations after a
failed load.
load_module() reuses ``plugin_<id>`` from sys.modules, so a module
left behind by a load that failed after import (instantiation,
validate_config, on_enable) would keep serving the old code even
after the user fixes the plugin and reloads it. Never raises.
"""
try:
sys.modules.pop(f"plugin_{plugin_id.replace('-', '_')}", None)
self.plugin_loader.unregister_plugin_modules(plugin_id)
except Exception as e: # pragma: no cover - defensive
self.logger.debug("Could not drop modules of %s: %s", plugin_id, e)
try:
if self.font_manager is not None and hasattr(self.font_manager, 'forget_manager_fonts'):
self.font_manager.forget_manager_fonts(plugin_id)
except Exception as e:
self.logger.debug("Could not forget fonts of %s: %s", plugin_id, e)
#: Config keys the **core** reads out of a plugin's own config block. The #: Config keys the **core** reads out of a plugin's own config block. The
#: plugin never declares them, so a schema with #: plugin never declares them, so a schema with
#: ``"additionalProperties": false`` — most published ones do — reports #: ``"additionalProperties": false`` — 37 of the 42 published ones — reports
#: them as violations and the plugin gets flagged degraded in the web UI for #: them as violations and the plugin gets flagged degraded in the web UI for
#: using a documented core feature. #: using a documented core feature.
#: #:
@@ -635,28 +594,6 @@ class PluginManager:
self.logger.warning("Plugin %s not loaded", plugin_id) self.logger.warning("Plugin %s not loaded", plugin_id)
return False return False
# Take the plugin's lock so cleanup()/on_disable() can't run while
# the update worker is mid-update() on this instance. Bounded: an
# update() that hangs past PluginExecutor's timeout keeps holding the
# lock from its lingering thread, and unload must still go through.
lock = self.get_plugin_lock(plugin_id)
lock_acquired = lock.acquire(timeout=self.UNLOAD_LOCK_TIMEOUT)
if not lock_acquired:
self.logger.warning(
"Plugin %s still busy after %.1fs; unloading without its lock",
plugin_id, self.UNLOAD_LOCK_TIMEOUT)
try:
return self._unload_plugin_locked(plugin_id)
finally:
if lock_acquired:
lock.release()
def _unload_plugin_locked(self, plugin_id: str) -> bool:
"""Body of unload_plugin(); caller holds (or gave up on) the plugin lock."""
if plugin_id not in self.plugins: # unloaded while we waited
self.logger.warning("Plugin %s not loaded", plugin_id)
return False
try: try:
plugin = self.plugins[plugin_id] plugin = self.plugins[plugin_id]
@@ -807,13 +744,7 @@ class PluginManager:
if plugin: if plugin:
info['loaded'] = True info['loaded'] = True
if hasattr(plugin, 'get_info'): if hasattr(plugin, 'get_info'):
# One plugin's get_info() raising must not take down the info['runtime_info'] = plugin.get_info()
# whole installed-plugins listing (/api/v3/plugins/installed).
try:
info['runtime_info'] = plugin.get_info()
except Exception as e:
self.logger.warning("Plugin %s get_info() failed: %s", plugin_id, e)
info['runtime_info'] = {}
else: else:
info['loaded'] = False info['loaded'] = False
@@ -955,14 +886,6 @@ class PluginManager:
updating, since a scheduler that propagates a plugin bug stops every updating, since a scheduler that propagates a plugin bug stops every
other plugin too. other plugin too.
Precedence, first match wins: the ``get_update_interval()`` hook, then
``update_interval`` in the plugin's **manifest**, then
``update_interval`` in the plugin's section of config.json, then 60s.
So a config value only drives the scheduler for a plugin whose
manifest sets none; when the manifest sets one, the config value is
ignored here (a plugin may still read it itself, e.g. to skip fetches
inside update()).
The static result is cached per plugin_id after the first lookup, so The static result is cached per plugin_id after the first lookup, so
the manifest/config resolution is not repeated on every scheduling the manifest/config resolution is not repeated on every scheduling
tick of the display loop. A change to ``update_interval`` in tick of the display loop. A change to ``update_interval`` in
@@ -1264,15 +1187,6 @@ class PluginManager:
return return
finished['done'] = True finished['done'] = True
try: try:
# The plugin was unloaded (or reloaded as a new instance)
# while this update() ran: unload_plugin() already cleared its
# lifecycle state, so recording success/failure here would
# resurrect a torn-down plugin as ENABLED. Only release.
if self.plugins.get(plugin_id) is not plugin_instance:
if lock is not None:
with self._pending_lock:
self._pending_updates.discard(plugin_id)
return
# Drop the queue reservation *before* the state goes back to # Drop the queue reservation *before* the state goes back to
# ENABLED. The other order leaves a window where a scheduler # ENABLED. The other order leaves a window where a scheduler
# sees ENABLED, reserves the plugin, then finds it still in # sees ENABLED, reserves the plugin, then finds it still in
+1 -1
View File
@@ -17,7 +17,7 @@ from src.logging_config import get_logger
class PluginState(Enum): class PluginState(Enum):
"""Plugin state enumeration.""" """Plugin state enumeration."""
UNLOADED = "unloaded" # Plugin not loaded UNLOADED = "unloaded" # Plugin not loaded
LOADED = "loaded" # load_plugin() in progress: set before the module is imported LOADED = "loaded" # Plugin module loaded but not instantiated
ENABLED = "enabled" # Plugin instantiated and enabled ENABLED = "enabled" # Plugin instantiated and enabled
RUNNING = "running" # Plugin is currently executing RUNNING = "running" # Plugin is currently executing
ERROR = "error" # Plugin encountered an error ERROR = "error" # Plugin encountered an error
+7 -66
View File
@@ -5,14 +5,12 @@ Tracks resource usage (memory, CPU, execution time) for plugins.
Provides resource limits and performance monitoring. Provides resource limits and performance monitoring.
""" """
import math
import time import time
import logging
import threading import threading
from typing import Dict, Optional, Any, Callable, cast from typing import Dict, Optional, Any, Callable
from dataclasses import dataclass, field, fields from dataclasses import dataclass, field, fields
from src.logging_config import get_logger
try: try:
import psutil import psutil
PSUTIL_AVAILABLE = True PSUTIL_AVAILABLE = True
@@ -33,52 +31,6 @@ class ResourceLimits:
warning_threshold: float = 0.8 # Warning at 80% of limit warning_threshold: float = 0.8 # Warning at 80% of limit
_LIMIT_FIELDS = ('max_memory_mb', 'max_cpu_percent', 'max_execution_time',
'warning_threshold')
def invalid_limit_field(data: Any) -> Optional[str]:
"""The first field of a limits mapping that isn't a valid limit, or None.
``"limits"`` when ``data`` isn't a mapping at all. Separate from
limits_from_dict so a caller can report the problem without passing an
exception's text back to a client.
"""
if not isinstance(data, dict):
return 'limits'
for name in _LIMIT_FIELDS:
value = data.get(name)
if value is None:
continue
# bool is an int subclass; True is not a limit anyone meant.
if (isinstance(value, bool) or not isinstance(value, (int, float))
or not math.isfinite(value) or value < 0):
return name
return None
def limits_from_dict(data: Any) -> ResourceLimits:
"""Build ResourceLimits from a JSON-shaped mapping, validating each value.
A dataclass does not enforce its annotations, so ResourceLimits built from
raw request JSON or a cached record happily stores ``"50"`` -- and then
every monitored update() raises TypeError comparing a float with it. Each
``max_*`` value must be absent/None (no limit) or a non-negative number;
``warning_threshold`` defaults to 0.8. Unknown keys are ignored.
Raises:
ValueError: naming the first offending field.
"""
bad = invalid_limit_field(data)
if bad == 'limits':
raise ValueError(f"limits must be an object, got {type(data).__name__}")
if bad:
raise ValueError(
f"{bad} must be a non-negative number or null, got {data.get(bad)!r}")
return ResourceLimits(**{name: data[name] for name in _LIMIT_FIELDS
if data.get(name) is not None})
@dataclass @dataclass
class ResourceMetrics: class ResourceMetrics:
"""Resource usage metrics for a plugin. """Resource usage metrics for a plugin.
@@ -134,12 +86,11 @@ class PluginResourceMonitor:
""" """
self.cache_manager = cache_manager self.cache_manager = cache_manager
self.enable_monitoring = enable_monitoring and PSUTIL_AVAILABLE self.enable_monitoring = enable_monitoring and PSUTIL_AVAILABLE
self.logger = get_logger(__name__) self.logger = logging.getLogger(__name__)
# Resource metrics per plugin # Resource metrics per plugin
self._metrics: Dict[str, ResourceMetrics] = {} self._metrics: Dict[str, ResourceMetrics] = {}
self._limits: Dict[str, ResourceLimits] = {} self._limits: Dict[str, ResourceLimits] = {}
self._bad_limits_warned: set = set()
# When each plugin's metrics last reached the cache. Metrics change on # When each plugin's metrics last reached the cache. Metrics change on
# every call, so they cannot be de-duplicated the way health state can; # every call, so they cannot be de-duplicated the way health state can;
# they are rate-limited instead. See _METRICS_PERSIST_INTERVAL. # they are rate-limited instead. See _METRICS_PERSIST_INTERVAL.
@@ -209,7 +160,7 @@ class PluginResourceMonitor:
# (not \"int\") to str"). Coerce here, where there is still a cache # (not \"int\") to str"). Coerce here, where there is still a cache
# key to name in the warning. # key to name in the warning.
declared = {f.name: f.type for f in fields(ResourceMetrics)} declared = {f.name: f.type for f in fields(ResourceMetrics)}
usable: Dict[str, Any] = {} usable = {}
for key, value in cached.items(): for key, value in cached.items():
if key not in known: if key not in known:
continue continue
@@ -279,17 +230,7 @@ class PluginResourceMonitor:
cache_key = self._get_limits_key(plugin_id) cache_key = self._get_limits_key(plugin_id)
cached = self.cache_manager.get(cache_key, max_age=None) cached = self.cache_manager.get(cache_key, max_age=None)
if cached: if cached:
try: self._limits[plugin_id] = ResourceLimits(**cached)
self._limits[plugin_id] = limits_from_dict(cached)
except ValueError as e:
# Treat as no limits rather than letting every update
# of this plugin raise; warn once, not on every call.
if plugin_id not in self._bad_limits_warned:
self._bad_limits_warned.add(plugin_id)
self.logger.warning(
"Ignoring cached resource limits for %s: %s",
plugin_id, e)
return None
else: else:
return None return None
return self._limits[plugin_id] return self._limits[plugin_id]
@@ -299,7 +240,7 @@ class PluginResourceMonitor:
if not self.enable_monitoring or self._process is None: if not self.enable_monitoring or self._process is None:
return 0.0 return 0.0
try: try:
return cast(float, self._process.memory_info().rss / 1024 / 1024) return self._process.memory_info().rss / 1024 / 1024
except Exception: except Exception:
return 0.0 return 0.0
@@ -313,7 +254,7 @@ class PluginResourceMonitor:
if not self.enable_monitoring or self._process is None: if not self.enable_monitoring or self._process is None:
return 0.0 return 0.0
try: try:
return cast(float, self._process.cpu_percent(interval=None)) return self._process.cpu_percent(interval=None)
except Exception: except Exception:
return 0.0 return 0.0
+4 -4
View File
@@ -5,11 +5,11 @@ Manages saved GitHub repository URLs for easy plugin discovery and installation.
""" """
import json import json
import logging
import os import os
from pathlib import Path from pathlib import Path
from typing import List, Dict, Optional, cast from typing import List, Dict, Optional
from src.logging_config import get_logger
from src.plugin_system.repo_urls import normalize_repo_url from src.plugin_system.repo_urls import normalize_repo_url
@@ -24,7 +24,7 @@ class SavedRepositoriesManager:
config_path: Path to JSON file storing saved repositories config_path: Path to JSON file storing saved repositories
""" """
self.config_path = Path(config_path) self.config_path = Path(config_path)
self.logger = get_logger(__name__) self.logger = logging.getLogger(__name__)
self.repositories = self._load_repositories() self.repositories = self._load_repositories()
def _load_repositories(self) -> List[Dict[str, str]]: def _load_repositories(self) -> List[Dict[str, str]]:
@@ -37,7 +37,7 @@ class SavedRepositoriesManager:
if isinstance(data, list): if isinstance(data, list):
return data return data
elif isinstance(data, dict) and 'repositories' in data: elif isinstance(data, dict) and 'repositories' in data:
return cast(List[Dict[str, str]], data['repositories']) return data['repositories']
else: else:
return [] return []
return [] return []
+8 -62
View File
@@ -8,7 +8,6 @@ Provides utilities for extracting defaults, validating configurations, and manag
import copy import copy
import json import json
import logging import logging
import time
from pathlib import Path from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple from typing import Any, Dict, List, Optional, Tuple
import jsonschema import jsonschema
@@ -16,8 +15,6 @@ from jsonschema import Draft7Validator, ValidationError
from src.core_config_keys import CORE_CONFIG_KEYS from src.core_config_keys import CORE_CONFIG_KEYS
from src.element_style import expand_style_elements from src.element_style import expand_style_elements
from src.logging_config import get_logger
from src.plugin_system.plugin_dirs import resolve_plugin_dir
def _renders_as_object(prop: Dict[str, Any]) -> bool: def _renders_as_object(prop: Dict[str, Any]) -> bool:
@@ -370,7 +367,7 @@ class SchemaManager:
device-wide ``location`` that seeds plugin location defaults. device-wide ``location`` that seeds plugin location defaults.
Omitting it simply leaves schema defaults untouched. Omitting it simply leaves schema defaults untouched.
""" """
self.logger = logger or get_logger(__name__) self.logger = logger or logging.getLogger(__name__)
self.plugins_dir = plugins_dir self.plugins_dir = plugins_dir
self.project_root = project_root or Path.cwd() self.project_root = project_root or Path.cwd()
self.config_manager = config_manager self.config_manager = config_manager
@@ -380,67 +377,23 @@ class SchemaManager:
# Default config cache: plugin_id -> default config dict # Default config cache: plugin_id -> default config dict
self._defaults_cache: Dict[str, Dict[str, Any]] = {} self._defaults_cache: Dict[str, Dict[str, Any]] = {}
# Schema-path misses: plugin_id -> monotonic time of the miss. A
# lookup now scans each search directory's manifests, and plugins
# without a schema are asked about on every page render.
self._schema_path_misses: Dict[str, float] = {}
self._schema_miss_logged: set = set()
#: How long a "no schema" answer is reused before the directories are
#: searched again -- short, so a plugin installed by a path that doesn't
#: call invalidate_cache() (a dev symlink, a manual copy) still shows up.
SCHEMA_MISS_TTL = 30.0
def get_schema_path(self, plugin_id: str) -> Optional[Path]: def get_schema_path(self, plugin_id: str) -> Optional[Path]:
""" """
Get the path to a plugin's config_schema.json file. Get the path to a plugin's config_schema.json file.
Each search directory -- plugins_dir, then PROJECT_ROOT/plugins, then Tries multiple locations in order:
PROJECT_ROOT/plugin-repos -- is first resolved the way the plugin
loader resolves it (``plugin_dirs.resolve_plugin_dir``: the directory
whose manifest declares the id, else ``<id>`` / ``ledmatrix-<id>``,
case-insensitively). Only if none of those holds a schema are the
literal locations tried:
1. plugins_dir / plugin_id / config_schema.json 1. plugins_dir / plugin_id / config_schema.json
2. PROJECT_ROOT / plugins / plugin_id / config_schema.json 2. PROJECT_ROOT / plugins / plugin_id / config_schema.json
3. PROJECT_ROOT / plugin-repos / plugin_id / config_schema.json 3. PROJECT_ROOT / plugin-repos / plugin_id / config_schema.json
4. a case-insensitive match of plugin_id in plugins/ and plugin-repos/
A miss is remembered for SCHEMA_MISS_TTL seconds (or until
invalidate_cache()) and logged once, at DEBUG: PluginManager already
warns at load time about a plugin that ships no schema.
Args: Args:
plugin_id: Plugin identifier plugin_id: Plugin identifier
Returns: Returns:
Path to schema file or None if not found Path to schema file or None if not found
""" """
missed_at = self._schema_path_misses.get(plugin_id)
if missed_at is not None and time.monotonic() - missed_at < self.SCHEMA_MISS_TTL:
return None
search_dirs = []
if self.plugins_dir:
search_dirs.append(Path(self.plugins_dir))
search_dirs.extend([self.project_root / 'plugins',
self.project_root / 'plugin-repos'])
# Resolved the way the loader does, so a plugin installed as
# ``ledmatrix-<id>`` or under a directory named differently from its
# manifest id still gets its schema. One directory at a time keeps
# the documented plugins/-before-plugin-repos/ order.
possible_paths = [] possible_paths = []
for search_dir in search_dirs:
try:
resolved = resolve_plugin_dir(
plugin_id, [search_dir], prefix=True, case_insensitive=True)
except Exception as e: # pragma: no cover - defensive
self.logger.debug(f"Could not resolve {plugin_id} in {search_dir}: {e}")
resolved = None
if resolved is not None:
possible_paths.append(resolved / 'config_schema.json')
# Try plugins_dir if set # Try plugins_dir if set
if self.plugins_dir: if self.plugins_dir:
@@ -463,14 +416,9 @@ class SchemaManager:
for path in possible_paths: for path in possible_paths:
if path.exists(): if path.exists():
self.logger.debug(f"Found schema for {plugin_id} at {path}") self.logger.debug(f"Found schema for {plugin_id} at {path}")
self._schema_path_misses.pop(plugin_id, None)
self._schema_miss_logged.discard(plugin_id)
return path return path
self._schema_path_misses[plugin_id] = time.monotonic() self.logger.warning(f"Schema file not found for plugin {plugin_id}")
if plugin_id not in self._schema_miss_logged:
self._schema_miss_logged.add(plugin_id)
self.logger.debug(f"Schema file not found for plugin {plugin_id}")
return None return None
def load_schema(self, plugin_id: str, use_cache: bool = True) -> Optional[Dict[str, Any]]: def load_schema(self, plugin_id: str, use_cache: bool = True) -> Optional[Dict[str, Any]]:
@@ -533,12 +481,10 @@ class SchemaManager:
if plugin_id: if plugin_id:
self._schema_cache.pop(plugin_id, None) self._schema_cache.pop(plugin_id, None)
self._defaults_cache.pop(plugin_id, None) self._defaults_cache.pop(plugin_id, None)
self._schema_path_misses.pop(plugin_id, None)
self.logger.debug(f"Invalidated cache for plugin {plugin_id}") self.logger.debug(f"Invalidated cache for plugin {plugin_id}")
else: else:
self._schema_cache.clear() self._schema_cache.clear()
self._defaults_cache.clear() self._defaults_cache.clear()
self._schema_path_misses.clear()
self.logger.debug("Invalidated all schema caches") self.logger.debug("Invalidated all schema caches")
def extract_defaults_from_schema(self, schema: Dict[str, Any], prefix: str = '') -> Dict[str, Any]: def extract_defaults_from_schema(self, schema: Dict[str, Any], prefix: str = '') -> Dict[str, Any]:
+11 -15
View File
@@ -13,7 +13,6 @@ from datetime import datetime
from dataclasses import dataclass, asdict from dataclasses import dataclass, asdict
from enum import Enum from enum import Enum
from src.config_manager_atomic import atomic_write_text
from src.logging_config import get_logger from src.logging_config import get_logger
@@ -284,10 +283,6 @@ class PluginStateManager:
return return
try: try:
# The write stays under the lock and goes through a temp file:
# Flask serves requests on threads, and two saves racing on a
# plain open('w') could interleave or leave a truncated file
# that _load_state then drops wholesale.
with self._lock: with self._lock:
# Convert states to dicts # Convert states to dicts
states_data = { states_data = {
@@ -301,15 +296,16 @@ class PluginStateManager:
'last_updated': datetime.now().isoformat() 'last_updated': datetime.now().isoformat()
} }
# Ensure directory exists with proper permissions # Ensure directory exists with proper permissions
from src.common.permission_utils import ( from src.common.permission_utils import (
ensure_directory_permissions, ensure_directory_permissions,
get_config_dir_mode get_config_dir_mode
) )
ensure_directory_permissions(self.state_file.parent, get_config_dir_mode()) ensure_directory_permissions(self.state_file.parent, get_config_dir_mode())
# Write to file # Write to file
atomic_write_text(self.state_file, json.dumps(state_data, indent=2)) with open(self.state_file, 'w') as f:
json.dump(state_data, f, indent=2)
except Exception as e: except Exception as e:
self.logger.error(f"Error saving plugin state: {e}", exc_info=True) self.logger.error(f"Error saving plugin state: {e}", exc_info=True)
@@ -320,7 +316,7 @@ class PluginStateManager:
return return
try: try:
with open(self.state_file, 'r', encoding='utf-8') as f: with open(self.state_file, 'r') as f:
state_data = json.load(f) state_data = json.load(f)
with self._lock: with self._lock:
+4 -5
View File
@@ -9,7 +9,7 @@ Detects and fixes inconsistencies between:
""" """
import json import json
from typing import Dict, Any, List, Set, cast from typing import Dict, Any, List, Set
from dataclasses import dataclass from dataclasses import dataclass
from enum import Enum from enum import Enum
from pathlib import Path from pathlib import Path
@@ -237,7 +237,7 @@ class StateReconciliation:
state_manager_state = self._get_state_manager_state() state_manager_state = self._get_state_manager_state()
# Find all unique plugin IDs # Find all unique plugin IDs
all_plugin_ids: Set[str] = set() all_plugin_ids = set()
all_plugin_ids.update(config_state.keys()) all_plugin_ids.update(config_state.keys())
all_plugin_ids.update(disk_state.keys()) all_plugin_ids.update(disk_state.keys())
all_plugin_ids.update(manager_state.keys()) all_plugin_ids.update(manager_state.keys())
@@ -380,7 +380,7 @@ class StateReconciliation:
state_manager_state: Dict[str, Dict[str, Any]] state_manager_state: Dict[str, Dict[str, Any]]
) -> List[Inconsistency]: ) -> List[Inconsistency]:
"""Check consistency for a single plugin.""" """Check consistency for a single plugin."""
inconsistencies: List[Inconsistency] = [] inconsistencies = []
if plugin_id in CORE_CONFIG_KEYS: if plugin_id in CORE_CONFIG_KEYS:
# A plugin whose id is a core setting's key ('display', 'sync', # A plugin whose id is a core setting's key ('display', 'sync',
@@ -496,8 +496,7 @@ class StateReconciliation:
# Bring the state manager in sync with config rather than the reverse, # Bring the state manager in sync with config rather than the reverse,
# so that manual config edits (or the state left behind after an # so that manual config edits (or the state left behind after an
# uninstall+reinstall cycle) don't silently override the user's intent. # uninstall+reinstall cycle) don't silently override the user's intent.
# Always set for this type (see _check_plugin_consistency). config_enabled = inconsistency.expected_state.get('enabled')
config_enabled = cast(bool, inconsistency.expected_state.get('enabled'))
success = self.state_manager.set_plugin_enabled(inconsistency.plugin_id, config_enabled) success = self.state_manager.set_plugin_enabled(inconsistency.plugin_id, config_enabled)
if success: if success:
self.logger.info( self.logger.info(
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
-852
View File
@@ -1,852 +0,0 @@
"""Plugin store: the plugin registry, GitHub metadata, search and manifest
validation.
Part of PluginStoreManager (store_manager.py), which mixes this class in;
methods reach shared state and helpers through ``self``.
"""
import json
import requests
import time
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime
from pathlib import Path
from typing import List, Dict, Optional, Any
from jsonschema import Draft7Validator, ValidationError
from src.plugin_system.repo_urls import (
github_api_headers, github_owner_repo, normalize_repo_url,
)
class _RegistryMixin:
"""PluginStoreManager methods: see the module docstring."""
def _load_github_token(self) -> Optional[str]:
"""
Load GitHub API token from config_secrets.json if available.
Returns:
GitHub token or None if not configured
"""
try:
config_path = Path(__file__).parent.parent.parent / "config" / "config_secrets.json"
if config_path.exists():
with open(config_path, 'r', encoding='utf-8') as f:
config = json.load(f)
token = config.get('github', {}).get('api_token', '').strip()
# The config template's placeholder, not a credential.
if token and token != "YOUR_GITHUB_PERSONAL_ACCESS_TOKEN": # nosec B105 # nosemgrep
return token
except Exception as e:
self.logger.debug(f"Could not load GitHub token: {e}")
return None
def _validate_github_token(self, token: str) -> tuple[bool, Optional[str]]:
"""
Validate a GitHub token by making a lightweight API call.
Args:
token: GitHub personal access token to validate
Returns:
Tuple of (is_valid, error_message)
- is_valid: True if token is valid, False otherwise
- error_message: None if valid, error description if invalid
"""
if not token:
return (False, "No token provided")
# Check cache first
cache_key = token[:10] # Use first 10 chars as cache key for privacy
if cache_key in self._token_validation_cache:
cached_valid, cached_time, cached_error = self._token_validation_cache[cache_key]
if time.time() - cached_time < self._token_validation_cache_timeout:
return (cached_valid, cached_error)
# Validate token by making a lightweight API call to /user endpoint
try:
api_url = "https://api.github.com/user"
response = requests.get(api_url, headers=github_api_headers(token), timeout=5)
if response.status_code == 200:
# Token is valid
result = (True, None)
self._token_validation_cache[cache_key] = (True, time.time(), None)
return result
elif response.status_code == 401:
# Token is invalid or expired
error_msg = "Token is invalid or expired"
result = (False, error_msg)
self._token_validation_cache[cache_key] = (False, time.time(), error_msg)
return result
elif response.status_code == 403:
# Rate limit or forbidden (but token might be valid)
# Check if it's a rate limit issue
if 'rate limit' in response.text.lower():
# Rate limit: return error but don't cache (rate limits are temporary)
error_msg = "Rate limit exceeded"
result = (False, error_msg)
return result
else:
# Token lacks permissions: cache the result (permissions don't change)
error_msg = "Token lacks required permissions"
result = (False, error_msg)
self._token_validation_cache[cache_key] = (False, time.time(), error_msg)
return result
else:
# Other error
error_msg = f"GitHub API error: {response.status_code}"
result = (False, error_msg)
self._token_validation_cache[cache_key] = (False, time.time(), error_msg)
return result
except requests.exceptions.Timeout:
error_msg = "GitHub API request timed out"
result = (False, error_msg)
# Don't cache timeout errors
return result
except requests.exceptions.RequestException as e:
error_msg = f"Network error: {str(e)}"
result = (False, error_msg)
# Don't cache network errors
return result
except Exception as e:
error_msg = f"Unexpected error: {str(e)}"
result = (False, error_msg)
# Don't cache unexpected errors
return result
@staticmethod
def _iso_to_date(iso_timestamp: str) -> str:
"""Convert an ISO timestamp to YYYY-MM-DD string."""
if not iso_timestamp:
return ""
try:
dt = datetime.fromisoformat(iso_timestamp.replace('Z', '+00:00'))
return dt.strftime('%Y-%m-%d')
except Exception:
return ""
@staticmethod
def _distinct_sequence(values: List[str]) -> List[str]:
"""Return list preserving order while removing duplicates and falsey entries."""
seen = set()
ordered = []
for value in values:
if not value:
continue
if value in seen:
continue
seen.add(value)
ordered.append(value)
return ordered
def _validate_manifest_version_fields(self, manifest: Dict[str, Any]) -> List[str]:
"""
Validate version-related fields in manifest for consistency.
Checks:
- compatible_versions is present and is an array
- Standardized field names are used (min_ledmatrix_version, max_ledmatrix_version)
- Deprecated fields are not used (ledmatrix_version)
- versions array entries use ledmatrix_min_version instead of ledmatrix_min
Args:
manifest: Manifest dictionary to validate
Returns:
List of validation error/warning messages (empty if valid)
"""
errors = []
# Check compatible_versions is an array
if 'compatible_versions' in manifest:
if not isinstance(manifest['compatible_versions'], list):
errors.append("compatible_versions must be an array")
elif len(manifest['compatible_versions']) == 0:
errors.append("compatible_versions array cannot be empty")
# Warn about deprecated ledmatrix_version field
if 'ledmatrix_version' in manifest:
errors.append("ledmatrix_version is deprecated, use compatible_versions instead")
# Check versions array entries use standardized field names
if 'versions' in manifest and isinstance(manifest['versions'], list):
for i, version_entry in enumerate(manifest['versions']):
if not isinstance(version_entry, dict):
continue
# Check for old ledmatrix_min field
if 'ledmatrix_min' in version_entry and 'ledmatrix_min_version' not in version_entry:
errors.append(f"versions[{i}] uses deprecated 'ledmatrix_min', should use 'ledmatrix_min_version'")
return errors
def _validate_manifest_schema(self, manifest: Dict[str, Any], plugin_id: str) -> List[str]:
"""
Validate manifest against JSON schema if available.
Args:
manifest: Manifest dictionary to validate
plugin_id: Plugin ID for error messages
Returns:
List of validation error messages (empty if valid or schema unavailable)
"""
try:
# Load manifest schema
schema_path = Path(__file__).parent.parent.parent / "schema" / "manifest_schema.json"
if not schema_path.exists():
return [] # Schema not available, skip validation
with open(schema_path, 'r', encoding='utf-8') as f:
schema = json.load(f)
# Validate schema itself
Draft7Validator.check_schema(schema)
# Validate manifest against schema
validator = Draft7Validator(schema)
errors = []
for error in validator.iter_errors(manifest):
error_path = '.'.join(str(p) for p in error.path)
errors.append(f"{error_path}: {error.message}")
return errors
except json.JSONDecodeError as e:
self.logger.warning(f"Could not parse manifest schema: {e}")
return []
except ValidationError as e:
self.logger.warning(f"Manifest schema is invalid: {e}")
return []
except Exception as e:
self.logger.debug(f"Error validating manifest schema for {plugin_id}: {e}")
return []
_EMPTY_REPO_INFO: Dict[str, Any] = {
'stars': 0,
'forks': 0,
'open_issues': 0,
'updated_at_iso': '',
'last_commit_iso': '',
'last_commit_date': '',
'language': '',
'license': '',
'default_branch': 'main',
}
def _get_github_repo_info(self, repo_url: str) -> Dict[str, Any]:
"""GitHub metadata for a repository (stars, default branch, last push).
Returns zeroed defaults (``_EMPTY_REPO_INFO``) for a non-GitHub URL or
when GitHub cannot be asked and nothing is cached.
"""
try:
owner_repo = github_owner_repo(repo_url)
if owner_repo is None:
return dict(self._EMPTY_REPO_INFO)
owner, repo = owner_repo
cache_key = f"{owner}/{repo}"
if cache_key in self.github_cache:
cached_time, cached_data = self.github_cache[cache_key]
if time.time() - cached_time < self.cache_timeout:
return cached_data
api_url = f"https://api.github.com/repos/{owner}/{repo}"
try:
response = requests.get(
api_url, headers=github_api_headers(self.github_token), timeout=10)
except requests.RequestException as req_err:
# Network error: prefer a stale cache hit over an empty
# default so the UI keeps working on a flaky Pi WiFi link.
# Bump the cached entry's timestamp into a short backoff
# window so subsequent requests serve the stale payload
# cheaply instead of re-hitting the network on every request.
if cache_key in self.github_cache:
_, stale = self.github_cache[cache_key]
self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale)
self.logger.warning(
"GitHub repo info fetch failed for %s (%s); serving stale cache.",
cache_key, req_err,
)
return stale
raise
if response.status_code == 200:
data = response.json()
pushed_at = data.get('pushed_at', '') or data.get('updated_at', '')
repo_info = {
'stars': data.get('stargazers_count', 0),
'forks': data.get('forks_count', 0),
'open_issues': data.get('open_issues_count', 0),
'updated_at_iso': data.get('updated_at', ''),
'last_commit_iso': pushed_at,
'last_commit_date': self._iso_to_date(pushed_at),
'language': data.get('language', ''),
'license': data.get('license', {}).get('name', '') if data.get('license') else '',
'default_branch': data.get('default_branch', 'main')
}
self.github_cache[cache_key] = (time.time(), repo_info)
return repo_info
if response.status_code == 403:
# Rate limit or authentication issue. A stale star count is
# better than a reset to zero, and the backoff bump stops the
# store hammering the API while rate-limited.
if cache_key in self.github_cache:
_, stale = self.github_cache[cache_key]
self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale)
self.logger.warning(
"GitHub API 403 for %s; serving stale cache.", cache_key,
)
return stale
if not self.github_token:
self.logger.warning(
"GitHub API rate limit likely exceeded (403). "
"Add a GitHub personal access token to config/config_secrets.json "
"under 'github.api_token' to increase rate limits from 60 to 5000/hour."
)
else:
self.logger.warning(
f"GitHub API request failed: 403 for {api_url}. "
f"Your token may have insufficient permissions or rate limit exceeded."
)
else:
self.logger.warning(f"GitHub API request failed: {response.status_code} for {api_url}")
if cache_key in self.github_cache:
_, stale = self.github_cache[cache_key]
self._record_cache_backoff(self.github_cache, cache_key, self.cache_timeout, stale)
return stale
return dict(self._EMPTY_REPO_INFO)
except requests.exceptions.RequestException as e:
# Offline, DNS or a timeout reaching GitHub: the listing still
# works without the extra repo info, so this is not an error.
self.logger.warning("GitHub repo info unavailable for %s: %s", repo_url, e)
return dict(self._EMPTY_REPO_INFO)
except Exception as e:
self.logger.error(f"Error fetching GitHub repo info for {repo_url}: {e}")
return dict(self._EMPTY_REPO_INFO)
def _http_get_with_retries(self, url: str, *, timeout: int = 10, stream: bool = False, headers: Dict[str, str] = None, max_retries: int = 3, backoff_sec: float = 0.75):
"""
HTTP GET with simple retry strategy and exponential backoff.
Returns a requests.Response or raises the last exception.
"""
last_exc = None
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, timeout=timeout, stream=stream, headers=headers)
return resp
except requests.RequestException as e:
last_exc = e
self.logger.warning(f"HTTP GET failed (attempt {attempt}/{max_retries}) for {url}: {e}")
if attempt < max_retries:
time.sleep(backoff_sec * attempt)
# Exhausted retries
raise last_exc
def fetch_registry_from_url(self, repo_url: str) -> Optional[Dict]:
"""
Fetch a registry-style plugins.json from a custom GitHub repository URL.
This allows users to point to a registry-style monorepo (like the official
ledmatrix-plugins repo) and browse/install plugins from it.
Args:
repo_url: GitHub repository URL (e.g., https://github.com/user/ledmatrix-plugins)
Returns:
Registry dict with plugins list, or None if not found/invalid
"""
try:
repo_url = normalize_repo_url(repo_url)
# plugins.json or registry.json at the root of main, then master.
registry_urls = []
owner_repo = github_owner_repo(repo_url)
if owner_repo is not None:
owner, repo = owner_repo
for branch in ['main', 'master']:
registry_urls.append(f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/plugins.json")
registry_urls.append(f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/registry.json")
for url in registry_urls:
try:
response = self._http_get_with_retries(url, timeout=10)
if response.status_code == 200:
registry = response.json()
# Validate it looks like a registry
if isinstance(registry, dict) and 'plugins' in registry:
self.logger.info(f"Successfully fetched registry from {url}")
return registry
except Exception as e:
self.logger.debug(f"Failed to fetch from {url}: {e}")
continue
self.logger.warning(f"No valid registry found at {repo_url}")
return None
except Exception as e:
self.logger.error(f"Error fetching registry from URL: {e}", exc_info=True)
return None
def fetch_registry(self, force_refresh: bool = False, raise_on_failure: bool = False) -> Dict:
"""
Fetch the plugin registry from GitHub.
Args:
force_refresh: Force refresh even if cached
raise_on_failure: If True, re-raise network / JSON errors instead
of silently falling back to stale cache / empty dict. UI
callers prefer the stale-fallback default so the plugin
list keeps working on flaky WiFi; the state reconciler
needs the explicit failure signal so it can distinguish
"plugin genuinely not in registry" from "I couldn't reach
the registry at all" and not mark everything unrecoverable.
Returns:
Registry data with list of available plugins
Raises:
requests.RequestException / json.JSONDecodeError when
``raise_on_failure`` is True and the fetch fails.
"""
# Check if cache is still valid (within timeout)
current_time = time.time()
if (self.registry_cache and self.registry_cache_time and
not force_refresh and
(current_time - self.registry_cache_time) < self.registry_cache_timeout):
return self.registry_cache
with self._registry_fetch_lock:
# Re-check inside the lock — a concurrent caller that was waiting
# may have already populated the cache while we blocked.
current_time = time.time()
if (self.registry_cache and self.registry_cache_time and
not force_refresh and
(current_time - self.registry_cache_time) < self.registry_cache_timeout):
return self.registry_cache
try:
self.logger.info(f"Fetching plugin registry from {self.REGISTRY_URL}")
response = self._http_get_with_retries(self.REGISTRY_URL, timeout=10)
response.raise_for_status()
self.registry_cache = response.json()
self.registry_cache_time = current_time
self.logger.info(f"Fetched registry with {len(self.registry_cache.get('plugins', []))} plugins")
return self.registry_cache
except requests.RequestException as e:
self.logger.error(f"Error fetching registry: {e}")
if raise_on_failure:
raise
# Prefer stale cache over an empty list so the plugin list UI
# keeps working on a flaky connection (e.g. Pi on WiFi). Bump
# registry_cache_time into a short backoff window so the next
# request serves the stale payload cheaply instead of
# re-hitting the network on every request (matches the
# pattern used by github_cache / commit_info_cache).
if self.registry_cache:
self.logger.warning("Falling back to stale registry cache")
self.registry_cache_time = (
time.time() + self._failure_backoff_seconds - self.registry_cache_timeout
)
return self.registry_cache
return {"plugins": []}
except json.JSONDecodeError as e:
self.logger.error(f"Error parsing registry JSON: {e}")
if raise_on_failure:
raise
if self.registry_cache:
self.registry_cache_time = (
time.time() + self._failure_backoff_seconds - self.registry_cache_timeout
)
return self.registry_cache
return {"plugins": []}
def search_plugins(self, query: str = "", category: str = "", tags: List[str] = None, fetch_commit_info: bool = True, include_saved_repos: bool = True, saved_repositories_manager = None) -> List[Dict]:
"""
Search for plugins in the registry with enhanced metadata.
GitHub supplies live metadata such as stars and last commit
timestamps; the registry supplies descriptive information (name,
description, repo URL, etc.).
Args:
query: Search query string (searches name, description, id, author)
category: Filter by category (e.g., 'sports', 'weather', 'time')
tags: Filter by tags (matches any tag in list)
fetch_commit_info: If True (default), fetch commit metadata from GitHub.
include_saved_repos: If True (default), also search the
registry-style repositories the user saved.
saved_repositories_manager: The SavedRepositoriesManager holding
those repositories; without it only the official registry is
searched.
Returns:
List of matching plugin metadata enriched with GitHub information
"""
if tags is None:
tags = []
# Fetch from official registry
registry = self.fetch_registry()
plugins = registry.get('plugins', []) or []
# Also fetch from saved repositories if enabled
if include_saved_repos and saved_repositories_manager:
saved_repos = saved_repositories_manager.get_registry_repositories()
for repo_info in saved_repos:
repo_url = repo_info.get('url')
if repo_url:
try:
custom_registry = self.fetch_registry_from_url(repo_url)
if custom_registry:
custom_plugins = custom_registry.get('plugins', []) or []
# Mark these as from custom repository
for plugin in custom_plugins:
plugin['_source'] = 'custom_repository'
plugin['_repository_url'] = repo_url
plugin['_repository_name'] = repo_info.get('name', repo_url)
plugins.extend(custom_plugins)
except Exception as e:
self.logger.warning(f"Failed to fetch plugins from saved repository {repo_url}: {e}")
# First pass: apply cheap filters (category/tags/query) so we only
# fetch GitHub metadata for plugins that will actually be returned.
filtered: List[Dict] = []
for plugin in plugins:
if category and plugin.get('category') != category:
continue
if tags and not any(tag in plugin.get('tags', []) for tag in tags):
continue
if query:
query_lower = query.lower()
searchable_text = ' '.join([
plugin.get('name', ''),
plugin.get('description', ''),
plugin.get('id', ''),
plugin.get('author', ''),
]).lower()
if query_lower not in searchable_text:
continue
filtered.append(plugin)
def _enrich(plugin: Dict) -> Dict:
"""Enrich a single plugin with GitHub metadata.
Called concurrently from a ThreadPoolExecutor. Both HTTP helpers
(``_get_github_repo_info`` / ``_get_latest_commit_info``) are
thread-safe -- they use ``requests`` and write their own cache
keys on Python dicts, which is atomic under the GIL for
single-key assignments.
"""
enhanced_plugin = plugin.copy()
repo_url = plugin.get('repo', '')
if not repo_url:
return enhanced_plugin
github_info = self._get_github_repo_info(repo_url)
enhanced_plugin['stars'] = github_info.get('stars', plugin.get('stars', 0))
enhanced_plugin['default_branch'] = github_info.get('default_branch', plugin.get('branch', 'main'))
enhanced_plugin['last_updated_iso'] = github_info.get('last_commit_iso')
enhanced_plugin['last_updated'] = github_info.get('last_commit_date')
if fetch_commit_info:
branch = plugin.get('branch') or github_info.get('default_branch', 'main')
commit_info = self._get_latest_commit_info(repo_url, branch)
if commit_info:
enhanced_plugin['last_commit'] = commit_info.get('short_sha')
enhanced_plugin['last_commit_sha'] = commit_info.get('sha')
enhanced_plugin['last_updated'] = commit_info.get('date') or enhanced_plugin.get('last_updated')
enhanced_plugin['last_updated_iso'] = commit_info.get('date_iso') or enhanced_plugin.get('last_updated_iso')
enhanced_plugin['last_commit_message'] = commit_info.get('message')
enhanced_plugin['last_commit_author'] = commit_info.get('author')
enhanced_plugin['branch'] = commit_info.get('branch', branch)
enhanced_plugin['last_commit_branch'] = commit_info.get('branch')
# Intentionally NO per-plugin manifest.json fetch here.
# The registry's plugins.json already carries ``description``
# (it is generated from each plugin's manifest by
# ``update_registry.py``), and ``last_updated`` is filled in
# from the commit info above. Fetching manifest.json per
# plugin costs one extra HTTPS round trip per result; on a Pi4
# with a flaky WiFi link the tail retries of that one call
# (_http_get_with_retries does 3 attempts with exponential
# backoff) dominate wall time even with the thread pool.
return enhanced_plugin
# Fan out the per-plugin GitHub enrichment. Serially, a Pi4 with ~15
# plugins and a cold cache makes 30+ HTTP requests in strict sequence
# (the "connecting to display" hang users reported). With a thread
# pool, latency is dominated by the slowest request rather than
# their sum. Workers capped at 10 to stay well under the
# unauthenticated GitHub rate limit burst and avoid overwhelming a
# Pi's WiFi link.
if not filtered:
return []
# Not worth the pool overhead for tiny workloads. Parenthesized to
# make Python's default ``and`` > ``or`` precedence explicit: a
# single plugin, OR a small batch where we don't need commit info.
if (len(filtered) == 1) or ((not fetch_commit_info) and (len(filtered) < 4)):
return [_enrich(p) for p in filtered]
max_workers = min(10, len(filtered))
with ThreadPoolExecutor(max_workers=max_workers, thread_name_prefix='plugin-search') as executor:
# executor.map preserves input order, which the UI relies on.
return list(executor.map(_enrich, filtered))
def _fetch_manifest_from_github(self, repo_url: str, branch: str = "master", manifest_path: str = "manifest.json", force_refresh: bool = False) -> Optional[Dict]:
"""
Fetch manifest.json directly from a GitHub repository.
Args:
repo_url: GitHub repository URL
branch: Branch name (default: master)
manifest_path: Path to manifest within the repo (default: manifest.json).
For monorepo plugins this will be e.g. "plugins/football-scoreboard/manifest.json".
force_refresh: If True, bypass the cache.
Returns:
Manifest data or None if not found
"""
try:
owner_repo = github_owner_repo(repo_url)
if owner_repo is None:
return None
owner, repo = owner_repo
cache_key = f"{owner}/{repo}:{branch}:{manifest_path}"
if not force_refresh and cache_key in self.manifest_cache:
cached_time, cached_data = self.manifest_cache[cache_key]
if time.time() - cached_time < self.manifest_cache_timeout:
return cached_data
raw_url = f"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{manifest_path}"
response = self._http_get_with_retries(raw_url, timeout=10)
if response.status_code == 200:
result = response.json()
self.manifest_cache[cache_key] = (time.time(), result)
return result
if response.status_code == 404 and branch != "main":
raw_url = f"https://raw.githubusercontent.com/{owner}/{repo}/main/{manifest_path}"
response = self._http_get_with_retries(raw_url, timeout=10)
if response.status_code == 200:
result = response.json()
self.manifest_cache[cache_key] = (time.time(), result)
return result
# Cache the miss too, so a plugin without a manifest at this path
# is not re-fetched on every browse.
self.manifest_cache[cache_key] = (time.time(), None)
except Exception as e:
self.logger.debug(f"Could not fetch manifest from GitHub for {repo_url}: {e}")
return None
def _get_latest_commit_info(self, repo_url: str, branch: str = "main", force_refresh: bool = False) -> Optional[Dict[str, Any]]:
"""Return metadata about the latest commit on the given branch."""
try:
owner_repo = github_owner_repo(repo_url)
if owner_repo is None:
return None
owner, repo = owner_repo
cache_key = f"{owner}/{repo}:{branch}"
if not force_refresh and cache_key in self.commit_info_cache:
cached_time, cached_data = self.commit_info_cache[cache_key]
if time.time() - cached_time < self.commit_cache_timeout:
return cached_data
branches_to_try = self._distinct_sequence([branch, 'main', 'master'])
headers = github_api_headers(self.github_token)
last_error = None
for branch_name in branches_to_try:
api_url = f"https://api.github.com/repos/{owner}/{repo}/commits/{branch_name}"
try:
response = requests.get(api_url, headers=headers, timeout=10)
except requests.RequestException as req_err:
# Network failure: fall back to a stale cache hit if
# available so the plugin store UI keeps populating
# commit info on a flaky WiFi link. Bump the cached
# timestamp into the backoff window so we don't
# re-retry on every request.
if cache_key in self.commit_info_cache:
_, stale = self.commit_info_cache[cache_key]
if stale is not None:
self._record_cache_backoff(
self.commit_info_cache, cache_key,
self.commit_cache_timeout, stale,
)
self.logger.warning(
"GitHub commit fetch failed for %s (%s); serving stale cache.",
cache_key, req_err,
)
return stale
last_error = str(req_err)
continue
if response.status_code == 200:
commit_data = response.json()
commit_sha_full = commit_data.get('sha', '')
commit_sha_short = commit_sha_full[:7] if commit_sha_full else ''
commit_meta = commit_data.get('commit', {})
commit_author = commit_meta.get('author', {})
commit_date_iso = commit_author.get('date', '')
result = {
'branch': branch_name,
'sha': commit_sha_full,
'short_sha': commit_sha_short,
'date_iso': commit_date_iso,
'date': self._iso_to_date(commit_date_iso),
'author': commit_author.get('name', ''),
'message': commit_meta.get('message', ''),
}
self.commit_info_cache[cache_key] = (time.time(), result)
return result
if response.status_code == 403 and not self.github_token:
self.logger.debug("GitHub commit API rate limited (403). Consider adding a token.")
last_error = response.text
else:
last_error = response.text
if last_error:
self.logger.debug(f"Unable to fetch commit info for {repo_url}: {last_error}")
# All branches returned a non-200 response (e.g. 404 on every
# candidate, or a transient 5xx). If we already had a good
# cached value, prefer serving that — overwriting it with
# None here would wipe out commit info the UI just showed
# on the previous request. Bump the timestamp into the
# backoff window so subsequent lookups hit the cache.
if cache_key in self.commit_info_cache:
_, prior = self.commit_info_cache[cache_key]
if prior is not None:
self._record_cache_backoff(
self.commit_info_cache, cache_key,
self.commit_cache_timeout, prior,
)
return prior
# No prior good value — cache the negative result so we don't
# hammer a plugin that genuinely has no reachable commits.
self.commit_info_cache[cache_key] = (time.time(), None)
except Exception as e:
self.logger.debug(f"Error fetching latest commit metadata for {repo_url}: {e}")
return None
def get_plugin_info(self, plugin_id: str, fetch_latest_from_github: bool = True, force_refresh: bool = False) -> Optional[Dict]:
"""
Get detailed information about a plugin from the registry.
GitHub provides authoritative metadata such as stars and the latest
commit. The registry supplies descriptive information (name, id, repo URL).
Args:
plugin_id: Plugin identifier
fetch_latest_from_github: If True (default), augment with GitHub commit metadata.
force_refresh: If True, bypass caches for commit/manifest data.
Returns:
Plugin metadata or None if not found
"""
registry = self.fetch_registry()
plugins = registry.get('plugins', []) or []
plugin_info = self._match_registry_entry(plugins, plugin_id)
if not plugin_info:
return None
if fetch_latest_from_github:
repo_url = plugin_info.get('repo')
if repo_url:
plugin_info = plugin_info.copy()
github_info = self._get_github_repo_info(repo_url)
branch = plugin_info.get('branch') or github_info.get('default_branch', 'main')
plugin_info['default_branch'] = github_info.get('default_branch', branch)
plugin_info['stars'] = github_info.get('stars', plugin_info.get('stars', 0))
plugin_info['last_updated'] = github_info.get('last_commit_date', plugin_info.get('last_updated'))
plugin_info['last_updated_iso'] = github_info.get('last_commit_iso', plugin_info.get('last_updated_iso'))
commit_info = self._get_latest_commit_info(repo_url, branch, force_refresh=force_refresh)
if commit_info:
plugin_info['last_commit'] = commit_info.get('short_sha')
plugin_info['last_commit_sha'] = commit_info.get('sha')
plugin_info['last_commit_message'] = commit_info.get('message')
plugin_info['last_commit_author'] = commit_info.get('author')
plugin_info['last_updated'] = commit_info.get('date') or plugin_info.get('last_updated')
plugin_info['last_updated_iso'] = commit_info.get('date_iso') or plugin_info.get('last_updated_iso')
plugin_info['branch'] = commit_info.get('branch', branch)
plugin_info['last_commit_branch'] = commit_info.get('branch')
plugin_subpath = plugin_info.get('plugin_path', '')
manifest_rel = f"{plugin_subpath}/manifest.json" if plugin_subpath else "manifest.json"
github_manifest = self._fetch_manifest_from_github(repo_url, branch, manifest_rel, force_refresh=force_refresh)
if github_manifest:
if 'last_updated' in github_manifest and not plugin_info.get('last_updated'):
plugin_info['last_updated'] = github_manifest['last_updated']
if 'description' in github_manifest:
plugin_info['description'] = github_manifest['description']
return plugin_info
@staticmethod
def _match_registry_entry(plugins: List[Dict], plugin_id: str) -> Optional[Dict]:
"""Find a registry entry by its id, or by the directory it installs to.
Four shipped plugins have a registry ``id`` that differs from the ``id``
in their own manifest: ``weather`` installs to ``plugins/ledmatrix-weather``,
and likewise stocks, music and leaderboard. Installation already prefers
the manifest id for the directory name, so on disk, in ``config.json``
and in a backup manifest those plugins are called ``ledmatrix-weather``.
Only the registry calls them ``weather``, and nothing resolved that in
reverse: restoring a backup asked the store for ``ledmatrix-weather``
and got "Plugin not found in registry", silently dropping four enabled
plugins from a restored device.
Matching ``plugin_path`` fixes it without renaming any published id,
which would orphan ``plugin_state.json`` entries keyed on the old ones.
Exact id always wins, so an entry whose *path* happens to collide with
another entry's id cannot shadow it.
"""
if not plugin_id:
return None
exact = next((p for p in plugins if p.get('id') == plugin_id), None)
if exact is not None:
return exact
for entry in plugins:
path = (entry.get('plugin_path') or '').rstrip('/')
if path and path.rsplit('/', 1)[-1] == plugin_id:
return entry
return None
def get_registry_info(self, plugin_id: str) -> Optional[Dict]:
"""
Get plugin information from the registry cache only (no GitHub API calls).
Use this for lightweight lookups where only registry fields are needed
(e.g., verified status, latest_version).
Args:
plugin_id: Plugin identifier
Returns:
Plugin metadata from registry or None if not found
"""
registry = self.fetch_registry()
plugins = registry.get('plugins', []) or []
return self._match_registry_entry(plugins, plugin_id)
-732
View File
@@ -1,732 +0,0 @@
"""Plugin store: updating installed plugins, with rollback, and reading their
local git state.
Part of PluginStoreManager (store_manager.py), which mixes this class in;
methods reach shared state and helpers through ``self``.
"""
import json
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
from pathlib import Path
from typing import Dict, Optional, Tuple
from src.plugin_system.plugin_dirs import BACKUP_MARKER
from src.plugin_system.repo_urls import same_repo
class _UpdateMixin:
"""PluginStoreManager methods: see the module docstring."""
def _git_cache_signature(self, git_dir: Path) -> Optional[Tuple]:
"""Build a cache signature that invalidates on the kind of updates
a plugin user actually cares about.
Caching on ``.git/HEAD`` mtime alone is not enough: a ``git pull``
that fast-forwards the current branch updates
``.git/refs/heads/<branch>`` (or ``.git/packed-refs``) but leaves
HEAD's contents and mtime untouched. And the cached ``result``
dict includes ``remote_url`` — a value read from ``.git/config`` —
so a config-only change (e.g. a monorepo-migration re-pointing
``remote.origin.url``) must also invalidate the cache.
Signature components:
- HEAD contents (catches detach / branch switch)
- HEAD mtime
- if HEAD points at a ref, that ref file's mtime (catches
fast-forward / reset on the current branch)
- packed-refs mtime as a coarse fallback for repos using packed refs
- .git/config contents + mtime (catches remote URL changes and
any other config-only edit that affects what the cached
``remote_url`` field should contain)
Returns ``None`` if HEAD cannot be read at all (caller will skip
the cache and take the slow path).
"""
head_file = git_dir / 'HEAD'
try:
head_mtime = head_file.stat().st_mtime
head_contents = head_file.read_text(encoding='utf-8', errors='replace').strip()
except OSError:
return None
ref_mtime = None
if head_contents.startswith('ref: '):
ref_path = head_contents[len('ref: '):].strip()
# ``ref_path`` looks like ``refs/heads/main``. It lives either
# as a loose file under .git/ or inside .git/packed-refs.
loose_ref = git_dir / ref_path
try:
ref_mtime = loose_ref.stat().st_mtime
except OSError:
ref_mtime = None
packed_refs_mtime = None
if ref_mtime is None:
try:
packed_refs_mtime = (git_dir / 'packed-refs').stat().st_mtime
except OSError:
packed_refs_mtime = None
config_mtime = None
config_contents = None
config_file = git_dir / 'config'
try:
config_mtime = config_file.stat().st_mtime
config_contents = config_file.read_text(encoding='utf-8', errors='replace').strip()
except OSError:
config_mtime = None
config_contents = None
return (
head_contents, head_mtime,
ref_mtime, packed_refs_mtime,
config_contents, config_mtime,
)
def _get_local_git_info(self, plugin_path: Path) -> Optional[Dict[str, str]]:
"""Return local git branch, commit hash, and commit date if the plugin is a git checkout.
Results are cached keyed on a signature that includes HEAD
contents plus the mtime of HEAD AND the resolved ref (or
packed-refs). Repeated calls skip the ``git log`` subprocess when
nothing has changed, and a ``git pull`` that fast-forwards the
branch correctly invalidates the cache.
"""
git_dir = plugin_path / '.git'
if not git_dir.exists():
return None
cache_key = str(plugin_path)
signature = self._git_cache_signature(git_dir)
if signature is not None:
cached = self._git_info_cache.get(cache_key)
if cached is not None and cached[0] == signature:
return cached[1]
try:
# .git may be a file (worktree / submodule) containing "gitdir: <path>".
# Resolve it to the actual git directory before reading any files.
try:
if git_dir.is_file():
pointer = git_dir.read_text(encoding='utf-8', errors='replace').strip()
if pointer.startswith('gitdir:'):
resolved = (plugin_path / pointer[len('gitdir:'):].strip()).resolve()
if resolved.is_dir():
git_dir = resolved
else:
return None
else:
return None
except (OSError, NotADirectoryError):
return None
# Read branch directly from .git/HEAD (no subprocess).
branch = ''
try:
head_text = (git_dir / 'HEAD').read_text(encoding='utf-8', errors='replace').strip()
if head_text.startswith('ref: refs/heads/'):
branch = head_text[len('ref: refs/heads/'):]
elif head_text.startswith('ref: '):
branch = head_text[len('ref: '):]
# else: detached HEAD — branch stays ''
except (OSError, NotADirectoryError):
pass
# Remote URL from .git/config — parse [remote "origin"] url line.
remote_url = None
try:
config_text = (git_dir / 'config').read_text(encoding='utf-8', errors='replace')
in_origin = False
for line in config_text.splitlines():
stripped = line.strip()
if stripped == '[remote "origin"]':
in_origin = True
elif stripped.startswith('['):
in_origin = False
elif in_origin and stripped.startswith('url') and '=' in stripped:
remote_url = stripped.split('=', 1)[1].strip()
break
except (OSError, NotADirectoryError):
pass
# Single subprocess: SHA + commit date in one call.
log_result = subprocess.run(
['git', '-C', str(plugin_path), 'log', '-1', '--format=%H%n%cI', 'HEAD'],
capture_output=True,
text=True,
timeout=10,
check=True
)
lines = log_result.stdout.strip().splitlines()
sha = lines[0] if lines else ''
commit_date_iso = lines[1] if len(lines) > 1 else ''
result = {
'sha': sha,
'short_sha': sha[:7] if sha else '',
'branch': branch,
}
if remote_url:
result['remote_url'] = remote_url
if commit_date_iso:
result['date_iso'] = commit_date_iso
result['date'] = self._iso_to_date(commit_date_iso)
if signature is not None:
self._git_info_cache[cache_key] = (signature, result)
return result
except subprocess.CalledProcessError as err:
self.logger.debug(f"Failed to read git info for {plugin_path.name}: {err}")
except subprocess.TimeoutExpired:
self.logger.debug(f"Timed out reading git info for {plugin_path.name}")
return None
def _gate_pulled_commit(self, plugin_id: str, plugin_path: Path,
previous_sha: Optional[str]) -> bool:
"""Apply the compatibility gate to a commit that arrived via git pull.
Every other route into an installed plugin goes through
``install_plugin``, which gates in ``_install_plugin_impl``. This one
did not: a ``git pull`` could deliver a manifest flooring above this
core and nothing would notice until the plugin failed to load, which
surfaces as one line in the journal and a scoreboard that silently
stopped appearing.
Checked after the pull rather than before it, for the same reason
``_install_plugin_impl`` checks after the download: the registry
carries no compatibility field, so the incoming floor is only knowable
once the new commit is on disk.
Undone with ``git reset --hard`` rather than by removing the directory.
This is a live checkout, the previous commit is still in the object
store, and the reset leaves the user on the exact version they were
already running -- the same promise ``_reinstall_with_rollback`` makes,
reached by the means this path actually has. It is also the gentler
option: no window in which the plugin directory does not exist, and no
``.standalone-backup-`` debris if the process dies mid-way.
A manifest that cannot be read is not evidence of incompatibility, so
it allows. ``compatibility.check`` refuses only on evidence for the
same reason: a wrong refusal breaks a working install, while a wrong
allowance degrades to exactly the behaviour this path had before the
gate existed.
"""
manifest_path = plugin_path / "manifest.json"
try:
with open(manifest_path, 'r', encoding='utf-8') as mf:
manifest = json.load(mf)
except (OSError, ValueError) as e:
self.logger.warning(
"Could not read %s after updating %s (%s); allowing the "
"update, as an unreadable manifest declares no floor",
manifest_path, plugin_id, e)
return True
from src.plugin_system import compatibility
core_version = compatibility.current_core_version()
compatible, reason = compatibility.check(manifest, core_version)
if compatible:
return True
self.logger.error("Refusing the update to %s: %s", plugin_id, reason)
if not previous_sha:
self.logger.error(
"Cannot roll %s back: the commit it was on before the pull is "
"unknown. It is now on a version this core cannot run — "
"reinstall it from the plugin store.", plugin_id)
return False
# Safe by construction: update_plugin returns before pulling unless the
# tree was clean or successfully stashed, so there are no uncommitted
# tracked edits for --hard to discard. The stash is not popped on the
# success path either, so the reset leaves the working tree exactly
# where a successful pull would have. Say "commit", not "changes".
reset = subprocess.run(
['git', '-C', str(plugin_path), 'reset', '--hard', previous_sha],
capture_output=True, text=True, timeout=60, check=False)
if reset.returncode != 0:
self.logger.error(
"CRITICAL: could not roll %s back to commit %s: %s. It is left "
"on a version this core cannot run; "
"`git -C %s reset --hard %s` restores it.",
plugin_id, previous_sha[:7],
(reset.stderr or reset.stdout or '').strip(),
plugin_path, previous_sha)
else:
self.logger.info(
"Rolled %s back to commit %s; it stays on the version it was "
"already running.", plugin_id, previous_sha[:7])
return False
def _reinstall_with_rollback(self, plugin_id: str, plugin_path: Path) -> bool:
"""Replace an installed plugin with a fresh install, atomically.
The old install is renamed aside (not deleted) until the new install
succeeds, then removed; on ANY install failure the old directory is
restored. Deleting first turns a failed download into a destroyed
plugin: during the monorepo migration a Pi with broken DNS lost every
old-remote plugin that way, with none able to be re-downloaded.
The aside name embeds BACKUP_MARKER ('.standalone-backup-') so every
plugin directory lookup (src/plugin_system/plugin_dirs.py) ignores it
even though it still contains a manifest.json.
Held for the whole operation under a per-plugin_id lock: two
overlapping requests for the same plugin (double-click, two
browser tabs — the web UI runs Flask with threaded=True) must not
interleave their renames, or the second could steal the first's
rollback safety net mid-install. Other plugin_ids are unaffected.
"""
with self._get_reinstall_lock(plugin_id):
backup_path = plugin_path.with_name(
f"{plugin_path.name}{BACKUP_MARKER}migrating")
problem = self._set_aside(plugin_path, backup_path)
if problem:
self.logger.error(
"Not updating %s: %s; the installed version is left in place",
plugin_id, problem)
return False
try:
installed = self.install_plugin(plugin_id)
except Exception as e:
self.logger.error(f"Reinstall of {plugin_id} raised: {e}")
installed = False
if installed:
self._discard_backup(plugin_id, backup_path, "update")
return True
# Bad network, registry error...: the user keeps a working plugin.
self._restore_backup(plugin_id, plugin_path, backup_path, "Reinstall")
return False
def update_plugin(self, plugin_id: str) -> bool:
"""
Update a plugin to the latest commit on its upstream branch.
"""
plugin_path = self._find_plugin_path(plugin_id)
if plugin_path is None or not plugin_path.exists():
self.logger.error(f"Plugin not installed: {plugin_id}")
return False
try:
self.logger.info(f"Checking for updates to plugin {plugin_id}")
# Check if this is a bundled/unmanaged plugin (no registry entry, no git remote)
# These are plugins shipped with LEDMatrix itself and updated via LEDMatrix updates.
metadata_path = plugin_path / ".plugin_metadata.json"
if metadata_path.exists():
try:
with open(metadata_path, 'r', encoding='utf-8') as f:
metadata = json.load(f)
if metadata.get('install_type') == 'bundled':
self.logger.info(f"Plugin {plugin_id} is a bundled plugin; updates are delivered via LEDMatrix itself")
return True
except (OSError, ValueError) as e:
self.logger.debug(f"[PluginStore] Could not read metadata for {plugin_id} at {metadata_path}: {e}")
# First check if it's a git repository - if so, we can update directly
git_info = self._get_local_git_info(plugin_path)
if git_info:
# Plugin is a git repository - try to update via git
local_branch = git_info.get('branch') or 'main'
local_sha = git_info.get('sha')
# Try to get remote info from registry (optional)
self.fetch_registry(force_refresh=True)
plugin_info_remote = self.get_plugin_info(plugin_id, fetch_latest_from_github=True, force_refresh=True)
# Try without 'ledmatrix-' prefix (monorepo migration)
resolved_id = plugin_id
if not plugin_info_remote and plugin_id.startswith('ledmatrix-'):
alt_id = plugin_id[len('ledmatrix-'):]
plugin_info_remote = self.get_plugin_info(alt_id, fetch_latest_from_github=True, force_refresh=True)
if plugin_info_remote:
resolved_id = alt_id
self.logger.info(f"Plugin {plugin_id} found in registry as {resolved_id}")
remote_branch = None
remote_sha = None
if plugin_info_remote:
remote_branch = plugin_info_remote.get('branch') or plugin_info_remote.get('default_branch')
remote_sha = plugin_info_remote.get('last_commit_sha')
# Check if the local git remote still matches the registry repo URL.
# After monorepo migration, old clones point to archived individual repos
# while the registry now points to the monorepo. Detect this and reinstall.
registry_repo = plugin_info_remote.get('repo', '')
local_remote = git_info.get('remote_url', '')
if local_remote and registry_repo and not same_repo(local_remote, registry_repo):
self.logger.info(
f"Plugin {resolved_id} git remote ({local_remote}) differs from registry ({registry_repo}). "
f"Reinstalling from registry to migrate to new source."
)
return self._reinstall_with_rollback(resolved_id, plugin_path)
# Check if already up to date
if remote_sha and local_sha and remote_sha.startswith(local_sha):
self.logger.info(f"Plugin {plugin_id} already matches remote commit {remote_sha[:7]}")
return True
# Update via git pull
self.logger.info(f"Updating {plugin_id} via git pull (local branch: {local_branch})...")
try:
# Fetch latest changes first to get all remote branch info
# If fetch fails, we'll still try to pull (might work with existing remote refs)
fetch_result = subprocess.run(
['git', '-C', str(plugin_path), 'fetch', 'origin'],
capture_output=True,
text=True,
timeout=60,
check=False
)
if fetch_result.returncode != 0:
self.logger.warning(f"Git fetch failed for {plugin_id}: {fetch_result.stderr or fetch_result.stdout}. Will still attempt pull.")
else:
self.logger.debug(f"Successfully fetched remote changes for {plugin_id}")
# Determine which remote branch to pull from
# Strategy: Use what the local branch is tracking, or find the best match
remote_pull_branch = None
# First, check what the local branch is tracking
tracking_result = subprocess.run(
['git', '-C', str(plugin_path), 'rev-parse', '--abbrev-ref', '--symbolic-full-name', f'{local_branch}@{{upstream}}'],
capture_output=True,
text=True,
timeout=10,
check=False
)
if tracking_result.returncode == 0 and tracking_result.stdout.strip():
# Local branch is tracking a remote branch
tracking_ref = tracking_result.stdout.strip()
# Extract branch name from refs/remotes/origin/branch-name or origin/branch-name
if tracking_ref.startswith('refs/remotes/origin/'):
remote_pull_branch = tracking_ref.replace('refs/remotes/origin/', '')
self.logger.info(f"Local branch {local_branch} is tracking origin/{remote_pull_branch}")
elif tracking_ref.startswith('origin/'):
remote_pull_branch = tracking_ref.replace('origin/', '')
self.logger.info(f"Local branch {local_branch} is tracking origin/{remote_pull_branch}")
# If not tracking anything, try to find the best remote branch match
if not remote_pull_branch:
# Check if remote branch from registry exists
if remote_branch:
remote_check = subprocess.run(
['git', '-C', str(plugin_path), 'ls-remote', '--heads', 'origin', remote_branch],
capture_output=True,
text=True,
timeout=10,
check=False
)
if remote_check.returncode == 0 and remote_check.stdout.strip():
remote_pull_branch = remote_branch
self.logger.info(f"Using remote branch {remote_branch} from registry")
# If registry branch doesn't exist, check if local branch name exists on remote
if not remote_pull_branch:
local_as_remote_check = subprocess.run(
['git', '-C', str(plugin_path), 'ls-remote', '--heads', 'origin', local_branch],
capture_output=True,
text=True,
timeout=10,
check=False
)
if local_as_remote_check.returncode == 0 and local_as_remote_check.stdout.strip():
remote_pull_branch = local_branch
self.logger.info(f"Using local branch name {local_branch} as remote branch")
# Last resort: try to get remote's default branch
if not remote_pull_branch:
default_branch_result = subprocess.run(
['git', '-C', str(plugin_path), 'symbolic-ref', 'refs/remotes/origin/HEAD'],
capture_output=True,
text=True,
timeout=10,
check=False
)
if default_branch_result.returncode == 0:
default_ref = default_branch_result.stdout.strip()
if default_ref.startswith('refs/remotes/origin/'):
remote_pull_branch = default_ref.replace('refs/remotes/origin/', '')
self.logger.info(f"Using remote default branch {remote_pull_branch}")
# If we still don't have a remote branch, use local branch name (git will handle it)
if not remote_pull_branch:
remote_pull_branch = local_branch
self.logger.info(f"Falling back to local branch name {local_branch} for pull")
# Ensure we're on the local branch
checkout_result = subprocess.run(
['git', '-C', str(plugin_path), 'checkout', local_branch],
capture_output=True,
text=True,
timeout=30,
check=False
)
if checkout_result.returncode != 0:
self.logger.warning(f"Git checkout to {local_branch} failed for {plugin_id}: {checkout_result.stderr or checkout_result.stdout}. Will still attempt pull.")
# Check for local changes and untracked files that might conflict
# First, check for untracked files that would be overwritten
try:
# Check for untracked files
untracked_result = subprocess.run(
['git', '-C', str(plugin_path), 'status', '--porcelain', '--untracked-files=all'],
capture_output=True,
text=True,
timeout=30,
check=False
)
untracked_files = []
if untracked_result.returncode == 0:
for line in untracked_result.stdout.strip().split('\n'):
if line.startswith('??'):
# Untracked file
file_path = line[3:].strip()
untracked_files.append(file_path)
# Check for tracked file changes
status_result = subprocess.run(
['git', '-C', str(plugin_path), 'status', '--porcelain', '--untracked-files=no'],
capture_output=True,
text=True,
timeout=30,
check=False
)
has_changes = bool(status_result.stdout.strip())
# If there are untracked files, stash them
if untracked_files:
self.logger.info(f"Found {len(untracked_files)} untracked files in {plugin_id}, will stash them")
has_changes = True
except subprocess.TimeoutExpired:
# If status check times out, assume there might be changes and proceed
self.logger.warning(f"Git status check timed out for {plugin_id}, proceeding with update")
has_changes = True
stash_info = ""
# Whether the pull can be undone without destroying work.
tree_is_recoverable = not has_changes
if has_changes:
self.logger.info(f"Stashing local changes in {plugin_id} before update")
try:
# Use -u to include untracked files in stash
stash_result = subprocess.run(
['git', '-C', str(plugin_path), 'stash', 'push', '-u', '-m', f'LEDMatrix auto-stash before update {plugin_id}'],
capture_output=True,
text=True,
timeout=30,
check=False
)
if stash_result.returncode == 0:
stash_info = " (local changes were stashed)"
tree_is_recoverable = True
self.logger.info(f"Stashed local changes (including untracked files) for {plugin_id}")
else:
self.logger.warning(f"Failed to stash local changes for {plugin_id}: {stash_result.stderr}")
except subprocess.TimeoutExpired:
self.logger.warning(f"Stash operation timed out for {plugin_id}, proceeding with pull")
# Do not pull what cannot be un-pulled.
#
# The compatibility gate below can refuse the commit this
# pull brings down, and its only way back is `git reset
# --hard`, which discards uncommitted tracked edits. Those
# edits are exactly what the stash above exists to protect,
# so a stash that failed or timed out leaves the rollback
# unable to run without destroying them.
#
# A pull does not necessarily refuse on a dirty tree -- git
# merges happily as long as the incoming commit touches
# different files -- so without this the update would
# succeed, the gate would refuse, and the reset would take
# the user's work with it. Refusing here costs an update in
# a case that already went wrong; the alternative costs
# data.
if not tree_is_recoverable:
self.logger.error(
"Refusing to update %s: it has local changes that could "
"not be stashed, and an incompatible update could then "
"only be rolled back by discarding them. Commit or stash "
"them by hand, then update.", plugin_id)
return False
# Pull from the determined remote branch
self.logger.info(f"Pulling from origin/{remote_pull_branch} for {plugin_id}...")
pull_result = subprocess.run(
['git', '-C', str(plugin_path), 'pull', 'origin', remote_pull_branch],
capture_output=True,
text=True,
timeout=120,
check=True
)
pull_message = pull_result.stdout.strip() or f"Pulled latest changes for {plugin_id}"
if stash_info:
pull_message += stash_info
self.logger.info(pull_message)
updated_git_info = self._get_local_git_info(plugin_path) or {}
updated_sha = updated_git_info.get('sha', '')
if remote_sha and updated_sha and remote_sha.startswith(updated_sha):
self.logger.info(f"Plugin {plugin_id} now at remote commit {remote_sha[:7]}{stash_info}")
elif updated_sha:
self.logger.info(f"Plugin {plugin_id} updated to commit {updated_sha[:7]}{stash_info}")
# The install gate, at the only point on this path where
# it can be answered. Every other route in goes through
# install_plugin, which gates in _install_plugin_impl; this
# one did not, so a pull could deliver a manifest flooring
# above this core and nothing would notice.
if not self._gate_pulled_commit(plugin_id, plugin_path, local_sha):
return False
self._install_dependencies(plugin_path)
return True
except subprocess.CalledProcessError as git_error:
error_output = git_error.stderr or git_error.stdout or "Unknown error"
cmd_str = ' '.join(git_error.cmd)
self.logger.error(f"Git update failed for {plugin_id}")
self.logger.error(f"Command: {cmd_str}")
self.logger.error(f"Return code: {git_error.returncode}")
self.logger.error(f"Error output: {error_output}")
# Check for specific error conditions
error_lower = error_output.lower()
if "would be overwritten" in error_output or "local changes" in error_lower:
self.logger.warning(f"Plugin {plugin_id} has local changes that prevent update. Consider committing or stashing changes manually.")
elif "refusing to merge unrelated histories" in error_lower:
self.logger.error(f"Plugin {plugin_id} has unrelated git histories. Plugin may need to be reinstalled.")
elif "authentication" in error_lower or "permission denied" in error_lower:
self.logger.error(f"Authentication failed for {plugin_id}. Check git credentials or repository permissions.")
elif "not found" in error_lower or "does not exist" in error_lower:
self.logger.error(f"Remote branch or repository not found for {plugin_id}. Check repository URL and branch name.")
elif "conflict" in error_lower:
self.logger.error(f"Merge conflict detected for {plugin_id}. Resolve conflicts manually or reinstall plugin.")
return False
except subprocess.TimeoutExpired:
self.logger.warning(f"Git update timed out for {plugin_id}")
return False
# A plugin with its own .git that _get_local_git_info could not
# read (e.g. no commits yet) may still name a remote to reinstall
# from. Without its own .git, `git -C <plugin>` walks up and finds
# the enclosing LEDMatrix checkout when plugins live in
# plugin-repos/ -- `--local` does not prevent that -- and the
# "plugin's" remote would be LEDMatrix itself.
repo_url = None
if (plugin_path / '.git').exists():
try:
remote_url_result = subprocess.run(
['git', '-C', str(plugin_path), 'config', '--local', '--get', 'remote.origin.url'],
capture_output=True,
text=True,
timeout=10,
check=False
)
if remote_url_result.returncode == 0:
repo_url = remote_url_result.stdout.strip() or None
if repo_url:
self.logger.info(f"Found git remote URL for {plugin_id}: {repo_url}")
except (OSError, subprocess.SubprocessError) as e:
self.logger.debug(f"Could not get git remote URL: {e}")
# Try registry-based update
self.logger.info(f"Plugin {plugin_id} is not a git repository, checking registry...")
self.fetch_registry(force_refresh=True)
plugin_info_remote = self.get_plugin_info(plugin_id, fetch_latest_from_github=True, force_refresh=True)
# If not found, try without 'ledmatrix-' prefix (monorepo migration)
registry_id = plugin_id
if not plugin_info_remote and plugin_id.startswith('ledmatrix-'):
alt_id = plugin_id[len('ledmatrix-'):]
plugin_info_remote = self.get_plugin_info(alt_id, fetch_latest_from_github=True, force_refresh=True)
if plugin_info_remote:
registry_id = alt_id
self.logger.info(f"Plugin {plugin_id} found in registry as {alt_id}")
# If not in registry but we have a repo URL, try reinstalling from that URL
if not plugin_info_remote and repo_url:
self.logger.info(f"Plugin {plugin_id} not in registry but has git remote URL. Reinstalling from {repo_url} to enable updates...")
try:
# Get current branch if possible
branch_result = subprocess.run(
['git', '-C', str(plugin_path), 'rev-parse', '--abbrev-ref', 'HEAD'],
capture_output=True,
text=True,
timeout=10,
check=False
)
branch = branch_result.stdout.strip() if branch_result.returncode == 0 else None
if branch == 'HEAD' or not branch:
branch = 'main'
# Reinstall from URL
result = self.install_from_url(repo_url, plugin_id=plugin_id, branch=branch)
if result.get('success'):
self.logger.info(f"Successfully reinstalled {plugin_id} from {repo_url} as git repository")
return True
else:
self.logger.warning(f"Failed to reinstall {plugin_id} from {repo_url}: {result.get('error')}")
except Exception as e:
self.logger.error(f"Error reinstalling {plugin_id} from URL: {e}")
if not plugin_info_remote:
self.logger.warning(f"Plugin {plugin_id} not found in registry and not a git repository; cannot update automatically")
if not repo_url:
self.logger.warning("Plugin may have been installed via ZIP download. Try reinstalling from GitHub URL to enable updates.")
return False
repo_url = plugin_info_remote.get('repo')
remote_sha = plugin_info_remote.get('last_commit_sha')
remote_branch = plugin_info_remote.get('branch') or plugin_info_remote.get('default_branch')
# Compare local manifest version against registry latest_version
# to avoid unnecessary reinstalls for monorepo plugins. Uses the
# same semantic comparator as the web UI's update badge, so
# equivalent spellings ("v1.2.0" vs "1.2.0") never trigger a
# reinstall and a locally-ahead version is never downgraded.
try:
local_manifest_path = plugin_path / "manifest.json"
if local_manifest_path.exists():
with open(local_manifest_path, 'r', encoding='utf-8') as f:
local_manifest = json.load(f)
local_version = local_manifest.get('version', '')
remote_version = plugin_info_remote.get('latest_version', '')
from src.plugin_system.compatibility import is_update_available
# No truthiness gate: the shared comparator already treats
# a missing version on either side as "no update", and the
# store must agree with the UI badge in that case too. A
# missing manifest (not just a missing version field)
# still falls through to the reinstall recovery path.
if not is_update_available(local_version, remote_version):
self.logger.info(
f"Plugin {plugin_id} already at latest version "
f"(installed {local_version}, registry {remote_version})")
return True
except Exception as e:
self.logger.debug(f"Could not compare versions for {plugin_id}: {e}")
# Plugin is not a git repo but is in registry and has a newer version - reinstall
self.logger.info(f"Plugin {plugin_id} not installed via git; re-installing latest archive (registry id: {registry_id})")
# Reinstall with the old version kept aside until the new
# download succeeds — this is the path every routine store
# update takes, and a mid-update network failure must not
# destroy the user's plugin.
return self._reinstall_with_rollback(registry_id, plugin_path)
except Exception as e:
self.logger.error(f"Error updating plugin {plugin_id}: {e}", exc_info=True)
return False
+4 -4
View File
@@ -7,7 +7,7 @@ plugin discovery / manifest / config-default logic lives in exactly one place.
import json import json
from pathlib import Path from pathlib import Path
from typing import Any, Dict, Optional, Sequence, Union, cast from typing import Any, Dict, Optional, Sequence, Union
def find_plugin_dir(plugin_id: str, search_dirs: Sequence[Union[str, Path]]) -> Optional[Path]: def find_plugin_dir(plugin_id: str, search_dirs: Sequence[Union[str, Path]]) -> Optional[Path]:
@@ -39,7 +39,7 @@ def load_manifest(plugin_dir: Union[str, Path]) -> Dict[str, Any]:
if not manifest_path.exists(): if not manifest_path.exists():
raise FileNotFoundError(f"No manifest.json in {plugin_dir}") raise FileNotFoundError(f"No manifest.json in {plugin_dir}")
with open(manifest_path, 'r', encoding='utf-8') as f: with open(manifest_path, 'r', encoding='utf-8') as f:
return cast(Dict[str, Any], json.load(f)) return json.load(f)
def merge_config(base: Dict[str, Any], override: Dict[str, Any]) -> Dict[str, Any]: def merge_config(base: Dict[str, Any], override: Dict[str, Any]) -> Dict[str, Any]:
@@ -64,7 +64,7 @@ def load_schema(plugin_dir: Union[str, Path]) -> Optional[Dict[str, Any]]:
if not schema_path.exists(): if not schema_path.exists():
return None return None
with open(schema_path, 'r', encoding='utf-8') as f: with open(schema_path, 'r', encoding='utf-8') as f:
return cast(Optional[Dict[str, Any]], json.load(f)) return json.load(f)
def load_config_defaults(plugin_dir: Union[str, Path]) -> Dict[str, Any]: def load_config_defaults(plugin_dir: Union[str, Path]) -> Dict[str, Any]:
@@ -124,7 +124,7 @@ def load_harness_spec(plugin_dir: Union[str, Path]) -> Dict[str, Any]:
if not spec_path.exists(): if not spec_path.exists():
return {} return {}
with open(spec_path, 'r', encoding='utf-8') as f: with open(spec_path, 'r', encoding='utf-8') as f:
spec: Dict[str, Any] = json.load(f) spec = json.load(f)
# Resolve mock_data path and inline its contents for convenience. # Resolve mock_data path and inline its contents for convenience.
mock_rel = spec.get('mock_data') mock_rel = spec.get('mock_data')
+7 -28
View File
@@ -5,21 +5,9 @@ Provides mock implementations of display_manager, cache_manager, config_manager,
and plugin_manager for use in plugin unit tests. and plugin_manager for use in plugin unit tests.
""" """
import warnings from typing import Dict, Any, Optional
from typing import Dict, Any, List, Optional
from PIL import Image from PIL import Image
#: Why draw_image() warns. Kept (rather than removed) so existing plugin test
#: suites that call it keep passing, but a plugin that calls it passes its
#: tests and then crashes on the Pi.
DRAW_IMAGE_DEPRECATION = (
"display_manager.draw_image() exists only on the test doubles; the real "
"DisplayManager has no such method, so this raises AttributeError on a "
"device. Paste onto the canvas instead: "
"display_manager.image.paste(img, (x, y)) (with the image as mask, "
"image.paste(rgba, (x, y), rgba), for transparency)."
)
class MockDisplayManager: class MockDisplayManager:
"""Mock display manager for testing.""" """Mock display manager for testing."""
@@ -32,7 +20,7 @@ class MockDisplayManager:
self.image = Image.new('RGB', (width, height), color=(0, 0, 0)) self.image = Image.new('RGB', (width, height), color=(0, 0, 0))
self.clear_called = False self.clear_called = False
self.update_called = False self.update_called = False
self.draw_calls: List[Dict[str, Any]] = [] self.draw_calls = []
def clear(self): def clear(self):
"""Clear the display.""" """Clear the display."""
@@ -43,16 +31,8 @@ class MockDisplayManager:
"""Update the display.""" """Update the display."""
self.update_called = True self.update_called = True
def draw_text(self, text: str, x: Optional[int] = None, y: Optional[int] = None, color: tuple = (255, 255, 255), def draw_text(self, text: str, x: int, y: int, color: tuple = (255, 255, 255), font=None):
font=None, small_font: bool = False, centered: bool = False): """Draw text on the display."""
"""Draw text on the display.
Accepts every argument the real ``DisplayManager.draw_text`` does, so
a plugin passing ``small_font``/``centered`` (or leaving x/y to
default) doesn't fail here while working on the device. ``font``
stays fifth for callers of the old mock signature; pass the rest by
keyword, as the real method's positional order differs.
"""
self.draw_calls.append({ self.draw_calls.append({
'type': 'text', 'type': 'text',
'text': text, 'text': text,
@@ -63,8 +43,7 @@ class MockDisplayManager:
}) })
def draw_image(self, image: Image.Image, x: int, y: int): def draw_image(self, image: Image.Image, x: int, y: int):
"""Draw an image on the display. Deprecated: see DRAW_IMAGE_DEPRECATION.""" """Draw an image on the display."""
warnings.warn(DRAW_IMAGE_DEPRECATION, DeprecationWarning, stacklevel=2)
self.draw_calls.append({ self.draw_calls.append({
'type': 'image', 'type': 'image',
'image': image, 'image': image,
@@ -166,8 +145,8 @@ class MockConfigManager:
def __init__(self, config: Optional[Dict[str, Any]] = None): def __init__(self, config: Optional[Dict[str, Any]] = None):
self._config = config or {} self._config = config or {}
self.load_config_calls: List[Dict[str, Any]] = [] self.load_config_calls = []
self.save_config_calls: List[Dict[str, Any]] = [] self.save_config_calls = []
def load_config(self) -> Dict[str, Any]: def load_config(self) -> Dict[str, Any]:
"""Load configuration.""" """Load configuration."""
@@ -18,7 +18,7 @@ get_font_height, get_text_width, draw_text,
draw_text_with_icons, draw_weather_icon (and the _draw_sun/_draw_cloud/ draw_text_with_icons, draw_weather_icon (and the _draw_sun/_draw_cloud/
_draw_rain/_draw_snow/_draw_storm family), format_date_with_ordinal, _draw_rain/_draw_snow/_draw_storm family), format_date_with_ordinal,
capture_mode, set_scrolling_state, is_currently_scrolling, capture_mode, set_scrolling_state, is_currently_scrolling,
process_deferred_updates, update_display, render_size, offscreen. A behavior process_deferred_updates, update_display, render_size. A behavior
change to any of those in DisplayManager must be mirrored here, or change to any of those in DisplayManager must be mirrored here, or
plugin visual tests will pass against stale behavior. plugin visual tests will pass against stale behavior.
@@ -29,7 +29,6 @@ through src/common/bdf_font.py, so those pixels cannot drift.
import math import math
import os import os
import time import time
import warnings
from contextlib import contextmanager from contextlib import contextmanager
from pathlib import Path from pathlib import Path
from typing import Any, List, Optional, Tuple from typing import Any, List, Optional, Tuple
@@ -39,12 +38,9 @@ from src.common.bdf_font import draw_bdf_text, load_bdf_face
from src.common.font_layout import crisp_size, load_truetype from src.common.font_layout import crisp_size, load_truetype
from src.logging_config import get_logger from src.logging_config import get_logger
from src.plugin_system.testing.mocks import DRAW_IMAGE_DEPRECATION
logger = get_logger(__name__) logger = get_logger(__name__)
_draw_image_warning_logged = False
class _MatrixProxy: class _MatrixProxy:
"""Lightweight proxy so plugins can access display_manager.matrix.width/height.""" """Lightweight proxy so plugins can access display_manager.matrix.width/height."""
@@ -247,39 +243,11 @@ class VisualTestDisplayManager:
wraps every off-screen content fetch in this context, so the harness wraps every off-screen content fetch in this context, so the harness
must provide it for that code path to be exercisable in tests. must provide it for that code path to be exercisable in tests.
""" """
was_active = self._capture_mode_active
self._capture_mode_active = True self._capture_mode_active = True
try: try:
yield yield
finally: finally:
self._capture_mode_active = was_active self._capture_mode_active = False
@contextmanager
def offscreen(self, width: Optional[int] = None, height: Optional[int] = None):
"""
Interface parity with DisplayManager.offscreen().
Vegas mode's PluginAdapter draws every plugin on a canvas of its own.
The real display manager keeps that canvas per thread; the harness is
single-threaded, so it swaps a fresh canvas in and restores the old one,
which is all a test can observe.
"""
prev = (self.image, self.draw, self._width, self._height,
self.matrix, self._capture_mode_active)
target_w = max(1, min(int(width), self._width)) if width else self._width
target_h = max(1, min(int(height), self._height)) if height else self._height
try:
self._width, self._height = target_w, target_h
self.matrix = _MatrixProxy(target_w, target_h)
self.image = Image.new('RGB', (target_w, target_h), (0, 0, 0))
self.draw = ImageDraw.Draw(self.image)
# Match production: 1-bit text, so goldens show what the panel shows.
self.draw.fontmode = "1"
self._capture_mode_active = True
yield self
finally:
(self.image, self.draw, self._width, self._height,
self.matrix, self._capture_mode_active) = prev
def draw_text(self, text: str, x: Optional[int] = None, y: Optional[int] = None, def draw_text(self, text: str, x: Optional[int] = None, y: Optional[int] = None,
color: Tuple[int, int, int] = (255, 255, 255), small_font: bool = False, color: Tuple[int, int, int] = (255, 255, 255), small_font: bool = False,
@@ -325,26 +293,17 @@ class VisualTestDisplayManager:
else: else:
self.draw.text((x, y), text, font=current_font, fill=color) self.draw.text((x, y), text, font=current_font, fill=color)
except Exception as e: except Exception as e:
# WARNING, not DEBUG: the real DisplayManager logs this at ERROR, logger.debug(f"Error drawing text: {e}")
# and a test double that hides it lets a broken draw pass.
logger.warning(f"Error drawing text: {e}")
def draw_image(self, image: Image.Image, x: int, y: int): def draw_image(self, image: Image.Image, x: int, y: int):
"""Draw an image on the display. Deprecated: see DRAW_IMAGE_DEPRECATION.""" """Draw an image on the display."""
warnings.warn(DRAW_IMAGE_DEPRECATION, DeprecationWarning, stacklevel=2)
global _draw_image_warning_logged
if not _draw_image_warning_logged:
# Also logged once: the dev preview server drives this class
# outside pytest, where DeprecationWarning is hidden by default.
_draw_image_warning_logged = True
logger.warning(DRAW_IMAGE_DEPRECATION)
self.draw_calls.append({ self.draw_calls.append({
'type': 'image', 'image': image, 'x': x, 'y': y, 'type': 'image', 'image': image, 'x': x, 'y': y,
}) })
try: try:
self.image.paste(image, (x, y)) self.image.paste(image, (x, y))
except Exception as e: except Exception as e:
logger.warning(f"Error drawing image: {e}") logger.debug(f"Error drawing image: {e}")
def _draw_bdf_text(self, text, x, y, color=(255, 255, 255), font=None): def _draw_bdf_text(self, text, x, y, color=(255, 255, 255), font=None):
"""Draw text in a BDF ``freetype.Face`` with (x, y) as its top-left. """Draw text in a BDF ``freetype.Face`` with (x, y) as its top-left.
+9 -28
View File
@@ -268,21 +268,15 @@ class StartupValidator:
except Exception as e: except Exception as e:
self.warnings.append(f"Could not validate display configuration: {e}") self.warnings.append(f"Could not validate display configuration: {e}")
def _validate_plugins(self, discovered_plugins=None) -> None: def _validate_plugins(self) -> None:
"""Validate plugin configurations and dependencies. """Validate plugin configurations and dependencies."""
``discovered_plugins`` is a list the caller already got from
``discover_plugins()``; passing it skips a second directory scan (and
its duplicate log lines) at startup.
"""
if not self.plugin_manager: if not self.plugin_manager:
return return
try: try:
# Get enabled plugins from config # Get enabled plugins from config
config = self.config_manager.get_config() config = self.config_manager.get_config()
if discovered_plugins is None: discovered_plugins = self.plugin_manager.discover_plugins()
discovered_plugins = self.plugin_manager.discover_plugins()
# Check for enabled plugins that don't exist # Check for enabled plugins that don't exist
for plugin_id, plugin_config in config.items(): for plugin_id, plugin_config in config.items():
@@ -300,11 +294,7 @@ class StartupValidator:
# Validate plugin configurations # Validate plugin configurations
for plugin_id in discovered_plugins: for plugin_id in discovered_plugins:
plugin_config = config.get(plugin_id) plugin_config = config.get(plugin_id, {})
# A null block ("my-plugin": null) is not an enabled plugin;
# .get() on it raised and abandoned every remaining check.
if not isinstance(plugin_config, dict):
continue
if plugin_config.get('enabled', False): if plugin_config.get('enabled', False):
# Check if plugin can be loaded (without actually loading it) # Check if plugin can be loaded (without actually loading it)
plugin_dir = self.plugin_manager.get_plugin_directory(plugin_id) plugin_dir = self.plugin_manager.get_plugin_directory(plugin_id)
@@ -318,21 +308,12 @@ class StartupValidator:
def raise_on_errors(self) -> None: def raise_on_errors(self) -> None:
""" """
Raise one exception if validation errors exist; return None if not. Raise exceptions if validation errors exist.
Nothing in core calls this (see the module docstring). Errors are
grouped by a keyword in their message, not by which check produced
them, and only the first non-empty group is raised, in the order
config > cache > plugin: a "plugin ... config" message counts as a
config error, and cache/plugin errors are not reported while a
config error exists. The raised exception's ``context['errors']``
holds that group's messages only.
Raises: Raises:
ConfigError: If any message mentions config/configuration, or if ConfigError: If configuration validation fails
none matches any group CacheError: If cache validation fails
CacheError: If a message mentions cache (and none config) PluginError: If plugin validation fails
PluginError: If a message mentions plugin (and none of the above)
""" """
if not self.errors: if not self.errors:
return return
-31
View File
@@ -74,31 +74,6 @@ class VegasModeConfig:
# precedence over smooth_scroll's whole-pixel pacing when on. # precedence over smooth_scroll's whole-pixel pacing when on.
sub_pixel_blend: bool = False sub_pixel_blend: bool = False
# Render every plugin's ticker content on the background prefetch thread,
# each on a canvas of its own (DisplayManager.offscreen), instead of
# handing plugins that draw on the display canvas to the render thread one
# at a time. Each of those cost the scroll a 40-600ms pause. False restores
# that path; it is kept for one release in case a plugin misbehaves when
# drawn off the render thread. See docs/OFFSCREEN_RENDERING.md.
offscreen_prefetch: bool = True
# How long another thread may hold the GIL before the render thread's
# request forces it to yield, in ms, while Vegas runs. CPython's default is
# 5ms. Plugin rendering on the prefetch thread and plugin updates hold the
# GIL in Pillow and Python code, and a frame waiting its turn for 5ms at a
# time misses its refresh. 0 leaves the interpreter default alone.
# Experimental. On hdpi it did less than prefetch_gate (0.90% -> 0.78% late
# against 0.60%; see docs/OFFSCREEN_RENDERING.md), so it stays off.
switch_interval_ms: float = 0.0
# Let the prefetch thread run Python only while the render thread is
# blocked waiting for vsync, and park it the rest of the time, so the
# render thread never waits for the GIL when its refresh comes round. Needs
# a binding that releases the GIL in SwapOnVSync; off otherwise. On hdpi
# it cut frames two or more refreshes late eightfold, and late frames
# overall from 0.90% to 0.60%. See src/common/render_gate.py.
prefetch_gate: bool = True
# Keep one continuous strip, extending it with the next group of plugins as # Keep one continuous strip, extending it with the next group of plugins as
# the scroll approaches the end, instead of composing a fresh strip and # the scroll approaches the end, instead of composing a fresh strip and
# swapping it in. A swap stops the motion, substitutes every pixel at once # swapping it in. A swap stops the motion, substitutes every pixel at once
@@ -232,9 +207,6 @@ class VegasModeConfig:
smooth_scroll=get('smooth_scroll', d.smooth_scroll), smooth_scroll=get('smooth_scroll', d.smooth_scroll),
sub_pixel_blend=bool(get('sub_pixel_blend', d.sub_pixel_blend)), sub_pixel_blend=bool(get('sub_pixel_blend', d.sub_pixel_blend)),
continuous_scroll=get('continuous_scroll', d.continuous_scroll), continuous_scroll=get('continuous_scroll', d.continuous_scroll),
offscreen_prefetch=bool(get('offscreen_prefetch', d.offscreen_prefetch)),
switch_interval_ms=float(get('switch_interval_ms', d.switch_interval_ms) or 0.0),
prefetch_gate=bool(get('prefetch_gate', d.prefetch_gate)),
extend_threshold_screens=float( extend_threshold_screens=float(
get('extend_threshold_screens', d.extend_threshold_screens)), get('extend_threshold_screens', d.extend_threshold_screens)),
auto_trim=get('auto_trim', d.auto_trim), auto_trim=get('auto_trim', d.auto_trim),
@@ -278,9 +250,6 @@ class VegasModeConfig:
'smooth_scroll': self.smooth_scroll, 'smooth_scroll': self.smooth_scroll,
'sub_pixel_blend': self.sub_pixel_blend, 'sub_pixel_blend': self.sub_pixel_blend,
'continuous_scroll': self.continuous_scroll, 'continuous_scroll': self.continuous_scroll,
'offscreen_prefetch': self.offscreen_prefetch,
'switch_interval_ms': self.switch_interval_ms,
'prefetch_gate': self.prefetch_gate,
'extend_threshold_screens': self.extend_threshold_screens, 'extend_threshold_screens': self.extend_threshold_screens,
'auto_trim': self.auto_trim, 'auto_trim': self.auto_trim,
'trim_threshold': self.trim_threshold, 'trim_threshold': self.trim_threshold,
+36 -140
View File
@@ -13,16 +13,15 @@ Supports three display modes per plugin:
import logging import logging
import math import math
import sys
import time import time
import threading import threading
from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING from typing import Optional, Dict, Any, List, Callable, TYPE_CHECKING
from src.common import render_gate
from src.vegas_mode.config import VegasModeConfig from src.vegas_mode.config import VegasModeConfig
from src.vegas_mode.plugin_adapter import PluginAdapter from src.vegas_mode.plugin_adapter import PluginAdapter
from src.vegas_mode.stream_manager import StreamManager from src.vegas_mode.stream_manager import StreamManager
from src.vegas_mode.render_pipeline import RenderPipeline from src.vegas_mode.render_pipeline import RenderPipeline
from src.plugin_system.base_plugin import VegasDisplayMode
if TYPE_CHECKING: if TYPE_CHECKING:
from src.plugin_system.plugin_manager import PluginManager from src.plugin_system.plugin_manager import PluginManager
@@ -77,14 +76,8 @@ class VegasModeCoordinator:
- Provide status and control interface - Provide status and control interface
""" """
#: How long a STATIC pause waits for the plugin's lock (held while its
#: update() runs) before skipping that turn.
STATIC_LOCK_TIMEOUT = 1.0
# Class-level so coordinators built without __init__ (tests) have it. # Class-level so coordinators built without __init__ (tests) have it.
_last_live_check: float = float('-inf') _last_live_check: float = float('-inf')
# Set only while Vegas has changed the GIL switch interval; read with getattr.
_saved_switch_interval: Optional[float]
def __init__( def __init__(
self, self,
@@ -108,8 +101,7 @@ class VegasModeCoordinator:
self.plugin_manager = plugin_manager self.plugin_manager = plugin_manager
# Initialize components # Initialize components
self.plugin_adapter = PluginAdapter( self.plugin_adapter = PluginAdapter(display_manager, self.vegas_config)
display_manager, self.vegas_config, plugin_manager=plugin_manager)
self.stream_manager = StreamManager( self.stream_manager = StreamManager(
self.vegas_config, self.vegas_config,
plugin_manager, plugin_manager,
@@ -278,18 +270,12 @@ class VegasModeCoordinator:
self._is_active = True self._is_active = True
self._should_stop = False self._should_stop = False
# A pause belongs to the run it happened in; carrying it into a
# new run would have run_frame() refuse every frame.
self._is_paused = False
self._live_priority_active = False
self._start_time = time.time() self._start_time = time.time()
# A fresh run starts with a clean health slate: no stale # A fresh run starts with a clean health slate: no stale
# "was degraded" from the previous run, and a heartbeat that is # "was degraded" from the previous run, and a heartbeat that is
# due immediately so the first sample confirms the marquee is up. # due immediately so the first sample confirms the marquee is up.
self._fps_last_health_log = 0.0 self._fps_last_health_log = 0.0
self._fps_was_degraded = False self._fps_was_degraded = False
self._apply_switch_interval()
self._install_render_gate()
# Line up the next group immediately, so the first extension is already # Line up the next group immediately, so the first extension is already
# warm rather than stalling the scroll to fetch it. # warm rather than stalling the scroll to fetch it.
@@ -307,16 +293,11 @@ class VegasModeCoordinator:
self._should_stop = True self._should_stop = True
self._is_active = False self._is_active = False
self._is_paused = False
self._live_priority_active = False
if self._start_time: if self._start_time:
self.stats['total_runtime_seconds'] += time.time() - self._start_time self.stats['total_runtime_seconds'] += time.time() - self._start_time
self._start_time = None self._start_time = None
self._restore_switch_interval()
self._remove_render_gate()
# Cleanup components # Cleanup components
self.render_pipeline.reset() self.render_pipeline.reset()
self.stream_manager.reset() self.stream_manager.reset()
@@ -324,57 +305,6 @@ class VegasModeCoordinator:
logger.info("Vegas mode stopped") logger.info("Vegas mode stopped")
def _apply_switch_interval(self) -> None:
"""Shorten the GIL switch interval for the run; see VegasModeConfig."""
ms = self.vegas_config.switch_interval_ms
if not ms or ms <= 0:
return
if getattr(self, '_saved_switch_interval', None) is None:
self._saved_switch_interval = sys.getswitchinterval()
sys.setswitchinterval(ms / 1000.0)
logger.info("Vegas: GIL switch interval %.1fms (was %.1fms)",
ms, self._saved_switch_interval * 1000.0) # type: ignore[operator] # set just above; getattr hides it
def _restore_switch_interval(self) -> None:
saved = getattr(self, '_saved_switch_interval', None)
if saved is not None:
sys.setswitchinterval(saved)
self._saved_switch_interval = None
def _install_render_gate(self) -> None:
"""Gate the prefetch thread on the render thread's swaps; see VegasModeConfig."""
if not self.vegas_config.prefetch_gate:
return
if getattr(self.display_manager, 'render_gate', None) is not None:
return
releases = render_gate.swap_releases_gil()
if releases is None:
logger.debug("Vegas: no prefetch gate -- no hardware binding loaded")
return
if not releases:
# On by default, so this is every stock install: say so once per
# run, not as a warning.
logger.info("Vegas: no prefetch gate -- this rgbmatrix binding keeps "
"the GIL in SwapOnVSync (scripts/build_rgbmatrix_nogil.sh)")
return
gate = render_gate.RenderGate()
# Locks the render thread takes too: never park the prefetch holding one.
gate.guard(self._state_lock,
getattr(self.stream_manager, '_buffer_lock', None),
getattr(self.render_pipeline, '_buffer_lock', None),
getattr(self.render_pipeline, '_prefetch_lock', None),
getattr(self.plugin_adapter, '_cache_lock', None))
self.display_manager.render_gate = gate
logger.info("Vegas: prefetch gated on vsync")
def _remove_render_gate(self) -> None:
gate = getattr(self.display_manager, 'render_gate', None)
if gate is None:
return
self.display_manager.render_gate = None
logger.info("Vegas: prefetch gate parked the prefetch %d times, %.1fs in all",
gate.parks, gate.parked_seconds)
def pause(self) -> None: def pause(self) -> None:
"""Pause Vegas mode (for live priority interruption).""" """Pause Vegas mode (for live priority interruption)."""
with self._state_lock: with self._state_lock:
@@ -484,20 +414,6 @@ class VegasModeCoordinator:
if not self.start(): if not self.start():
return False return False
# A live-priority pause is only ever lifted by _check_live_priority(),
# and run_frame() returns before reaching it while paused -- so once
# paused, every later iteration returned False at its first frame and
# the ticker never came back until a restart. The display controller
# only calls run_iteration() when nothing preempts Vegas (no live mode,
# or live content is kept in the ticker), so being called at all means
# the live content that paused us has ended.
with self._state_lock:
paused_for_live = self._is_paused and self._live_priority_active
if paused_for_live:
self._live_priority_active = False
self.resume()
logger.info("Live priority ended - resuming Vegas")
if self.vegas_config.continuous_scroll: if self.vegas_config.continuous_scroll:
# The strip is continuously extended and trimmed, so its width says # The strip is continuously extended and trimmed, so its width says
# nothing about how long to run. This is only how often control # nothing about how long to run. This is only how often control
@@ -506,11 +422,7 @@ class VegasModeCoordinator:
duration = float(self.vegas_config.max_cycle_duration) duration = float(self.vegas_config.max_cycle_duration)
else: else:
duration = self.render_pipeline.get_dynamic_duration() duration = self.render_pipeline.get_dynamic_duration()
# Monotonic for the same reason as the per-frame clock below: this start_time = time.time()
# bounds how long the iteration runs, and an NTP step on an RTC-less
# Pi would otherwise end it at once (forward) or stretch it by the
# size of the correction (backward).
start_time = time.monotonic()
frame_count = 0 frame_count = 0
fps_log_interval = 5.0 # Sample FPS every 5 seconds fps_log_interval = 5.0 # Sample FPS every 5 seconds
# Health state lives on the coordinator, not here: run_iteration() is # Health state lives on the coordinator, not here: run_iteration() is
@@ -519,8 +431,10 @@ class VegasModeCoordinator:
# of every iteration rather than once per interval, and a recovery # of every iteration rather than once per interval, and a recovery
# that crossed an iteration boundary was never reported at all -- # that crossed an iteration boundary was never reported at all --
# was_degraded had already gone back to False. # was_degraded had already gone back to False.
# Monotonic. Never mix it with a wall-clock value: every delta would # Monotonic, and deliberately not start_time: start_time is wall
# be hugely negative and silence the frame-rate reporting altogether. # clock and is used below to report the iteration's duration. Mixing
# the two here would make every delta hugely negative and silence the
# frame-rate reporting altogether.
last_fps_log_time = time.monotonic() last_fps_log_time = time.monotonic()
fps_frame_count = 0 fps_frame_count = 0
# A mean hides stutter completely. At 120fps a five-second window is # A mean hides stutter completely. At 120fps a five-second window is
@@ -547,7 +461,8 @@ class VegasModeCoordinator:
if not self._handle_static_pause(static_plugin): if not self._handle_static_pause(static_plugin):
# Static pause was interrupted # Static pause was interrupted
return False return False
# The trigger consumed the plugin's marker; carry on scrolling. # After static pause, skip this segment and continue
self.stream_manager.get_next_segment() # Consume the segment
continue continue
# Run frame # Run frame
@@ -609,10 +524,6 @@ class VegasModeCoordinator:
fps, target, fps_frame_count, fps, target, fps_frame_count,
p99 * 1000.0, frame_worst * 1000.0 p99 * 1000.0, frame_worst * 1000.0
) )
gate = getattr(self.display_manager, 'render_gate', None)
if gate is not None:
logger.info("Vegas: prefetch parked %d times, %.1fs in all",
gate.parks, gate.parked_seconds)
self._fps_last_health_log = current_time self._fps_last_health_log = current_time
else: else:
logger.debug( logger.debug(
@@ -659,7 +570,7 @@ class VegasModeCoordinator:
).start() ).start()
# Check elapsed time # Check elapsed time
elapsed = time.monotonic() - start_time elapsed = time.time() - start_time
if elapsed >= duration: if elapsed >= duration:
break break
@@ -678,7 +589,7 @@ class VegasModeCoordinator:
# cycle content multiple times within one iteration — acceptable for # cycle content multiple times within one iteration — acceptable for
# a continuous ticker. # a continuous ticker.
logger.info("Vegas iteration completed after %.1fs", time.monotonic() - start_time) logger.info("Vegas iteration completed after %.1fs", time.time() - start_time)
return True return True
def _check_live_priority(self) -> bool: def _check_live_priority(self) -> bool:
@@ -826,30 +737,32 @@ class VegasModeCoordinator:
""" """
Check if a STATIC mode plugin should take over display. Check if a STATIC mode plugin should take over display.
Called every frame. The render pipeline marks where each STATIC Called during iteration to detect when scroll should pause
plugin's turn falls in the strip, and this reports the one the scroll for a static plugin display.
has just reached.
This used to peek at the front of the stream manager's segment
buffer, which continuous scrolling (the default) never advances: it
extends the strip with take_next_group() instead. The same first
segment was examined on every frame, so a STATIC plugin paused the
scroll only if it happened to be first, once, at startup -- and
otherwise just scrolled past as ordinary content. Swap mode fared no
better: nothing advanced the buffer mid-cycle either.
Returns: Returns:
Plugin instance if static pause should begin, None otherwise Plugin instance if static pause should begin, None otherwise
""" """
plugin_id = self.render_pipeline.next_static_trigger() # Get the next plugin that would be displayed
if not plugin_id: next_segment = self.stream_manager.peek_next_segment()
if not next_segment:
return None return None
plugin: Optional['BasePlugin'] = self.plugin_manager.get_plugin(plugin_id)
plugin_id = next_segment.plugin_id
plugin = self.plugin_manager.get_plugin(plugin_id)
if not plugin: if not plugin:
logger.debug("[%s] STATIC turn reached, but the plugin is no longer loaded",
plugin_id)
return None return None
return plugin
# Check if this plugin is configured for STATIC mode
try:
display_mode = plugin.get_vegas_display_mode()
if display_mode == VegasDisplayMode.STATIC:
return plugin
except (AttributeError, TypeError):
logger.exception("Error checking vegas mode for %s", plugin_id)
return None
def _handle_static_pause(self, plugin: 'BasePlugin') -> bool: def _handle_static_pause(self, plugin: 'BasePlugin') -> bool:
""" """
@@ -879,32 +792,15 @@ class VegasModeCoordinator:
self.display_manager.set_scrolling_state(False) self.display_manager.set_scrolling_state(False)
try: try:
# Display the plugin using its standard display() method, under # Display the plugin using its standard display() method
# its plugin lock like every other display() call: without it this plugin.display(force_clear=True)
# could draw while the update worker is inside the plugin's
# update(). If update() holds the lock past the wait, skip this
# turn rather than stall the marquee.
get_lock = getattr(self.plugin_manager, 'get_plugin_lock', None)
plugin_lock = get_lock(plugin_id) if get_lock else None
if plugin_lock is not None and not plugin_lock.acquire(
timeout=self.STATIC_LOCK_TIMEOUT):
logger.info("Static pause skipped for %s: its update() is still running",
plugin_id)
return True
try:
plugin.display(force_clear=True)
finally:
if plugin_lock is not None:
plugin_lock.release()
self.display_manager.update_display() self.display_manager.update_display()
# Wait for the plugin's display duration. Monotonic, like the # Wait for the plugin's display duration
# iteration clock: an NTP step on an RTC-less Pi would otherwise
# end the pause at once or stretch it by the correction.
duration = plugin.get_display_duration() duration = plugin.get_display_duration()
start = time.monotonic() start = time.time()
while time.monotonic() - start < duration: while time.time() - start < duration:
# Check for interruptions # Check for interruptions
if self._should_stop: if self._should_stop:
logger.info("Static pause interrupted by stop request") logger.info("Static pause interrupted by stop request")
@@ -924,7 +820,7 @@ class VegasModeCoordinator:
logger.info( logger.info(
"Static pause completed for %s after %.1fs", "Static pause completed for %s after %.1fs",
plugin_id, time.monotonic() - start plugin_id, time.time() - start
) )
except Exception: except Exception:
+57 -114
View File
@@ -8,7 +8,7 @@ implement get_vegas_content() and fallback capture of display() output.
import logging import logging
import threading import threading
import time import time
from contextlib import contextmanager, nullcontext from contextlib import nullcontext
from typing import Optional, List, Any, Tuple, Union, TYPE_CHECKING from typing import Optional, List, Any, Tuple, Union, TYPE_CHECKING
from PIL import Image from PIL import Image
@@ -33,13 +33,7 @@ class PluginAdapter:
2. Fallback: Capture display_manager.image after calling plugin.display() 2. Fallback: Capture display_manager.image after calling plugin.display()
""" """
#: How long a background fetch waits for a plugin's update() to finish def __init__(self, display_manager: Any, config: Optional[Any] = None):
#: before skipping the plugin this round. Off the render thread waiting
#: costs nothing visible; it only delays that one plugin's content.
PLUGIN_LOCK_TIMEOUT = 2.0
def __init__(self, display_manager: Any, config: Optional[Any] = None,
plugin_manager: Optional[Any] = None):
""" """
Initialize the plugin adapter. Initialize the plugin adapter.
@@ -48,13 +42,8 @@ class PluginAdapter:
config: VegasModeConfig controlling trim behaviour. When omitted, config: VegasModeConfig controlling trim behaviour. When omitted,
trimming runs with the dataclass defaults, so existing callers trimming runs with the dataclass defaults, so existing callers
and tests keep working unchanged. and tests keep working unchanged.
plugin_manager: Source of the per-plugin lock that keeps a
background fetch from running a plugin's display() while its
update() is mid-flight. Optional: without it, fetches take no
lock, as they always did.
""" """
self.display_manager = display_manager self.display_manager = display_manager
self.plugin_manager = plugin_manager
if config is None: if config is None:
from src.vegas_mode.config import VegasModeConfig from src.vegas_mode.config import VegasModeConfig
config = VegasModeConfig() config = VegasModeConfig()
@@ -102,14 +91,13 @@ class PluginAdapter:
Args: Args:
plugin: Plugin instance to get content from plugin: Plugin instance to get content from
plugin_id: Plugin identifier for logging plugin_id: Plugin identifier for logging
offscreen_only: The caller is off the render thread. Every content offscreen_only: Skip every path that touches the shared display
path draws on a canvas of its own (DisplayManager.offscreen), canvas, for callers running off the render thread. The canvas
so all of them are safe there; the fetch also takes the and the matrix proxy are process-wide mutable state, so
plugin's lock, waiting up to PLUGIN_LOCK_TIMEOUT for a running narrowing or capturing through them from another thread would
update() to finish. With ``offscreen_prefetch`` switched off, corrupt the frame the render loop is pushing. Returns None when
the old behaviour applies instead: paths that need a canvas the plugin can only be served that way, leaving the caller to
return None, leaving the caller to fetch the plugin on the fetch it on the render thread.
render thread.
Returns: Returns:
List of PIL Images representing plugin content, or None if no content List of PIL Images representing plugin content, or None if no content
@@ -129,77 +117,11 @@ class PluginAdapter:
) )
return cached return cached
# The old contract, kept behind the switch: background callers may
# not draw, so anything needing a canvas is left for the render thread.
restricted = offscreen_only and not getattr(
self.config, 'offscreen_prefetch', True)
if not offscreen_only or restricted:
return self._fetch_content(plugin, plugin_id, restricted)
with self._plugin_lock(plugin_id) as acquired:
if not acquired:
logger.warning(
"[%s] update() still running after %.0fs; skipping it this "
"round", plugin_id, self.PLUGIN_LOCK_TIMEOUT
)
return None
return self._fetch_content(plugin, plugin_id, restricted=False)
@contextmanager
def _plugin_lock(self, plugin_id: str):
"""Hold the plugin's update/display lock, waiting a bounded time.
Yields whether it was acquired. Yields True, holding nothing, when
there is no plugin manager to ask -- the behaviour before the lock was
taken here at all.
"""
if not hasattr(self.plugin_manager, 'get_plugin_lock'):
yield True
return
lock = self.plugin_manager.get_plugin_lock(plugin_id)
acquired = lock.acquire(timeout=self.PLUGIN_LOCK_TIMEOUT)
try:
yield acquired
finally:
if acquired:
lock.release()
@contextmanager
def _isolated_canvas(self, width: Optional[int] = None):
"""A canvas for the plugin to draw on that nothing else sees.
DisplayManager.offscreen() gives the calling thread its own canvas, so
this is safe on any thread and leaves the shared canvas untouched.
Older display managers and test doubles without it get the previous
behaviour: capture on the shared canvas, narrowed with render_size,
then restore it -- which is only safe on the render thread.
"""
offscreen = getattr(self.display_manager, 'offscreen', None)
if offscreen is not None:
with offscreen(width):
yield
return
original_image = self.display_manager.image.copy()
try:
with self._capture(), self._render_at(width or self.display_width):
yield
finally:
self.display_manager.image = original_image
def _fetch_content(
self, plugin: 'BasePlugin', plugin_id: str, restricted: bool
) -> Optional[List[Image.Image]]:
"""Every content path in order: native, scroll helper, display capture.
``restricted`` is the pre-offscreen contract for background callers:
skip every path that needs a canvas and return None instead.
"""
# Try native Vegas content method first # Try native Vegas content method first
has_native = hasattr(plugin, 'get_vegas_content') has_native = hasattr(plugin, 'get_vegas_content')
logger.debug("[%s] Has get_vegas_content: %s", plugin_id, has_native) logger.debug("[%s] Has get_vegas_content: %s", plugin_id, has_native)
if has_native: if has_native:
content = self._get_native_content(plugin, plugin_id, restricted) content = self._get_native_content(plugin, plugin_id, offscreen_only)
if content: if content:
total_width = sum(img.width for img in content) total_width = sum(img.width for img in content)
logger.debug( logger.debug(
@@ -212,7 +134,7 @@ class PluginAdapter:
# Try to get scroll_helper's cached image (for scrolling plugins like stocks/odds) # Try to get scroll_helper's cached image (for scrolling plugins like stocks/odds)
has_scroll_helper = hasattr(plugin, 'scroll_helper') has_scroll_helper = hasattr(plugin, 'scroll_helper')
logger.debug("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper) logger.debug("[%s] Has scroll_helper: %s", plugin_id, has_scroll_helper)
content = self._get_scroll_helper_content(plugin, plugin_id, restricted) content = self._get_scroll_helper_content(plugin, plugin_id, offscreen_only)
if content: if content:
total_width = sum(img.width for img in content) total_width = sum(img.width for img in content)
logger.debug( logger.debug(
@@ -223,8 +145,8 @@ class PluginAdapter:
if has_scroll_helper: if has_scroll_helper:
logger.debug("[%s] ScrollHelper content returned None", plugin_id) logger.debug("[%s] ScrollHelper content returned None", plugin_id)
if restricted: if offscreen_only:
# Display capture needs a canvas; leave it to the caller. # Display capture needs the shared canvas; leave it to the caller.
logger.debug( logger.debug(
"[%s] Needs display capture, deferring to the render thread", "[%s] Needs display capture, deferring to the render thread",
plugin_id plugin_id
@@ -760,7 +682,7 @@ class PluginAdapter:
return img.crop((start, 0, end, img.height)) return img.crop((start, 0, end, img.height))
def _get_native_content( def _get_native_content(
self, plugin: 'BasePlugin', plugin_id: str, restricted: bool = False self, plugin: 'BasePlugin', plugin_id: str, offscreen_only: bool = False
) -> Optional[List[Image.Image]]: ) -> Optional[List[Image.Image]]:
""" """
Get content via plugin's native get_vegas_content() method. Get content via plugin's native get_vegas_content() method.
@@ -789,21 +711,22 @@ class PluginAdapter:
plugin._vegas_render_width = render_width plugin._vegas_render_width = render_width
try: try:
# On a canvas of its own even at full width. Building Vegas # capture_mode unconditionally, even at full width. Building
# content is an off-screen operation, but a plugin is free to # Vegas content is an off-screen operation, but a plugin is free
# call update_display() while doing it, and on the shared canvas # to call update_display() while doing it — and outside
# that write would land on the hardware, flashing the panel # capture_mode that write lands on the hardware, flashing the
# mid-scroll. # panel mid-scroll. The narrowing context is separate because it
if restricted: # is a no-op at full width.
# Restricted (offscreen_prefetch off): no canvas of our own, if offscreen_only:
# so no narrowing. _vegas_render_width is set regardless: a # _render_at swaps the shared canvas, so it is unsafe here.
# plugin reading get_vegas_render_width() still gets its # _vegas_render_width is set regardless: a plugin reading
# narrow size, and one that only reads matrix.width renders # get_vegas_render_width() still gets its narrow size, and
# full width and is trimmed instead. # one that only reads matrix.width renders full width and is
# trimmed instead.
with self._capture(): with self._capture():
result = plugin.get_vegas_content() result = plugin.get_vegas_content()
else: else:
with self._isolated_canvas(render_width): with self._capture(), self._render_at(render_width):
result = plugin.get_vegas_content() result = plugin.get_vegas_content()
finally: finally:
plugin._vegas_render_width = None plugin._vegas_render_width = None
@@ -884,7 +807,7 @@ class PluginAdapter:
return None return None
def _get_scroll_helper_content( def _get_scroll_helper_content(
self, plugin: 'BasePlugin', plugin_id: str, restricted: bool = False self, plugin: 'BasePlugin', plugin_id: str, offscreen_only: bool = False
) -> Optional[List[Image.Image]]: ) -> Optional[List[Image.Image]]:
""" """
Get content from plugin's scroll_helper if available. Get content from plugin's scroll_helper if available.
@@ -918,7 +841,7 @@ class PluginAdapter:
"[%s] scroll_helper.cached_image is None, triggering content generation", "[%s] scroll_helper.cached_image is None, triggering content generation",
plugin_id plugin_id
) )
if restricted: if offscreen_only:
# Generating it calls display(), which needs the canvas. # Generating it calls display(), which needs the canvas.
logger.debug( logger.debug(
"[%s] scroll_helper cache empty; deferring generation " "[%s] scroll_helper cache empty; deferring generation "
@@ -1068,8 +991,12 @@ class PluginAdapter:
Returns: Returns:
The generated cached_image or None The generated cached_image or None
""" """
original_image = None
try: try:
with self._isolated_canvas(): # Save display state to restore after
original_image = self.display_manager.image.copy()
with self._capture():
# Method 1: Try _create_scrolling_display (stocks pattern) # Method 1: Try _create_scrolling_display (stocks pattern)
if hasattr(plugin, '_create_scrolling_display'): if hasattr(plugin, '_create_scrolling_display'):
logger.debug( logger.debug(
@@ -1125,6 +1052,11 @@ class PluginAdapter:
logger.exception("[%s] Error triggering scroll content", plugin_id) logger.exception("[%s] Error triggering scroll content", plugin_id)
return None return None
finally:
# Restore original display state
if original_image is not None:
self.display_manager.image = original_image
def _capture_display_content( def _capture_display_content(
self, plugin: 'BasePlugin', plugin_id: str self, plugin: 'BasePlugin', plugin_id: str
) -> Optional[List[Image.Image]]: ) -> Optional[List[Image.Image]]:
@@ -1138,7 +1070,12 @@ class PluginAdapter:
Returns: Returns:
List with single captured image, or None List with single captured image, or None
""" """
original_image = None
try: try:
# Save current display state
original_image = self.display_manager.image.copy()
logger.debug("[%s] Fallback: saved original display state", plugin_id)
# Ensure plugin has fresh data before capturing # Ensure plugin has fresh data before capturing
has_update_data = hasattr(plugin, 'update_data') has_update_data = hasattr(plugin, 'update_data')
logger.debug("[%s] Fallback: has update_data=%s", plugin_id, has_update_data) logger.debug("[%s] Fallback: has update_data=%s", plugin_id, has_update_data)
@@ -1149,12 +1086,12 @@ class PluginAdapter:
except (AttributeError, RuntimeError, OSError): except (AttributeError, RuntimeError, OSError):
logger.exception("[%s] Fallback: update_data() failed", plugin_id) logger.exception("[%s] Fallback: update_data() failed", plugin_id)
# Clear and call plugin display on a canvas of its own: nothing it # Clear and call plugin display — use capture_mode to suppress hardware writes
# draws, and no update_display() it calls, reaches the panel. # that plugins may trigger internally via update_display().
# #
# The canvas is render_width wide, so a plugin that spreads across # render_size narrows the canvas the plugin lays out against, so a
# the whole panel produces a compact arrangement rather than one # plugin that spreads across the whole panel produces a compact
# that has to be cropped afterwards. # arrangement rather than one that has to be cropped afterwards.
render_width = self.resolve_render_width(plugin, plugin_id) render_width = self.resolve_render_width(plugin, plugin_id)
if render_width != self.display_width: if render_width != self.display_width:
logger.debug( logger.debug(
@@ -1162,7 +1099,7 @@ class PluginAdapter:
plugin_id, render_width, self.display_width plugin_id, render_width, self.display_width
) )
with self._isolated_canvas(render_width): with self._capture(), self._render_at(render_width):
self.display_manager.clear() self.display_manager.clear()
logger.debug("[%s] Fallback: display cleared, calling display()", plugin_id) logger.debug("[%s] Fallback: display cleared, calling display()", plugin_id)
@@ -1196,7 +1133,7 @@ class PluginAdapter:
plugin_id plugin_id
) )
# Try once more with force_clear=True # Try once more with force_clear=True
with self._isolated_canvas(render_width): with self._capture(), self._render_at(render_width):
self.display_manager.clear() self.display_manager.clear()
plugin.display(force_clear=True) plugin.display(force_clear=True)
captured = self.display_manager.image.copy() captured = self.display_manager.image.copy()
@@ -1233,6 +1170,12 @@ class PluginAdapter:
) )
return None return None
finally:
# Always restore original image to prevent display corruption
if original_image is not None:
self.display_manager.image = original_image
logger.debug("[%s] Fallback: restored original display state", plugin_id)
def _is_blank_image( def _is_blank_image(
self, img: Image.Image, return_ratio: bool = False self, img: Image.Image, return_ratio: bool = False
) -> Union[bool, Tuple[bool, float]]: ) -> Union[bool, Tuple[bool, float]]:

Some files were not shown because too many files have changed in this diff Show More