rick/LEDMatrix
1
0
Fork 0
mirror of https://github.com/ChuckBuilds/LEDMatrix.git synced 2026-10-04 06:15:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(vegas): render plugin content off the render thread, and keep it off the GIL when the panel needs it (#630)

Browse Source 
DisplayManager.offscreen() gives a thread its own canvas, so Vegas renders every plugin's ticker content on its prefetch thread instead of pausing the scroll for canvas-bound plugins on the render thread. A render gate (src/common/render_gate.py, vegas_scroll.prefetch_gate, on by default with the GIL-releasing binding) lets the prefetch thread run Python only while the render thread waits in SwapOnVSync: on hdpi, frames 2+ refreshes late fell eightfold and late frames overall from 0.90% to 0.60%. See docs/OFFSCREEN_RENDERING.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-09-24 19:57:03 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 865d62f67b
commit 9964dd2183
13 changed files with 1903 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/OFFSCREEN_RENDERING.md
+388
View File
@@ -0,0 +1,388 @@
# Offscreen Rendering
**Status (2026-09-24):** step 1, offscreen rendering, is implemented
(`DisplayManager.offscreen()`, the adapter on the prefetch thread, the plugin
lock). Steps 2 and 3 are proposed. When all three land, this file becomes the
reference for how plugin content is rendered off the render thread.
First soak of step 1 on hdpi (50 px/s, `pwm_bits` 7, preview open, 8-minute
runs, A/B/B/A):
| build | late | by 1 | 2 | 3–5 | 6+ | freezes | render-thread fetches |
|---|---|---|---|---|---|---|---|
| #628 | 0.53% | 82 | 2 | 3 | 2 | 3 | 6 |
| step 1 | 0.63% | 78 | 63 | 17 | 2 | 1 | 0 |
| step 1 | 0.42% | 77 | 23 | 10 | 0 | 0 | 0 |
| #628 | 0.37% | 84 | 5 | 3 | 3 | 2 | 14 |
It does what it was built to: no plugin is fetched on the render thread, and
freezes fell from 5 to 1. But frames 2–5 refreshes late rose. The rendering
moved to the prefetch thread still needs the GIL, and the render thread waits
for it (risk 5 below). The late rate did not improve overall. The 1–2 s
freezes appear in both builds and have a separate, not yet identified cause.
The GIL fix, measured on hdpi (90 px/s, `pwm_bits` 8, preview open, 8-minute
runs after a 2-minute warm-up, order A B C C B A, 2026-09-24). Each arm pools
two runs, about 81,000 frames:
| arm | late | by 1 | 2 | 3–5 | 6+ | 2+ late per 10k frames | freezes |
|---|---|---|---|---|---|---|---|
| A: step 1 as is | 0.90% | 575 | 64 | 91 | 9 | 20.1 | 0 |
| B: `switch_interval_ms` 1 | 0.78% | 510 | 105 | 23 | 2 | 15.8 | 0 |
| C: `prefetch_gate` | **0.60%** | 471 | 11 | 7 | 2 | **2.5** | 0 |
The gate removes the frames the render thread spent waiting for the GIL, and
it costs the prefetch nothing that shows: it parked the thread for 3–6 s per
run, and the next group was ready at every strip extension in every arm.
`prefetch_gate` is therefore on by default; `switch_interval_ms` stays an
off-by-default experiment. What is left is almost all one refresh late, which
is the per-frame budget (a 6.75 ms p50 blit in a refresh the panel holds at
83–85 Hz while rendering), not contention.
The runs restart the service, so the hourly sports refresh never fell inside
one. That refresh is its own case: about twenty ESPN chunk-fetch threads at
once, which the gate does not cover (it gates only the prefetch thread).
## The problem
Vegas mode builds its ticker from every plugin's content. Most of that work
already happens on a background prefetch thread
(`RenderPipeline.start_prefetch`). But any plugin whose content needs the
**shared display canvas** is deferred to the render thread
(`RenderPipeline.drain_deferred`), one plugin every two seconds. The code's
own comments put each of those at 40–600 ms, and the render thread presents no
frames while one runs.
On hdpi (Pi 4, 512×64) most plugins take that path: geochron, tide-display,
news, hockey-scoreboard, ledmatrix-stocks, incoming-packages, clock-simple,
countdown, birdnet-go, ledmatrix-music and odds-ticker. They arrive in bursts
("Whole group deferred; strip will extend as it drains") every minute or so,
12 fetches in five minutes. That is the "occasional pause" a viewer sees.
An 8-minute soak (`scripts/frame_soak.py --preview`) of the #628 build on
hdpi:
| late by | frames |
|---|---|
| 1 refresh | 238 |
| 2 | 32 |
| 3–5 | 30 |
| 6+ | 5 |
| freezes ≥ 250 ms | 2 (0.97 s total) |
The 3+ rows and the freezes are the pauses. The single-refresh row is a
separate problem: the blit is 6 ms of a 10 ms refresh, so there is little
slack. It is covered under *What this does not fix*.
## Why a plugin is canvas-bound
The plugin-facing canvas is a set of shared attributes on `DisplayManager`:
`image`, `draw`, `matrix`, and the `width`/`height` properties that read from
`matrix`. Three adapter paths (`src/vegas_mode/plugin_adapter.py`) need them,
and each returns `None` under `offscreen_only=True` so the plugin is queued for
the render thread:
1. **Display capture** (`_capture_display_content`): clear the canvas, call
`plugin.display()`, copy `display_manager.image`. Used by any plugin
without `get_vegas_content()` or a populated `scroll_helper`.
2. **Scroll-content generation** (`_trigger_scroll_content_generation`): a
ticker plugin whose `scroll_helper.cached_image` is empty is made to build
it by calling `display(force_clear=True)` or `_create_scrolling_display()`.
Both draw on the canvas.
3. **Narrowed rendering** (`DisplayManager.render_size`): swaps the shared
`matrix`, `image` and `draw` for a narrower set so the plugin lays out for
`render_width_pct`. The render thread would see the swap mid-frame.
The render thread keeps the canvas coherent only because nothing else touches
it at the same time. A background thread can't use it.
## The design: a per-thread render target
`capture_mode()` is already per-thread (#423 made its state a
`threading.local`, so a background capture no longer suppresses the render
loop's pushes). The same move applies to the canvas itself:
```python
with display_manager.offscreen(width=None, height=None) as surface:
plugin.display(force_clear=True)
content = surface.image.copy()
```
For the **calling thread only**, inside the block:
| accessor | resolves to |
|---|---|
| `display_manager.image`, `.draw` | the surface's own image and draw: a fresh black canvas, `fontmode = "1"` |
| `display_manager.matrix` | a logical proxy reporting the surface size, so `width`/`height` and plugins that read `matrix.width` follow it. Hardware calls through it (`SetImage`, `SwapOnVSync`, `Clear`, brightness writes) are inert. |
| `update_display()`, `clear()` | canvas-only: the block implies capture mode, which is already per-thread |
| `set_scrolling_state()`, `set_frame_hold()` | no-ops, so a plugin's `display()` cannot re-pace the live scroll. Today it can, when it is captured on the render thread. |
Every other thread sees the real canvas, unchanged. The render loop in
particular keeps presenting while a plugin draws elsewhere.
### Implementation sketch
- `image`, `draw` and `matrix` become properties over `_image`, `_draw` and
`_matrix`, plus a thread-local current surface. The getter returns the
surface's value when the calling thread has one, else the shared one; setters
mirror that. That costs about 0.1 µs per access, and `update_display()` reads
each a handful of times per frame. Every existing `self.image = ...` in
`DisplayManager` (`clear()`, setup, fallback) keeps working and becomes
thread-correct for free.
- `render_size()` is rebuilt on `offscreen()`: it creates or narrows the
calling thread's surface instead of swapping shared state.
- `offscreen()` nests and always restores on exit, including when the plugin
raises.
- `VisualDisplayManager` (the plugin test harness) gets the same method, for
parity.
### Adapter changes
- `get_content(offscreen_only=True)` stops returning `None` for the three
paths above. Each runs inside `display_manager.offscreen(render_width)`.
- `_capture_display_content` and `_trigger_scroll_content_generation` drop
their "copy the shared image, restore it afterwards" bookkeeping, since the
shared image is never touched.
- **Take the plugin's lock.** `PluginManager.get_plugin_lock()` keeps
`update()` and `display()` mutually exclusive in normal rotation, but Vegas
never takes it, so today's render-thread captures already race
`update()`. Off the render thread the adapter can afford to wait: blocking
acquire with a timeout (proposed 2 s). On timeout it keeps the cached segment
and tries again next group.
- `drain_deferred()` and the deferred queue are deleted. The only render-thread
fetch left is the inline fallback when no prepared group is ready, which in
practice is the first extension. Prefetching at start removes that too.
## Keeping live content fresh
Offscreen rendering is also what makes fresh sports scores possible. Today a
plugin's segment is drawn when its group is prefetched, and the strip carries
7,000–10,000 px of content ahead of the viewport (hdpi logs: "7153px still
ahead", "9842px ahead"). At ~100 px/s, a score drawn now reaches the screen
70–100 seconds later. When a plugin reports new data, Vegas only drops its
cache (`invalidate_pending_updates`), so the change is drawn on the plugin's
*next* turn, several minutes later. A segment already in the strip scrolls by
with the data it was drawn with.
That was the right trade while every redraw of a canvas-bound plugin stalled
the scroll. Off the render thread a redraw costs the scroll nothing, so the
strip can afford three things.
### 1. Refresh at the gate
Before a segment enters the viewport, check whether its plugin has updated
since the segment was drawn. If it has, redraw it offscreen and replace it
while it is still out of sight. Width changes are fine here, because
everything from that segment onward is still invisible.
The gate sits `lead` pixels ahead of the viewport's right edge:
`lead = max(one screen, speed × (render time + margin))`. The render time is
the plugin's own, measured on each render (sports cards take the longest,
hundreds of ms up to seconds per the prefetch notes). A plugin whose render
does not finish before its segment reaches the viewport keeps the old segment.
The scroll never waits for it.
Content is then at most `lead / speed` seconds old when it appears, a few
seconds instead of minutes, without changing how far ahead the rotation
fetches.
### 2. Replace ahead of the screen
When a plugin reports new data (the Vegas update tick already names them), any
of its segments that are **anywhere ahead of the viewport** are redrawn and
replaced straight away, not only at the gate. That covers the long stretch of
strip between prefetch and the gate.
### 3. Update on screen
A segment that is already **visible** is patched in place when the redrawn
version has the same geometry: the same total width, and the same width for
each card (a sports plugin returns one image per game, joined with
`intra_plugin_gap`). Scoreboard cards keep a fixed layout, so a score change
patches in and the digits update as the card scrolls past. The patch is a
pixel copy of one card (a 150×64 card is ~29 KB) applied by the render thread
between frames, so a frame never shows half of a patch.
When the geometry differs (a game added or dropped, a card that grew), the
visible part cannot change without a jump. Only the cards not yet on screen
are replaced, and only if the geometry up to that point is unchanged. Otherwise
the segment keeps its snapshot until it has scrolled off.
### Avoiding wasted work
- **Change detection.** `run_scheduled_updates_with_changes()` names a plugin
whenever its `update()` ran, not when its data changed. On hdpi
`clock-simple` and `ledmatrix-music` are named on every 4-second tick. A
redraw whose pixels hash the same as the segment's is discarded without a
swap.
- **Redraw on real updates only.** Vegas makes no API calls. Each plugin
fetches on its own schedule, and a redraw is triggered only when the
plugin's `update()` has run since its segment was drawn. On hdpi live
football, baseball and hockey poll every 30 s (live odds every 60 s,
everything else hourly), so a live sports card is redrawn once per poll.
- **Floor.** A plugin is redrawn at most once per
`vegas_scroll.refresh_min_interval` (proposed 10 s), and never while its
previous redraw is still running. The floor never holds back a sports card
polling every 30 s. It exists for chatty plugins: `clock-simple` updates
every second and `ledmatrix-music` polls every 2 s.
- **One worker.** Redraws go through the same background worker as prefetch,
one plugin at a time at `nice 10`, under the plugin's lock.
Data freshness is still bounded by each plugin's own fetch interval (how often
it polls live scores). Drawing faster cannot beat the data source.
### The strip becomes a list of segments
All three need the strip to be replaceable by segment. Today it is one
image (`ScrollHelper.cached_array`, 8,000–20,000 px wide, 1.5–3.8 MB), and
`append_content()` rebuilds the whole thing on the render thread for every
appended block. That is also a pause source.
Proposed `SegmentStrip`, used by Vegas in place of the single image:
- an ordered list of segments: plugin id, card boundaries, a pixel array, the
render time, and the plugin data version it was drawn from, plus its
x-offset in the strip;
- `visible(x, width)` assembles the viewport by slicing across at most a few
segments: the same ~100 KB copy per frame that slicing the single image
costs today;
- append and trim become O(block) list operations, not a copy of the strip;
- replace swaps one list entry and shifts the offsets of the segments after it
(dozens at most). A same-geometry patch copies pixels into the existing array.
Every mutation is prepared off the render thread and applied by the render
thread at a frame boundary, so the strip the render loop reads is never
half-changed.
### Multi-display sync
The follower renders from its own copy of the strip, offset from the leader's
scroll position. Today the leader sends that copy whole, and only in
`start_new_cycle()` (`send_scroll_image`), plus the scroll position every
frame. Continuous scroll, the default, extends and trims the strip without
starting a new cycle, and nothing sends those changes. From reading the code,
the follower therefore probably falls out of step after the first extension
already, before any of this design. That is untested; it needs a two-Pi rig.
With a segment strip, keeping the follower identical becomes **replaying the
leader's operations**:
- Every strip mutation (append, trim, replace, patch) is one operation in
strip coordinates. The leader applies it and sends the same operation to the
follower over the existing TCP channel. Segments are small: a card is ~29 KB
raw and compresses well.
- Operations on off-screen segments apply on arrival. A patch to a segment
that is on either panel carries an *apply at scroll position X* stamp a
couple of hundred milliseconds ahead. Both sides apply it when their scroll
position passes X, so both panels change on the same frame, within the
existing position-sync jitter.
- Each operation carries a sequence number. A follower that sees a gap (a
reconnect, a dropped message) asks for a full snapshot, which is today's
`send_scroll_image` path.
That also fixes the probable continuous-mode gap as a side effect, since
appends and trims become operations too. Until it is in place, fresh-content
updates are disabled while sync is active.
## Risks, and what was checked
1. **Plugins holding their own reference to the shared `draw` or `image`.**
They would keep drawing into the shared canvas, and routing by thread can't
redirect them. A grep of the 49 plugins installed on hdpi found none storing
`display_manager.draw` or `.image` in an attribute (a pattern search, so
indirect aliasing would slip past it). A plugin that did would
draw into an image nobody displays, which trims to a blank segment. That is
not corruption, and it is no worse than today.
2. **Plugins calling the matrix directly.** None in the audit. Inside
`offscreen()` the proxy makes it inert anyway.
3. **Font thread-safety.** `FontManager` shares font objects across plugins.
Measured on Pillow 12.3, two threads rendering text take 1.94× as long as
one, so text rendering holds the GIL and FreeType is never entered
concurrently. Re-check if Pillow changes that.
4. **Plugin thread-safety.** `display()` moves to the prefetch thread. The
plugin lock makes it exclusive with `update()`, which is more protection
than it has today. Threads a plugin starts itself are not covered, as today.
5. **The GIL.** Moving 40–600 ms of plugin rendering off the render thread
removes the pauses, but the work still needs the GIL. Pillow drawing holds
it, and a waiting thread only gets it back after the switch interval
(default 5 ms). Expect some single-refresh late frames while a prefetch
runs. Measure with the soak. A render process separate from plugin work
is the structural answer (the "native presenter" step). Two experiments
get most of the way first (results under Status, above):
- `vegas_scroll.switch_interval_ms` lowers the switch interval for a Vegas
run (1 ms is the obvious try), so the render thread waits at most that
long behind bytecode. It does nothing for a C call that keeps the GIL.
- `vegas_scroll.prefetch_gate` (`src/common/render_gate.py`) lets the
prefetch thread run Python only while the render thread is blocked in
`SwapOnVSync`, up to just before the refresh the swap returns on, and
parks it the rest of the time. That covers C calls too, since the gate is
checked before each one starts. It never parks the thread while it holds
a lock the render thread takes, and never for more than 50 ms. It needs
the rebuilt binding, which releases the GIL during the swap. On by
default.
## What this does not fix
- **The blit.** Copying a 512×64 frame into the matrix (`SetImage`) is ~6 ms at
8 PWM bits on a Pi 4, leaving ~4 ms of slack per refresh. That is the main
source of the single-refresh late frames. Holding frames for two refreshes
(≈50 px/s) doubles the budget. Cutting the blit itself is the native-presenter
step.
- **Live refreshes pushed from `update()`.** Some sports plugins call
`display()` and `update_display()` from inside `update()`, which runs on the
update worker and can push to the panel mid-Vegas. That is a separate
hazard. `offscreen()` gives a tool for it (run the update worker offscreen
while Vegas owns the panel), but it is out of scope here.
## Test plan
- **Unit, `DisplayManager`:** one thread inside `offscreen()` draws while
another reads `image`/`draw`/`matrix`/`width`/`height` and sees the real
canvas. Also: `update_display()` and `set_scrolling_state()` are inert inside;
`render_size()` narrows only the calling thread; nesting and exceptions
restore state.
- **Unit, adapter:** a stub display-capture plugin and a stub scroll-helper
plugin both return content with `offscreen_only=True`, and nothing is queued
for the render thread. The plugin lock is taken, and a timeout keeps the cached
segment.
- **Emulator integration:** a stub canvas-bound plugin whose `display()` sleeps
300 ms. The Vegas render loop never goes a frame without presenting (frame
timing recorder: zero freezes).
- **Unit, `SegmentStrip`:** the viewport assembled across segment boundaries
matches slicing one concatenated image, pixel for pixel. Append, trim,
replace-ahead and same-geometry patch each leave every other column
unchanged. A geometry-changing patch of a visible segment is refused.
- **Freshness:** a stub sports plugin whose score changes every second. The
score on screen is never older than `lead / speed` plus the plugin's fetch
interval. A visible card's digits change without the frame-timing recorder
seeing a late frame. An unchanged redraw is discarded.
- **Hardware:** an hdpi soak, A/B against the #628 build, alternating order.
Targets: no freezes, an empty 6+ bucket, the 3–5 bucket near zero, and the late
rate below 0.66%. Plus, for freshness: log each segment's age when it enters
the viewport, and compare the median and max before and after.
## Rollout
Three changes, each soaked on hdpi before the next:
1. **Offscreen rendering:** `offscreen()`, the adapter on the prefetch thread,
and the plugin lock. Removes the render-thread pauses.
2. **`SegmentStrip`:** Vegas's strip becomes a list of segments. Removes the
whole-strip copy on append. No visible behaviour change.
3. **Fresh content:** refresh at the gate, replace ahead, patch on screen,
with change detection and the rate limit.
`display.vegas_scroll.offscreen_prefetch` (default `true`) restores today's
deferred path when `false`, and `display.vegas_scroll.live_refresh` (default
`true`) turns off step 3. Keep both for one release, then delete the old paths.
## Open questions
1. Keep the kill switch, or ship without one?
2. Plugin lock timeout: skip the plugin and keep its cached segment (proposed),
or wait longer?
3. `refresh_min_interval`: 10 s proposed. It only limits chatty plugins;
live sports are redrawn once per 30 s poll regardless.
4. Multi-display sync: is there a two-Pi rig to test on? Operation replay is
proposed as part of the segment strip (step 2), with fresh content
disabled under sync until it has been verified on real hardware.