LEDMatrix/docs/OFFSCREEN_RENDERING.md

# Offscreen Rendering

**Status (2026-09-30):** offscreen rendering is implemented
(`DisplayManager.offscreen()`, the adapter on the prefetch thread, the plugin
lock), and so are live elements, which grew out of steps 2 and 3 below: see
*Live elements*. The segment strip proposed as step 2 was not needed; *Why not
a SegmentStrip* says why.

First soak of step 1 on hdpi (50 px/s, `pwm_bits` 7, preview open, 8-minute
runs, A/B/B/A):

| build | late | by 1 | 2 | 3–5 | 6+ | freezes | render-thread fetches |
|---|---|---|---|---|---|---|---|
| #628 | 0.53% | 82 | 2 | 3 | 2 | 3 | 6 |
| step 1 | 0.63% | 78 | 63 | 17 | 2 | 1 | 0 |
| step 1 | 0.42% | 77 | 23 | 10 | 0 | 0 | 0 |
| #628 | 0.37% | 84 | 5 | 3 | 3 | 2 | 14 |

It does what it was built to: no plugin is fetched on the render thread, and
freezes fell from 5 to 1. But frames 2–5 refreshes late rose. The rendering
moved to the prefetch thread still needs the GIL, and the render thread waits
for it (risk 5 below). The late rate did not improve overall. The 1–2 s
freezes appear in both builds and have a separate, not yet identified cause.

The GIL fix, measured on hdpi (90 px/s, `pwm_bits` 8, preview open, 8-minute
runs after a 2-minute warm-up, order A B C C B A, 2026-09-24). Each arm pools
two runs, about 81,000 frames:

| arm | late | by 1 | 2 | 3–5 | 6+ | 2+ late per 10k frames | freezes |
|---|---|---|---|---|---|---|---|
| A: step 1 as is | 0.90% | 575 | 64 | 91 | 9 | 20.1 | 0 |
| B: `switch_interval_ms` 1 | 0.78% | 510 | 105 | 23 | 2 | 15.8 | 0 |
| C: `prefetch_gate` | **0.60%** | 471 | 11 | 7 | 2 | **2.5** | 0 |

The gate removes the frames the render thread spent waiting for the GIL, and
it costs the prefetch nothing that shows: it parked the thread for 3–6 s per
run, and the next group was ready at every strip extension in every arm.
`prefetch_gate` is therefore on by default; `switch_interval_ms` stays an
off-by-default experiment. What is left is almost all one refresh late, which
is the per-frame budget (a 6.75 ms p50 blit in a refresh the panel holds at
83–85 Hz while rendering), not contention.

The runs restart the service, so the hourly sports refresh never fell inside
one. That refresh is its own case: about twenty ESPN chunk-fetch threads at
once, which the gate does not cover (it gates only the prefetch thread).

## The problem

Vegas mode builds its ticker from every plugin's content. Most of that work
already happens on a background prefetch thread
(`RenderPipeline.start_prefetch`). But any plugin whose content needs the
**shared display canvas** is deferred to the render thread
(`RenderPipeline.drain_deferred`), one plugin every two seconds. The code's
own comments put each of those at 40–600 ms, and the render thread presents no
frames while one runs.

On hdpi (Pi 4, 512×64) most plugins take that path: geochron, tide-display,
news, hockey-scoreboard, ledmatrix-stocks, incoming-packages, clock-simple,
countdown, birdnet-go, ledmatrix-music and odds-ticker. They arrive in bursts
("Whole group deferred; strip will extend as it drains") every minute or so,
12 fetches in five minutes. That is the "occasional pause" a viewer sees.

An 8-minute soak (`scripts/frame_soak.py --preview`) of the #628 build on
hdpi:

| late by | frames |
|---|---|
| 1 refresh | 238 |
| 2 | 32 |
| 3–5 | 30 |
| 6+ | 5 |
| freezes ≥ 250 ms | 2 (0.97 s total) |

The 3+ rows and the freezes are the pauses. The single-refresh row is a
separate problem: the blit is 6 ms of a 10 ms refresh, so there is little
slack. It is covered under *What this does not fix*.

## Why a plugin is canvas-bound

The plugin-facing canvas is a set of shared attributes on `DisplayManager`:
`image`, `draw`, `matrix`, and the `width`/`height` properties that read from
`matrix`. Three adapter paths (`src/vegas_mode/plugin_adapter.py`) need them,
and each returns `None` under `offscreen_only=True` so the plugin is queued for
the render thread:

1. **Display capture** (`_capture_display_content`): clear the canvas, call
   `plugin.display()`, copy `display_manager.image`. Used by any plugin
   without `get_vegas_content()` or a populated `scroll_helper`.
2. **Scroll-content generation** (`_trigger_scroll_content_generation`): a
   ticker plugin whose `scroll_helper.cached_image` is empty is made to build
   it by calling `display(force_clear=True)` or `_create_scrolling_display()`.
   Both draw on the canvas.
3. **Narrowed rendering** (`DisplayManager.render_size`): swaps the shared
   `matrix`, `image` and `draw` for a narrower set so the plugin lays out for
   `render_width_pct`. The render thread would see the swap mid-frame.

The render thread keeps the canvas coherent only because nothing else touches
it at the same time. A background thread can't use it.

## The design: a per-thread render target

`capture_mode()` is already per-thread (#423 made its state a
`threading.local`, so a background capture no longer suppresses the render
loop's pushes). The same move applies to the canvas itself:

```python
with display_manager.offscreen(width=None, height=None) as surface:
    plugin.display(force_clear=True)
    content = surface.image.copy()
```

For the **calling thread only**, inside the block:

| accessor | resolves to |
|---|---|
| `display_manager.image`, `.draw` | the surface's own image and draw: a fresh black canvas, `fontmode = "1"` |
| `display_manager.matrix` | a logical proxy reporting the surface size, so `width`/`height` and plugins that read `matrix.width` follow it. Hardware calls through it (`SetImage`, `SwapOnVSync`, `Clear`, brightness writes) are inert. |
| `update_display()`, `clear()` | canvas-only: the block implies capture mode, which is already per-thread |
| `set_scrolling_state()`, `set_frame_hold()` | no-ops, so a plugin's `display()` cannot re-pace the live scroll. Today it can, when it is captured on the render thread. |

Every other thread sees the real canvas, unchanged. The render loop in
particular keeps presenting while a plugin draws elsewhere.

### Implementation sketch

- `image`, `draw` and `matrix` become properties over `_image`, `_draw` and
  `_matrix`, plus a thread-local current surface. The getter returns the
  surface's value when the calling thread has one, else the shared one; setters
  mirror that. That costs about 0.1 µs per access, and `update_display()` reads
  each a handful of times per frame. Every existing `self.image = ...` in
  `DisplayManager` (`clear()`, setup, fallback) keeps working and becomes
  thread-correct for free.
- `render_size()` is rebuilt on `offscreen()`: it creates or narrows the
  calling thread's surface instead of swapping shared state.
- `offscreen()` nests and always restores on exit, including when the plugin
  raises.
- `VisualDisplayManager` (the plugin test harness) gets the same method, for
  parity.

### Adapter changes

- `get_content(offscreen_only=True)` stops returning `None` for the three
  paths above. Each runs inside `display_manager.offscreen(render_width)`.
- `_capture_display_content` and `_trigger_scroll_content_generation` drop
  their "copy the shared image, restore it afterwards" bookkeeping, since the
  shared image is never touched.
- **Take the plugin's lock.** `PluginManager.get_plugin_lock()` keeps
  `update()` and `display()` mutually exclusive in normal rotation, but Vegas
  never takes it, so today's render-thread captures already race
  `update()`. Off the render thread the adapter can afford to wait: blocking
  acquire with a timeout (proposed 2 s). On timeout it keeps the cached segment
  and tries again next group.
- `drain_deferred()` and the deferred queue are deleted. The only render-thread
  fetch left is the inline fallback when no prepared group is ready, which in
  practice is the first extension. Prefetching at start removes that too.

## Live elements: content that changes while it scrolls

Offscreen rendering is also what makes fresh content possible. A plugin's
segment is drawn when its group is prefetched, and the strip carries
7,000-10,000 px of content ahead of the viewport, so at ~100 px/s a score drawn
then reaches the screen 70-100 seconds later -- and once in the strip it never
changed: when a plugin reported new data, Vegas only dropped its caches, so the
change appeared on the plugin's *next* turn, minutes later.

A plugin can now hand Vegas **live elements** instead of pictures
(`BasePlugin.get_vegas_elements()`, see "Live Vegas elements" in
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md#live-vegas-elements)):
named, fixed-width pieces of content -- one per game card, one for a map. Vegas
records where each lands in the strip and, when the plugin's data changes,
redraws just the changed ones off the render thread and copies their pixels
over the old ones between two frames. A card already crossing the panel
changes; nothing next to it moves.

### Why not redraw every frame

On a Pi the render thread has about 4 ms of slack per refresh at 512x64 after
the ~6 ms blit, and a scoreboard card is ~29 ms of Pillow work that holds the
GIL. Drawing on the render thread is out of the question at any rate, so the
render thread only ever *copies* pixels that are already drawn. Measured on a
Pi 4 (ledpi): writing a 35 KB card into a 20,000 px strip takes 8.5 µs, a
101 KB map 17 µs, four cards (the per-frame cap) 34 µs -- against 124 µs for
the viewport slice every frame already does.

### How an update reaches the screen

1. A plugin's `update()` completes. The update worker calls
   `PluginManager._note_update_completed`, which calls the update listeners
   (`add_update_listener`) there and then, with the plugin's lock still held.
   Vegas's listener moves the plugin's **epoch** on
   (`src/vegas_mode/elements.py`, `LiveEpochs`) and wakes the live worker.
2. The **live worker** (`src/vegas_mode/live_worker.py`), the one background
   thread that draws for the strip once it holds a live element, finds the
   plugin's elements whose recorded epoch is older than its current one,
   nearest the screen first, and calls `get_vegas_elements()` under the
   plugin's lock (0.25 s wait, then a 1 s backoff). Elements whose `version`
   is unchanged cost nothing; the rest are pinned and checksummed, and each
   whose pixels changed becomes a patch in a one-per-element slot (the latest
   wins).
3. Between two frames the render thread
   (`RenderPipeline.apply_live_patches`, from `coordinator.run_frame`) pops at
   most four patches or two screens of bytes and copies each into the strip
   with `ScrollHelper.patch_columns`. It takes no lock and draws nothing. A
   patch made for an older strip, for an element trimmed away or already
   behind the screen, or from older data than the strip shows, is dropped.

End to end, a new score reaches a card already on screen within one poll of
the data source (30 s for live games) plus about a second: the listener is
immediate, and while live elements exist the update tick that schedules
plugins runs every second instead of every four.

Elements that change with **time** rather than data (an aircraft moving
between position reports) ask for `refresh_hz`; the worker calls
`redraw_vegas_element()` -- without the plugin's lock, from state the plugin
publishes in one assignment -- that often while the element is on or within
`live_lead_screens` of the screen, capped by `live_max_hz` (5), at 1 Hz
without the render gate, and halved for an element whose redraws average over
50 ms.

### Geometry

A live element is never trimmed to its ink: the adapter pads it with
`content_padding` black columns either side and pins its width, and a redraw
at any other width is refused (it shows the next time the plugin comes round).
Records keep **absolute** strip columns -- the strip column plus everything
trimmed off the front since the strip was composed -- so a trim moves one
origin rather than every record. Nothing on screen is ever moved, inserted or
resized; a game added to a slate appears on the plugin's next turn.

### Why not a SegmentStrip

The proposal here was to replace the single strip with a list of segments.
In-place patching of the single strip meets every goal without that: the
patch is O(element) and the strip layout never changes. What a segment list
would still buy is cheaper extensions, and most of that came from making the
strip's PIL copy lazy instead (`ScrollHelper.cached_image`: an extension used
to rebuild it twice, 1.7-3.8 ms each on a Pi 4). The `extend` row of
`frame_soak.py`'s "after work" table says whether the rest is worth it.

### When it is off

- `display.vegas_scroll.live_refresh: false` (the kill switch; also in the
  web UI), or `vegas_live: false` in one plugin's section.
- Always under multi-display sync: the follower mirrors whole strips only, so
  a patch would never reach it. (Continuous-mode sync has a separate problem:
  the follower is not sent extensions or trims at all.)
- In swap mode (`continuous_scroll: false`) and with `offscreen_prefetch:
  false`.
- For plugins without the hook, which are drawn and placed exactly as before,
  and on the paths that fetch without the plugin's lock (the first strip of a
  run, the render thread's fallback fetch), which use `get_vegas_content()`.

## Risks, and what was checked

1. **Plugins holding their own reference to the shared `draw` or `image`.**
   They would keep drawing into the shared canvas, and routing by thread can't
   redirect them. A grep of the 49 plugins installed on hdpi found none storing
   `display_manager.draw` or `.image` in an attribute (a pattern search, so
   indirect aliasing would slip past it). A plugin that did would
   draw into an image nobody displays, which trims to a blank segment. That is
   not corruption, and it is no worse than today.
2. **Plugins calling the matrix directly.** None in the audit. Inside
   `offscreen()` the proxy makes it inert anyway.
3. **Font thread-safety.** `FontManager` shares font objects across plugins.
   Measured on Pillow 12.3, two threads rendering text take 1.94× as long as
   one, so text rendering holds the GIL and FreeType is never entered
   concurrently. Re-check if Pillow changes that.
4. **Plugin thread-safety.** `display()` moves to the prefetch thread. The
   plugin lock makes it exclusive with `update()`, which is more protection
   than it has today. Threads a plugin starts itself are not covered, as today.
5. **The GIL.** Moving 40–600 ms of plugin rendering off the render thread
   removes the pauses, but the work still needs the GIL. Pillow drawing holds
   it, and a waiting thread only gets it back after the switch interval
   (default 5 ms). Expect some single-refresh late frames while a prefetch
   runs. Measure with the soak. A render process separate from plugin work
   is the structural answer (the "native presenter" step). Two experiments
   get most of the way first (results under Status, above):
   - `vegas_scroll.switch_interval_ms` lowers the switch interval for a Vegas
     run (1 ms is the obvious try), so the render thread waits at most that
     long behind bytecode. It does nothing for a C call that keeps the GIL.
   - `vegas_scroll.prefetch_gate` (`src/common/render_gate.py`) lets the
     prefetch thread run Python only while the render thread is blocked in
     `SwapOnVSync`, up to just before the refresh the swap returns on, and
     parks it the rest of the time. That covers C calls too, since the gate is
     checked before each one starts. It never parks the thread while it holds
     a lock the render thread takes, and never for more than 50 ms. It needs
     the rebuilt binding, which releases the GIL during the swap. On by
     default.

## What this does not fix

- **The blit.** Copying a 512×64 frame into the matrix (`SetImage`) is ~6 ms at
  8 PWM bits on a Pi 4, leaving ~4 ms of slack per refresh. That is the main
  source of the single-refresh late frames. Holding frames for two refreshes
  (≈50 px/s) doubles the budget. Cutting the blit itself is the native-presenter
  step.
- **Multi-display sync in continuous mode.** The follower is sent the whole
  strip only at a new cycle and on connect, never the extensions and trims of
  continuous mode, so it drifts from the leader after the first extension.
  Live elements stay off under sync for that reason.

## Test plan

- **Unit, `DisplayManager`:** one thread inside `offscreen()` draws while
  another reads `image`/`draw`/`matrix`/`width`/`height` and sees the real
  canvas. Also: `update_display()` and `set_scrolling_state()` are inert inside;
  `render_size()` narrows only the calling thread; nesting and exceptions
  restore state.
- **Unit, adapter:** a stub display-capture plugin and a stub scroll-helper
  plugin both return content with `offscreen_only=True`, and nothing is queued
  for the render thread. The plugin lock is taken, and a timeout keeps the cached
  segment.
- **Emulator integration:** a stub canvas-bound plugin whose `display()` sleeps
  300 ms. The Vegas render loop never goes a frame without presenting (frame
  timing recorder: zero freezes).
- **Live elements** (`test/test_vegas_live_*.py`,
  `test/test_vegas_elements_*.py`, `test/test_scroll_helper_patch.py`): every
  record points at exactly its element's pixels through any sequence of
  compose, extend and trim; a patch changes only its element's columns (a
  property test against a twin strip that is never patched); the render
  thread's apply takes no lock and draws nothing; the worker's priorities,
  floors, backoff and hand-over; and, end to end on the emulator with the stub
  plugin (`test/fixtures/plugins/vegas-live-stub`), an update changes a card
  already in the strip and an animated element moves with no update at all.
- **Hardware:** an hdpi soak, A/B against the #628 build, alternating order.
  Targets: no freezes, an empty 6+ bucket, the 3–5 bucket near zero, and the late
  rate below 0.66%. Plus, for freshness: log each segment's age when it enters
  the viewport, and compare the median and max before and after.

## Rollout

1. **Offscreen rendering** (shipped): `offscreen()`, the adapter on the
   prefetch thread, and the plugin lock. Removed the render-thread pauses.
2. **Measurement and the lazy strip image:** late frames attributed to the
   render-thread work before them (`FrameTimingRecorder.note_op`, the "after
   work" table), and extensions no longer rebuilding the strip's PIL copy.
3. **Live elements:** the plugin API, the records, the worker and in-place
   patches, with the sports scoreboards and the flight map adopting it.
4. **Live games in the ticker by default:** `live_in_ticker` true, so a live
   game's cards update in the marquee instead of the full-screen scoreboard
   replacing it; existing configs are switched once (`ConfigManager`).

`display.vegas_scroll.offscreen_prefetch` (default `true`) restores the
deferred path when `false`, and `display.vegas_scroll.live_refresh` (default
`true`) turns live elements off. Keep both for one release, then delete the
old paths.

## Open questions

1. Multi-display sync: is there a two-Pi rig to test on? Replaying strip
   operations to the follower (append, trim, patch, in absolute columns) would
   fix continuous-mode sync and let live elements run under it.
2. Is the `extend` cost worth a segment list after the lazy image? The soak's
   "after work" table answers it per rig.