LEDMatrix/docs/OFFSCREEN_RENDERING.md

# Offscreen Rendering

**Status: proposed (2026-09-24).** Nothing here is implemented yet. This is the
design for review. When it lands, this file becomes the reference for how
plugin content is rendered off the render thread.

## The problem

Vegas mode builds its ticker from every plugin's content. Most of that work
already happens on a background prefetch thread
(`RenderPipeline.start_prefetch`). But any plugin whose content needs the
**shared display canvas** is deferred to the render thread
(`RenderPipeline.drain_deferred`), one plugin every two seconds. The code's
own comments put each of those at 40–600 ms, and the render thread presents no
frames while one runs.

On hdpi (Pi 4, 512×64) most plugins take that path: geochron, tide-display,
news, hockey-scoreboard, ledmatrix-stocks, incoming-packages, clock-simple,
countdown, birdnet-go, ledmatrix-music and odds-ticker. They arrive in bursts
("Whole group deferred; strip will extend as it drains") every minute or so,
12 fetches in five minutes. That is the "occasional pause" a viewer sees.

An 8-minute soak (`scripts/frame_soak.py --preview`) of the #628 build on
hdpi:

| late by | frames |
|---|---|
| 1 refresh | 238 |
| 2 | 32 |
| 3–5 | 30 |
| 6+ | 5 |
| freezes ≥ 250 ms | 2 (0.97 s total) |

The 3+ rows and the freezes are the pauses. The single-refresh row is a
separate problem: the blit is 6 ms of a 10 ms refresh, so there is little
slack. It is covered under *What this does not fix*.

## Why a plugin is canvas-bound

The plugin-facing canvas is a set of shared attributes on `DisplayManager`:
`image`, `draw`, `matrix`, and the `width`/`height` properties that read from
`matrix`. Three adapter paths (`src/vegas_mode/plugin_adapter.py`) need them,
and each returns `None` under `offscreen_only=True` so the plugin is queued for
the render thread:

1. **Display capture** (`_capture_display_content`): clear the canvas, call
   `plugin.display()`, copy `display_manager.image`. Used by any plugin
   without `get_vegas_content()` or a populated `scroll_helper`.
2. **Scroll-content generation** (`_trigger_scroll_content_generation`): a
   ticker plugin whose `scroll_helper.cached_image` is empty is made to build
   it by calling `display(force_clear=True)` or `_create_scrolling_display()`.
   Both draw on the canvas.
3. **Narrowed rendering** (`DisplayManager.render_size`): swaps the shared
   `matrix`, `image` and `draw` for a narrower set so the plugin lays out for
   `render_width_pct`. The render thread would see the swap mid-frame.

The render thread keeps the canvas coherent only because nothing else touches
it at the same time. A background thread can't use it.

## The design: a per-thread render target

`capture_mode()` is already per-thread (#423 made its state a
`threading.local`, so a background capture no longer suppresses the render
loop's pushes). The same move applies to the canvas itself:

```python
with display_manager.offscreen(width=None, height=None) as surface:
    plugin.display(force_clear=True)
    content = surface.image.copy()
```

For the **calling thread only**, inside the block:

| accessor | resolves to |
|---|---|
| `display_manager.image`, `.draw` | the surface's own image and draw: a fresh black canvas, `fontmode = "1"` |
| `display_manager.matrix` | a logical proxy reporting the surface size, so `width`/`height` and plugins that read `matrix.width` follow it. Hardware calls through it (`SetImage`, `SwapOnVSync`, `Clear`, brightness writes) are inert. |
| `update_display()`, `clear()` | canvas-only: the block implies capture mode, which is already per-thread |
| `set_scrolling_state()`, `set_frame_hold()` | no-ops, so a plugin's `display()` cannot re-pace the live scroll. Today it can, when it is captured on the render thread. |

Every other thread sees the real canvas, unchanged. The render loop in
particular keeps presenting while a plugin draws elsewhere.

### Implementation sketch

- `image`, `draw` and `matrix` become properties over `_image`, `_draw` and
  `_matrix`, plus a thread-local current surface. The getter returns the
  surface's value when the calling thread has one, else the shared one; setters
  mirror that. That costs about 0.1 µs per access, and `update_display()` reads
  each a handful of times per frame. Every existing `self.image = ...` in
  `DisplayManager` (`clear()`, setup, fallback) keeps working and becomes
  thread-correct for free.
- `render_size()` is rebuilt on `offscreen()`: it creates or narrows the
  calling thread's surface instead of swapping shared state.
- `offscreen()` nests and always restores on exit, including when the plugin
  raises.
- `VisualDisplayManager` (the plugin test harness) gets the same method, for
  parity.

### Adapter changes

- `get_content(offscreen_only=True)` stops returning `None` for the three
  paths above. Each runs inside `display_manager.offscreen(render_width)`.
- `_capture_display_content` and `_trigger_scroll_content_generation` drop
  their "copy the shared image, restore it afterwards" bookkeeping, since the
  shared image is never touched.
- **Take the plugin's lock.** `PluginManager.get_plugin_lock()` keeps
  `update()` and `display()` mutually exclusive in normal rotation, but Vegas
  never takes it, so today's render-thread captures already race
  `update()`. Off the render thread the adapter can afford to wait: blocking
  acquire with a timeout (proposed 2 s). On timeout it keeps the cached segment
  and tries again next group.
- `drain_deferred()` and the deferred queue are deleted. The only render-thread
  fetch left is the inline fallback when no prepared group is ready, which in
  practice is the first extension. Prefetching at start removes that too.

## Risks, and what was checked

1. **Plugins holding their own reference to the shared `draw` or `image`.**
   They would keep drawing into the shared canvas, and routing by thread can't
   redirect them. A grep of the 49 plugins installed on hdpi found none storing
   `display_manager.draw` or `.image` in an attribute (a pattern search, so
   indirect aliasing would slip past it). A plugin that did would
   draw into an image nobody displays, which trims to a blank segment. That is
   not corruption, and it is no worse than today.
2. **Plugins calling the matrix directly.** None in the audit. Inside
   `offscreen()` the proxy makes it inert anyway.
3. **Font thread-safety.** `FontManager` shares font objects across plugins.
   Measured on Pillow 12.3, two threads rendering text take 1.94× as long as
   one, so text rendering holds the GIL and FreeType is never entered
   concurrently. Re-check if Pillow changes that.
4. **Plugin thread-safety.** `display()` moves to the prefetch thread. The
   plugin lock makes it exclusive with `update()`, which is more protection
   than it has today. Threads a plugin starts itself are not covered, as today.
5. **The GIL.** Moving 40–600 ms of plugin rendering off the render thread
   removes the pauses, but the work still needs the GIL. Pillow drawing holds
   it, and a waiting thread only gets it back after the switch interval
   (default 5 ms). Expect some single-refresh late frames while a prefetch
   runs. Measure with the soak. Lowering `sys.setswitchinterval` during Vegas
   (e.g. 1 ms) is a one-line experiment, and a render process separate from
   plugin work is the structural answer (the "native presenter" step).

## What this does not fix

- **The blit.** Copying a 512×64 frame into the matrix (`SetImage`) is ~6 ms at
  8 PWM bits on a Pi 4, leaving ~4 ms of slack per refresh. That is the main
  source of the single-refresh late frames. Holding frames for two refreshes
  (≈50 px/s) doubles the budget. Cutting the blit itself is the native-presenter
  step.
- **Appending to the strip.** `append_content()` rebuilds the whole strip
  (8,000–20,000 px wide, 1.5–3.8 MB) on the render thread for every appended
  block. Not yet measured in isolation. Candidates: build the extended strip on
  the prefetch thread and swap one reference, or keep the strip as a list of
  chunks so an append is O(block). Measure first.
- **Live refreshes pushed from `update()`.** Some sports plugins call
  `display()` and `update_display()` from inside `update()`, which runs on the
  update worker and can push to the panel mid-Vegas. That is a separate
  hazard. `offscreen()` gives a tool for it (run the update worker offscreen
  while Vegas owns the panel), but it is out of scope here.

## Test plan

- **Unit, `DisplayManager`:** one thread inside `offscreen()` draws while
  another reads `image`/`draw`/`matrix`/`width`/`height` and sees the real
  canvas. Also: `update_display()` and `set_scrolling_state()` are inert inside;
  `render_size()` narrows only the calling thread; nesting and exceptions
  restore state.
- **Unit, adapter:** a stub display-capture plugin and a stub scroll-helper
  plugin both return content with `offscreen_only=True`, and nothing is queued
  for the render thread. The plugin lock is taken, and a timeout keeps the cached
  segment.
- **Emulator integration:** a stub canvas-bound plugin whose `display()` sleeps
  300 ms. The Vegas render loop never goes a frame without presenting (frame
  timing recorder: zero freezes).
- **Hardware:** an hdpi soak, A/B against the #628 build, alternating order.
  Targets: no freezes, an empty 6+ bucket, the 3–5 bucket near zero, and the late
  rate below 0.66%.

## Rollout

`display.vegas_scroll.offscreen_prefetch` (default `true`) restores today's
deferred path when `false`. Keep it for one release, then delete it along with
the deferred path.

## Open questions

1. Keep the kill switch, or ship without one?
2. Plugin lock timeout: skip the plugin and keep its cached segment (proposed),
   or wait longer?
3. Strip appends: in this change, or measured and handled separately (proposed)?