Files
LEDMatrix/src/vegas_mode/elements.py
T
ChuckandClaude Opus 5.5 f4bda50710 feat(vegas): live elements update in place while they scroll (#697)
* perf(timing): say which render-thread work a late frame followed

The soak already says how often a moving frame reached the panel late, but
not what the render thread was doing just before it. Vegas does two kinds of
work there between frames -- building its strip (compose, extend) and, with
live elements, patching changed pixels into it -- and deciding whether either
is affordable needs their own numbers.

- FrameTimingRecorder.note_op(kind, nbytes) tags the next presented frame.
  Totals gain op_frames, late_op_frames, op_freezes and op_bytes per kind;
  aggregate() still takes frames without ops. The file schema is unchanged.
- Vegas tags compose and every strip extension (with the bytes it copied).
- frame_soak prints an "after work" table: frames, late %, freezes and MB
  moved per kind, only when something tagged its work.
- render_bench gains --strip-screens (Vegas-sized strips), --patch-bytes /
  --patch-every / --patch-where (in-place column writes, as a live element
  update does) and --extend-every-screens / --extend-width (append + trim on
  a fixed cadence that holds the strip's width).

No runtime behaviour changes: this is the measurement gate for live Vegas
elements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): note the frame-op attribution and bench modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scroll): build the strip's PIL image only when something reads it

Every Vegas strip extension rebuilt ScrollHelper.cached_image from
cached_array in full, twice (append, then trim), on the render thread:
Image.fromarray is 1.7ms for an 8,000px strip and 3.8ms for 20,000px on a
Pi 4 (measured on ledpi), about two thirds of an extension's render-thread
cost. Nothing on the frame path reads the image's pixels; every frame is cut
from the array.

cached_image is now a property. append_content and drop_scrolled_prefix
defer it; the first read builds it from the array it started with and keeps
it only if the strip has not changed meanwhile, so a sync push racing an
extension cannot leave a stale image cached. Assigning cached_image stores
exactly what was assigned, as before. has_strip() says whether there is a
strip without building its image; the helper's frame path, Vegas and the
adapter's scroll-cache invalidation use it. The strip is also no longer held
in memory twice.

In Vegas the image is now built only by a multi-display sync push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements -- a plugin API for content that changes while it scrolls

Vegas bakes each plugin's pictures into one strip, so a card already on its
way across the panel keeps what it showed when it was drawn. This adds the
API and bookkeeping for content that can be updated in place; the worker
that redraws and swaps it follows separately. No shipped plugin implements
the hook yet, so nothing changes for users.

Plugin API (core 3.8.0), all no-ops by default:
- BasePlugin.get_vegas_elements() -> [VegasElement(key, image, version,
  live, refresh_hz)]: named, fixed-width pieces of Vegas content.
- BasePlugin.redraw_vegas_element(key, width, height, at): a lock-free
  redraw for content that changes with time.
- BasePlugin.notify_vegas_data_changed(): data that lands outside update().
- src/plugin_system/vegas_elements.py (VegasElement, re-exported from
  base_plugin).

Core:
- PluginAdapter asks a plugin that implements the hook for elements on the
  background fetch only (under its lock, on its own canvas); every other
  path keeps get_vegas_content(). Live elements are pinned (padded with
  content_padding, never trimmed), tagged with their key, digest and data
  epoch in Image.info so the existing cache and group plumbing carry them
  unchanged, and untagged if a width budget crops them.
- RenderPipeline records where each live element lands (ElementRecord), in
  absolute strip columns a trim does not move; the block-start arithmetic
  is shared with the STATIC markers.
- PluginManager update listeners (add/remove_update_listener,
  notify_data_changed): told the moment update() completes, not at the
  next ~4s Vegas poll. The coordinator uses one to move each plugin's data
  epoch on.
- vegas_scroll.live_refresh (kill switch), live_max_hz, live_min_interval,
  live_lead_screens; per-plugin core-owned vegas_live. Live elements are
  off under multi-display sync, in swap mode and with offscreen_prefetch off.
- scripts/check_plugin.py checks the element contract
  (src/plugin_system/testing/vegas.py); test/fixtures/plugins/vegas-live-stub
  is a working example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(vegas): live elements update in place while they scroll

One background worker (src/vegas_mode/live_worker.py) redraws a plugin's
live elements when its data epoch moves on (update listener) or on their
refresh_hz, nearest the screen first, and hands changed pixels lock-free to
the render thread, which copies them into the strip between frames
(RenderPipeline.apply_live_patches, ScrollHelper.patch_columns): at most
four patches or two screens of bytes a frame, no drawing or locks there.
The worker takes over group prefetch once a live element is placed, runs
inside the render gate, and is supervised. Update tick 1s while live
elements exist. Web UI switch for live_refresh. OFFSCREEN_RENDERING.md
describes what was built and why SegmentStrip was not needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 21:07:31 -04:00

174 lines
5.9 KiB
Python

"""Bookkeeping for live Vegas elements (see src/plugin_system/vegas_elements.py).
A live element travels through the same plumbing as any other Vegas content --
the adapter's cache, a prefetched group, the pipeline's join -- as a PIL image.
What makes it live rides along in the image's ``info`` dict (:data:`INFO_KEY`),
which Pillow copies through ``copy()``, ``crop()``, ``convert()`` and
``resize()``, so none of that plumbing has to change shape. The pipeline reads
the tag back when it places the image in the strip and keeps an
:class:`ElementRecord` of where it went.
Geometry is pinned: a live element is never trimmed to its ink. It is padded
with ``content_padding`` black columns each side, the margin trimming would
have left, so its width in the strip is its image width plus twice that, for
as long as its key is there. That is what lets a redraw be swapped in place.
"""
from __future__ import annotations
import itertools
import threading
import zlib
from typing import Dict, NamedTuple, Optional, Tuple
import numpy as np
from PIL import Image
#: Where a live element's :class:`ElementMeta` rides in ``Image.info``.
INFO_KEY = "ledmatrix.vegas_element"
class ElementMeta(NamedTuple):
"""What the pipeline needs to know about one live element's pixels."""
plugin_id: str
key: str
#: The plugin's data epoch (LiveEpochs) the pixels were drawn from.
epoch: int
#: pixel_digest() of the pinned pixels.
digest: Tuple[Tuple[int, ...], int]
#: time.monotonic() when drawn.
rendered_at: float
refresh_hz: float
#: The plugin's own version for the pixels, or None.
version: object = None
class ElementRecord(NamedTuple):
"""Where one live element sits in the strip.
``abs_x`` is in absolute strip columns: the strip's own column plus every
column trimmed off its front since it was composed (the pipeline's
``_strip_origin``). Trimming therefore never moves a record.
"""
seq: int
plugin_id: str
key: str
abs_x: int
width: int
epoch: int
digest: Tuple[Tuple[int, ...], int]
refresh_hz: float
class RenderedElement(NamedTuple):
"""One live element freshly redrawn by the worker, ready to compare and swap."""
key: str
#: The plugin's data epoch it was drawn from.
epoch: int
version: object
#: Pinned pixels (see pin_element), read-only.
pixels: np.ndarray
digest: Tuple[Tuple[int, ...], int]
#: Pinned width, the width it would occupy in the strip.
width: int
class LivePatch(NamedTuple):
"""A redraw handed from the worker to the render thread for one record."""
seq: int
#: The strip generation it was made against; a patch for an older strip
#: is dropped.
strip_gen: int
epoch: int
pixels: np.ndarray
digest: Tuple[Tuple[int, ...], int]
made_at: float
class LiveView(NamedTuple):
"""Where the viewport is, in absolute strip columns, published every frame."""
abs_left: int
abs_right: int
#: The end of the strip: how far ahead content exists.
abs_end: int
#: time.monotonic() when published. An old one means frames have stopped.
t_mono: float
def tag(image: Image.Image, meta: ElementMeta) -> Image.Image:
"""Mark ``image`` as the live element ``meta`` describes. Returns it."""
image.info[INFO_KEY] = meta
return image
def meta_of(image: object) -> Optional[ElementMeta]:
"""The live-element tag on ``image``, or None for plain content."""
info = getattr(image, 'info', None)
if not isinstance(info, dict):
return None
meta = info.get(INFO_KEY)
return meta if isinstance(meta, ElementMeta) else None
def untag(image: Image.Image) -> Image.Image:
"""Make ``image`` plain content again (e.g. after cropping it). Returns it."""
image.info.pop(INFO_KEY, None)
return image
def pin_element(image: Image.Image, padding: int) -> Tuple[Image.Image, np.ndarray]:
"""An element's pixels as they will sit in the strip, as image and array.
RGB, with ``padding`` black columns each side. The array is what a live
patch writes into the strip; it is read-only, so a patch in flight cannot
be changed under the render thread.
"""
if image.mode != 'RGB':
image = image.convert('RGB')
pad = max(0, int(padding))
if pad:
pinned = Image.new('RGB', (image.width + 2 * pad, image.height), (0, 0, 0))
pinned.paste(image, (pad, 0))
else:
pinned = image.copy()
array = np.ascontiguousarray(np.asarray(pinned))
array.setflags(write=False)
return pinned, array
def pixel_digest(array: np.ndarray) -> Tuple[Tuple[int, ...], int]:
"""A cheap fingerprint of an element's pixels: its shape and a CRC.
Two redraws with the same digest are treated as the same pixels and the
second is not swapped in. CRC-32 rather than Adler-32: a changed digit is
a small, local change, which is exactly where Adler-32 is weakest.
"""
data = np.ascontiguousarray(array)
return tuple(data.shape), zlib.crc32(memoryview(data).cast('B'))
class LiveEpochs:
"""A counter per plugin that moves on whenever its data may have changed.
Bumped when a plugin's update() completes (PluginManager's update
listener) or when it calls notify_vegas_data_changed(). Every live element
is tagged with the epoch it was drawn from; one drawn from an older epoch
than the plugin's current one is due a redraw. Epochs are the truth and
wake-ups only hints, so a missed wake-up delays a redraw but never loses
one.
"""
def __init__(self) -> None:
self._counter = itertools.count(1)
self._epochs: Dict[str, int] = {}
self._lock = threading.Lock()
def bump(self, plugin_id: str) -> int:
with self._lock:
epoch = next(self._counter)
self._epochs[plugin_id] = epoch
return epoch
def get(self, plugin_id: str) -> int:
return self._epochs.get(plugin_id, 0)