perf(display): skip preview work nobody reads (lazy checksum, 1 Hz snapshot writer) (#717)

Mid-scroll, update_display() no longer checksums every frame: it asks is_currently_scrolling() once per frame and reuses the answer, hashes only when dirty tracking can skip a static frame, and the preview snapshot asks its policy first and hashes only when a write or touch could follow (decide() is monotone, pinned by a property test). With the preview open the snapshot is written at most once a second (VIEWER_INTERVAL 1.0 s, was 0.2 s; the SSE stream re-read it once a second, so four encodes in five went unread); the stream now polls its mtime every 0.25 s (VIEWER_POLL_INTERVAL), so the preview stays about as fresh. The PNG is written at compress_level=1. --preview soaks are not comparable across this change.

Merged with #716: _scan_segments takes the frame's one scrolling answer and #716's static-handover pass-through, as soaked on ledpi (A B B A, 20 min each: main 0.118% / 0.113% late, with #716 and this 0.107% / 0.104%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-10-01 20:27:20 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 1ecf3aba03
commit f72d69c2b0
13 changed files with 804 additions and 47 deletions
+32
View File
@@ -146,6 +146,38 @@ policies are unchanged.
until the next minute, because the once-a-minute schedule check had
already run that minute and the session had overridden its answer.
### Web preview: less work per frame
- Mid-scroll, `update_display()` no longer checksums every frame. The
checksum (`tobytes()` plus `adler32` over the whole framebuffer: ~0.17 ms a
frame at 256x64 on a Pi 4, so roughly twice that at 512x64 and well under
0.1 ms at 128x32) fed only the dirty-tracking skip, which never applies
while scrolling, and the preview snapshot's changed-frame check. The
snapshot now asks its policy first and hashes the frame only when a write
or touch could follow. That changes no snapshot decision:
`snapshot_policy.decide()` is monotone in `frame_changed`, and a test holds
it to that. Two small differences on the panel: the first static frame
after a scroll is pushed even when it matches the scroll's last frame (one
extra swap), and the frame on which a scroll that never said it stopped
times out is presented at a hold of 1 rather than the scroll's hold.
- With the web preview open, the display writes the snapshot at most once a
second (`snapshot_policy.VIEWER_INTERVAL`, was 0.2 s). The preview already
showed at most one frame a second: its SSE stream re-read the file once a
second, so four PNG encodes in five were overwritten unread. The stream now
checks the file's mtime every 0.25 s (new `VIEWER_POLL_INTERVAL`) and sends
each frame soon after it is written, so the preview stays about as fresh;
it still touches the viewer marker once a second, and with no snapshot
file it still sends its placeholder once a second. A screen that animates
faster than once a second without marking itself as scrolling (a GIF, say)
was encoded on the render thread up to five times a second while the
preview was open, 12-14 ms each at 512x64 on a Pi 4; now at most once.
- The snapshot PNG is written at `compress_level=1`. On a desktop that
encoded a text-dense 512x64 frame in about half Pillow's default time, into
a larger file (12 KB instead of 7 KB); sparser frames gain less.
- `scripts/frame_soak.py --preview` soaks are not comparable across this
change: an open preview now costs at most one encode a second, not up to
five. Take both sides of an A/B pair on the same side of it.
### Scroller-to-static handovers
- A static plugin screen that follows a scroller no longer starts with the
+2 -2
View File
@@ -51,8 +51,8 @@ each other. They share three things:
| Fetch statistics (requests per plugin and host) | cache `fetch_stats_snapshot` | display: `FetchStatsPublisher` ([`src/common/fetch_service.py`](../src/common/fetch_service.py)), at most once a minute on change | web: `read_fetch_stats()` for `/api/v3/plugins/fetch-stats` |
| Plugin health | cache `plugin_health:<id>` | display (web writes on reset) | web: `/api/v3/plugins/health` |
| Plugin runtime (loaded, state, last error, version) | cache `plugin_runtime_snapshot` | display: `PluginRuntimePublisher` ([`src/plugin_system/plugin_runtime.py`](../src/plugin_system/plugin_runtime.py)) | web: `read_plugin_runtime()` for `/api/v3/plugins/installed`, `/plugins/state`, reconciliation |
| Preview frame | `/tmp/led_matrix_preview.png` | display: `DisplayManager`, gated by [`snapshot_policy`](../src/common/snapshot_policy.py) | web: display SSE stream, `/api/v3/health` (file age) |
| Preview viewer marker | `/tmp/led_matrix_preview_viewer` | web, while a preview is open | display: writes full-rate snapshots only while it is fresh |
| Preview frame | `/tmp/led_matrix_preview.png` | display: `DisplayManager`, gated by [`snapshot_policy`](../src/common/snapshot_policy.py): a changed frame at most once a second with a viewer, every 30 s without | web: display SSE stream (checks the mtime every 0.25 s), `/api/v3/health` (file age) |
| Preview viewer marker | `/tmp/led_matrix_preview_viewer` | web, about once a second while a preview is open | display: writes viewer-rate snapshots only while it is fresh (5 s) |
| Hardware init status | `/tmp/led_matrix_hw_status.json` | display | web: `/api/v3/hardware/status` |
| Render-loop heartbeat | `/run/ledmatrix/display-heartbeat.json` (tmpfs) | display: the render thread, via [`display_watchdog`](../src/display_watchdog.py) | web: `/api/v3/health` (`checks.display_loop`); the update health check |
+6 -2
View File
@@ -335,8 +335,12 @@ python3 scripts/frame_soak.py --json a.json # keep the report to compare later
It runs as any user next to the display service and stops nothing. It needs
something to *scroll* during the run: a live game holding a static scoreboard
on screen gives no verdict. `--preview` keeps the web preview's viewer marker
fresh, which puts the preview's PNG encoding at full rate -- run it as the web
service's user.
fresh, which puts the preview's PNG encoding at the viewer rate, as an open
preview does -- run it as the web service's user. That rate is at most one
frame a second. Through 3.8.0 it was up to five, so a `--preview` soak taken
before that change is not comparable with one taken after it (the hdpi
results below are from before it): take both sides of an A/B pair on
the same side of it.
| line | what it tells you |
|---|---|
+6 -2
View File
@@ -9,7 +9,10 @@ reports the difference. Nothing is stopped, restarted or drawn.
python3 scripts/frame_soak.py
# the same with the web preview open (the preview's PNG encodes are one of
# the things that used to make the render loop miss refreshes)
# the things that used to make the render loop miss refreshes). An open
# preview is encoded at most once a second; through 3.8.0 it was up to
# five times, so a --preview run from before that change is not comparable
# with one from after it
python3 scripts/frame_soak.py --preview
# quick look at the totals since the service started
@@ -65,7 +68,8 @@ from src.common.frame_timing import ( # noqa: E402
)
#: Touched by the web UI while someone has the preview open; a fresh marker
#: puts the display service's snapshot writer at full rate. Same path as
#: puts the display service's snapshot writer at the viewer rate
#: (snapshot_policy.VIEWER_INTERVAL). Same path as
#: DisplayManager._viewer_marker_path.
VIEWER_MARKER = "/tmp/led_matrix_preview_viewer" # nosec B108 - fixed path shared with the service
+2 -1
View File
@@ -231,7 +231,8 @@ rather than the `set_*` methods. Vegas mode reads a plugin's
[`snapshot_policy.py`](snapshot_policy.py). Core-internal. `decide()`
tells `DisplayManager` whether to write `/tmp/led_matrix_preview.png`, only
touch its mtime, or skip, based on whether a browser is watching the preview.
The web health check reads the file's age.
The web health check reads the file's age, and the web preview stream checks
its mtime every `VIEWER_POLL_INTERVAL`.
### sports_card
+6 -3
View File
@@ -19,8 +19,9 @@ Only intervals between two consecutive *scrolling* frames count: a static
screen that changes once a second has no timing to get wrong, and the first
frame of a scroll has no predecessor worth measuring against.
"Scrolling" is DisplayManager's scroll state when the frame is presented, and
that state can go missing in the middle of a scroll. It expires after 2s
"Scrolling" is the scroll state ``DisplayManager.update_display`` acted on for
the frame, sampled once before the blit and swap, and that state can go missing
in the middle of a scroll. It expires after 2s
without scroll activity, which a long enough stall outlasts, and any thread can
clear it: plugins call ``set_scrolling_state(False)`` from their own
``display()``, and Vegas captures some of those on the render thread between
@@ -393,7 +394,9 @@ class FrameTimingRecorder:
:param blit: seconds spent copying the frame into the canvas.
:param wait: seconds SwapOnVSync blocked.
:param hold: the refreshes this frame was held for.
:param scrolling: whether a scroll was running when it was presented.
:param scrolling: whether a scroll was running for this frame: the
scroll state ``update_display`` acted on, sampled once before the
blit and swap.
:param presented_at: ``time.perf_counter()`` when the swap returned.
"""
previous = self._previous
+21 -2
View File
@@ -26,14 +26,33 @@ Policy:
- Unchanged frames are never re-encoded; the mtime is touched every
TOUCH_INTERVAL so the health check (60s threshold) never degrades.
The writer and the SSE reader (web_interface/app.py) have two periods, not
one shared value. The reader sends each write it sees, so the preview shows
at most one frame per VIEWER_INTERVAL. (That was 0.2 s while the reader
slept 1 s between reads, so four encodes in five were overwritten unread.)
The reader checks the file's mtime every VIEWER_POLL_INTERVAL, which is only
a stat, and so sends each write within that long of it landing. Equal
periods would alias: two unsynchronised 1 s clocks leave the preview up to a
second stale, and now and then 2 s between frames.
decide() is monotone in frame_changed: SKIP for a changed frame means SKIP
for an unchanged one. DisplayManager relies on that to skip hashing the
frame when even a changed one would be skipped; test_snapshot_policy.py
checks it.
If any constant here changes, re-check the health threshold in
api_v3/misc.py (get_hardware_status) — TOUCH_INTERVAL must stay well under it.
"""
from enum import Enum
# Snapshot cadence with a browser preview open (seconds).
VIEWER_INTERVAL = 0.2
# Snapshot cadence with a browser preview open (seconds): the shortest gap
# between two preview frames.
VIEWER_INTERVAL = 1.0
# How often the web SSE reader checks the snapshot's mtime (seconds). Must
# stay well under VIEWER_INTERVAL -- half of it at most -- or the two clocks
# alias (see above).
VIEWER_POLL_INTERVAL = 0.25
# Snapshot cadence with no viewers — cheap freshness for page-open (seconds).
IDLE_INTERVAL = 30.0
# Max age of the last write/touch before bumping mtime for the health
+81 -23
View File
@@ -80,6 +80,15 @@ _CALENDAR_FONT_PX = 7
#: frame, so a fault that persists would otherwise log ~100 lines a second.
_UPDATE_ERROR_LOG_INTERVAL = 60.0
#: zlib level for the preview snapshot PNG. The fastest level: each file is
#: read by the web UI and soon replaced by the next, so encode time (paid on
#: the render thread for a static screen) matters more than its size.
#: Lossless at any level. Against Pillow's default (6), on a desktop with
#: Pillow 12.3, a text-dense 512x64 frame encoded in about half the time,
#: into 12 KB instead of 7 KB; sparser frames saved less time (10-20%) and
#: stayed under 2 KB.
_SNAPSHOT_PNG_COMPRESS_LEVEL = 1
def _bdf_native_size(face) -> int:
"""The pixel height a BDF Face declares, or 0 if it does not say.
@@ -296,8 +305,8 @@ class DisplayManager:
self._TEXT_WIDTH_CACHE_MAX = 1024
# Snapshot mirror for web preview + health check (service writes, web
# reads). Cadence/skip decisions live in src/common/snapshot_policy.py:
# full rate only while the web SSE broadcaster keeps the viewer marker
# fresh; unchanged frames are never re-encoded, only mtime-touched.
# the viewer rate only while the web SSE broadcaster keeps the viewer
# marker fresh; unchanged frames are never re-encoded, only mtime-touched.
self._snapshot_path = "/tmp/led_matrix_preview.png" # nosec B108 - fixed path intentional; web UI reads same path
self._viewer_marker_path = "/tmp/led_matrix_preview_viewer" # nosec B108 - touched by web SSE broadcaster
self._last_snapshot_ts = 0.0
@@ -946,16 +955,32 @@ class DisplayManager:
self._write_snapshot_if_due()
return
# Asked once per frame and the answer reused below: the call
# has side effects (it expires a stale scroll and drops its
# frame hold), so asking again further down could disagree
# with what this frame was already treated as. Asked first,
# so the dirty check, the scan-order segments, the pacing gate,
# the swaps and frame timing all see one answer and the frame
# hold it leaves.
scrolling = self.is_currently_scrolling()
digest = None
frame_checksum = None
if self._dirty_tracking_enabled:
# No digest mid-scroll. The skip it feeds is never taken while
# scrolling (see below), so all it bought there was the
# snapshot's changed-frame check -- a tobytes() plus adler32
# over the whole framebuffer every frame (~0.17ms at 256x64
# on a Pi 4, twice that at 512x64) for a decision acted on at
# most once a second. _write_snapshot_if_due hashes for
# itself when a write or touch is actually due. The cost: the
# first static frame after a scroll is always pushed, once.
if self._dirty_tracking_enabled and not scrolling:
try:
brightness = getattr(self.matrix, 'brightness', None)
except AttributeError:
brightness = None
frame_checksum = zlib.adler32(self.image.tobytes())
digest = (frame_checksum, brightness)
if digest == self._last_pushed_digest and not self.is_currently_scrolling():
if digest == self._last_pushed_digest:
# Nothing changed since the last push — the panel is
# already showing exactly this frame.
#
@@ -982,7 +1007,7 @@ class DisplayManager:
if self._double_sided is not None:
segments = [(self._composite_double_sided(), self._frame_hold)]
else:
segments = self._scan_segments(self.image)
segments = self._scan_segments(self.image, scrolling)
gate = self.render_gate
blit_time = swap_time = 0.0
# Usually one segment: the frame, held for _frame_hold
@@ -1008,7 +1033,7 @@ class DisplayManager:
self._last_blit_seconds = blit_time / len(segments)
self.frame_timing.record(
blit_time, swap_time,
self._frame_hold, self.is_currently_scrolling(), presented_at)
self._frame_hold, scrolling, presented_at)
self._last_pushed_digest = digest
@@ -1055,7 +1080,9 @@ class DisplayManager:
", ".join(f"rows {top}-{bottom - 1} show {lag} refresh(es) behind"
for top, bottom, lag in bands))
def _scan_segments(self, image: Image.Image) -> List[Tuple[Image.Image, int]]:
def _scan_segments(self, image: Image.Image,
scrolling: Optional[bool] = None
) -> List[Tuple[Image.Image, int]]:
"""What to present for this frame: ``[(image, refreshes), ...]``.
Mid-scroll with compensation on, lagging rows are taken from earlier
@@ -1068,10 +1095,17 @@ class DisplayManager:
after a scroll, while the scroll state is still set (see
end_scroll_for_static_screen): one segment, held for the scroll's
hold, with no rows from the scroller's frames.
``scrolling`` is the caller's is_currently_scrolling() answer for this
frame. update_display() asks once, before calling this, so the frame
hold read here is the one that answer left (an expired scroll's hold
is already dropped). None asks here.
"""
hold = self._frame_hold
bands = getattr(self, '_scan_lag_bands', None)
if (not bands or not self.is_currently_scrolling()
if (not bands
or not (scrolling if scrolling is not None
else self.is_currently_scrolling())
or self._static_handover):
if bands:
self._scan_history.clear()
@@ -1563,9 +1597,10 @@ class DisplayManager:
frame -- the bottom half of the old ticker under the new screen on a
96x48 panel. At hold 1 that frame stays up for a whole second; at a
longer hold its first refresh flashes the old rows. For a second frame
in the same call, the rows would come from the first. Dirty tracking
compares frames as drawn, so once the scroll is over it skips every
identical 1 Hz redraw of such a frame, and nothing would replace it.
in the same call, the rows would come from the first, shown for as long
as the first's would be. Dirty tracking does not keep such a frame up
past the screen's next redraw: a frame pushed while the scroll state
is set leaves it no digest to match, so that redraw is pushed.
The rest of that scroll is left on purpose, until the controller ends
it:
@@ -1729,11 +1764,14 @@ class DisplayManager:
Args:
frame_checksum: adler32 of the current frame, when the caller has
already computed one. Dirty tracking checksums every frame a
few lines above the call site, and re-deriving it here meant a
second tobytes() plus a second pass over the whole framebuffer
on every single frame — ~0.17ms per frame of the two combined
at 256x64, paid 100 times a second to reach the same number.
already computed one. Dirty tracking checksums every static
frame a few lines above the call site, and re-deriving it here
meant a second tobytes() plus a second pass over the whole
framebuffer on every single frame — ~0.17ms per frame of the
two combined at 256x64, paid 100 times a second to reach the
same number. None (mid-scroll, dirty tracking off, no
hardware): the frame is hashed here, and only when the policy
could act on it.
"""
try:
now = time.time()
@@ -1744,11 +1782,29 @@ class DisplayManager:
self._last_snapshot_ts = 0.0
self._viewer_was_fresh = viewer_fresh
digest = (frame_checksum if frame_checksum is not None
else zlib.adler32(self.image.tobytes()))
action = snapshot_policy.decide(
now, self._last_snapshot_ts, self._last_snapshot_touch_ts,
viewer_fresh, digest != self._last_snapshot_digest)
if frame_checksum is not None:
digest = frame_checksum
action = snapshot_policy.decide(
now, self._last_snapshot_ts, self._last_snapshot_touch_ts,
viewer_fresh, digest != self._last_snapshot_digest)
else:
# Ask as if the frame had changed before paying to find out.
# decide() is monotone in frame_changed -- a SKIP for a
# changed frame is a SKIP for an unchanged one too (its touch
# branch ignores frame_changed) -- so returning here gives the
# same answer the hash would have, and on most frames the hash
# is never taken. test_snapshot_policy.py holds decide() to it.
action = snapshot_policy.decide(
now, self._last_snapshot_ts, self._last_snapshot_touch_ts,
viewer_fresh, True)
if action is snapshot_policy.SnapshotAction.SKIP:
return
digest = zlib.adler32(self.image.tobytes())
if digest == self._last_snapshot_digest:
# Unchanged after all: the decision an unchanged frame gets.
action = snapshot_policy.decide(
now, self._last_snapshot_ts,
self._last_snapshot_touch_ts, viewer_fresh, False)
if action is snapshot_policy.SnapshotAction.SKIP:
return
if (action is snapshot_policy.SnapshotAction.TOUCH
@@ -1826,7 +1882,8 @@ class DisplayManager:
prefix=f".{snapshot_path_obj.name}.", suffix=".tmp")
try:
with os.fdopen(_fd, "wb") as _f:
image.save(_f, format='PNG')
image.save(_f, format='PNG',
compress_level=_SNAPSHOT_PNG_COMPRESS_LEVEL)
os.chmod(tmp_path, 0o644)
os.replace(tmp_path, self._snapshot_path)
except Exception:
@@ -1837,7 +1894,8 @@ class DisplayManager:
except OSError:
pass
# Fallback to direct save if replace not supported
image.save(self._snapshot_path, format='PNG')
image.save(self._snapshot_path, format='PNG',
compress_level=_SNAPSHOT_PNG_COMPRESS_LEVEL)
# Set proper file permissions after saving
try:
ensure_file_permissions(snapshot_path_obj, get_assets_file_mode())
+392
View File
@@ -68,6 +68,7 @@ class _SwapSpy:
self.matrix = matrix
self.count = 0
self.last_frame_hold = None
self.holds = []
self._orig = matrix.SwapOnVSync
def __enter__(self):
@@ -77,6 +78,7 @@ class _SwapSpy:
# the real binding or the spy hides a TypeError as a failed push.
self.count += 1
self.last_frame_hold = args[0] if args else 1
self.holds.append(self.last_frame_hold)
return self._orig(canvas, *args)
self.matrix.SwapOnVSync = counting
return self
@@ -501,3 +503,393 @@ class TestSnapshotOffRenderThread:
dm.set_scrolling_state(False)
dm.update_display()
assert threads == [threading.current_thread().name]
class TestLazyDigest:
"""Mid-scroll the frame is hashed only when the preview snapshot is due.
The digest feeds two things: the dirty-tracking skip, which is never
taken while scrolling, and the snapshot's changed-frame check, which acts
at most once a second. Hashing every scrolled frame for them cost a
tobytes() plus adler32 over the whole framebuffer on every frame (~0.3ms
at 512x64 on a Pi 4). update_display() now leaves the digest out while
scrolling, and _write_snapshot_if_due() hashes only when the policy could
act on the answer.
"""
@pytest.fixture
def hashes(self, dm, tmp_path, monkeypatch):
"""Every adler32 display_manager takes, while no viewer is watching."""
import types
import zlib
import src.display_manager as display_manager_module
calls = []
def counting(data, *args):
calls.append(len(data))
return zlib.adler32(data, *args)
monkeypatch.setattr(display_manager_module, "zlib",
types.SimpleNamespace(adler32=counting))
# The fixture's marker is the real /tmp one, which a preview open on
# this machine would keep fresh. Point it somewhere nobody touches.
monkeypatch.setattr(dm, "_viewer_marker_path", str(tmp_path / "no-viewer"))
monkeypatch.setattr(dm, "_snapshot_path", str(tmp_path / "snap.png"))
dm._viewer_check_ts = 0.0
dm._viewer_fresh = False
dm._viewer_was_fresh = False
# TestSnapshotOffRenderThread's cleanup test stops this shared
# manager's snapshot writer for good; queued writes here need one.
with dm._snapshot_cond:
dm._snapshot_stop = False
dm._snapshot_pending = None
yield calls
dm.set_scrolling_state(False)
dm._viewer_check_ts = 0.0
dm._viewer_fresh = False
dm._viewer_was_fresh = False
def _scroll(self, dm, frames):
"""Present ``frames`` scrolled frames, each different from the last."""
for shade in range(frames):
dm.set_scrolling_state(True) # as a scroller does every frame
dm.draw.rectangle([0, 0, 10, 10], fill=(shade * 10 % 256, 40, 0))
dm.update_display()
def test_scrolled_frames_are_not_hashed_when_no_snapshot_is_due(
self, dm, hashes):
now = time.time()
dm._last_snapshot_ts = now # written a moment ago
dm._last_snapshot_touch_ts = now
with _SwapSpy(dm.matrix) as spy:
self._scroll(dm, 20)
assert spy.count == 20, "every scrolled frame must still be pushed"
assert hashes == []
def test_a_due_snapshot_hashes_the_frame_once(
self, dm, hashes, monkeypatch):
import threading
wrote = threading.Event()
monkeypatch.setattr(dm, "_save_snapshot", lambda image: wrote.set())
dm._last_snapshot_ts = 0.0 # a write is due (no viewer:
dm._last_snapshot_touch_ts = 0.0 # the idle keepalive)
dm._last_snapshot_digest = None
self._scroll(dm, 20)
assert wrote.wait(5), "the due snapshot was never written"
assert len(hashes) == 1
def test_with_a_viewer_a_scroll_is_hashed_once_per_write(
self, dm, hashes, tmp_path, monkeypatch):
import threading
from src.common import snapshot_policy
marker = tmp_path / "viewer"
marker.touch()
monkeypatch.setattr(dm, "_viewer_marker_path", str(marker))
writes = []
wrote = threading.Event()
def save(image):
writes.append(image)
wrote.set()
monkeypatch.setattr(dm, "_save_snapshot", save)
dm._last_snapshot_ts = time.time() # the viewer opening resets it
dm._last_snapshot_touch_ts = time.time()
dm._last_snapshot_digest = None # every frame here is new
started = time.time()
self._scroll(dm, 30)
elapsed = time.time() - started
assert wrote.wait(5)
# One write as the viewer appears, then at most one per interval.
assert 1 <= len(hashes) <= 1 + int(elapsed / snapshot_policy.VIEWER_INTERVAL)
assert len(hashes) < 30
def test_an_unchanged_scrolled_frame_is_not_re_encoded(
self, dm, hashes, monkeypatch):
# A write would be due for a changed frame, but this one is already
# on disk: the lazy hash must re-ask the policy for an unchanged frame
# (SKIP here) rather than act on the changed-frame answer (WRITE).
import zlib
saved, touched = [], []
monkeypatch.setattr(dm, "_save_snapshot", lambda image: saved.append(image))
monkeypatch.setattr(os, "utime", lambda *a, **k: touched.append(a))
dm.set_scrolling_state(True)
dm.draw.rectangle([0, 0, 10, 10], fill=(3, 3, 3))
digest = zlib.adler32(dm.image.tobytes())
dm._last_snapshot_digest = digest
dm._saved_snapshot_digest = digest
last_write = time.time() - 1000 # a changed frame would write
dm._last_snapshot_ts = last_write
dm._last_snapshot_touch_ts = time.time() # no touch due
dm.update_display()
# A write, even one only queued, would have moved the bookkeeping.
assert dm._last_snapshot_ts == last_write
assert dm._snapshot_pending is None
assert saved == [] and touched == []
assert len(hashes) == 1
def test_an_unchanged_scrolled_frame_is_touched_only_once_on_disk(
self, dm, hashes, monkeypatch):
# A touch vouches for the file on disk, so through the lazy hash too it
# must only happen once the frame has actually been saved.
import threading
import zlib
from src.common import snapshot_policy
saved, touched = [], []
wrote = threading.Event()
def save(image):
saved.append(image)
wrote.set()
monkeypatch.setattr(dm, "_save_snapshot", save)
monkeypatch.setattr(os, "utime", lambda *a, **k: touched.append(a))
dm.set_scrolling_state(True)
dm.draw.rectangle([0, 0, 10, 10], fill=(5, 5, 5))
digest = zlib.adler32(dm.image.tobytes())
dm._last_snapshot_digest = digest # queued earlier...
dm._saved_snapshot_digest = 12345 # ...but an older frame is on disk
# A touch is due; a write is not (no viewer: the idle interval).
stale = time.time() - snapshot_policy.TOUCH_INTERVAL - 1.0
dm._last_snapshot_ts = stale
dm._last_snapshot_touch_ts = stale
dm.update_display()
assert wrote.wait(5), "a frame not yet on disk must be written"
deadline = time.time() + 5
while dm._saved_snapshot_digest != digest and time.time() < deadline:
time.sleep(0.01)
assert touched == [] and len(saved) == 1
# Once it is on disk, the same frame is only touched.
dm._last_snapshot_ts = stale
dm._last_snapshot_touch_ts = stale
dm.set_scrolling_state(True)
dm.update_display()
assert len(touched) == 1 and len(saved) == 1
def test_the_first_static_frame_after_a_scroll_is_pushed_once(
self, dm, hashes):
# No digest is kept mid-scroll, so a static frame identical to the
# scroll's last one is pushed once (one swap), and then dirty
# tracking skips its repeats as before.
dm.set_scrolling_state(True)
dm.draw.rectangle([0, 0, 18, 18], fill=(90, 90, 0))
dm.update_display() # the scroll's last frame
dm.set_scrolling_state(False)
with _SwapSpy(dm.matrix) as spy:
dm.update_display()
dm.update_display()
dm.update_display()
assert spy.count == 1
def test_a_scrolled_push_forgets_the_last_static_frame(
self, dm, hashes):
# Scrolled frames carry no digest, so a scrolled push must clear the
# last static one. Were it kept, a static screen drawing what it
# showed before the scroll would be skipped as already on the panel,
# and the scroll's last frame would stay up until the content changed.
dm.set_scrolling_state(False)
dm.draw.rectangle([0, 0, 18, 18], fill=(10, 120, 30))
dm.update_display() # static frame A
dm.set_scrolling_state(True)
dm.draw.rectangle([0, 0, 18, 18], fill=(200, 0, 60))
dm.update_display() # a scrolled frame B
assert dm._last_pushed_digest is None
dm.set_scrolling_state(False)
dm.draw.rectangle([0, 0, 18, 18], fill=(10, 120, 30))
with _SwapSpy(dm.matrix) as spy:
dm.update_display() # A again, over B
assert spy.count == 1, "A must replace the scroll's last frame"
class TestOneScrollingAnswerPerFrame:
"""update_display() asks is_currently_scrolling() once per frame and uses
that answer throughout. The call has side effects -- a scroll that has
gone quiet past its threshold is expired there and its frame hold
dropped -- so asking again partway through let one frame be presented at
the scroll's hold and recorded as static."""
def test_a_scroll_that_times_out_is_presented_without_its_hold(
self, dm, monkeypatch):
recorded = []
monkeypatch.setattr(
dm.frame_timing, "record",
lambda blit, wait, hold, scrolling, at: recorded.append(
(hold, scrolling)))
dm.set_scrolling_state(True, frame_hold=3)
# Quiet past the inactivity threshold, and nothing has asked since.
dm._scrolling_state['last_scroll_activity'] -= (
dm._scrolling_state['scroll_inactivity_threshold'] + 1.0)
dm.draw.rectangle([0, 0, 8, 8], fill=(31, 62, 93))
try:
with _SwapSpy(dm.matrix) as spy:
dm.update_display()
finally:
dm.set_scrolling_state(False)
assert spy.count == 1
assert spy.last_frame_hold == 1
assert recorded == [(1, False)]
def test_scan_compensation_uses_the_frames_answer(
self, dm, tmp_path, monkeypatch):
from collections import deque
monkeypatch.setattr(dm, "_scan_lag_bands", [(16, 32, 1)])
monkeypatch.setattr(dm, "_scan_history", deque(maxlen=1))
# No preview and no snapshot due, so only the frame itself asks.
monkeypatch.setattr(dm, "_viewer_marker_path", str(tmp_path / "no-viewer"))
dm._viewer_check_ts = 0.0
dm._last_snapshot_ts = dm._last_snapshot_touch_ts = time.time()
asked = []
real = dm.is_currently_scrolling
def counting():
asked.append(1)
return real()
monkeypatch.setattr(dm, "is_currently_scrolling", counting)
dm.set_scrolling_state(True)
try:
dm.draw.rectangle([0, 0, 8, 8], fill=(17, 34, 51))
dm.update_display()
finally:
dm.set_scrolling_state(False)
dm._viewer_check_ts = 0.0
assert len(asked) == 1
@staticmethod
def _scan_banded(dm, tmp_path, monkeypatch):
"""Scan-order compensation on, no preview, no snapshot due, and every
frame handed to the canvases kept (update_display alternates them)."""
from collections import deque
monkeypatch.setattr(dm, "_scan_lag_bands", [(16, 32, 1)])
monkeypatch.setattr(dm, "_scan_history", deque(maxlen=1))
monkeypatch.setattr(dm, "_viewer_marker_path", str(tmp_path / "no-viewer"))
dm._viewer_check_ts = 0.0
dm._last_snapshot_ts = dm._last_snapshot_touch_ts = time.time()
shown = []
for canvas in (dm.offscreen_canvas, dm.current_canvas):
def capture(image, *args, _real=canvas.SetImage, **kwargs):
shown.append(image.copy())
return _real(image, *args, **kwargs)
monkeypatch.setattr(canvas, "SetImage", capture)
return shown
def test_a_held_frame_is_split_on_the_frames_answer(
self, dm, tmp_path, monkeypatch):
# Scan-order compensation presents a held frame as two swaps, the
# lagging half stepping one refresh after the rest. The split is
# decided on the frame's one answer, exactly as when it asked itself.
shown = self._scan_banded(dm, tmp_path, monkeypatch)
asked, recorded = [], []
real = dm.is_currently_scrolling
def counting():
asked.append(1)
return real()
monkeypatch.setattr(dm, "is_currently_scrolling", counting)
monkeypatch.setattr(
dm.frame_timing, "record",
lambda blit, wait, hold, scrolling, at: recorded.append(
(hold, scrolling)))
dm.set_scrolling_state(True, frame_hold=2)
try:
# The fixture is shared: a slow blit left by an earlier test
# would send this frame out whole and drop the history.
dm._last_blit_seconds = 0.0
dm.draw.rectangle([0, 0, dm.width - 1, dm.height - 1],
fill=(10, 0, 0))
dm.update_display() # the previous frame
asked.clear()
recorded.clear()
before = len(shown)
dm._last_blit_seconds = 0.0 # a blit that fits a refresh
dm.set_scrolling_state(True, frame_hold=2)
dm.draw.rectangle([0, 0, dm.width - 1, dm.height - 1],
fill=(20, 0, 0))
with _SwapSpy(dm.matrix) as spy:
dm.update_display()
finally:
dm.set_scrolling_state(False)
dm._viewer_check_ts = 0.0
assert len(asked) == 1
assert spy.holds == [1, 1]
first, second = shown[before:]
assert first.getpixel((0, 0)) == (20, 0, 0)
assert first.getpixel((0, 31)) == (10, 0, 0) # lagging half: old
assert second.getpixel((0, 31)) == (20, 0, 0) # a refresh later
assert recorded == [(2, True)]
def test_a_held_scroll_that_times_out_goes_out_whole_at_hold_1(
self, dm, tmp_path, monkeypatch):
# The expiry frame with scan-order compensation on: one swap of the
# frame as drawn, at hold 1, the history dropped -- and the pacing
# gate, the swap and frame timing all see that one hold. (Asking
# partway through instead had the swap use the dead scroll's hold
# while the gate and frame timing were told 1.)
shown = self._scan_banded(dm, tmp_path, monkeypatch)
gate_holds, recorded = [], []
class _Gate:
def before_swap(self, hold):
gate_holds.append(('before', hold))
def after_swap(self, hold):
gate_holds.append(('after', hold))
monkeypatch.setattr(
dm.frame_timing, "record",
lambda blit, wait, hold, scrolling, at: recorded.append(
(hold, scrolling)))
dm.set_scrolling_state(True, frame_hold=3)
try:
dm.draw.rectangle([0, 0, dm.width - 1, dm.height - 1],
fill=(30, 0, 0))
dm.update_display() # the scroll's last frame
recorded.clear()
before = len(shown)
# Quiet past the inactivity threshold, and nothing has asked since.
dm._scrolling_state['last_scroll_activity'] -= (
dm._scrolling_state['scroll_inactivity_threshold'] + 1.0)
dm._last_blit_seconds = 0.0 # a split would fit
monkeypatch.setattr(dm, "render_gate", _Gate())
dm.draw.rectangle([0, 0, dm.width - 1, dm.height - 1],
fill=(40, 0, 0))
with _SwapSpy(dm.matrix) as spy:
dm.update_display()
finally:
dm.set_scrolling_state(False)
dm._viewer_check_ts = 0.0
assert spy.holds == [1]
assert gate_holds == [('before', 1), ('after', 1)]
assert recorded == [(1, False)]
(frame,) = shown[before:]
assert frame.getpixel((0, 31)) == (40, 0, 0) # not composed
assert len(dm._scan_history) == 0
class TestSnapshotEncoding:
def test_the_snapshot_is_written_at_the_fastest_level_and_lossless(
self, dm, tmp_path, monkeypatch):
# The encode is paid on the render thread for a static screen, and
# the file is read once; level 1 cuts the encode, and PNG is lossless
# at any level, so the preview shows exactly the frame.
from PIL import Image, ImageChops
levels = []
real_save = Image.Image.save
def recording(image, fp, format=None, **params):
levels.append(params.get("compress_level"))
return real_save(image, fp, format, **params)
monkeypatch.setattr(Image.Image, "save", recording)
monkeypatch.setattr(dm, "_snapshot_path", str(tmp_path / "snap.png"))
frame = Image.new("RGB", (dm.width, dm.height))
frame.paste((200, 10, 90), (3, 4, 40, 20))
frame.putpixel((0, 0), (1, 2, 3))
dm._save_snapshot(frame)
assert levels == [1]
with Image.open(dm._snapshot_path) as saved:
assert saved.format == "PNG"
assert ImageChops.difference(
saved.convert("RGB"), frame).getbbox() is None
+57 -6
View File
@@ -151,12 +151,58 @@ class TestTheFirstStaticFrame:
[(_blit, _wait, hold, scrolling, _at)] = records
assert (hold, scrolling) == (3, True)
def test_it_is_decided_on_the_frames_one_answer_and_not_hashed(
self, dm, monkeypatch, tmp_path):
# The handover frame goes through update_display like any frame while
# the scroll state is set: it asks is_currently_scrolling() once, takes
# no digest, and that answer and its hold reach the swap and the
# record -- while the frame itself goes out as drawn.
import time
import zlib
import src.display_manager as display_manager_module
dm._scan_lag_bands = [(24, 48, 1)]
# No preview and no snapshot due: only the frame itself asks or hashes.
dm._viewer_marker_path = str(tmp_path / "no-viewer")
dm._viewer_check_ts = 0.0
dm.set_scrolling_state(True, 2)
_push(dm, (255, 0, 0)) # the ticker's last frame
dm._last_snapshot_ts = dm._last_snapshot_touch_ts = time.time()
asked, hashed, records = [], [], []
real_asked = dm.is_currently_scrolling
def asking():
asked.append(1)
return real_asked()
def hashing(data, *args):
hashed.append(len(data))
return zlib.adler32(data, *args)
real_record = dm.frame_timing.record
def record(*args, **kwargs):
records.append(args[2:4])
return real_record(*args, **kwargs)
monkeypatch.setattr(dm, "is_currently_scrolling", asking)
monkeypatch.setattr(display_manager_module, "zlib",
types.SimpleNamespace(adler32=hashing))
monkeypatch.setattr(dm.frame_timing, "record", record)
holds = _watch_swaps(dm)
before = len(dm._presented)
dm._last_blit_seconds = 0.0 # fast enough to split
dm.end_scroll_for_static_screen()
_push(dm, (0, 0, 0))
assert len(asked) == 1
assert hashed == []
assert holds == [2]
assert records == [(2, True)]
assert len(dm._presented) - before == 1
assert dm._presented[-1].getpixel((10, 30)) == (0, 0, 0)
def test_nor_its_own_first_frame_under_its_second(self, dm):
# A first display() that pushes two frames: a clear, then the screen.
# Composed, the second would show the first's rows -- and once the
# scroll is over, dirty tracking (which compares frames as drawn, not
# as composed) skips every identical 1 Hz redraw, so that half-black
# frame would stay up for the whole turn.
# Composed, the second would show the first's rows: a half-black frame
# up for the whole second until the 1 Hz loop's next redraw.
dm._scan_lag_bands = [(24, 48, 1)]
dm.set_scrolling_state(True, 1)
_push(dm, (255, 0, 0)) # the ticker's last frame
@@ -166,8 +212,13 @@ class TestTheFirstStaticFrame:
assert shown.getpixel((10, 30)) == (0, 0, 255)
dm.set_scrolling_state(False) # the controller, after the dispatch
pushed = len(dm._presented)
_push(dm, (0, 0, 255)) # the 1 Hz redraw: skipped
assert len(dm._presented) == pushed
# The 1 Hz redraw is pushed once, as drawn: a frame pushed while the
# scroll state was set leaves dirty tracking no digest to match. The
# next identical one is skipped.
assert _push(dm, (0, 0, 255)).getpixel((10, 30)) == (0, 0, 255)
assert len(dm._presented) == pushed + 1
_push(dm, (0, 0, 255))
assert len(dm._presented) == pushed + 1
def test_the_next_scroll_is_compensated_again(self, dm):
dm._scan_lag_bands = [(24, 48, 1)]
+45 -1
View File
@@ -5,7 +5,9 @@ The invariants that matter:
frames at 5 fps, 24/7)
- the file mtime never goes stale enough to trip the health check's 60s
degraded threshold (api_v3 get_hardware_status)
- a viewer gets full cadence; no viewer drops to the idle keepalive
- a viewer gets the viewer cadence; no viewer drops to the idle keepalive
- decide() is monotone in frame_changed, which DisplayManager's lazy
frame hash relies on
"""
import os
@@ -19,6 +21,8 @@ from src.common.snapshot_policy import ( # noqa: E402
IDLE_INTERVAL,
TOUCH_INTERVAL,
VIEWER_INTERVAL,
VIEWER_MARKER_FRESH_SEC,
VIEWER_POLL_INTERVAL,
SnapshotAction,
decide,
)
@@ -89,5 +93,45 @@ class TestHealthKeepalive:
assert action in (SnapshotAction.WRITE, SnapshotAction.TOUCH)
class TestMonotoneInFrameChanged:
"""DisplayManager asks decide() as if the frame had changed, and only
hashes the frame when that answer is not SKIP. That is exact only while
a SKIP for a changed frame is also a SKIP for an unchanged one."""
def test_skip_for_a_changed_frame_is_skip_for_an_unchanged_one(self):
# Ages that straddle every threshold decide() compares against, plus
# a timestamp in the future (a wall-clock step back) and "never".
eps = 1e-6
ages = sorted({0.0, -1.0, 1000.0, 100.0} | {
base + delta
for base in (VIEWER_INTERVAL, IDLE_INTERVAL, TOUCH_INTERVAL)
for delta in (-eps, 0.0, eps)
})
now = 1000.0
checked = 0
for write_age in ages:
for touch_age in ages:
for viewer in (True, False):
args = (now, now - write_age, now - touch_age, viewer)
if decide(*args, frame_changed=True) is SnapshotAction.SKIP:
checked += 1
assert decide(*args, frame_changed=False) is SnapshotAction.SKIP, args
assert checked, "the grid never reached a SKIP; it tests nothing"
class TestPreviewPolling:
def test_the_reader_polls_faster_than_the_writer_writes(self):
"""Equal periods alias: two unsynchronised clocks of the same period
leave the preview up to a period stale, with the odd double gap.
Polling at least twice per interval keeps each frame at most half an
interval late."""
assert VIEWER_POLL_INTERVAL <= VIEWER_INTERVAL / 2
def test_polling_keeps_the_viewer_marker_fresh(self):
# The SSE reader touches the marker about once a second however fast
# it polls; the display must still count that as a live viewer.
assert max(VIEWER_POLL_INTERVAL, 1.0) < VIEWER_MARKER_FRESH_SEC
if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-v"]))
@@ -0,0 +1,135 @@
"""The /stream/display generator's pace (web_interface/app.py).
display_preview_generator() used to sleep a flat second between passes, while
the display wrote a snapshot up to five times a second for it: four encodes
in five were overwritten unread. The display now writes about once a second
(snapshot_policy.VIEWER_INTERVAL), so the generator checks the snapshot's
mtime every snapshot_policy.VIEWER_POLL_INTERVAL instead -- a stat -- and
sends each frame soon after it lands. Polling equal to the write period would
alias the two clocks into stale frames and the odd two-second gap.
What must not change with it: the viewer marker is touched about once a
second (more would only be extra writes to /tmp), and a pass with no snapshot
or an error still sends its message once a second, not four times.
"""
import os
import sys
import types
from contextlib import nullcontext
from pathlib import Path
from unittest.mock import MagicMock
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from src.common import snapshot_policy # noqa: E402
from web_interface import display_preview # noqa: E402
MARKER = "/tmp/led_matrix_preview_viewer" # nosec B108 - the path the generator touches
class _Stop(Exception):
"""Raised from the fake sleep to end the generator's endless loop."""
class _Stream:
"""Drives display_preview_generator() on a fake clock.
The marker is a fixed /tmp path shared with a display service that may be
running on this machine, so touches are recorded rather than made.
"""
def __init__(self, monkeypatch, snapshot_path, os_path=None):
import web_interface.app as web_app
self.now = 1000.0
self.sleeps = []
self.touches = []
self.at_sleep = {}
self.stop_after = 0
monkeypatch.setattr(web_app, "time", types.SimpleNamespace(
sleep=self._sleep, monotonic=lambda: self.now, time=lambda: self.now))
monkeypatch.setattr(web_app, "open", self._open, raising=False)
monkeypatch.setattr(web_app, "os", types.SimpleNamespace(
path=os_path or os.path, utime=lambda *a, **k: None))
monkeypatch.setattr(web_app, "config_manager",
MagicMock(**{"load_config.return_value": {}}))
monkeypatch.setattr(display_preview, "SNAPSHOT_PATH", str(snapshot_path))
self._generator = web_app.display_preview_generator
def _open(self, path, mode="r", *args, **kwargs):
assert path == MARKER and mode == "a"
self.touches.append(self.now)
return nullcontext()
def _sleep(self, seconds):
self.sleeps.append(seconds)
self.now += seconds
if len(self.sleeps) >= self.stop_after:
raise _Stop
action = self.at_sleep.get(len(self.sleeps))
if action:
action()
def run(self, passes):
"""The messages sent over ``passes`` passes, each with the pass number."""
self.stop_after = passes
sent = []
try:
for message in self._generator():
sent.append((len(self.sleeps), message))
except _Stop:
pass
return sent
def _write(path, mtime):
path.write_bytes(b"\x89PNG not really")
os.utime(path, (mtime, mtime))
def test_frames_go_out_on_the_next_poll_and_the_marker_is_touched_once_a_second(
monkeypatch, tmp_path):
snapshot = tmp_path / "led_matrix_preview.png"
_write(snapshot, 500.0)
stream = _Stream(monkeypatch, snapshot)
# The display writes a new frame once a second: every fourth poll.
for n, sleep_no in enumerate((4, 8, 12)):
stream.at_sleep[sleep_no] = lambda m=501.0 + n: _write(snapshot, m)
sent = stream.run(16)
assert stream.sleeps == [snapshot_policy.VIEWER_POLL_INTERVAL] * 16
# The first pass sends what is there; each write goes out on the poll
# straight after it, not up to a second later.
assert [pass_no for pass_no, _ in sent] == [0, 4, 8, 12]
assert all(message["image"] for _, message in sent)
# Sixteen quarter-second polls are four seconds: four touches, a second
# apart, well inside VIEWER_MARKER_FRESH_SEC.
assert stream.touches == [1000.0, 1001.0, 1002.0, 1003.0]
def test_without_a_snapshot_the_placeholder_still_goes_out_once_a_second(
monkeypatch, tmp_path):
stream = _Stream(monkeypatch, tmp_path / "missing.png")
sent = stream.run(5)
assert stream.sleeps == [1.0] * 5
assert [pass_no for pass_no, _ in sent] == [0, 1, 2, 3, 4]
assert all(message["image"] is None for _, message in sent)
assert stream.touches == [1000.0, 1001.0, 1002.0, 1003.0, 1004.0]
def test_an_error_keeps_the_one_second_pace(monkeypatch, tmp_path):
def broken(_path):
raise RuntimeError("stat failed")
stream = _Stream(monkeypatch, tmp_path / "snap.png",
os_path=types.SimpleNamespace(exists=lambda _p: True,
getmtime=broken))
sent = stream.run(3)
assert stream.sleeps == [1.0] * 3
assert [message for _, message in sent] == [
{"error": "An error occurred; see server logs"}] * 3
+19 -5
View File
@@ -33,6 +33,7 @@ from src.web_interface.error_handler import describe_exception
from src.common.path_safety import (
resolve_under, safe_path_component, safe_relative_parts,
)
from src.common import snapshot_policy
from werkzeug.exceptions import HTTPException
from src.exceptions import ConfigError
from src.plugin_system.plugin_catalog import PluginCatalog
@@ -797,12 +798,19 @@ def display_preview_generator():
"""Generate display preview updates from snapshot file"""
snapshot_path = display_preview.SNAPSHOT_PATH
# Viewer marker: this generator only runs while the broadcaster has
# subscribers (it exits with no clients), so touching the marker each
# loop tells the DISPLAY service a browser is actually watching — it
# only pays for full-rate PNG snapshot encodes while this stays fresh
# subscribers (it exits with no clients), so touching the marker tells
# the DISPLAY service a browser is actually watching — it only pays for
# viewer-rate PNG snapshot encodes while this stays fresh
# (see src/common/snapshot_policy.py).
viewer_marker_path = "/tmp/led_matrix_preview_viewer" # nosec B108 - fixed path matches display_manager
last_modified = None
# While there is a snapshot, its mtime is checked every
# VIEWER_POLL_INTERVAL, so a new frame goes out soon after it lands
# rather than up to a second later. The rest keeps its old once-a-second
# pace: the marker touch (the display counts it fresh for
# VIEWER_MARKER_FRESH_SEC), and passes with no snapshot or an error,
# each of which sends a message.
last_marker_touch = None
def _touch_viewer_marker():
try:
@@ -821,10 +829,15 @@ def display_preview_generator():
width, height = logical_size({})
while True:
delay = 1.0
try:
_touch_viewer_marker()
now = time.monotonic()
if last_marker_touch is None or now - last_marker_touch >= 1.0:
_touch_viewer_marker()
last_marker_touch = now
# Check if snapshot file exists and has been modified
if os.path.exists(snapshot_path):
delay = snapshot_policy.VIEWER_POLL_INTERVAL
current_modified = os.path.getmtime(snapshot_path)
# Only read if file is new or has been updated
@@ -842,10 +855,11 @@ def display_preview_generator():
yield display_preview.preview_payload(width, height, None)
except Exception:
delay = 1.0
app.logger.error("SSE generator error", exc_info=True)
yield {'error': 'An error occurred; see server logs'}
time.sleep(1.0) # the snapshot is re-read only when its mtime changes
time.sleep(delay) # the snapshot is re-read only when its mtime changes
# Logs generator for SSE
def logs_generator():