mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-04 14:25:08 +00:00
feat(perf): frame timing for every presented frame, a soak tool, a render bench and a stall watchdog (#629)
src/common/frame_timing.py times every frame the display presents, whoever drew it, and writes cumulative counters to /dev/shm. scripts/frame_soak.py grades a running service (late frames, freezes, where the time goes) and scripts/render_bench.py the hardware and render path alone. A stall watchdog logs the stacks behind any scroll held up for 250 ms or more (LEDMATRIX_STALL_WATCHDOG_MS lowers that). See docs/SCROLL_PERFORMANCE.md, "Soaking a rig". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
@@ -0,0 +1,609 @@
|
||||
"""System-wide frame timing: one set of numbers for every presented frame.
|
||||
|
||||
Each scroller already logs its own stats line (ScrollHelper.log_frame_rate,
|
||||
the Vegas coordinator's "Vegas FPS"), but in different formats, per source,
|
||||
and Vegas only logs a healthy window at DEBUG. None of that answers the
|
||||
question a release has to answer on each rig: *over a long run, how often did
|
||||
a moving frame reach the panel late?*
|
||||
|
||||
Every frame reaches the panel through ``DisplayManager.update_display``, so it
|
||||
is recorded there, once, whoever drew it. The render thread only appends a
|
||||
tuple; a worker thread aggregates, and every ``flush_interval`` seconds writes
|
||||
cumulative counters and histograms to a small JSON file -- in ``/dev/shm`` where
|
||||
it exists, so a stats file refreshed all day costs no SD-card writes.
|
||||
``scripts/frame_soak.py`` reads it twice and reports the difference.
|
||||
|
||||
What is counted
|
||||
---------------
|
||||
Only intervals between two consecutive *scrolling* frames count: a static
|
||||
screen that changes once a second has no timing to get wrong, and the first
|
||||
frame of a scroll has no predecessor worth measuring against.
|
||||
|
||||
"Scrolling" is DisplayManager's scroll state when the frame is presented, and
|
||||
that state can go missing in the middle of a scroll. It expires after 2s
|
||||
without scroll activity, which a long enough stall outlasts, and any thread can
|
||||
clear it: plugins call ``set_scrolling_state(False)`` from their own
|
||||
``display()``, and Vegas captures some of those on the render thread between
|
||||
two of its frames. The frame after that is recorded as static, and the interval
|
||||
it ends -- the stall, or the capture -- would vanish from the report. So a
|
||||
single static frame between two scrolling ones, with the scroll picking up
|
||||
again within ``RESUME_SECONDS``, is treated as a frame of the scroll: both of
|
||||
its intervals count. A second static frame in a row means the scroll really
|
||||
ended. (On hdpi on 2026-09-24 the watchdog logged a 1.9s stall that the soak
|
||||
report did not have; this is how.)
|
||||
|
||||
A frame held for ``hold`` refreshes should arrive ``hold`` refresh periods
|
||||
after the one before it. One that arrives a whole refresh or more after that is
|
||||
**late**: the panel showed the previous frame again, which on a moving strip is
|
||||
a visible hitch. ``missed_refreshes`` sums how many refreshes late.
|
||||
|
||||
An interval of ``FREEZE_SECONDS`` or more is a **freeze** instead -- a
|
||||
recompose, a plugin handover, a blocking call on the render thread. Those are
|
||||
counted separately, both because they are a different fault and because
|
||||
folding a single 400ms handover into the late count as "40 missed refreshes"
|
||||
would drown the jitter the late count exists to measure. ``freeze_by`` splits
|
||||
them by length. Intervals of ``GAP_SECONDS`` or more are ignored as not being
|
||||
frames of one scroll at all.
|
||||
|
||||
A frame that arrives a whole refresh or more *early* means the swap did not
|
||||
wait for the panel: the emulator, the fallback display, or a hold that was not
|
||||
the one in effect. Those are counted as **early**, and a run with more than a
|
||||
trace of them was not locked to the panel, so its late count means nothing.
|
||||
|
||||
The refresh period is estimated from the frames themselves: swaps that block
|
||||
on vsync can only land on refresh boundaries, so the low end of
|
||||
interval / hold is the period. It is the smallest per-window 10th percentile
|
||||
seen so far, over windows with enough frames to trust -- except that a window
|
||||
cutting it by more than ``MAX_REFRESH_DROP`` is ignored. A panel's refresh does
|
||||
not jump like that; swaps that stopped blocking do, and adopting their period
|
||||
would make every early frame look on time.
|
||||
|
||||
A caller that has measured the panel independently -- ``scripts/render_bench.py``
|
||||
times bare swaps first with :func:`measure_refresh_hz` -- passes that rate in
|
||||
as ``refresh_hz``. The estimate then starts from it instead of from the frames,
|
||||
which is what catches a loop that never locked at all: one that free-runs
|
||||
faster than the panel (every frame early) or sits at half its rate (every
|
||||
frame late), both of which look self-consistent to an estimate taken from
|
||||
their own intervals.
|
||||
|
||||
Stall watchdog
|
||||
--------------
|
||||
Counting a freeze says that it happened, not why. ``StallWatchdog`` watches the
|
||||
same frames from its own thread and, when a scroll's last frame is more than
|
||||
``STALL_SECONDS`` old, logs the stack of the thread that presented it and the
|
||||
top of every other thread's, so the log names what the render thread was
|
||||
waiting on. It also measures how late its own wake-up was: if the watchdog was
|
||||
held up as long as the render thread, the whole interpreter was blocked (C
|
||||
code holding the GIL, or the process not scheduled), not one thread on a lock.
|
||||
Set ``LEDMATRIX_STALL_WATCHDOG=0`` to turn it off, or
|
||||
``LEDMATRIX_STALL_WATCHDOG_MS`` to dump at a lower threshold -- 30 catches
|
||||
frames three refreshes late, which is where GIL contention shows. It polls
|
||||
three times per threshold, so keep it to diagnostic runs, not soaks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
from typing import Any, Callable, Dict, List, Optional, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
#: Bumped when a field changes meaning, so a reader can refuse stale files.
|
||||
SCHEMA_VERSION = 1
|
||||
|
||||
#: Histogram resolution. 64ms of range covers any frame worth drawing a
|
||||
#: distribution of; everything beyond lands in the last bucket.
|
||||
BUCKET_MS = 0.25
|
||||
BUCKET_COUNT = 256
|
||||
|
||||
#: See the module docstring.
|
||||
FREEZE_SECONDS = 0.25
|
||||
|
||||
#: Intervals this long are not frames of one scroll. This used to be 1s,
|
||||
#: which silently dropped every 1-2s stall inside a scroll. It is now only a
|
||||
#: sanity bound.
|
||||
GAP_SECONDS = 5.0
|
||||
|
||||
#: A frame recorded as static between two scrolling frames is a frame of the
|
||||
#: scroll whose state went missing, if the scroll resumes within this long.
|
||||
#: See "What is counted".
|
||||
RESUME_SECONDS = 1.0
|
||||
|
||||
#: Buckets for freeze length, as cumulative counters a soak can difference.
|
||||
FREEZE_BUCKETS = ((0.5, "<0.5s"), (1.0, "0.5-1s"), (2.0, "1-2s"),
|
||||
(float("inf"), "2s+"))
|
||||
|
||||
#: A window may lower the refresh-period estimate by at most this fraction.
|
||||
MAX_REFRESH_DROP = 0.2
|
||||
|
||||
#: A window needs this many scrolling frames before its refresh estimate is
|
||||
#: trusted -- about a second of scrolling.
|
||||
MIN_FRAMES_FOR_REFRESH = 90
|
||||
|
||||
FLUSH_INTERVAL = 10.0
|
||||
|
||||
#: A scroll's last frame older than this is a stall worth a stack dump.
|
||||
STALL_SECONDS = 0.25
|
||||
#: How often the watchdog looks. Also the resolution of its starvation check.
|
||||
WATCHDOG_POLL_SECONDS = 0.05
|
||||
#: At most one stack dump per this many seconds: a stall that repeats every
|
||||
#: extension would otherwise write the same stacks to the SD card all day.
|
||||
STALL_LOG_INTERVAL = 30.0
|
||||
|
||||
#: Written by the display service, read by scripts/frame_soak.py and anything
|
||||
#: else that wants the numbers. The web UI's viewer marker lives in /tmp; this
|
||||
#: goes to RAM where there is some, since it is rewritten all day.
|
||||
STATS_FILENAME = "ledmatrix_frame_stats.json"
|
||||
|
||||
|
||||
def default_stats_path() -> str:
|
||||
# A fixed name in a shared directory is safe here: write() creates its
|
||||
# temp file with mkstemp and os.replace()s it over this path, which swaps
|
||||
# out whatever is there -- a planted symlink included -- without following it.
|
||||
base = "/dev/shm" if os.path.isdir("/dev/shm") else tempfile.gettempdir() # nosec B108
|
||||
return os.path.join(base, STATS_FILENAME)
|
||||
|
||||
|
||||
def _bucket(seconds: float) -> int:
|
||||
index = int(seconds * 1000.0 / BUCKET_MS)
|
||||
return min(max(index, 0), BUCKET_COUNT - 1)
|
||||
|
||||
|
||||
def binding_releases_gil() -> Optional[bool]:
|
||||
"""Whether the loaded rgbmatrix binding releases the GIL, or None.
|
||||
|
||||
The stock binding blocks in SwapOnVSync holding the GIL, which starves
|
||||
every other thread for most of each frame (docs/SCROLL_PERFORMANCE.md).
|
||||
scripts/build_rgbmatrix_nogil.sh rebuilds it, and the rebuilt module links
|
||||
PyEval_SaveThread where the stock one never does -- a crude test, but the
|
||||
only one that needs neither a probe on the panel nor the source tree the
|
||||
module was built from. None when no hardware binding is loaded.
|
||||
"""
|
||||
module = sys.modules.get("rgbmatrix.core")
|
||||
path = getattr(module, "__file__", None)
|
||||
if not path:
|
||||
return None
|
||||
try:
|
||||
with open(path, "rb") as handle:
|
||||
return b"PyEval_SaveThread" in handle.read()
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def _pi_model() -> Optional[str]:
|
||||
try:
|
||||
with open("/proc/device-tree/model", "rb") as handle:
|
||||
return handle.read().rstrip(b"\0").decode("ascii", "replace").strip()
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def measure_refresh_hz(matrix: Any, seconds: float = 4.0) -> float:
|
||||
"""The panel's refresh rate with nothing else running, by timing bare swaps.
|
||||
|
||||
``SwapOnVSync`` blocks until the panel's next refresh, so a loop that does
|
||||
nothing else runs at exactly the panel's rate. ``limit_refresh_rate_hz`` is
|
||||
a *cap*, and a long chain, a high ``pwm_bits`` or an older Pi will sit well
|
||||
under it. Solving scroll speeds against a cap the panel cannot reach is
|
||||
what produces "3px every 4 refreshes" and the judder that comes with it.
|
||||
|
||||
This is the idle rate. The panel refreshes a few percent slower while the
|
||||
Pi is also pushing frames into it (100.4Hz idle against 96.3Hz scrolling on
|
||||
a Pi 4 driving 512x64), which is why the recorder reads the rendering rate
|
||||
back from the frames rather than trusting this.
|
||||
|
||||
Pass the matrix the display is already running on rather than opening a
|
||||
second one: the GPIO has a single owner, and the options in force change
|
||||
the answer.
|
||||
|
||||
:returns: measured Hz, or 0.0 if the matrix cannot be swapped (no
|
||||
hardware, a stub, a mock).
|
||||
"""
|
||||
try:
|
||||
canvas = matrix.CreateFrameCanvas()
|
||||
# Discard the first swap: it carries construction and first-touch costs
|
||||
# that have nothing to do with the steady-state refresh.
|
||||
canvas = matrix.SwapOnVSync(canvas)
|
||||
except Exception: # pylint: disable=broad-except
|
||||
return 0.0
|
||||
|
||||
frames = 0
|
||||
started = time.perf_counter()
|
||||
while time.perf_counter() - started < seconds:
|
||||
canvas = matrix.SwapOnVSync(canvas)
|
||||
frames += 1
|
||||
elapsed = time.perf_counter() - started
|
||||
if elapsed <= 0 or frames <= 0:
|
||||
return 0.0
|
||||
return frames / elapsed
|
||||
|
||||
class FrameTimingRecorder:
|
||||
"""Collects per-frame timings on the render thread; aggregates elsewhere.
|
||||
|
||||
``record`` is the only method the render thread calls, and it does no more
|
||||
than compare two floats and append a tuple.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
path: Optional[str] = None,
|
||||
flush_interval: float = FLUSH_INTERVAL,
|
||||
info: Optional[Dict[str, Any]] = None,
|
||||
refresh_hz: Optional[float] = None,
|
||||
):
|
||||
"""
|
||||
:param refresh_hz: the panel's rate, measured independently (see the
|
||||
module docstring). Omit it to estimate from the frames alone, as
|
||||
the display service does.
|
||||
"""
|
||||
self.path = path or default_stats_path()
|
||||
self.flush_interval = flush_interval
|
||||
self.info = dict(info or {})
|
||||
|
||||
# Render-thread state.
|
||||
self._pending: List[Tuple[float, float, float, int]] = []
|
||||
self._static_frames = 0
|
||||
self._previous: Optional[Tuple[float, bool, int]] = None
|
||||
# The interval ended by a static frame that followed a scrolling one,
|
||||
# until the next frame shows whether the scroll went on.
|
||||
self._unsure: Optional[Tuple[float, float, float, int]] = None
|
||||
self._last_flush: Optional[float] = None
|
||||
self._queue: "queue.SimpleQueue" = queue.SimpleQueue()
|
||||
self._worker: Optional[threading.Thread] = None
|
||||
|
||||
# Worker-thread state. Nothing on the render thread reads these.
|
||||
self.started = time.time()
|
||||
self.refresh_period: Optional[float] = (
|
||||
1.0 / refresh_hz if refresh_hz and refresh_hz > 0 else None)
|
||||
# The first estimate, until a second window agrees with it.
|
||||
self._refresh_candidate: Optional[float] = None
|
||||
self.totals: Dict[str, Any] = {
|
||||
"static_frames": 0,
|
||||
"scroll_frames": 0,
|
||||
"late_frames": 0,
|
||||
"missed_refreshes": 0,
|
||||
"late_by": {"1": 0, "2": 0, "3-5": 0, "6+": 0},
|
||||
"early_frames": 0,
|
||||
# Frames judged against a known refresh period: the denominator
|
||||
# for the late and early rates. Frames before the period is known
|
||||
# are neither, and must not dilute them.
|
||||
"timed_frames": 0,
|
||||
"freezes": 0,
|
||||
"freeze_seconds": 0.0,
|
||||
"freeze_by": {label: 0 for _, label in FREEZE_BUCKETS},
|
||||
"worst_interval_ms": 0.0,
|
||||
}
|
||||
self.histograms: Dict[str, Dict[int, int]] = {
|
||||
"blit": {}, "wait": {}, "work": {}, "interval_per_hold": {},
|
||||
}
|
||||
self._binding_gil: Optional[bool] = None
|
||||
self._binding_checked = False
|
||||
|
||||
# Read by the stall watchdog from its own thread: one tuple assignment,
|
||||
# so it always sees a consistent (time, scrolling, thread) triple.
|
||||
self.last_frame: Optional[Tuple[float, bool, int]] = None
|
||||
#: Whether a scroll is running *now*, supplied by the display manager.
|
||||
#: The last frame's flag alone would call the end of every scroll a
|
||||
#: stall.
|
||||
self.scrolling_now: Optional[Callable[[], bool]] = None
|
||||
self.watchdog: Optional["StallWatchdog"] = None
|
||||
|
||||
def close(self) -> None:
|
||||
"""Stop the stall watchdog, if one was started."""
|
||||
watchdog, self.watchdog = self.watchdog, None
|
||||
if watchdog is not None:
|
||||
watchdog.stop()
|
||||
|
||||
# -- render thread ------------------------------------------------------
|
||||
|
||||
def record(self, blit: float, wait: float, hold: int, scrolling: bool,
|
||||
presented_at: float) -> None:
|
||||
"""One frame reached the panel.
|
||||
|
||||
:param blit: seconds spent copying the frame into the canvas.
|
||||
:param wait: seconds SwapOnVSync blocked.
|
||||
:param hold: the refreshes this frame was held for.
|
||||
:param scrolling: whether a scroll was running when it was presented.
|
||||
:param presented_at: ``time.perf_counter()`` when the swap returned.
|
||||
"""
|
||||
previous = self._previous
|
||||
self._previous = (presented_at, scrolling, hold)
|
||||
self.last_frame = (presented_at, scrolling, threading.get_ident())
|
||||
if not scrolling:
|
||||
self._static_frames += 1
|
||||
# The scroll ended, or its state went missing for this frame: the
|
||||
# next frame says which. Its hold may have been dropped with the
|
||||
# state, so the interval is due at the scroll's own.
|
||||
self._unsure = None
|
||||
if previous is not None and previous[1]:
|
||||
self._unsure = (presented_at - previous[0], blit, wait, previous[2])
|
||||
elif self.watchdog is None and self.scrolling_now is not None \
|
||||
and os.environ.get("LEDMATRIX_STALL_WATCHDOG", "1") != "0":
|
||||
self.watchdog = StallWatchdog(self, **watchdog_settings())
|
||||
self.watchdog.start()
|
||||
elif previous is not None:
|
||||
interval = presented_at - previous[0]
|
||||
unsure, self._unsure = self._unsure, None
|
||||
if previous[1]:
|
||||
if interval < GAP_SECONDS:
|
||||
self._pending.append((interval, blit, wait, hold))
|
||||
elif unsure is not None and interval < RESUME_SECONDS:
|
||||
# One static frame between two scrolling ones: the scroll never
|
||||
# stopped, only its state did. Both intervals were motion.
|
||||
self._static_frames -= 1
|
||||
if unsure[0] < GAP_SECONDS:
|
||||
self._pending.append(unsure)
|
||||
self._pending.append((interval, blit, wait, hold))
|
||||
|
||||
if self._last_flush is None:
|
||||
self._last_flush = presented_at
|
||||
elif presented_at - self._last_flush >= self.flush_interval:
|
||||
self._hand_off()
|
||||
self._last_flush = presented_at
|
||||
|
||||
def _hand_off(self) -> None:
|
||||
batch, self._pending = self._pending, []
|
||||
static, self._static_frames = self._static_frames, 0
|
||||
self._queue.put((batch, static))
|
||||
if self._worker is None or not self._worker.is_alive():
|
||||
self._worker = threading.Thread(
|
||||
target=self._run, daemon=True, name="frame-timing")
|
||||
self._worker.start()
|
||||
|
||||
def drain(self) -> None:
|
||||
"""Aggregate everything recorded so far, on the calling thread.
|
||||
|
||||
For a caller that owns the recorder outright and wants exact numbers at
|
||||
a moment of its choosing -- the benchmark, between warm-up and run and
|
||||
at the end. Construct it with ``flush_interval=float('inf')`` so the
|
||||
worker never runs; the two must not aggregate at once.
|
||||
"""
|
||||
batch, self._pending = self._pending, []
|
||||
static, self._static_frames = self._static_frames, 0
|
||||
self.aggregate(batch, static)
|
||||
|
||||
# -- worker thread ------------------------------------------------------
|
||||
|
||||
def _run(self) -> None:
|
||||
while True:
|
||||
batch, static = self._queue.get()
|
||||
try:
|
||||
self.aggregate(batch, static)
|
||||
self.write()
|
||||
except Exception: # never let telemetry take anything down
|
||||
logger.debug("Frame timing flush failed", exc_info=True)
|
||||
|
||||
def aggregate(self, batch: List[Tuple[float, float, float, int]],
|
||||
static: int) -> None:
|
||||
"""Fold one window of frames into the running totals."""
|
||||
totals = self.totals
|
||||
totals["static_frames"] += static
|
||||
|
||||
per_hold = sorted(interval / max(1, hold)
|
||||
for interval, _, _, hold in batch
|
||||
if interval < FREEZE_SECONDS)
|
||||
if len(per_hold) >= MIN_FRAMES_FOR_REFRESH:
|
||||
estimate = per_hold[len(per_hold) // 10]
|
||||
current = self.refresh_period
|
||||
if estimate <= 0:
|
||||
pass
|
||||
elif current is None:
|
||||
# Adopt the first period only once two windows in a row agree:
|
||||
# one loaded window at startup, most of its frames a refresh
|
||||
# late, would otherwise fix a period twice the real one for
|
||||
# the life of the process, since later windows may only lower
|
||||
# it by MAX_REFRESH_DROP.
|
||||
candidate = self._refresh_candidate
|
||||
if candidate and abs(estimate - candidate) <= candidate * MAX_REFRESH_DROP:
|
||||
self.refresh_period = min(candidate, estimate)
|
||||
else:
|
||||
self._refresh_candidate = estimate
|
||||
elif current * (1.0 - MAX_REFRESH_DROP) <= estimate < current:
|
||||
self.refresh_period = estimate
|
||||
period = self.refresh_period
|
||||
|
||||
histograms = self.histograms
|
||||
for interval, blit, wait, hold in batch:
|
||||
totals["worst_interval_ms"] = max(totals["worst_interval_ms"],
|
||||
interval * 1000.0)
|
||||
if interval >= FREEZE_SECONDS:
|
||||
totals["freezes"] += 1
|
||||
totals["freeze_seconds"] += interval
|
||||
label = next(name for limit, name in FREEZE_BUCKETS
|
||||
if interval < limit)
|
||||
totals["freeze_by"][label] += 1
|
||||
continue
|
||||
totals["scroll_frames"] += 1
|
||||
for name, value in (("blit", blit), ("wait", wait),
|
||||
("work", max(0.0, interval - blit - wait)),
|
||||
("interval_per_hold", interval / max(1, hold))):
|
||||
bucket = _bucket(value)
|
||||
histogram = histograms[name]
|
||||
histogram[bucket] = histogram.get(bucket, 0) + 1
|
||||
if period:
|
||||
totals["timed_frames"] += 1
|
||||
missed = round(interval / period) - hold
|
||||
if missed >= 1:
|
||||
totals["late_frames"] += 1
|
||||
totals["missed_refreshes"] += missed
|
||||
key = ("1" if missed == 1 else "2" if missed == 2
|
||||
else "3-5" if missed <= 5 else "6+")
|
||||
totals["late_by"][key] += 1
|
||||
elif missed <= -1:
|
||||
totals["early_frames"] += 1
|
||||
|
||||
def snapshot(self) -> Dict[str, Any]:
|
||||
"""The JSON document: cumulative since this process started."""
|
||||
if not self._binding_checked:
|
||||
self._binding_gil = binding_releases_gil()
|
||||
self._binding_checked = True
|
||||
info = dict(self.info)
|
||||
info.setdefault("pi_model", _pi_model())
|
||||
period = self.refresh_period
|
||||
return {
|
||||
"version": SCHEMA_VERSION,
|
||||
"pid": os.getpid(),
|
||||
"started": self.started,
|
||||
"updated": time.time(),
|
||||
"bucket_ms": BUCKET_MS,
|
||||
"freeze_seconds": FREEZE_SECONDS,
|
||||
"measured_refresh_hz": round(1.0 / period, 2) if period else None,
|
||||
"binding_releases_gil": self._binding_gil,
|
||||
"info": info,
|
||||
"totals": copy.deepcopy(self.totals),
|
||||
# JSON keys are strings; readers convert back.
|
||||
"histograms": {name: {str(k): v for k, v in sorted(h.items())}
|
||||
for name, h in self.histograms.items()},
|
||||
}
|
||||
|
||||
def write(self) -> None:
|
||||
"""Replace the stats file atomically with the current snapshot."""
|
||||
directory = os.path.dirname(self.path) or "."
|
||||
fd, tmp = tempfile.mkstemp(dir=directory, prefix=".frame_stats.",
|
||||
suffix=".tmp")
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(self.snapshot(), handle)
|
||||
os.chmod(tmp, 0o644)
|
||||
os.replace(tmp, self.path)
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def watchdog_settings() -> Dict[str, float]:
|
||||
"""StallWatchdog arguments from ``LEDMATRIX_STALL_WATCHDOG_MS``, if set.
|
||||
|
||||
The poll comes down with the threshold, or a stall shorter than one poll
|
||||
would go unseen.
|
||||
"""
|
||||
try:
|
||||
ms = float(os.environ.get("LEDMATRIX_STALL_WATCHDOG_MS") or 0)
|
||||
except ValueError:
|
||||
ms = 0.0
|
||||
if ms <= 0:
|
||||
return {}
|
||||
threshold = ms / 1000.0
|
||||
return {"threshold": threshold,
|
||||
"poll": min(WATCHDOG_POLL_SECONDS, threshold / 3)}
|
||||
|
||||
|
||||
class StallWatchdog:
|
||||
"""Log what the render thread is doing when a scroll stops presenting.
|
||||
|
||||
See the module docstring. Polls; never touches the render thread.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
recorder: FrameTimingRecorder,
|
||||
threshold: float = STALL_SECONDS,
|
||||
poll: float = WATCHDOG_POLL_SECONDS,
|
||||
log_interval: float = STALL_LOG_INTERVAL,
|
||||
clock: Callable[[], float] = time.perf_counter,
|
||||
):
|
||||
self.recorder = recorder
|
||||
self.threshold = threshold
|
||||
self.poll = poll
|
||||
self.log_interval = log_interval
|
||||
self.clock = clock
|
||||
self.stalls = 0
|
||||
self._last_dump: Optional[float] = None
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
self._stop = threading.Event()
|
||||
|
||||
def start(self) -> None:
|
||||
self._thread = threading.Thread(
|
||||
target=self._run, daemon=True, name="stall-watchdog")
|
||||
self._thread.start()
|
||||
|
||||
def stop(self, timeout: float = 1.0) -> None:
|
||||
"""End the polling thread (DisplayManager.cleanup calls this)."""
|
||||
self._stop.set()
|
||||
thread = self._thread
|
||||
if thread is not None and thread is not threading.current_thread():
|
||||
thread.join(timeout)
|
||||
|
||||
def _run(self) -> None:
|
||||
last_wake = self.clock()
|
||||
stall_from: Optional[float] = None # presented_at of the stalled frame
|
||||
dumped = False
|
||||
while not self._stop.wait(self.poll):
|
||||
now = self.clock()
|
||||
late = max(0.0, now - last_wake - self.poll)
|
||||
last_wake = now
|
||||
try:
|
||||
stall_from, dumped = self.check(now, late, stall_from, dumped)
|
||||
except Exception: # never let a diagnostic take anything down
|
||||
logger.debug("Stall watchdog check failed", exc_info=True)
|
||||
|
||||
def check(self, now: float, late: float, stall_from: Optional[float],
|
||||
dumped: bool) -> Tuple[Optional[float], bool]:
|
||||
"""One look. Returns the updated (stall_from, dumped) state."""
|
||||
frame = self.recorder.last_frame
|
||||
if frame is None:
|
||||
return None, False
|
||||
presented_at, scrolling, ident = frame
|
||||
|
||||
if stall_from is not None and presented_at != stall_from:
|
||||
# A frame arrived: the stall is over.
|
||||
if dumped:
|
||||
logger.warning(
|
||||
"Render stall over: no frame for %.0fms",
|
||||
(presented_at - stall_from) * 1000.0)
|
||||
return None, False
|
||||
|
||||
scrolling_now = self.recorder.scrolling_now
|
||||
if (stall_from is not None and now - stall_from >= GAP_SECONDS
|
||||
and (scrolling_now is None or not scrolling_now())):
|
||||
# The scroll ended without another frame: nothing more to time.
|
||||
# Only past GAP_SECONDS: the scroll state expires after 2s without
|
||||
# activity, which a stall outlasts, and its end still wants saying.
|
||||
return None, False
|
||||
age = now - presented_at
|
||||
if (stall_from is None and scrolling and age >= self.threshold
|
||||
and scrolling_now is not None and scrolling_now()):
|
||||
self.stalls += 1
|
||||
if self._last_dump is None or now - self._last_dump >= self.log_interval:
|
||||
self._last_dump = now
|
||||
logger.warning(self.describe(ident, age, late))
|
||||
return presented_at, True
|
||||
return presented_at, False
|
||||
return stall_from, dumped
|
||||
|
||||
def describe(self, ident: int, age: float, late: float) -> str:
|
||||
"""The stack dump: the stalled thread in full, the rest in brief."""
|
||||
names = {t.ident: t.name for t in threading.enumerate()}
|
||||
frames = sys._current_frames()
|
||||
lines = [
|
||||
f"Render stall: no frame for {age * 1000.0:.0f}ms mid-scroll "
|
||||
f"(watchdog woke {late * 1000.0:.0f}ms late"
|
||||
+ ("; the interpreter itself was blocked" if late >= age / 2 else "")
|
||||
+ ")",
|
||||
f"-- {names.get(ident, ident)} (presents frames):",
|
||||
]
|
||||
stalled = frames.get(ident)
|
||||
if stalled is not None:
|
||||
lines.extend(line.rstrip() for line in
|
||||
traceback.format_stack(stalled, limit=12))
|
||||
for other, frame in frames.items():
|
||||
if other in (ident, threading.get_ident()):
|
||||
continue
|
||||
top = traceback.extract_stack(frame, limit=3)
|
||||
where = " <- ".join(
|
||||
f"{os.path.basename(f.filename)}:{f.lineno} {f.name}"
|
||||
for f in reversed(top))
|
||||
lines.append(f"-- {names.get(other, other)}: {where}")
|
||||
return "\n".join(lines)
|
||||
@@ -51,6 +51,7 @@ import zlib
|
||||
import freetype
|
||||
|
||||
from src.common import snapshot_policy
|
||||
from src.common.frame_timing import FrameTimingRecorder
|
||||
from src.deprecation import deprecated
|
||||
from src.logging_config import get_logger
|
||||
from src.common.permission_utils import (
|
||||
@@ -236,6 +237,11 @@ class DisplayManager:
|
||||
# See src/common/scroll_config.py and scripts/scroll_speeds.py.
|
||||
self._frame_hold = 1
|
||||
|
||||
# Timing of every presented frame, whoever drew it, for
|
||||
# scripts/frame_soak.py. See src/common/frame_timing.py.
|
||||
self.frame_timing = FrameTimingRecorder(info=self._frame_timing_info())
|
||||
self.frame_timing.scrolling_now = self._scrolling_now
|
||||
|
||||
self._scrolling_state = {
|
||||
'is_scrolling': False,
|
||||
'last_scroll_activity': 0,
|
||||
@@ -798,15 +804,21 @@ class DisplayManager:
|
||||
|
||||
# Copy the current image to the offscreen canvas. In double-sided
|
||||
# mode the logical screen is first tiled across the full chain.
|
||||
blit_started = time.perf_counter()
|
||||
if self._double_sided is not None:
|
||||
self.offscreen_canvas.SetImage(self._composite_double_sided())
|
||||
else:
|
||||
self.offscreen_canvas.SetImage(self.image)
|
||||
blit_done = time.perf_counter()
|
||||
|
||||
# Swap buffers immediately. framerate_fraction holds the frame
|
||||
# for N refreshes; SwapOnVSync blocks for all of them, which is
|
||||
# what paces the render loop to the chosen frame rate.
|
||||
self.matrix.SwapOnVSync(self.offscreen_canvas, self._frame_hold)
|
||||
presented_at = time.perf_counter()
|
||||
self.frame_timing.record(
|
||||
blit_done - blit_started, presented_at - blit_done,
|
||||
self._frame_hold, self.is_currently_scrolling(), presented_at)
|
||||
|
||||
# Swap our canvas references
|
||||
self.offscreen_canvas, self.current_canvas = self.current_canvas, self.offscreen_canvas
|
||||
@@ -1295,6 +1307,9 @@ class DisplayManager:
|
||||
self._new_canvas(self.width, self.height)
|
||||
except (OSError, RuntimeError, ValueError, MemoryError):
|
||||
logger.debug("Canvas reset during cleanup failed", exc_info=True)
|
||||
# The stall watchdog would otherwise outlive this manager.
|
||||
if getattr(self, 'frame_timing', None) is not None:
|
||||
self.frame_timing.close()
|
||||
# Reset the singleton state when cleaning up
|
||||
DisplayManager._instance = None
|
||||
|
||||
@@ -1390,6 +1405,27 @@ class DisplayManager:
|
||||
value = 0.0
|
||||
return value if value > 0 else 100.0
|
||||
|
||||
def _scrolling_now(self) -> bool:
|
||||
"""Whether a scroll is running, without is_currently_scrolling()'s
|
||||
side effect of expiring the state -- safe from the stall watchdog's
|
||||
thread."""
|
||||
state = self._scrolling_state
|
||||
return bool(state['is_scrolling']) and (
|
||||
time.time() - state['last_scroll_activity']
|
||||
<= state['scroll_inactivity_threshold'])
|
||||
|
||||
def _frame_timing_info(self) -> Dict[str, Any]:
|
||||
"""What the frame-timing stats were measured on, for the soak report."""
|
||||
display = self.config.get('display') or {}
|
||||
hardware = display.get('hardware') or {}
|
||||
runtime = display.get('runtime') or {}
|
||||
info = {key: hardware.get(key) for key in (
|
||||
'rows', 'cols', 'chain_length', 'parallel', 'pwm_bits',
|
||||
'hardware_mapping', 'limit_refresh_rate_hz', 'pixel_mapper_config')}
|
||||
info['gpio_slowdown'] = runtime.get('gpio_slowdown')
|
||||
info['emulator'] = os.environ.get('EMULATOR', 'false') == 'true'
|
||||
return info
|
||||
|
||||
def set_frame_hold(self, refreshes: int) -> None:
|
||||
"""Hold each pushed frame for this many panel refreshes (>=1).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user