mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-08-12 22:28:06 +00:00
fix(vegas): let the watchdog see stalls that hold the GIL
The watchdog only noticed a late heartbeat, which a whole class of freeze can never produce: if the loop is inside one long C call that holds the GIL, this thread cannot run during the stall, and by the time it does the loop has already checked in. On the dev rig that hid a recurring 3.2s freeze completely -- twenty minutes of watching produced one dump, for an unrelated 0.4s stall. What it can still observe is that its own sleep ran long. A badly overshot wait is now reported as a stall in its own right. The stacks are stale by then and the message says so, but knowing the freeze is GIL-holding is most of the diagnosis: it rules out lock contention and scheduling, and points at a single long C call. This also explains why lowering sys.setswitchinterval changed nothing -- the switch interval cannot preempt a C call that never releases the GIL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Udr6MfaFLUPhX5Fgo67Jf5
This commit is contained in:
co-authored by
Claude Opus 5
parent
256925a806
commit
d754847d8b
@@ -71,11 +71,30 @@ class _StallWatchdog:
|
||||
self._stop.set()
|
||||
|
||||
def _watch(self) -> None:
|
||||
while not self._stop.wait(self.threshold / 4.0):
|
||||
poll = self.threshold / 4.0
|
||||
while True:
|
||||
woke_at = time.time()
|
||||
if self._stop.wait(poll):
|
||||
break
|
||||
with self._lock:
|
||||
last = self._beat
|
||||
stalled = time.time() - last
|
||||
if stalled < self.threshold or last == self._dumped_for:
|
||||
now = time.time()
|
||||
stalled = now - last
|
||||
# A stall inside a C call that holds the GIL never shows up as a
|
||||
# late beat: this thread cannot run during it, and by the time it
|
||||
# does the loop has already checked in. What it can see is that
|
||||
# its own sleep ran long. Treat a badly overshot wait as a stall
|
||||
# in its own right -- the stacks are stale by then, but knowing
|
||||
# the freeze is GIL-holding is itself the diagnosis.
|
||||
overshoot = (now - woke_at) - poll
|
||||
if overshoot > self.threshold:
|
||||
logger.warning(
|
||||
"render loop stalled %.2fs holding the GIL -- no Python "
|
||||
"frames ran, so the stacks below are from after it ended; "
|
||||
"look for one long C call (a large PIL operation, a "
|
||||
"compress, a big allocation)", overshoot)
|
||||
stalled = overshoot
|
||||
elif stalled < self.threshold or last == self._dumped_for:
|
||||
continue
|
||||
self._dumped_for = last # one dump per stall, not per poll
|
||||
frames = sys._current_frames()
|
||||
|
||||
Reference in New Issue
Block a user