fix(vegas): let the watchdog see stalls that hold the GIL

The watchdog only noticed a late heartbeat, which a whole class of
freeze can never produce: if the loop is inside one long C call that
holds the GIL, this thread cannot run during the stall, and by the time
it does the loop has already checked in. On the dev rig that hid a
recurring 3.2s freeze completely -- twenty minutes of watching produced
one dump, for an unrelated 0.4s stall.

What it can still observe is that its own sleep ran long. A badly
overshot wait is now reported as a stall in its own right. The stacks
are stale by then and the message says so, but knowing the freeze is
GIL-holding is most of the diagnosis: it rules out lock contention and
scheduling, and points at a single long C call.

This also explains why lowering sys.setswitchinterval changed nothing --
the switch interval cannot preempt a C call that never releases the GIL.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Udr6MfaFLUPhX5Fgo67Jf5
This commit is contained in:
ChuckBuilds
2026-08-12 08:45:06 -04:00
co-authored by Claude Opus 5
parent 256925a806
commit d754847d8b
+22 -3
View File
@@ -71,11 +71,30 @@ class _StallWatchdog:
self._stop.set() self._stop.set()
def _watch(self) -> None: def _watch(self) -> None:
while not self._stop.wait(self.threshold / 4.0): poll = self.threshold / 4.0
while True:
woke_at = time.time()
if self._stop.wait(poll):
break
with self._lock: with self._lock:
last = self._beat last = self._beat
stalled = time.time() - last now = time.time()
if stalled < self.threshold or last == self._dumped_for: stalled = now - last
# A stall inside a C call that holds the GIL never shows up as a
# late beat: this thread cannot run during it, and by the time it
# does the loop has already checked in. What it can see is that
# its own sleep ran long. Treat a badly overshot wait as a stall
# in its own right -- the stacks are stale by then, but knowing
# the freeze is GIL-holding is itself the diagnosis.
overshoot = (now - woke_at) - poll
if overshoot > self.threshold:
logger.warning(
"render loop stalled %.2fs holding the GIL -- no Python "
"frames ran, so the stacks below are from after it ended; "
"look for one long C call (a large PIL operation, a "
"compress, a big allocation)", overshoot)
stalled = overshoot
elif stalled < self.threshold or last == self._dumped_for:
continue continue
self._dumped_for = last # one dump per stall, not per poll self._dumped_for = last # one dump per stall, not per poll
frames = sys._current_frames() frames = sys._current_frames()