Compare commits

...
2 Commits
Author SHA1 Message Date
ChuckandClaude Opus 5 e26dff149e docs(sports): state what ESPN_CHUNK_WORKERS was measured to do, not more
The comment claimed the sequential fetch made scoreboards blow the 20s
startup update() timeout. A boot on this branch still deferred 12 plugins
and timed out baseball-scoreboard while its season fetches took 0.74s and
1.12s: the startup budget is spent on other per-plugin work. Say what was
measured -- 17.7s sequential, 2.6-3.3s concurrent -- and nothing else.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 08:37:21 -04:00
ChuckandClaude Opus 5 f7e5a6a4d0 fix(sports): drop capped month payloads before fetching their days
Review of the concurrent chunk fetch found it raised the worst-case peak
memory more than the concurrency explains. The old loop discarded a month
that came back at the 500-event cap the moment it saw it; the rewrite kept
every capped month alive in `results`/`slots` until all of their day
requests had finished.

Measured on a Pi 4 fetching 20260201-20260531 college baseball (four capped
months, 5462 events), peak RSS growth over the call:

  sequential (main)               83 MB
  concurrent, months retained    121 MB  (+43)
  concurrent, one worker         108 MB  -- the retention alone was +25
  concurrent, months dropped      98-100 MB (+16)

docs/LOW_MEMORY_BOARDS.md puts a 1 GB Pi 3B+ at under 200 MB of headroom,
where running out makes the board unreachable until a power cycle, so the
difference matters. The remaining +16 MB is six responses parsing at once;
three workers saved about 6 MB more, within run-to-run noise, so the worker
count stays at six.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 08:28:08 -04:00
+13 -6
View File
@@ -46,10 +46,9 @@ ESPN_MAX_LIMIT = 500
# How long a rejected range keeps later ranges from being tried as ranges.
RANGE_RETRY_SECONDS = 6 * 60 * 60
# How many chunk requests may be in flight at once. A cold college-baseball
# season is ~130 chunks once the busy months are re-asked day by day; asking
# for them one at a time took long enough that scoreboard plugins blew the
# 20s update() timeout on first run. Kept under requests' default
# How many chunk requests may be in flight at once. Four busy months of
# college baseball are ~130 chunks once each is re-asked day by day: 17.7s one
# at a time on a Pi 4, 2.6-3.3s six at a time. Kept under requests' default
# pool_maxsize of 10 so the shared Session never has to discard connections.
ESPN_CHUNK_WORKERS = 6
@@ -272,9 +271,10 @@ def fetch_espn_date_chunks(
# A month that came back at the cap is truncated; its days replace it in
# place, so merged events stay in chunk order however the requests raced.
slots: List[Any] = list(results)
slots: List[Any] = results
capped: Dict[int, List[str]] = {}
for index, (chunk, payload) in enumerate(zip(chunks, results)):
for index, chunk in enumerate(chunks):
payload = slots[index]
if payload is None or len(chunk) != 6:
continue
events = payload.get("events") if isinstance(payload, dict) else None
@@ -285,6 +285,13 @@ def fetch_espn_date_chunks(
chunk, ESPN_MAX_LIMIT,
)
capped[index] = _days_of_month(chunk)
# Drop the truncated month now rather than after its days arrive:
# a capped college-baseball month is ~2MB of parsed JSON, and
# holding four of them through ~120 day requests added ~25MB to
# the peak -- more than the concurrency itself. Low-memory boards
# (docs/LOW_MEMORY_BOARDS.md) have under 200MB of headroom.
slots[index] = None
payload = events = None
if capped:
days = [day for index in sorted(capped) for day in capped[index]]