docs(sports): family 7 is adopted (core 3.8.4, ledmatrix-plugins #643) (#788)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

docs/SPORTS_UNIFICATION.md

@@ -350,7 +350,7 @@ compare abbreviations exactly, and
 abbreviation (or by `_team_in` where a plugin has one) instead of through
 `_favorite_key`. The result-colour helpers also wait (decision above).
 
-### Family 7: the other-games rotation (core done; adoption waits for a release)
+### Family 7: the other-games rotation (done: core 3.8.4, adopted)
 
 ledmatrix-plugins `scripts/test_other_games_rotation.py` (#640) pinned 96 rows
 across the nine plugins first: `_by_importance` per rankings table, core's
@@ -468,7 +468,7 @@ release.
 | 4 | Identical sweep | `manager.py`: `_dispatch_switch_refresh`, `_favorite_team_is_live`, `get_vegas_priority_weight`, `_game_involves`, `_favorite_scan_targets`, `_favorite_scan_games`, `_get_total_games_for_manager` (all nine, 1); the live-scroll helpers `_preserving_scroll_position`, `_refresh_live_scroll_managers`, `_live_scroll_managers`, `_note_live_scroll_built`, `_live_scroll_needs_rebuild`, `_live_scroll_fields` (eight, 1). `sports.py`: `_card_option`, `_filtered_or_all`, `_effective_live_duration`, `_recent_date_text` (eight, 1). 58 identical families in all | Nothing to decide; brings `manager.py` into core as a `SportsPluginHostMixin`. `_resolve_font_path` (identical in nine `sports.py` and eight renderers) becomes `sports_font_path.resolve_font_path`, not `font_layout.resolve_asset_path`, which skips the cwd. Done: core 3.8.0, adopted (ledmatrix-plugins #594); see [Stage 4](#stage-4-the-identical-sweep-done-core-380-adopted) |
 | 5 | Game-over check | `SportsLive._is_game_really_over` (5) | Pure logic, no pixels; one seam, `FINAL_PERIOD`. The pilot for the procedure. Reconciled to one body and promoted as `sports_game_over`; shipped in 3.8.1 and adopted. See [Family 5](#family-5-the-game-over-check-done-core-381-adopted) |
 | 6 | Favourite matching | `_is_favorite_game` (7 across three classes), `_select_games_for_display` (2: nrl), `_select_recent_games_for_display` (3) | Everything that asks "is this a favourite" goes through the 3.5.0 `_favorite_key` seam. Reconciled to one body each and promoted as `sports_favorites`; shipped in 3.8.2 and adopted. See [Family 6](#family-6-favourite-matching-done-core-382-adopted) |
-| 7 | Other-games rotation | `_by_importance`, `_other_games_window`, `_advance_other_games_if_due` (2 each: football), `_rotate_other_games_on_display` (2: ufc), with `_attach_odds_to_rotated_games` (3; ufc had none) | One outlier each; football carried two fixes the other eight lacked. Reconciled to one body each, on a `_rankings_loaded` seam, and promoted as `sports_rotation`; adoption waits for the release that ships it. See [Family 7](#family-7-the-other-games-rotation-core-done-adoption-waits-for-a-release) |
+| 7 | Other-games rotation | `_by_importance`, `_other_games_window`, `_advance_other_games_if_due` (2 each: football), `_rotate_other_games_on_display` (2: ufc), with `_attach_odds_to_rotated_games` (3; ufc had none) | One outlier each; football carried two fixes the other eight lacked. Reconciled to one body each, on a `_rankings_loaded` seam, and promoted as `sports_rotation`; done: core 3.8.4, adopted (ledmatrix-plugins #643). See [Family 7](#family-7-the-other-games-rotation-done-core-384-adopted) |
 | 8 | Rankings | `_fetch_team_rankings` (3), `_choose_poll` (3), `_load_division_team_ids`, `_passes_other_filters`, `_best_rank`, `_is_ranked_game` (2 each: football) | Needs 7; the rank badge and the "ranked only" filter read it |
 | 9 | Live fetch and odds | `_fetch_todays_games` (5), `_fetch_odds` (3), `_attach_odds_to_rotated_games` (3) | The prerequisite for one shared ESPN poller across plugins |
 | 10 | View model | `_extract_game_details_common` (9 of 9) | Every renderer reads it; its keys are additive-only, so reconcile to the superset and leave sport extras in `_extract_game_details` |

src/common/espn_dates.py

@@ -66,19 +66,39 @@ entry older than the reader's own ``max_age``, whoever wrote it and whatever
 ttl they stored with it. Old keys are passed as ``legacy_keys`` and read
 after the canonical one, so an upgrade does not refetch everything at once;
 they can go one release after the one that added this.
+
+Chunks whose days are long over are kept in memory between fetches. The
+scoreboards re-fetch their whole Recent/Upcoming window (14 days back, 7
+ahead) every hour, and since ranges went away that is 22 day requests per
+league. Measured on hdpi on 2026-10-02 (NFL, college football, MLB, college
+baseball, NHL): the hourly window refresh was ~270 of 321 ESPN requests and
+~21 of 24.6MB in the hour, and the 12 days that ended three or more days ago
+were 68% of those bytes (6.9 of 10.2MB per copy of the five windows). A
+settled chunk is answered from memory for ``SETTLED_CHUNK_TTL_SECONDS``,
+stored as zlib-compressed JSON (~13x smaller than the body, and far smaller
+than the parsed objects), so the hourly refresh only goes to ESPN for the
+days that can still change.
 """
 
 import contextvars
+import json
 import logging
 import math
 import re
 import threading
 import time
+import zlib
+from collections import OrderedDict
 from concurrent.futures import ThreadPoolExecutor
-from datetime import date, datetime, timedelta, tzinfo
+from datetime import date, datetime, timedelta, timezone, tzinfo
 from functools import partial
 from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, cast
 
+try:
+    import orjson
+except ImportError:  # optional; the stdlib parser gives the same objects
+    orjson = None
+
 try:
     from src.common.json_body import response_json
 except ImportError:
@@ -162,12 +182,38 @@ _range_lock = threading.Lock()
 # answered -- to learn what every start learns.
 _ranges_rejected_until = time.monotonic() + RANGE_RETRY_SECONDS
 
+# A chunk is "settled" once its last day is this many UTC days back. ESPN
+# files games under the US Eastern date, and a late West-coast game ends after
+# midnight UTC; three days leaves a full day of margin past both, so nothing
+# still being played, finalised or rescheduled is ever served from memory.
+SETTLED_AFTER_DAYS = 3
+
+# How long a settled chunk is trusted. A day's finals do not change, but a
+# rare correction (or an empty answer during an ESPN outage) should not live
+# forever: once a day is plenty, and still skips 23 of every 24 hourly asks.
+SETTLED_CHUNK_TTL_SECONDS = 24 * 60 * 60
+
+# Bounds on the settled-chunk memory. A settled day measured 90KB (NHL) to
+# 990KB (a college-football Saturday) of JSON and 9-74KB compressed; the five
+# windows on hdpi need 60 entries and ~0.55MB. The caps only matter for a
+# board fetching whole past seasons.
+SETTLED_CACHE_MAX_ENTRIES = 512
+SETTLED_CACHE_MAX_BYTES = 8 * 1024 * 1024
+
+_settled_lock = threading.Lock()
+# key -> (stored_at monotonic, compressed JSON)
+_settled_chunks: "OrderedDict[Any, Tuple[float, bytes]]" = OrderedDict()
+_settled_bytes = 0
+
 __all__ = [
     "ESPN_MAX_LIMIT",
     "ESPN_CHUNK_WORKERS",
     "ESPN_MONTH_COVER_MIN_DAYS",
     "RANGE_RETRY_SECONDS",
+    "SETTLED_AFTER_DAYS",
+    "SETTLED_CHUNK_TTL_SECONDS",
     "clamp_espn_limit",
+    "clear_settled_chunk_cache",
     "parse_espn_date_range",
     "espn_date_chunks",
     "espn_request_chunks",
@@ -260,6 +306,88 @@ def _days_of_month(chunk: str) -> List[str]:
     return days
 
 
+def _utc_today() -> date:
+    return datetime.now(timezone.utc).date()
+
+
+def _chunk_last_day(chunk: str) -> Optional[date]:
+    try:
+        if len(chunk) == 8:
+            return date(int(chunk[:4]), int(chunk[4:6]), int(chunk[6:]))
+        if len(chunk) == 6:
+            first = date(int(chunk[:4]), int(chunk[4:6]), 1)
+            return _first_of_next_month(first) - timedelta(days=1)
+    except ValueError:
+        pass
+    return None
+
+
+def _settled_key(url: str, params: Dict[str, Any], chunk: str) -> Optional[Any]:
+    """Memory key for a chunk that can no longer change, else None."""
+    last_day = _chunk_last_day(chunk)
+    if last_day is None:
+        return None
+    if last_day > _utc_today() - timedelta(days=SETTLED_AFTER_DAYS):
+        return None
+    # dates is the chunk itself and limit is always ESPN_MAX_LIMIT here;
+    # anything else (groups=80 for FBS, a team filter) changes the answer.
+    rest = tuple(sorted(
+        (str(k), str(v)) for k, v in params.items() if k not in ("dates", "limit")
+    ))
+    return (url, rest, chunk)
+
+
+def _settled_get(key: Any) -> Optional[Dict[str, Any]]:
+    with _settled_lock:
+        entry = _settled_chunks.get(key)
+        if entry is None:
+            return None
+        if time.monotonic() - entry[0] > SETTLED_CHUNK_TTL_SECONDS:
+            _settled_drop(key)
+            return None
+        _settled_chunks.move_to_end(key)
+        blob = entry[1]
+    # Decompress and parse outside the lock: every hit gets its own objects,
+    # so a caller mutating its payload cannot reach another caller's.
+    body = zlib.decompress(blob)
+    return cast(Dict[str, Any], orjson.loads(body) if orjson else json.loads(body))
+
+
+def _settled_drop(key: Any) -> None:
+    """Remove one entry. Caller holds _settled_lock."""
+    global _settled_bytes
+    entry = _settled_chunks.pop(key, None)
+    if entry is not None:
+        _settled_bytes -= len(entry[1])
+
+
+def _settled_put(key: Any, response: Any, payload: Dict[str, Any]) -> None:
+    global _settled_bytes
+    body = getattr(response, "content", None)
+    if not isinstance(body, (bytes, bytearray)):
+        body = json.dumps(payload).encode("utf-8")
+    blob = zlib.compress(bytes(body), 6)
+    if len(blob) > SETTLED_CACHE_MAX_BYTES:
+        return
+    with _settled_lock:
+        _settled_drop(key)
+        _settled_chunks[key] = (time.monotonic(), blob)
+        _settled_bytes += len(blob)
+        while _settled_chunks and (
+            len(_settled_chunks) > SETTLED_CACHE_MAX_ENTRIES
+            or _settled_bytes > SETTLED_CACHE_MAX_BYTES
+        ):
+            _settled_drop(next(iter(_settled_chunks)))
+
+
+def clear_settled_chunk_cache() -> None:
+    """Forget every remembered settled chunk (tests, or a manual refresh)."""
+    global _settled_bytes
+    with _settled_lock:
+        _settled_chunks.clear()
+        _settled_bytes = 0
+
+
 def espn_date_chunks(start: date, end: date) -> List[str]:
     """Cover ``[start, end]`` inclusive with ``dates=`` values ESPN accepts.
 
@@ -391,6 +519,10 @@ def _fetch_one_chunk(
     One bad chunk must not sink the rest of the season, so every error is
     logged and swallowed here rather than raised to the gather below.
 
+    A chunk whose days are settled (see ``SETTLED_AFTER_DAYS``) is answered
+    from memory when it was fetched in the last day. What is remembered is
+    the month as ESPN sent it, so each window still trims it to its own days.
+
     A month that comes back at the cap is truncated: it returns ``_CAPPED``,
     its payload dropped here before it is ever held beside the others. A
     month in ``trims`` loses its events outside the days given there.
@@ -398,24 +530,34 @@ def _fetch_one_chunk(
     The request holds one of the process-wide ``_chunk_slots`` while it runs.
     """
     try:
-        with _chunk_slots:
+        settled = _settled_key(url, params, chunk)
-            response = fetch_get(
+        payload = _settled_get(settled) if settled is not None else None
-                session,
+        if payload is None:
-                url,
+            with _chunk_slots:
-                params=dict(params, dates=chunk, limit=ESPN_MAX_LIMIT),
+                response = fetch_get(
-                headers=headers,
+                    session,
-                timeout=timeout,
+                    url,
-                **_memo_kwargs(cache_max_age),
+                    params=dict(params, dates=chunk, limit=ESPN_MAX_LIMIT),
-            )
+                    headers=headers,
-            response.raise_for_status()
+                    timeout=timeout,
-            payload = response_json(response)
+                    **_memo_kwargs(cache_max_age),
+                )
+                response.raise_for_status()
+                payload = response_json(response)
+            if len(chunk) == 6 and isinstance(payload, dict):
+                if len(payload.get("events") or []) >= ESPN_MAX_LIMIT:
+                    return _CAPPED
+            if settled is not None and isinstance(payload, dict):
+                events = payload.get("events")
+                # A capped month never gets here; a day at the cap is
+                # truncated too, and remembering it would only cost memory.
+                if isinstance(events, list) and len(events) < ESPN_MAX_LIMIT:
+                    _settled_put(settled, response, payload)
     except Exception as exc:  # noqa: BLE001 - see docstring
         if logger:
             logger.warning("ESPN chunk %s failed, skipping it: %s", chunk, exc)
         return None
     if len(chunk) == 6 and isinstance(payload, dict):
-        if len(payload.get("events") or []) >= ESPN_MAX_LIMIT:
-            return _CAPPED
         trim = (trims or {}).get(chunk)
         if trim is not None:
             payload = _trim_to_days(payload, *trim)

test/conftest.py

@@ -343,6 +343,24 @@ def _hermetic_unit_refresh(monkeypatch, tmp_path_factory):
     monkeypatch.setattr(unit_refresh, 'SYSTEMD_DIR', str(tmp_path_factory.getbasetemp() / 'no-systemd'))
 
 
+@pytest.fixture(autouse=True)
+def _forget_settled_espn_chunks(monkeypatch):
+    """src.common.espn_dates remembers past days process-wide; tests fake
+    different answers for the same dates, so none may inherit another's.
+
+    "Today" is also pinned to 2000-01-01, so no date a test uses counts as
+    settled unless the test says so (by pinning _utc_today itself). Without
+    that, a test asking for last month twice passes while that month is
+    recent and fails once it is three days old: the second ask is answered
+    from memory."""
+    from datetime import date
+    from src.common import espn_dates
+    monkeypatch.setattr(espn_dates, "_utc_today", lambda: date(2000, 1, 1))
+    espn_dates.clear_settled_chunk_cache()
+    yield
+    espn_dates.clear_settled_chunk_cache()
+
+
 @pytest.fixture(autouse=True)
 def reset_logging():
     """Reset logging configuration before each test."""

test/test_espn_dates.py

@@ -39,6 +39,14 @@ def forget_rejected_ranges(monkeypatch):
     monkeypatch.setattr(espn_dates, "_ranges_rejected_until", 0.0)
 
 
+@pytest.fixture(autouse=True)
+def nothing_is_settled_yet(monkeypatch):
+    """Pin "today" before every date these tests use, so the settled-chunk
+    memory stays out of tests that are not about it whatever the real date.
+    TestSettledChunkCache moves it forward."""
+    monkeypatch.setattr(espn_dates, "_utc_today", lambda: date(2000, 1, 1))
+
+
 class FakeResponse:
     def __init__(self, status_code=200, payload=None):
         self.status_code = status_code
@@ -491,6 +499,100 @@ class TestConcurrency:
         assert live["peak"] > 1, "chunks should actually overlap"
 
 
+class TestSettledChunkCache:
+    """Days that ended three or more days ago are fetched once a day, not hourly.
+
+    The scoreboards re-fetch a 22-day window every hour; on hdpi (2026-10-02)
+    the 12 settled days were 68% of that window's bytes.
+    """
+
+    TODAY = date(2026, 10, 2)
+    # The scoreboards' default window on that day: 14 back, 7 ahead.
+    WINDOW = "20260918-20261009"
+
+    @pytest.fixture(autouse=True)
+    def frozen_today(self, monkeypatch):
+        monkeypatch.setattr(espn_dates, "_utc_today", lambda: self.TODAY)
+        # These count day requests; whole edge months are covered below.
+        monkeypatch.setattr(espn_dates, "ESPN_MONTH_COVER_MIN_DAYS", 0)
+
+    def _events(self):
+        days = [(9, d) for d in range(18, 31)] + [(10, d) for d in range(1, 10)]
+        return {"2026%02d%02d" % (m, d): [{"id": f"{m}-{d}"}] for m, d in days}
+
+    def test_the_second_refresh_only_asks_for_unsettled_days(self):
+        session = FakeSession(self._events())
+        first = fetch_espn_scoreboard(session, URL, params={"dates": self.WINDOW})
+        session.calls.clear()
+
+        second = fetch_espn_scoreboard(session, URL, params={"dates": self.WINDOW})
+
+        asked = sorted(call["dates"] for call in session.calls)
+        # Sep 29 is the last settled day (today minus three).
+        assert asked == ["20260930"] + ["202610%02d" % d for d in range(1, 10)]
+        assert second["events"] == first["events"]  # same events, same order
+        assert len(second["events"]) == 22
+
+    def test_a_hit_is_a_fresh_copy(self):
+        session = FakeSession(self._events())
+        fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        hit = fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        hit["events"][0]["id"] = "mutated"
+        again = fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        assert again["events"][0]["id"] == "9-18"
+
+    def test_other_params_are_part_of_the_key(self):
+        session = FakeSession(self._events())
+        fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW, "groups": 80})
+        session.calls.clear()
+        fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        assert len(session.calls) == 22  # a different question, nothing reused
+
+    def test_entries_expire_after_a_day(self, monkeypatch):
+        clock = [1000.0]
+        monkeypatch.setattr(espn_dates.time, "monotonic", lambda: clock[0])
+        session = FakeSession(self._events())
+        fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        clock[0] += espn_dates.SETTLED_CHUNK_TTL_SECONDS + 1
+        session.calls.clear()
+        fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        assert len(session.calls) == 22
+
+    def test_failed_chunks_are_not_remembered(self):
+        session = FakeSession(self._events(), fail_chunks={"20260920"})
+        fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        session.fail_chunks.clear()
+        session.calls.clear()
+        data = fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        assert "20260920" in [call["dates"] for call in session.calls]
+        assert len(data["events"]) == 22
+
+    def test_a_capped_month_is_not_remembered_but_its_days_are(self):
+        full = [{"id": f"x{i}"} for i in range(ESPN_MAX_LIMIT)]
+        session = FakeSession({"202608": full, "20260801": [{"id": "d1"}]})
+        fetch_espn_date_chunks(session, URL, params={"dates": "20260801-20260831"})
+        session.calls.clear()
+        data = fetch_espn_date_chunks(session, URL, params={"dates": "20260801-20260831"})
+        assert [call["dates"] for call in session.calls] == ["202608"]
+        assert [event["id"] for event in data["events"]] == ["d1"]
+
+    def test_memory_is_bounded(self, monkeypatch):
+        monkeypatch.setattr(espn_dates, "SETTLED_CACHE_MAX_ENTRIES", 5)
+        session = FakeSession(self._events())
+        fetch_espn_date_chunks(session, URL, params={"dates": self.WINDOW})
+        assert len(espn_dates._settled_chunks) == 5
+        assert espn_dates._settled_bytes == sum(
+            len(blob) for _, blob in espn_dates._settled_chunks.values())
+
+    def test_single_day_requests_are_untouched(self):
+        # The live path asks for today (or one day) as a plain request; that
+        # never goes through chunks or the memory.
+        session = FakeSession(self._events())
+        for _ in range(2):
+            fetch_espn_scoreboard(session, URL, params={"dates": "20260918"})
+        assert len(session.calls) == 2
+
+
 class TestEdgeMonths:
     """A window's partial edge months are asked whole and trimmed.
 
@@ -573,6 +675,25 @@ class TestEdgeMonths:
         # An event with no readable date is kept, never dropped on a guess.
         assert [e["id"] for e in data["events"]] == ["first", "late", "oct1", "last", "undated"]
 
+    def test_a_remembered_month_is_trimmed_per_window(self, monkeypatch):
+        """The settled-chunk memory holds the month whole; each window trims it."""
+        monkeypatch.setattr(espn_dates, "_utc_today", lambda: date(2026, 12, 20))
+        espn_dates.clear_settled_chunk_cache()
+        november = [
+            {"id": "early", "date": "2026-11-02T18:00Z"},
+            {"id": "late", "date": "2026-11-25T18:00Z"},
+        ]
+        session = FakeSession({"202611": november})
+
+        late = fetch_espn_date_chunks(session, URL, params={"dates": "20261120-20261215"})
+        session.calls.clear()
+        early = fetch_espn_date_chunks(session, URL, params={"dates": "20261101-20261110"})
+
+        assert [e["id"] for e in late["events"]] == ["late"]
+        assert [e["id"] for e in early["events"]] == ["early"]
+        assert not any(call["dates"] == "202611" for call in session.calls)
+        espn_dates.clear_settled_chunk_cache()
+
     def test_eastern_standard_time_is_honoured_after_the_clocks_change(self):
         # 2026-11-01 ends daylight saving: Eastern is UTC-5 from then on.
         november = [