feat(fetch): shared fetch service, stage 1 (pooling, merging, host budgets, counters) (#702)

Core's own HTTP fetch paths (APIHelper, fetch_espn_scoreboard and its date chunks, BackgroundDataService, BaseOddsManager.get_odds) go through one service in src/common/fetch_service.py: shared connection pools per retry policy, merged identical in-flight GETs, per-host token-bucket budgets (fetch_service.rate_limits), and per-plugin request counters published to GET /api/v3/plugins/fetch-stats. Return values, exceptions, cache keys, TTLs and retry policies are unchanged. Core-internal in this release; plugins should not import it directly yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Chuck
2026-10-01 15:38:11 -04:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 9edeb6da14
commit 4be53d048b
19 changed files with 2407 additions and 31 deletions
+1
View File
@@ -48,6 +48,7 @@ each other. They share three things:
| Plugin errors | cache `plugin_error_snapshot` | display: `ErrorSnapshotPublisher` ([`src/error_aggregator.py`](../src/error_aggregator.py)) | web: `read_error_report()` for `/api/v3/errors/*` |
| Error clear | cache `plugin_error_clear_request` | web | display |
| Font usage | cache `font_usage_snapshot` | display: `FontUsagePublisher` ([`src/font_usage.py`](../src/font_usage.py)) | web: Fonts tab |
| Fetch statistics (requests per plugin and host) | cache `fetch_stats_snapshot` | display: `FetchStatsPublisher` ([`src/common/fetch_service.py`](../src/common/fetch_service.py)), at most once a minute on change | web: `read_fetch_stats()` for `/api/v3/plugins/fetch-stats` |
| Plugin health | cache `plugin_health:<id>` | display (web writes on reset) | web: `/api/v3/plugins/health` |
| Plugin runtime (loaded, state, last error, version) | cache `plugin_runtime_snapshot` | display: `PluginRuntimePublisher` ([`src/plugin_system/plugin_runtime.py`](../src/plugin_system/plugin_runtime.py)) | web: `read_plugin_runtime()` for `/api/v3/plugins/installed`, `/plugins/state`, reconciliation |
| Preview frame | `/tmp/led_matrix_preview.png` | display: `DisplayManager`, gated by [`snapshot_policy`](../src/common/snapshot_policy.py) | web: display SSE stream, `/api/v3/health` (file age) |
+58
View File
@@ -14,6 +14,7 @@ Complete API reference for plugin developers. This document describes all method
- [Display Manager](#display-manager)
- [Cache Manager](#cache-manager)
- [Plugin Manager](#plugin-manager)
- [Fetching data](#fetching-data)
- [Deprecated APIs](#deprecated-apis)
---
@@ -1030,6 +1031,63 @@ if weather is not None and weather.enabled:
---
## Fetching data
Use the core helpers for HTTP rather than a `requests.Session` of your own:
`APIHelper` (`from src.common import APIHelper`) for JSON APIs, and
`fetch_espn_scoreboard()` (`src.common.espn_dates`) or
`BackgroundDataService` for ESPN scoreboards. Since the release after 3.7.0
these go through the core **fetch service** (`src/common/fetch_service.py`),
so a plugin that uses them gets the following with no code change. Return
values, exceptions and retries are what they were.
- **Shared connections.** Core sessions with the same retry policy share one
connection pool per host, instead of one pool per helper.
- **Merged requests.** Identical GETs in flight at the same time (same URL
and query, headers, timeout and retry policy) go to the network once, and
every caller gets its own copy of the response, or the same exception.
- **Host budgets.** A host can have a token-bucket budget. A request past it
waits for a token, but never longer than `max_wait_seconds` (2 s by
default). Only ESPN hosts have one by default (20 requests a second, burst
200), which normal use never reaches.
- **Conditional GET.** When a server sends `ETag` or `Last-Modified`, the
next identical request revalidates, and a `304 Not Modified` comes back to
your code as the original `200` with its body. ESPN currently sends
neither, so this does nothing there.
- **Counters.** Requests, merged requests, bytes, 304s, errors and time spent
waiting are counted per plugin and per host, and published for the web UI
at `GET /api/v3/plugins/fetch-stats` (see
[REST_API_REFERENCE.md](REST_API_REFERENCE.md#get-fetch-statistics)). A
request is counted against your plugin when it runs inside your
`update()`/`display()`, your constructor or `on_enable()`, or anywhere in
code under your plugin's directory, including threads you start.
What is not covered yet: requests a plugin makes with its own `requests.get()`
or `Session.get()` calls. They work as before but are invisible to the
budgets and counters.
The settings live in `config.json` under `fetch_service`, read when the
display starts and on a config reload:
```json
"fetch_service": {
"enabled": true,
"max_wait_seconds": 2,
"rate_limits": {
"*.espn.com": {"per_second": 20, "burst": 200},
"api.example.com": {"per_second": 1, "burst": 5}
}
}
```
`rate_limits` keys are a host or a `*.domain` pattern (which also matches
the bare domain); `"per_second": 0` removes a budget. `"enabled": false`
turns the whole service into a plain `session.get()`. Two further switches,
`"single_flight": false` and `"conditional_get": false`, turn off merging and
revalidation.
---
## Best Practices
### Caching
+57
View File
@@ -980,6 +980,63 @@ Metrics for one plugin; `data` has the same fields as one entry above.
Reset metrics for a plugin.
### Get Fetch Statistics
**GET** `/api/v3/plugins/fetch-stats`
Network requests made through the core fetch service
(`src/common/fetch_service.py`), per plugin and per host, cumulative since
the display started. Read-only. The display publishes the counters at most
once a minute when they change (every 10 minutes otherwise), so they can be
up to a minute old. Requests a plugin makes with its own `requests` calls,
outside `APIHelper`, `espn_dates`, `BackgroundDataService` and
`BaseOddsManager`, are not counted yet.
`data.status` is `live`, `stale` (no publish for longer than
`stale_after`), `stopped` (the display exited; the last counters are kept)
or `unknown` (nothing published; `data.data` is `null`).
**Response**:
```json
{
"status": "success",
"data": {
"status": "live",
"age_seconds": 12.4,
"data": {
"schema": 1,
"running": true,
"published_at": 1790000000.0,
"stale_after": 720.0,
"since": 1789990000.0,
"totals": {"requests": 412, "merged": 3, "not_modified": 0,
"errors": 1, "http_errors": 2, "retries": 0,
"throttled": 0, "overruns": 0, "bytes": 18234011,
"wait_seconds": 0.0},
"plugins": {
"football-scoreboard": {"requests": 240, "merged": 2, "bytes": 9120330,
"hosts": {"site.api.espn.com": 180,
"sports.core.api.espn.com": 62},
"...": "the other counters, as in totals"}
},
"hosts": {
"site.api.espn.com": {"requests": 301, "...": "as in totals"}
},
"validators": {"entries": 0, "bytes": 0},
"config": {"enabled": true, "single_flight": true,
"conditional_get": true, "max_wait_seconds": 2.0,
"rate_limits": {"*.espn.com": {"per_second": 20.0, "burst": 200.0}}}
}
}
}
```
`requests` counts round trips sent (retries inside the HTTP adapter are in
`retries`), `merged` requests answered by an identical one already in
flight, `not_modified` 304s served from the stored body, `errors` transport
failures and `http_errors` responses with status 400 or above. `bytes` is the
decoded body size. `core` is everything no plugin made.
### Get/Set Plugin Limits
**GET** `/api/v3/plugins/limits/<plugin_id>`