mirror of
https://github.com/ChuckBuilds/LEDMatrix.git
synced 2026-10-04 14:25:08 +00:00
feat(fetch): shared fetch service, stage 1 (pooling, merging, host budgets, counters) (#702)
Core's own HTTP fetch paths (APIHelper, fetch_espn_scoreboard and its date chunks, BackgroundDataService, BaseOddsManager.get_odds) go through one service in src/common/fetch_service.py: shared connection pools per retry policy, merged identical in-flight GETs, per-host token-bucket budgets (fetch_service.rate_limits), and per-plugin request counters published to GET /api/v3/plugins/fetch-stats. Return values, exceptions, cache keys, TTLs and retry policies are unchanged. Core-internal in this release; plugins should not import it directly yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -48,6 +48,7 @@ each other. They share three things:
|
||||
| Plugin errors | cache `plugin_error_snapshot` | display: `ErrorSnapshotPublisher` ([`src/error_aggregator.py`](../src/error_aggregator.py)) | web: `read_error_report()` for `/api/v3/errors/*` |
|
||||
| Error clear | cache `plugin_error_clear_request` | web | display |
|
||||
| Font usage | cache `font_usage_snapshot` | display: `FontUsagePublisher` ([`src/font_usage.py`](../src/font_usage.py)) | web: Fonts tab |
|
||||
| Fetch statistics (requests per plugin and host) | cache `fetch_stats_snapshot` | display: `FetchStatsPublisher` ([`src/common/fetch_service.py`](../src/common/fetch_service.py)), at most once a minute on change | web: `read_fetch_stats()` for `/api/v3/plugins/fetch-stats` |
|
||||
| Plugin health | cache `plugin_health:<id>` | display (web writes on reset) | web: `/api/v3/plugins/health` |
|
||||
| Plugin runtime (loaded, state, last error, version) | cache `plugin_runtime_snapshot` | display: `PluginRuntimePublisher` ([`src/plugin_system/plugin_runtime.py`](../src/plugin_system/plugin_runtime.py)) | web: `read_plugin_runtime()` for `/api/v3/plugins/installed`, `/plugins/state`, reconciliation |
|
||||
| Preview frame | `/tmp/led_matrix_preview.png` | display: `DisplayManager`, gated by [`snapshot_policy`](../src/common/snapshot_policy.py) | web: display SSE stream, `/api/v3/health` (file age) |
|
||||
|
||||
@@ -14,6 +14,7 @@ Complete API reference for plugin developers. This document describes all method
|
||||
- [Display Manager](#display-manager)
|
||||
- [Cache Manager](#cache-manager)
|
||||
- [Plugin Manager](#plugin-manager)
|
||||
- [Fetching data](#fetching-data)
|
||||
- [Deprecated APIs](#deprecated-apis)
|
||||
|
||||
---
|
||||
@@ -1030,6 +1031,63 @@ if weather is not None and weather.enabled:
|
||||
|
||||
---
|
||||
|
||||
## Fetching data
|
||||
|
||||
Use the core helpers for HTTP rather than a `requests.Session` of your own:
|
||||
`APIHelper` (`from src.common import APIHelper`) for JSON APIs, and
|
||||
`fetch_espn_scoreboard()` (`src.common.espn_dates`) or
|
||||
`BackgroundDataService` for ESPN scoreboards. Since the release after 3.7.0
|
||||
these go through the core **fetch service** (`src/common/fetch_service.py`),
|
||||
so a plugin that uses them gets the following with no code change. Return
|
||||
values, exceptions and retries are what they were.
|
||||
|
||||
- **Shared connections.** Core sessions with the same retry policy share one
|
||||
connection pool per host, instead of one pool per helper.
|
||||
- **Merged requests.** Identical GETs in flight at the same time (same URL
|
||||
and query, headers, timeout and retry policy) go to the network once, and
|
||||
every caller gets its own copy of the response, or the same exception.
|
||||
- **Host budgets.** A host can have a token-bucket budget. A request past it
|
||||
waits for a token, but never longer than `max_wait_seconds` (2 s by
|
||||
default). Only ESPN hosts have one by default (20 requests a second, burst
|
||||
200), which normal use never reaches.
|
||||
- **Conditional GET.** When a server sends `ETag` or `Last-Modified`, the
|
||||
next identical request revalidates, and a `304 Not Modified` comes back to
|
||||
your code as the original `200` with its body. ESPN currently sends
|
||||
neither, so this does nothing there.
|
||||
- **Counters.** Requests, merged requests, bytes, 304s, errors and time spent
|
||||
waiting are counted per plugin and per host, and published for the web UI
|
||||
at `GET /api/v3/plugins/fetch-stats` (see
|
||||
[REST_API_REFERENCE.md](REST_API_REFERENCE.md#get-fetch-statistics)). A
|
||||
request is counted against your plugin when it runs inside your
|
||||
`update()`/`display()`, your constructor or `on_enable()`, or anywhere in
|
||||
code under your plugin's directory, including threads you start.
|
||||
|
||||
What is not covered yet: requests a plugin makes with its own `requests.get()`
|
||||
or `Session.get()` calls. They work as before but are invisible to the
|
||||
budgets and counters.
|
||||
|
||||
The settings live in `config.json` under `fetch_service`, read when the
|
||||
display starts and on a config reload:
|
||||
|
||||
```json
|
||||
"fetch_service": {
|
||||
"enabled": true,
|
||||
"max_wait_seconds": 2,
|
||||
"rate_limits": {
|
||||
"*.espn.com": {"per_second": 20, "burst": 200},
|
||||
"api.example.com": {"per_second": 1, "burst": 5}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`rate_limits` keys are a host or a `*.domain` pattern (which also matches
|
||||
the bare domain); `"per_second": 0` removes a budget. `"enabled": false`
|
||||
turns the whole service into a plain `session.get()`. Two further switches,
|
||||
`"single_flight": false` and `"conditional_get": false`, turn off merging and
|
||||
revalidation.
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Caching
|
||||
|
||||
@@ -980,6 +980,63 @@ Metrics for one plugin; `data` has the same fields as one entry above.
|
||||
|
||||
Reset metrics for a plugin.
|
||||
|
||||
### Get Fetch Statistics
|
||||
|
||||
**GET** `/api/v3/plugins/fetch-stats`
|
||||
|
||||
Network requests made through the core fetch service
|
||||
(`src/common/fetch_service.py`), per plugin and per host, cumulative since
|
||||
the display started. Read-only. The display publishes the counters at most
|
||||
once a minute when they change (every 10 minutes otherwise), so they can be
|
||||
up to a minute old. Requests a plugin makes with its own `requests` calls,
|
||||
outside `APIHelper`, `espn_dates`, `BackgroundDataService` and
|
||||
`BaseOddsManager`, are not counted yet.
|
||||
|
||||
`data.status` is `live`, `stale` (no publish for longer than
|
||||
`stale_after`), `stopped` (the display exited; the last counters are kept)
|
||||
or `unknown` (nothing published; `data.data` is `null`).
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"status": "live",
|
||||
"age_seconds": 12.4,
|
||||
"data": {
|
||||
"schema": 1,
|
||||
"running": true,
|
||||
"published_at": 1790000000.0,
|
||||
"stale_after": 720.0,
|
||||
"since": 1789990000.0,
|
||||
"totals": {"requests": 412, "merged": 3, "not_modified": 0,
|
||||
"errors": 1, "http_errors": 2, "retries": 0,
|
||||
"throttled": 0, "overruns": 0, "bytes": 18234011,
|
||||
"wait_seconds": 0.0},
|
||||
"plugins": {
|
||||
"football-scoreboard": {"requests": 240, "merged": 2, "bytes": 9120330,
|
||||
"hosts": {"site.api.espn.com": 180,
|
||||
"sports.core.api.espn.com": 62},
|
||||
"...": "the other counters, as in totals"}
|
||||
},
|
||||
"hosts": {
|
||||
"site.api.espn.com": {"requests": 301, "...": "as in totals"}
|
||||
},
|
||||
"validators": {"entries": 0, "bytes": 0},
|
||||
"config": {"enabled": true, "single_flight": true,
|
||||
"conditional_get": true, "max_wait_seconds": 2.0,
|
||||
"rate_limits": {"*.espn.com": {"per_second": 20.0, "burst": 200.0}}}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`requests` counts round trips sent (retries inside the HTTP adapter are in
|
||||
`retries`), `merged` requests answered by an identical one already in
|
||||
flight, `not_modified` 304s served from the stored body, `errors` transport
|
||||
failures and `http_errors` responses with status 400 or above. `bytes` is the
|
||||
decoded body size. `core` is everything no plugin made.
|
||||
|
||||
### Get/Set Plugin Limits
|
||||
|
||||
**GET** `/api/v3/plugins/limits/<plugin_id>`
|
||||
|
||||
Reference in New Issue
Block a user