Compare commits

..
7 Commits
Author SHA1 Message Date
ChuckandClaude Sonnet 5.5 e51ee0070e chore: prepare the 3.9.0 release (#803)
Bumps src.__version__ to 3.9.0 and turns Unreleased into ## 3.9.0:
per-plugin schedules (#799), the removal of the cache-key mailboxes (#773,
breaking), the Vegas pause fix (#798) and the Update Code index.lock fix
(#801, #802).

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-29 14:19:15 -04:00
ChuckandClaude Sonnet 5.5 237d611b89 fix(update): stop a timed-out git from leaving .git/index.lock behind (#802)
* fix(update): stop a timed-out git from leaving .git/index.lock behind

subprocess.run's timeout SIGKILLs only the direct git child, so the lock
is left behind and git's helpers are orphaned. On a slow SD card the same
pull then times out on every retry and recreates the lock, so removing it
by hand does not help.

- run_git starts git in its own session, stops the group with SIGTERM
  before SIGKILL, and clears the lock if nothing owns it. Update Code's
  pull, checkout, reset and stash go through it, with a 600s pull timeout.
- Read-only calls (describe --dirty, status, the update-check fetch) set
  GIT_OPTIONAL_LOCKS=0 so a timed-out poll cannot take the lock.
- A timed-out pull returns a clear error; a rebase left by an interrupted
  update is aborted before pulling.
- The manual-fix hint says to stop the services and kill running git.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(update): address review of the index.lock fix

- The automatic update's status check now sets GIT_OPTIONAL_LOCKS=0
  whichever runner it is given, not only the default one.
- Update Code no longer runs `git rebase --abort` on any rebase dir: a
  rebase in progress (detached HEAD) holds the user's conflict work and
  blocks the update; an abandoned one is dropped with `rebase --quit`,
  as the automatic update does, and a failure to clear it stops the update.
- run_git waits out the lock-freshness window after stopping a timed-out
  git, so a lock created just before the timeout is not left behind.
- Mark the Popen/subprocess use with the repo's nosec comments.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* chore(update): mark run_git's CompletedProcess as a result object, not a subprocess call

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-29 14:13:09 -04:00
ChuckandClaude Opus 5.5 fa51ebab5f fix(update): clear a stale .git/index.lock before Update Code (#801)
* fix(update): clear a stale .git/index.lock before Update Code

A git that subprocess.run's timeout kills (SIGKILL) cannot remove
.git/index.lock, so every later update failed with "Unable to create
.../.git/index.lock: File exists" until the file was removed by hand.

Update Code and the automatic update's rollback now remove the lock when
no git process is running in the checkout and the file is more than a
few seconds old. While another git is running, Update Code says so
instead of failing; when the lock could not be checked, the error names
the rm command that fixes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvFtgthKUNEPhJu6RuX9Sk

* fix(update): count a git that names the checkout in its arguments as running

git --git-dir/--work-tree can work on the checkout from any working
directory, so the cwd check alone could take such a git's live lock for
a stale one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvFtgthKUNEPhJu6RuX9Sk

* fix(update): resolve a git's relative path arguments against its cwd

`git -C LEDMatrix ...` or a relative --git-dir started outside the
checkout works on it too; resolve each argument against the process's
working directory before deciding a lock is stale, and count unreadable
arguments as naming the checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvFtgthKUNEPhJu6RuX9Sk

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-10-26 20:13:40 -04:00
ChuckandClaude Opus 5.5 4c883ec2c2 test(ipc): wait for the state subscription before publishing in the end-to-end test (#800)
The first current-status read can be answered by a one-shot state.get
while the subscription's own snapshot (taken before the publish) is still
in flight. The poll loop then saw 'stocks' from a one-shot, broke, and the
next read got the subscription's older 'weather' snapshot, before its
event for the change arrived (CI run 38774723656, 3.13, failed in 0.7 s).

Wait until the subscription holds a snapshot before publishing, then
until the change reaches it, and only then assert on the route.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 11:02:10 -04:00
ChuckandClaude Opus 5.5 1fe7965f5c feat(schedule): per-plugin schedules and hide-while-live (#799)
* feat(schedule): per-plugin schedules and hide-while-live

Users asked to turn off news and stocks on gameday so only scores show.
Each plugin can now have its own schedule (global or per-day window in
which it may show) plus hide_when_live, which hides it whenever any
plugin reports a live game. A hidden plugin is skipped in the rotation,
the live takeover and the Vegas ticker; it stays loaded, and on-demand
still shows it.

- src/plugin_schedule.py: the window rule, now shared with the display
  schedule (_check_schedule calls schedule_state)
- core config key plugin_schedules; GET/POST /api/v3/config/plugin-schedules
- Schedule page: Plugin Schedules card, one picker per enabled plugin

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schedule): tolerate wrong-shaped schedules, quiet skipped modes

- schedule_state treats a non-string mode, non-dict days or a non-dict
  day entry as missing instead of raising in the display loop
- the run loop no longer warns "No plugin manager found" for a mode it
  skips: _plugin_to_draw / _plugin_for_mode already log why (a
  scheduled-off plugin at DEBUG)
- Schedule page DOM test: every form is marked data-reports-result, not
  exactly two (each plugin adds one)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schedule): guard plugin_schedules shape and the cross-thread cache

- a plugin_schedules that is not a dict (list, string) hides nothing
  instead of raising in the display loop
- the hidden-set cache keys on the schedules themselves and is published
  as one tuple, so a Vegas-thread call still holding the pre-reload
  config cannot leave its answer standing after a reload; the separate
  invalidation in _refresh_config_cache is gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schedule): no live hold on a scheduled-off mode; blank an empty rotation

- _still_live ignores a plugin its schedule hides, so a rotation landing
  on a scheduled-off live mode advances instead of holding a mode it skips
- once a whole rotation has had nothing to show (every plugin empty or
  scheduled off), the panel is cleared once instead of leaving the last
  screen up

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schedule): push the empty-rotation blank to the web preview

clear() alone blanked the matrix but left the preview snapshot on the
last frame (seen on ledpi with every plugin scheduled off). Follow it
with update_display(), as the scheduled-off blank does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schedule): write the empty-rotation blank to the preview

The preview throttle skipped the blank frame (written 12 s after the
last snapshot), and write_owed_snapshot() only runs after a display()
call, which an all-empty rotation never makes: on ledpi the preview
kept the startup splash while the panel was dark. Flush the owed
snapshot on each paused empty pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schedule): quiet the once-a-second logs when nothing can show

With every plugin scheduled off (seen on ledpi), the loop retries once
a second and journaled three to four lines each time: Vegas's
"No plugins available" WARNING, the coordinator's "Failed to initialize
stream manager" ERROR, the 0-plugin rotation summary and "Switching to
mode". Vegas now warns once until its list fills again, the coordinator
line is DEBUG (initialize only fails for an empty list), and the
mode-switch line is DEBUG once a whole rotation has been empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(schedule): end the scroll before the empty blank; Vegas sees hidden changes sooner

- the empty-rotation blank is controller-drawn, so it ends the last
  scroll's state first (_end_scroll_before_core_screen), as the
  scheduled-off blank does
- StreamManager.refresh() rebuilds the plugin list when the hidden set
  changed, even inside its 30 s interval, so a newly hidden plugin leaves
  at the next scroll group rather than staying for later ones

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 16:16:47 -04:00
ChuckandClaude Opus 5.5 7c51c93707 fix(vegas): a static pause redraws its plugin for the whole pause (#798)
* fix(vegas): a static pause redraws its plugin for the whole pause

The pause called display(force_clear=True) once and slept out the
duration, so a plugin that rotates its screens or animates (Fantasy
Blitz's playlist) was frozen on its first frame. The pause now calls
display() again at the rotation's rate: once a second, or the high-FPS
interval for a plugin that declares needs_high_fps (or scrolls). Only
the first call clears. A redraw that finds update() holding the plugin
lock is skipped; the pause carries on. Interrupt checks and the
watchdog beat still run every 0.1 s; a raise still ends the pause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(vegas): legacy static-image pauses redraw at high FPS

Match DisplayController._needs_high_fps: a static-image that does not
declare needs_high_fps gets the high-FPS loop for its GIFs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:38:59 -04:00
ChuckandClaude Opus 5.5 397333ae59 feat(ipc)!: remove the cache-key mailboxes, control socket stage 5 (for the release after 3.8.1) (#773)
* feat(ipc)!: remove the cache-key mailboxes (control socket stage 5)

The control socket is now the only way the web interface sends the display a
command. The display stops reading display_on_demand_request and
plugin_error_clear_request, and the web interface stops writing them.

- Display: no mailbox poll (MailboxWatch, the 1 s / 0.25 s cadence,
  _consume_on_demand_request, the deprecation log) and no persisted
  display_on_demand_processed_id guard; the error publisher reads no clear
  request. CacheManager.file_signature and MailboxWatch are removed.
- A write to either retired key is dropped by CacheManager.save_cache and
  logged once per writer, naming the plugin from the call stack (or the
  request's plugin_id), with the API to move to.
- Web: on-demand start with no display listening starts the service (when
  start_service) and sends the request again once the socket answers (45 s,
  10 s for a running service without a socket yet); every other failure is
  a 503 (400 for invalid_args). Stop answers 503 when no display listens,
  unless stop_service. errors/clear answers 503 with a reason-specific
  message instead of writing a request; clear_pending is always false.
  src.ipc.client.should_fall_back is replaced by display_not_listening.
- Kept: display_current_state, display_on_demand_state,
  plugin_runtime_snapshot and the heartbeat (read whenever the socket cannot
  answer), and display_on_demand_config (the display's resume record).

Tests: mailbox-only tests removed (test_on_demand_mailbox.py, the mailbox
cadence, file_signature and MailboxWatch tests); tests that injected
requests through the mailbox now use the socket queue or a plugin's
in-process request. The run-loop harness sends on-demand requests over its
fake control socket, so four golden traces change: on-demand starts and
stops land at the request instant instead of the next 0.25 s mailbox look
(one frame fewer on the screen they end), and in vegas.json within one
frame instead of 263 ms, which shifts the later 1 s-throttled WiFi-notice
check by under a second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: the on-demand routes' other callers ack over the socket, not the mailbox

test_api_v3_bool_coercion, test_api_v3_lazy_plugin_discovery and
test_web_api's stop test checked the mailbox write; they now get an ack
from a mocked control socket. The stop route touches no manager any more,
so it leaves the removed-catch-all sample. A request that was sent is
never 'not listening' (pins the sent check the mutation run let through).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(web): a start that waits for the display answers 202 and is delivered in the background

The start route held a request open for up to 45 s while a cold-started
display loaded its plugins; the MQTT bridge (15 s timeout) and browsers
reported a failure for a request that was then delivered.

Now, when no display is listening, the route starts the service if asked
and answers 202 with status "starting" at once. A single worker in the web
process (web_interface/on_demand_dispatch.py) sends the request until the
display acknowledges it or the wait runs out (45 s cold start, 10 s for a
running service without a socket yet). A newer start supersedes the
pending one; a stop cancels it (and succeeds, with cancelled_request_id,
even with no display listening). The outcome is reported by
/display/on-demand/status (source "web": starting, or error with
start-timeout or the socket's reason, until the display publishes
something newer) and by /display/current-status as on_demand_pending.

Callers: the web UI's on-demand modal and "Preview on display" treat
"starting" as taken (an info toast); the MQTT bridge already treats any
non-error 2xx as success (now pinned by a test).

Tests: the dispatcher (ack, retry then ack, start-timeout, other failures,
superseded, an in-flight ack for a superseded start, stop while pending,
a per-start wait, outcome lifetime); the routes (202, status routes while
pending and after a timeout, a later display state replacing the failure,
stop while pending, a new start superseding); a JS suite for app.js.
Mutation check: 20 mutants on the worker, the routes and app.js, 20 killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a cancelled on-demand start never reaches the display after its replacement

CI caught a race in test_a_new_start_supersedes_the_pending_one: the
dispatcher's worker could read the old start, the route cancel it and send
the new one, and the worker's send of the old one land after it. cancel()
now waits out a send already in flight (the worker holds a send lock while
it reads the pending start and sends it), so whatever the caller sends next
lands after it. Pinned by test_a_cancel_waits_for_a_send_in_flight, which
fails without the wait.

The Linux-only TestRealSocket test for a display that went away now
expects the 202 and the start-timeout that follows, as the route answers
since the background dispatcher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): a delivered on-demand start reads as starting until the display acts on it

On ledpi (three cold starts) the display acknowledged the start as its
socket opened, then took ~5 s to act on it while Vegas built its first
strip; the status routes meanwhile showed the display's own idle state, so
a UI polling every 700 ms flashed idle.

The display's on-demand state now names the request it answers
(request_id). A delivered start keeps reading as status "starting" with
delivered: true, in /display/on-demand/status and as on_demand_pending in
/display/current-status, until the display publishes state for that
request id (a display without the field: any state newer than the
delivery), for at most DELIVERED_SHOWN_SECONDS (30 s). The display's
startup state, which can be published after the acknowledgement, names no
request and does not end it.

Tests: stays starting against the startup idle state (no id, an older id);
the matching active state and the matching error take over; an older
display's newer state takes over; the 30 s cap; the display's state names
its request. Mutation check: 11 mutants, 11 killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(display): on_demand_request_id has a class default for controllers built without __init__

_on_demand_state now publishes it, and test_state_stream_readers builds
controllers with __new__.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: the case-spelling on-demand test patches _send_on_demand now the mailbox is gone

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* docs: blank line before 3.8.7 heading

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:31:35 -04:00
74 changed files with 4016 additions and 2065 deletions
+106
View File
@@ -19,6 +19,112 @@ accepts both, but the store flags the old spelling as deprecated
## Unreleased
## 3.9.0
Per-plugin schedules (#799), the end of the cache-key mailboxes (#773,
**breaking**: see "Removed" below; restart the display and the web interface
together after updating), a Vegas pause fix (#798), and a fix for Update Code
failing with `.git/index.lock: File exists` on a slow Pi (#801, #802).
### Fixed: Update Code left `.git/index.lock` behind
A git that Python's timeout killed could not remove `.git/index.lock`, so
every later update failed with "Unable to create '.../.git/index.lock': File
exists", and removing the file by hand only lasted until the next timeout.
Update Code now runs git in its own process group, stops it with SIGTERM
before SIGKILL and clears the lock it left; pulls get 600 s instead of 60 s;
the read-only version and status checks set `GIT_OPTIONAL_LOCKS=0` so a
timed-out poll cannot take the lock. A rebase in progress (detached HEAD) now
stops Update Code with a message instead of failing inside git; an abandoned
one is quit first. If a device is already stuck on an older release, stop
the services, kill any running git, remove `.git/index.lock` and pull by hand
once. (#801, #802)
### Added: plugin schedules
Schedule → Plugin Schedules sets when each plugin may show, per day or with
one time range, and can hide a plugin whenever any game is live ("no news or
stocks on gameday"). A hidden plugin is skipped in the rotation, the live
takeover and Vegas mode, but stays loaded. New core config key
`plugin_schedules`, new `GET/POST /api/v3/config/plugin-schedules`, new
module `src/plugin_schedule.py` (the display schedule's window rule now
lives there too).
### Removed: the cache-key mailboxes (control socket stage 5) -- breaking
The control socket (`docs/IPC_CONTROL_SOCKET.md`) is now the only way the
web interface sends the display a command. The file mailboxes it fell back
to for one release are gone.
- **`display_on_demand_request` and `plugin_error_clear_request` are no
longer written or read.** The display stops polling the on-demand mailbox
(`MailboxWatch`, `MAILBOX_POLL_INTERVAL_WITH_SOCKET`,
`_consume_on_demand_request` and the persisted
`display_on_demand_processed_id` guard are removed), and the error
publisher stops reading the clear request. `CacheManager.file_signature`,
`src.cache_manager.MailboxWatch`, `src.ipc.client.should_fall_back` and
`src.error_aggregator.ERROR_CLEAR_REQUEST_KEY` are removed.
- **A write to either key is dropped, with one warning per writer.**
`CacheManager.save_cache` (and so `set`) refuses `RETIRED_MAILBOX_KEYS`
and logs `Ignored a write to the retired '<key>' cache key by plugin
'<id>'`, naming the plugin from the call stack or the request. Plugins
must use `BasePlugin.request_on_demand()` / `end_on_demand()` (3.8.1).
In the official monorepo, birdnet-go, mqtt-notifications, on-air and
pomodoro-timer still write the mailbox, but only as their fallback when
those methods are missing or answer `None`.
- **On-demand routes without a listening display.** `POST
/api/v3/display/on-demand/start` with no display listening starts the
service (when `start_service`, the default) and answers **`202`** with
`status: "starting"` at once; a single background worker in the web
process (`web_interface/on_demand_dispatch.py`) sends the request until
the display acknowledges it, for up to 45 s (10 s for a service that is
running but has no socket yet). `GET /display/on-demand/status` reports it
(`starting`; still `starting` with `delivered: true` after the display
acknowledges it, until the display publishes the state for that
`request_id`, now part of its on-demand state, for at most 30 s; then the
display's state, or `error` / `start-timeout`), and
`/display/current-status` adds `on_demand_pending`. A newer start replaces
a pending one and a stop cancels it (`cancelled_request_id`). The web UI
and the MQTT bridge treat `202` as taken. With the service stopped and
`start_service` false it answers `400`. Every other socket failure (`unknown_command`
from an older display, `disabled`/`unsupported`, `busy`, a timeout) is a
`503`. `/stop` answers `503` when no display is listening, unless
`stop_service` stops the service. `transport` is always `"socket"`; the
`"mailbox"` value is gone.
- **`POST /api/v3/errors/clear` without the socket answers `503`** (with
`context.socket_error` and a message saying why) instead of recording a
request. `clear_pending` in the error routes is now always `false`;
`src.error_aggregator.read_error_report()` returns only the snapshot, and
`error_summary_from_report()` / `plugin_health_from_report()` /
`request_error_clear()` lose their clear-request arguments.
- **Kept:** the display still writes `display_current_state`,
`display_on_demand_state`, `plugin_runtime_snapshot` and the heartbeat
file, because the web interface reads them whenever the socket cannot
answer (a stopped or starting display, a web user not yet in the socket's
group, Windows), and `display_on_demand_config`, its own record for
resuming a session after a restart.
- **Windows and `LEDMATRIX_CONTROL_SOCKET=off`:** with no socket, the web
interface can no longer start or stop on-demand sessions or clear errors
on a running display (the mailbox used to carry them).
### Fixes
- Vegas: a `pause` (STATIC) turn redraws its plugin for the whole pause,
once a second, or at the high-FPS rate for a plugin that declares
`needs_high_fps` (or scrolls) -- the rate the rotation uses. The pause used
to call `display()` once and sleep out the duration, so a plugin that
rotates its screens or animates was frozen on its first frame. The first
draw still passes `force_clear=True`; later ones pass nothing. A redraw
that finds the plugin's `update()` holding its lock is skipped; the pause
carries on.
- Update Code no longer fails forever with `fatal: Unable to create
'.../.git/index.lock': File exists`. A git killed mid-command (a pull or
checkout that hit its timeout is SIGKILLed; a power cut does the same)
leaves that lock behind. Update Code and the automatic update's rollback
now remove it when no git process is running in the checkout and the
file is more than a few seconds old (new `web_interface/git_index_lock.py`);
while another git is running, Update Code says so instead of failing.
## 3.8.7
Sports consolidation family 10: `src.common.sports_view_model`, the game
+1
View File
@@ -47,6 +47,7 @@
}
}
},
"plugin_schedules": {},
"dim_schedule": {
"enabled": false,
"dim_brightness": 30,
+21 -45
View File
@@ -21,7 +21,10 @@ Each plugin has a *Vegas participation*:
**`pause`:**
- The scroll stops when the plugin's turn comes round
- The plugin draws the whole panel for its display duration, then the
scroll resumes
scroll resumes. `display()` is called again through the pause at the
rotation's rate (once a second, or the high-FPS rate for a plugin that
declares `needs_high_fps`), so a plugin can rotate screens or animate in
it; only the first call passes `force_clear=True`
- Best for content that needs to be read in full, or alerts
**`exclude`:**
@@ -362,7 +365,7 @@ Vegas mode consists of four core components working together to provide smooth 1
**Main Loop:**
1. Check for interrupts (live priority, on-demand, config updates)
2. If static pause active, wait for duration
2. If a static pause is due, redraw that plugin for its duration
3. Otherwise, delegate to render pipeline for frame rendering
4. Sleep to maintain target FPS
@@ -650,11 +653,10 @@ When nothing is running on demand, `data.state` is
> above (or the web UI buttons). The API handlers
> (`start_on_demand_display()` / `stop_on_demand_display()` in
> `web_interface/blueprints/api_v3/display.py`) send the request over the
> display's control socket ([IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)).
> Only when the socket cannot carry it do they write it into the cache
> manager under the `display_on_demand_request` key, which
> `DisplayController._poll_on_demand_requests()`
> (`src/display_controller.py`) picks up. A separate
> display's control socket ([IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)),
> the only way in (the `display_on_demand_request` cache-key mailbox is
> gone). A plugin asks for the screen with `BasePlugin.request_on_demand()`
> (see [PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md)). A separate
> `display_on_demand_config` key is used by the controller itself
> during activation (`_activate_on_demand()`) to track what's
> currently running, and is cleared by `_clear_on_demand()`.
@@ -737,29 +739,12 @@ keys helps troubleshoot stuck states.
### Cache Keys
**1. display_on_demand_request** (TTL: 1 hour)
```json
{
"request_id": "uuid-string",
"action": "start|stop",
"plugin_id": "plugin-name",
"mode": "mode-name",
"duration": 30.0,
"pinned": true,
"timestamp": 1234567890.123
}
```
**Purpose:** Communication from web interface to display controller, as the
fallback when the control socket cannot carry the request (deprecated; it
will be removed in a later release)
**When Set:** API endpoint receives a request and the display's control
socket is unavailable (display stopped, or older than the socket or the
command); some plugins also write it directly
**Read:** once a second while the display serves the control socket (0.25 s
without it), and only when the file changed since the last look
**Auto-Cleared:** After processing or 1 hour TTL
Requests are not cache keys: they go over the control socket. The
`display_on_demand_request` and `display_on_demand_processed_id` keys of
earlier releases are no longer written or read; a leftover file is
harmless and can be deleted.
**2. display_on_demand_config** (No TTL)
**1. display_on_demand_config** (No TTL)
```json
{
"mode": "mode-name",
@@ -771,7 +756,7 @@ without it), and only when the file changed since the last look
**When Set:** Controller processes start request
**Auto-Cleared:** When on-demand stops
**3. display_on_demand_state** (Continuously updated)
**2. display_on_demand_state** (Continuously updated)
```json
{
"active": true,
@@ -785,31 +770,24 @@ without it), and only when the file changed since the last look
**When Set:** Every display loop iteration
**Auto-Cleared:** Never (continuously updated)
**4. display_on_demand_processed_id** (TTL: 1 hour)
```text
"uuid-string-of-last-processed-request"
```
**Purpose:** Prevents duplicate request processing
**When Set:** After processing request
**Auto-Cleared:** After 1 hour TTL
### When Manual Clearing is Needed
**Scenario 1: Stuck in On-Demand State**
- Symptom: Display stays on one plugin, won't return to rotation
- Clear: `config`, `state`, `request`
- Clear: `config`, `state`
**Scenario 2: Mode Switching Issues**
- Symptom: Can't change to different plugin
- Clear: `request`, `processed_id`, `state`
- Clear: `state`, then restart the display
**Scenario 3: On-Demand Not Activating**
- Symptom: Button click does nothing
- Clear: `processed_id`, `request`
- Symptom: Button click does nothing, or answers an error
- Check the error's `socket_error` (see [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md),
"Checking it on a device"); no cache key is involved
**Scenario 4: After Service Crash**
- Symptom: Strange behavior after crash/restart
- Clear: All four keys
- Clear: both keys
### Manual Recovery Procedures
@@ -846,8 +824,6 @@ from src.cache_manager import CacheManager
cache = CacheManager()
cache.clear_cache('display_on_demand_config')
cache.clear_cache('display_on_demand_state')
cache.clear_cache('display_on_demand_request')
cache.clear_cache('display_on_demand_processed_id')
```
> `CacheManager` also has a `delete(key)` method — a thin wrapper over
+14 -15
View File
@@ -42,11 +42,11 @@ each other. They share three things:
| State | Where | Written by | Read by |
|---|---|---|---|
| On-demand command | control socket `/run/ledmatrix/control.sock` ([IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)) | web: `start_on_demand_display()` / `stop_on_demand_display()` in [`api_v3/display.py`](../web_interface/blueprints/api_v3/display.py), via [`src/ipc/client.py`](../src/ipc/client.py) | display: [`src/ipc/server.py`](../src/ipc/server.py) acks; the render thread applies it in `_poll_on_demand_requests()` |
| On-demand request (fallback) | cache `display_on_demand_request` | web, only when the socket could not carry the request (`should_fall_back`); four plugins write it directly | display: `_poll_on_demand_requests()`, a `stat()` every 1 s while the socket is up (0.25 s without), read only when the file changed |
| On-demand request from a plugin | in memory: `BasePlugin.request_on_demand()` / `end_on_demand()` | a plugin in the display process | display: `submit_plugin_on_demand()` queues it; the render thread applies it in `_poll_on_demand_requests()` |
| On-demand state | cache `display_on_demand_state` | display: `_publish_on_demand_state()` | web: `/api/v3/display/on-demand/status` |
| Current screen | cache `display_current_state` | display | web: `/api/v3/display/current-status` |
| Plugin errors | cache `plugin_error_snapshot` | display: `ErrorSnapshotPublisher` ([`src/error_aggregator.py`](../src/error_aggregator.py)) | web: `read_error_report()` for `/api/v3/errors/*` |
| Error clear | control socket `errors.clear`; cache `plugin_error_clear_request` as the fallback | web: `POST /api/v3/errors/clear` | display: applied before the socket answers; the mailbox on the error publisher's 5 s tick, read only when the file changed |
| Error clear | control socket `errors.clear` | web: `POST /api/v3/errors/clear` | display: applied before the socket answers |
| Font usage | cache `font_usage_snapshot` | display: `FontUsagePublisher` ([`src/font_usage.py`](../src/font_usage.py)) | web: Fonts tab |
| Fetch statistics (requests per plugin and host) | cache `fetch_stats_snapshot` | display: `FetchStatsPublisher` ([`src/common/fetch_service.py`](../src/common/fetch_service.py)), at most once a minute on change | web: `read_fetch_stats()` for `/api/v3/plugins/fetch-stats` |
| Plugin health | cache `plugin_health:<id>` | display (web writes on reset) | web: `/api/v3/plugins/health` |
@@ -58,18 +58,17 @@ each other. They share three things:
The on-demand start route starts `ledmatrix.service` when it is not running
(`start_service`, on by default) but never restarts a running one. The routes
send the command over the display's control socket and get an ack; only when
the socket could not carry it (a stopped display, one older than the socket
or the command) do they write the mailbox instead. A display that had the
request and refused it is answered with the error, not posted a mailbox
copy. The display looks at the mailbox every
`MAILBOX_POLL_INTERVAL_WITH_SOCKET` (1 s) while it serves the socket, and
every `ON_DEMAND_POLL_INTERVAL` (0.25 s) without one, from its dwell sleep,
its render loops and Vegas's interrupt check as well as the main loop; a
look is one `stat()` unless the file changed. Both ways end in the same
handler, `_handle_on_demand_request()`.
send the command over the display's control socket and get an ack. That is
the only way in: the cache-key mailboxes (`display_on_demand_request`,
`plugin_error_clear_request`) are gone, and a write to either is dropped
with a warning. When no display is listening yet, the start route starts the
service (if asked) and answers `202`; the web process's dispatcher
([`on_demand_dispatch.py`](../web_interface/on_demand_dispatch.py)) sends the
request once the socket is up, and the status routes report the outcome.
Any other failure is answered as an error. Socket commands and plugins'
in-process requests end in the same handler, `_handle_on_demand_request()`.
The socket's handlers only queue; see [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)
for the protocol, the permission model and the plan to retire the mailboxes.
for the protocol and the permission model.
### Web and display processes: who runs plugins
@@ -118,8 +117,8 @@ other web-UI action runs its script as a subprocess. A later, explicit
The **control socket** from the web process to the display
([IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)) carries on-demand
commands and reloads an updated plugin; its next stages stream the
display's state and retire the cache-key mailboxes. The plugin web-entry
commands, reloads an updated plugin and streams the display's state; it
replaced the cache-key mailboxes. The plugin web-entry
contract above is still to come.
### Plugin state: desired, observed, and who owns it
+35
View File
@@ -55,6 +55,41 @@ saved via `POST /api/v3/config/dim-schedule`). The display returns to
boundaries as `schedule`: dimmed from `start_time` up to, but not including,
`end_time`.
## `plugin_schedules` — when each plugin may show
Maps a plugin id to a schedule shaped like `schedule` (the window in which
that plugin may show), plus:
| Key | Type / default | Meaning |
|---|---|---|
| `hide_when_live` | bool, `false` | Hide the plugin whenever any plugin reports a live game (its `has_live_content()`), live priority or not |
A plugin with no entry always shows. Outside its window, or during a live
game with `hide_when_live`, a plugin is skipped in the rotation, in the live
takeover and in Vegas mode; it stays loaded, and an on-demand request still shows it. Example, news off on
Sundays and during any live game:
```json
"plugin_schedules": {
"news": {"enabled": true, "mode": "per-day", "hide_when_live": true,
"start_time": "00:00", "end_time": "23:59",
"days": {"sunday": {"enabled": false}}}
}
```
As in `schedule`, a day missing from `days` uses the top-level times
(`07:00`–`23:00` when those are missing too). The window ends at `end_time`,
so `00:00`–`23:59` leaves the plugin out for that last minute.
A change takes effect at the next screen in the rotation or live takeover,
and in Vegas mode at its next plugin-list refresh: the next scroll group,
or at the latest the next scroll cycle (about 6 minutes on ledpi with 8
plugins). Content already composed into the ticker scrolls out first.
Read by `src/plugin_schedule.py` through `DisplayController._hidden_plugins()`.
Managed in the web UI under Schedule → Plugin Schedules, saved via
`POST /api/v3/config/plugin-schedules`.
## `display.hardware` — matrix panel hardware
All keys map to the corresponding `rpi-rgb-led-matrix` options and are read
+119 -90
View File
@@ -1,18 +1,18 @@
# Control socket (web → display)
The display process serves a Unix socket that the web interface uses to send
it commands and get an answer back. It replaces the cache-file "mailboxes" on
it commands and get an answer back. It replaced the cache-file "mailboxes" on
the SD card one command at a time. Stage 1 carries on-demand start, stop and
status. Stage 2 makes those commands land within a frame on every kind of
screen, and adds `brightness.set` and `plugin.reload`. Stage 3 adds a state
stream (`state.get`, `state.subscribe`), so the web interface reads what the
display is doing from the socket instead of from cache files the display
wrote to the SD card. Stage 4 makes the socket the only way a command goes
while it works: the web interface writes a mailbox only when the socket
cannot carry the request, `errors.clear` replaces the last command that
always went through a mailbox, and the display looks at the mailboxes once
a second, with a `stat()`. The file mailboxes and the cache keys stay as a
fallback for one release.
display wrote to the SD card. Stage 4 makes the socket the only way a command goes
while it works, and `errors.clear` replaces the last command that always
went through a mailbox. Stage 5 removes the mailboxes: the web interface no
longer writes them and the display no longer reads them, so the socket is
the only way a command reaches the display (see "Without the socket"). The
cache keys the display writes for readers stay as their fallback.
| | |
|---|---|
@@ -145,14 +145,17 @@ than the display's wait, so `pending` arrives before the client gives up.
`v` the display does not speak gets `unsupported_version`. `hello` is checked
by its `versions` list instead, and its result names the highest version both
sides share, so a client can find out what a display supports before it
relies on anything newer. The client sends `v: 1` and falls back to the
mailbox when the display refuses it. It does not send `hello` first, which
relies on anything newer. The client sends `v: 1`, and a route answers an
error when the display refuses it. It does not send `hello` first, which
saves a round trip.
New commands are added within a version, so stage 2 is still version 1. A
display that does not know a command answers `unknown_command`, which the
web interface treats like any other socket failure and falls back from, and
`hello` lists the commands a display knows. The version changes only when the
`hello` lists the commands a display knows.
(Since stage 5, a command the display does not know is an error the route
reports, telling the user to restart the display; there is no mailbox left
to fall back to.) The version changes only when the
envelope or the meaning of an existing command changes.
**Events.** `state.subscribe` is the one command with more than one message
@@ -217,7 +220,8 @@ socket.
}}
```
- `display` and `on_demand` are the dicts the cache keys hold, `plugins` is
- `display` and `on_demand` are the dicts the cache keys hold (`on_demand`
includes `request_id`, the request it answers), `plugins` is
the runtime snapshot (`build_runtime_snapshot`), and `brightness` is the
configured level, what the panel shows now, and whether the dim schedule
has it dimmed. A section not published yet is `null`.
@@ -368,13 +372,12 @@ request, validates it against the contract, and then does one of two things:
- For a query, it answers from a status snapshot the display provides
(`DisplayController._control_status`). The snapshot only reads attributes.
The render thread drains the queue in `_poll_on_demand_requests()`, the same
place it reads the mailbox:
The render thread drains the queue in `_poll_on_demand_requests()`:
- An on-demand command goes to `_handle_on_demand_request()`, which is the
mailbox's own handler. The two paths share all of their code: activation,
the processed-id guard, error publishing, and resuming the rotation
afterwards.
- An on-demand command goes to `_handle_on_demand_request()`, which also
handles plugins' own requests. The two paths share all of their code:
activation, the request-id guard, error publishing, and resuming the
rotation afterwards.
- `brightness.set` is applied there and then (`_apply_control_brightness`),
and the current frame is pushed again so the panel shows it.
- `plugin.reload` starts at the top of the next loop pass, the place where
@@ -407,14 +410,13 @@ place it reads the mailbox:
unloads it mid-load; a disable saved meanwhile is applied once the
reload is done. A second reload of the same plugin runs after the first.
The floor on the mailbox read (0.25 s, 1 s since stage 4) does not apply to the queue, because
draining it costs no disk read. A queued command also lets
`_service_pending_changes()` skip its own floor.
Draining the queue costs no disk read, so it has no floor. A queued command
also lets `_service_pending_changes()` skip its own 0.25 s floor.
### Waking the render thread (stage 2)
Stage 1 made the socket answer, but not land sooner: a queued command waited
for the same polls the mailbox does. Measured on ledpi (Pi 4, 24 fps Vegas),
for the same polls the mailbox did. Measured on ledpi (Pi 4, 24 fps Vegas),
a start took 1.02 s on a static screen and about 0.4 s in Vegas either way.
Now the queue wakes the render thread:
@@ -434,8 +436,7 @@ Now the queue wakes the render thread:
- **Scrolling screens** already service pending changes every frame.
So a command lands within a millisecond or so on a static screen and in a
dwell, and within one frame in Vegas and on a scrolling screen. The mailbox
is slower on purpose (see "The mailboxes now"). Commands still run only on the render thread: the
dwell, and within one frame in Vegas and on a scrolling screen. Commands still run only on the render thread: the
connection threads only queue them and set the event. The one exception is
the slow half of `plugin.reload` (tearing down and loading the plugin),
which runs on its own thread. Every change to the display's state still
@@ -453,74 +454,101 @@ bookkeeping. A client's send to the render thread waking took 0.72 ms median
Without a socket (Windows, `LEDMATRIX_CONTROL_SOCKET=off`) the waits are the
plain sleeps they were.
**Exactly once.** Since stage 4 the web interface writes the mailbox only
when the display never had the request (see "When the web interface falls
back"), so a request goes one way or the other, never both. A command and a
mailbox write for the same request still share one `request_id`, and the
`on_demand_request_id` and processed-id checks still drop a second copy: an
older web interface (before stage 4) wrote the mailbox after a reply timed
out, too. The display takes such a copy out of the mailbox when it drops it.
**Exactly once.** A request goes over the socket once. The web route sends
it again only while no display is listening (see "Without the socket"), so
a display gets it at most once; a start with a request id the display has
just processed is dropped anyway (`on_demand_request_id`). The persisted
`display_on_demand_processed_id` guard against a mailbox replayed after a
restart went with the mailbox.
## When the web interface falls back (stage 4)
## Without the socket (stage 5)
The client tells a request the display never had from one it had and then
failed. `ControlError.sent` is True once the whole request was written to a
connected display; a refusal the display sends before reading anything
(`forbidden`, too many connections) carries no request id, and leaves it
False. `src.ipc.client.should_fall_back()` is the one rule every route uses:
False. `src.ipc.client.display_not_listening()` picks out the one case a
later retry can fix: nothing is listening (`no_socket`, `refused`) and the
request was never sent.
| What happened | Example reasons | Mailbox? | The route answers |
|---|---|---|---|
| The display never had it | `no_socket`, `refused`, `disabled`, `unsupported`, a connect or send that timed out, `forbidden` / `busy` at the door, `invalid_request` (refused by the client itself) | yes | success, `transport: "mailbox"`, `socket_error` |
| A display too old to know it (the upgrade case) | `unknown_command`, `unsupported_version` | yes | as above |
| The display had it and failed | `busy` (queue full), `invalid_args`, `internal`, a timeout or hang-up after the send, `bad_response` | no | `503` (`400` for `invalid_args`), `socket_error` |
| What happened | Example reasons | On-demand start | On-demand stop | `errors.clear` |
|---|---|---|---|---|
| No display listening | `no_socket`, `refused` | service stopped: `400` without `start_service`; with it, start the service and answer `202` (`status: "starting"`) at once; the dispatcher sends the request until the display takes it (up to 45 s), else `start-timeout`. Service running (still starting): the same `202`, sent for up to 10 s | `503` ("not running" / "may still be starting"); with `stop_service` the service is stopped and the route succeeds | `503` ("not running"; its errors are the last run's, and the next run starts with none) |
| A display too old to know the command | `unknown_command`, `unsupported_version` | `503` | `503` | `503`, "restart it" |
| No socket in this process | `disabled`, `unsupported` (Windows, `LEDMATRIX_CONTROL_SOCKET=off`) | `503` | `503` | `503` |
| The display had it and failed, turned it away, or never answered | `busy`, `invalid_args`, `internal`, a timeout, a hang-up, `bad_response`, `forbidden` | `503` (`400` for `invalid_args`) | `503` (with `stop_service`: stopped anyway) | `503` |
A display that had the request may have applied it (a reply that timed out),
or would refuse the mailbox copy as well (bad arguments), or is stuck and
would not read the mailbox either (a full queue). Writing the copy anyway
only turned that into a "success". An on-demand stop with `stop_service`
still stops the service, which ends on-demand whatever happened.
Every error answer carries `socket_error` (a reason code, or `other`).
Nothing is written to the cache in any of these cases.
**Waiting for a display that is starting.** The socket comes up when the
display's run loop starts, after every plugin has loaded, which can take
longer than a client waits (the MQTT bridge gives up after 15 s). So the
start route never waits: it answers `202` with `status: "starting"`, and
hands the request to the web process's one dispatcher
([`web_interface/on_demand_dispatch.py`](../web_interface/on_demand_dispatch.py)).
Its worker thread sends the request every 0.5 s while nothing is listening,
until the display acknowledges it or the wait runs out (45 s after a cold
start, `START_WAIT_SECONDS`; 10 s for a service that was already running,
`ON_DEMAND_SOCKET_WAIT_RUNNING_SECONDS`). Any other failure ends it at once.
One start is pending at a time: a newer start replaces it, and a stop
cancels it (the stop then succeeds even with no display listening, and
reports `cancelled_request_id`).
The outcome is reported where clients already look:
`GET /display/on-demand/status` answers the pending start's state
(`source: "web"`, `status: "starting"`, or `status: "error"` with `error:
"start-timeout"` or the socket's reason) until the display publishes
something newer, and `GET /display/current-status` adds it as
`on_demand_pending`.
A delivered start keeps reading as `status: "starting"`, now with
`delivered: true`, until the display publishes the state that answers it.
The display acknowledges a start as soon as its socket opens, but its run
loop acts on it only after the first screen is built (about 5 s on ledpi,
while Vegas renders its first strip), and meanwhile it publishes its own
idle state. The display's on-demand state names the request it answers
(`request_id`), so "answers it" means the id matches. A display older than
that field answers with any state published after the delivery. Either
way the delivered start is reported for at most 30 s
(`DELIVERED_SHOWN_SECONDS`).
Brightness and plugin reload never had a mailbox: without the socket, the
config watcher applies the saved brightness and a reload becomes the
restart banner, as before.
### The mailboxes now
### The mailboxes are gone
| Mailbox | Written by | Read by the display | While the socket is up |
|---|---|---|---|
| `display_on_demand_request` | the web interface, only on fallback; plugins that predate `BasePlugin.request_on_demand()`, or run on a core without it | the render thread, `_poll_on_demand_requests()` | looked at every 1 s (`MAILBOX_POLL_INTERVAL_WITH_SOCKET`), 0.25 s without a socket |
| `plugin_error_clear_request` | the web interface, only on fallback | the error publisher's thread, every 5 s tick | unchanged rate |
| Former mailbox | Last written by | Now |
|---|---|---|
| `display_on_demand_request` | the web interface on fallback (stage 4); plugins that predate `BasePlugin.request_on_demand()` | not read. A write is dropped by `CacheManager.save_cache` (`RETIRED_MAILBOX_KEYS`) and logged once per writer as a warning that names the plugin when it can be told (the plugin instance on the call stack, else the `plugin_id` in the request) |
| `plugin_error_clear_request` | the web interface on fallback (stage 4) | not read; a write is dropped and logged the same way |
A look is one `stat()` of the mailbox file (`CacheManager.file_signature`):
`(inode, mtime, size)`, and every write renames a new file into place, so a
new write always looks different. `MailboxWatch` reads the file only when
that changed since the last look, so a mailbox that holds nothing new, or
nothing at all, costs no open and no parse. A socket command never reads or
deletes the on-demand mailbox. A start already processed is taken out of
the mailbox instead of being re-read until it expires.
A file left on the SD card by an older version is never read again, so it
is harmless. `MailboxWatch` and
`CacheManager.file_signature`, which made a look at a mailbox one `stat()`,
went with them.
A request that comes through the on-demand mailbox while the socket is up
is logged once per writer (`came through the file mailbox although the
control socket is up`), which names the plugins that still write it.
The four plugins that wrote `display_on_demand_request` (birdnet-go,
mqtt-notifications, on-air, pomodoro-timer) use `request_on_demand()` on a
core that has it, and fall back to the mailbox only when that method is
missing or answers `None` (no display in the process, or a full queue).
On a stage-5 core such a fallback write is dropped with the warning above.
### Plugins in the display process
A plugin asks for the screen with `BasePlugin.request_on_demand()` and gives
it back with `end_on_demand()` (see "On-demand display" in
[PLUGIN_API_REFERENCE.md](PLUGIN_API_REFERENCE.md)). Neither goes through
the socket or a file: `PluginManager` hands the mailbox-shaped request,
the socket or a file: `PluginManager` hands the on-demand request,
marked `source: 'plugin'`, to `DisplayController.submit_plugin_on_demand`,
which queues it in memory (at most `PLUGIN_ON_DEMAND_QUEUE_SIZE`, 32) from
whatever thread the plugin called on, and wakes the render thread through
the socket's queue flag (`ControlServer.wake()`). The render thread applies
it in `_drain_control_commands`, after the socket's commands, through the
same `_handle_on_demand_request`, so it lands within a frame like a socket
command. Without a socket it lands on the next pending-changes pass (typically
within 0.25 s). A plugin's stop ends only a session that plugin owns. The four
plugins that wrote the mailbox (birdnet-go, mqtt-notifications, on-air,
pomodoro-timer) use it where the core has it and write the mailbox
otherwise.
command. Without a socket it lands on the next pending-changes pass. A
plugin's stop ends only a session that plugin owns.
## Robustness
@@ -538,8 +566,7 @@ block the render loop or crash it:
found. A client that disconnects mid-message is dropped silently. No
exception from a handler leaves the connection thread.
- **Full queue.** When the queue is full, the client gets `busy`, and the web
interface answers `503` rather than write the mailbox, which the stuck
render thread would not read either. A full queue means the render thread
interface answers `503`. A full queue means the render thread
is stuck, and the systemd watchdog deals with that.
- **Awaited commands.** The wait for an awaited command's outcome happens on
its connection thread and is bounded (`AWAIT_SECONDS`), so a stuck render
@@ -554,8 +581,8 @@ block the render loop or crash it:
process created.
- **Never fatal.** If the server cannot start (Windows, no `AF_UNIX`, a bind
failure, `LEDMATRIX_CONTROL_SOCKET=off`), it logs that and the display runs
as before. The web interface then uses the mailbox, and reads the cache
keys and the heartbeat file.
as before. The web interface then cannot send it commands (the routes
answer `503`), and reads the cache keys and the heartbeat file.
- **Subscribers (stage 3).** A `state.subscribe` connection gives its request
slot back and takes one of 4 subscriber slots (`MAX_SUBSCRIBERS`). A fifth
gets `busy`. So a few browsers' web processes holding streams can never
@@ -613,11 +640,9 @@ device never touches the live display.
## Stage plan
1. **On-demand, with acks (done, #706).** Contract, server, client.
`on_demand.start`/`stop`/`status`, `hello`, `ping`. The REST routes try the
socket first and report `transport: "socket" | "mailbox"` (plus
`socket_error` on fallback). The mailbox is unchanged, and the plugins that
write it directly (birdnet-go, mqtt-notifications, on-air, pomodoro-timer)
keep working.
`on_demand.start`/`stop`/`status`, `hello`, `ping`. The REST routes tried
the socket first and reported `transport: "socket" | "mailbox"` (plus
`socket_error` on fallback).
2. **Commands that were restarts or polls (done).**
- The render thread waits on the queue instead of sleeping, and Vegas
checks it every frame, so a command lands within a frame on every kind
@@ -660,12 +685,12 @@ device never touches the live display.
4. **The mailboxes become a fallback (done).**
- The web interface writes a mailbox only when the socket could not carry
the request (`should_fall_back`); a display that had it and failed is
answered as that (see "When the web interface falls back").
answered as that.
- `errors.clear` replaces `plugin_error_clear_request` as the way a clear
reaches the display.
- The display looks at the on-demand mailbox once a second while the
socket is up, reads either mailbox only when its file changed, and logs
who still writes the on-demand one (see "The mailboxes now").
who still writes the on-demand one.
- Not changed, deliberately: config saves (the schedule, the dim
schedule, plugin settings) still reach the display through
`config.json` and its watcher, which is the setting itself rather than
@@ -674,15 +699,20 @@ device never touches the live display.
already stats at most once a second. Plugin health and metrics resets
write the persisted record the display publishes and do not reach the
running display (their routes say so); they are not mailboxes.
5. **Remove the mailboxes (next release).** Once every device has run a
display with stage 4, the web interface stops writing both mailboxes and
the display stops reading them. The four plugins that wrote
`display_on_demand_request` now have an in-process way to ask for the
screen (`BasePlugin.request_on_demand()` / `end_on_demand()`, see
"Plugins in the display process"); they keep the mailbox write only as
their fallback on older cores. The display also stops writing `display_current_state`,
`display_on_demand_state` and `plugin_runtime_snapshot` once the web
interface no longer falls back to them.
5. **Remove the mailboxes (done, the release after 3.8.1).** The web
interface no longer writes `display_on_demand_request` or
`plugin_error_clear_request`, and the display no longer reads them (see
"Without the socket"). When no display is listening, the start route
starts the service if asked, answers `202`, and the web process's
dispatcher sends the request once the socket is up; every other failure
is an error the route reports. A write to
either key is dropped with a one-time warning naming the writer. The
display still writes `display_current_state`, `display_on_demand_state`
and `plugin_runtime_snapshot`: the web interface reads them whenever the
socket cannot answer (a stopped or starting display, a web user not yet
in the socket's group, a platform without Unix sockets), and
`display_on_demand_config` is the display's own record for resuming a
session after a restart. Retiring those keys is left for later.
## Checking it on a device
@@ -694,12 +724,11 @@ curl -s -X POST localhost:5000/api/v3/display/on-demand/start \
# ... "transport": "socket"
```
If the response says `"transport": "mailbox"`, `socket_error` gives the
reason. `no_socket` means the display is stopped or predates the socket.
`refused` usually means the web user is not in the socket's group, which
takes effect when the web service restarts after the user is added. A `503`
with `"transport": "socket"` means the display had the request and did not
take it (`busy`, `timeout`, ...): nothing was written to the mailbox.
On an error, `socket_error` gives the reason. `no_socket` means the display
is stopped or still starting. `refused` usually means the web user is not in
the socket's group, which takes effect when the web service restarts after
the user is added. `busy`, `timeout` and the like mean the display had the
request and did not take it. Nothing is ever written to a mailbox.
An error clear:
@@ -707,7 +736,7 @@ An error clear:
curl -s -X POST localhost:5000/api/v3/errors/clear \
-H 'Content-Type: application/json' -d '{"all":true}'
# ... "applied": true, "transport": "socket"
sudo journalctl -u ledmatrix | grep -E "Cleared .* plugin error|file mailbox"
sudo journalctl -u ledmatrix | grep -E "Cleared .* plugin error|retired"
```
Brightness and a plugin reload:
+31 -21
View File
@@ -328,7 +328,7 @@ Each plugin takes part in Vegas mode in one of three ways:
| Participation | What Vegas does |
|---|---|
| `'scroll'` | The plugin's content (`get_vegas_content()`) scrolls by with everything else |
| `'pause'` | The scroll stops when the plugin's turn comes round; its `display()` draws it full screen for `get_display_duration()` seconds, then the scroll resumes |
| `'pause'` | The scroll stops when the plugin's turn comes round; its `display()` draws it full screen for `get_display_duration()` seconds, then the scroll resumes. `display()` is called again through the pause, as in the rotation: once a second, or at the high-FPS rate if the plugin declares `needs_high_fps`; the first call passes `force_clear=True` |
| `'exclude'` | The plugin is left out of Vegas mode |
Declare the plugin's default in `manifest.json`:
@@ -522,42 +522,52 @@ Returns the request id once queued, or `None` as above.
These methods are new after core 3.8.0 (see `CHANGELOG.md`). Before them,
plugins wrote the `display_on_demand_request` cache key (the "mailbox")
themselves. The display reads it only once a second while the control
socket is up, and it will be removed in a future release (see
[IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md), stage 5). A plugin that
must keep working on older cores checks for the method, and writes the
mailbox only when the method is missing or answers `None`:
themselves. **The mailbox is gone** (see
[IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md), stage 5): the display no
longer reads it, and a write to it is dropped with a warning in the log,
once per plugin:
```
Ignored a write to the retired 'display_on_demand_request' cache key by plugin 'my-plugin': ...
```
A plugin that only needs to run on cores with these methods calls them and
treats `None` as "no display took it":
```python
def _show_alert(self):
if self.request_on_demand(mode="my_alert", duration=15) is None:
self.logger.info("No display to show the alert on")
```
A plugin that must also work on cores before them can keep the mailbox
write as its fallback, guarded by `hasattr`: on those cores the display
still reads it, and on a current core the write is only dropped and logged
(when the method is missing it never runs at all). Raise
`ledmatrix_min_version` to the release that added the methods once you no
longer need it.
```python
import time, uuid
def _show_alert(self):
if hasattr(self, "request_on_demand") and self.request_on_demand(
mode="my_alert", duration=15):
if hasattr(self, "request_on_demand"):
self.request_on_demand(mode="my_alert", duration=15)
return
# Older core, or no display in this process: the mailbox, as before.
# A core older than request_on_demand(): the mailbox it still reads.
self.cache_manager.set("display_on_demand_request", {
"request_id": str(uuid.uuid4()), "action": "start",
"plugin_id": self.plugin_id, "mode": "my_alert",
"duration": 15, "pinned": False, "timestamp": time.time(),
})
def _release(self):
if hasattr(self, "end_on_demand") and self.end_on_demand():
return
self.cache_manager.set("display_on_demand_request", {
"request_id": str(uuid.uuid4()), "action": "stop",
"plugin_id": self.plugin_id, "timestamp": time.time(),
})
```
Keep `ledmatrix_min_version` where it is: the fallback is what keeps the
plugin working on older cores. A mailbox stop ends any on-demand session,
whoever started it; `end_on_demand()` ends only the plugin's own.
`end_on_demand()` ends only the plugin's own session; a stop written to the
mailbox on an older core ends any session, whoever started it.
Both methods answer a request id only when the plugin manager returned a
string, so a test that gives the plugin a `MagicMock()` plugin manager gets
`None` and exercises the mailbox path. To test the new path, set
`None`. To test the path where the display takes the request, set
`plugin_manager.request_on_demand.return_value = "some-id"`.
> The full source for `BasePlugin` lives in
+74 -34
View File
@@ -466,7 +466,7 @@ Request a specific plugin to display on-demand.
- `mode` (string, optional): Display mode name (plugin_id inferred if not provided)
- `duration` (number, optional): Duration in seconds (0 = until stopped)
- `pinned` (boolean, optional): Pin display (pause rotation)
- `start_service` (boolean, optional): Start the display service if it is not running (default: true). A running service is never restarted: it picks the request up within a frame over its control socket (within about a second through the mailbox fallback). When false and the service is stopped, the route returns 400.
- `start_service` (boolean, optional): Start the display service if it is not running (default: true). A running service is never restarted: it picks the request up within a frame over its control socket. A stopped one is started, and the route answers `202` at once (see below); the request is sent once the display's socket is up, which can take as long as the display takes to load its plugins. When false and the service is stopped, the route returns 400.
**Response**:
```json
@@ -486,24 +486,54 @@ Request a specific plugin to display on-demand.
`service` is `null` when `start_service` is false.
`transport` says how the request reached the display: `"socket"` means the
display's control socket acknowledged it (it is queued for the render thread,
which wakes for it and applies it within a frame; see
[IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)), `"mailbox"` means it was
written to the cache mailbox the display polls, as before the socket existed.
The mailbox is used only when the socket could not carry the request. With
`"mailbox"`, `socket_error` gives the reason (`no_socket` when the display is
stopped or predates the socket, `refused`, a connect `timeout`,
`unknown_command` from a display too old for the command, ...). Either way
the request is applied the same way; `request_id` is the same id in both.
`transport` is always `"socket"`: the display's control socket acknowledged
the request (it is queued for the render thread, which wakes for it and
applies it within a frame; see [IPC_CONTROL_SOCKET.md](IPC_CONTROL_SOCKET.md)).
The `"mailbox"` value earlier releases could answer is gone with the
mailbox: nothing is written to the cache.
When the display had the request and did not take it -- a full queue
(`busy`), bad arguments (`invalid_args`), no answer after the request was
sent (`timeout`, `closed`) -- the route answers `503` (`400` for
`invalid_args`) with `status: "error"` and `data: {request_id, transport:
"socket", socket_error}`, and writes nothing to the mailbox. The stop route
does the same, except that with `stop_service: true` it still stops the
service and answers success.
**No display listening yet** (`no_socket`, `refused`: the service is
stopped, or still loading its plugins). With the service stopped and
`start_service` false, `400`. Otherwise the route starts the service if
needed and answers at once:
```json
{
"status": "starting",
"message": "The display service is starting; ...",
"data": {"request_id": "uuid-here", "plugin_id": "football-scoreboard", "mode": "nfl_live",
"duration": 45, "pinned": true, "service": {"active": true, "started": true},
"transport": "socket", "socket_error": "no_socket",
"pending": true, "wait_seconds": 45.0}
}
```
with HTTP `202`. The web process sends the request until the display takes
it, for up to `wait_seconds` (45 after a cold start, 10 when the service was
already running). Follow it with `GET /api/v3/display/on-demand/status`:
its `state` is `{status: "starting", source: "web", request_id, ...}` while
it waits, and still `starting` with `delivered: true` once the display has
acknowledged it but not yet published the state for that `request_id` (at
most 30 s), then the display's own state, or `{status: "error",
error: "start-timeout"}` (or the socket's reason) if it never was;
`GET /api/v3/display/current-status` carries the same as
`on_demand_pending`. A newer start replaces a pending one; a stop cancels it.
Otherwise, when the display did not take the request, the route answers an
error with `status: "error"` and `data: {request_id, transport: "socket",
socket_error}`:
- a full queue (`busy`), no answer after the request was sent (`timeout`,
`closed`), a display older than the command (`unknown_command`), no
socket in the web process (`disabled`, `unsupported`): `503` at once;
- bad arguments (`invalid_args`): `400`.
The stop route answers the same errors (`503` when no display is
listening, with a message saying whether the service is stopped), except
that with `stop_service: true` it still stops the service and answers
success, with `socket_error` set, and that a stop which cancelled a pending
start succeeds with `cancelled_request_id` even when no display is
listening.
### Stop On-Demand Display
@@ -533,7 +563,7 @@ Stop the current on-demand display.
}
```
`transport` and `socket_error` are as for start.
`transport` and the errors are as for start; a stop is never retried.
---
@@ -2199,7 +2229,7 @@ Every response below adds three fields to the shape it always had:
|---|---|
| `snapshot_available` | `false` until the display service has reported (for example, it is not running). Counts are then zero. |
| `generated_at` | When the display service produced the snapshot (ISO, the Pi's local time), or `null`. |
| `clear_pending` | A clear has been requested and the display service has not applied it yet. |
| `clear_pending` | Always `false`: a clear is applied before its route answers. Kept for compatibility. |
### Get Error Summary
@@ -2280,21 +2310,17 @@ before it answers: `applied` is `true`, `transport` is `"socket"`, and
}
```
When the socket cannot carry it (the display is stopped, or older than
`errors.clear`) the clear is asynchronous, as before: the web interface
records a request (`plugin_error_clear_request` in the shared cache),
`applied` is `false` and `transport` is `"mailbox"`, and the display service
applies it within about 5 seconds. Reads hide the cleared errors from the
moment the request is recorded. Until the display service applies an
age-based clear, `recent_errors` and `active_patterns` are already filtered
but the counts are the old ones, and `clear_pending` is `true`. Then
`cleared_count` is how many of the reported errors the clear hides, and
`null` when that cannot be known before the display service applies it (an
age-based clear over more errors than the report lists).
`clear_requested`, `applied` and `transport` are always `true`, `true` and
`"socket"`, and are kept for compatibility.
A request that could not be written to the shared cache answers `500`. A
display that had the request and failed it (`internal`, a timeout after the
request was sent) answers `503`, with `context.socket_error`.
When the socket cannot carry the clear, the route answers `503` with
`context.socket_error`, and nothing is cleared or recorded: the
`plugin_error_clear_request` mailbox earlier releases fell back to is gone.
The message says why: the display service is not running (its errors are
then the last run's, and its next run starts with none), it is too old for
`errors.clear` (restart it), the web process has no socket, or the display
had the request and failed it (`internal`, a timeout after the request was
sent).
---
@@ -2383,6 +2409,20 @@ enabled with default times `20:00`-`07:00`. As for the schedule above, an
enabled dim schedule needs at least one day enabled and a disabled one may
have every day off.
### Plugin Schedules
**GET** `/api/v3/config/plugin-schedules`
Every plugin's schedule (`plugin_schedules` in `docs/CONFIG_REFERENCE.md`),
keyed by plugin id.
**POST** `/api/v3/config/plugin-schedules`
Save one plugin's schedule: `plugin_id`, `hide_when_live`, and the same
fields as `POST /api/v3/config/schedule`. A schedule that is off and does
not hide during live games is removed. 400 for a core section name or a bad
time, 404 for a plugin that is not installed.
---
## Integrations
+1 -1
View File
@@ -4,5 +4,5 @@ LEDMatrix Display System
Core source package for the LED Matrix Display project.
"""
__version__ = "3.8.7"
__version__ = "3.9.0"
+54 -48
View File
@@ -25,10 +25,11 @@ Typical plugin usage::
import json
import os
import sys
import time
from datetime import datetime
import pytz
from typing import Any, Dict, List, Optional, Tuple
from typing import Any, Dict, List, Optional
import logging
import threading
import tempfile
@@ -72,41 +73,60 @@ def _outlived(record: Any, max_age: Optional[float], now: float) -> bool:
return False
_NOT_SEEN: Any = object()
#: Cache keys that were file "mailboxes" from the web interface (and some
#: plugins) to the display. The control socket replaced them, and nothing
#: reads them any more, so a write is refused rather than left on the SD card
#: for nobody: see :func:`_refuse_retired_mailbox_write`.
RETIRED_MAILBOX_KEYS = frozenset({'display_on_demand_request', 'plugin_error_clear_request'})
#: (key, writer) pairs already warned about, so a plugin that writes on every
#: event logs once per process, not once per write.
_retired_writers_warned: set = set()
_retired_writers_lock = threading.Lock()
class MailboxWatch:
"""Tells the poller of a mailbox key whether its file changed since the
last look, from one stat() (:meth:`CacheManager.file_signature`).
def _retired_mailbox_writer(data: Any) -> str:
"""Name whoever is writing a retired mailbox key, as well as can be told.
The display polls the mailboxes the web interface falls back to. Reading
one is an open and a JSON parse; with this a poll that finds the same file
(or none) costs a stat, and the file is read only after a new write. A
cache without ``file_signature`` (a test double) is read every time.
The plugin instance on the call stack when there is one (a ``self`` with
a string ``plugin_id`` and a ``cache_manager``: what BasePlugin gives
every plugin), else the ``plugin_id`` the request itself names, else
``'unknown'``.
"""
frame = sys._getframe(2) # pylint: disable=protected-access
depth = 0
while frame is not None and depth < 25:
owner = frame.f_locals.get('self')
plugin_id = getattr(owner, 'plugin_id', None) if owner is not None else None
if isinstance(plugin_id, str) and plugin_id and hasattr(owner, 'cache_manager'):
return f"plugin '{plugin_id}'"
frame = frame.f_back
depth += 1
payload = data.get('data', data) if isinstance(data, dict) else None
named = payload.get('plugin_id') if isinstance(payload, dict) else None
if isinstance(named, str) and named:
return f"plugin '{named}' (named in the request)"
return 'unknown'
def __init__(self, key: str):
self.key = key
self._seen: Any = _NOT_SEEN
def changed(self, cache_manager: Any) -> bool:
"""True when the poller should read the key now."""
signature = getattr(cache_manager, 'file_signature', None)
sig = signature(self.key) if callable(signature) else _NOT_SEEN
if sig is not None and not isinstance(sig, tuple):
return True # cannot tell: read it
if sig is None:
self._seen = None
return False # no file, nothing to read
if sig == self._seen:
return False
self._seen = sig
return True
def forget(self) -> None:
"""Read the key on the next poll even if its file has not changed
(the last read failed)."""
self._seen = _NOT_SEEN
def _refuse_retired_mailbox_write(logger: logging.Logger, key: str, data: Any) -> None:
"""Warn, once per writer, that a write to a retired mailbox key was dropped."""
try:
writer = _retired_mailbox_writer(data)
except Exception: # pylint: disable=broad-except
writer = 'unknown'
with _retired_writers_lock:
if (key, writer) in _retired_writers_warned:
return
_retired_writers_warned.add((key, writer))
if key == 'display_on_demand_request':
hint = ("call self.request_on_demand() / self.end_on_demand() instead "
"(BasePlugin, LEDMatrix 3.8.1 and later)")
else:
hint = "clear errors through POST /api/v3/errors/clear instead"
logger.warning("Ignored a write to the retired '%s' cache key by %s: the display no "
"longer reads this file mailbox. Update it to %s. (Logged once per writer.)",
key, writer, hint)
class CacheManager:
@@ -330,24 +350,6 @@ class CacheManager:
"""Get the path for a cache file."""
return self._disk_cache_component.get_cache_path(key)
def file_signature(self, key: str) -> Optional[Tuple[int, int, int]]:
"""``(st_ino, st_mtime_ns, st_size)`` of ``key``'s file, or None when
there is no file (the key is absent, or this cache has no disk tier).
One stat(), no read: a poller of a mailbox another process writes
compares it with the last one it saw and reads the file only when it
changed. Every write replaces the file (a temp file renamed into
place), so a new write always has a new inode, however fast it came.
"""
path = self._get_cache_path(key)
if not path:
return None
try:
st = os.stat(path)
except OSError:
return None
return (st.st_ino, st.st_mtime_ns, st.st_size)
def get_cached_data(self, key: str, max_age: int = 300, memory_ttl: Optional[int] = None) -> Optional[Dict[str, Any]]:
"""Get data from cache (memory first, then disk) honoring TTLs.
@@ -385,6 +387,10 @@ class CacheManager:
key: Cache key
data: Data to cache
"""
if key in RETIRED_MAILBOX_KEYS:
_refuse_retired_mailbox_write(self.logger, key, data)
return
# Periodic cleanup before adding new entries
self._cleanup_memory_cache()
+1
View File
@@ -23,6 +23,7 @@ CORE_CONFIG_KEYS = frozenset({
'auto_update',
'schedule',
'dim_schedule',
'plugin_schedules',
'timezone',
'target_fps',
'location',
+144 -285
View File
@@ -20,6 +20,7 @@ Entry point: :func:`main` — instantiates :class:`DisplayController` and calls
:meth:`~DisplayController.run`.
"""
import logging
import time
import os
import inspect
@@ -37,6 +38,7 @@ import pytz
from src import display_watchdog
from src.malloc_tuning import MallocTrimmer
from src.plugin_schedule import hidden_plugins, in_window, schedule_state, wants_live_scan
from src.display_arbiter import (
Arbiter, ArbiterInputs, ArbiterState, FramePolicy,
ScreenPlan, Source, WifiNotice, live_pick, live_takeover, on_demand_bound, rotation_plan,
@@ -48,7 +50,7 @@ from src.screen_runner import (
from src.display_manager import DisplayManager
from src.config_manager import ConfigManager
from src.config_service import ConfigService
from src.cache_manager import CacheManager, MailboxWatch
from src.cache_manager import CacheManager
from src.font_manager import FontManager
from src.logging_config import get_logger
from src.exceptions import PluginError
@@ -69,10 +71,6 @@ from src.vegas_mode.render_pipeline import SYNC_SEND_INTERVAL
# Get logger with consistent configuration
logger = get_logger(__name__)
# The on-demand file mailbox: the fallback for a web interface that cannot
# reach the control socket, and how some plugins still ask for the screen.
ON_DEMAND_MAILBOX_KEY = 'display_on_demand_request'
# How often the unchanged current mode is republished for the web UI, which
# treats display_current_state older than 120 s as unknown.
CURRENT_STATE_REFRESH_SECONDS = 30
@@ -412,19 +410,15 @@ class DisplayController:
# coordinator exists (Vegas was off at startup). The render thread
# creates it in _is_vegas_mode_active(), never the watcher thread.
self._pending_vegas_init = False
# Monotonic stamp of the last mailbox disk read; see
# _poll_on_demand_requests. None means "never polled", so the first
# call always goes through.
self._last_on_demand_poll: Optional[float] = None
# Monotonic stamp of the last _service_pending_changes pass; same
# "None means never" convention as _last_on_demand_poll.
# Monotonic stamp of the last _service_pending_changes pass. None
# means "never", so the first call always goes through.
self._last_pending_service: Optional[float] = None
# Monotonic stamp of the last scheduled-update pass; see
# _tick_plugin_updates_if_due. Same "None means never" convention.
self._last_plugin_update_tick: Optional[float] = None
# The control socket (src/ipc), started by run(). None when it is not
# served (Windows, LEDMATRIX_CONTROL_SOCKET=off, a bind failure);
# the file mailbox works either way.
# then only plugins in this process can start on-demand sessions.
self._control_server = None
# A brightness set_brightness() refused, so the periodic service pass
# doesn't retry (and log) the same failure several times a second.
@@ -713,6 +707,11 @@ class DisplayController:
self._schedule_checked_minute: Optional[tuple] = None
self._dim_checked_minute: Optional[tuple] = None
self._cached_target_brightness: int = self._normal_brightness
# Per-plugin schedules (config['plugin_schedules'], src/plugin_schedule.py):
# (key, hidden set), see _hidden_plugins.
self._plugin_hidden_cache: Tuple[Optional[tuple], FrozenSet[str]] = (None, frozenset())
self._game_live = False
self._game_live_at = float('-inf')
# Register controller-level hot-reload callback so cached config values
# (_normal_brightness, _scroll_speed, _tz, minute-gates) stay in sync
@@ -759,6 +758,8 @@ class DisplayController:
# Set up live priority checker
self.vegas_coordinator.set_live_priority_checker(self._check_live_priority)
# Plugins scheduled off sit out the ticker too.
self.vegas_coordinator.stream_manager.hidden_plugins = self._hidden_plugins
# Set up interrupt checker for on-demand/wifi status and follower mode
def _vegas_interrupt():
@@ -920,38 +921,19 @@ class DisplayController:
self._tz = pytz.UTC
return self._tz
@staticmethod
def _in_window(start, end, now) -> bool:
"""Whether ``now`` is within [start, end), a window that may span midnight.
Half-open: on from the start minute, off at exactly the end minute.
A closed end made the end minute count as inside, and because ``now``
carries seconds, only a check at hh:mm:00.000 saw it that way -- so
whether the panel went off at the start or the end of that minute
depended on when the minute's one check ran. ``start == end`` is an
empty window, as it effectively was before.
"""
if start <= end:
return start <= now < end
return now >= start or now < end
# The window rule, shared with the dim schedule and plugin schedules.
_in_window = staticmethod(in_window)
def _check_schedule(self):
"""Check if display should be active based on schedule."""
schedule_config = self.config.get('schedule', {})
# No schedule configured: always active.
if not schedule_config:
# No schedule, or a disabled one: always active.
if not schedule_config or not schedule_config.get('enabled', True):
self.is_display_active = True
self._was_display_active = True
return
# A schedule without an 'enabled' key counts as enabled.
if 'enabled' in schedule_config and not schedule_config.get('enabled', True):
self.is_display_active = True
self._was_display_active = True
logger.debug("Schedule is disabled - display always active")
return
current_time = datetime.now(self._timezone())
# Gate: schedule state can only change on a minute boundary, so skip
# all the strptime / comparison work if we already evaluated this minute.
@@ -960,81 +942,15 @@ class DisplayController:
return
self._schedule_checked_minute = current_minute_key
current_day = current_time.strftime('%A').lower() # e.g. 'monday'
current_time_only = current_time.time()
# Check if per-day schedule is configured
days_config = schedule_config.get('days')
# Determine which schedule to use. Respect an explicit 'mode' field
# (like the dim schedule does) so a stray/legacy 'days' dict left over
# from config migration or a prior per-day setup can't silently
# override a user's Global schedule selection.
mode = schedule_config.get('mode')
mode_normalized = mode.replace('_', '-') if mode else None
use_per_day = False
if mode_normalized == 'global':
use_per_day = False
elif mode_normalized == 'per-day':
use_per_day = bool(days_config and current_day in days_config)
elif days_config:
# No explicit mode recorded (legacy config) - fall back to
# inferring from presence of a 'days' dict for the current day.
if current_day in days_config:
use_per_day = True
active, why = schedule_state(schedule_config, current_time)
if active != self._was_display_active:
logger.info("Schedule activated: Display is now %s (%s)%s",
"ACTIVE" if active else "INACTIVE", why,
"" if active else ". Display will be blanked.")
else:
logger.debug("Per-day schedule exists but %s not configured, using global schedule", current_day)
if use_per_day:
day_config = days_config[current_day]
if not day_config.get('enabled', True):
was_active = self._was_display_active
self.is_display_active = False
if was_active:
logger.info("Schedule activated: Display is now INACTIVE (%s is disabled in schedule). Display will be blanked.", current_day)
else:
logger.debug("Display inactive - %s is disabled in schedule", current_day)
self._was_display_active = self.is_display_active
return
start_time_str = day_config.get('start_time', '07:00')
end_time_str = day_config.get('end_time', '23:00')
schedule_type = f"per-day ({current_day})"
else:
start_time_str = schedule_config.get('start_time', '07:00')
end_time_str = schedule_config.get('end_time', '23:00')
schedule_type = "global"
try:
start_time = datetime.strptime(start_time_str, '%H:%M').time()
end_time = datetime.strptime(end_time_str, '%H:%M').time()
self.is_display_active = self._in_window(start_time, end_time, current_time_only)
was_active = self._was_display_active
if not self.is_display_active:
if was_active:
logger.info("Schedule activated: Display is now INACTIVE (outside %s schedule window %s - %s). Display will be blanked.",
schedule_type, start_time_str, end_time_str)
else:
logger.debug("Display inactive - outside %s schedule window (%s - %s)",
schedule_type, start_time_str, end_time_str)
else:
if not was_active:
logger.info("Schedule activated: Display is now ACTIVE (within %s schedule window %s - %s)",
schedule_type, start_time_str, end_time_str)
else:
logger.debug("Display active - within %s schedule window (%s - %s)",
schedule_type, start_time_str, end_time_str)
self._was_display_active = self.is_display_active
except ValueError as e:
logger.warning("Invalid schedule format for %s schedule: %s (start: %s, end: %s). Defaulting to active.",
schedule_type, e, start_time_str, end_time_str)
self.is_display_active = True
self._was_display_active = True
logger.debug("Display %s - %s", "active" if active else "inactive", why)
self.is_display_active = active
self._was_display_active = active
def _check_dim_schedule(self) -> int:
"""
@@ -1513,8 +1429,18 @@ class DisplayController:
if self._empty_pass_streak < rotation:
return
if self._empty_pass_streak == rotation:
# Blank once: with every plugin empty or scheduled off, the last
# screen drawn would otherwise stay up indefinitely. A blank the
# controller draws, so the last scroll's state ends first.
self._end_scroll_before_core_screen()
self.display_manager.clear()
self.display_manager.update_display()
logger.info("No mode has anything to show; checking one mode every %.0fs "
"until one does", self.EMPTY_ROTATION_PAUSE)
# The preview throttle may have skipped the blank frame, and with no
# display() call to follow, nothing else writes it (see
# DisplayManager.write_owed_snapshot). Cheap when nothing is owed.
self.display_manager.write_owed_snapshot()
self._sleep_with_plugin_updates(self.EMPTY_ROTATION_PAUSE)
#: Plugins already warned about a display duration that is not a number,
@@ -1785,6 +1711,9 @@ class DisplayController:
'status': self.on_demand_status,
'error': self.on_demand_last_error,
'last_event': self.on_demand_last_event,
# The request this state answers: lets the web interface tell
# the outcome of a start it delivered from an older state.
'request_id': self.on_demand_request_id,
'remaining': self._get_on_demand_remaining(),
'last_updated': time.time()
}
@@ -1859,34 +1788,16 @@ class DisplayController:
self.force_change = True
self._publish_on_demand_state()
#: Shortest gap between mailbox disk reads. This is called after every
#: frame -- about 125 times a second on a scrolling mode -- and the read
#: below is deliberately uncached, so without a floor it was 125 disk reads
#: per second to find nothing. An on-demand request comes from a person
#: clicking in the web UI, so a quarter second of latency is not
#: perceptible, and it cuts the read rate by 30x.
ON_DEMAND_POLL_INTERVAL = 0.25
#: The mailbox poll while the control socket is up. The web interface then
#: writes the mailbox only when it could not reach the socket (a display
#: being restarted, a web user not yet in the socket's group), and the
#: plugins that still write it get the screen within this long. A look is
#: one stat() of the mailbox file (MailboxWatch).
MAILBOX_POLL_INTERVAL_WITH_SOCKET = 1.0
#: Shortest gap between _service_pending_changes passes. The same floor
#: the mailbox poll had before the socket, since that read was the only
#: real cost in the pass: the schedule checks are gated to once per clock
#: minute and the rest is attribute compares. Callers run at frame rate,
#: Shortest gap between _service_pending_changes passes. The schedule
#: checks are gated to once per clock minute and the rest is attribute
#: compares. Callers run at frame rate,
#: so between passes the whole cost is one monotonic-clock compare.
PENDING_CHANGES_INTERVAL = 0.25
#: Class-level defaults for controllers built without __init__ (tests).
_control_server: Optional[ControlServer] = None
#: Created on the first poll; see _poll_on_demand_requests.
_on_demand_mailbox: Optional[MailboxWatch] = None
#: Writers whose mailbox requests have been logged (_note_mailbox_request).
_mailbox_writers_logged: FrozenSet[str] = frozenset()
#: The last on-demand request handled; published in _on_demand_state.
on_demand_request_id: Optional[str] = None
#: Most plugin on-demand requests waiting for the render thread at once.
#: A plugin that asks faster than the display drains (four times a
#: second at worst) is refused, not queued without end.
@@ -2029,44 +1940,11 @@ class DisplayController:
on_demand_plugin_id, len(enabled_plugins))
return enabled_plugins
def _consume_on_demand_request(self, request_id: str) -> None:
"""Remove the request we just handled from the mailbox.
Leaving it on disk meant a restart replayed the previous request: the
fresh controller read it, activated it and cached it, so the request
the caller had just made was ignored and the panel silently showed the
earlier plugin.
Compare before deleting. The web process can post a newer request
between the read and this delete; an unconditional delete threw that
one away and it was never processed -- the user's second click did
nothing. Re-reading uncached and only deleting our own request_id
leaves a newer request in the mailbox for the next poll instead.
This narrows the window rather than closing it: a request landing
between the re-read and the delete is still lost. Closing it properly
needs an atomic claim (a rename, or a compare-and-delete primitive)
that the cache layer does not currently offer, so the honest fix is a
smaller window plus this note, not a bigger lock. For start requests
processed_id still guards against reprocessing if the delete fails.
"""
try:
current = self.cache_manager.get(ON_DEMAND_MAILBOX_KEY,
max_age=3600, memory_ttl=0)
if not current or current.get('request_id') == request_id:
self.cache_manager.delete(ON_DEMAND_MAILBOX_KEY)
else:
logger.debug("Newer on-demand request %s arrived while processing "
"%s; leaving it in the mailbox",
current.get('request_id'), request_id)
except (OSError, AttributeError, KeyError) as err:
logger.debug("Could not clear the on-demand request mailbox: %s", err)
def _start_control_server(self) -> None:
"""Serve the control socket (src/ipc/server.py). Never raises.
Its handlers only queue commands; _drain_control_commands applies
them on the render thread, where the mailbox is read.
them on the render thread.
"""
if self._control_server is not None:
return
@@ -2079,7 +1957,8 @@ class DisplayController:
state_hub=hub,
handlers={ControlCommand.ERRORS_CLEAR: apply_error_clear})
except Exception: # pylint: disable=broad-except
logger.exception("Control socket not started; using the file mailbox only")
logger.exception("Control socket not started; the web interface cannot "
"send this display commands")
if self._control_server is not None:
self._start_state_stream(hub)
@@ -2107,10 +1986,8 @@ class DisplayController:
def _drain_control_commands(self) -> None:
"""Apply the commands that arrived over the control socket.
On-demand commands go through _handle_on_demand_request, the
mailbox's own handler, so both ways in behave the same, and a
request that came both ways (a client that timed out and fell back)
has one request id and is processed once. A brightness is applied
On-demand commands go through _handle_on_demand_request, as plugins'
own requests do, so both ways in behave the same. A brightness is applied
here. A plugin reload waits for the top of the next loop pass, where
no plugin is on the stack (_apply_pending_plugin_reloads); until
then the current screen ends early (_plugin_reload_pending).
@@ -2143,7 +2020,7 @@ class DisplayController:
``PluginManager.request_on_demand`` / ``end_on_demand`` (which
BasePlugin's methods of the same names call) build ``request``: the
mailbox's shape, with ``source: 'plugin'`` and the asking plugin's
on-demand request shape, with ``source: 'plugin'`` and the asking plugin's
id. Nothing here touches the panel or the on-demand state; the render
thread applies the request where it applies a socket command
(_drain_control_commands), through _handle_on_demand_request, and is
@@ -2438,105 +2315,35 @@ class DisplayController:
}
command.succeed(dict(result))
def _mailbox_poll_interval(self) -> float:
"""How often the on-demand mailbox is looked at: its old 0.25 s when
it is the only way in, MAILBOX_POLL_INTERVAL_WITH_SOCKET while the
control socket carries the web interface's commands."""
if self._control_server is not None:
return self.MAILBOX_POLL_INTERVAL_WITH_SOCKET
return self.ON_DEMAND_POLL_INTERVAL
def _poll_on_demand_requests(self) -> None:
"""Apply on-demand requests: the control socket's, then the mailbox's.
"""Apply on-demand requests: the control socket's, then plugins' own.
Socket commands are in memory and are applied at once. The file
mailbox (``display_on_demand_request``) is the fallback for a web
interface that could not reach the socket, and the way four plugins
still ask for the screen. It is looked at once per poll interval
(_mailbox_poll_interval), and read only when its file changed
(MailboxWatch): a look that finds nothing new is one stat().
Both are in memory (no disk read), so this has no floor and is
cheap to call every frame. The file mailbox
(``display_on_demand_request``) is gone: nothing reads it, and a
write to it is dropped with a warning (CacheManager).
"""
# Socket commands are already in memory: no disk read, so no floor.
self._drain_control_commands()
now = time.monotonic()
if (self._last_on_demand_poll is not None
and now - self._last_on_demand_poll < self._mailbox_poll_interval()):
return
self._last_on_demand_poll = now
watch = self._on_demand_mailbox
if watch is None:
watch = self._on_demand_mailbox = MailboxWatch(ON_DEMAND_MAILBOX_KEY)
if not watch.changed(self.cache_manager):
return
try:
# Use a long max_age (1 hour) to ensure requests aren't expired before processing
# The request_id check prevents duplicate processing.
#
# memory_ttl=0 is required, not optional: this key is a mailbox the
# web process writes and this process reads. get() defaults the
# in-memory TTL to max_age, so without it the first request read was
# pinned in memory for the full hour and every later poll returned
# that stale copy -- meaning no second on-demand request was honoured
# for an hour, while the API still reported success.
request = self.cache_manager.get(ON_DEMAND_MAILBOX_KEY,
max_age=3600, memory_ttl=0)
except (OSError, RuntimeError, ValueError, TypeError) as err:
watch.forget() # read it again next time
logger.error("Failed to read on-demand request: %s", err, exc_info=True)
return
if not isinstance(request, dict):
return
self._note_mailbox_request(request)
self._handle_on_demand_request(request)
def _note_mailbox_request(self, request: Dict[str, Any]) -> None:
"""Log, once per writer, an on-demand request that came through the
mailbox while the control socket is up.
The web interface writes the mailbox only when the socket fails, so
this is mostly a plugin that writes ``display_on_demand_request``
itself. The mailbox is going away; the log says who still uses it.
"""
if self._control_server is None:
return
writer = request.get('plugin_id') or request.get('mode') or 'unknown'
if not isinstance(writer, str):
writer = 'unknown'
if writer in self._mailbox_writers_logged:
return
self._mailbox_writers_logged = self._mailbox_writers_logged | {writer}
logger.info("On-demand %s request %s (for %s) came through the file mailbox "
"although the control socket is up. The mailbox is deprecated: it "
"is read every %.1fs and will be removed in a future release.",
request.get('action'), request.get('request_id'), writer,
self.MAILBOX_POLL_INTERVAL_WITH_SOCKET)
def _handle_on_demand_request(self, request: Dict[str, Any]) -> None:
"""Process one on-demand request, from the mailbox or the control socket.
"""Process one on-demand request, from the control socket or a plugin.
A socket command carries ``source: 'socket'``, and a plugin's own
request (submit_plugin_on_demand) ``source: 'plugin'``. Only a
mailbox request is removed from the mailbox afterwards: the others
never put anything there, so that would be a disk read and maybe a
delete for nothing.
request (submit_plugin_on_demand) ``source: 'plugin'``.
A plugin's stop ends only that plugin's own session: a plugin
releasing the screen must not end one the user started for
another plugin. (A stop through the mailbox ends any session, as it
always has.)
another plugin. A stop from the socket (the web interface) ends any
session.
"""
request_id = request.get('request_id')
if not request_id:
return
source = request.get('source')
from_mailbox = source not in ('socket', 'plugin')
action = request.get('action')
# For stop requests, always process them (don't check processed_id)
# For stop requests, always process them (no request-id guard)
# This allows stopping even if the same stop request was sent before
if action == 'stop':
if source == 'plugin' and not (
@@ -2562,42 +2369,22 @@ class DisplayController:
# without this the status route kept reporting it until
# the state aged out (120s) or another request came in.
self._clear_on_demand(reason='requested-stop')
# Stop requests are deliberately exempt from the request_id/
# processed_id guards above, so that a second click stops a mode
# that a race left running. Consuming the mailbox is therefore the
# only thing that ends the request: without it the same stop was
# re-read and re-processed on every poll, forever, logging at
# ON_DEMAND_POLL_INTERVAL for the life of the process.
if from_mailbox:
self._consume_on_demand_request(request_id)
# Stop requests are deliberately exempt from the request_id
# guard below, so that a second click stops a mode that a race
# left running.
return
# For start requests, check if already processed. A duplicate in the
# mailbox (a copy of a socket command, or one read before a restart)
# is taken out of it too, so it is not read again.
# A start already processed (a client that sent the same request
# id twice) is not applied a second time.
if request_id == self.on_demand_request_id:
logger.debug("On-demand start request %s already processed (instance check)", request_id)
if from_mailbox:
self._consume_on_demand_request(request_id)
return
# Also check persistent processed_id (for restart scenarios)
processed_request_id = self.cache_manager.get('display_on_demand_processed_id', max_age=3600)
if request_id == processed_request_id:
logger.debug("On-demand start request %s already processed (persisted check)", request_id)
if from_mailbox:
self._consume_on_demand_request(request_id)
logger.debug("On-demand start request %s already processed", request_id)
return
logger.info("Received on-demand request %s: %s (plugin_id=%s, mode=%s, via %s)",
request_id, action, request.get('plugin_id'), request.get('mode'),
'mailbox' if from_mailbox else source)
source or 'unknown')
# Mark as processed BEFORE processing (to prevent duplicate processing)
self.cache_manager.set('display_on_demand_processed_id', request_id, ttl=3600)
self.on_demand_request_id = request_id
if from_mailbox:
self._consume_on_demand_request(request_id)
if action == 'start':
logger.info("Processing on-demand start request for plugin: %s", request.get('plugin_id'))
@@ -3137,6 +2924,54 @@ class DisplayController:
self.force_change = True
logger.info("Live priority ended - resuming rotation at %s", self.current_display_mode)
#: How long a "is any game live?" answer is reused for plugin schedules.
GAME_LIVE_TTL = 5.0
def _any_game_live(self) -> bool:
"""Whether any plugin reports live content, asked at most every GAME_LIVE_TTL s.
Unlike _collect_live_modes this asks every plugin, live priority or
not: a scoreboard without live priority still makes it gameday.
"""
now = time.monotonic()
if now - self._game_live_at >= self.GAME_LIVE_TTL:
self._game_live_at = now
live = False
for plugin in {id(p): p for p in list(self.plugin_modes.values())}.values():
try:
if hasattr(plugin, 'has_live_content') and plugin.has_live_content():
live = True
break
except Exception as e: # pylint: disable=broad-except
logger.warning("Error checking live content for %s: %s",
getattr(plugin, 'plugin_id', plugin), e)
self._game_live = live
return self._game_live
def _hidden_plugins(self) -> FrozenSet[str]:
"""Plugin ids their plugin_schedules entry keeps off the panel right now.
Recomputed when the schedules, the minute or the live-game answer
change. Also Vegas's filter (StreamManager.hidden_plugins), from its
own thread: the schedules are part of the key and the cache is one
tuple, so a call still holding the config from before a reload
cannot leave its answer standing for the new one.
"""
schedules = self.config.get('plugin_schedules')
if not isinstance(schedules, dict) or not schedules:
return frozenset()
now = datetime.now(self._timezone())
game_live = wants_live_scan(schedules) and self._any_game_live()
key = (schedules, now.hour, now.minute, game_live)
cached_key, hidden = self._plugin_hidden_cache
if key != cached_key:
previous = hidden
hidden = hidden_plugins(schedules, now, game_live)
if hidden != previous:
logger.info("Plugin schedules: hidden now %s", sorted(hidden) or "none")
self._plugin_hidden_cache = (key, hidden)
return hidden
def _collect_live_modes(self):
"""Return every currently live-priority mode, in registration order.
@@ -3153,10 +2988,14 @@ class DisplayController:
# Asked once per plugin per scan, not once per mode key: a scoreboard
# registered under several modes computes the same answer each time.
is_live: Dict[int, bool] = {}
# A scoreboard scheduled off does not win the panel by going live.
hidden = self._hidden_plugins()
for mode_name, plugin_instance in self.plugin_modes.items():
if not (hasattr(plugin_instance, 'has_live_priority')
and hasattr(plugin_instance, 'has_live_content')):
continue
if getattr(plugin_instance, 'plugin_id', mode_name) in hidden:
continue
try:
key = id(plugin_instance)
if key not in is_live:
@@ -3615,6 +3454,20 @@ class DisplayController:
logger.debug(f"Found plugin manager for mode {mode}: {type(plugin_instance).__name__}")
return plugin_instance
def _plugin_to_draw(self, plan: ScreenPlan):
"""The plugin to draw ``plan``'s mode, or None to skip it this pass.
_plugin_for_mode, and None for a plugin its plugin_schedules entry
keeps off the panel right now: skipped like an empty screen. An
on-demand plan is the user asking, so it still shows.
"""
plugin = self._plugin_for_mode(plan.mode)
if (plugin is not None and plan.source is not Source.ON_DEMAND
and getattr(plugin, 'plugin_id', plan.mode) in self._hidden_plugins()):
logger.debug("%s is scheduled off, skipping", plan.mode)
return None
return plugin
def _start_screen_handover(self, plugin, active_mode: str) -> bool:
"""Before a screen's first dispatch: if the screen is static, keep
the last scroll's leftovers off its first frame.
@@ -4067,14 +3920,22 @@ class DisplayController:
elif nxt is not state:
self._adopt_state(nxt)
self.force_change = True
logger.info("Switching to mode: %s", self.current_display_mode)
# DEBUG once a whole rotation has been empty (_note_empty_pass):
# it then steps once a second, and every plugin scheduled off
# would otherwise journal a line a second.
logger.log(logging.DEBUG if self._empty_pass_streak >= max(1, len(self.available_modes))
else logging.INFO,
"Switching to mode: %s", self.current_display_mode)
def _still_live(self, mode: Optional[str]) -> bool:
"""The live hold: ``mode``'s plugin has live priority and live content,
so the rotation stays on it."""
so the rotation stays on it. Never for a plugin scheduled off: the
hold would keep the rotation on a mode it skips."""
if mode not in self.plugin_modes:
return False
plugin_instance = self.plugin_modes[mode]
if getattr(plugin_instance, 'plugin_id', mode) in self._hidden_plugins():
return False
if not (hasattr(plugin_instance, 'has_live_priority')
and hasattr(plugin_instance, 'has_live_content')):
return False
@@ -4348,10 +4209,8 @@ class DisplayController:
logger.debug("Loaded plugin modes: %s", list(self.plugin_modes.keys()))
# Handle plugin-based display modes
manager_to_display = self._plugin_for_mode(active_mode)
if not manager_to_display:
logger.warning(f"No plugin manager found for mode {active_mode} - skipping display and rotating to next mode")
manager_to_display = None
# None skips the mode; _plugin_to_draw has logged why.
manager_to_display = self._plugin_to_draw(plan)
# One screen: the first frame, the frame loop, the make-up
# dwell (src/screen_runner.py).
+46 -167
View File
@@ -19,7 +19,7 @@ import uuid
from collections import defaultdict
from dataclasses import dataclass, field
from datetime import datetime, timedelta
from typing import Dict, List, Optional, Any, Callable, Tuple
from typing import Dict, List, Optional, Any, Callable
import logging
from src.exceptions import LEDMatrixError
@@ -449,33 +449,28 @@ def record_error(
# service publishes to the shared cache directory -- the same channel, and the
# same file permissions, as display_current_state and plugin_metrics_snapshot: files
# are 0660 and carry the cache directory's group, so root writes and the web
# user reads, and the other way round for the clear request.
# user reads.
#
# ERROR_SNAPSHOT_KEY written by the display service only
# ERROR_CLEAR_REQUEST_KEY written by the web interface only, as a fallback
#
# A clear goes over the control socket (``errors.clear``): the display applies
# it (clear_before) and republishes the snapshot before it answers. Only when
# the socket cannot carry it (no socket, or a display older than the command)
# does the web interface record a request in the mailbox, which the display
# applies on its next tick; its tick reads that file only when it changed.
# Until it has, the web interface hides whatever the snapshot shows from
# before the cutoff, so a clear takes effect for readers immediately and a
# snapshot published just before the request cannot bring old errors back.
# The web interface never writes the snapshot itself: two writers would race,
# and a snapshot owned by the web user is one more file root's write has to
# replace.
# it (clear_before) and republishes the snapshot before it answers. When the
# socket cannot carry it, the clear fails and the route says so: the
# ``plugin_error_clear_request`` file mailbox it used to fall back to is gone.
# A stopped display's errors go anyway: its next run publishes an empty
# snapshot over the old one. The web interface never writes the snapshot
# itself: two writers would race, and a snapshot owned by the web user is one
# more file root's write has to replace.
ERROR_SNAPSHOT_KEY = "plugin_error_snapshot"
ERROR_CLEAR_REQUEST_KEY = "plugin_error_clear_request"
#: Shortest gap between two snapshot writes, in seconds. A plugin failing in
#: a tight loop changes the aggregator many times a second; the snapshot is
#: rewritten at most this often, and only when something changed.
SNAPSHOT_MIN_INTERVAL = 10.0
#: How often the display service checks for changes and clear requests. A
#: check is an in-memory comparison plus reading one small file.
#: How often the display service checks for changes. A check is an
#: in-memory comparison.
SNAPSHOT_TICK_INTERVAL = 5.0
_SNAPSHOT_RECENT_ERRORS = 20
@@ -544,8 +539,8 @@ class ErrorSnapshotPublisher:
Runs in the display service only. tick() is the whole job; start() just
calls it from a daemon thread every SNAPSHOT_TICK_INTERVAL seconds, which
also means errors recorded while a write was being throttled still reach
the cache once the interval has passed, and a clear request is applied
even when no new error arrives to trigger a publish.
the cache once the interval has passed. A clear (``errors.clear`` over
the control socket) is applied by :meth:`clear_now`.
Nothing here raises: a failure to read or write the cache is logged at
debug and retried on a later tick.
@@ -563,11 +558,6 @@ class ErrorSnapshotPublisher:
self._published_version: Optional[int] = None
self._last_attempt: Optional[float] = None
self._applied_clear_id: Optional[str] = None
# The widest cutoff applied in this process: a clear request at or
# before it has nothing left to clear (see _pending_cutoff).
self._applied_clear_cutoff: Optional[float] = None
from src.cache_manager import MailboxWatch # the display's cache, loaded already
self._mailbox = MailboxWatch(ERROR_CLEAR_REQUEST_KEY)
self._tick_lock = threading.Lock()
self._stop = threading.Event()
self._thread: Optional[threading.Thread] = None
@@ -579,39 +569,9 @@ class ErrorSnapshotPublisher:
cleared = self.aggregator.clear_before(datetime.fromtimestamp(cutoff))
_snapshot_logger.info("Cleared %d plugin error record(s) as requested (%s)",
cleared, request_id)
if self._applied_clear_cutoff is None or cutoff > self._applied_clear_cutoff:
self._applied_clear_cutoff = cutoff
# A malformed request is acknowledged too, so it is not retried forever.
self._applied_clear_id = request_id
return cleared
def _apply_clear_request(self) -> bool:
"""Honour a mailbox clear request we have not applied yet. True if one was.
The mailbox is the fallback for a web interface that could not use
the control socket (``errors.clear``, :meth:`clear_now`). It is read
only when its file changed since the last tick; otherwise a tick
costs one stat().
"""
if not self._mailbox.changed(self.cache_manager):
return False
try:
request = self.cache_manager.get(ERROR_CLEAR_REQUEST_KEY, max_age=None, memory_ttl=0)
except Exception:
self._mailbox.forget()
raise
if not isinstance(request, dict):
return False
request_id = request.get("request_id")
if not isinstance(request_id, str) or not request_id or request_id == self._applied_clear_id:
return False
try:
cutoff = float(request.get("cutoff"))
except (TypeError, ValueError):
cutoff = float("nan")
self._clear(request_id, cutoff)
return True
def clear_now(self, request_id: str, cutoff: float) -> int:
"""``errors.clear`` over the control socket: apply a clear at once and
republish the snapshot, so the web interface's next read has it.
@@ -629,18 +589,15 @@ class ErrorSnapshotPublisher:
self._last_attempt = now
snapshot = self.aggregator.build_snapshot()
snapshot["applied_clear_id"] = self._applied_clear_id
snapshot["applied_clear_cutoff"] = self._applied_clear_cutoff
self.cache_manager.set(ERROR_SNAPSHOT_KEY, snapshot)
self._published_version = version
def tick(self) -> bool:
"""Apply a pending clear and publish if due. True if a snapshot was written."""
"""Publish if due. True if a snapshot was written."""
with self._tick_lock:
try:
cleared = self._apply_clear_request()
version = self.aggregator.version
now = self._clock()
if not cleared:
if version == self._published_version:
return False
if (self._last_attempt is not None
@@ -716,16 +673,13 @@ def apply_error_clear(request_id: str, args: Any) -> Dict[str, Any]:
# --- Reading side (web interface) -------------------------------------------
def read_error_report(cache_manager: Any) -> Tuple[Optional[Dict[str, Any]], Optional[Dict[str, Any]]]:
"""The display service's latest snapshot and the latest clear request.
def read_error_report(cache_manager: Any) -> Optional[Dict[str, Any]]:
"""The display service's latest snapshot, or None before it has published.
memory_ttl=0: both keys are written by the other process, so only the
file is current.
memory_ttl=0: the display writes the key, so only the file is current.
"""
snapshot = cache_manager.get(ERROR_SNAPSHOT_KEY, max_age=None, memory_ttl=0)
clear_request = cache_manager.get(ERROR_CLEAR_REQUEST_KEY, max_age=None, memory_ttl=0)
return (snapshot if isinstance(snapshot, dict) else None,
clear_request if isinstance(clear_request, dict) else None)
return snapshot if isinstance(snapshot, dict) else None
def _epoch(iso: Any) -> Optional[float]:
@@ -738,36 +692,6 @@ def _epoch(iso: Any) -> Optional[float]:
return None
def _pending_cutoff(snapshot: Optional[Dict[str, Any]],
clear_request: Optional[Dict[str, Any]]) -> Optional[float]:
"""The cutoff of a clear the snapshot has not applied yet, if any."""
if not clear_request:
return None
request_id = clear_request.get("request_id")
if not request_id:
return None
if snapshot is not None and snapshot.get("applied_clear_id") == request_id:
return None
try:
cutoff = float(clear_request.get("cutoff"))
except (TypeError, ValueError):
return None
if not math.isfinite(cutoff):
return None
# A wider clear has been applied since (over the control socket): this
# older request has nothing left to hide.
applied = snapshot.get("applied_clear_cutoff") if snapshot is not None else None
if (isinstance(applied, (int, float)) and not isinstance(applied, bool)
and applied >= cutoff):
return None
return cutoff
def _is_after(item: Any, field_name: str, cutoff: float) -> bool:
when = _epoch(item.get(field_name)) if isinstance(item, dict) else None
return when is not None and when > cutoff
def _empty_summary(snapshot: Optional[Dict[str, Any]]) -> Dict[str, Any]:
return {
"session_start": snapshot.get("session_start") if snapshot else None,
@@ -780,11 +704,11 @@ def _empty_summary(snapshot: Optional[Dict[str, Any]]) -> Dict[str, Any]:
}
def error_summary_from_report(snapshot: Optional[Dict[str, Any]],
clear_request: Optional[Dict[str, Any]]) -> Dict[str, Any]:
def error_summary_from_report(snapshot: Optional[Dict[str, Any]]) -> Dict[str, Any]:
"""The /errors/summary payload: get_error_summary()'s shape plus
``generated_at``, ``snapshot_available`` and ``clear_pending``."""
cutoff = _pending_cutoff(snapshot, clear_request)
``generated_at``, ``snapshot_available`` and ``clear_pending`` (always
False now: a clear is applied before its route answers; kept for API
compatibility)."""
summary = _empty_summary(snapshot)
if snapshot is not None:
for name, default in summary.items():
@@ -793,32 +717,17 @@ def error_summary_from_report(snapshot: Optional[Dict[str, Any]],
isinstance(default, float) and isinstance(value, int)) or (
name == "session_start" and isinstance(value, str)):
summary[name] = value
if cutoff is not None:
recent = summary["recent_errors"]
newest = _epoch(recent[-1].get("timestamp")) if recent and isinstance(recent[-1], dict) else None
if newest is None or newest <= cutoff:
# Everything the display has reported predates the clear.
summary = _empty_summary(snapshot)
else:
# Only part of it does. The lists can be filtered exactly; the
# counts cannot, and stay as reported until the display
# applies the clear (clear_pending says so).
summary["recent_errors"] = [r for r in recent if _is_after(r, "timestamp", cutoff)]
summary["active_patterns"] = {
k: p for k, p in summary["active_patterns"].items()
if _is_after(p, "last_seen", cutoff)
}
summary["generated_at"] = snapshot.get("generated_at") if snapshot else None
summary["snapshot_available"] = snapshot is not None
summary["clear_pending"] = cutoff is not None
summary["clear_pending"] = False
return summary
def plugin_health_from_report(snapshot: Optional[Dict[str, Any]],
clear_request: Optional[Dict[str, Any]],
plugin_id: str) -> Dict[str, Any]:
"""The /errors/plugin/<id> payload: get_plugin_health()'s shape plus
``generated_at``, ``snapshot_available`` and ``clear_pending``."""
``generated_at``, ``snapshot_available`` and ``clear_pending`` (always
False, as in error_summary_from_report)."""
health: Dict[str, Any] = {
"plugin_id": plugin_id,
"status": "healthy",
@@ -827,19 +736,15 @@ def plugin_health_from_report(snapshot: Optional[Dict[str, Any]],
"recent_error_count": 0,
"last_error": None,
}
cutoff = _pending_cutoff(snapshot, clear_request)
table = snapshot.get("plugin_health") if snapshot else None
entry = table.get(plugin_id) if isinstance(table, dict) else None
if isinstance(entry, dict):
# last_error is the plugin's newest error: if even that predates a
# pending clear, so does everything else the display reported for it.
if cutoff is None or _is_after(entry.get("last_error"), "timestamp", cutoff):
for name in ("status", "total_errors", "error_types", "recent_error_count", "last_error"):
if name in entry:
health[name] = entry[name]
health["generated_at"] = snapshot.get("generated_at") if snapshot else None
health["snapshot_available"] = snapshot is not None
health["clear_pending"] = cutoff is not None
health["clear_pending"] = False
return health
@@ -861,61 +766,35 @@ def _count_cleared(summary: Dict[str, Any], cutoff: float) -> Optional[int]:
#: ``send(request_id, cutoff)`` hands a clear to the display over the control
#: socket and returns its ErrorsClearResult, or None when the socket could
#: not carry it and the mailbox should be written instead. Any exception it
#: raises reaches the caller: the display had the request and failed it.
ClearSender = Callable[[str, float], Optional[Dict[str, Any]]]
#: socket and returns its ErrorsClearResult. It raises when the socket could
#: not carry it or the display failed it (``src.ipc.client.ControlError``).
ClearSender = Callable[[str, float], Dict[str, Any]]
def request_error_clear(cache_manager: Any, cutoff: float,
send: Optional[ClearSender] = None) -> Dict[str, Any]:
send: ClearSender) -> Dict[str, Any]:
"""Ask the display service to forget errors recorded at or before ``cutoff``.
Over the control socket when ``send`` is given and carries it: the
display applies the clear and republishes its snapshot before it
answers, so nothing is written here. Otherwise (no socket, or a display
older than ``errors.clear``) a request is written to the
``plugin_error_clear_request`` mailbox, which the display applies on
its next tick, and readers hide the cleared errors until then.
Over the control socket: the display applies the clear and republishes
its snapshot before it answers, so nothing is written here. Whatever
``send`` raises reaches the caller.
Returns ``request_id``, ``cutoff`` (ISO, local time), ``cleared_count``,
``clear_requested``, ``applied`` (the display has already cleared them)
and ``transport`` (``socket`` or ``mailbox``). ``cleared_count`` is the
display's own count over the socket, else an estimate from the snapshot
(see _count_cleared). Raises OSError when a mailbox request did not reach
the shared cache, since a cache without a usable directory accepts set()
and keeps nothing.
A request the display has not applied yet is only ever widened: a later,
narrower one ("older than 24 hours" after "everything") overwriting it
would otherwise bring back the errors the first one hid.
Returns ``request_id``, ``cutoff`` (ISO, local time), ``cleared_count``
(the display's own count, else an estimate from the snapshot; see
_count_cleared), and ``clear_requested``, ``applied`` and ``transport``,
which are always True, True and ``"socket"`` now and are kept for API
compatibility.
"""
snapshot, clear_request = read_error_report(cache_manager)
pending = _pending_cutoff(snapshot, clear_request)
if pending is not None:
cutoff = max(cutoff, pending)
before = error_summary_from_report(snapshot, clear_request)
before = error_summary_from_report(read_error_report(cache_manager))
request_id = uuid.uuid4().hex
answer = {
result = send(request_id, cutoff)
count = result.get("cleared") if isinstance(result, dict) else None
return {
"clear_requested": True,
"request_id": request_id,
"cutoff": datetime.fromtimestamp(cutoff).isoformat(),
"applied": True,
"transport": "socket",
"cleared_count": (count if isinstance(count, int) and not isinstance(count, bool)
else _count_cleared(before, cutoff)),
}
if send is not None:
result = send(request_id, cutoff)
if result is not None:
count = result.get("cleared")
return dict(answer, applied=True, transport="socket",
cleared_count=count if isinstance(count, int) and not isinstance(count, bool)
else _count_cleared(before, cutoff))
request = {
"request_id": request_id,
"cutoff": cutoff,
"requested_at": time.time(),
}
cache_manager.set(ERROR_CLEAR_REQUEST_KEY, request)
stored = cache_manager.get(ERROR_CLEAR_REQUEST_KEY, max_age=None, memory_ttl=0)
if not isinstance(stored, dict) or stored.get("request_id") != request["request_id"]:
raise OSError("the clear request was not stored in the shared cache")
return dict(answer, applied=False, transport="mailbox",
cleared_count=_count_cleared(before, cutoff))
+23 -28
View File
@@ -5,11 +5,11 @@ a refused or timed-out connection, a reply that breaks the contract, or an
error the display returned -- raises :class:`ControlError` with a short
``reason``. Nothing here blocks for longer than ``timeout`` in total.
Whether the caller may then write the file mailbox instead is
:func:`should_fall_back`: only when the display never took the request (it
could not be reached, or it is too old to know the command). A display that
took the request and then failed, refused or went quiet is answered as
that, not posted a second time through the mailbox.
There is no other way to reach the display: the file mailboxes the web
interface used to fall back to are gone. :func:`display_not_listening` tells
a caller when no display is listening yet (it is stopped, or still
starting), the one case where sending the same request again later, once a
display is up, can work.
"""
from __future__ import annotations
@@ -43,8 +43,7 @@ from src.ipc.contract import (
#: Total budget for one request: connect, send and the reply. The display
#: answers from a thread that does no rendering, normally within a few
#: milliseconds; this only bounds a wedged one. The web route then falls back
#: to the mailbox, so a timeout costs this much latency and nothing else.
#: milliseconds; this only bounds a wedged one.
DEFAULT_TIMEOUT_SECONDS = 1.0
@@ -74,28 +73,25 @@ class ControlError(Exception):
#: Answers from a display that read the request but does not speak it: one
#: older than the command (an upgrade in progress) or the protocol version.
#: It did nothing, so the mailbox is the way to reach it.
UPGRADE_REASONS = frozenset({ErrorCode.UNKNOWN_COMMAND, ErrorCode.UNSUPPORTED_VERSION})
#: Transport reasons that mean nothing is listening at the socket: no socket
#: file (the display is stopped, or has not reached its run loop), or a file
#: nobody accepts on (a stale socket) or that this user may not open.
NOT_LISTENING_REASONS = frozenset({'no_socket', 'refused'})
def should_fall_back(error: BaseException) -> bool:
"""May the caller write the file mailbox after ``error``?
Yes when the display never took the request: there is no socket (the
display is stopped, predates the socket, or it is switched off), the
connection was refused or timed out, the display turned the connection
away before reading it, or it is too old to know the command
(:data:`UPGRADE_REASONS`). Also for an error that is not a
:class:`ControlError` (a bug in the client), as before.
No once the display had the request: a ``busy`` queue, ``invalid_args``,
an ``internal`` error, or a timeout or hang-up after the request was
sent. The display may have applied it, or would refuse it from the
mailbox too, so a second copy there only hides the failure.
def display_not_listening(error: BaseException) -> bool:
"""True when ``error`` says no display took the request because none is
listening: it never reached one (``sent`` is False) and the reason is in
:data:`NOT_LISTENING_REASONS`. A display that is started, or finishes
starting, may take the same request later. False for everything else:
a display that had the request and failed it, one too old to know the
command, a client that cannot use the socket at all (``disabled``,
``unsupported``), and an exception that is not a :class:`ControlError`.
"""
if not isinstance(error, ControlError):
return True
return not error.sent or error.reason in UPGRADE_REASONS
return (isinstance(error, ControlError) and not error.sent
and error.reason in NOT_LISTENING_REASONS)
def request(cmd: str, args: Optional[Mapping[str, Any]] = None, *,
@@ -218,8 +214,8 @@ def on_demand_start(request_id: str, plugin_id: Optional[str], mode: Optional[st
paths: Optional[Sequence[str]] = None) -> Dict[str, Any]:
"""Ask the display to show a plugin now. Returns the ack; raises :class:`ControlError`.
``request_id`` doubles as the on-demand request id, so a request that a
timed-out caller then also writes to the mailbox is processed only once.
``request_id`` doubles as the on-demand request id, so a request sent
twice with the same id is processed only once.
"""
args = {'plugin_id': plugin_id, 'mode': mode, 'duration': duration, 'pinned': pinned}
return request(Command.ON_DEMAND_START, args, request_id=request_id,
@@ -281,8 +277,7 @@ def errors_clear(request_id: str, cutoff: float, *,
Returns :class:`~src.ipc.contract.ErrorsClearResult` once it is done.
Raises :class:`ControlError`: ``unknown_command`` from a display older
than the command, which still reads the ``plugin_error_clear_request``
mailbox.
than the command.
"""
return request(Command.ERRORS_CLEAR, {'cutoff': cutoff}, request_id=request_id,
timeout=timeout, paths=paths)
+10 -11
View File
@@ -85,8 +85,8 @@ DEFAULT_SOCKET_PATH = DEFAULT_SOCKET_DIR + '/' + SOCKET_NAME
#: Overrides the socket path for both processes (a dev checkout, a second
#: instance, tests). One of :data:`DISABLED_VALUES` turns the socket off: the
#: display does not serve it and the web interface goes straight to the
#: file mailbox.
#: display does not serve it, and the web interface cannot send it commands
#: (it still reads the state the display writes to the cache).
SOCKET_PATH_ENV = 'LEDMATRIX_CONTROL_SOCKET'
DISABLED_VALUES = frozenset({'off', '0', 'false', 'no', 'none', 'disabled'})
@@ -375,8 +375,8 @@ def _optional_name(args: Mapping[str, Any], key: str) -> Optional[str]:
def _optional_duration(value: Any) -> Optional[float]:
"""Seconds, or None for "until stopped". 0 means the same as None.
Numbers and numeric strings are accepted, the same as the REST route and
the file mailbox take them; anything else is refused rather than guessed.
Numbers and numeric strings are accepted, the same as the REST route
takes them; anything else is refused rather than guessed.
"""
if value is None or value == '':
return None
@@ -415,7 +415,7 @@ class HelloArgs:
class OnDemandStartArgs:
"""``on_demand.start``: show a plugin (or one of its modes) now.
The same fields the file mailbox carries. At least one of ``plugin_id``
The same fields as the REST route's body. At least one of ``plugin_id``
and ``mode`` is required; the display resolves the other.
"""
plugin_id: Optional[str] = None
@@ -600,13 +600,12 @@ def parse_args(cmd: str, args: Mapping[str, Any]) -> CommandArgs:
def on_demand_request(request_id: str, args: Union[OnDemandStartArgs, OnDemandStopArgs],
timestamp: float) -> Dict[str, Any]:
"""The file-mailbox payload for a queued on-demand command.
"""The on-demand request dict for a queued on-demand command.
The display hands socket commands to the same code that handles the
mailbox (``DisplayController._handle_on_demand_request``), so a command
behaves identically whichever way it arrived, and a request that came
both ways (a client that timed out and fell back) is processed once: the
request id is the same.
The display hands socket commands to the same code that handles
plugins' own requests (``DisplayController._handle_on_demand_request``),
so a command behaves identically whichever way it arrived. (This was
the file mailbox's payload, which the display no longer reads.)
"""
if isinstance(args, OnDemandStartArgs):
return {'request_id': request_id, 'action': 'start', 'plugin_id': args.plugin_id,
+13 -12
View File
@@ -3,9 +3,9 @@
A small threaded server on a Unix stream socket (``/run/ledmatrix/control.sock``
by default; see :mod:`src.ipc.contract` for the protocol). It never touches
rendering: a command that changes the panel is validated, put on a bounded
queue and acknowledged, and the render thread drains that queue at the point
where it reads the file mailbox (``DisplayController._poll_on_demand_requests``),
handing each command to the same code. Queries (``on_demand.status``) are
queue and acknowledged, and the render thread drains that queue
(``DisplayController._poll_on_demand_requests``), handing each on-demand
command to the code that handles plugins' own requests. Queries (``on_demand.status``) are
answered from a snapshot callable the display provides, and the few commands
that touch nothing the render thread owns (``errors.clear``) by a handler the
display registers, on the connection thread.
@@ -110,8 +110,8 @@ MAX_CLIENTS = 8
#: Commands waiting for the render thread. It drains them at least every
#: 0.25 s, so a full queue means the render thread is stuck, and the client
#: is told ``busy`` instead of piling up work. The mailbox would not be read
#: either, so the web interface reports the failure rather than fall back.
#: is told ``busy`` instead of piling up work, and the web interface reports
#: the failure.
QUEUE_SIZE = 16
#: Timeout for one recv()/send() on a connection.
@@ -185,7 +185,7 @@ class QueuedCommand:
outcome: Optional[CommandOutcome] = field(default=None, compare=False, repr=False)
def as_on_demand_request(self) -> Dict[str, Any]:
"""The mailbox-shaped payload the display's on-demand handler takes."""
"""The on-demand request dict the display's on-demand handler takes."""
if not isinstance(self.args, (OnDemandStartArgs, OnDemandStopArgs)):
raise TypeError(f'{self.cmd} is not an on-demand command')
return on_demand_request(self.request_id, self.args, self.received_at)
@@ -619,8 +619,8 @@ class ControlServer:
def start(self) -> bool:
"""Bind and start serving. False (logged) when the socket cannot be served.
Never raises: without the socket the web interface uses the file
mailbox, exactly as before.
Never raises: without the socket the display runs, but the web
interface cannot send it commands.
"""
if not socket_supported():
logger.debug("Control socket not started: no Unix sockets on this platform")
@@ -632,7 +632,7 @@ class ControlServer:
self._bind()
except OSError as e:
logger.warning("Control socket not started at %s (%s); the web interface "
"will use the file mailbox", self.path, e)
"cannot send this display commands", self.path, e)
self._close_socket()
return False
self._stopping.clear()
@@ -1134,12 +1134,13 @@ def start_control_server(status_provider: Optional[StatusProvider] = None,
"""Start the display's control socket, or return None when it can't run.
None covers Windows, ``LEDMATRIX_CONTROL_SOCKET=off`` and any failure to
bind; in every case the web interface falls back to the file mailbox
and to the cache keys the display still writes.
bind; in every case the web interface cannot send the display commands,
and reads the cache keys the display still writes.
"""
path = server_socket_path(environ)
if path is None:
logger.debug("Control socket disabled or unsupported here; using the file mailbox only")
logger.debug("Control socket disabled or unsupported here; the web interface "
"cannot send this display commands")
return None
server = ControlServer(path, status_provider, resolve_socket_group(cache_dir),
state_hub=state_hub, handlers=handlers)
+104
View File
@@ -0,0 +1,104 @@
"""Schedule windows: the display's on/off schedule and per-plugin schedules.
A schedule is the dict config.json keeps under ``schedule``::
{"enabled": true, "mode": "global" | "per-day",
"start_time": "07:00", "end_time": "23:00",
"days": {"monday": {"enabled": true, "start_time": ..., "end_time": ...}, ...}}
``config['plugin_schedules']`` maps a plugin id to one of these, the window in
which that plugin may take a turn on the panel, plus ``hide_when_live``: hide
the plugin while any plugin reports a live game ("no news or stocks on
gameday"). A plugin without an entry always shows. Hiding is a skip at
display time, never the plugin's ``enabled`` flag: flipping that unloads and
reloads the plugin and throws away its data.
"""
import logging
from datetime import datetime, time
from typing import Any, FrozenSet, Mapping, Tuple
logger = logging.getLogger(__name__)
def in_window(start: time, end: time, now: time) -> bool:
"""Whether ``now`` is within [start, end), a window that may span midnight.
Half-open: on from the start minute, off at exactly the end minute.
A closed end made the end minute count as inside, and because ``now``
carries seconds, only a check at hh:mm:00.000 saw it that way -- so
whether the panel went off at the start or the end of that minute
depended on when the minute's one check ran. ``start == end`` is an
empty window, as it effectively was before.
"""
if start <= end:
return start <= now < end
return now >= start or now < end
def schedule_state(schedule: Mapping[str, Any], now: datetime) -> Tuple[bool, str]:
"""Whether ``now`` is inside ``schedule``, and a phrase saying why.
No schedule, or ``enabled: false``, is always inside; a schedule without
an ``enabled`` key counts as enabled. ``mode`` "global" ignores ``days``;
"per-day" (or "per_day") uses today's entry and falls back to the global
times when today has none; with no ``mode`` (older configs) a ``days``
entry for today wins. A disabled day is outside all day. A time that does
not parse is logged and counts as inside, so a bad edit never blanks the
panel.
"""
if not schedule or not schedule.get('enabled', True):
return True, 'no schedule'
day = now.strftime('%A').lower()
# Wrong-shaped values (the raw config editor allows any) are treated as
# missing rather than raised: this runs in the display loop.
days = schedule.get('days')
if not isinstance(days, Mapping):
days = {}
mode = schedule.get('mode')
mode = mode.replace('_', '-') if isinstance(mode, str) else None
# An explicit Global mode beats a stray 'days' dict left from a past
# per-day setup.
per_day = mode != 'global' and isinstance(days.get(day), Mapping)
if per_day:
day_config = days[day]
if not day_config.get('enabled', True):
return False, f'{day} is disabled'
start_str = day_config.get('start_time', '07:00')
end_str = day_config.get('end_time', '23:00')
kind = f'per-day ({day})'
else:
start_str = schedule.get('start_time', '07:00')
end_str = schedule.get('end_time', '23:00')
kind = 'global'
try:
start = datetime.strptime(start_str, '%H:%M').time()
end = datetime.strptime(end_str, '%H:%M').time()
except (TypeError, ValueError) as e:
logger.warning("Invalid %s schedule: %s (start: %s, end: %s). Treating as inside.",
kind, e, start_str, end_str)
return True, f'invalid {kind} schedule'
return in_window(start, end, now.time()), f'{kind} window {start_str} - {end_str}'
def wants_live_scan(plugin_schedules: Mapping[str, Any]) -> bool:
"""Whether any plugin hides during live games, so live content is worth asking about."""
return any(isinstance(s, Mapping) and s.get('hide_when_live')
for s in plugin_schedules.values())
def hidden_plugins(plugin_schedules: Mapping[str, Any], now: datetime,
game_live: bool) -> FrozenSet[str]:
"""The plugin ids that may not show at ``now``.
A plugin is hidden outside its schedule window, or while ``game_live``
when its entry sets ``hide_when_live``.
"""
return frozenset(
plugin_id for plugin_id, schedule in plugin_schedules.items()
if isinstance(schedule, Mapping)
and ((game_live and schedule.get('hide_when_live'))
or not schedule_state(schedule, now)[0]))
+7 -7
View File
@@ -1114,16 +1114,16 @@ class BasePlugin(ABC):
Returns:
The request id once the display has queued it, or None when
there is no display in this process to ask (the web interface,
scripts/check_plugin.py) or its queue is full. A plugin that
also runs on cores without this method writes the
``display_on_demand_request`` mailbox on None, as before; see
"On-demand display" in docs/PLUGIN_API_REFERENCE.md.
scripts/check_plugin.py) or its queue is full. Nothing else can
take the request then: the ``display_on_demand_request`` file
mailbox older cores read is gone, and a write to it is dropped
with a warning. See "On-demand display" in
docs/PLUGIN_API_REFERENCE.md.
Example::
if not (hasattr(self, 'request_on_demand')
and self.request_on_demand(mode='my_alert', duration=15)):
self._write_on_demand_mailbox(...) # older cores
if self.request_on_demand(mode='my_alert', duration=15) is None:
self.logger.info("No display to show the alert on")
"""
request = getattr(getattr(self, 'plugin_manager', None), 'request_on_demand', None)
if not callable(request):
+1 -1
View File
@@ -1869,7 +1869,7 @@ class PluginManager:
"""Route plugins' on-demand requests to ``handler`` (None: nowhere).
The display controller sets its ``submit_plugin_on_demand`` here
before any plugin loads. The handler takes a mailbox-shaped request
before any plugin loads. The handler takes an on-demand request dict
from any thread, queues it for the render thread and returns True,
or False when it could not. A plugin manager with no handler (the
web interface's, a test's, scripts/check_plugin.py's) has no screen
+56 -12
View File
@@ -23,6 +23,7 @@ from typing import Optional, Dict, Any, FrozenSet, List, Callable, TYPE_CHECKING
from src import display_watchdog
from src.common import render_gate
from src.plugin_system.base_plugin import finite_seconds
from src.screen_runner import HIGH_FPS_INTERVAL, STATIC_INTERVAL
from src.vegas_mode.config import VegasModeConfig
from src.vegas_mode.elements import LiveEpochs
from src.vegas_mode.plugin_adapter import PluginAdapter
@@ -302,7 +303,9 @@ class VegasModeCoordinator:
# Initialize stream manager
if not self.stream_manager.initialize():
logger.error("Failed to initialize stream manager")
# Only fails with no plugin to show (every one excluded or
# scheduled off), which initialize() has logged once.
logger.debug("Vegas not started: no plugins to scroll")
return False
# Compose initial content
@@ -970,25 +973,38 @@ class VegasModeCoordinator:
# turn rather than stall the marquee.
get_lock = getattr(self.plugin_manager, 'get_plugin_lock', None)
plugin_lock = get_lock(plugin_id) if get_lock else None
if plugin_lock is not None and not plugin_lock.acquire(
timeout=self.STATIC_LOCK_TIMEOUT):
if not self._draw_static(plugin, plugin_lock, self.STATIC_LOCK_TIMEOUT,
force_clear=True):
logger.info("Static pause skipped for %s: its update() is still running",
plugin_id)
return True
try:
plugin.display(force_clear=True)
finally:
if plugin_lock is not None:
plugin_lock.release()
self.display_manager.update_display()
# Wait for the plugin's display duration. Monotonic, like the
# iteration clock: an NTP step on an RTC-less Pi would otherwise
# end the pause at once or stretch it by the correction.
duration = self._static_pause_duration(plugin)
start = time.monotonic()
# Redraw for the whole pause at the rotation's own rate. One
# display() used to be all a pause got, so a plugin that rotates
# its screens or animates was frozen on its first frame.
interval = (HIGH_FPS_INTERVAL if self._static_needs_high_fps(plugin)
else STATIC_INTERVAL)
next_draw = start + interval
next_check = start
while (now := time.monotonic()) - start < duration:
if now >= next_draw:
next_draw = max(next_draw, now) + interval
# No wait for the lock: update() holding it costs this
# frame only, and the next one tries again.
self._draw_static(plugin, plugin_lock, 0)
if now < next_check:
time.sleep(max(0.0, min(next_draw, next_check) - time.monotonic()))
continue
# Every 0.1 s, however fast the redraws: the live-priority
# check asks the plugins.
next_check = now + 0.1
while time.monotonic() - start < duration:
# Check for interruptions
if self._should_stop:
logger.info("Static pause interrupted by stop request")
@@ -1003,9 +1019,8 @@ class VegasModeCoordinator:
logger.info("Static pause interrupted by the display controller")
return False
# Sleep in small increments to remain responsive
time.sleep(0.1)
display_watchdog.beat()
time.sleep(max(0.0, min(next_draw, next_check) - time.monotonic()))
logger.info(
"Static pause completed for %s after %.1fs",
@@ -1021,6 +1036,35 @@ class VegasModeCoordinator:
return True
def _draw_static(self, plugin: 'BasePlugin', plugin_lock: Any,
timeout: float, **kwargs: Any) -> bool:
"""One display() of a static pause, under the plugin's lock like
every other display() call: without it this could draw while the
update worker is inside the plugin's update(). False when update()
held the lock past ``timeout`` and nothing was drawn."""
if plugin_lock is not None and not plugin_lock.acquire(timeout=timeout):
return False
try:
plugin.display(**kwargs)
finally:
if plugin_lock is not None:
plugin_lock.release()
self.display_manager.update_display()
return True
@staticmethod
def _static_needs_high_fps(plugin: 'BasePlugin') -> bool:
"""Whether the rotation would run ``plugin`` in its high-FPS loop:
DisplayController._needs_high_fps's rule. Older static-image
versions, which do not declare needs_high_fps, get high FPS for
their GIFs there too."""
declared = getattr(plugin, 'needs_high_fps', None)
if declared is not None:
return bool(declared)
if getattr(plugin, 'plugin_id', None) == 'static-image':
return True
return bool(getattr(plugin, 'enable_scrolling', False))
def _static_pause_duration(self, plugin: 'BasePlugin') -> float:
"""Seconds a static pause shows ``plugin``: its display duration,
read the way the rotation reads it.
+24 -3
View File
@@ -16,7 +16,7 @@ BasePlugin.get_vegas_participation):
import logging
import threading
import time
from typing import Optional, List, Dict, Deque, Tuple, TYPE_CHECKING
from typing import Callable, Optional, List, Dict, Deque, FrozenSet, Tuple, TYPE_CHECKING
from collections import deque
from dataclasses import dataclass, field
from PIL import Image
@@ -75,6 +75,16 @@ class StreamManager:
self.config = config
self.plugin_manager = plugin_manager
self.plugin_adapter = plugin_adapter
# Plugin ids to leave out right now (config['plugin_schedules']);
# the display controller sets it. Read at each list refresh.
# A change to it rebuilds the list at the next refresh() even inside
# the refresh interval; segments already composed still scroll out.
self.hidden_plugins: Optional[Callable[[], FrozenSet[str]]] = None
self._applied_hidden: FrozenSet[str] = frozenset()
# True once "no plugins" has been logged, until the list fills again:
# with every plugin scheduled off the display retries every second,
# and three log lines a second is a full journal on an SD card.
self._empty_logged = False
# Segments composed into the current cycle (swap mode only).
self._active_buffer: Deque[ContentSegment] = deque()
@@ -106,8 +116,11 @@ class StreamManager:
self._refresh_plugin_list()
if not self._ordered_plugins:
if not self._empty_logged:
logger.warning("No plugins available for Vegas scroll")
self._empty_logged = True
return False
self._empty_logged = False
# Fill the buffer to a whole cycle's worth of plugins. This used to be
# buffer_ahead + 1, which conflated prefetch depth with cycle size and
@@ -268,7 +281,9 @@ class StreamManager:
Called periodically to pick up new plugins or config changes.
"""
current_time = time.time()
if current_time - self._last_refresh < self._refresh_interval:
hidden = self.hidden_plugins() if self.hidden_plugins else frozenset()
if (current_time - self._last_refresh < self._refresh_interval
and hidden == self._applied_hidden):
return
self._last_refresh = current_time
@@ -295,10 +310,15 @@ class StreamManager:
# own thread (DisplayController._start_plugin_reload).
plugins = list(self.plugin_manager.plugins.items())
loaded = len(plugins)
hidden = self.hidden_plugins() if self.hidden_plugins else frozenset()
self._applied_hidden = hidden
for plugin_id, plugin in plugins:
if not getattr(plugin, 'enabled', False):
logger.debug("[%s] Vegas: skipped (not enabled)", plugin_id)
continue
if plugin_id in hidden:
logger.debug("[%s] Vegas: skipped (scheduled off)", plugin_id)
continue
# 'pause' plugins stay in the rotation: they pause the scroll
# for their turn rather than contributing to it.
@@ -329,7 +349,8 @@ class StreamManager:
slots = (f", {len(ordered_plugins)} slots"
if len(ordered_plugins) != len(set(ordered_plugins)) else "")
logger.info(
logger.log(
logging.DEBUG if (self._empty_logged and not ordered_plugins) else logging.INFO,
"Vegas rotation: %d of %d loaded plugin(s)%s: %s",
len(set(ordered_plugins)), loaded, slots, ', '.join(ordered_plugins)
)
+15 -9
View File
@@ -155,13 +155,6 @@ class FakeCache:
self._writes += 1
self._written[key] = self._writes
def file_signature(self, key):
"""CacheManager.file_signature: None without a file, else a value
that changes with every write."""
if key not in self.data:
return None
return (self._written.get(key, 0), 0, 0)
def delete(self, key):
self.data.pop(key, None)
@@ -729,10 +722,23 @@ class RunLoopHarness:
self.controller.available_modes.append(mode)
def on_demand_request(self, t: float, request_id: str, action: str = "start", **fields):
"""An on-demand start or stop from the web interface at ``t``: a
command on the control socket (served for the run if no test did),
which is the only way the web interface reaches the display."""
from src.ipc.contract import Command, parse_args
from src.ipc.server import QueuedCommand
server = self.controller._control_server
if not isinstance(server, FakeControlServer):
server = self.control_socket()
cmd = Command.ON_DEMAND_START if action == "start" else Command.ON_DEMAND_STOP
command = QueuedCommand(request_id=request_id, cmd=cmd,
args=parse_args(cmd, fields if action == "start" else {}),
received_at=0.0)
def post():
self.log("request", f"{action}:{request_id}")
self.cache.set("display_on_demand_request",
{"request_id": request_id, "action": action, **fields})
server.queue.append(command)
self.clock.at(t, post)
def restore_on_demand(self, plugin_id: str, mode: Optional[str] = None,
+14
View File
@@ -328,6 +328,20 @@ def _hermetic_control_socket(monkeypatch):
monkeypatch.setenv(SOCKET_PATH_ENV, 'off')
@pytest.fixture(autouse=True)
def _no_pending_on_demand_dispatch():
"""Drop the web process's on-demand dispatcher after each test, so a
start one test left pending is not still being sent in the next."""
yield
module = sys.modules.get('web_interface.on_demand_dispatch')
if module is None:
return
dispatcher = module.current()
if dispatcher is not None:
dispatcher.cancel('test-teardown')
module.reset_for_tests()
@pytest.fixture(autouse=True)
def _hermetic_unit_refresh(monkeypatch, tmp_path_factory):
"""Keep updates' systemd unit refresh off the host.
+17
View File
@@ -159,6 +159,23 @@
"POST"
]
],
[
"/api/v3/config/plugin-schedules",
"api_v3.get_plugin_schedules",
[
"GET",
"HEAD",
"OPTIONS"
]
],
[
"/api/v3/config/plugin-schedules",
"api_v3.save_plugin_schedule",
[
"OPTIONS",
"POST"
]
],
[
"/api/v3/config/raw/main",
"api_v3.save_raw_main_config",
+3 -3
View File
@@ -1,16 +1,16 @@
{
"screens": [
[0.0, "clock", 20.0, "duration", 20, false],
[20.0, "weather", 5.0, "on-demand-start", 6, true],
[20.0, "weather", 5.0, "on-demand-start", 5, true],
[25.0, "sports_recent", 15.0, "duration", 15, true],
[40.0, "sports_upcoming", 15.0, "duration", 15, true],
[55.0, "sports_recent", 15.0, "duration", 15, true],
[70.0, "sports_upcoming", 15.0, "duration", 15, true],
[85.0, "sports_recent", 10.0, "on-demand-requested-stop", 11, true],
[85.0, "sports_recent", 10.0, "on-demand-requested-stop", 10, true],
[95.0, "weather", 20.0, "duration", 20, true],
[115.0, "sports_recent", 15.0, "duration", 15, true],
[130.0, "sports_upcoming", 15.0, "duration", 15, true],
[145.0, "clock", 5.0, "on-demand-start", 6, true],
[145.0, "clock", 5.0, "on-demand-start", 5, true],
[150.0, "weather", 20.0, "duration", 20, true],
[170.0, "weather", 10.0, "on-demand-expired", 10, true],
[180.0, "clock", 20.0, "duration", 20, true],
+4 -4
View File
@@ -1,18 +1,18 @@
{
"screens": [
[0.0, "clock", 5.0, "on-demand-start", 6, false],
[0.0, "clock", 5.0, "on-demand-start", 5, false],
[5.0, "sports_live", 15.0, "duration", 15, true],
[20.0, "sports_recent", 15.0, "duration", 15, true],
[35.0, "sports_upcoming", 5.0, "on-demand-requested-stop", 6, true],
[35.0, "sports_upcoming", 5.0, "on-demand-requested-stop", 5, true],
[40.0, "clock", 20.0, "duration", 20, true],
[60.0, "sports_live", 15.0, "display-false", 11, true],
[75.0, "sports_recent", 15.0, "duration", 15, true],
[90.0, "sports_upcoming", 10.0, "on-demand-start", 11, true],
[90.0, "sports_upcoming", 10.0, "on-demand-start", 10, true],
[100.0, "sports_live", 0.0, "empty", 1, true],
[100.0, "sports_recent", 15.0, "duration", 15, true],
[115.0, "sports_upcoming", 15.0, "duration", 15, true],
[130.0, "sports_live", 0.0, "empty", 1, true],
[130.0, "sports_recent", 10.0, "on-demand-requested-stop", 11, true],
[130.0, "sports_recent", 10.0, "on-demand-requested-stop", 10, true],
[140.0, "sports_upcoming", 15.0, "duration", 15, true],
[155.0, "clock", 5.0, "horizon", 5, true]
],
+2 -2
View File
@@ -1,11 +1,11 @@
{
"screens": [
[0.0, "clock", 12.0, "on-demand-start", 13, false],
[0.0, "clock", 12.0, "on-demand-start", 12, false],
[12.0, "sports_upcoming", 15.0, "duration", 15, true],
[27.0, "sports_upcoming", 15.0, "duration", 15, true],
[42.0, "sports_upcoming", 15.0, "duration", 15, true],
[57.0, "sports_upcoming", 15.0, "duration", 15, true],
[72.0, "sports_upcoming", 8.0, "on-demand-start", 9, true],
[72.0, "sports_upcoming", 8.0, "on-demand-start", 8, true],
[80.0, "app_a", 0.0, "empty", 1, true],
[80.0, "app_b", 10.0, "duration", 10, true],
[90.0, "app_a", 0.0, "empty", 1, true],
+11 -11
View File
@@ -6,22 +6,22 @@
[70.255, "sports_live", 20.0, "duration", 20, true],
[90.255, "sports_live", 20.0, "display-false", 11, false],
[110.255, "<vegas>", 30.008, "duration", 3751, null],
[140.263, "<vegas>", 10.0, "on-demand-start", 1250, null],
[150.263, "clock", 20.0, "duration", 20, true],
[170.263, "clock", 5.0, "on-demand-expired", 5, true],
[175.263, "<vegas>", 24.959, "vegas-interrupt", 3120, null],
[200.222, "<wifi>", 3.0, "duration", 6, null],
[203.222, "<vegas>", 30.008, "duration", 3751, null],
[233.23, "<vegas>", 26.77, "horizon", 3347, null]
[140.263, "<vegas>", 9.744, "on-demand-start", 1218, null],
[150.007, "clock", 20.0, "duration", 20, true],
[170.007, "clock", 5.0, "on-demand-expired", 5, true],
[175.007, "<vegas>", 25.999, "vegas-interrupt", 3250, null],
[201.006, "<wifi>", 2.0, "duration", 4, null],
[203.006, "<vegas>", 30.008, "duration", 3751, null],
[233.014, "<vegas>", 26.986, "horizon", 3374, null]
],
"events": [
[70.255, "vegas-live"],
[70.255, "live", "sports_live"],
[150.0, "request", "start:v1"],
[150.263, "on-demand-start", "clock"],
[150.263, "vegas-interrupt"],
[175.263, "on-demand-expired"],
[150.007, "on-demand-start", "clock"],
[150.007, "vegas-interrupt"],
[175.007, "on-demand-expired"],
[200.0, "wifi-file", "Connected to HomeNet"],
[200.222, "vegas-interrupt"]
[201.006, "vegas-interrupt"]
]
}
+5 -2
View File
@@ -39,8 +39,11 @@ const ok = (l, c, x) => c ? (pass++, console.log(' ok ' + l))
ok('the partial has no inline handlers', !/\son(click|input|change|submit)=/i.test(partial));
ok('the forms carry no hx-on handler', !/hx-on/i.test(partial));
ok('the partial root is data-page="schedule"', /data-page="schedule"/.test(partial));
ok('both forms are marked data-reports-result',
(partial.match(/<form[^>]*data-reports-result/g) || []).length === 2);
// The two schedule forms, plus one per plugin in the Plugin Schedules card.
ok('every form is marked data-reports-result',
(partial.match(/<form[^>]*data-reports-result/g) || []).length
=== (partial.match(/<form[\s>]/g) || []).length
&& (partial.match(/<form[\s>]/g) || []).length >= 2);
const errs = [];
const logged = [];
+1
View File
@@ -33,6 +33,7 @@ const UNIT = ['unit/test_list_filter.js', 'unit/test_render_cards.js',
'unit/test_overview_reconciliation_poll.js',
'unit/test_display_partial_ids.js',
'unit/test_general_web_login_token.js',
'unit/test_on_demand_starting.js',
'unit/test_general_display_autostart.js'];
const DOM = ['dom/test_installed_dom.js', 'dom/test_store_dom.js', 'dom/test_no_double_fetch.js',
'dom/test_tools_sections.js', 'dom/test_cache_page.js',
+69
View File
@@ -0,0 +1,69 @@
// POST /api/v3/display/on-demand/start answers 202 with status "starting"
// when the display service has to be started first: the request is taken,
// and the web process sends it once the display listens. "Preview on
// display" (app.js) must read that as taken -- an info toast and the
// floating preview opened -- not as a failure. Runs the shipped app.js in a
// vm with a minimal fake DOM, as test_restart_banner.js does.
const fs = require('fs');
const path = require('path');
const vm = require('vm');
const V3 = path.resolve(__dirname, '../../../web_interface/static/v3');
let pass = 0, fail = 0;
const ok = (label, cond, extra) => cond
? (pass++, console.log(' ok ' + label))
: (fail++, console.log(' FAIL ' + label + (extra !== undefined ? ' ' + JSON.stringify(extra) : '')));
function load(answer) {
const notes = [];
const opened = [];
const noop = () => {};
const document = {
body: { addEventListener: noop },
addEventListener: noop,
getElementById: () => null,
querySelector: () => null,
querySelectorAll: () => [],
};
const window = { addEventListener: noop, getApp: () => null };
const context = {
window, document, console,
sessionStorage: { setItem: noop, removeItem: noop, getItem: () => null },
showNotification: (m, t) => notes.push([m, t]),
setTimeout: () => 0,
fetch: () => Promise.resolve({ json: () => Promise.resolve(answer) }),
};
vm.createContext(context);
vm.runInContext(fs.readFileSync(path.join(V3, 'app.js'), 'utf8'), context);
window.toggleFloatingPreview = (open) => opened.push(open);
return { window, notes, opened };
}
async function preview(answer) {
const t = load(answer);
t.window.previewPluginNow('weather');
for (let i = 0; i < 5; i++) await Promise.resolve();
return t;
}
(async () => {
console.log('\npreviewPluginNow');
{
const t = await preview({ status: 'starting', message: 'The display service is starting',
data: { request_id: 'r1', pending: true } });
ok('a 202 "starting" answer is an info toast, not an error',
t.notes.length === 1 && t.notes[0][1] === 'info', t.notes);
ok('and the preview opens', t.opened.length === 1 && t.opened[0] === true, t.opened);
}
{
const t = await preview({ status: 'success', data: { request_id: 'r1' } });
ok('a 200 success still opens it', t.opened.length === 1 && t.notes[0][1] === 'success', t.notes);
}
{
const t = await preview({ status: 'error', message: 'no display' });
ok('an error does not', t.opened.length === 0 && t.notes[0][1] === 'error', t.notes);
}
console.log(`\n${pass} passed, ${fail} failed`);
process.exit(fail ? 1 : 0);
})();
+4 -1
View File
@@ -54,7 +54,10 @@ class TestOnDemandStart:
def service(self, api_v3_module):
api_v3_module.api_v3.plugin_catalog = None
api_v3_module.api_v3.config_manager = None
with patch("web_interface.blueprints.api_v3.display._get_display_service_status",
# The display answers the control socket (the only way in).
with patch("web_interface.blueprints.api_v3.display.control_client.on_demand_start",
side_effect=lambda request_id, *a: {"accepted": True}), \
patch("web_interface.blueprints.api_v3.display._get_display_service_status",
return_value={"active": True}), \
patch("web_interface.blueprints.api_v3.display._stop_display_service") as stop, \
patch("web_interface.blueprints.api_v3.display._ensure_display_service_running",
+1 -2
View File
@@ -160,8 +160,7 @@ class TestOneBadConfigSectionDoesNotBlankTheList:
class TestOnDemandUsesTheRegisteredSpelling:
def test_a_mode_differing_in_case_is_sent_as_registered(self, client):
with patch('web_interface.blueprints.api_v3.display._deliver_on_demand',
return_value=('socket', None)) as deliver:
with patch('web_interface.blueprints.api_v3.display._send_on_demand') as deliver:
response = client.post('/api/v3/display/on-demand/start',
json={'plugin_id': 'football-scoreboard',
'mode': 'NFL_LIVE', 'start_service': False})
+15 -4
View File
@@ -77,11 +77,22 @@ def fresh_web_process(api_v3_module, plugins_dir):
@pytest.fixture
def display_service(api_v3_module):
"""Keep on-demand start away from systemctl and the real cache."""
cache = api_v3_module.api_v3.cache_manager = MagicMock()
with patch('web_interface.blueprints.api_v3.display._get_display_service_status') as status:
"""Keep on-demand start away from systemctl and the real cache; the
display's control socket is a mock that acks. What it was sent is
recorded as ``.set(cmd, request)`` calls on the yielded mock."""
api_v3_module.api_v3.cache_manager = MagicMock()
sent = MagicMock()
def ack(request_id, plugin_id, mode, *a, **kw):
sent.set('on_demand.start', {'request_id': request_id,
'plugin_id': plugin_id, 'mode': mode})
return {'accepted': True}
with patch('web_interface.blueprints.api_v3.display._get_display_service_status') as status, \
patch('web_interface.blueprints.api_v3.display.control_client.on_demand_start',
side_effect=ack):
status.return_value = {'active': True}
yield cache
yield sent
def _start(client, **body):
+77 -84
View File
@@ -5,18 +5,20 @@ the web UI and the MQTT bridge send) as "restart": with the service running it
ran ``systemctl stop``, slept 1.5s and started it again. Every on-demand or
"Preview on display" click therefore cold-restarted the display process --
every plugin reloaded, the panel blank for seconds -- to deliver a request the
running process polls for every ON_DEMAND_POLL_INTERVAL anyway (see
test_on_demand_mailbox.py and test_display_pending_changes.py for the display
side: the mailbox is read mid-dwell, mid-screen and mid-Vegas-iteration).
running process takes within a frame over its control socket anyway (see
test_display_pending_changes.py for the display side: commands land
mid-dwell, mid-screen and mid-Vegas-iteration).
The restart did not buy anything either: a freshly started display restores
only the on-demand session it saved itself (``display_on_demand_config``), so
the new request reached it through the same mailbox, one cold start later.
only the on-demand session it saved itself (``display_on_demand_config``).
This file previously pinned that restart path (it guarded a broken
``import _pkg.time`` inside it). The path is gone; these tests pin its
replacement: a running service is left alone, a stopped one is started (only
when start_service is set), and the request lands in the mailbox either way.
when start_service is set), and the request goes over the control socket
either way -- to a stopped display once it has started and its socket is up,
sent by the web process's dispatcher after the route has answered 202.
Nothing is ever written to the cache: the file mailbox is gone (stage 5).
The service helpers are patched where they run. display.py binds
_get_display_service_status by value, while _ensure_display_service_running
@@ -25,6 +27,7 @@ _run_systemctl_command is the one place a systemctl command is issued.
"""
import sys
import time
from pathlib import Path
from unittest.mock import patch
@@ -34,6 +37,8 @@ sys.path.insert(0, str(Path(__file__).parent.parent))
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
from src.ipc import client as control_client # noqa: E402
START_URL = "/api/v3/display/on-demand/start"
STOP_URL = "/api/v3/display/on-demand/stop"
MAILBOX = "display_on_demand_request"
@@ -46,7 +51,10 @@ def service(api_v3_module):
plugin_manager and config_manager are None so the route skips plugin
resolution (not what is under test here). The cache is the blueprint's
MagicMock cache_manager, so mailbox writes are visible as set() calls.
MagicMock cache_manager, so a write would be visible as a set() call.
The control socket answers while the service is active and is missing
(``no_socket``) while it is not; ``sent`` records what it carried.
"""
api_v3_module.api_v3.plugin_catalog = None
api_v3_module.api_v3.config_manager = None
@@ -62,7 +70,32 @@ def service(api_v3_module):
state["active"] = False
return {"returncode": 0, "stdout": "", "stderr": ""}
with patch("web_interface.blueprints.api_v3._get_display_service_status",
sent = []
def socket(action):
def call(request_id, *args, **kwargs):
if not state["active"]:
raise control_client.ControlError("no_socket", "x", sent=False)
sent.append((action, request_id) + args)
return {"accepted": True, "request_id": request_id}
return call
class Clock:
now = 1000.0
def monotonic(self):
return self.now
def time(self):
return self.now
def sleep(self, seconds):
self.now += seconds
with patch("web_interface.blueprints.api_v3.time", Clock()), \
patch(f"{DISPLAY}.control_client.on_demand_start", side_effect=socket("start")), \
patch(f"{DISPLAY}.control_client.on_demand_stop", side_effect=socket("stop")), \
patch("web_interface.blueprints.api_v3._get_display_service_status",
side_effect=status), \
patch("web_interface.blueprints.api_v3.display._get_display_service_status",
side_effect=status), \
@@ -74,6 +107,7 @@ def service(api_v3_module):
"systemctl": run_systemctl,
"stop_service": stop_service,
"cache": api_v3_module.api_v3.cache_manager,
"sent": sent,
}
@@ -99,19 +133,15 @@ class TestStartWhileTheServiceIsRunning:
assert _systemctl_verbs(service["systemctl"]) == [], (
"a running display service was sent a systemctl command")
def test_the_request_is_posted_for_the_running_display(self, api_v3_client, service):
def test_the_request_is_sent_to_the_running_display(self, api_v3_client, service):
response = api_v3_client.post(
START_URL, json={"plugin_id": "weather", "mode": "weather_current",
"duration": 60, "pinned": True})
data = response.get_json()["data"]
writes = _mailbox_writes(service["cache"])
assert len(writes) == 1
assert writes[0]["action"] == "start"
assert writes[0]["request_id"] == data["request_id"]
assert writes[0]["plugin_id"] == "weather"
assert writes[0]["mode"] == "weather_current"
assert writes[0]["duration"] == 60
assert writes[0]["pinned"] is True
assert service["sent"] == [("start", data["request_id"], "weather",
"weather_current", 60, True)]
assert data["transport"] == "socket"
assert _mailbox_writes(service["cache"]) == []
def test_the_response_reports_the_service_was_not_started(self, api_v3_client, service):
data = api_v3_client.post(START_URL, json={"plugin_id": "weather"}).get_json()["data"]
@@ -129,11 +159,20 @@ class TestStartWhileTheServiceIsStopped:
def test_start_service_starts_it_once_and_never_stops_it(self, api_v3_client, service):
service["state"]["active"] = False
response = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert response.status_code == 200, response.get_json()
# Answered at once: the web process sends the request in the
# background once the started display listens.
assert response.status_code == 202, response.get_json()
assert response.get_json()["status"] == "starting"
assert _systemctl_verbs(service["systemctl"]) == ["start"]
service["stop_service"].assert_not_called()
# Written before the start, so the new process finds it on its first poll.
assert len(_mailbox_writes(service["cache"])) == 1
from web_interface import on_demand_dispatch
dispatcher = on_demand_dispatch.current()
deadline = time.monotonic() + 5
while dispatcher.pending() and time.monotonic() < deadline:
time.sleep(0.01)
assert dispatcher.status()["status"] == "delivered"
assert [s[0] for s in service["sent"]] == ["start"]
assert _mailbox_writes(service["cache"]) == []
def test_without_start_service_it_is_left_stopped(self, api_v3_client, service):
service["state"]["active"] = False
@@ -141,6 +180,7 @@ class TestStartWhileTheServiceIsStopped:
START_URL, json={"plugin_id": "weather", "start_service": "false"})
assert response.status_code == 400
assert _systemctl_verbs(service["systemctl"]) == []
assert service["sent"] == []
def test_a_start_that_fails_is_reported(self, api_v3_client, service):
service["state"]["active"] = False
@@ -151,31 +191,12 @@ class TestStartWhileTheServiceIsStopped:
assert response.get_json()["status"] == "error"
class _Mailbox:
"""The CacheManager calls the routes make, over a dict."""
def __init__(self):
self.entries = {}
def set(self, key, value, ttl=None):
self.entries[key] = value
def get(self, key, max_age=300, memory_ttl=None):
return self.entries.get(key)
def delete(self, key):
self.entries.pop(key, None)
class TestARefusedStartLeavesNoRequestBehind:
"""A start the route answers with an error must not run later.
The request was posted (to the mailbox, with the display stopped) before
the route refused it, and the display reads the mailbox for an hour
without looking at a request's age. So "Display service is not running"
(start_service off) or "Failed to start display service" left the
request waiting, and the next time the display started -- minutes later,
by hand -- it ran that plugin, pinned if the request said so.
With the mailbox, the request was posted before the route refused it, and
a display started later ran it. Now nothing is written anywhere: the
request only ever goes over the socket, to a display that answers.
A socket acknowledgement is the other side of it: the display answered,
so it is running and has the request, whatever systemd says (a display
@@ -184,76 +205,48 @@ class TestARefusedStartLeavesNoRequestBehind:
"""
@pytest.fixture
def mailbox(self, api_v3_module, service):
box = _Mailbox()
api_v3_module.api_v3.cache_manager = box
def stopped(self, service):
service["state"]["active"] = False
return box
return service
@pytest.mark.parametrize("body", [
{"plugin_id": "weather", "start_service": False},
{"plugin_id": "weather"}, # start_service defaults on
])
def test_a_socket_ack_is_a_success_whatever_systemd_says(
self, api_v3_client, service, mailbox, body):
self, api_v3_client, stopped, body):
with patch(f"{DISPLAY}.control_client.on_demand_start",
side_effect=lambda request_id, *a: {"accepted": True}):
response = api_v3_client.post(START_URL, json=body)
assert response.status_code == 200, response.get_json()
assert response.get_json()["data"]["transport"] == "socket"
assert MAILBOX not in mailbox.entries
assert _systemctl_verbs(service["systemctl"]) == [], (
assert _systemctl_verbs(stopped["systemctl"]) == [], (
"a unit was started beside a display that answered the socket")
def test_without_start_service_the_request_is_taken_back(
self, api_v3_client, service, mailbox):
def test_without_start_service_nothing_is_left_behind(self, api_v3_client, stopped):
response = api_v3_client.post(START_URL, json={
"plugin_id": "weather", "pinned": True, "start_service": False})
assert response.status_code == 400
assert response.get_json()["status"] == "error"
assert MAILBOX not in mailbox.entries
assert stopped["cache"].set.call_count == 0
assert stopped["sent"] == []
def test_a_start_that_fails_takes_its_request_back(self, api_v3_client, service, mailbox):
service["systemctl"].side_effect = lambda args: {
def test_a_start_that_fails_leaves_nothing_behind(self, api_v3_client, stopped):
stopped["systemctl"].side_effect = lambda args: {
"returncode": 1, "stdout": "", "stderr": "denied"}
response = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert response.status_code == 500
assert MAILBOX not in mailbox.entries
def test_a_newer_request_is_left_alone_on_the_400(self, api_v3_client, service, mailbox):
newer = {"request_id": "someone-else", "action": "start", "plugin_id": "clock"}
def stopped_and_another_post_lands(*args):
mailbox.entries[MAILBOX] = newer
return {"active": False}
with patch(f"{DISPLAY}._get_display_service_status",
side_effect=stopped_and_another_post_lands):
response = api_v3_client.post(START_URL, json={
"plugin_id": "weather", "start_service": False})
assert response.status_code == 400
assert mailbox.entries[MAILBOX] is newer
def test_a_newer_request_is_left_alone_on_the_500(self, api_v3_client, service, mailbox):
newer = {"request_id": "someone-else", "action": "start", "plugin_id": "clock"}
def start_fails_after_another_post(args):
mailbox.entries[MAILBOX] = newer
return {"returncode": 1, "stdout": "", "stderr": "denied"}
service["systemctl"].side_effect = start_fails_after_another_post
response = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert response.status_code == 500
assert mailbox.entries[MAILBOX] is newer
assert stopped["cache"].set.call_count == 0
assert stopped["sent"] == []
class TestStop:
def test_stop_posts_a_stop_request_and_leaves_the_service_running(
def test_stop_sends_a_stop_request_and_leaves_the_service_running(
self, api_v3_client, service):
response = api_v3_client.post(STOP_URL, json={})
assert response.status_code == 200, response.get_json()
writes = _mailbox_writes(service["cache"])
assert [w["action"] for w in writes] == ["stop"]
assert [s[0] for s in service["sent"]] == ["stop"]
assert _mailbox_writes(service["cache"]) == []
service["stop_service"].assert_not_called()
assert _systemctl_verbs(service["systemctl"]) == []
+398 -85
View File
@@ -1,15 +1,12 @@
"""POST /display/on-demand/start and /stop: control socket first, mailbox fallback.
"""POST /display/on-demand/start and /stop: the control socket is the only way.
The routes hand the request to the display over the control socket
(src/ipc) and get an acknowledgement. When the socket could not carry the
request -- no socket (a stopped display, or one older than the socket), a
refused or timed-out connect, a display too old to know the command, a bug
in the client -- they write the file mailbox exactly as they did before the
socket existed. When the display had the request and failed it (a full
queue, bad arguments, no answer in time) the route says so and writes
nothing. These tests pin each path, that at most one of them is used, that
the response says which, and that the request id is the same either way (the
display deduplicates on it).
(src/ipc) and get an acknowledgement. The file mailbox they used to fall
back to (``display_on_demand_request``) is gone (stage 5), so nothing is
ever written to the cache. When no display is listening (a stopped one, or
one still starting) the start route starts the service if asked to and
sends the request again once the socket is up; every other failure is
answered as what it is.
The socket client is patched at the route's module attribute; the last class
runs a real server on a temp socket (Linux/macOS only).
@@ -17,6 +14,8 @@ runs a real server on a temp socket (Linux/macOS only).
import os
import sys
import time
import types
from pathlib import Path
from unittest.mock import patch
@@ -33,11 +32,12 @@ START_URL = "/api/v3/display/on-demand/start"
STOP_URL = "/api/v3/display/on-demand/stop"
MAILBOX = "display_on_demand_request"
CLIENT = "web_interface.blueprints.api_v3.display.control_client"
DISPLAY = "web_interface.blueprints.api_v3.display"
@pytest.fixture
def service(api_v3_module):
"""A running display service; records systemctl calls and mailbox writes."""
"""A running display service; records systemctl calls and cache writes."""
api_v3_module.api_v3.plugin_catalog = None
api_v3_module.api_v3.config_manager = None
state = {"active": True}
@@ -101,85 +101,398 @@ class TestSocketPath:
assert _mailbox_writes(service["cache"]) == []
class TestMailboxFallback:
@pytest.mark.parametrize("reason", [
"no_socket", "refused", "timeout", "closed", "bad_response", "invalid_request",
"busy", "forbidden", "unknown_command", "unsupported_version", "disabled",
"unsupported",
])
def test_a_request_the_socket_never_carried_writes_the_mailbox(
self, api_v3_client, service, reason):
# sent=False: the display never had it (no socket, a refused or
# timed-out connect, turned away at the door).
with patch(f"{CLIENT}.on_demand_start",
side_effect=control_client.ControlError(reason, "x", sent=False)):
resp = api_v3_client.post(START_URL, json={
"plugin_id": "weather", "mode": "weather_current",
"duration": 60, "pinned": True})
assert resp.status_code == 200
data = resp.get_json()["data"]
assert data["transport"] == "mailbox"
assert data["socket_error"] == reason
[write] = _mailbox_writes(service["cache"])
assert write["request_id"] == data["request_id"]
assert write["action"] == "start"
assert (write["plugin_id"], write["mode"], write["duration"], write["pinned"]) == \
("weather", "weather_current", 60, True)
class FakeTime:
"""Stands in for the route's ``time`` module: sleep() moves the clock."""
def test_a_client_bug_still_falls_back(self, api_v3_client, service):
def __init__(self):
self.now = 1000.0
self.sleeps = 0
def monotonic(self):
return self.now
def time(self):
return self.now
def sleep(self, seconds):
self.sleeps += 1
self.now += seconds
@pytest.fixture
def clock():
fake = FakeTime()
with patch("web_interface.blueprints.api_v3.time", fake):
yield fake
def _attempts(*outcomes, calls=None, clock=None):
"""A fake on_demand_start/stop that answers ``outcomes`` in turn (an
exception is raised, anything else acks); the last one repeats."""
outcomes = list(outcomes)
seen = calls if calls is not None else []
def attempt(request_id, *a, **kw):
seen.append(clock.now if clock is not None else None)
outcome = outcomes.pop(0) if len(outcomes) > 1 else outcomes[0]
if isinstance(outcome, BaseException):
raise outcome
return _ack(request_id)
return attempt
def _until(predicate, timeout=5.0):
end = time.monotonic() + timeout
while time.monotonic() < end:
if predicate():
return True
time.sleep(0.005)
return False
def _no_socket():
return control_client.ControlError("no_socket", "x", sent=False)
class TestNoDisplayListening:
"""No socket to talk to: the display is stopped, or still starting.
The route answers at once (202, ``status: "starting"``) and the web
process's dispatcher (web_interface/on_demand_dispatch.py) sends the
request until the display acknowledges it; the status routes report the
outcome. The dispatcher runs on real time with short waits here.
"""
@pytest.fixture(autouse=True)
def quick(self, monkeypatch, service):
from web_interface import on_demand_dispatch
monkeypatch.setattr(on_demand_dispatch, "RETRY_INTERVAL", 0.01)
monkeypatch.setattr(on_demand_dispatch, "START_WAIT_SECONDS", 2.0)
monkeypatch.setattr(f"{DISPLAY}.ON_DEMAND_SOCKET_WAIT_RUNNING_SECONDS", 1.0)
service["cache"].get.return_value = None # the display has published nothing
@staticmethod
def _outcome(status=None):
from web_interface import on_demand_dispatch
d = on_demand_dispatch.current()
assert d is not None
assert _until(lambda: not d.pending()), "the dispatcher never finished"
return d.status()
def test_a_display_still_starting_gets_the_request_once_it_listens(
self, api_v3_client, service):
calls = []
with patch(f"{CLIENT}.on_demand_start",
side_effect=_attempts(_no_socket(), _no_socket(), "ack", calls=calls)):
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 202, resp.get_json()
body = resp.get_json()
assert body["status"] == "starting"
data = body["data"]
assert data["pending"] is True and data["socket_error"] == "no_socket"
assert data["service"]["started"] is False
outcome = self._outcome()
assert outcome["status"] == "delivered"
assert outcome["request_id"] == data["request_id"]
assert len(calls) == 3
assert not [c for c in service["calls"] if c[0] == "systemctl"]
assert _mailbox_writes(service["cache"]) == []
def test_a_stopped_display_is_started_and_answered_at_once(self, api_v3_client, service):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_start",
side_effect=_attempts(*[_no_socket()] * 6, "ack")) as start:
started = time.monotonic()
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather",
"duration": 30})
answered = time.monotonic() - started
assert resp.status_code == 202, resp.get_json()
data = resp.get_json()["data"]
assert data["service"]["started"] is True
from web_interface import on_demand_dispatch
assert data["wait_seconds"] == on_demand_dispatch.START_WAIT_SECONDS
outcome = self._outcome()
assert answered < 1.0, "the route waited for the display"
assert service["calls"] == [("systemctl", "start")]
assert outcome["status"] == "delivered"
# The same request, the same id, every time.
assert {call.args[0] for call in start.call_args_list} == {data["request_id"]}
assert _mailbox_writes(service["cache"]) == []
def test_while_pending_the_status_routes_say_starting(self, api_v3_client, service):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_start", side_effect=_no_socket()):
rid = api_v3_client.post(START_URL, json={"plugin_id": "weather"}) \
.get_json()["data"]["request_id"]
status = api_v3_client.get("/api/v3/display/on-demand/status").get_json()["data"]
assert status["source"] == "web"
assert status["state"]["status"] == "starting"
assert status["state"]["request_id"] == rid
assert status["state"]["plugin_id"] == "weather"
current = api_v3_client.get("/api/v3/display/current-status").get_json()["data"]
assert current["on_demand_pending"]["status"] == "starting"
def test_a_display_that_never_comes_up_is_a_start_timeout(self, api_v3_client, service):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_start", side_effect=_no_socket()):
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 202
outcome = self._outcome()
assert outcome["status"] == "error" and outcome["error"] == "start-timeout"
status = api_v3_client.get("/api/v3/display/on-demand/status").get_json()["data"]
assert status["state"]["status"] == "error"
assert status["state"]["error"] == "start-timeout"
current = api_v3_client.get("/api/v3/display/current-status").get_json()["data"]
assert current["on_demand_pending"]["error"] == "start-timeout"
def test_a_later_display_state_replaces_the_failure(self, api_v3_client, service):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_start", side_effect=_no_socket()):
api_v3_client.post(START_URL, json={"plugin_id": "weather"})
failed_at = self._outcome()["last_updated"]
later = {"active": True, "status": "active", "plugin_id": "clock",
"last_updated": failed_at + 5}
service["cache"].get.return_value = later
status = api_v3_client.get("/api/v3/display/on-demand/status").get_json()["data"]
assert status["state"]["plugin_id"] == "clock" and status["source"] == "cache"
def test_a_running_service_without_a_socket_is_waited_for_less(
self, api_v3_client, service):
with patch(f"{CLIENT}.on_demand_start", side_effect=_no_socket()):
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 202
assert resp.get_json()["data"]["wait_seconds"] == 1.0
assert self._outcome()["error"] == "start-timeout"
assert not [c for c in service["calls"] if c[0] == "systemctl"]
def test_a_different_failure_while_waiting_ends_it(self, api_v3_client, service):
calls = []
busy = control_client.ControlError("busy", "x", sent=True)
with patch(f"{CLIENT}.on_demand_start",
side_effect=_attempts(_no_socket(), _no_socket(), busy, calls=calls)):
assert api_v3_client.post(START_URL, json={"plugin_id": "weather"}).status_code == 202
outcome = self._outcome()
assert outcome["status"] == "error" and outcome["error"] == "busy"
assert len(calls) == 3
def test_a_stop_while_pending_cancels_it(self, api_v3_client, service):
service["state"]["active"] = False
calls = []
with patch(f"{CLIENT}.on_demand_start",
side_effect=_attempts(_no_socket(), calls=calls)), \
patch(f"{CLIENT}.on_demand_stop", side_effect=_no_socket()):
rid = api_v3_client.post(START_URL, json={"plugin_id": "weather"}) \
.get_json()["data"]["request_id"]
resp = api_v3_client.post(STOP_URL, json={})
assert resp.status_code == 200, resp.get_json()
data = resp.get_json()["data"]
assert data["cancelled_request_id"] == rid
outcome = self._outcome()
n = len(calls)
time.sleep(0.05)
assert len(calls) == n, "the cancelled start was still being sent"
assert outcome["status"] == "idle" and outcome["last_event"] == "requested-stop"
status = api_v3_client.get("/api/v3/display/on-demand/status").get_json()["data"]
assert status["state"]["status"] == "idle" and status["source"] != "web"
# -- delivered, but the display has not acted on it yet ----------------
# The display acknowledges a start when its socket opens and acts on it
# seconds later (Vegas builds its first strip first: ~5 s on ledpi);
# meanwhile it publishes its own idle state, which must not flash.
def _delivered(self, api_v3_client, service):
with patch(f"{CLIENT}.on_demand_start",
side_effect=_attempts(_no_socket(), "ack")):
data = api_v3_client.post(START_URL, json={"plugin_id": "weather"}) .get_json()["data"]
outcome = self._outcome()
assert outcome["status"] == "delivered"
return data["request_id"], outcome["last_updated"]
@staticmethod
def _status(api_v3_client):
return api_v3_client.get("/api/v3/display/on-demand/status").get_json()["data"]
def test_a_delivered_start_stays_starting_until_the_display_answers(
self, api_v3_client, service):
rid, delivered_at = self._delivered(api_v3_client, service)
# The display's startup state: idle, published after the ack, for
# no request (or an older one).
for older in (None, "an-older-request"):
service["cache"].get.return_value = {
"active": False, "status": "idle", "request_id": older,
"last_updated": delivered_at + 1}
data = self._status(api_v3_client)
assert data["source"] == "web", older
assert data["state"]["status"] == "starting"
assert data["state"]["delivered"] is True
assert data["state"]["request_id"] == rid
current = api_v3_client.get("/api/v3/display/current-status").get_json()["data"]
assert current["on_demand_pending"]["delivered"] is True
def test_the_display_state_for_the_request_takes_over(self, api_v3_client, service):
rid, delivered_at = self._delivered(api_v3_client, service)
service["cache"].get.return_value = {
"active": True, "status": "active", "plugin_id": "weather",
"request_id": rid, "last_updated": delivered_at + 5}
data = self._status(api_v3_client)
assert data["source"] == "cache" and data["state"]["status"] == "active"
current = api_v3_client.get("/api/v3/display/current-status").get_json()["data"]
assert "on_demand_pending" not in current
def test_its_error_takes_over_too(self, api_v3_client, service):
rid, delivered_at = self._delivered(api_v3_client, service)
service["cache"].get.return_value = {
"active": False, "status": "error", "error": "load-failed",
"request_id": rid, "last_updated": delivered_at + 5}
assert self._status(api_v3_client)["state"]["error"] == "load-failed"
def test_an_older_display_answers_with_any_newer_state(self, api_v3_client, service):
# A display before request_id was published: timestamps decide.
_, delivered_at = self._delivered(api_v3_client, service)
service["cache"].get.return_value = {"active": False, "status": "idle",
"last_updated": delivered_at - 1}
assert self._status(api_v3_client)["source"] == "web"
service["cache"].get.return_value = {"active": True, "status": "active",
"last_updated": delivered_at + 1}
assert self._status(api_v3_client)["source"] == "cache"
def test_a_delivered_start_is_shown_for_a_limited_time(
self, api_v3_client, service, monkeypatch):
from web_interface import on_demand_dispatch
_, delivered_at = self._delivered(api_v3_client, service)
service["cache"].get.return_value = {"active": False, "status": "idle",
"request_id": None,
"last_updated": delivered_at + 1}
cap = on_demand_dispatch.DELIVERED_SHOWN_SECONDS
assert cap == 30.0
clock = types.SimpleNamespace(time=lambda: delivered_at + cap - 1,
monotonic=time.monotonic, sleep=time.sleep)
monkeypatch.setattr("web_interface.blueprints.api_v3.time", clock)
assert self._status(api_v3_client)["source"] == "web"
clock.time = lambda: delivered_at + cap + 1
data = self._status(api_v3_client)
assert data["source"] == "cache" and data["state"]["status"] == "idle"
current = api_v3_client.get("/api/v3/display/current-status").get_json()["data"]
assert "on_demand_pending" not in current
def test_a_new_start_supersedes_the_pending_one(self, api_v3_client, service):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_start", side_effect=_no_socket()):
old = api_v3_client.post(START_URL, json={"plugin_id": "weather"}) \
.get_json()["data"]["request_id"]
sent = []
service["state"]["active"] = True
with patch(f"{CLIENT}.on_demand_start",
side_effect=lambda rid, *a: sent.append(rid) or {"accepted": True}):
resp = api_v3_client.post(START_URL, json={"plugin_id": "clock"})
assert resp.status_code == 200
outcome = self._outcome()
# The old start is cancelled before the new one is sent, and
# cancel() waits out a send of it in flight: nothing of it lands
# after the new one.
new = resp.get_json()["data"]["request_id"]
assert sent[-1] == new and sent.count(new) == 1
assert outcome["status"] == "idle" and outcome["last_event"] == "superseded"
def test_a_service_that_will_not_start_is_an_error(self, api_v3_client, service, clock):
service["state"]["active"] = False
with patch("web_interface.blueprints.api_v3._run_systemctl_command",
return_value={"returncode": 1, "stdout": "", "stderr": "nope"}), \
patch(f"{CLIENT}.on_demand_start", side_effect=_no_socket()) as start:
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 500
assert "Failed to start display service" in resp.get_json()["message"]
assert start.call_count == 1
def test_stop_with_no_display_running_is_an_error(self, api_v3_client, service, clock):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_stop", side_effect=_no_socket()) as stop:
resp = api_v3_client.post(STOP_URL, json={})
assert resp.status_code == 503
body = resp.get_json()
assert "not running" in body["message"]
assert body["data"]["socket_error"] == "no_socket"
assert stop.call_count == 1 and clock.sleeps == 0
assert service["calls"] == []
assert _mailbox_writes(service["cache"]) == []
def test_stop_with_a_running_service_but_no_socket_is_an_error(
self, api_v3_client, service, clock):
with patch(f"{CLIENT}.on_demand_stop",
side_effect=control_client.ControlError("refused", "x")):
resp = api_v3_client.post(STOP_URL, json={})
assert resp.status_code == 503
assert "still be starting" in resp.get_json()["message"]
def test_stop_with_stop_service_stops_it_anyway(self, api_v3_client, service, clock):
with patch(f"{CLIENT}.on_demand_stop", side_effect=_no_socket()), \
patch(f"{DISPLAY}._stop_display_service",
return_value={"active": False}) as stop:
resp = api_v3_client.post(STOP_URL, json={"stop_service": True})
assert resp.status_code == 200
assert resp.get_json()["data"]["socket_error"] == "no_socket"
stop.assert_called_once()
assert _mailbox_writes(service["cache"]) == []
def test_the_socket_is_off_in_the_test_suite(self, api_v3_client, service):
# conftest's _hermetic_control_socket: a suite run on a device must
# not drive the live display. Nothing is retried or started for it.
assert os.environ[c.SOCKET_PATH_ENV] == "off"
service["state"]["active"] = False
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 503
assert resp.get_json()["data"]["socket_error"] in ("disabled", "unsupported")
assert service["calls"] == []
class TestNothingElseIsRetried:
@pytest.mark.parametrize("reason,sent,status", [
("timeout", False, 503), ("busy", False, 503), ("forbidden", False, 503),
("invalid_request", False, 503), ("disabled", False, 503),
("unsupported", False, 503), ("unknown_command", True, 503),
("unsupported_version", True, 503),
])
def test_answered_at_once_and_nothing_written(self, api_v3_client, service, clock,
reason, sent, status):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_start",
side_effect=control_client.ControlError(reason, "x", sent=sent)) as start:
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == status
body = resp.get_json()
assert body["status"] == "error"
assert body["data"]["socket_error"] == reason
assert start.call_count == 1 and clock.sleeps == 0
assert service["calls"] == []
def test_a_client_bug_is_an_error(self, api_v3_client, service):
with patch(f"{CLIENT}.on_demand_start", side_effect=RuntimeError("boom")):
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 200
assert resp.status_code == 503
assert resp.get_json()["data"]["socket_error"] == "internal"
assert len(_mailbox_writes(service["cache"])) == 1
assert _mailbox_writes(service["cache"]) == []
def test_an_unknown_reason_is_reported_as_other(self, api_v3_client, service):
# Only known codes are echoed back; anything else stays server-side.
with patch(f"{CLIENT}.on_demand_start",
side_effect=control_client.ControlError("/run/secret/path", "x")):
data = api_v3_client.post(START_URL, json={"plugin_id": "weather"}).get_json()["data"]
assert data["transport"] == "mailbox"
assert data["socket_error"] == "other"
assert len(_mailbox_writes(service["cache"])) == 1
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 503
assert resp.get_json()["data"]["socket_error"] == "other"
assert "/run/secret" not in resp.get_data(as_text=True)
def test_every_display_error_code_is_reportable(self):
from web_interface.blueprints.api_v3 import display
codes = {v for k, v in vars(c.ErrorCode).items() if not k.startswith("_")}
assert codes <= set(display._REPORTABLE_SOCKET_REASONS)
def test_stop_falls_back(self, api_v3_client, service):
with patch(f"{CLIENT}.on_demand_stop",
side_effect=control_client.ControlError("timeout")):
data = api_v3_client.post(STOP_URL, json={}).get_json()["data"]
assert data["transport"] == "mailbox"
[write] = _mailbox_writes(service["cache"])
assert write == {"request_id": data["request_id"], "action": "stop",
"timestamp": write["timestamp"]}
def test_a_stopped_display_gets_the_mailbox_before_it_is_started(
self, api_v3_client, service):
service["state"]["active"] = False
with patch(f"{CLIENT}.on_demand_start",
side_effect=control_client.ControlError("no_socket")):
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
assert resp.status_code == 200
assert service["calls"] == [("cache", MAILBOX), ("systemctl", "start")]
def test_the_socket_is_off_in_the_test_suite(self, api_v3_client, service):
# conftest's _hermetic_control_socket: a suite run on a device must
# not drive the live display.
assert os.environ[c.SOCKET_PATH_ENV] == "off"
data = api_v3_client.post(START_URL, json={"plugin_id": "weather"}).get_json()["data"]
assert data["transport"] == "mailbox"
assert data["socket_error"] in ("disabled", "unsupported") # Linux, Windows
class TestTheDisplayHadIt:
"""Once the display has the request, its answer stands: no mailbox copy.
"""Once the display has the request, its answer stands.
A busy queue, a refusal or silence after the request was sent mean the
display may have applied it, or would refuse the mailbox copy too, so
the route reports the failure instead of posting it a second time.
display may have applied it, so the route reports the failure and does
not send it again.
"""
@pytest.mark.parametrize("reason,status", [
@@ -217,16 +530,6 @@ class TestTheDisplayHadIt:
stop.assert_called_once()
assert _mailbox_writes(service["cache"]) == []
@pytest.mark.parametrize("reason", ["unknown_command", "unsupported_version"])
def test_an_older_display_that_does_not_speak_it_gets_the_mailbox(
self, api_v3_client, service, reason):
# The upgrade case: new web interface, display still on an old build.
with patch(f"{CLIENT}.on_demand_start",
side_effect=control_client.ControlError(reason, "x", sent=True)):
data = api_v3_client.post(START_URL, json={"plugin_id": "weather"}).get_json()["data"]
assert data["transport"] == "mailbox" and data["socket_error"] == reason
assert len(_mailbox_writes(service["cache"])) == 1
@pytest.mark.skipif(not c.socket_supported(), reason="AF_UNIX sockets are Linux/macOS only")
class TestRealSocket:
@@ -259,11 +562,21 @@ class TestRealSocket:
assert data["transport"] == "socket"
assert [x.request_id for x in live.drain()] == [data["request_id"]]
def test_a_display_that_went_away_falls_back(self, api_v3_client, service, live):
def test_a_display_that_went_away_is_given_up_on(self, api_v3_client, service, live,
monkeypatch):
from web_interface import on_demand_dispatch
monkeypatch.setattr(f"{DISPLAY}.ON_DEMAND_SOCKET_WAIT_RUNNING_SECONDS", 0.3)
monkeypatch.setattr(on_demand_dispatch, "RETRY_INTERVAL", 0.01)
live.close()
data = api_v3_client.post(START_URL, json={"plugin_id": "weather"}).get_json()["data"]
assert data["transport"] == "mailbox" and data["socket_error"] == "no_socket"
assert len(_mailbox_writes(service["cache"])) == 1
resp = api_v3_client.post(START_URL, json={"plugin_id": "weather"})
# The service still reads as running: answered at once, and the
# web process gives up once the wait is over.
assert resp.status_code == 202
assert resp.get_json()["data"]["socket_error"] == "no_socket"
d = on_demand_dispatch.current()
assert _until(lambda: not d.pending())
assert d.status()["error"] == "start-timeout"
assert _mailbox_writes(service["cache"]) == []
def test_a_full_queue_is_reported_not_mailed(self, api_v3_client, service, monkeypatch):
import shutil
+24
View File
@@ -9,9 +9,11 @@ Git runs for real against a throwaway origin/device clone pair; systemd, sudo
and the health check service are faked.
"""
import json
import os
import shutil
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import MagicMock, patch
@@ -579,6 +581,27 @@ class TestUpdateIsVerified:
assert repo.head() == old
assert 'rolled back' in result['core_message']
@pytest.mark.skipif(not Path('/proc/self/comm').exists(), reason='needs /proc')
def test_a_pull_killed_by_its_timeout_is_still_rolled_back(self, tmp_path):
"""subprocess.run SIGKILLs a timed-out git, which leaves .git/index.lock
behind; the rollback's reset would fail on it."""
repo = Repo(tmp_path)
old = repo.head()
repo.publish()
pull = real_pull(repo.device)
lock = repo.device / '.git' / 'index.lock'
def killed(**kwargs):
pull()
lock.write_text('')
stale = time.time() - 60
os.utime(lock, (stale, stale))
raise subprocess.TimeoutExpired(['git', 'pull'], 60)
result = Harness(tmp_path, repo, core_update=killed).updater.run()
assert repo.head() == old
assert 'rolled back' in result['core_message']
assert not lock.exists()
@pytest.mark.parametrize('how', ['partial_pull', 'check_never_started'])
def test_a_failed_rollback_leaves_plugins_and_the_display_alone(self, tmp_path, how):
"""update_core returns rollback_failed itself on two paths. The device
@@ -800,6 +823,7 @@ def test_core_update_names_the_requirement_files_that_failed():
pkg.api_v3.plugin_store_manager = None
failed_install = subprocess.CompletedProcess([], 1, stdout='', stderr='boom')
with patch.object(pkg.subprocess, 'run', run), \
patch.object(system.git_index_lock, 'run_git', run), \
patch.object(system, '_pip_install_requirements', return_value=failed_install):
result = system.perform_core_update()
assert result['status'] == 'success'
@@ -8,6 +8,7 @@ now pauses between passes, and it stops pausing as soon as something shows.
"""
import os
from unittest.mock import MagicMock
os.environ.setdefault("EMULATOR", "true")
@@ -21,6 +22,7 @@ def _bare_controller(modes, on_demand_modes=None):
dc.on_demand_modes = list(on_demand_modes or [])
dc.sleeps = []
dc._sleep_with_plugin_updates = lambda duration, tick_interval=1.0: dc.sleeps.append(duration)
dc.display_manager = MagicMock()
return dc
@@ -37,6 +39,19 @@ class TestNoteEmptyPass:
dc._note_empty_pass()
# Passes 3, 4 and 5 are at or past one full rotation.
assert dc.sleeps == [DisplayController.EMPTY_ROTATION_PAUSE] * 3
# Blanked once, so the last screen drawn does not stay up.
assert dc.display_manager.clear.call_count == 1
# A controller-drawn blank: the last scroll's state is ended first.
dc.display_manager.set_scrolling_state.assert_called_once_with(False)
assert dc.display_manager.update_display.call_count == 1
# ...and the preview gets the blank frame however the throttle fell.
assert dc.display_manager.write_owed_snapshot.call_count == 3
def test_a_partly_empty_rotation_never_blanks(self):
dc = _bare_controller(["a", "b", "c"])
dc._note_empty_pass()
dc._note_empty_pass()
dc.display_manager.clear.assert_not_called()
def test_content_resets_the_streak(self):
dc = _bare_controller(["a", "b"])
+36 -37
View File
@@ -6,7 +6,7 @@ a Vegas iteration runs for vegas_scroll.max_cycle_duration (240s here). On a
real Pi on 2026-09-23:
* an on-demand request posted at 10:54:27 was activated at 10:57:24, when the
Vegas iteration it arrived during finally ended -- nothing read the mailbox
Vegas iteration it arrived during finally ended -- nothing read the request
in between, because _check_vegas_interrupt only looked at a flag that the
main-loop read sets;
* two brightness saves 12s apart inside one 30s screen never showed at all.
@@ -15,6 +15,11 @@ _service_pending_changes is the fix: a throttled pass the dwell sleep, the
render loops and the Vegas interrupt check all call. These tests drive those
long stretches on a fake clock and check a change lands within one throttle
interval -- and that the throttle holds, since the callers run at frame rate.
The on-demand requests here come from a plugin in the display process
(submit_plugin_on_demand), the way in that needs no socket; the file mailbox
these tests used to write is gone (stage 5). A queued request skips the
throttle, so it lands at the next pass's call, not the next interval.
"""
import threading
@@ -28,7 +33,6 @@ from src.vegas_mode.config import VegasModeConfig
from src.vegas_mode.coordinator import VegasModeCoordinator
VEGAS_ITERATION_SECONDS = 240
REQUEST_KEY = 'display_on_demand_request'
class FakeClock:
@@ -75,26 +79,15 @@ def clock(monkeypatch):
@pytest.fixture
def controller(test_display_controller, clock):
"""A controller at rest: no schedule, full brightness, empty mailbox."""
"""A controller at rest: no schedule, full brightness, nothing queued."""
c = test_display_controller
c._refresh_config_cache({'display': {'hardware': {'brightness': 90}}})
c.current_brightness = 90
c.is_display_active = True
c._check_wifi_status_message = MagicMock(return_value=None)
c.mailbox = {} # what the web process has written
def cache_get(key, *args, **kwargs):
if key == REQUEST_KEY:
return c.mailbox.get('request')
return None
def cache_delete(key):
if key == REQUEST_KEY:
c.mailbox.pop('request', None)
c.cache_manager.get = MagicMock(side_effect=cache_get)
c.cache_manager.delete = MagicMock(side_effect=cache_delete)
c.cache_manager.get = MagicMock(return_value=None)
c.cache_manager.delete = MagicMock()
c.cache_manager.set = MagicMock()
c.display_manager.set_brightness = MagicMock(return_value=True)
c.display_manager.update_display = MagicMock()
@@ -110,11 +103,12 @@ def controller(test_display_controller, clock):
return c
def post_request(controller, request_id='r1', mode='clock'):
controller.mailbox['request'] = {
'request_id': request_id, 'action': 'start',
'plugin_id': mode, 'mode': mode,
}
def post_request(controller, request_id='r1', mode='clock', action='start'):
"""A plugin asking for the screen (or giving it back) from its thread."""
assert controller.submit_plugin_on_demand({
'request_id': request_id, 'action': action,
'plugin_id': mode, 'mode': mode, 'source': 'plugin',
})
def save_brightness(controller, brightness):
@@ -124,9 +118,11 @@ def save_brightness(controller, brightness):
{'display': {'hardware': {'brightness': brightness}}})
def mailbox_reads(controller):
return sum(1 for call in controller.cache_manager.get.call_args_list
if call.args and call.args[0] == REQUEST_KEY)
def count_passes(controller):
"""Count the pending-changes passes that got past the throttle."""
controller._poll_on_demand_requests = MagicMock(
wraps=controller._poll_on_demand_requests)
return controller._poll_on_demand_requests
def vegas_coordinator(controller):
@@ -206,15 +202,16 @@ class TestOnDemandDuringVegas:
"interrupt checker never saw the request")
assert controller.on_demand_active
def test_an_empty_mailbox_lets_the_iteration_run_out(self, controller, clock):
def test_a_quiet_iteration_runs_out(self, controller, clock):
coord = vegas_coordinator(controller)
passes = count_passes(controller)
assert coord.run_iteration() is True
assert not controller.on_demand_active
# And the read is throttled: at most one per interval, not per check.
max_reads = VEGAS_ITERATION_SECONDS / controller.PENDING_CHANGES_INTERVAL + 1
assert 0 < mailbox_reads(controller) <= max_reads
# And the pass is throttled: at most one per interval, not per check.
max_passes = VEGAS_ITERATION_SECONDS / controller.PENDING_CHANGES_INTERVAL + 1
assert 0 < passes.call_count <= max_passes
checks = coord.frames // coord._interrupt_check_interval
assert mailbox_reads(controller) < checks / 2
assert passes.call_count < checks / 2
class TestBrightnessIsAppliedMidScreen:
@@ -294,23 +291,25 @@ class TestBrightnessIsAppliedMidScreen:
class TestThrottle:
def test_no_reads_or_brightness_calls_between_passes(self, controller, clock):
def test_no_passes_or_brightness_calls_between_passes(self, controller, clock):
controller._service_pending_changes()
reads = mailbox_reads(controller)
passes = count_passes(controller)
save_brightness(controller, 40)
post_request(controller)
for _ in range(500): # a few seconds of frames, all inside one interval
controller._service_pending_changes()
clock.t += controller.PENDING_CHANGES_INTERVAL / 1000
assert mailbox_reads(controller) == reads
assert passes.call_count == 0
controller.display_manager.set_brightness.assert_not_called()
assert not controller.on_demand_active
clock.t += controller.PENDING_CHANGES_INTERVAL
controller._service_pending_changes()
# The poll, plus the consume step's re-read of the request it acted on.
assert mailbox_reads(controller) > reads
assert passes.call_count == 1
controller.display_manager.set_brightness.assert_called_once_with(40)
def test_a_queued_request_skips_the_throttle(self, controller, clock):
controller._service_pending_changes()
post_request(controller)
controller._service_pending_changes() # inside the interval
assert controller.on_demand_active
def test_between_passes_it_does_no_work_at_all(self, controller, clock):
@@ -397,7 +396,7 @@ class TestScheduleAndDwells:
controller._service_pending_changes()
assert controller.on_demand_active
clock.t += controller.PENDING_CHANGES_INTERVAL
controller.mailbox['request'] = {'request_id': 'r2', 'action': 'stop'}
post_request(controller, request_id='r2', action='stop')
start = clock.t
controller._sleep_with_plugin_updates(30)
assert not controller.on_demand_active
+127 -200
View File
@@ -24,7 +24,7 @@ sys.path.insert(0, str(Path(__file__).parent.parent))
from src.cache_manager import CacheManager # noqa: E402
from src import error_aggregator as errors # noqa: E402
from src.error_aggregator import ( # noqa: E402
ERROR_CLEAR_REQUEST_KEY, ERROR_SNAPSHOT_KEY, ErrorAggregator,
ERROR_SNAPSHOT_KEY, ErrorAggregator,
ErrorSnapshotPublisher,
)
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: F401,E402
@@ -314,138 +314,19 @@ class TestRoutes:
assert secret not in published, secret
class TestClear:
def test_clear_is_applied_by_the_display_and_republished(self, web, display):
aggregator, publisher, _ = display
_fail(aggregator)
_fail(aggregator)
publisher.tick()
response = web.post("/api/v3/errors/clear", json={"all": True})
assert response.status_code == 200
body = response.get_json()["data"]
assert body["clear_requested"] is True
assert body["cleared_count"] == 2
# The display applies it on its next tick, throttle or not...
assert publisher.tick() is True
assert aggregator.get_error_summary()["total_errors"] == 0
data = _summary(web)
assert data["total_errors"] == 0 and data["clear_pending"] is False
# ...and only once.
assert publisher.tick() is False
def test_summary_hides_cleared_errors_before_the_display_applies_it(self, web, display):
aggregator, publisher, _ = display
for _ in range(5):
_fail(aggregator)
publisher.tick()
web.post("/api/v3/errors/clear", json={"all": True})
# No display tick yet.
data = _summary(web)
assert data["clear_pending"] is True
assert data["total_errors"] == 0
assert data["recent_errors"] == [] and data["active_patterns"] == {}
assert data["plugin_error_counts"] == {}
plugin = web.get("/api/v3/errors/plugin/p1").get_json()["data"]
assert plugin["status"] == "healthy" and plugin["total_errors"] == 0
def test_a_snapshot_written_just_before_the_clear_cannot_bring_errors_back(
self, web, display, shared_cache):
# The race: the display builds a snapshot, the user clicks Clear, and
# the display's write lands after the request.
aggregator, publisher, _ = display
display_cache, _, _ = shared_cache
for _ in range(3):
_fail(aggregator)
stale = aggregator.build_snapshot()
web.post("/api/v3/errors/clear", json={"all": True})
stale["applied_clear_id"] = None
display_cache.set(ERROR_SNAPSHOT_KEY, stale)
assert _summary(web)["total_errors"] == 0
def test_errors_after_the_clear_are_kept(self, web, display):
aggregator, publisher, clock = display
_fail(aggregator, plugin_id="before")
publisher.tick()
web.post("/api/v3/errors/clear", json={"all": True})
# An error lands after the request but before the display applies it.
for record in aggregator._records:
record.timestamp -= timedelta(seconds=5)
_fail(aggregator, plugin_id="after")
publisher.tick()
data = _summary(web)
assert data["plugin_error_counts"] == {"after": {"ValueError": 1}}
assert data["clear_pending"] is False
def test_age_based_clear(self, web, display):
aggregator, publisher, _ = display
_fail(aggregator, plugin_id="old")
_fail(aggregator, plugin_id="old")
for record in aggregator._records:
record.timestamp -= timedelta(hours=3)
_fail(aggregator, plugin_id="new")
publisher.tick()
body = web.post("/api/v3/errors/clear", json={"max_age_hours": 1}).get_json()["data"]
assert body["cleared_count"] == 2
pending = _summary(web)
assert pending["clear_pending"] is True
assert [r["plugin_id"] for r in pending["recent_errors"]] == ["new"]
publisher.tick()
data = _summary(web)
assert data["plugin_error_counts"] == {"new": {"ValueError": 1}}
assert data["total_errors"] == 1
def test_a_narrower_clear_does_not_undo_a_pending_wider_one(self, web, display):
aggregator, publisher, _ = display
for _ in range(3):
_fail(aggregator)
publisher.tick()
web.post("/api/v3/errors/clear", json={"all": True})
web.post("/api/v3/errors/clear", json={"max_age_hours": 24})
assert _summary(web)["total_errors"] == 0
publisher.tick()
assert aggregator.get_error_summary()["total_errors"] == 0
def test_default_body_still_means_older_than_24_hours(self, web, display):
aggregator, publisher, _ = display
_fail(aggregator)
publisher.tick()
response = web.post("/api/v3/errors/clear")
assert response.status_code == 200
assert response.get_json()["data"]["cleared_count"] == 0
publisher.tick()
assert _summary(web)["total_errors"] == 1
def test_validation_is_unchanged_but_all_skips_it(self, web):
assert web.post("/api/v3/errors/clear", json={"max_age_hours": 0}).status_code == 400
assert web.post("/api/v3/errors/clear", json={"max_age_hours": 9000}).status_code == 400
assert web.post("/api/v3/errors/clear",
json={"all": True, "max_age_hours": "junk"}).status_code == 200
def test_a_request_that_did_not_reach_the_cache_is_an_error(self, web, api_v3_module): # noqa: F811
cache = MagicMock()
cache.get.return_value = None
api_v3_module.api_v3.cache_manager = cache
response = web.post("/api/v3/errors/clear", json={"all": True})
assert response.status_code == 500
assert "clear request" in response.get_json()["message"]
CLIENT = "web_interface.blueprints.api_v3.control_client"
RETIRED_CLEAR_KEY = "plugin_error_clear_request"
def _mailbox_file(shared_cache):
_, _, directory = shared_cache
return directory / f"{ERROR_CLEAR_REQUEST_KEY}.json"
return directory / f"{RETIRED_CLEAR_KEY}.json"
class TestClearOverTheSocket:
"""``errors.clear``: the display applies the clear before it answers, and
the mailbox is written only when the socket could not carry it."""
@pytest.fixture
def socket_up(self, display, monkeypatch):
"""The control socket, as the display serves it: errors_clear runs
the display's own handler against its publisher."""
def socket_up(display, monkeypatch):
"""The control socket, as the display serves it: errors_clear runs the
display's own handler against its publisher."""
from src.ipc import client as control_client
from src.ipc.contract import ErrorsClearArgs
_, publisher, _ = display
@@ -460,6 +341,10 @@ class TestClearOverTheSocket:
assert control_client.errors_clear is errors_clear
return calls
class TestClear:
"""``errors.clear``: the display applies the clear before it answers."""
def test_socket_clear_is_applied_before_the_answer(self, web, display, socket_up,
shared_cache):
aggregator, publisher, _ = display
@@ -471,6 +356,7 @@ class TestClearOverTheSocket:
body = response.get_json()
data = body["data"]
assert data["transport"] == "socket" and data["applied"] is True
assert data["clear_requested"] is True
assert data["cleared_count"] == 3
assert body["message"] == "Cleared all errors"
[(request_id, _)] = socket_up
@@ -479,90 +365,134 @@ class TestClearOverTheSocket:
assert aggregator.get_error_summary()["total_errors"] == 0
summary = _summary(web)
assert summary["total_errors"] == 0 and summary["clear_pending"] is False
# And no mailbox file.
assert not _mailbox_file(shared_cache).exists()
# Nothing left for a tick to do.
assert publisher.tick() is False
def test_errors_after_the_clear_are_kept(self, web, display, socket_up):
aggregator, publisher, _ = display
_fail(aggregator, plugin_id="before")
for record in aggregator._records:
record.timestamp -= timedelta(seconds=5)
publisher.tick()
web.post("/api/v3/errors/clear", json={"all": True})
_fail(aggregator, plugin_id="after")
publisher.min_interval = 0
publisher.tick()
data = _summary(web)
assert data["plugin_error_counts"] == {"after": {"ValueError": 1}}
assert data["clear_pending"] is False
def test_age_based_clear(self, web, display, socket_up):
aggregator, publisher, _ = display
_fail(aggregator, plugin_id="old")
_fail(aggregator, plugin_id="old")
for record in aggregator._records:
record.timestamp -= timedelta(hours=3)
_fail(aggregator, plugin_id="new")
publisher.tick()
body = web.post("/api/v3/errors/clear", json={"max_age_hours": 1}).get_json()["data"]
assert body["cleared_count"] == 2
data = _summary(web)
assert data["plugin_error_counts"] == {"new": {"ValueError": 1}}
assert data["total_errors"] == 1
def test_default_body_still_means_older_than_24_hours(self, web, display, socket_up):
aggregator, publisher, _ = display
_fail(aggregator)
publisher.tick()
response = web.post("/api/v3/errors/clear")
assert response.status_code == 200
assert response.get_json()["data"]["cleared_count"] == 0
assert _summary(web)["total_errors"] == 1
def test_validation_is_unchanged_but_all_skips_it(self, web, socket_up):
assert web.post("/api/v3/errors/clear", json={"max_age_hours": 0}).status_code == 400
assert web.post("/api/v3/errors/clear", json={"max_age_hours": 9000}).status_code == 400
assert web.post("/api/v3/errors/clear",
json={"all": True, "max_age_hours": "junk"}).status_code == 200
class TestClearWithoutTheSocket:
"""Stage 5: no mailbox to fall back to. The route says why it failed and
writes nothing; the errors stay as the display last reported them."""
def _post(self, web, display, monkeypatch, error):
monkeypatch.setattr(f"{CLIENT}.errors_clear", MagicMock(side_effect=error))
aggregator, publisher, _ = display
_fail(aggregator)
publisher.tick()
return web.post("/api/v3/errors/clear", json={"all": True})
@pytest.mark.parametrize("reason", ["no_socket", "refused"])
def test_a_stopped_display_is_an_error(self, web, display, shared_cache, monkeypatch,
reason):
from src.ipc import client as control_client
response = self._post(web, display, monkeypatch,
control_client.ControlError(reason, sent=False))
assert response.status_code == 503
body = response.get_json()
assert body["context"]["socket_error"] == reason
assert "not running" in body["message"]
assert not _mailbox_file(shared_cache).exists()
# Nothing hides them: they are still the display's last report.
summary = _summary(web)
assert summary["total_errors"] == 1 and summary["clear_pending"] is False
@pytest.mark.parametrize("reason", ["disabled", "unsupported"])
def test_no_socket_here_is_an_error(self, web, display, shared_cache, monkeypatch, reason):
from src.ipc import client as control_client
response = self._post(web, display, monkeypatch,
control_client.ControlError(reason, sent=False))
assert response.status_code == 503
assert "not available" in response.get_json()["message"]
assert not _mailbox_file(shared_cache).exists()
def test_an_older_mailbox_request_does_not_read_as_pending(self, web, display, socket_up,
shared_cache, monkeypatch):
# A clear that went to the mailbox while the socket was down, then a
# wider one over the socket: the old request has nothing left to hide.
from unittest.mock import patch
def test_an_older_display_is_told_to_restart(self, web, display, shared_cache,
monkeypatch):
from src.ipc import client as control_client
aggregator, publisher, _ = display
_fail(aggregator)
publisher.tick()
with patch(f"{CLIENT}.errors_clear",
side_effect=control_client.ControlError("no_socket")):
data = web.post("/api/v3/errors/clear", json={"max_age_hours": 1}).get_json()["data"]
assert data["transport"] == "mailbox"
assert _mailbox_file(shared_cache).exists()
data = web.post("/api/v3/errors/clear", json={"all": True}).get_json()["data"]
assert data["transport"] == "socket"
assert _summary(web)["clear_pending"] is False
@pytest.mark.parametrize("reason", ["no_socket", "refused", "disabled", "unsupported"])
def test_no_socket_writes_the_mailbox(self, web, display, shared_cache, monkeypatch, reason):
from src.ipc import client as control_client
monkeypatch.setattr(f"{CLIENT}.errors_clear", MagicMock(
side_effect=control_client.ControlError(reason, sent=False)))
aggregator, publisher, _ = display
_fail(aggregator)
publisher.tick()
data = web.post("/api/v3/errors/clear", json={"all": True}).get_json()["data"]
assert data["transport"] == "mailbox" and data["applied"] is False
assert _mailbox_file(shared_cache).exists()
assert _summary(web)["clear_pending"] is True
publisher.tick()
assert aggregator.get_error_summary()["total_errors"] == 0
def test_an_older_display_gets_the_mailbox(self, web, display, shared_cache, monkeypatch):
# The upgrade case: new web interface, a display from before errors.clear.
from src.ipc import client as control_client
monkeypatch.setattr(f"{CLIENT}.errors_clear", MagicMock(
side_effect=control_client.ControlError("unknown_command", sent=True)))
aggregator, publisher, _ = display
_fail(aggregator)
publisher.tick()
data = web.post("/api/v3/errors/clear", json={"all": True}).get_json()["data"]
assert data["transport"] == "mailbox"
publisher.tick()
assert aggregator.get_error_summary()["total_errors"] == 0
response = self._post(web, display, monkeypatch,
control_client.ControlError("unknown_command", sent=True))
assert response.status_code == 503
assert "restart" in response.get_json()["message"]
assert not _mailbox_file(shared_cache).exists()
@pytest.mark.parametrize("reason", ["internal", "timeout", "busy", "invalid_args"])
def test_a_display_that_had_it_and_failed_is_an_error(self, web, display, shared_cache,
monkeypatch, reason):
from src.ipc import client as control_client
monkeypatch.setattr(f"{CLIENT}.errors_clear", MagicMock(
side_effect=control_client.ControlError(reason, sent=True)))
response = web.post("/api/v3/errors/clear", json={"all": True})
response = self._post(web, display, monkeypatch,
control_client.ControlError(reason, sent=True))
assert response.status_code == 503
assert response.get_json()["context"]["socket_error"] == reason
body = response.get_json()
assert body["context"]["socket_error"] == reason
assert body["message"] == "The display service did not apply the clear"
assert not _mailbox_file(shared_cache).exists()
class TestPublisherMailboxPoll:
def test_the_mailbox_is_read_only_when_its_file_changed(self, display, shared_cache):
_, publisher, _ = display
_, web_cache, _ = shared_cache
def test_the_default_test_setup_has_no_socket(self, web, display, shared_cache):
# conftest turns the socket off: the real client answers "disabled".
aggregator, publisher, _ = display
_fail(aggregator)
publisher.tick()
response = web.post("/api/v3/errors/clear", json={"all": True})
assert response.status_code == 503
assert response.get_json()["context"]["socket_error"] in ("disabled", "unsupported")
class TestPublisher:
def test_a_tick_reads_no_clear_request(self, display, shared_cache):
_, publisher, clock = display
_, web_cache, directory = shared_cache
# An old web interface's leftover request file is ignored.
(directory / f"{RETIRED_CLEAR_KEY}.json").write_text(
'{"timestamp": 1, "data": {"request_id": "old", "cutoff": 9e9}}')
publisher.cache_manager = MagicMock(wraps=publisher.cache_manager)
def reads():
return [c for c in publisher.cache_manager.get.call_args_list
if c.args[0] == ERROR_CLEAR_REQUEST_KEY]
for _ in range(5):
clock.now += 60
publisher.tick()
assert reads() == [] # no file: a stat per tick, no read
errors.request_error_clear(web_cache, 1.0)
publisher.tick()
assert len(reads()) == 1
for _ in range(5):
publisher.tick()
assert len(reads()) == 1 # unchanged file: not read again
errors.request_error_clear(web_cache, 2.0)
publisher.tick()
assert len(reads()) == 2
publisher.cache_manager.get.assert_not_called()
snapshot = web_cache.get(ERROR_SNAPSHOT_KEY, max_age=None, memory_ttl=0)
assert snapshot["applied_clear_id"] is None
def test_clear_now_publishes_what_it_applied(self, display, shared_cache):
aggregator, publisher, _ = display
@@ -572,7 +502,6 @@ class TestPublisherMailboxPoll:
snapshot = web_cache.get(ERROR_SNAPSHOT_KEY, max_age=None, memory_ttl=0)
assert snapshot["applied_clear_id"] == "sock-1"
assert snapshot["total_errors"] == 0
assert snapshot["applied_clear_cutoff"] is not None
def test_the_handler_needs_a_running_publisher(self, monkeypatch):
from src.ipc.contract import ErrorsClearArgs
@@ -583,12 +512,10 @@ class TestPublisherMailboxPoll:
@pytest.mark.skipif(not hasattr(os, "fchmod") or os.name == "nt",
reason="POSIX file modes")
def test_both_files_are_group_readable(web, display, shared_cache):
def test_the_snapshot_is_group_readable(web, display, shared_cache):
aggregator, publisher, _ = display
_, _, directory = shared_cache
_fail(aggregator)
publisher.tick()
web.post("/api/v3/errors/clear", json={"all": True})
for key in (ERROR_SNAPSHOT_KEY, ERROR_CLEAR_REQUEST_KEY):
mode = stat.S_IMODE(os.stat(directory / f"{key}.json").st_mode)
assert mode == 0o660, (key, oct(mode))
mode = stat.S_IMODE(os.stat(directory / f"{ERROR_SNAPSHOT_KEY}.json").st_mode)
assert mode == 0o660, oct(mode)
+325
View File
@@ -0,0 +1,325 @@
"""A stale .git/index.lock must not block Update Code forever.
subprocess.run's timeout SIGKILLs git, and a killed git cannot remove its
index.lock. Every later update then failed with "Unable to create
'.../.git/index.lock': File exists" until someone removed the file over SSH.
The update now removes the lock when no git process can still own it.
"""
import os
import shutil
import subprocess
import sys
import time
from pathlib import Path
import contextlib
from unittest.mock import patch
import pytest
from flask import Flask
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from web_interface import git_index_lock as gil # noqa: E402
from web_interface.blueprints import api_v3 as mod # noqa: E402
from web_interface.blueprints.api_v3 import api_v3 # noqa: E402
needs_git = pytest.mark.skipif(shutil.which('git') is None, reason='git not installed')
OLD = 1_000_000.0
@contextlib.contextmanager
def _patched(fake):
"""Fake git for both the plain subprocess calls and the timeout-safe runner."""
with patch.object(mod.subprocess, 'run', fake), patch.object(mod.git_index_lock, 'run_git', fake):
yield
def _checkout(tmp_path, lock_age=None):
repo = tmp_path / 'LEDMatrix'
(repo / '.git').mkdir(parents=True)
if lock_age is not None:
lock = repo / '.git' / 'index.lock'
lock.write_text('')
os.utime(lock, (OLD, OLD))
return repo
def _proc(tmp_path, *procs):
"""A fake /proc: ``procs`` are (comm, cwd[, argv]); cwd None means unreadable."""
root = tmp_path / 'proc'
root.mkdir()
(root / 'self').mkdir() # non-numeric entries are skipped
for pid, (comm, cwd, *argv) in enumerate(procs, start=100):
entry = root / str(pid)
entry.mkdir()
(entry / 'comm').write_text(comm + '\n')
(entry / 'cmdline').write_bytes(b'\0'.join(a.encode() for a in [comm, *argv]) + b'\0')
if cwd is not None:
(entry / 'cwd').symlink_to(cwd)
else:
(entry / 'cwd').mkdir() # readlink on a directory fails: unreadable
return root
class TestClearStaleIndexLock:
def test_no_lock_is_left_alone(self, tmp_path):
repo = _checkout(tmp_path)
assert gil.clear_stale_index_lock(repo, proc_root=_proc(tmp_path)) == gil.ABSENT
def test_a_stale_lock_is_removed(self, tmp_path):
repo = _checkout(tmp_path, lock_age=60)
proc = _proc(tmp_path, ('python3', str(repo)), ('git', str(tmp_path / 'elsewhere')))
assert gil.clear_stale_index_lock(repo, proc_root=proc, now=OLD + 60) == gil.REMOVED
assert not (repo / '.git' / 'index.lock').exists()
@pytest.mark.parametrize('comm', ['git', 'git-remote-http'])
def test_a_lock_held_by_a_running_git_is_kept(self, tmp_path, comm):
repo = _checkout(tmp_path, lock_age=60)
proc = _proc(tmp_path, (comm, str(repo / 'src')))
assert gil.clear_stale_index_lock(repo, proc_root=proc, now=OLD + 60) == gil.IN_USE
assert (repo / '.git' / 'index.lock').exists()
def test_a_git_pointed_at_the_checkout_from_elsewhere_is_running(self, tmp_path):
repo = _checkout(tmp_path, lock_age=60)
proc = _proc(tmp_path, ('git', str(tmp_path), f'--git-dir={repo.resolve()}/.git',
f'--work-tree={repo.resolve()}', 'checkout', 'v3.8.7'))
assert gil.clear_stale_index_lock(repo, proc_root=proc, now=OLD + 60) == gil.IN_USE
assert (repo / '.git' / 'index.lock').exists()
@pytest.mark.parametrize('argv', [['-C', 'LEDMatrix', 'status'],
['--git-dir=LEDMatrix/.git', '--work-tree=LEDMatrix', 'add', '.']])
def test_a_git_pointed_at_the_checkout_by_a_relative_path_is_running(self, tmp_path, argv):
repo = _checkout(tmp_path, lock_age=60)
proc = _proc(tmp_path, ('git', str(repo.resolve().parent), *argv))
assert gil.clear_stale_index_lock(repo, proc_root=proc, now=OLD + 60) == gil.IN_USE
def test_a_git_working_on_another_repository_is_not(self, tmp_path):
repo = _checkout(tmp_path, lock_age=60)
proc = _proc(tmp_path, ('git', str(tmp_path), '-C', 'other', 'pull'),
('git', str(tmp_path), '--git-dir=LEDMatrix-backup/.git', 'fetch'))
assert gil.clear_stale_index_lock(repo, proc_root=proc, now=OLD + 60) == gil.REMOVED
def test_a_git_whose_cwd_cannot_be_read_counts_as_running(self, tmp_path):
repo = _checkout(tmp_path, lock_age=60)
proc = _proc(tmp_path, ('git', None))
assert gil.clear_stale_index_lock(repo, proc_root=proc, now=OLD + 60) == gil.IN_USE
assert (repo / '.git' / 'index.lock').exists()
def test_a_brand_new_lock_is_kept(self, tmp_path):
"""A git may have created it between the process scan and the removal."""
repo = _checkout(tmp_path, lock_age=0)
assert gil.clear_stale_index_lock(repo, proc_root=_proc(tmp_path), now=OLD + 1) == gil.IN_USE
assert (repo / '.git' / 'index.lock').exists()
def test_without_proc_nothing_is_removed(self, tmp_path):
repo = _checkout(tmp_path, lock_age=60)
missing = tmp_path / 'no-proc'
assert gil.clear_stale_index_lock(repo, proc_root=missing, now=OLD + 60) == gil.UNKNOWN
assert (repo / '.git' / 'index.lock').exists()
def test_a_checkout_without_a_git_directory_is_left_alone(self, tmp_path):
assert gil.clear_stale_index_lock(tmp_path, proc_root=_proc(tmp_path)) == gil.ABSENT
@needs_git
@pytest.mark.skipif(not Path('/proc/self/comm').exists(), reason='needs /proc')
def test_git_works_again_after_a_real_stale_lock_is_cleared(tmp_path):
repo = tmp_path / 'repo'
env = {**os.environ, 'GIT_AUTHOR_NAME': 't', 'GIT_AUTHOR_EMAIL': 't@t',
'GIT_COMMITTER_NAME': 't', 'GIT_COMMITTER_EMAIL': 't@t'}
def git(*args):
return subprocess.run(['git', *args], cwd=repo, env=env, capture_output=True, text=True)
repo.mkdir()
git('init', '-q')
(repo / 'a.txt').write_text('a')
lock = repo / '.git' / 'index.lock'
lock.write_text('')
old = time.time() - 60
os.utime(lock, (old, old))
assert 'index.lock' in git('add', 'a.txt').stderr
assert gil.clear_stale_index_lock(repo) == gil.REMOVED
assert git('add', 'a.txt').returncode == 0
# -- Update Code ----------------------------------------------------------------
@pytest.fixture
def client():
app = Flask(__name__)
app.config['TESTING'] = True
app.register_blueprint(api_v3, url_prefix='/api/v3')
api_v3.plugin_store_manager = None
api_v3.config_manager = None
return app.test_client()
def _fake_git(calls, pull_rc=0, pull_err=''):
def run(args, **kwargs):
calls.append(list(args))
if args[:2] == ['git', 'rev-parse'] and args[-1] == 'HEAD':
return subprocess.CompletedProcess(args, 0, stdout='aaa111\n', stderr='')
if 'symbolic-full-name' in args or '@{u}' in args:
return subprocess.CompletedProcess(args, 0, stdout='origin/main\n', stderr='')
if args[:2] == ['git', 'pull']:
return subprocess.CompletedProcess(args, pull_rc, stdout='', stderr=pull_err)
return subprocess.CompletedProcess(args, 0, stdout='', stderr='')
return run
def _update(client):
return client.post('/api/v3/system/action', json={'action': 'git_pull'}).get_json()
def test_update_code_clears_the_lock_before_running_git(client):
seen = []
def clear(project_dir):
seen.append(project_dir)
return gil.REMOVED
calls = []
with patch.object(gil, 'clear_stale_index_lock', clear), \
_patched(_fake_git(calls)):
data = _update(client)
assert seen == [str(mod.PROJECT_ROOT)]
assert data['status'] == 'success'
def test_update_code_waits_for_a_git_that_is_still_running(client):
calls = []
with patch.object(gil, 'clear_stale_index_lock', lambda d: gil.IN_USE), \
_patched(_fake_git(calls)):
data = _update(client)
assert data['status'] == 'error'
assert 'Another git command is running' in data['message']
assert calls == [], "nothing may touch the checkout while another git owns it"
def test_a_lock_it_could_not_clear_gets_the_manual_fix(client):
err = "fatal: Unable to create '/home/ledpi/LEDMatrix/.git/index.lock': File exists.\n"
calls = []
with patch.object(gil, 'clear_stale_index_lock', lambda d: gil.UNKNOWN), \
_patched(_fake_git(calls, pull_rc=128, pull_err=err)):
data = _update(client)
assert data['status'] == 'error'
assert data['message'].startswith('Update failed: fatal: Unable to create')
assert 'rm -f ' in data['message'] and 'index.lock' in data['message']
# -- run_git: a timed-out git must not leave a lock or an orphaned helper ------
def _lock_holder_script(tmp_path, lock):
"""A stand-in for a slow git: takes the lock, spawns a helper, then hangs."""
script = tmp_path / 'slow.sh'
script.write_text(f'#!/bin/sh\n: > "{lock}"\nsleep 60 &\necho $! > "{tmp_path}/helper.pid"\nwait\n')
script.chmod(0o755)
return script
def test_run_git_returns_a_completed_process(tmp_path):
result = gil.run_git([sys.executable, '-c', 'print("hi")'], cwd=str(tmp_path), timeout=30)
assert (result.returncode, result.stdout.strip()) == (0, 'hi')
@pytest.mark.skipif(not os.path.isdir('/proc'), reason='needs /proc')
def test_run_git_timeout_stops_the_group_and_clears_the_lock(tmp_path, monkeypatch):
(tmp_path / '.git').mkdir()
lock = tmp_path / '.git' / 'index.lock'
script = _lock_holder_script(tmp_path, lock)
monkeypatch.setattr(gil, 'MIN_STALE_AGE_SECONDS', 0)
monkeypatch.setattr(gil, 'TERM_GRACE_SECONDS', 1)
# The scan only sees processes named git, so the stand-in must not match;
# the lock is then provably unowned once the group is gone.
with pytest.raises(subprocess.TimeoutExpired):
gil.run_git([str(script)], cwd=str(tmp_path), timeout=1)
assert not lock.exists()
helper = int((tmp_path / 'helper.pid').read_text())
time.sleep(0.2)
with pytest.raises(ProcessLookupError):
os.kill(helper, 0)
def test_read_only_env_disables_optional_locks():
assert gil.read_only_env()['GIT_OPTIONAL_LOCKS'] == '0'
def test_run_git_waits_out_the_freshness_window_before_giving_up_on_the_lock(tmp_path, monkeypatch):
# The lock is created a moment before the timeout, so on age alone it is
# still "in use" once git is stopped; it must not be left to block the retry.
(tmp_path / '.git').mkdir()
lock = tmp_path / '.git' / 'index.lock'
script = tmp_path / 'slow.sh'
script.write_text(f'#!/bin/sh\n: > "{lock}"\nexec sleep 60\n')
script.chmod(0o755)
monkeypatch.setattr(gil, 'MIN_STALE_AGE_SECONDS', 2)
monkeypatch.setattr(gil, 'TERM_GRACE_SECONDS', 1)
with pytest.raises(subprocess.TimeoutExpired):
gil.run_git([str(script)], cwd=str(tmp_path), timeout=1)
assert not lock.exists()
# -- a rebase in the checkout ----------------------------------------------------
def _rebase_checkout(tmp_path, monkeypatch):
from web_interface.blueprints.api_v3 import system
(tmp_path / '.git' / 'rebase-merge').mkdir(parents=True)
monkeypatch.setattr(system, 'PROJECT_ROOT', tmp_path)
return tmp_path / '.git' / 'rebase-merge'
def _rebase_git(calls, on_branch, quit_rc=0, remove_dir=None):
base = _fake_git(calls)
def run(args, **kwargs):
if args[:2] == ['git', 'symbolic-ref']:
calls.append(list(args))
return subprocess.CompletedProcess(args, 0 if on_branch else 1, stdout='', stderr='')
if args[:3] == ['git', 'rebase', '--quit']:
calls.append(list(args))
if quit_rc == 0 and remove_dir is not None:
remove_dir.rmdir()
return subprocess.CompletedProcess(args, quit_rc, stdout='', stderr='cannot quit' if quit_rc else '')
return base(args, **kwargs)
return run
def test_a_rebase_in_progress_is_left_alone(client, tmp_path, monkeypatch):
_rebase_checkout(tmp_path, monkeypatch)
calls = []
with _patched(_rebase_git(calls, on_branch=False)):
data = _update(client)
assert data['status'] == 'error' and 'rebase is in progress' in data['message']
assert not any(c[:2] in (['git', 'pull'], ['git', 'checkout']) or c[:3] == ['git', 'rebase', '--quit']
for c in calls), "a paused rebase holds the user's conflict work"
def test_an_abandoned_rebase_is_quit_before_the_pull(client, tmp_path, monkeypatch):
rebase = _rebase_checkout(tmp_path, monkeypatch)
calls = []
with _patched(_rebase_git(calls, on_branch=True, remove_dir=rebase)):
_update(client)
names = [c[1] for c in calls]
assert ['git', 'rebase', '--quit'] in calls
assert names.index('rebase') < names.index('pull')
def test_a_rebase_that_cannot_be_cleared_stops_the_update(client, tmp_path, monkeypatch):
_rebase_checkout(tmp_path, monkeypatch)
calls = []
with _patched(_rebase_git(calls, on_branch=True, quit_rc=1)):
data = _update(client)
assert data['status'] == 'error' and 'could not be cleared' in data['message']
assert not any(c[:2] == ['git', 'pull'] for c in calls)
def test_the_automatic_status_check_is_read_only_whatever_the_runner(tmp_path):
from web_interface import auto_update
seen = {}
def run(args, **kwargs):
seen.update(kwargs)
return subprocess.CompletedProcess(args, 0, stdout='', stderr='')
auto_update.local_changes(tmp_path, run=run)
assert seen['env']['GIT_OPTIONAL_LOCKS'] == '0'
+3 -3
View File
@@ -3,7 +3,7 @@
Pure data, so every test here runs on every platform. What they pin:
* a request and a response survive encode -> decode -> parse unchanged, and
the on-demand arguments carry exactly what the file mailbox carries;
the on-demand arguments carry exactly what the REST route sends;
* the envelope and the arguments refuse what the display could not act on
(missing ids, wrong types, a non-finite duration) with a stable error code;
* framing never holds more than one message's worth of bytes, however the
@@ -186,8 +186,8 @@ class TestOnDemandArgs:
class TestMailboxShape:
"""Socket commands are handed to the mailbox's own handler, so they must
look exactly like what the web route writes to the mailbox."""
"""Socket commands are handed to the display's on-demand handler, so they
must look exactly like the request dict it takes."""
def test_start(self):
args = OnDemandStartArgs(plugin_id='clock', mode='clock_main', duration=60.0,
+34 -58
View File
@@ -1,14 +1,13 @@
"""DisplayController's side of the control socket.
The server's handlers only queue; the render thread drains the queue where
it reads the file mailbox (_poll_on_demand_requests) and hands each command
to the mailbox's own handler (_handle_on_demand_request). These tests pin
that hook:
The server's handlers only queue; the render thread drains the queue
(_poll_on_demand_requests) and hands each on-demand command to
_handle_on_demand_request, which plugins' own requests use too. These tests
pin that hook:
* a socket command is applied by the same code as a mailbox request, with
its request id, and without waiting for the mailbox's 0.25 s read floor;
* a request that arrives both ways (a client that timed out after the
command was queued, then wrote the mailbox) is activated once;
* a socket command is applied with its request id, at once, and touches no
cache key (the file mailbox and the persisted processed id are gone);
* a start sent twice with one request id is activated once;
* a command that fails is contained, and the ones after it still run;
* cleanup closes the socket; a disabled socket changes nothing.
"""
@@ -58,28 +57,15 @@ def controller(test_display_controller):
c_ = test_display_controller
c_.on_demand_active = False
c_.on_demand_request_id = None
c_._last_on_demand_poll = None
mailbox = {'value': None}
def fake_get(key, *a, **kw):
if key == 'display_on_demand_request':
return mailbox['value']
return None
def fake_delete(key):
if key == 'display_on_demand_request':
mailbox['value'] = None
c_.cache_manager.get = MagicMock(side_effect=fake_get)
c_.cache_manager.get = MagicMock(return_value=None)
c_.cache_manager.set = MagicMock()
c_.cache_manager.delete = MagicMock(side_effect=fake_delete)
c_.cache_manager.delete = MagicMock()
c_._activate_on_demand = MagicMock()
c_.mailbox = mailbox
return c_
class TestDrain:
def test_a_socket_start_goes_through_the_mailbox_handler(self, controller):
def test_a_socket_start_is_activated_with_its_request_id(self, controller):
controller._control_server = FakeServer(_start('sock-1', duration=30.0, pinned=True))
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_called_once()
@@ -89,47 +75,28 @@ class TestDrain:
assert request['plugin_id'] == 'clock'
assert request['duration'] == 30.0 and request['pinned'] is True
assert controller.on_demand_request_id == 'sock-1'
# The same restart-replay guard as a mailbox request.
controller.cache_manager.set.assert_any_call(
'display_on_demand_processed_id', 'sock-1', ttl=3600)
# No mailbox, and no persisted processed id.
keys = {call.args[0] for m in (controller.cache_manager.get,
controller.cache_manager.set,
controller.cache_manager.delete)
for call in m.call_args_list}
assert not keys & {'display_on_demand_request', 'display_on_demand_processed_id'}
def test_socket_commands_skip_the_mailbox_floor(self, controller):
def test_socket_commands_land_at_once(self, controller):
server = FakeServer()
controller._control_server = server
controller._poll_on_demand_requests() # reads the mailbox, sets the floor
reads = controller.cache_manager.get.call_count
controller._poll_on_demand_requests()
server.commands.append(_start('quick'))
controller._poll_on_demand_requests() # within the floor
controller._activate_on_demand.assert_called_once()
mailbox_reads = [call for call in controller.cache_manager.get.call_args_list[reads:]
if call.args[0] == 'display_on_demand_request']
# Only _consume_on_demand_request's compare-before-delete re-read.
assert len(mailbox_reads) <= 1
def test_a_request_that_came_both_ways_is_activated_once(self, controller):
controller._control_server = FakeServer(_start('dup'))
controller.mailbox['value'] = {'request_id': 'dup', 'action': 'start',
'plugin_id': 'clock'}
controller._poll_on_demand_requests()
controller._last_on_demand_poll = None
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_called_once()
assert controller.mailbox['value'] is None, "the duplicate was left in the mailbox"
def test_a_fallback_write_landing_later_is_ignored(self, controller):
controller._control_server = FakeServer(_start('late'))
controller._poll_on_demand_requests()
controller.mailbox['value'] = {'request_id': 'late', 'action': 'start',
'plugin_id': 'clock'}
controller._last_on_demand_poll = None
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_called_once()
def test_the_mailbox_still_works_alongside(self, controller):
controller._control_server = FakeServer()
controller.mailbox['value'] = {'request_id': 'mb', 'action': 'start', 'plugin_id': 'p'}
def test_a_start_sent_twice_is_activated_once(self, controller):
server = FakeServer(_start('dup'))
controller._control_server = server
controller._poll_on_demand_requests()
assert controller._activate_on_demand.call_args.args[0]['request_id'] == 'mb'
server.commands.append(_start('dup'))
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_called_once()
def test_a_socket_stop_ends_on_demand(self, controller):
controller.on_demand_active = True
@@ -159,10 +126,11 @@ class TestDrain:
controller._poll_on_demand_requests()
assert calls == ['bad', 'good']
def test_no_server_means_mailbox_only(self, controller):
def test_no_server_means_nothing_to_apply(self, controller):
controller._control_server = None
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_not_called()
controller.cache_manager.get.assert_not_called()
class TestPendingChangesFloor:
@@ -194,6 +162,14 @@ class TestLifecycle:
assert status['on_demand']['plugin_id'] == 'clock'
c.encode_message(status) # it has to fit on the wire
def test_the_state_names_the_request_it_answers(self, controller):
# The web interface keeps reporting a delivered start as "starting"
# until the display publishes state for that request id.
assert controller._on_demand_state()['request_id'] is None
controller._control_server = FakeServer(_start('sock-7'))
controller._poll_on_demand_requests()
assert controller._on_demand_state()['request_id'] == 'sock-7'
def test_cleanup_closes_the_socket(self, controller):
server = FakeServer()
controller._control_server = server
+139 -204
View File
@@ -1,14 +1,13 @@
"""Stage 4 of the control socket: the file mailboxes are only a fallback.
"""Stages 4 and 5 of the control socket: the socket is the only way in.
* The client knows whether the display had the request (``ControlError.sent``)
and ``should_fall_back`` allows a mailbox write only when it did not, or
when the display is too old to know the command (the upgrade case).
and ``display_not_listening`` says when no display was there to take it
(stopped or still starting), the one case a later retry can fix.
* ``errors.clear`` is answered on the connection thread by a handler the
display registers; a display without one answers like an older display.
* The display looks at the on-demand mailbox once a second while the socket
is up (0.25 s without it), reads it only when its file changed, never
touches it for a socket command, and logs who still writes it.
* ``CacheManager.file_signature`` / ``MailboxWatch`` make a look one stat().
* Stage 5: the display no longer reads the ``display_on_demand_request``
mailbox at all, and the cache refuses writes to the retired mailbox keys,
warning once per writer.
The web routes are covered in test_api_v3_on_demand_socket.py and
test_error_snapshot_cross_process.py.
@@ -23,7 +22,8 @@ from unittest.mock import MagicMock, patch
import pytest
from src.cache_manager import CacheManager, MailboxWatch
from src import cache_manager as cache_module
from src.cache_manager import CacheManager
from src.ipc import client
from src.ipc import contract as c
from src.ipc.contract import Command, ErrorsClearArgs, OnDemandStartArgs, ProtocolError
@@ -81,38 +81,41 @@ class TestSent:
def test_an_answer_is_returned(self):
assert _call(FakeSock([_reply('rid-1', ok=True, result={'pong': True})])) == {'pong': True}
@pytest.mark.parametrize('reason', ['no_socket', 'refused', 'timeout', 'busy'])
def test_a_failed_connect_was_not_sent(self, reason):
@pytest.mark.parametrize('reason,listening', [('no_socket', False), ('refused', False),
('timeout', True), ('busy', True)])
def test_a_failed_connect_was_not_sent(self, reason, listening):
e = _error(connect_error=client.ControlError(reason))
assert e.reason == reason and e.sent is False
assert client.should_fall_back(e)
# Only "nothing there" is worth waiting for: a timeout or a full
# backlog is a display that is there and stuck.
assert client.display_not_listening(e) is not listening
def test_a_send_that_timed_out_was_not_sent(self):
e = _error(sock=FakeSock(send_error=socket.timeout()))
assert e.reason == 'timeout' and e.sent is False
assert client.should_fall_back(e)
assert not client.display_not_listening(e)
def test_silence_after_the_request_was_sent(self):
e = _error(sock=FakeSock(recv_error=socket.timeout()))
assert e.reason == 'timeout' and e.sent is True
assert not client.should_fall_back(e)
assert not client.display_not_listening(e)
def test_a_hang_up_after_the_request_was_sent(self):
e = _error(sock=FakeSock([]))
assert e.reason == 'closed' and e.sent is True
assert not client.should_fall_back(e)
assert not client.display_not_listening(e)
def test_a_garbled_reply(self):
e = _error(sock=FakeSock([b'not json\n']))
assert e.reason == 'bad_response' and e.sent is True
assert not client.should_fall_back(e)
assert not client.display_not_listening(e)
@pytest.mark.parametrize('code', ['busy', 'invalid_args', 'internal', 'pending', 'failed'])
def test_a_display_error_with_an_id_was_sent(self, code):
sock = FakeSock([_reply('rid-1', ok=False, error={'code': code, 'message': 'x'})])
e = _error(sock=sock)
assert e.reason == code and e.sent is True
assert not client.should_fall_back(e)
assert not client.display_not_listening(e)
@pytest.mark.parametrize('code', ['forbidden', 'busy'])
def test_a_refusal_at_the_door_was_not_sent(self, code):
@@ -122,23 +125,33 @@ class TestSent:
'error': {'code': code, 'message': 'x'}}) + '\n').encode()])
e = _error(sock=sock)
assert e.reason == code and e.sent is False
assert client.should_fall_back(e)
assert not client.display_not_listening(e)
@pytest.mark.parametrize('code', ['unknown_command', 'unsupported_version'])
def test_an_older_display_is_fallen_back_from(self, code):
def test_an_older_display_is_listening(self, code):
sock = FakeSock([_reply('rid-1', ok=False, error={'code': code, 'message': 'x'})])
e = _error(sock=sock)
assert e.sent is True
assert client.should_fall_back(e)
assert not client.display_not_listening(e)
def test_a_request_refused_locally_never_left(self):
with pytest.raises(client.ControlError) as e:
client.request(Command.ERRORS_CLEAR, {'cutoff': 'soon'}, paths=['/x.sock'])
assert e.value.reason == 'invalid_request' and e.value.sent is False
assert client.should_fall_back(e.value)
assert not client.display_not_listening(e.value)
def test_a_client_bug_falls_back(self):
assert client.should_fall_back(RuntimeError('boom'))
@pytest.mark.parametrize('reason', ['disabled', 'unsupported'])
def test_a_client_without_the_socket_is_not_waiting_for_a_display(self, reason):
assert not client.display_not_listening(client.ControlError(reason))
@pytest.mark.parametrize('reason', sorted(client.NOT_LISTENING_REASONS))
def test_a_request_that_was_sent_had_a_display(self, reason):
# Only a request that never left can be waited on and sent again: one
# that was sent may have been applied.
assert not client.display_not_listening(client.ControlError(reason, sent=True))
def test_a_client_bug_is_not_a_missing_display(self):
assert not client.display_not_listening(RuntimeError('boom'))
# -- errors.clear on the server ------------------------------------------------------
@@ -204,37 +217,28 @@ class TestErrorsClearOnTheServer:
assert Command.ERRORS_CLEAR in result['commands']
# -- the display's mailbox poll ------------------------------------------------------
# -- stage 5: the display reads no mailbox --------------------------------------------
class SignedCache:
"""The slice of CacheManager the poll uses, counting what it costs."""
class CountingCache:
"""The slice of CacheManager the on-demand path uses, counting every call."""
def __init__(self):
self.data = {}
self.writes = 0
self.version = {}
self.reads = []
self.stats = 0
self.deletes = []
self.sets = []
self.calls = []
def file_signature(self, key):
self.stats += 1
return (self.version[key], 0, 0) if key in self.data else None
def __getattr__(self, name):
# Any other method (file_signature, delete, ...) is recorded too.
def call(*a, **kw):
self.calls.append((name, a[0] if a else None))
return call
def get(self, key, *a, **kw):
self.reads.append(key)
self.calls.append(('get', key))
return self.data.get(key)
def set(self, key, value, *a, **kw):
self.sets.append(key)
self.calls.append(('set', key))
self.data[key] = value
self.writes += 1
self.version[key] = self.writes
def delete(self, key):
self.deletes.append(key)
self.data.pop(key, None)
class FakeServer:
@@ -250,209 +254,140 @@ class FakeServer:
return out
class Clock:
def __init__(self):
self.t = 1000.0
def __call__(self):
return self.t
@pytest.fixture
def controller(test_display_controller, monkeypatch):
def controller(test_display_controller):
dc = test_display_controller
dc.cache_manager = SignedCache()
dc.cache_manager = CountingCache()
dc._activate_on_demand = MagicMock()
dc.on_demand_active = False
dc.on_demand_request_id = None
dc._last_on_demand_poll = None
dc._on_demand_mailbox = None
dc._mailbox_writers_logged = frozenset()
clock = Clock()
monkeypatch.setattr('src.display_controller.time.monotonic', clock)
dc.clock = clock
return dc
def _post(dc, rid, action='start', **fields):
dc.cache_manager.set(MAILBOX, dict({'request_id': rid, 'action': action}, **fields))
def _poll_for(dc, seconds, step=1 / 16): # exact in binary: no drift past a floor
end = dc.clock.t + seconds
while dc.clock.t < end:
dc._poll_on_demand_requests()
dc.clock.t += step
class TestMailboxCadence:
def test_without_a_socket_it_is_looked_at_every_quarter_second(self, controller):
controller._control_server = None
_poll_for(controller, 10.0)
assert 38 <= controller.cache_manager.stats <= 42
def test_with_a_socket_it_is_looked_at_once_a_second(self, controller):
controller._control_server = FakeServer()
_poll_for(controller, 10.0)
assert 9 <= controller.cache_manager.stats <= 11
def test_a_look_that_finds_nothing_reads_nothing(self, controller):
controller._control_server = FakeServer()
_poll_for(controller, 10.0)
assert controller.cache_manager.reads == []
def test_an_unchanged_mailbox_is_not_read_again(self, controller):
# An already-processed start the delete could not remove, say.
controller._control_server = FakeServer()
controller.cache_manager.delete = MagicMock() # the file stays
_post(controller, 'once', plugin_id='clock')
_poll_for(controller, 10.0)
assert controller.cache_manager.reads.count(MAILBOX) <= 2 # the read + the re-check
controller._activate_on_demand.assert_called_once()
def test_a_mailbox_request_lands_within_a_second_with_the_socket_up(self, controller):
# The upgrade case the other way round: a new display, and a web
# interface (or a plugin) that still writes the mailbox.
controller._control_server = FakeServer()
class TestNoMailbox:
@pytest.mark.parametrize('with_socket', [False, True])
def test_polling_touches_no_cache_key(self, controller, with_socket):
controller._control_server = FakeServer() if with_socket else None
controller.cache_manager.data[MAILBOX] = {'request_id': 'left', 'action': 'start',
'plugin_id': 'clock'}
for _ in range(200):
controller._poll_on_demand_requests()
controller.clock.t += 0.1
_post(controller, 'old-web', plugin_id='clock')
posted = controller.clock.t
while not controller._activate_on_demand.called:
controller._poll_on_demand_requests()
controller.clock.t += 0.05
assert controller.clock.t - posted < 1.5
assert controller.clock.t - posted <= controller.MAILBOX_POLL_INTERVAL_WITH_SOCKET + 0.06
assert MAILBOX in controller.cache_manager.deletes # consumed
assert controller.cache_manager.calls == []
controller._activate_on_demand.assert_not_called()
def test_socket_commands_still_land_at_once(self, controller):
def test_socket_commands_land_at_once(self, controller):
server = controller._control_server = FakeServer()
controller._poll_on_demand_requests()
server.commands.append(QueuedCommand('sock', Command.ON_DEMAND_START,
OnDemandStartArgs(plugin_id='clock'), time.time()))
controller._poll_on_demand_requests() # inside the mailbox interval
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_called_once()
class TestSocketCommandsLeaveTheMailboxAlone:
def test_a_socket_start_reads_and_deletes_no_mailbox(self, controller):
def test_a_start_is_applied_once_and_writes_no_processed_id(self, controller):
server = controller._control_server = FakeServer()
controller._poll_on_demand_requests()
before = list(controller.cache_manager.reads)
server.commands.append(QueuedCommand('s1', Command.ON_DEMAND_START,
for _ in range(2):
server.commands.append(QueuedCommand('same', Command.ON_DEMAND_START,
OnDemandStartArgs(plugin_id='clock'), time.time()))
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_called_once()
assert MAILBOX not in controller.cache_manager.reads[len(before):]
assert controller.cache_manager.deletes == []
assert controller.cache_manager.calls == []
def test_a_socket_stop_reads_and_deletes_no_mailbox(self, controller):
def test_a_socket_stop_touches_no_cache_key(self, controller):
from src.ipc.contract import OnDemandStopArgs
controller.on_demand_active = True
controller._clear_on_demand = MagicMock()
server = controller._control_server = FakeServer()
server.commands.append(QueuedCommand('s2', Command.ON_DEMAND_STOP,
OnDemandStopArgs(), time.time()))
controller.clock.t += 5
controller._drain_control_commands()
controller._clear_on_demand.assert_called_once()
assert MAILBOX not in controller.cache_manager.reads
assert controller.cache_manager.deletes == []
assert controller.cache_manager.calls == []
def test_a_mailbox_copy_of_a_socket_command_is_dropped(self, controller):
# An older web interface timed out after the display queued the
# command, then wrote the mailbox too.
server = controller._control_server = FakeServer()
server.commands.append(QueuedCommand('both', Command.ON_DEMAND_START,
OnDemandStartArgs(plugin_id='clock'), time.time()))
controller._poll_on_demand_requests()
_post(controller, 'both', plugin_id='clock')
_poll_for(controller, 2.0)
controller._activate_on_demand.assert_called_once()
assert MAILBOX not in controller.cache_manager.data
def test_the_mailbox_helpers_are_gone(self):
from src import display_controller as dcm
from src import error_aggregator as ea
for name in ('ON_DEMAND_MAILBOX_KEY', 'MailboxWatch'):
assert not hasattr(dcm, name)
assert not hasattr(ea, 'ERROR_CLEAR_REQUEST_KEY')
assert not hasattr(cache_module, 'MailboxWatch')
assert not hasattr(CacheManager, 'file_signature')
assert not hasattr(client, 'should_fall_back')
for name in ('_consume_on_demand_request', '_note_mailbox_request',
'_mailbox_poll_interval', 'MAILBOX_POLL_INTERVAL_WITH_SOCKET'):
assert not hasattr(dcm.DisplayController, name)
class TestDeprecationLog:
def test_each_mailbox_writer_is_logged_once(self, controller, caplog):
controller._control_server = FakeServer()
caplog.set_level(logging.INFO, logger='src.display_controller')
for i, plugin in enumerate(['on-air', 'on-air', 'pomodoro-timer']):
_post(controller, f'r{i}', plugin_id=plugin)
_poll_for(controller, 1.2)
lines = [r.getMessage() for r in caplog.records if 'file mailbox' in r.getMessage()]
assert len(lines) == 2
assert 'on-air' in lines[0] and 'pomodoro-timer' in lines[1]
def test_nothing_is_logged_without_a_socket(self, controller, caplog):
controller._control_server = None
caplog.set_level(logging.INFO, logger='src.display_controller')
_post(controller, 'r', plugin_id='on-air')
_poll_for(controller, 1.0)
controller._activate_on_demand.assert_called_once()
assert not [r for r in caplog.records if 'file mailbox' in r.getMessage()]
# -- file_signature and MailboxWatch -------------------------------------------------
# -- stage 5: writes to the retired keys are refused, with one warning per writer -----
@pytest.fixture
def real_cache(tmp_path, monkeypatch):
monkeypatch.setattr(CacheManager, '_get_writable_cache_dir', lambda self: str(tmp_path))
monkeypatch.setattr(cache_module, '_retired_writers_warned', set())
cache = CacheManager()
yield cache
cache.stop_cleanup_thread()
class TestFileSignature:
def test_absent_key(self, real_cache):
assert real_cache.file_signature('nothing') is None
class OldPlugin:
"""What an old plugin looks like on the stack: BasePlugin gives every
plugin ``plugin_id`` and ``cache_manager``."""
def test_every_write_is_a_new_signature(self, real_cache):
seen = set()
for i in range(20):
# Same size each time, written as fast as possible.
real_cache.set(MAILBOX, {'request_id': f'r{i:02d}'})
sig = real_cache.file_signature(MAILBOX)
assert isinstance(sig, tuple)
seen.add(sig)
assert len(seen) == 20
def __init__(self, plugin_id, cache):
self.plugin_id = plugin_id
self.cache_manager = cache
def test_gone_after_a_delete(self, real_cache):
real_cache.set(MAILBOX, {'a': 1})
real_cache.delete(MAILBOX)
assert real_cache.file_signature(MAILBOX) is None
def trigger(self, target=None):
self.cache_manager.set(MAILBOX, {'request_id': 'r', 'action': 'start',
'plugin_id': target or self.plugin_id})
class TestMailboxWatch:
def test_reads_once_per_write(self, real_cache):
watch = MailboxWatch(MAILBOX)
assert watch.changed(real_cache) is False # no file
real_cache.set(MAILBOX, {'request_id': 'a'})
assert watch.changed(real_cache) is True
assert watch.changed(real_cache) is False
real_cache.set(MAILBOX, {'request_id': 'b'})
assert watch.changed(real_cache) is True
def _warnings(caplog):
return [r.getMessage() for r in caplog.records
if r.levelno == logging.WARNING and 'retired' in r.getMessage()]
def test_forget_reads_again(self, real_cache):
watch = MailboxWatch(MAILBOX)
real_cache.set(MAILBOX, {'request_id': 'a'})
assert watch.changed(real_cache) is True
watch.forget()
assert watch.changed(real_cache) is True
def test_a_rewrite_after_a_delete_is_seen(self, real_cache):
watch = MailboxWatch(MAILBOX)
real_cache.set(MAILBOX, {'request_id': 'a'})
assert watch.changed(real_cache)
real_cache.delete(MAILBOX)
assert watch.changed(real_cache) is False
real_cache.set(MAILBOX, {'request_id': 'a'})
assert watch.changed(real_cache) is True
class TestRetiredKeys:
@pytest.mark.parametrize('key', sorted(cache_module.RETIRED_MAILBOX_KEYS))
def test_a_write_stores_nothing(self, real_cache, key):
real_cache.set(key, {'request_id': 'r'})
real_cache.save_cache(key, {'request_id': 'r2'})
assert real_cache.get(key, max_age=None, memory_ttl=0) is None
assert not [f for f in os.listdir(real_cache.cache_dir) if key in f]
def test_a_cache_that_cannot_tell_is_read_every_time(self):
watch = MailboxWatch(MAILBOX)
assert watch.changed(MagicMock()) is True
assert watch.changed(MagicMock()) is True
assert watch.changed(object()) is True
def test_other_keys_are_unaffected(self, real_cache):
real_cache.set('display_on_demand_state', {'active': True})
assert real_cache.get('display_on_demand_state', max_age=None,
memory_ttl=0) == {'active': True}
def test_the_writing_plugin_is_named_once(self, real_cache, caplog):
caplog.set_level(logging.WARNING)
on_air = OldPlugin('on-air', real_cache)
for _ in range(3):
on_air.trigger()
OldPlugin('pomodoro-timer', real_cache).trigger()
lines = _warnings(caplog)
assert len(lines) == 2
assert "plugin 'on-air'" in lines[0] and 'display_on_demand_request' in lines[0]
assert 'request_on_demand' in lines[0]
assert "plugin 'pomodoro-timer'" in lines[1]
def test_the_writer_is_the_caller_not_the_target(self, real_cache, caplog):
caplog.set_level(logging.WARNING)
OldPlugin('mqtt-notifications', real_cache).trigger(target='clock')
(line,) = _warnings(caplog)
assert "plugin 'mqtt-notifications'" in line and 'clock' not in line
def test_without_a_plugin_on_the_stack_the_request_names_it(self, real_cache, caplog):
caplog.set_level(logging.WARNING)
real_cache.set(MAILBOX, {'request_id': 'r', 'action': 'start', 'plugin_id': 'gif-player'})
(line,) = _warnings(caplog)
assert "plugin 'gif-player' (named in the request)" in line
def test_otherwise_unknown(self, real_cache, caplog):
caplog.set_level(logging.WARNING)
real_cache.set('plugin_error_clear_request', {'request_id': 'r', 'cutoff': 1.0})
real_cache.set('plugin_error_clear_request', {'request_id': 'r2', 'cutoff': 2.0})
(line,) = _warnings(caplog)
assert 'by unknown' in line and '/api/v3/errors/clear' in line
# -- end to end over a real socket ---------------------------------------------------
@@ -479,24 +414,24 @@ class TestOverTheSocket:
finally:
server.close()
def test_an_older_display_is_an_upgrade_fallback(self, sock_path):
def test_an_older_display_is_listening(self, sock_path):
server = ControlServer(sock_path) # no errors.clear handler
assert server.start()
try:
with pytest.raises(client.ControlError) as e:
client.errors_clear('clr-2', 1.0, paths=[sock_path])
assert e.value.reason == 'unknown_command' and e.value.sent is True
assert client.should_fall_back(e.value)
assert not client.display_not_listening(e.value)
finally:
server.close()
def test_no_display_is_a_fallback(self, sock_path):
def test_no_display_is_not_listening(self, sock_path):
with pytest.raises(client.ControlError) as e:
client.errors_clear('clr-3', 1.0, paths=[sock_path])
assert e.value.reason == 'no_socket' and e.value.sent is False
assert client.should_fall_back(e.value)
assert client.display_not_listening(e.value)
def test_a_full_queue_is_not_a_fallback(self, sock_path):
def test_a_full_queue_is_listening(self, sock_path):
server = ControlServer(sock_path, queue_size=1)
assert server.start()
try:
@@ -504,6 +439,6 @@ class TestOverTheSocket:
with pytest.raises(client.ControlError) as e:
client.on_demand_start('q2', 'clock', None, paths=[sock_path])
assert e.value.reason == 'busy' and e.value.sent is True
assert not client.should_fall_back(e.value)
assert not client.display_not_listening(e.value)
finally:
server.close()
+24
View File
@@ -327,3 +327,27 @@ class TestCleartextIsCalledOut:
def test_tls_on_does_not_warn(self, bridge_module):
assert bridge_module.warn_if_cleartext(
{"mqtt_tls": True, "mqtt_password": "hunter2"}) is False
class TestTheApiClientReadsStartingAsTaken:
"""A cold start answers 202 with ``status: "starting"``: the request is
taken and the web process delivers it once the display listens. The
bridge must report that as success, not as a failure."""
def _client(self, bridge_module, status_code, body):
response = MagicMock(status_code=status_code)
response.json.return_value = body
session = MagicMock()
session.request.return_value = response
return bridge_module.LEDMatrixClient("http://pi:5000", session=session)
def test_202_starting_is_returned_not_raised(self, bridge_module):
api = self._client(bridge_module, 202, {
"status": "starting", "message": "starting",
"data": {"request_id": "r1", "pending": True}})
assert api.start_on_demand(mode="clock") == {"request_id": "r1", "pending": True}
def test_an_error_is_still_raised(self, bridge_module):
api = self._client(bridge_module, 503, {"status": "error", "message": "no display"})
with pytest.raises(RuntimeError, match="no display"):
api.start_on_demand(mode="clock")
+2 -7
View File
@@ -346,13 +346,8 @@ class TestResumingAfterTheSession:
class TestStopClearsAnError:
def _post_stop(self, c):
stop = {'request_id': 'S1', 'action': 'stop'}
c._last_on_demand_poll = None
c.cache_manager.get = MagicMock(
side_effect=lambda key, *a, **kw:
stop if key == 'display_on_demand_request' else None)
c.cache_manager.delete = MagicMock()
c._poll_on_demand_requests()
c._handle_on_demand_request({'request_id': 'S1', 'action': 'stop',
'source': 'socket'})
def test_a_stop_after_a_failed_request_clears_the_error(self, controller):
_start(controller, plugin_id='uninstalled')
+272
View File
@@ -0,0 +1,272 @@
"""The web process's on-demand dispatcher (web_interface/on_demand_dispatch.py).
A start that finds no display listening -- the service was just started, or
is still loading its plugins -- is answered at once (202), and the
dispatcher's one worker thread sends it again until the display
acknowledges it or the wait runs out. These tests drive the worker with a
fake ``send`` and short waits:
* acknowledged: delivered once, and reported as such;
* nothing listening for the whole wait: ``start-timeout``;
* any other failure: reported at once, not retried;
* a newer start supersedes the pending one; a stop cancels it;
* the outcome is reported for a while, then forgotten.
"""
import threading
import time
import pytest
from src.ipc import client as control_client
from web_interface import on_demand_dispatch
from web_interface.on_demand_dispatch import OnDemandDispatcher
def _not_listening():
return control_client.ControlError("no_socket", "x", sent=False)
def _payload(rid, plugin_id="weather"):
return {"request_id": rid, "action": "start", "plugin_id": plugin_id,
"mode": plugin_id, "duration": 30, "pinned": False}
class FakeSend:
"""Answers with ``outcomes`` in turn (an exception is raised, anything
else acks); the last one repeats. Records the request ids it was sent."""
def __init__(self, *outcomes):
self.outcomes = list(outcomes) or ["ack"]
self.sent = []
self.lock = threading.Lock()
def __call__(self, payload):
with self.lock:
self.sent.append(payload["request_id"])
outcome = self.outcomes.pop(0) if len(self.outcomes) > 1 else self.outcomes[0]
if isinstance(outcome, BaseException):
raise outcome
if callable(outcome):
return outcome(payload)
return {"accepted": True}
def _until(predicate, timeout=5.0):
end = time.monotonic() + timeout
while time.monotonic() < end:
if predicate():
return True
time.sleep(0.005)
return False
def _settled(d):
return _until(lambda: not d.pending() and d._thread is None)
@pytest.fixture
def make():
made = []
def build(send, wait_seconds=2.0, retry_interval=0.01):
d = OnDemandDispatcher(send, wait_seconds=wait_seconds, retry_interval=retry_interval)
made.append(d)
return d
yield build
for d in made:
d.cancel("teardown")
class TestDelivery:
def test_an_acknowledged_start_is_delivered_once(self, make):
send = FakeSend("ack")
d = make(send)
d.submit(_payload("r1"))
assert _settled(d)
assert send.sent == ["r1"]
status = d.status()
assert status["status"] == "delivered" and status["request_id"] == "r1"
assert status["source"] == "web"
def test_it_is_sent_again_until_the_display_listens(self, make):
send = FakeSend(_not_listening(), _not_listening(), "ack")
d = make(send)
d.submit(_payload("r1"))
assert _settled(d)
assert send.sent == ["r1", "r1", "r1"]
assert d.status()["status"] == "delivered"
def test_while_it_waits_it_reports_starting(self, make):
gate = threading.Event()
def blocked(payload):
gate.wait(5)
raise _not_listening()
send = FakeSend(blocked, "ack")
d = make(send)
d.submit(_payload("r1"))
status = d.status()
assert status["status"] == "starting" and status["active"] is False
assert (status["plugin_id"], status["mode"], status["duration"]) == ("weather", "weather", 30)
gate.set()
assert _settled(d)
assert d.status()["status"] == "delivered"
class TestGivingUp:
def test_nothing_listening_for_the_whole_wait_is_a_start_timeout(self, make):
send = FakeSend(_not_listening())
d = make(send, wait_seconds=0.2)
started = time.monotonic()
d.submit(_payload("r1"))
assert _settled(d)
waited = time.monotonic() - started
status = d.status()
assert status["status"] == "error" and status["error"] == "start-timeout"
assert 0.15 <= waited < 2.0
assert len(send.sent) > 3 # it kept trying in between
def test_a_start_can_carry_its_own_wait(self, make):
# The route passes a shorter wait for a service that was already
# running; the dispatcher's default must not override it.
d = make(FakeSend(_not_listening()), wait_seconds=30.0)
d.submit(_payload("r1"), wait_seconds=0.1)
assert _until(lambda: not d.pending(), timeout=3.0), "it waited the default"
assert d.status()["error"] == "start-timeout"
@pytest.mark.parametrize("reason,sent", [("busy", True), ("unknown_command", True),
("timeout", True), ("forbidden", False)])
def test_any_other_failure_is_reported_at_once(self, make, reason, sent):
send = FakeSend(control_client.ControlError(reason, "x", sent=sent))
d = make(send)
d.submit(_payload("r1"))
assert _settled(d)
assert send.sent == ["r1"]
status = d.status()
assert status["status"] == "error" and status["error"] == reason
def test_a_client_bug_is_internal(self, make):
d = make(FakeSend(RuntimeError("boom")))
d.submit(_payload("r1"))
assert _settled(d)
assert d.status()["error"] == "internal"
class TestOneAtATime:
def test_a_newer_start_supersedes_the_pending_one(self, make):
send = FakeSend(_not_listening())
d = make(send)
d.submit(_payload("old"))
assert _until(lambda: "old" in send.sent)
d.submit(_payload("new", plugin_id="clock"))
assert d.status()["request_id"] == "new"
n = len(send.sent)
send.outcomes = ["ack"]
assert _settled(d)
assert send.sent[n:] and set(send.sent[n + 1:]) <= {"new"}
assert send.sent[-1] == "new"
status = d.status()
assert status["status"] == "delivered" and status["plugin_id"] == "clock"
def test_an_answer_for_a_superseded_start_is_not_reported(self, make):
# The old start's send is in flight when the new one arrives; its
# ack must not mark the new one delivered.
in_flight, release = threading.Event(), threading.Event()
def slow(payload):
in_flight.set()
release.wait(5)
return {"accepted": True}
send = FakeSend(slow, _not_listening())
d = make(send, wait_seconds=0.3)
d.submit(_payload("old"))
assert in_flight.wait(5)
d.submit(_payload("new"))
release.set()
assert _settled(d)
status = d.status()
assert status["request_id"] == "new"
assert status["status"] == "error" and status["error"] == "start-timeout"
def test_a_stop_cancels_the_pending_start(self, make):
send = FakeSend(_not_listening())
d = make(send)
d.submit(_payload("r1"))
assert _until(lambda: send.sent)
assert d.cancel("requested-stop") == "r1"
assert _settled(d)
n = len(send.sent)
time.sleep(0.05)
assert len(send.sent) == n, "it kept sending a cancelled start"
status = d.status()
assert status["status"] == "idle" and status["last_event"] == "requested-stop"
def test_a_cancel_waits_for_a_send_in_flight(self, make):
# Whatever the caller sends after cancel() must land after the
# cancelled start, not race it to the display.
in_flight, release = threading.Event(), threading.Event()
def slow(payload):
in_flight.set()
release.wait(5)
raise _not_listening()
send = FakeSend(slow)
d = make(send)
d.submit(_payload("old"))
assert in_flight.wait(5)
done = threading.Event()
threading.Thread(target=lambda: (d.cancel("superseded"), done.set()),
daemon=True).start()
assert not done.wait(0.1), "cancel returned while the old send was in flight"
release.set()
assert done.wait(5)
assert _settled(d)
assert send.sent == ["old"]
def test_a_cancel_with_nothing_pending_does_nothing(self, make):
d = make(FakeSend("ack"))
assert d.cancel() is None
assert d.status() is None
def test_a_cancel_after_delivery_leaves_the_outcome(self, make):
d = make(FakeSend("ack"))
d.submit(_payload("r1"))
assert _settled(d)
assert d.cancel() is None
assert d.status()["status"] == "delivered"
class TestOutcomeLifetime:
def test_an_outcome_is_forgotten_after_a_while(self, make, monkeypatch):
d = make(FakeSend("ack"))
d.submit(_payload("r1"))
assert _settled(d)
assert d.status() is not None
monkeypatch.setattr(on_demand_dispatch, "OUTCOME_SECONDS", 0.0)
time.sleep(0.01)
assert d.status() is None
def test_a_pending_start_never_expires(self, make, monkeypatch):
monkeypatch.setattr(on_demand_dispatch, "OUTCOME_SECONDS", 0.0)
gate = threading.Event()
d = make(FakeSend(lambda p: gate.wait(5) and {"accepted": True}))
d.submit(_payload("r1"))
time.sleep(0.01)
assert d.status()["status"] == "starting"
gate.set()
assert _settled(d)
def test_the_process_has_one_dispatcher():
on_demand_dispatch.reset_for_tests()
try:
assert on_demand_dispatch.current() is None
first = on_demand_dispatch.get_dispatcher(FakeSend())
assert on_demand_dispatch.get_dispatcher(FakeSend()) is first
assert on_demand_dispatch.current() is first
finally:
on_demand_dispatch.reset_for_tests()
-106
View File
@@ -1,106 +0,0 @@
"""The on-demand request mailbox: how often it is read, and how it is consumed.
The mailbox is a cache key the web process writes and the display process
reads. Two properties matter and neither is obvious from the call site:
* it is polled after every rendered frame, so an uncached read here is a
disk read at frame rate;
* consuming it must not throw away a request that arrived while the previous
one was being processed.
"""
from unittest.mock import MagicMock
import pytest
class TestPollingIsBounded:
"""_poll_on_demand_requests runs ~125x/second on a scrolling mode.
The read is deliberately uncached (memory_ttl=0) because a cached one
pinned the first request for an hour. That makes the call a real disk read,
so it needs a floor -- without one it was ~125 reads per second to find
nothing at all.
"""
def test_first_call_always_reads(self, test_display_controller):
c = test_display_controller
c.cache_manager.get = MagicMock(return_value=None)
c._poll_on_demand_requests()
assert c.cache_manager.get.call_count == 1
def test_immediate_second_call_does_not_read(self, test_display_controller):
c = test_display_controller
c.cache_manager.get = MagicMock(return_value=None)
c._poll_on_demand_requests()
for _ in range(50):
c._poll_on_demand_requests()
assert c.cache_manager.get.call_count == 1, "polling was not bounded"
def test_reads_again_once_the_interval_has_passed(self, test_display_controller, monkeypatch):
c = test_display_controller
c.cache_manager.get = MagicMock(return_value=None)
clock = {"t": 1000.0}
monkeypatch.setattr("src.display_controller.time.monotonic", lambda: clock["t"])
c._poll_on_demand_requests()
clock["t"] += c.ON_DEMAND_POLL_INTERVAL / 2
c._poll_on_demand_requests()
assert c.cache_manager.get.call_count == 1, "read before the interval elapsed"
clock["t"] += c.ON_DEMAND_POLL_INTERVAL
c._poll_on_demand_requests()
assert c.cache_manager.get.call_count == 2
def test_the_interval_is_short_enough_to_feel_instant(self, test_display_controller):
# A person clicking in the web UI must not notice the floor.
assert test_display_controller.ON_DEMAND_POLL_INTERVAL <= 0.5
class TestMailboxIsConsumedByIdentity:
"""Deleting whatever is in the mailbox loses a request that raced in."""
def _arrange(self, controller, first, later):
"""Mailbox returns `first`, then `later` on the pre-delete re-read."""
controller.on_demand_active = False
controller.on_demand_request_id = None
controller._last_on_demand_poll = None
reads = iter([first, later])
def fake_get(key, *a, **kw):
if key == 'display_on_demand_request':
return next(reads, later)
return None # processed-id lookup
controller.cache_manager.get = MagicMock(side_effect=fake_get)
controller.cache_manager.set = MagicMock()
controller.cache_manager.delete = MagicMock()
controller._activate_on_demand = MagicMock()
REQ_A = {'request_id': 'A', 'action': 'start', 'plugin_id': 'p', 'mode': 'm'}
REQ_B = {'request_id': 'B', 'action': 'start', 'plugin_id': 'p', 'mode': 'm'}
def test_own_request_is_deleted(self, test_display_controller):
c = test_display_controller
self._arrange(c, self.REQ_A, self.REQ_A)
c._poll_on_demand_requests()
c.cache_manager.delete.assert_called_once_with('display_on_demand_request')
def test_a_newer_request_is_left_for_the_next_poll(self, test_display_controller):
c = test_display_controller
self._arrange(c, self.REQ_A, self.REQ_B)
c._poll_on_demand_requests()
assert c.cache_manager.delete.call_count == 0, \
"request B was deleted without ever being processed"
def test_an_already_empty_mailbox_is_still_cleared(self, test_display_controller):
c = test_display_controller
self._arrange(c, self.REQ_A, None)
c._poll_on_demand_requests()
c.cache_manager.delete.assert_called_once_with('display_on_demand_request')
def test_the_request_is_still_processed(self, test_display_controller):
c = test_display_controller
self._arrange(c, self.REQ_A, self.REQ_B)
c._poll_on_demand_requests()
c._activate_on_demand.assert_called_once()
+13 -38
View File
@@ -8,8 +8,9 @@ Three separate gaps, all reachable from the web UI's force-display dialog:
* restarting while on-demand was active loaded *only* the on-demand plugin,
so normal rotation had nothing to return to for the life of the process;
* a stop request was exempt from the duplicate guards on purpose and was
never removed from the mailbox, so it was re-processed on every poll
forever.
never removed from the file mailbox, so it was re-processed on every poll
forever. The mailbox is gone (stage 5); a stop still skips the guards,
so a second click stops a session a race left running.
"""
from unittest.mock import MagicMock
@@ -185,51 +186,25 @@ class TestRestartDoesNotStarveTheOtherPlugins:
assert controller.on_demand_active is False
class TestStopRequestsAreConsumed:
"""A stop request is exempt from the duplicate guards, so the mailbox
delete is the only thing that ends it."""
STOP = {'request_id': 'S1', 'action': 'stop'}
class TestStopRequestsSkipTheDuplicateGuard:
"""A stop is exempt from the request-id guard: every one is acted on."""
def _arrange(self, controller, active):
controller.on_demand_active = active
controller.on_demand_status = 'active' if active else 'idle'
controller._last_on_demand_poll = None
controller.cache_manager.get = MagicMock(
side_effect=lambda key, *a, **kw:
self.STOP if key == 'display_on_demand_request' else None)
controller.cache_manager.set = MagicMock()
controller.cache_manager.delete = MagicMock()
controller._clear_on_demand = MagicMock()
def test_a_handled_stop_is_removed_from_the_mailbox(self, test_display_controller):
def test_the_stop_is_acted_on(self, test_display_controller):
c = test_display_controller
self._arrange(c, active=True)
c._poll_on_demand_requests()
c.cache_manager.delete.assert_called_once_with('display_on_demand_request')
def test_a_stop_arriving_while_idle_is_also_removed(self, test_display_controller):
"""Otherwise a stop sent to an idle display re-fires forever."""
c = test_display_controller
self._arrange(c, active=False)
c._poll_on_demand_requests()
c.cache_manager.delete.assert_called_once_with('display_on_demand_request')
def test_the_stop_is_still_acted_on(self, test_display_controller):
c = test_display_controller
self._arrange(c, active=True)
c._poll_on_demand_requests()
c._handle_on_demand_request({'request_id': 'S1', 'action': 'stop',
'source': 'socket'})
c._clear_on_demand.assert_called_once_with(reason='requested-stop')
def test_a_start_racing_in_behind_a_stop_is_not_discarded(self, test_display_controller):
"""The compare-before-delete applies to stops too."""
def test_the_same_stop_twice_is_acted_on_twice(self, test_display_controller):
c = test_display_controller
self._arrange(c, active=True)
newer = {'request_id': 'S2', 'action': 'start', 'plugin_id': 'p', 'mode': 'm'}
reads = iter([self.STOP, newer])
c.cache_manager.get = MagicMock(
side_effect=lambda key, *a, **kw:
next(reads, newer) if key == 'display_on_demand_request' else None)
c._poll_on_demand_requests()
assert c.cache_manager.delete.call_count == 0
for _ in range(2):
c._handle_on_demand_request({'request_id': 'S1', 'action': 'stop',
'source': 'socket'})
assert c._clear_on_demand.call_count == 2
+38 -50
View File
@@ -2,19 +2,17 @@
and end_on_demand().
A plugin running in the display process used to write the
``display_on_demand_request`` mailbox, which the display reads once a second
while the control socket is up. These tests pin the way in that replaces it:
``display_on_demand_request`` mailbox, which the display no longer reads
(stage 5). These tests pin the way in that replaced it:
* BasePlugin -> PluginManager -> DisplayController.submit_plugin_on_demand,
which only queues, from any thread;
* the render thread applies the queue where it applies socket commands,
through the mailbox's own handler, without the mailbox's read floor, and
woken by the control socket when it is up;
through _handle_on_demand_request, at once, and woken by the control
socket when it is up;
* a plugin's stop ends only its own session;
* no display to ask (the web interface's plugin manager, an old core's
plugin manager) answers None, which is a plugin's cue to fall back to the
mailbox;
* the mailbox still works for plugins that write it.
plugin manager) answers None.
"""
import logging
@@ -73,19 +71,10 @@ def controller(test_display_controller):
c_ = test_display_controller
c_.on_demand_active = False
c_.on_demand_request_id = None
c_._last_on_demand_poll = None
mailbox = {'value': None}
def fake_get(key, *a, **kw):
if key == 'display_on_demand_request':
return mailbox['value']
return None
c_.cache_manager.get = MagicMock(side_effect=fake_get)
c_.cache_manager.get = MagicMock(return_value=None)
c_.cache_manager.set = MagicMock()
c_.cache_manager.delete = MagicMock()
c_._activate_on_demand = MagicMock()
c_.mailbox = mailbox
return c_
@@ -101,7 +90,7 @@ class TestWiring:
controller.plugin_manager.set_on_demand_handler.assert_called_once_with(
controller.submit_plugin_on_demand)
def test_a_start_reaches_the_mailbox_handler(self, wired):
def test_a_start_reaches_the_on_demand_handler(self, wired):
controller, manager = wired
rid = _plugin('pomodoro-timer', manager).request_on_demand(
mode='pomodoro', duration=30, pinned=True)
@@ -139,15 +128,6 @@ class TestWiring:
controller._poll_on_demand_requests()
assert seen == ['a', 'b', 'c']
def test_the_mailbox_still_works_for_older_plugins(self, wired):
controller, manager = wired
controller.mailbox['value'] = {'request_id': 'mb', 'action': 'start',
'plugin_id': 'birdnet-go'}
_plugin('on-air', manager).request_on_demand()
controller._poll_on_demand_requests()
ids = [call.args[0]['request_id'] for call in controller._activate_on_demand.call_args_list]
assert 'mb' in ids and len(ids) == 2
def test_a_failing_request_is_contained(self, wired):
controller, manager = wired
calls = []
@@ -174,10 +154,10 @@ class TestPromptness:
controller._service_pending_changes() # well inside the 0.25 s floor
controller._activate_on_demand.assert_called_once()
def test_a_plugin_request_skips_the_mailbox_floor(self, wired):
def test_a_plugin_request_lands_on_the_next_poll(self, wired):
controller, manager = wired
controller._control_server = _WakeServer()
controller._poll_on_demand_requests() # sets the 1 s mailbox floor
controller._poll_on_demand_requests()
_plugin('p', manager).request_on_demand()
controller._poll_on_demand_requests()
controller._activate_on_demand.assert_called_once()
@@ -286,14 +266,13 @@ class TestStop:
controller._poll_on_demand_requests()
controller._clear_on_demand.assert_not_called()
def test_a_mailbox_stop_still_ends_any_session(self, wired):
def test_a_socket_stop_still_ends_any_session(self, wired):
controller, _ = wired
controller.on_demand_active = True
controller.on_demand_plugin_id = 'clock'
controller._clear_on_demand = MagicMock()
controller.mailbox['value'] = {'request_id': 's', 'action': 'stop',
'plugin_id': 'on-air'}
controller._poll_on_demand_requests()
controller._handle_on_demand_request({'request_id': 's', 'action': 'stop',
'source': 'socket'})
controller._clear_on_demand.assert_called_once_with(reason='requested-stop')
def test_start_then_stop_from_one_thread_ends_the_session(self, wired):
@@ -314,7 +293,7 @@ class TestStop:
class TestNoDisplay:
"""None is a plugin's cue to write the mailbox instead."""
"""None: no display in this process took the request."""
def test_a_manager_with_no_handler_answers_none(self):
plugin = _plugin('p', _manager())
@@ -353,21 +332,30 @@ class TestNoDisplay:
assert bare._plugin_on_demand_pending() is False
bare._drain_plugin_on_demand() # nothing to do, no error
def test_the_feature_detection_pattern(self):
"""The hasattr pattern from docs/PLUGIN_API_REFERENCE.md."""
writes = []
class OldCorePlugin: # an older core's BasePlugin has no such method
pass
for plugin, expect_mailbox in ((OldCorePlugin(), True),
(_plugin('p', _manager()), True),
(_plugin('p', _manager(lambda r: True)), False)):
writes.clear()
if not (hasattr(plugin, 'request_on_demand')
and plugin.request_on_demand(mode='m')):
writes.append('mailbox')
assert (writes == ['mailbox']) is expect_mailbox
def test_a_mailbox_write_after_none_is_dropped_with_a_warning(self, tmp_path,
monkeypatch, caplog):
"""What a plugin written for older cores does on None now: its
fallback write to the retired key stores nothing, and the log names
it once."""
from src import cache_manager as cache_module
from src.cache_manager import CacheManager
monkeypatch.setattr(CacheManager, '_get_writable_cache_dir',
lambda self: str(tmp_path))
monkeypatch.setattr(cache_module, '_retired_writers_warned', set())
cache = CacheManager()
try:
plugin = _plugin('birdnet-go', _manager())
plugin.cache_manager = cache
caplog.set_level(logging.WARNING)
for _ in range(2):
if plugin.request_on_demand(mode='m') is None:
plugin.cache_manager.set('display_on_demand_request', {
'request_id': 'r', 'action': 'start', 'plugin_id': 'birdnet-go'})
assert cache.get('display_on_demand_request', max_age=None, memory_ttl=0) is None
lines = [r.getMessage() for r in caplog.records if 'retired' in r.getMessage()]
assert len(lines) == 1 and "plugin 'birdnet-go'" in lines[0]
finally:
cache.stop_cleanup_thread()
class TestArguments:
@@ -401,7 +389,7 @@ class TestArguments:
class TestMockManagers:
def test_a_magicmock_manager_reads_as_not_taken(self):
"""A plugin's test with a MagicMock manager keeps its mailbox path."""
"""A plugin's test with a MagicMock manager reads as "not taken"."""
plugin = _plugin('p', MagicMock())
assert plugin.request_on_demand(mode='m') is None
assert plugin.end_on_demand() is None
+205
View File
@@ -0,0 +1,205 @@
"""Per-plugin schedules (config['plugin_schedules'], src/plugin_schedule.py).
The window rule itself is the display schedule's, covered case by case in
test_display_controller_schedule.py through _check_schedule; here: the
plugin-level answer, and the three places it is applied -- the rotation,
the live scan, and the Vegas ticker -- plus the save endpoint.
"""
import copy
import os
from datetime import datetime
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
os.environ.setdefault("EMULATOR", "true")
from src.display_arbiter import ScreenPlan, Source # noqa: E402
from src.display_controller import DisplayController # noqa: E402
from src.plugin_schedule import hidden_plugins, schedule_state, wants_live_scan # noqa: E402
from src.vegas_mode.config import VegasModeConfig # noqa: E402
from src.vegas_mode.stream_manager import StreamManager # noqa: E402
from test._api_v3_test_helpers import api_v3_client, api_v3_module # noqa: E402,F401
DAYS = ('monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday')
SUNDAY_1PM = datetime(2026, 10, 18, 13, 0) # a Sunday
MONDAY_1PM = datetime(2026, 10, 19, 13, 0)
def sundays_off(**extra):
"""The gameday schedule: every day all day, except Sunday."""
days = {d: {'enabled': d != 'sunday', 'start_time': '00:00', 'end_time': '23:59'}
for d in DAYS}
return {'enabled': True, 'mode': 'per-day', 'days': days, **extra}
ALWAYS_OFF = {'enabled': True, 'mode': 'per-day',
'days': {d: {'enabled': False} for d in DAYS}}
class TestHiddenPlugins:
def test_outside_window_is_hidden(self):
schedules = {'news': sundays_off(), 'stocks': sundays_off()}
assert hidden_plugins(schedules, SUNDAY_1PM, False) == {'news', 'stocks'}
assert hidden_plugins(schedules, MONDAY_1PM, False) == frozenset()
def test_hide_when_live_only_while_live(self):
schedules = {'news': {'enabled': False, 'hide_when_live': True}}
assert hidden_plugins(schedules, MONDAY_1PM, True) == {'news'}
assert hidden_plugins(schedules, MONDAY_1PM, False) == frozenset()
def test_disabled_schedule_and_junk_entries_hide_nothing(self):
schedules = {'news': {**ALWAYS_OFF, 'enabled': False}, 'bad': 'x', 'none': None}
assert hidden_plugins(schedules, SUNDAY_1PM, True) == frozenset()
def test_bad_time_shows_rather_than_hides(self):
assert schedule_state({'mode': 'global', 'start_time': '25:00'}, MONDAY_1PM)[0]
@pytest.mark.parametrize('schedule', [
{'mode': 1},
{'mode': 'per-day', 'days': 5},
{'mode': 'per-day', 'days': {'sunday': False}},
{'mode': 'per-day', 'days': ['sunday']},
])
def test_wrong_shapes_show_rather_than_raise(self, schedule):
# Falls back to the global 07:00-23:00 window, which 13:00 is inside.
assert schedule_state(schedule, SUNDAY_1PM)[0]
def test_global_mode_ignores_days(self):
schedule = {**sundays_off(), 'mode': 'global', 'start_time': '12:00', 'end_time': '14:00'}
assert schedule_state(schedule, SUNDAY_1PM)[0]
def test_wants_live_scan(self):
assert not wants_live_scan({'news': sundays_off()})
assert wants_live_scan({'news': sundays_off(), 'stocks': {'hide_when_live': True}})
def plugin(plugin_id, live=False, priority=False):
return SimpleNamespace(plugin_id=plugin_id, display=lambda **_: True,
has_live_content=lambda: live,
has_live_priority=lambda: priority,
get_live_modes=lambda: [f'{plugin_id}_live'])
def controller(schedules, plugin_modes):
dc = object.__new__(DisplayController)
dc.config = {'plugin_schedules': schedules}
dc._tz = None
dc.plugin_modes = plugin_modes
dc._plugin_hidden_cache = (None, frozenset())
dc._game_live = False
dc._game_live_at = float('-inf')
dc._health_tracker = lambda: None
return dc
class TestController:
def test_rotation_skips_hidden_but_on_demand_shows_it(self):
news = plugin('news')
dc = controller({'news': ALWAYS_OFF}, {'news': news})
assert dc._plugin_to_draw(ScreenPlan(Source.ROTATION, mode='news')) is None
assert dc._plugin_to_draw(ScreenPlan(Source.ON_DEMAND, mode='news')) is news
def test_no_schedules_changes_nothing(self):
news = plugin('news')
dc = controller({}, {'news': news})
assert dc._plugin_to_draw(ScreenPlan(Source.ROTATION, mode='news')) is news
def test_any_live_game_hides_news_even_without_live_priority(self):
modes = {'news': plugin('news'), 'nfl_live': plugin('nfl', live=True)}
dc = controller({'news': {'enabled': False, 'hide_when_live': True}}, modes)
assert dc._hidden_plugins() == {'news'}
def test_scoreboard_scheduled_off_does_not_win_live(self):
modes = {'nfl_live': plugin('nfl', live=True, priority=True)}
dc = controller({'nfl': ALWAYS_OFF}, modes)
assert dc._collect_live_modes() == []
dc.config = {}
assert dc._collect_live_modes() == ['nfl_live']
def test_a_scheduled_off_live_mode_does_not_hold_the_rotation(self):
modes = {'nfl_live': plugin('nfl', live=True, priority=True)}
dc = controller({'nfl': ALWAYS_OFF}, modes)
assert not dc._still_live('nfl_live')
dc.config = {}
assert dc._still_live('nfl_live')
@pytest.mark.parametrize('schedules', [['news'], 'news', 5])
def test_wrong_shaped_plugin_schedules_hide_nothing(self, schedules):
dc = controller(schedules, {'news': plugin('news')})
assert dc._hidden_plugins() == frozenset()
def test_a_reload_is_never_answered_from_the_old_schedules(self):
dc = controller({'news': ALWAYS_OFF}, {'news': plugin('news')})
assert dc._hidden_plugins() == {'news'}
# A new config in the same minute, with no cache reset.
dc.config = {'plugin_schedules': {'news': {'enabled': False}}}
assert dc._hidden_plugins() == frozenset()
class TestVegas:
def test_hidden_plugins_sit_out_the_ticker(self):
plugins = {pid: SimpleNamespace(enabled=True, config={}) for pid in ('news', 'nfl')}
stream = StreamManager(VegasModeConfig(), SimpleNamespace(plugins=plugins), MagicMock())
stream._refresh_plugin_list()
assert set(stream._ordered_plugins) == {'news', 'nfl'}
stream.hidden_plugins = lambda: frozenset({'news'})
stream._refresh_plugin_list()
assert set(stream._ordered_plugins) == {'nfl'}
def test_a_changed_hidden_set_skips_the_refresh_interval(self):
plugins = {pid: SimpleNamespace(enabled=True, config={}) for pid in ('news', 'nfl')}
stream = StreamManager(VegasModeConfig(), SimpleNamespace(plugins=plugins), MagicMock())
stream.refresh() # the interval's refresh
assert set(stream._ordered_plugins) == {'news', 'nfl'}
stream.refresh() # inside the interval, nothing changed: no rebuild
stream.hidden_plugins = lambda: frozenset({'news'})
stream.refresh() # still inside the interval, but the set changed
assert set(stream._ordered_plugins) == {'nfl'}
class TestSaveEndpoint:
@pytest.fixture
def store(self, api_v3_module, monkeypatch):
state = {'config': {}}
api_v3_module.api_v3.config_manager.load_config.side_effect = \
lambda *a, **k: copy.deepcopy(state['config'])
api_v3_module.api_v3.plugin_catalog.plugin_manifests = {'news': {}}
def fake_save(_manager, config, **_kwargs):
state['config'] = copy.deepcopy(config)
return True, ''
monkeypatch.setattr(api_v3_module, '_save_config_atomic', fake_save)
return state
def test_round_trip(self, api_v3_client, store):
body = {'plugin_id': 'news', 'hide_when_live': 'on', **sundays_off()}
resp = api_v3_client.post('/api/v3/config/plugin-schedules', json=body)
assert resp.status_code == 200, resp.get_json()
saved = store['config']['plugin_schedules']['news']
assert saved['hide_when_live'] is True
assert saved['days']['sunday']['enabled'] is False
got = api_v3_client.get('/api/v3/config/plugin-schedules').get_json()['data']
assert got == {'news': saved}
def test_schedule_that_does_nothing_is_removed(self, api_v3_client, store):
store['config'] = {'plugin_schedules': {'news': sundays_off()}}
resp = api_v3_client.post('/api/v3/config/plugin-schedules',
json={'plugin_id': 'news', 'enabled': False})
assert resp.status_code == 200, resp.get_json()
assert store['config']['plugin_schedules'] == {}
@pytest.mark.parametrize('body,status', [
({'plugin_id': 'display', 'enabled': True}, 400), # a core section
({'plugin_id': 'nope', 'enabled': True}, 404), # not installed
({'plugin_id': 'news', 'enabled': True, 'mode': 'global',
'start_time': '25:00', 'end_time': '23:00'}, 400), # bad time
({'enabled': True}, 400), # no id
])
def test_refusals(self, api_v3_client, store, body, status):
resp = api_v3_client.post('/api/v3/config/plugin-schedules', json=body)
assert resp.status_code == status, resp.get_json()
assert 'plugin_schedules' not in store['config']
+9 -18
View File
@@ -12,8 +12,8 @@ frame, so a command is applied:
* at the next frame in Vegas (8 ms here, at 125 fps).
Each test runs the real run() loop on the fake clock of
test/_run_loop_harness.py and compares the socket with the mailbox for the
same request. The latencies are the fake clock's, so they are exact.
test/_run_loop_harness.py. The latencies are the fake clock's, so they are
exact. (The file mailbox these were once compared with is gone: stage 5.)
"""
import os
@@ -32,13 +32,10 @@ def _event_time(trace, kind):
return next(e[0] for e in trace["events"] if e[1] == kind)
def _on_demand_latency(tmp_path, build, via, horizon=40):
def _on_demand_latency(tmp_path, build, horizon=40):
h = RunLoopHarness(tmp_path, horizon=horizon)
build(h)
if via == "socket":
h.control_socket().post(POSTED, Command.ON_DEMAND_START, {"plugin_id": "weather"})
else:
h.on_demand_request(POSTED, "mb1", plugin_id="weather")
trace = h.run()
return round(_event_time(trace, "on-demand-start") - POSTED, 3), trace
@@ -60,18 +57,12 @@ def _vegas(h):
h.enable_vegas(cycle=30)
@pytest.mark.parametrize("build, mailbox_latency", [
(_static, 0.7), # the next 1 s frame, at t=11
(_dwell, 0.2), # the next 0.25 s tick, at t=10.5
(_vegas, 0.02), # the next 10-frame check: 80 ms at 125 fps, ~0.4 s on a Pi 4
], ids=["static-screen", "dwell", "vegas"])
def test_a_socket_command_lands_at_once(tmp_path, build, mailbox_latency):
(tmp_path / "s").mkdir()
(tmp_path / "m").mkdir()
socket_latency, trace = _on_demand_latency(tmp_path / "s", build, "socket")
via_mailbox, _ = _on_demand_latency(tmp_path / "m", build, "mailbox")
assert via_mailbox == pytest.approx(mailbox_latency, abs=0.002)
@pytest.mark.parametrize("build", [_static, _dwell, _vegas],
ids=["static-screen", "dwell", "vegas"])
def test_a_socket_command_lands_at_once(tmp_path, build):
# Before stage 2 these waited for the next 1 s frame, the next 0.25 s
# dwell tick, or the next 10-frame Vegas check.
socket_latency, trace = _on_demand_latency(tmp_path, build)
if build is _vegas:
# One 8 ms frame: Vegas checks the queue every frame now.
assert socket_latency <= 0.008
+18 -6
View File
@@ -604,6 +604,15 @@ class TestFreshnessOverTicks:
# --- end to end over a real socket -------------------------------------------------------
def _until(predicate, timeout=5.0):
end = time.monotonic() + timeout
while time.monotonic() < end:
if predicate():
return True
time.sleep(0.005)
return False
@needs_unix_sockets
class TestEndToEnd:
def test_routes_read_the_socket_then_fall_back_when_it_goes(self, web, tmp_path,
@@ -624,14 +633,17 @@ class TestEndToEnd:
try:
data = _data(client, '/api/v3/display/current-status')
assert (data['mode'], data['source']) == ('weather', 'socket')
# The subscription picks up a change the display publishes.
# The subscription picks up a change the display publishes. That
# first answer can be a one-shot state.get with the subscription's
# own snapshot still on its way: a change published in between
# would read new (one-shot), then old (that snapshot), then new
# again (its event). So wait for the subscription first.
feed = display_state._subscription()
assert _until(lambda: feed.latest() is not None)
hub.publish('display', dict(hub.snapshot()['state']['display'], mode='stocks'),
volatile=('last_updated',))
deadline = time.monotonic() + 5
while time.monotonic() < deadline:
if _data(client, '/api/v3/display/current-status')['mode'] == 'stocks':
break
time.sleep(0.05)
assert _until(lambda: (display_state.current_status(feed.latest()) or {})
.get('mode') == 'stocks')
assert _data(client, '/api/v3/display/current-status')['mode'] == 'stocks'
assert hub.subscribers == 1 and hub.readers_active()
finally:
+14 -8
View File
@@ -9,6 +9,7 @@ restart is owed, and the UI raises the existing restart-pending banner.
import subprocess
import sys
from pathlib import Path
import contextlib
from unittest.mock import patch
import pytest
@@ -20,6 +21,13 @@ from web_interface.blueprints import api_v3 as mod # noqa: E402
from web_interface.blueprints.api_v3 import api_v3 # noqa: E402
@contextlib.contextmanager
def _patched(fake):
"""Fake git for both the plain subprocess calls and the timeout-safe runner."""
with patch.object(mod.subprocess, 'run', fake), patch.object(mod.git_index_lock, 'run_git', fake):
yield
@pytest.fixture
def client():
app = Flask(__name__)
@@ -62,7 +70,7 @@ def _pull(client):
class TestRestartIsRequestedWhenCodeChanged:
def test_a_pull_that_moved_head_asks_for_a_restart(self, client):
with patch.object(mod.subprocess, 'run', _git(['aaa111', 'bbb222'])):
with _patched(_git(['aaa111', 'bbb222'])):
data = _pull(client)
assert data['status'] == 'success'
assert data['restart_required'] is True, (
@@ -70,15 +78,14 @@ class TestRestartIsRequestedWhenCodeChanged:
"nothing told the user to restart")
def test_already_up_to_date_does_not(self, client):
with patch.object(mod.subprocess, 'run',
_git(['aaa111', 'aaa111'], pull_out='Already up to date.\n')):
with _patched(_git(['aaa111', 'aaa111'], pull_out='Already up to date.\n')):
data = _pull(client)
assert data['status'] == 'success'
assert data['restart_required'] is False, (
"prompting after a no-op update trains users to ignore the prompt")
def test_a_failed_pull_does_not(self, client):
with patch.object(mod.subprocess, 'run', _git(['aaa111'], pull_rc=1)):
with _patched(_git(['aaa111'], pull_rc=1)):
data = _pull(client)
assert data['status'] == 'error'
assert data['restart_required'] is False
@@ -100,19 +107,18 @@ class TestUnitsAreRefreshedWithTheCode:
return calls
def test_an_update_that_moved_head_refreshes_the_units(self, client, refresh):
with patch.object(mod.subprocess, 'run', _git(['aaa111', 'bbb222'])):
with _patched(_git(['aaa111', 'bbb222'])):
data = _pull(client)
assert refresh == [True]
assert data['unit_refresh']['status'] == 'refreshed'
assert 'Service settings updated' in data['message']
def test_nothing_new_touches_no_units(self, client, refresh):
with patch.object(mod.subprocess, 'run',
_git(['aaa111', 'aaa111'], pull_out='Already up to date.\n')):
with _patched(_git(['aaa111', 'aaa111'], pull_out='Already up to date.\n')):
data = _pull(client)
assert refresh == [] and data['unit_refresh'] is None
def test_a_failed_pull_touches_no_units(self, client, refresh):
with patch.object(mod.subprocess, 'run', _git(['aaa111'], pull_rc=1)):
with _patched(_git(['aaa111'], pull_rc=1)):
data = _pull(client)
assert refresh == [] and data['unit_refresh'] is None
+107 -1
View File
@@ -14,6 +14,7 @@ from collections import deque
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
from PIL import Image
from src.plugin_system.base_plugin import VegasDisplayMode
@@ -233,7 +234,7 @@ class TestCoordinatorStaticPause:
plugin.display.side_effect = lambda **k: seen.append(lock.locked())
coord = self._coord(plugin, lock)
assert coord._handle_static_pause(plugin) is True
assert seen == [True]
assert seen and all(seen)
assert not lock.locked()
def test_pause_is_skipped_while_update_holds_the_lock(self):
@@ -247,3 +248,108 @@ class TestCoordinatorStaticPause:
finally:
lock.release()
plugin.display.assert_not_called()
class FakeClock:
"""time.monotonic/time.sleep for the pause loop: sleeping moves the clock."""
def __init__(self):
self.now = 0.0
def monotonic(self):
return self.now
def sleep(self, seconds):
assert seconds >= 0
self.now += seconds
if self.now > 3600:
raise RuntimeError("the static pause never ended")
class TestStaticPauseRedraws:
"""A pause redraws its plugin for as long as it lasts.
It used to call display() once and sleep out the duration, so a plugin
that rotates its screens or animates (Fantasy Blitz's playlist, say) was
frozen on its first frame for the whole pause.
"""
def _run(self, monkeypatch, duration=10.0, high_fps=False, interrupt_at=None,
lock=None):
from src.vegas_mode import coordinator
clock = FakeClock()
monkeypatch.setattr(coordinator, 'time', clock)
plugin = MagicMock()
plugin.plugin_id = 'clock'
plugin.needs_high_fps = high_fps
plugin.get_display_duration.return_value = duration
calls = []
plugin.display.side_effect = lambda **k: calls.append((clock.now, k))
coord = TestCoordinatorStaticPause()._coord(plugin, lock)
if interrupt_at is not None:
coord._interrupt_check = lambda: clock.now >= interrupt_at
return coord._handle_static_pause(plugin), calls, clock
def test_redraws_once_a_second_for_the_whole_pause(self, monkeypatch):
done, calls, clock = self._run(monkeypatch, duration=10.0)
assert done is True
assert len(calls) == 10
assert [round(t, 6) for t, _ in calls] == [float(i) for i in range(10)]
assert 10.0 <= clock.now < 10.2
def test_only_the_first_draw_clears(self, monkeypatch):
_, calls, _ = self._run(monkeypatch, duration=3.0)
assert calls[0][1] == {'force_clear': True}
assert all(kwargs == {} for _, kwargs in calls[1:])
def test_a_high_fps_plugin_redraws_at_the_high_fps_rate(self, monkeypatch):
from src.screen_runner import HIGH_FPS_INTERVAL
_, calls, _ = self._run(monkeypatch, duration=1.0, high_fps=True)
assert len(calls) == pytest.approx(1.0 / HIGH_FPS_INTERVAL, abs=2)
def test_the_high_fps_rule_matches_the_rotation(self):
from src.vegas_mode.coordinator import VegasModeCoordinator
rule = VegasModeCoordinator._static_needs_high_fps
legacy_image = SimpleNamespace(plugin_id='static-image')
assert rule(legacy_image) is True
assert rule(SimpleNamespace(plugin_id='static-image', needs_high_fps=False)) is False
assert rule(SimpleNamespace(plugin_id='clock', enable_scrolling=True)) is True
assert rule(SimpleNamespace(plugin_id='clock')) is False
def test_an_interrupt_still_ends_the_pause(self, monkeypatch):
done, calls, clock = self._run(monkeypatch, duration=30.0, interrupt_at=2.5)
assert done is False
assert clock.now < 2.7
assert len(calls) == 3
def test_a_held_lock_skips_one_redraw_not_the_pause(self, monkeypatch):
lock = threading.Lock()
real_acquire = lock.acquire
held = {'count': 0}
class FlakyLock:
def acquire(self, timeout=-1):
held['count'] += 1
# update() holds the lock at the third draw only.
return False if held['count'] == 3 else real_acquire(timeout=timeout)
def release(self):
lock.release()
done, calls, _ = self._run(monkeypatch, duration=5.0, lock=FlakyLock())
assert done is True
assert [round(t) for t, _ in calls] == [0, 1, 3, 4]
def test_a_raise_from_a_redraw_ends_the_pause(self, monkeypatch):
from src.vegas_mode import coordinator
clock = FakeClock()
monkeypatch.setattr(coordinator, 'time', clock)
plugin = MagicMock()
plugin.plugin_id = 'clock'
plugin.needs_high_fps = False
plugin.get_display_duration.return_value = 30.0
plugin.display.side_effect = [None, RuntimeError('boom')]
coord = TestCoordinatorStaticPause()._coord(plugin)
assert coord._handle_static_pause(plugin) is False
assert clock.now < 1.2
assert coord._static_pause_active is False
+5 -2
View File
@@ -19,7 +19,7 @@ import logging
import os
import threading
from types import SimpleNamespace
from unittest.mock import MagicMock
from unittest.mock import MagicMock, call
os.environ.setdefault("EMULATOR", "true")
@@ -62,6 +62,7 @@ def _plugin(duration, plugin_id='clock-simple'):
plugin = MagicMock()
plugin.plugin_id = plugin_id
plugin.get_display_duration.return_value = duration
plugin.needs_high_fps = False
return plugin
@@ -104,7 +105,9 @@ class TestPauseLength:
completed, lasted = _pause(_coord(plugin), plugin, clock)
assert completed is True
assert lasted == pytest.approx(30.0, abs=0.15)
plugin.display.assert_called_once_with(force_clear=True)
# Redrawn once a second; only the first draw clears.
assert plugin.display.call_args_list[0] == call(force_clear=True)
assert plugin.display.call_count == 30
@pytest.mark.parametrize('value', NOT_ABOVE_ZERO, ids=repr)
def test_a_number_not_above_zero_pauses_for_15s(self, clock, value):
+19
View File
@@ -59,6 +59,25 @@ def test_a_refresh_logs_a_single_info_summary(caplog):
assert any('off' in m and 'not enabled' in m for m in debug)
def test_an_empty_list_is_logged_once_until_it_fills(caplog):
# Every plugin scheduled off: the display retries every second.
stream = _stream({'clock': _Plugin()})
stream.hidden_plugins = lambda: frozenset({'clock'})
with caplog.at_level(logging.DEBUG, logger=LOGGER):
for _ in range(3):
assert not stream.initialize()
loud = [r for r in caplog.records if r.name == LOGGER and r.levelno >= logging.INFO]
assert len(loud) == 2, [r.getMessage() for r in loud] # one summary, one warning
caplog.clear()
stream.hidden_plugins = lambda: frozenset()
with caplog.at_level(logging.DEBUG, logger=LOGGER):
assert stream.initialize()
stream.hidden_plugins = lambda: frozenset({'clock'})
assert not stream.initialize()
warnings = [r for r in caplog.records if r.levelno == logging.WARNING]
assert len(warnings) == 1 # warned again after the list had filled
def test_fetching_content_logs_nothing_at_info(caplog):
stream = _stream({'clock': _Plugin()})
with caplog.at_level(logging.DEBUG, logger=LOGGER):
+8 -5
View File
@@ -529,13 +529,16 @@ class TestDisplayAPI:
mock_cache_manager = api_v3.cache_manager = MagicMock()
with patch('web_interface.blueprints.api_v3.display.control_client.on_demand_stop',
side_effect=lambda request_id, **kw: {'accepted': True}) as stop:
response = client.post('/api/v3/display/on-demand/stop')
# May return 200 if successful or 500 on error
assert response.status_code in [200, 500]
# Verify stop request was set in cache if successful
if response.status_code == 200:
assert mock_cache_manager.set.called
assert response.status_code == 200
assert response.get_json()['data']['transport'] == 'socket'
stop.assert_called_once()
# The request goes over the control socket; nothing is written to
# the cache (the file mailbox is gone).
mock_cache_manager.set.assert_not_called()
class TestPluginsAPI:
@@ -81,7 +81,6 @@ REMOVED_CATCH_ALLS = [
("GET", "/api/v3/config/main", None),
("GET", "/api/v3/config/secrets", None),
("GET", "/api/v3/display/modes", None),
("POST", "/api/v3/display/on-demand/stop", {}),
("GET", "/api/v3/cache/list", None),
("GET", "/api/v3/plugins/installed", None),
("GET", "/api/v3/plugins/health", None),
+8 -3
View File
@@ -42,7 +42,7 @@ import time
from datetime import datetime
from pathlib import Path
from web_interface import update_channel
from web_interface import git_index_lock, update_channel
logger = logging.getLogger(__name__)
@@ -302,10 +302,12 @@ def local_changes(project_dir, run=None, timeout=30):
Returns the changed paths, or None when git could not tell. Raises what
``run`` raises (e.g. ``subprocess.TimeoutExpired``).
"""
run = run or subprocess.run
# A status that times out must not leave index.lock: don't refresh the index.
run = run or git_index_lock.run_git
result = run(['git', '-c', 'core.fileMode=false', 'status', '--porcelain',
'--untracked-files=no', '-z'],
cwd=str(project_dir), capture_output=True, text=True, timeout=timeout)
cwd=str(project_dir), capture_output=True, text=True, timeout=timeout,
env=git_index_lock.read_only_env())
if result.returncode != 0:
return None
changed = []
@@ -615,6 +617,9 @@ class AutoUpdater:
if core.get('status') != 'success':
message = core.get('message') or 'The LEDMatrix update failed.'
if new_head and new_head != old_head:
# A pull killed by its timeout leaves .git/index.lock behind,
# and the rollback's reset would fail on it.
git_index_lock.clear_stale_index_lock(self.project_root)
ok, detail = rollback()
message += (' The partial update was rolled back.' if ok
else f' Rolling back the partial update also failed: {detail}.')
+3 -1
View File
@@ -35,6 +35,7 @@ from typing import Dict, Any, Optional, Tuple, Type
from urllib.parse import urlparse, urlunparse
logger = logging.getLogger(__name__)
# Import new infrastructure
from web_interface import git_index_lock
from src.web_interface.api_helpers import (success_response, error_response,
exception_error_response, validate_request_json)
from src.web_interface.errors import ErrorCode
@@ -558,7 +559,8 @@ def get_git_version(project_dir=None):
capture_output=True,
text=True,
timeout=5,
cwd=str(project_dir)
cwd=str(project_dir),
env=git_index_lock.read_only_env()
)
if result.returncode == 0:
+114 -59
View File
@@ -224,43 +224,13 @@ def get_refresh_rate():
}})
@api_v3.route('/config/schedule', methods=['GET'])
def get_schedule_config():
"""Get current schedule configuration"""
try:
if not api_v3.config_manager:
return error_response(
ErrorCode.CONFIG_LOAD_FAILED,
'Config manager not initialized',
status_code=500
)
config = api_v3.config_manager.load_config()
schedule_config = config.get('schedule', {})
return success_response(data=schedule_config)
except Exception as e:
logger.error("%s failed", request.path, exc_info=True)
return error_response(
ErrorCode.CONFIG_LOAD_FAILED,
"An error occurred; see logs for details",
details=describe_exception(e),
status_code=500
)
@api_v3.route('/config/schedule', methods=['POST'])
def save_schedule_config():
"""Save schedule configuration"""
try:
if not api_v3.config_manager:
return jsonify({'status': 'error', 'message': 'Config manager not initialized'}), 500
data = request.get_json(silent=True)
if not data:
return jsonify({'status': 'error', 'message': 'No data provided'}), 400
# Load current config
current_config = api_v3.config_manager.load_config()
def _build_schedule(data):
"""``(schedule, None)`` from a schedule picker's POST body, or ``(None, message)``.
The display schedule and each plugin schedule (``plugin_schedules``)
share it. Times are validated; a per-day schedule that is on needs a day
that is on.
"""
# Build schedule configuration
# Handle enabled checkbox - can be True, False, or 'on'
enabled_value = data.get('enabled', False)
@@ -281,19 +251,11 @@ def save_schedule_config():
# Validate time formats
is_valid, error_msg = _validate_time_format(start_time)
if not is_valid:
return error_response(
ErrorCode.VALIDATION_ERROR,
error_msg,
status_code=400
)
return None, error_msg
is_valid, error_msg = _validate_time_format(end_time)
if not is_valid:
return error_response(
ErrorCode.VALIDATION_ERROR,
error_msg,
status_code=400
)
return None, error_msg
schedule_config['start_time'] = start_time
schedule_config['end_time'] = end_time
@@ -338,19 +300,11 @@ def save_schedule_config():
# Validate time formats
is_valid, error_msg = _validate_time_format(start_time)
if not is_valid:
return error_response(
ErrorCode.VALIDATION_ERROR,
f"Invalid start time for {day}: {error_msg}",
status_code=400
)
return None, f"Invalid start time for {day}: {error_msg}"
is_valid, error_msg = _validate_time_format(end_time)
if not is_valid:
return error_response(
ErrorCode.VALIDATION_ERROR,
f"Invalid end time for {day}: {error_msg}",
status_code=400
)
return None, f"Invalid end time for {day}: {error_msg}"
day_config['start_time'] = start_time
day_config['end_time'] = end_time
@@ -364,12 +318,52 @@ def save_schedule_config():
# config.template.json ships, so refusing it meant a fresh
# install could not post back the schedule GET returned.
if enabled_days_count == 0 and enabled_value:
return None, "At least one day must be enabled in per-day schedule mode"
return schedule_config, None
@api_v3.route('/config/schedule', methods=['GET'])
def get_schedule_config():
"""Get current schedule configuration"""
try:
if not api_v3.config_manager:
return error_response(
ErrorCode.VALIDATION_ERROR,
"At least one day must be enabled in per-day schedule mode",
status_code=400
ErrorCode.CONFIG_LOAD_FAILED,
'Config manager not initialized',
status_code=500
)
config = api_v3.config_manager.load_config()
schedule_config = config.get('schedule', {})
return success_response(data=schedule_config)
except Exception as e:
logger.error("%s failed", request.path, exc_info=True)
return error_response(
ErrorCode.CONFIG_LOAD_FAILED,
"An error occurred; see logs for details",
details=describe_exception(e),
status_code=500
)
@api_v3.route('/config/schedule', methods=['POST'])
def save_schedule_config():
"""Save schedule configuration"""
try:
if not api_v3.config_manager:
return jsonify({'status': 'error', 'message': 'Config manager not initialized'}), 500
data = request.get_json(silent=True)
if not data:
return jsonify({'status': 'error', 'message': 'No data provided'}), 400
# Load current config
current_config = api_v3.config_manager.load_config()
schedule_config, error_msg = _build_schedule(data)
if error_msg:
return error_response(ErrorCode.VALIDATION_ERROR, error_msg, status_code=400)
# Update and save config using atomic save
current_config['schedule'] = schedule_config
success, error_msg = _pkg._save_config_atomic(api_v3.config_manager, current_config, create_backup=True)
@@ -391,6 +385,67 @@ def save_schedule_config():
status_code=500, details=describe_exception(e)
)
@api_v3.route('/config/plugin-schedules', methods=['GET'])
def get_plugin_schedules():
"""Every plugin's schedule (config['plugin_schedules']), by plugin id."""
if not api_v3.config_manager:
return error_response(ErrorCode.CONFIG_LOAD_FAILED,
'Config manager not initialized', status_code=500)
config = api_v3.config_manager.load_config()
return success_response(data=config.get('plugin_schedules') or {})
@api_v3.route('/config/plugin-schedules', methods=['POST'])
def save_plugin_schedule():
"""Save one plugin's schedule (src/plugin_schedule.py).
The body is a schedule picker's fields plus ``plugin_id`` and
``hide_when_live``. A schedule that is off and does not hide during live
games changes nothing, so it is removed rather than stored.
"""
try:
if not api_v3.config_manager:
return error_response(ErrorCode.CONFIG_LOAD_FAILED,
'Config manager not initialized', status_code=500)
data = request.get_json(silent=True)
if not isinstance(data, dict):
return error_response(ErrorCode.VALIDATION_ERROR, 'No data provided', status_code=400)
plugin_id = data.get('plugin_id')
if not isinstance(plugin_id, str) or not plugin_id:
return error_response(ErrorCode.VALIDATION_ERROR, 'plugin_id is required', status_code=400)
not_plugin = _pkg._non_plugin_id_error(plugin_id)
if not_plugin is not None:
return not_plugin
if plugin_id not in _pkg._discovered_plugin_manifests(plugin_id):
return error_response(ErrorCode.PLUGIN_NOT_FOUND,
f"Plugin '{plugin_id}' is not installed", status_code=404)
schedule, error_msg = _build_schedule(data)
if error_msg:
return error_response(ErrorCode.VALIDATION_ERROR, error_msg, status_code=400)
schedule['hide_when_live'] = _coerce_to_bool(data.get('hide_when_live'))
current_config = api_v3.config_manager.load_config()
schedules = dict(current_config.get('plugin_schedules') or {})
if schedule['enabled'] or schedule['hide_when_live']:
schedules[plugin_id] = schedule
else:
schedules.pop(plugin_id, None)
current_config['plugin_schedules'] = schedules
success, error_msg = _pkg._save_config_atomic(api_v3.config_manager, current_config,
create_backup=True)
if not success:
return error_response(ErrorCode.CONFIG_SAVE_FAILED,
f"Failed to save plugin schedule: {error_msg}", status_code=500)
invalidate_cache()
return success_response(message=f"Schedule for {plugin_id} saved")
except Exception as e:
logger.error("Error saving plugin schedule", exc_info=True)
return error_response(ErrorCode.CONFIG_SAVE_FAILED,
"An error occurred; see logs for details",
status_code=500, details=describe_exception(e))
@api_v3.route('/config/dim-schedule', methods=['GET'])
def get_dim_schedule_config():
"""Get current dim schedule configuration"""
+202 -130
View File
@@ -9,7 +9,7 @@ from web_interface.blueprints.api_v3 import (
_get_display_service_status, _socket_reason_code, _stop_display_service, api_v3,
jsonify, logger, request, uuid,
)
from web_interface import display_preview, display_state
from web_interface import display_preview, display_state, on_demand_dispatch
import web_interface.blueprints.api_v3 as _pkg
from src.ipc import client as control_client
# Read through the module rather than bound by value: tests patch these
@@ -33,87 +33,97 @@ def _cache_manager():
class _NotDelivered(Exception):
"""The display took an on-demand request over the socket and did not
accept it (``busy``, ``invalid_args``, ...) or did not answer in time."""
def __init__(self, reason):
super().__init__(reason)
self.reason = reason
#: How long a start is sent again to a service that systemd reports running
#: but that has no socket yet (a display still loading its plugins, or one
#: someone else just restarted). A cold start gets the dispatcher's own
#: START_WAIT_SECONDS. Either way the route answers at once (202) and the
#: web process's dispatcher does the waiting.
ON_DEMAND_SOCKET_WAIT_RUNNING_SECONDS = 10.0
def _deliver_on_demand(payload):
"""Hand an on-demand request to the display: control socket, else mailbox.
def _dispatcher():
"""The web process's on-demand dispatcher (web_interface/on_demand_dispatch.py)."""
return on_demand_dispatch.get_dispatcher(_send_on_demand)
The socket (src/ipc) answers with an acknowledgement as soon as the
display has the command queued for its render thread. The file mailbox
is written only when the socket could not carry the request at all
(``control_client.should_fall_back``): no socket (the display is stopped
or predates it), a refused connection, or a display too old to know the
command. The display reads it within its mailbox poll interval.
A display that had the request and refused it, or did not answer in
time, raises :class:`_NotDelivered`: a mailbox copy would be refused
the same way, or hide a stuck display behind a "success".
def _pending_start_state():
"""A start the dispatcher is still delivering, one it delivered but the
display has not acted on yet, or one it gave up on: what the status
routes report instead of the display's own state (see _shadows). None
when there is none.
Returns ``(transport, socket_error)``: ``'socket'`` and None, or
``'mailbox'`` and the socket failure's reason code.
A delivered start reads as ``status: "starting"`` with ``delivered:
true`` for at most DELIVERED_SHOWN_SECONDS: the display acknowledges it
when its socket opens and acts on it seconds later, and until then
publishes its own idle state, which would flash in the UI.
"""
dispatcher = on_demand_dispatch.current()
status = dispatcher.status() if dispatcher is not None else None
if status is None:
return None
if status.get('status') == 'delivered':
delivered_at = status.get('last_updated') or 0
if _pkg.time.time() - delivered_at > on_demand_dispatch.DELIVERED_SHOWN_SECONDS:
return None
return dict(status, status='starting', delivered=True, delivered_at=delivered_at)
if status.get('status') not in ('starting', 'error'):
return None
return status
def _display_on_demand_state(snapshot):
"""The display's own on-demand state: (state, source). From the socket's
snapshot, else the cache key it also writes; state None when neither."""
state = display_state.on_demand_state(snapshot)
if state is not None:
return state, 'socket'
# memory_ttl=0: the display service writes this key, so only the file
# is current. This process's memory tier would keep serving the first
# copy it read for the full max_age -- "active" for two minutes after
# the display had already stopped.
return _cache_manager().get('display_on_demand_state', max_age=120, memory_ttl=0), 'cache'
def _send_on_demand(payload):
"""Hand an on-demand request to the display over the control socket.
Returns the display's acknowledgement: it has the command queued for
its render thread. Raises ``control_client.ControlError`` when it did
not take it; there is no other way to reach the display (the file
mailbox ``display_on_demand_request`` is gone).
"""
try:
if payload['action'] == 'start':
control_client.on_demand_start(
return control_client.on_demand_start(
payload['request_id'], payload.get('plugin_id'), payload.get('mode'),
payload.get('duration'), bool(payload.get('pinned', False)))
else:
control_client.on_demand_stop(payload['request_id'])
return 'socket', None
except control_client.ControlError as e:
reason = _socket_reason_code(e.reason)
if not control_client.should_fall_back(e):
logger.warning("The display did not accept on-demand %s %s (%s)",
payload['action'], payload['request_id'], e)
raise _NotDelivered(reason) from None
if reason in _QUIET_SOCKET_REASONS:
logger.debug("On-demand %s via the mailbox: %s", payload['action'], e)
else:
logger.warning("Control socket did not take on-demand %s (%s); "
"using the mailbox", payload['action'], e)
except Exception: # never let the socket path break the route
logger.exception("Control socket client failed; using the mailbox")
reason = 'internal'
_cache_manager().set('display_on_demand_request', payload)
return 'mailbox', reason
return control_client.on_demand_stop(payload['request_id'])
def _not_delivered_response(request_id, action, reason):
"""The answer when the display had the request and did not accept it."""
def _socket_error_response(request_id, action, reason, message=None, **extra):
"""The answer when the display did not take an on-demand request: ``400``
for arguments it refused, else ``503``."""
status = 400 if reason == 'invalid_args' else 503
data = {'request_id': request_id, 'transport': 'socket', 'socket_error': reason}
data.update(extra)
return jsonify({
'status': 'error',
'message': (f'The display service did not accept the on-demand {action} '
'message': message or (f'The display service did not accept the on-demand {action} '
f'request ({reason})'),
'data': {'request_id': request_id, 'transport': 'socket', 'socket_error': reason},
'data': data,
}), status
def _withdraw_on_demand(request_id):
"""Take a start request the route has refused back out of the mailbox.
The display reads the mailbox for an hour without looking at a
request's age, so one left there after an error answer ran whenever the
display next started. Only this request is removed: the mailbox is
re-read and cleared only while it still holds this request_id, as the
display's _consume_on_demand_request does, so a newer request posted in
the meantime stays for the display to take.
"""
cache = _cache_manager()
try:
current = cache.get('display_on_demand_request', max_age=3600, memory_ttl=0)
if isinstance(current, dict) and current.get('request_id') == request_id:
cache.delete('display_on_demand_request')
except Exception: # the route is answering an error already
logger.warning("Could not withdraw on-demand request %s from the mailbox",
request_id, exc_info=True)
def _socket_failure_reason(error):
"""The reportable reason code for an exception from _send_on_demand, logged."""
if isinstance(error, control_client.ControlError):
reason = _socket_reason_code(error.reason)
if reason in _QUIET_SOCKET_REASONS:
logger.debug("On-demand request not taken over the control socket: %s", error)
else:
logger.warning("On-demand request not taken over the control socket: %s", error)
return reason
logger.error("Control socket client failed", exc_info=error)
return 'internal'
@api_v3.route('/display/current', methods=['GET'])
@@ -218,16 +228,13 @@ def get_on_demand_status():
available (``source: "socket"``), else the cache key it also writes
(``source: "cache"``).
"""
state = display_state.on_demand_state(display_state.read_state())
source = 'socket'
if state is None:
source = 'cache'
cache = _cache_manager()
# memory_ttl=0: the display service writes this key, so only the file
# is current. This process's memory tier would keep serving the first
# copy it read for the full max_age -- "active" for two minutes after
# the display had already stopped.
state = cache.get('display_on_demand_state', max_age=120, memory_ttl=0)
state, source = _display_on_demand_state(display_state.read_state())
pending = _pending_start_state()
if pending is not None and _shadows(pending, state):
# A start the web process is still delivering, has delivered but
# the display has not answered yet, or gave up on (start-timeout):
# newer than anything the display has said.
state, source = pending, 'web'
if state is None:
state = {
'active': False,
@@ -243,6 +250,31 @@ def get_on_demand_status():
'source': source,
}
})
def _shadows(pending, state):
"""Whether the web process's start (or its failure) is newer than the
display's on-demand ``state``.
* Still being sent: always.
* Delivered: until the display publishes the state that answers it.
A display that names its request (``request_id``) answers when the
id matches; its startup state, published as the socket opens and so
possibly after the acknowledgement, names no request or an older one
and does not count. An older display without the field answers with
any state published after the delivery.
* Failed: until the display publishes something later.
"""
if not isinstance(state, dict):
return True
if pending.get('status') == 'starting' and not pending.get('delivered'):
return True
if pending.get('delivered') and 'request_id' in state:
return state.get('request_id') != pending.get('request_id')
shown = state.get('last_updated')
if not isinstance(shown, (int, float)) or isinstance(shown, bool):
return True
return shown < (pending.get('last_updated') or 0)
@api_v3.route('/display/on-demand/start', methods=['POST'])
def start_on_demand_display():
"""Request the display controller to run a specific plugin on-demand."""
@@ -300,11 +332,10 @@ def start_on_demand_display():
resolved_plugin,
)
# Deliver the request over the control socket, or post it to the
# mailbox the display process polls (DisplayController.
# _poll_on_demand_requests). Done before any service start: a stopped
# display has no socket, so the request lands in the mailbox, where a
# freshly started display finds it on its first poll.
# The request goes over the control socket, the only way to reach the
# display. A display that is not listening (stopped, or still starting)
# is started when start_service asks for it, and the request is sent
# again once its socket is up.
request_id = data.get('request_id') or str(uuid.uuid4())
request_payload = {
'request_id': request_id,
@@ -315,69 +346,85 @@ def start_on_demand_display():
'pinned': pinned,
'timestamp': _pkg.time.time()
}
# This start supersedes one the dispatcher is still delivering,
# whatever becomes of it: an older request must not land after it.
dispatcher = on_demand_dispatch.current()
if dispatcher is not None:
dispatcher.cancel('superseded')
try:
transport, socket_error = _deliver_on_demand(request_payload)
except _NotDelivered as e:
return _not_delivered_response(request_id, 'start', e.reason)
_send_on_demand(request_payload)
except Exception as e: # pylint: disable=broad-except
error = e
else:
# A socket acknowledgement is the display itself answering: it is
# running and has the request queued, whatever systemd says (a display
# run by hand or in the emulator has no active unit). So nothing is
# checked or started for it -- that answered "not running" for a request
# that had already taken effect. The service is still reported the way
# _ensure_display_service_running reports a running one.
if transport == 'socket':
# running and has the request queued, whatever systemd says (a
# display run by hand or in the emulator has no active unit). So
# nothing is checked or started for it. The service is still
# reported the way _ensure_display_service_running reports a
# running one.
service_result = (dict(_get_display_service_status(), started=False)
if start_service else None)
return _on_demand_started(request_id, resolved_plugin, resolved_mode,
duration, pinned, service_result, transport,
socket_error)
duration, pinned, service_result)
reason = _socket_failure_reason(error)
if not control_client.display_not_listening(error):
# The display had it and refused it, is too old for the command, or
# this web process cannot use the socket at all (switched off, no
# Unix sockets): starting a service would not change that.
return _socket_error_response(request_id, 'start', reason)
service_status = _get_display_service_status()
if not service_status.get('active') and not start_service:
# The request is in the mailbox, and the display reads it whenever
# it next starts: taken back out, or a request answered with this
# error ran later anyway.
_withdraw_on_demand(request_id)
return jsonify({
'status': 'error',
'message': 'Display service is not running. Please start the display service or enable "Start Service" option.',
'service_status': service_status
'service_status': service_status,
'data': {'request_id': request_id, 'transport': 'socket', 'socket_error': reason},
}), 400
# start_service means "start it if it is not running", as the UI's
# checkbox says; _ensure_display_service_running leaves a running service
# alone. This used to stop a running service, sleep 1.5s and start it
# again, so every on-demand or "Preview on display" click -- and every
# MQTT on-demand command, which posts here with the default -- cold-
# restarted the display process: every plugin reloaded and the panel was
# blank for seconds. The restart bought nothing. The running process
# looks at this mailbox at least once a second, from its dwell sleep,
# its render loops and Vegas's interrupt check as well as the main loop
# (DisplayController._mailbox_poll_interval), and a restarted one got
# the request the same way: the
# startup path only restores a session the display itself saved
# (display_on_demand_config), so it loaded nothing it would not have had.
# checkbox says; _ensure_display_service_running leaves a running
# service alone (restarting it cost seconds of blank panel for nothing).
# Either way the display has no socket yet. The route does not wait for
# it -- a cold start can outlast a client's timeout (the MQTT bridge's is
# 15 s) -- but answers 202 and leaves the sending to the dispatcher,
# whose outcome the status routes report.
wait = ON_DEMAND_SOCKET_WAIT_RUNNING_SECONDS
service_result = None
if start_service:
if not service_status.get('active'):
service_result = _ensure_display_service_running()
# Check if service actually started
if service_result and not service_result.get('active'):
_withdraw_on_demand(request_id)
return jsonify({
'status': 'error',
'message': 'Failed to start display service. Please check service logs or start it manually.',
'service_result': service_result
}), 500
wait = on_demand_dispatch.START_WAIT_SECONDS
elif start_service:
service_result = dict(service_status, started=False)
return _on_demand_started(request_id, resolved_plugin, resolved_mode,
duration, pinned, service_result, transport,
socket_error)
_dispatcher().submit(request_payload, wait_seconds=wait)
return jsonify({
'status': 'starting',
'message': ('The display service is starting; the request is sent to it as soon '
'as it is listening. Check the on-demand status for the outcome.'),
'data': {
'request_id': request_id,
'plugin_id': resolved_plugin,
'mode': resolved_mode,
'duration': duration,
'pinned': pinned,
'service': service_result,
'transport': 'socket',
'socket_error': reason,
'pending': True,
'wait_seconds': wait,
},
}), 202
def _on_demand_started(request_id, plugin_id, mode, duration, pinned,
service_result, transport, socket_error):
def _on_demand_started(request_id, plugin_id, mode, duration, pinned, service_result):
"""The success answer of /display/on-demand/start."""
response_data = {
'request_id': request_id,
@@ -386,10 +433,8 @@ def _on_demand_started(request_id, plugin_id, mode, duration, pinned,
'duration': duration,
'pinned': pinned,
'service': service_result,
'transport': transport,
'transport': 'socket',
}
if socket_error:
response_data['socket_error'] = socket_error
return jsonify({'status': 'success', 'data': response_data})
@api_v3.route('/display/on-demand/stop', methods=['POST'])
def stop_on_demand_display():
@@ -398,23 +443,38 @@ def stop_on_demand_display():
# _coerce_to_bool: bool("false") is True, which stopped the service.
stop_service = _coerce_to_bool(data.get('stop_service', False))
# The running display takes the stop over the control socket, or reads
# it from the mailbox within ON_DEMAND_POLL_INTERVAL, and resumes normal
# rotation in place (_clear_on_demand); nothing is restarted.
# The running display takes the stop over the control socket and
# resumes normal rotation in place (_clear_on_demand); nothing is
# restarted.
request_id = data.get('request_id') or str(uuid.uuid4())
request_payload = {
'request_id': request_id,
'action': 'stop',
'timestamp': _pkg.time.time()
}
socket_error = None
# A start the web process is still delivering is dropped first: the
# stop is newer, whatever happens to it below.
dispatcher = on_demand_dispatch.current()
cancelled = dispatcher.cancel('requested-stop') if dispatcher is not None else None
try:
transport, socket_error = _deliver_on_demand(request_payload)
except _NotDelivered as e:
if not stop_service:
return _not_delivered_response(request_id, 'stop', e.reason)
_send_on_demand(request_payload)
except Exception as e: # pylint: disable=broad-except
socket_error = _socket_failure_reason(e)
if not stop_service and not (cancelled and control_client.display_not_listening(e)):
if control_client.display_not_listening(e):
service_status = _get_display_service_status()
message = ('Display service is not running, so the stop could not be '
'delivered. If it resumes an on-demand session when it starts, '
'stop it then.' if not service_status.get('active') else
f'The display service is running but its control socket did '
f'not answer ({socket_error}). It may still be starting; '
f'try again shortly.')
return _socket_error_response(request_id, 'stop', socket_error, message,
service=service_status)
return _socket_error_response(request_id, 'stop', socket_error)
# Stopping the service ends on-demand too, whatever the display did
# with the request.
transport, socket_error = 'socket', e.reason
service_result = None
if stop_service:
@@ -423,11 +483,16 @@ def stop_on_demand_display():
response_data = {
'request_id': request_id,
'service': service_result,
'transport': transport,
'transport': 'socket',
}
if cancelled:
# The start it ended never reached the display.
response_data['cancelled_request_id'] = cancelled
if socket_error:
response_data['socket_error'] = socket_error
return jsonify({'status': 'success', 'data': response_data})
@api_v3.route('/display/current-status', methods=['GET'])
def get_current_display_status():
"""Return the display mode/plugin currently intended to be shown.
@@ -457,4 +522,11 @@ def get_current_display_status():
'plugin_id': None,
'last_updated': None,
}
return jsonify({'status': 'success', 'data': dict(state, source=source)})
data = dict(state, source=source)
pending = _pending_start_state()
if pending is not None and _shadows(pending, _display_on_demand_state(snapshot)[0]):
# An on-demand start the web process is still delivering, has
# delivered but the display has not acted on yet, or gave up on:
# what the panel is about to show, or why it will not.
data['on_demand_pending'] = pending
return jsonify({'status': 'success', 'data': data})
+25 -36
View File
@@ -370,25 +370,13 @@ def _redact_error_record(record):
def _read_errors():
snapshot, clear_request = _errors.read_error_report(_errors_cache())
return snapshot, clear_request
return _errors.read_error_report(_errors_cache())
def _send_error_clear(request_id, cutoff):
"""``errors.clear`` over the control socket: the display's answer, or None
when the socket could not carry it (no socket, or a display older than
the command) and the clear goes to the mailbox instead. A display that
took the request and failed raises ControlError (no second copy)."""
client = _pkg.control_client
try:
return client.errors_clear(request_id, cutoff)
except client.ControlError as e:
if not client.should_fall_back(e):
raise
_pkg._log_socket_failure('errors.clear', e, _pkg._socket_reason_code(e.reason))
except Exception: # never let the socket path break the route
logger.exception("Control socket client failed clearing errors; using the mailbox")
return None
"""``errors.clear`` over the control socket: the display's answer.
Raises ControlError when it did not take or apply it."""
return _pkg.control_client.errors_clear(request_id, cutoff)
@api_v3.route('/errors/summary', methods=['GET'])
@@ -402,7 +390,7 @@ def get_error_summary():
until it has reported; ``generated_at`` says when it did.
"""
try:
summary = _errors.error_summary_from_report(*_read_errors())
summary = _errors.error_summary_from_report(_read_errors())
summary['recent_errors'] = [_redact_error_record(r) for r in summary['recent_errors']]
for pattern in summary['active_patterns'].values():
if isinstance(pattern, dict) and isinstance(pattern.get('sample_messages'), list):
@@ -430,7 +418,7 @@ def get_plugin_errors(plugin_id):
recorded errors is "healthy".
"""
try:
health = _errors.plugin_health_from_report(*_read_errors(), plugin_id)
health = _errors.plugin_health_from_report(_read_errors(), plugin_id)
health['last_error'] = _redact_error_record(health['last_error'])
return success_response(data=health, message="Plugin health retrieved")
except Exception as e:
@@ -449,9 +437,10 @@ def clear_old_errors():
max_age_hours: Maximum age in hours (default: 24, max: 8760 = 1 year)
all: true clears every error recorded so far (max_age_hours ignored)
The errors live in the display service, so this records a clear request
that it applies within a few seconds. Reads hide the cleared errors from
the moment the request is recorded.
The errors live in the display service, so the clear is sent to it over
the control socket, and it has applied it when this answers. Without a
running display it fails with 503: the errors shown are then the last
run's, and the display's next run starts with none.
"""
try:
data = request.get_json(silent=True) or {}
@@ -492,28 +481,28 @@ def clear_old_errors():
send=_send_error_clear)
except _pkg.control_client.ControlError as e:
reason = _pkg._socket_reason_code(e.reason)
logger.warning("The display did not apply the error clear: %s", e)
_pkg._log_socket_failure('errors.clear', e, reason)
if _pkg.control_client.display_not_listening(e):
message = ("The display service is not running (or is still starting), "
"so the errors could not be cleared. The errors shown are "
"from its last run; it starts again with none.")
elif reason in _pkg.control_client.UPGRADE_REASONS:
message = ("The running display service is too old to clear errors; "
"restart it to pick up the installed version")
elif reason in _pkg._QUIET_SOCKET_REASONS:
message = ("The control socket is not available here, so the errors "
"could not be cleared")
else:
message = "The display service did not apply the clear"
return error_response(
error_code=ErrorCode.SYSTEM_ERROR,
message="The display service did not apply the clear",
message=message,
context={'socket_error': reason},
status_code=503
)
except OSError as e:
logger.error("Could not record an error clear request: %s", e)
return error_response(
error_code=ErrorCode.SYSTEM_ERROR,
message="Could not record the clear request in the shared cache",
status_code=500
)
scope = "all errors" if clear_all else f"errors older than {max_age_hours} hours"
if result.get('applied'):
message = f"Cleared {scope}"
else:
message = (f"Clear of {scope} requested; the display service applies it "
f"within about {int(_errors.SNAPSHOT_TICK_INTERVAL)} seconds")
return success_response(data=result, message=message)
return success_response(data=result, message=f"Cleared {scope}")
except Exception as e:
logger.error(f"Error clearing old errors: {e}", exc_info=True)
return error_response(
+47 -4
View File
@@ -14,6 +14,7 @@ from web_interface.blueprints.api_v3 import (
)
import threading
from web_interface import git_index_lock
from web_interface.cache import get_cached, set_cached
from web_interface.system_metrics import collect_system_metrics, format_uptime
import web_interface.blueprints.api_v3 as _pkg
@@ -163,6 +164,7 @@ def check_for_update():
['git', 'fetch', '--quiet', '--tags', '--force', update_channel.REMOTE,
update_channel.BETA_BRANCH],
capture_output=True, timeout=10, cwd=cwd,
env=git_index_lock.read_only_env(),
)
if fetch_result.returncode != 0:
stderr = fetch_result.stderr.decode(errors='replace').strip()
@@ -334,6 +336,37 @@ def _perform_core_update_locked(stash_local_changes=True):
project_dir = str(PROJECT_ROOT)
from web_interface import update_channel
# A git killed mid-command (a timed-out pull, a power cut) leaves
# .git/index.lock behind, and every later update then fails with
# "Unable to create .../index.lock: File exists". Remove it when no git
# process can still own it; refuse while one is running.
if git_index_lock.clear_stale_index_lock(project_dir) == git_index_lock.IN_USE:
return {'status': 'error', 'restart_required': False, 'dependency_failures': [],
'message': 'Another git command is running in the LEDMatrix folder; '
'try again in a minute.'}
# A pull killed mid-rebase leaves .git/rebase-merge behind, and every
# later `pull --rebase` then fails. Same rule as the automatic update's
# preflight: a rebase really in progress leaves HEAD detached and holds
# the user's conflict work, so leave it; with HEAD back on a branch it was
# abandoned, and `--quit` drops its state without touching the tree.
git_dir = Path(project_dir) / '.git'
rebase_dirs = [git_dir / name for name in ('rebase-merge', 'rebase-apply') if (git_dir / name).exists()]
if rebase_dirs:
def _git(*args):
return git_index_lock.run_git(['git', *args], capture_output=True, text=True,
timeout=60, cwd=project_dir)
if _git('symbolic-ref', '-q', 'HEAD').returncode != 0:
return {'status': 'error', 'restart_required': False, 'dependency_failures': [],
'message': 'A git rebase is in progress in the LEDMatrix folder. '
'Finish or abort it (git rebase --continue / --abort), then update again.'}
cleared = _git('rebase', '--quit')
if cleared.returncode != 0 or any(path.exists() for path in rebase_dirs):
return {'status': 'error', 'restart_required': False, 'dependency_failures': [],
'message': 'An abandoned git rebase in the LEDMatrix folder could not be cleared: '
f'{(cleared.stderr or "").strip() or "unknown error"}.'}
logger.warning("Cleared an abandoned git rebase from %s", ', '.join(str(p) for p in rebase_dirs))
# Which code to move to: the newest release (stable) or main (beta).
# See web_interface/update_channel.py; it never picks an older commit.
channel, fetch_error = channel_status(project_dir)
@@ -425,25 +458,33 @@ def _perform_core_update_locked(stash_local_changes=True):
# Stable checks out the release tag, carrying edits across the way the
# pull's --autostash does.
channel_note = ''
try:
if action == update_channel.ACTION_CHECKOUT_TAG:
result, autostash_note = update_channel.checkout_release(project_dir, channel.newest_release)
channel_note = f"Now on release {channel.newest_release} (stable channel). {autostash_note}".strip()
elif action == update_channel.ACTION_SWITCH_TO_BETA:
result, autostash_note = update_channel.checkout_beta_branch(project_dir)
if result.returncode == 0:
result = subprocess.run(['git', 'pull', '--rebase', '--autostash'],
capture_output=True, text=True, timeout=60, cwd=project_dir)
result = git_index_lock.run_git(['git', 'pull', '--rebase', '--autostash'],
capture_output=True, text=True,
timeout=git_index_lock.MUTATING_TIMEOUT_SECONDS, cwd=project_dir)
channel_note = f"Now following {update_channel.BETA_BRANCH} (beta channel). {autostash_note}".strip()
else:
result = subprocess.run(
result = git_index_lock.run_git(
pull_args,
capture_output=True,
text=True,
timeout=60,
timeout=git_index_lock.MUTATING_TIMEOUT_SECONDS,
cwd=project_dir
)
if channel.waiting and channel.newest_release:
channel_note = channel.message
except subprocess.TimeoutExpired:
# run_git has stopped git and removed the lock it left.
logger.warning("git timed out during Update Code")
return {'status': 'error', 'restart_required': False, 'dependency_failures': [],
'message': 'Update failed: git took too long (a slow network or SD card). '
'The partial update was cleaned up; try again.'}
# Give the branch tracking information so the next pull is a plain
# `git pull` — otherwise every update repeats the fallback.
@@ -554,6 +595,8 @@ def _perform_core_update_locked(stash_local_changes=True):
detail = next((ln.strip() for ln in (result.stderr or '').splitlines()
if ln.strip()), '')
pull_message = f"Update failed: {detail}" if detail else "Update failed; check logs for details"
if 'index.lock' in (result.stderr or ''):
pull_message = f"{pull_message} {git_index_lock.manual_fix_hint(project_dir)}"
# Nothing here restarts anything: the pull replaces files on
# disk while the display and web services keep running the code
+22 -1
View File
@@ -615,7 +615,28 @@ def _load_schedule_partial():
return render_template('v3/partials/schedule.html',
schedule_config=schedule_config,
dim_schedule_config=dim_schedule_config,
normal_brightness=normal_brightness)
normal_brightness=normal_brightness,
plugin_schedule_rows=_plugin_schedule_rows(main_config))
def _plugin_schedule_rows(main_config):
"""One row per enabled plugin, or plugin with a saved schedule, for the
Plugin Schedules card: ``{id, name, schedule}``, sorted by name."""
schedules = main_config.get('plugin_schedules') or {}
if not pages_v3.plugin_catalog:
return []
try:
pages_v3.plugin_catalog.discover_plugins()
infos = pages_v3.plugin_catalog.get_all_plugin_info()
except Exception:
logger.warning("schedule: could not enumerate plugins", exc_info=True)
return []
rows = [{'id': info['id'], 'name': info.get('name') or info['id'],
'schedule': schedules.get(info['id']) or {}}
for info in infos
if info.get('id') and ((main_config.get(info['id']) or {}).get('enabled', False)
or info['id'] in schedules)]
return sorted(rows, key=lambda r: r['name'].lower())
def _load_plugins_partial():
+220
View File
@@ -0,0 +1,220 @@
"""Clear a stale ``.git/index.lock`` before Update Code touches the checkout.
git creates ``.git/index.lock`` while it rewrites the index and removes it when
it finishes, or when it is stopped with SIGTERM/SIGINT. A git that dies any
other way leaves the file behind: ``subprocess.run(..., timeout=...)`` SIGKILLs
its child when the timeout expires (a slow pull or checkout on a Pi), and a
power cut or an OOM kill does the same. From then on every git command that
writes the index fails with::
fatal: Unable to create '.../.git/index.lock': File exists.
and Update Code can never succeed again until someone removes it over SSH.
``clear_stale_index_lock`` removes the file only when it is provably stale: no
git process is running in this checkout, and the file is old enough that no
git can have just created it. When it cannot tell (no ``/proc``), it leaves
the file alone. Standard library only.
"""
import logging
import os
import signal
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
import time
from pathlib import Path
logger = logging.getLogger(__name__)
#: A lock younger than this is left alone even when no git process is seen,
#: so a git that started between the process scan and the removal keeps it.
MIN_STALE_AGE_SECONDS = 5
#: ``clear_stale_index_lock`` outcomes.
ABSENT = 'absent' # no lock file: nothing to do
REMOVED = 'removed' # it was stale and has been removed
IN_USE = 'in_use' # a git process is running here (or the lock is brand new)
UNKNOWN = 'unknown' # could not tell whether git is running; left alone
#: How long a timed-out git gets to exit after SIGTERM (which lets it remove
#: its own lock) before the whole process group is SIGKILLed.
TERM_GRACE_SECONDS = 5
#: Timeout for commands that rewrite the checkout (pull, checkout, reset,
#: stash). A Pi on an SD card can take minutes; killing git is what leaves the
#: lock behind, so this is generous.
MUTATING_TIMEOUT_SECONDS = 600
_PROC = Path('/proc')
def index_lock_path(project_dir):
"""``<project_dir>/.git/index.lock``, or None when ``.git`` is not a directory."""
git_dir = Path(project_dir) / '.git'
return git_dir / 'index.lock' if git_dir.is_dir() else None
def _is_within(path, root):
try:
Path(path).relative_to(root)
return True
except ValueError:
return False
def git_running_in(project_dir, proc_root=_PROC):
"""True when a git process may be working in ``project_dir``.
Scans ``/proc`` for processes named ``git`` or ``git-*`` (``git-remote-https``
and friends) whose working directory is inside the checkout, or whose
arguments name a path in it (``git -C ...``, ``--git-dir=...``). A git process
whose working directory cannot be read (another user's) counts as running
here: it might be. Returns None when ``/proc`` is unavailable.
"""
proc_root = Path(proc_root)
if not proc_root.is_dir():
return None
root = Path(project_dir).resolve()
try:
entries = list(proc_root.iterdir())
except OSError:
return None
for entry in entries:
if not entry.name.isdigit():
continue
try:
comm = (entry / 'comm').read_text().strip()
except OSError:
continue # exited meanwhile
if comm != 'git' and not comm.startswith('git-'):
continue
try:
cwd = os.readlink(entry / 'cwd')
except FileNotFoundError:
continue # exited meanwhile
except OSError:
return True
if _is_within(cwd, root) or _names_checkout(entry, cwd, root):
return True
return False
def _names_checkout(entry, cwd, root):
"""True when the process's arguments name a path in the checkout
(``-C``, ``--git-dir``, ``--work-tree``, absolute or relative to its
working directory): git can work on it from anywhere. Unreadable
arguments count as naming it."""
try:
args = os.fsdecode((entry / 'cmdline').read_bytes()).split('\0')
except FileNotFoundError:
return False # exited meanwhile
except OSError:
return True
for arg in args[1:]:
value = arg.split('=', 1)[1] if arg.startswith('--') and '=' in arg else arg
if value and _is_within(os.path.normpath(os.path.join(cwd, value)), root):
return True
return False
def clear_stale_index_lock(project_dir, proc_root=_PROC, now=None):
"""Remove ``.git/index.lock`` when no git process can still own it.
Returns one of ABSENT, REMOVED, IN_USE or UNKNOWN.
"""
lock = index_lock_path(project_dir)
if lock is None:
return ABSENT
try:
mtime = lock.stat().st_mtime
except FileNotFoundError:
return ABSENT
except OSError:
return UNKNOWN
running = git_running_in(project_dir, proc_root)
if running is None:
return UNKNOWN
age = (time.time() if now is None else now) - mtime
if running or age < MIN_STALE_AGE_SECONDS:
return IN_USE
try:
lock.unlink()
except FileNotFoundError:
return ABSENT # its owner finished after all
except OSError as exc:
logger.warning("Could not remove stale git lock %s: %s", lock, exc)
return UNKNOWN
logger.warning("Removed stale git lock %s (%.0fs old, no git process running); "
"a previous git command was killed before it finished", lock, age)
return REMOVED
def _stop_group(proc):
"""SIGTERM the process group, then SIGKILL whatever is left of it."""
try:
os.killpg(proc.pid, signal.SIGTERM)
except (ProcessLookupError, PermissionError):
pass
try:
proc.wait(timeout=TERM_GRACE_SECONDS)
except subprocess.TimeoutExpired:
pass
# Also reaps git's children (git-remote-https, rebase helpers) that
# outlive a killed parent.
try:
os.killpg(proc.pid, signal.SIGKILL)
except (ProcessLookupError, PermissionError):
pass
def _clear_after_stop(project_dir):
"""Remove the lock a stopped git left, waiting out the freshness window.
A lock created within ``MIN_STALE_AGE_SECONDS`` of the stop reads as IN_USE
on age alone; with no git process left it only needs to age, so wait for
that rather than leave it to block the next attempt.
"""
deadline = time.monotonic() + MIN_STALE_AGE_SECONDS + 1
while clear_stale_index_lock(project_dir) == IN_USE and git_running_in(project_dir) is False:
if time.monotonic() >= deadline:
break
time.sleep(0.5)
def run_git(cmd, *, cwd=None, timeout=None, capture_output=True, text=True, **kwargs):
"""``subprocess.run`` for git, without leaving ``.git/index.lock`` behind.
``subprocess.run`` SIGKILLs only the direct child when ``timeout`` expires,
which skips git's cleanup and orphans its helpers. This starts git in its
own session, stops the whole group with SIGTERM before SIGKILL, removes a
lock nothing owns any more, then raises ``TimeoutExpired`` as
``subprocess.run`` would.
"""
if capture_output:
kwargs['stdout'] = kwargs['stderr'] = subprocess.PIPE
with subprocess.Popen(cmd, cwd=cwd, text=text, start_new_session=True, **kwargs) as proc: # nosec B603 - list-form argv # nosemgrep
try:
stdout, stderr = proc.communicate(timeout=timeout)
except subprocess.TimeoutExpired as exc:
_stop_group(proc)
exc.stdout, exc.stderr = proc.communicate()
if cwd is not None:
_clear_after_stop(cwd)
raise
except BaseException:
_stop_group(proc)
raise
return subprocess.CompletedProcess(proc.args, proc.returncode, stdout, stderr) # result object only, nothing is run # nosemgrep
def read_only_env():
"""Environment for git calls that only read: ``GIT_OPTIONAL_LOCKS=0``
stops ``status`` and ``describe --dirty`` refreshing (and so locking) the
index, so a poll that times out cannot leave a lock."""
return {**os.environ, 'GIT_OPTIONAL_LOCKS': '0'}
def manual_fix_hint(project_dir):
"""The sentence shown when a lock blocks the update and was not removed."""
lock = Path(project_dir) / '.git' / 'index.lock'
return (f"If it keeps coming back, stop the LEDMatrix services, kill any running git "
f"(ps aux | grep git), then remove it with rm -f {lock} and try again.")
+239
View File
@@ -0,0 +1,239 @@
"""Deliver an on-demand start to a display that is not listening yet.
``POST /api/v3/display/on-demand/start`` can find no display on the control
socket: the service is stopped (the route starts it) or still loading its
plugins. The socket comes up only when the display's run loop starts, which
can take longer than a client waits -- the MQTT bridge gives up after 15 s.
So the route answers at once (``202``, ``status: "starting"``) and hands
the request to the one :class:`OnDemandDispatcher` of the web process, whose
worker thread sends it again until the display acknowledges it or
:data:`START_WAIT_SECONDS` pass.
* One request at a time: a newer start replaces the pending one, and a
stop cancels it (:meth:`OnDemandDispatcher.cancel`).
* Its outcome is :meth:`OnDemandDispatcher.status`, which
``GET /display/on-demand/status`` and ``/display/current-status`` report:
``starting`` while it waits, ``delivered`` once acknowledged (the
display's own state takes over from there), or ``error`` with
``start-timeout`` or the socket's reason.
Nothing is written to disk: the file mailbox that once carried such a
request is gone (docs/IPC_CONTROL_SOCKET.md, stage 5).
"""
from __future__ import annotations
import threading
import time
from typing import Any, Callable, Dict, Optional
from src.ipc import client as control_client
from src.logging_config import get_logger
logger = get_logger(__name__)
#: How long the worker keeps sending a start before it gives up
#: (``start-timeout``). The socket comes up when the display's run loop
#: starts, after every plugin has loaded.
START_WAIT_SECONDS = 45.0
#: Gap between two sends while nothing is listening.
RETRY_INTERVAL = 0.5
#: How long a finished outcome (delivered, error, cancelled) is still
#: reported, so a client polling every few seconds sees it.
OUTCOME_SECONDS = 120.0
#: How long a delivered start is still reported as ``starting`` while the
#: display has not published the state that answers it. The display
#: acknowledges a start when its socket opens, but its run loop acts on it
#: only after its first screen is built (about 5 s on a Pi 4 in Vegas), and
#: until then it reports its own idle state. See
#: ``web_interface/blueprints/api_v3/display.py:_pending_start_state``.
DELIVERED_SHOWN_SECONDS = 30.0
#: ``send(payload)`` hands the request to the display (the route's
#: ``_send_on_demand``) and raises ``ControlError`` when it does not take it.
Sender = Callable[[Dict[str, Any]], Any]
class OnDemandDispatcher:
"""One pending on-demand start, and the worker thread that delivers it."""
def __init__(self, send: Sender, *,
wait_seconds: Optional[float] = None,
retry_interval: Optional[float] = None,
clock: Callable[[], float] = time.monotonic,
wall_clock: Callable[[], float] = time.time):
self._send = send
self.wait_seconds = START_WAIT_SECONDS if wait_seconds is None else wait_seconds
self.retry_interval = RETRY_INTERVAL if retry_interval is None else retry_interval
self._clock = clock
self._wall = wall_clock
self._lock = threading.Lock()
# Held by the worker while it reads the pending start and sends it,
# so cancel() can wait out a send already in flight: a cancelled (or
# superseded) start must not reach the display after the request
# that replaced it. Never taken while holding _lock.
self._send_lock = threading.Lock()
self._wake = threading.Event()
# Bumped by every submit and cancel: a send that started under an
# older generation does not report its result as the current one.
self._generation = 0
self._pending: Optional[Dict[str, Any]] = None
self._deadline = 0.0
self._status: Optional[Dict[str, Any]] = None
self._finished_at: Optional[float] = None
self._thread: Optional[threading.Thread] = None
# -- the routes' side ------------------------------------------------------
def submit(self, payload: Dict[str, Any], wait_seconds: Optional[float] = None) -> None:
"""Deliver ``payload`` (an on-demand start) in the background,
replacing any start still pending."""
with self._lock:
self._generation += 1
superseded = self._pending
self._pending = dict(payload)
self._deadline = self._clock() + (self.wait_seconds if wait_seconds is None
else wait_seconds)
self._status = self._describe('starting', payload)
self._finished_at = None
if self._thread is None or not self._thread.is_alive():
self._thread = threading.Thread(target=self._run, name='on-demand-dispatch',
daemon=True)
self._thread.start()
if superseded is not None:
logger.info("On-demand start %s superseded by %s before the display took it",
superseded.get('request_id'), payload.get('request_id'))
self._wake.set()
def cancel(self, reason: str = 'cancelled') -> Optional[str]:
"""Drop the pending start (a stop arrived). Returns its request id,
or None when nothing was pending."""
with self._lock:
pending = self._pending
if pending is None:
return None
self._generation += 1
self._pending = None
self._finish(self._describe('idle', pending, last_event=reason))
self._wake.set()
# A send of it may be in flight (at most the client's timeout, 1 s):
# let it finish, so whatever the caller sends next lands after it.
with self._send_lock:
pass
logger.info("On-demand start %s cancelled before the display took it (%s)",
pending.get('request_id'), reason)
return pending.get('request_id')
def status(self) -> Optional[Dict[str, Any]]:
"""The pending start's state, or its outcome for OUTCOME_SECONDS
after it finished; None otherwise. In the shape of the display's
on-demand state (``active``, ``status``, ``error``, ...), plus
``source: "web"``."""
with self._lock:
if self._status is None:
return None
if (self._finished_at is not None
and self._clock() - self._finished_at > OUTCOME_SECONDS):
return None
return dict(self._status)
def pending(self) -> bool:
with self._lock:
return self._pending is not None
# -- the worker ------------------------------------------------------------
def _describe(self, status: str, payload: Dict[str, Any], error: Optional[str] = None,
last_event: Optional[str] = None) -> Dict[str, Any]:
return {
'active': False,
'status': status,
'error': error,
'last_event': last_event,
'request_id': payload.get('request_id'),
'plugin_id': payload.get('plugin_id'),
'mode': payload.get('mode'),
'duration': payload.get('duration'),
'pinned': bool(payload.get('pinned', False)),
'last_updated': self._wall(),
'source': 'web',
}
def _finish(self, status: Dict[str, Any]) -> None:
"""Record an outcome. Caller holds _lock."""
self._status = status
self._finished_at = self._clock()
def _run(self) -> None:
while True:
with self._send_lock:
with self._lock:
payload, generation = self._pending, self._generation
deadline = self._deadline
if payload is None:
self._thread = None
return
outcome, error = self._attempt(payload)
with self._lock:
if generation != self._generation:
continue # superseded or cancelled meanwhile
if outcome == 'retry' and self._clock() + self.retry_interval > deadline:
outcome, error = 'error', 'start-timeout'
if outcome == 'delivered':
self._pending = None
self._finish(self._describe('delivered', payload,
last_event='delivered'))
elif outcome == 'error':
self._pending = None
self._finish(self._describe('error', payload, error=error))
else:
self._wake.clear()
if outcome == 'delivered':
logger.info("On-demand start %s delivered once the display was listening",
payload.get('request_id'))
elif outcome == 'error':
logger.warning("On-demand start %s not delivered: %s",
payload.get('request_id'), error)
else:
# A submit or cancel wakes the wait at once.
self._wake.wait(self.retry_interval)
def _attempt(self, payload: Dict[str, Any]):
try:
self._send(payload)
except control_client.ControlError as e:
if control_client.display_not_listening(e):
return 'retry', None
return 'error', str(e.reason)
except Exception: # pylint: disable=broad-except
logger.exception("On-demand start %s: the control socket client failed",
payload.get('request_id'))
return 'error', 'internal'
return 'delivered', None
_dispatcher: Optional[OnDemandDispatcher] = None
_dispatcher_lock = threading.Lock()
def get_dispatcher(send: Sender) -> OnDemandDispatcher:
"""The web process's dispatcher, created on first use."""
global _dispatcher
with _dispatcher_lock:
if _dispatcher is None:
_dispatcher = OnDemandDispatcher(send)
return _dispatcher
def current() -> Optional[OnDemandDispatcher]:
"""The dispatcher if one was created, without creating one."""
return _dispatcher
def reset_for_tests() -> None:
global _dispatcher
with _dispatcher_lock:
_dispatcher = None
+5 -2
View File
@@ -355,9 +355,12 @@ window.previewPluginNow = function(pluginId) {
})
.then(r => r.json())
.then(data => {
// 'starting' (202): the display service is starting and the request
// follows once it listens.
const starting = data.status === 'starting';
showNotification(data.message || ('Previewing ' + pluginId + ' for 60 seconds'),
data.status || 'success');
if (data.status === 'success') window.toggleFloatingPreview(true);
starting ? 'info' : (data.status || 'success'));
if (data.status === 'success' || starting) window.toggleFloatingPreview(true);
})
.catch(err => {
showNotification('Preview failed: ' + err.message, 'error');
+20 -1
View File
@@ -39,12 +39,26 @@ const PICKERS = [
},
];
// Each plugin's picker (the Plugin Schedules card): the window in which the
// plugin may show. Unset, it offers the whole day.
const PLUGIN_PICKER = { label: 'PluginSchedule', start: '00:00', end: '23:59', fillEmptyDays: true };
const FORMS = {
schedule_form: { saved: 'Schedule settings saved', failed: 'Error saving schedule' },
dim_schedule_form: { saved: 'Dim schedule settings saved', failed: 'Error saving dim schedule' },
};
const PLUGIN_FORM = { saved: 'Plugin schedule saved', failed: 'Error saving plugin schedule' };
/** The notification labels for a form on this page, or null. */
function formLabels(form) {
if (FORMS[form.id]) return FORMS[form.id];
return form.classList.contains('plugin-schedule-form') ? PLUGIN_FORM : null;
}
const WIDGET_OPTIONS = { showModeToggle: true, showEnableToggle: true, compactMode: false };
const PLUGIN_WIDGET_OPTIONS = Object.assign({}, WIDGET_OPTIONS, {
enableDescription: 'When enabled, this plugin only shows during the hours below.',
});
// The mounted page, for the deprecated aliases (boot.js).
let active = null;
@@ -126,6 +140,11 @@ function drawPickers(root, ctx) {
widget.render(container, { 'x-options': WIDGET_OPTIONS },
widgetValue(readConfig(root, spec.data), spec), { fieldId: spec.fieldId });
});
root.querySelectorAll('[data-plugin-schedule-config]').forEach(function(container) {
widget.render(container, { 'x-options': PLUGIN_WIDGET_OPTIONS },
widgetValue(readConfig(container, 'data-plugin-schedule-config'), PLUGIN_PICKER),
{ fieldId: container.id });
});
}
attempt();
}
@@ -135,7 +154,7 @@ export function init(root, ctx) {
root.addEventListener('htmx:afterRequest', function(event) {
const form = event.target && event.target.closest ? event.target.closest('form') : null;
const labels = form && root.contains(form) ? FORMS[form.id] : null;
const labels = form && root.contains(form) ? formLabels(form) : null;
if (labels && event.detail && event.detail.xhr) report(ctx.notify, event.detail.xhr, labels);
}, on);
@@ -188,7 +188,7 @@
class="h-4 w-4 text-blue-600 focus:ring-blue-500 border-gray-300 rounded">
<span class="ml-2 text-sm font-medium text-gray-900">Enable Schedule</span>
</label>
<p class="mt-1 text-sm text-gray-600">When enabled, the display will only operate during specified hours.</p>
<p class="mt-1 text-sm text-gray-600">${window.LEDEscape.html(xOptions.enableDescription || 'When enabled, the display will only operate during specified hours.')}</p>
</div>
`;
}
@@ -1739,6 +1739,13 @@ function submitOnDemandRequest(event) {
showNotification(`Requested on-demand mode for ${pluginName}`, 'success');
closeOnDemandModal();
setTimeout(() => loadOnDemandStatus(true), 700);
} else if (result.status === 'starting') {
// 202: the display service is starting; the web process sends
// the request once it listens. The status card shows the outcome.
const pluginName = resolvePluginDisplayName(currentOnDemandPluginId);
showNotification(`Starting the display for ${pluginName}…`, 'info');
closeOnDemandModal();
setTimeout(() => loadOnDemandStatus(true), 700);
} else {
console.error('[submitOnDemandRequest] Request failed:', result);
showNotification(result.message || 'Failed to start on-demand mode', 'error');
@@ -85,4 +85,51 @@
</div>
</form>
</div>
<!-- Plugin Schedules Section (config plugin_schedules, src/plugin_schedule.py) -->
<div class="bg-white rounded-lg shadow p-6 mt-6">
<div class="border-b border-gray-200 pb-4 mb-6">
<h2 class="text-lg font-semibold text-gray-900">Plugin Schedules</h2>
<p class="mt-1 text-sm text-gray-600">Choose when each plugin may show. Outside its schedule a plugin is skipped in the rotation and in Vegas mode; it stays loaded, and on-demand still shows it. "Hide while a game is live" skips it whenever any plugin has a live game, so on gameday you only see scores.</p>
</div>
{% if plugin_schedule_rows %}
<div class="space-y-3">
{% for row in plugin_schedule_rows %}
<details class="border border-gray-200 rounded-md">
<summary class="cursor-pointer px-4 py-3 text-sm font-medium text-gray-900">
{{ row.name }}
{% if row.schedule.enabled or row.schedule.hide_when_live %}
<span class="ml-2 text-xs text-blue-700">scheduled</span>
{% endif %}
</summary>
<form class="plugin-schedule-form px-4 pb-4 space-y-4"
hx-post="/api/v3/config/plugin-schedules"
hx-ext="json-enc"
hx-headers='{"Content-Type": "application/json"}'
hx-swap="none"
data-reports-result>
<input type="hidden" name="plugin_id" value="{{ row.id }}">
<label class="flex items-center text-sm text-gray-700">
<input type="checkbox" name="hide_when_live" class="mr-2"
{% if row.schedule.hide_when_live %}checked{% endif %}>
Hide while a game is live
</label>
<div id="plugin_schedule_picker_{{ loop.index }}"
data-plugin-schedule-config='{{ row.schedule | tojson }}'></div>
<div class="flex justify-end">
<button type="submit"
class="btn bg-blue-600 hover:bg-blue-700 text-white px-4 py-2 rounded-md">
<i class="fas fa-save mr-2"></i>
Save {{ row.name }} Schedule
</button>
</div>
</form>
</details>
{% endfor %}
</div>
{% else %}
<p class="text-sm text-gray-600">No enabled plugins.</p>
{% endif %}
</div>
</div>
+3 -1
View File
@@ -24,6 +24,8 @@ import re
import subprocess # nosec B404 - list-form argv only, no shell # nosemgrep
from pathlib import Path
from web_interface import git_index_lock
PROJECT_ROOT = Path(__file__).resolve().parent.parent
CHANNELS = ('stable', 'beta')
DEFAULT_CHANNEL = 'stable'
@@ -105,7 +107,7 @@ class ChannelStatus(dict):
def _git(project_dir, run, *args, timeout=30):
run = run or subprocess.run
run = run or git_index_lock.run_git
return run(['git', *args], cwd=str(project_dir), capture_output=True, text=True, timeout=timeout)